Module: Hasta::IdentityReducer

Defined in:
lib/hasta/identity_reducer.rb

Overview

Used by any EMR job that requires an identity reducer

Class Method Summary collapse

Class Method Details

.reduce(_, data_source, data_sink = InMemoryDataSink.new) ⇒ Object 



9
10
11
12
13
14
15
16
 
# File 'lib/hasta/identity_reducer.rb', line 9

def self.reduce(_, data_source, data_sink = InMemoryDataSink.new)
  Hasta.logger.debug "Starting Identity Reducer"
  SortedDataSource.new(data_source).each_line do |line|
    data_sink << line.rstrip
  end

  data_sink.close.tap { Hasta.logger.debug "Finished Identity Reducer" }
end