Class: Cosmos::ReducerModel

Inherits:
Object show all
Defined in:
lib/cosmos/models/reducer_model.rb

Overview

Tracks the files which are being stored in S3 for data reduction purposes. Files are stored in a Redis set by spliting their filenames and storing in a set named SCOPE__TARGET__reducer__TYPE, e.g. DEFAULT__INST__reducer__decom Where TYPE can be ‘decom’, ‘minute’, or ‘hour’. ‘day’ is not necessary because day is the final reduction state. As files are reduced they are removed from the set. Thus the sets contain the active set of files to be reduced.

Class Method Summary collapse

Class Method Details

.add_file(s3_key) ⇒ Object 



30
31
32
33
34
35
36
37
38
39
40
41
42
43
 
# File 'lib/cosmos/models/reducer_model.rb', line 30

def self.add_file(s3_key)
  # s3_key is formatted like STARTTIME__ENDTIME__SCOPE__TARGET__PACKET__TYPE.bin
  # e.g. 20211229191610578229500__20211229192610563836500__DEFAULT__INST__HEALTH_STATUS__rt__decom.bin
  _, _, scope, target, _ = s3_key.split('__')
  case s3_key
  when /__decom\.bin$/
    Store.sadd("#{scope}__#{target}__reducer__decom", s3_key)
  when /__minute\.bin$/
    Store.sadd("#{scope}__#{target}__reducer__minute", s3_key)
  when /__hour\.bin$/
    Store.sadd("#{scope}__#{target}__reducer__hour", s3_key)
  end
  # No else clause because add_file is called with raw files which are ignored
end

.all_files(type:, target:, scope:) ⇒ Object 



61
62
63
 
# File 'lib/cosmos/models/reducer_model.rb', line 61

def self.all_files(type:, target:, scope:)
  Store.smembers("#{scope}__#{target}__reducer__#{type.downcase}").sort
end

.rm_file(s3_key) ⇒ Object 



45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
 
# File 'lib/cosmos/models/reducer_model.rb', line 45

def self.rm_file(s3_key)
  _, _, scope, target, _ = s3_key.split('__')
  case s3_key
  when /__decom\.bin$/
    Store.srem("#{scope}__#{target}__reducer__decom", s3_key)
  when /__minute\.bin$/
    Store.srem("#{scope}__#{target}__reducer__minute", s3_key)
  when /__hour\.bin$/
    Store.srem("#{scope}__#{target}__reducer__hour", s3_key)
  else
    # We should only remove files that were previously in the set
    # Thus if we don't match the s3_key it is an error
    raise "Unknown file #{s3_key}"
  end
end