Module: ScraperUtils::RandomizeUtils

Defined in:
lib/scraper_utils/randomize_utils.rb

Overview

Provides utilities for randomizing processing order in scrapers, particularly helpful for distributing load and avoiding predictable patterns

Class Method Summary collapse

Class Method Details

.randomize_order(collection) ⇒ Array

Returns a randomized version of the input collection when in production mode, or the original collection when in test/sequential mode

Parameters:

  • collection (Array, Enumerable) —

    Collection of items to potentially randomize

Returns:

  • (Array) —

    Randomized or original collection depending on environment



12
13
14
15
16
 
# File 'lib/scraper_utils/randomize_utils.rb', line 12

def self.randomize_order(collection)
  return collection.to_a if sequential?

  collection.to_a.shuffle
end

.sequential=(value) ⇒ Boolean?

Explicitly set sequential mode for testing

Parameters:

  • value (Boolean, nil) —

    true to enable sequential mode, false to disable, nil to clear cache

Returns:

  • (Boolean, nil) 


30
31
32
 
# File 'lib/scraper_utils/randomize_utils.rb', line 30

def self.sequential=(value)
  @sequential = value
end

.sequential? ⇒ Boolean

Checks if sequential processing is enabled

Returns:

  • (Boolean) —

    true when in test mode or MORPH_PROCESS_SEQUENTIALLY is set



21
22
23
24
 
# File 'lib/scraper_utils/randomize_utils.rb', line 21

def self.sequential?
  @sequential = !ENV["MORPH_PROCESS_SEQUENTIALLY"].to_s.empty? if @sequential.nil?
  @sequential || false
end