Module: ScraperUtils::RandomizeUtils

Defined in:
lib/scraper_utils/randomize_utils.rb

Overview

Provides utilities for randomizing processing order in scrapers, particularly helpful for distributing load and avoiding predictable patterns

Class Attribute Summary collapse

Class Method Summary collapse

Class Attribute Details

.random ⇒ Boolean Also known as: random?

Note:

Defaults to true unless the MORPH_DISABLE_RANDOM ENV variable is set

Controls if processing order can be randomized

Returns:

  • (Boolean) —

    true if all processing is done sequentially, otherwise false



12
13
14
 
# File 'lib/scraper_utils/randomize_utils.rb', line 12

def random
  @random
end

Class Method Details

.randomize_order(collection) ⇒ Array

Returns a randomized version of the input collection unless .sequential? is true.

Parameters:

  • collection (Array, Enumerable) —

    Collection of items

Returns:

  • (Array) —

    Randomized unless sequential? is true, otherwise original order



31
32
33
34
35
 
# File 'lib/scraper_utils/randomize_utils.rb', line 31

def self.randomize_order(collection)
  return collection.to_a.shuffle if random?

  collection.to_a
end

.reset! ⇒ Object 



20
21
22
 
# File 'lib/scraper_utils/randomize_utils.rb', line 20

def self.reset!
  @random = ENV["MORPH_DISABLE_RANDOM"].to_s.empty?
end