Module: ScraperUtils

Defined in:

lib/scraper_utils.rb,
lib/scraper_utils/version.rb,
lib/scraper_utils/db_utils.rb,
lib/scraper_utils/log_utils.rb,
lib/scraper_utils/scheduler.rb,
lib/scraper_utils/cycle_utils.rb,
lib/scraper_utils/debug_utils.rb,
lib/scraper_utils/spec_support.rb,
lib/scraper_utils/authority_utils.rb,
lib/scraper_utils/mechanize_utils.rb,
lib/scraper_utils/randomize_utils.rb,
lib/scraper_utils/date_range_utils.rb,
lib/scraper_utils/mechanize_actions.rb,
lib/scraper_utils/scheduler/constants.rb,
lib/scraper_utils/data_quality_monitor.rb,
lib/scraper_utils/scheduler/thread_request.rb,
lib/scraper_utils/scheduler/process_request.rb,
lib/scraper_utils/scheduler/thread_response.rb,
lib/scraper_utils/scheduler/operation_worker.rb,
lib/scraper_utils/mechanize_utils/agent_config.rb,
lib/scraper_utils/scheduler/operation_registry.rb,
lib/scraper_utils/mechanize_utils/adaptive_delay.rb,
lib/scraper_utils/mechanize_utils/robots_checker.rb

Overview

Utilities for planningalerts scrapers

Defined Under Namespace

Modules: AuthorityUtils, CycleUtils, DbUtils, DebugUtils, LogUtils, MechanizeUtils, RandomizeUtils, Scheduler, SpecSupport Classes: DataQualityMonitor, DateRangeUtils, Error, MechanizeActions, UnprocessableRecord, UnprocessableSite

Constant Summary

AUSTRALIAN_PROXY_ENV_VAR =

Constants for configuration on Morph.io

"MORPH_AUSTRALIAN_PROXY"

VERSION =

"0.8.1"

Class Method Summary

• .australian_proxy ⇒ Object

Class Method Details

.australian_proxy ⇒ Object

# File 'lib/scraper_utils.rb', line 35

def self.australian_proxy
  ap = ENV[AUSTRALIAN_PROXY_ENV_VAR].to_s
  ap.empty? ? nil : ap
end