Class: RequestLogAnalyzer::Controller

Inherits:
Object
  • Object
show all
Defined in:
lib/request_log_analyzer/controller.rb

Overview

The RequestLogAnalyzer::Controller class creates a LogParser instance for the requested file format and connect it with sources and aggregators.

Sources are streams or files from which the requests will be parsed. Aggregators will handle every passed request to yield a meaningfull results.

  • Use the build -function to build a new controller instance.

  • Use the run! method to start the parser and send the requests to the aggregators.

Note that the order of sources can be imported if you have log files than succeed eachother. Requests that span over succeeding files will be parsed correctly if the sources are registered in the correct order. This can be helpful to parse requests from several logrotated log files.

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(source, options = {}) ⇒ Controller

Builds a new Controller for the given log file format. format Logfile format. Defaults to :rails Options are passd on to the LogParser.

  • :database Database the controller should use.

  • :yaml Yaml Dump the contrller should use.

  • :output All report outputs get << through this output.

  • :no_progress No progress bar



205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
 
# File 'lib/request_log_analyzer/controller.rb', line 205

def initialize(source, options = {})

  @source      = source
  @options     = options
  @aggregators = []
  @filters     = []
  @output      = options[:output]
  @interrupted = false
  
  # Register the request format for this session after checking its validity
  raise "Invalid file format!" unless @source.file_format.valid?

  # Install event handlers for wrnings, progress updates and source changes
  @source.warning        = lambda { |type, message, lineno|  @aggregators.each { |agg| agg.warning(type, message, lineno) } }
  @source.progress       = lambda { |message, value| handle_progress(message, value) } unless options[:no_progress]
  @source.source_changes = lambda { |change, filename| handle_source_change(change, filename) }
end

Instance Attribute Details

#aggregators ⇒ Object (readonly)

Returns the value of attribute aggregators.



18
19
20
 
# File 'lib/request_log_analyzer/controller.rb', line 18

def aggregators
  @aggregators
end

#filters ⇒ Object (readonly)

Returns the value of attribute filters.



18
19
20
 
# File 'lib/request_log_analyzer/controller.rb', line 18

def filters
  @filters
end

#options ⇒ Object (readonly)

Returns the value of attribute options.



18
19
20
 
# File 'lib/request_log_analyzer/controller.rb', line 18

def options
  @options
end

#output ⇒ Object (readonly)

Returns the value of attribute output.



18
19
20
 
# File 'lib/request_log_analyzer/controller.rb', line 18

def output
  @output
end

#source ⇒ Object (readonly)

Returns the value of attribute source.



18
19
20
 
# File 'lib/request_log_analyzer/controller.rb', line 18

def source
  @source
end

Class Method Details

.build(options) ⇒ Object

Build a new controller. Returns a new RequestLogAnalyzer::Controller object.

Options

  • :after Drop all requests after this date (Date, DateTime, Time, or a String in “YYYY-MM-DD hh:mm:ss” format)

  • :aggregator Array of aggregators (ATM: STRINGS OR SYMBOLS ONLY! - Defaults to [:summarizer]).

  • :boring Do not show color on STDOUT (Defaults to false).

  • :before Drop all requests before this date (Date, DateTime, Time or a String in “YYYY-MM-DD hh:mm:ss” format)

  • :database Database file to insert encountered requests to.

  • :debug Enables echo aggregator which will echo each request analyzed.

  • :file Filestring, File or StringIO.

  • :format :rails, => ‘FORMATSTRING’, :merb, etcetera or Format Class. (Defaults to :rails).

  • :mail Email the results to this email address.

  • :no_progress Do not display the progress bar (increases speed).

  • :output :fixed_width, :html or Output class. Defaults to fixed width.

  • :reject Reject specific => :value combination (expects a single hash).

  • :report_width Width or reports in characters. (Defaults to 80)

  • :reset_database Reset the database before starting.

  • :select Select specific => :value combination (expects a single hash).

  • :source_files Source files to analyze. Provide either File, array of files or STDIN.

  • :yaml Output to YAML file.

Example

RequestLogAnalyzer::Controller.build(

:output       => :HTML,
:mail         => 'root@localhost',
:after        => Time.now - 24*60*60, 
:source_files => '/var/log/passenger.log'

).run!

Todo

  • Check if defaults work (Aggregator defaults seem wrong).

  • Refactor :database => options, :dump => options away from contoller intialization.



105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
 
# File 'lib/request_log_analyzer/controller.rb', line 105

def self.build(options)
  # Defaults
  options[:output]        ||= 'fixed_width'
  options[:format]        ||= :rails
  options[:aggregator]    ||= [:summarizer]
  options[:report_width]  ||= 80
  options[:report_amount] ||= 20
  options[:report_sort]   ||= 'sum,mean'
  options[:boring]        ||= false
  
  # Deprecation warnings
  if options[:dump] && options[:yaml].blank?
    warn "[DEPRECATION] `:dump` is deprecated.  Please use `:yaml` instead."
    options[:yaml]          = options[:dump]
  end
  
  # Set the output class
  output_args   = {}
  output_object = nil
  if options[:output].is_a? Class
    output_class = options[:output]
  else
    output_class = RequestLogAnalyzer::Output::const_get(options[:output])
  end
  
  output_sort   = options[:report_sort].split(',').map { |s| s.to_sym }
  output_amount = options[:report_amount] == 'all' ? :all : options[:report_amount].to_i
  
  if options[:file]
    output_object = %w[File StringIO].include?(options[:file].class.name) ? options[:file] : File.new(options[:file], "w+")
    output_args   = {:width => 80, :color => false, :characters => :ascii, :sort => output_sort, :amount => output_amount }
  elsif options[:mail]
    output_object = RequestLogAnalyzer::Mailer.new(options[:mail])
    output_args   = {:width => 80, :color => false, :characters => :ascii, :sort => output_sort, :amount => output_amount  }
  else
    output_object = STDOUT
    output_args   = {:width => options[:report_width].to_i, :color => !options[:boring],
                    :characters => (options[:boring] ? :ascii : :utf), :sort => output_sort, :amount => output_amount }
  end
  
  output_instance = output_class.new(output_object, output_args)
  
  # Create the controller with the correct file format
  if options[:format].kind_of?(Hash)
    file_format = RequestLogAnalyzer::FileFormat.load(options[:format].keys[0], options[:format].values[0])
  else
    file_format = RequestLogAnalyzer::FileFormat.load(options[:format])
  end
  
  # Kickstart the controller
  controller = Controller.new(  RequestLogAnalyzer::Source::LogParser.new(file_format, :source_files => options[:source_files]),
                                { :output => output_instance,
                                  :database => options[:database],                # FUGLY!
                                  :yaml => options[:yaml], 
                                  :reset_database => options[:reset_database],
                                  :no_progress => options[:no_progress]})

  # register filters
  if options[:after] || options[:before]
    filter_options = {}
    [:after, :before].each do |filter|
      case options[filter]
      when Date, DateTime, Time
        filter_options[filter] = options[filter]
      when String
        filter_options[filter] = DateTime.parse(options[filter])
      end
    end
    controller.add_filter(:timespan, filter_options)
  end

  if options[:reject]
    options[:reject].each do |(field, value)|
      controller.add_filter(:field, :mode => :reject, :field => field, :value => value)
    end
  end

  if options[:reject]
    options[:select].each do |(field, value)|
      controller.add_filter(:field, :mode => :select, :field => field, :value => value)
    end
  end

  # register aggregators
  options[:aggregator].each { |agg| controller.add_aggregator(agg.to_sym) }
  controller.add_aggregator(:summarizer)          if options[:aggregator].empty?
  controller.add_aggregator(:echo)                if options[:debug]
  controller.add_aggregator(:database_inserter)   if options[:database] && !options[:aggregator].include?('database')

  file_format.setup_environment(controller)
  return controller
end

.build_from_arguments(arguments) ⇒ Object

Builds a RequestLogAnalyzer::Controller given parsed command line arguments <tt>arguments<tt> A CommandLine::Arguments hash containing parsed commandline parameters.



22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
 
# File 'lib/request_log_analyzer/controller.rb', line 22

def self.build_from_arguments(arguments)
  
  options = {}
  
  # Copy fields
  options[:database]       = arguments[:database]
  options[:reset_database] = arguments[:reset_database]
  options[:debug]          = arguments[:debug]
  options[:yaml]           = arguments[:dump]
  options[:mail]           = arguments[:mail]
  options[:no_progress]    = arguments[:no_progress]
  options[:format]         = arguments[:format]
  options[:output]         = arguments[:output]
  options[:file]           = arguments[:file]
  options[:format]         = arguments[:format]
  options[:after]          = arguments[:after]
  options[:before]         = arguments[:before]
  options[:reject]         = arguments[:reject]
  options[:select]         = arguments[:select]
  options[:boring]         = arguments[:boring]
  options[:aggregator]     = arguments[:aggregator]
  options[:report_width]   = arguments[:report_width]
  options[:report_sort]    = arguments[:report_sort]
  options[:report_amount]  = arguments[:report_amount]
  
  # Apache format workaround
  if arguments[:rails_format]
    options[:format] = {:rails => arguments[:rails_format]}
  elsif arguments[:apache_format]
    options[:format] = {:apache => arguments[:apache_format]}
  end
  
  # Register sources
  if arguments.parameters.length == 1
    file = arguments.parameters[0]
    if file == '-' || file == 'STDIN'
      options.store(:source_files, $stdin)
    elsif File.exist?(file)
      options.store(:source_files, file)
    else
      puts "File not found: #{file}"
      exit(0)
    end
  else
    options.store(:source_files, arguments.parameters)
  end
  
  build(options)
end

Instance Method Details

#add_aggregator(agg) ⇒ Object Also known as: >>

Adds an aggregator to the controller. The aggregator will be called for every request that is parsed from the provided sources (see add_source)



251
252
253
254
 
# File 'lib/request_log_analyzer/controller.rb', line 251

def add_aggregator(agg)
  agg = RequestLogAnalyzer::Aggregator.const_get(RequestLogAnalyzer::to_camelcase(agg)) if agg.kind_of?(Symbol)
  @aggregators << agg.new(@source, @options)
end

#add_filter(filter, filter_options = {}) ⇒ Object

Adds a request filter to the controller.



259
260
261
262
 
# File 'lib/request_log_analyzer/controller.rb', line 259

def add_filter(filter, filter_options = {})
  filter = RequestLogAnalyzer::Filter.const_get(RequestLogAnalyzer::to_camelcase(filter)) if filter.kind_of?(Symbol)
  @filters << filter.new(source.file_format, @options.merge(filter_options))
end

#aggregate_request(request) ⇒ Object

Push a request to all the aggregators (@aggregators). request The request to push to the aggregators.



277
278
279
280
281
 
# File 'lib/request_log_analyzer/controller.rb', line 277

def aggregate_request(request)
  return false unless request
  @aggregators.each { |agg| agg.aggregate(request) }
  return true
end

#filter_request(request) ⇒ Object

Push a request through the entire filterchain (@filters). request The request to filter. Returns the filtered request or nil.



267
268
269
270
271
272
273
 
# File 'lib/request_log_analyzer/controller.rb', line 267

def filter_request(request)
  @filters.each do |filter|
    request = filter.filter(request)
    return nil if request.nil?
  end
  return request
end

#handle_progress(message, value = nil) ⇒ Object

Progress function. Expects :started with file, :progress with current line and :finished or :interrupted when done. message Current state (:started, :finished, :interupted or :progress). value File or current line.



227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
 
# File 'lib/request_log_analyzer/controller.rb', line 227

def handle_progress(message, value = nil)
  case message
  when :started
    @progress_bar = CommandLine::ProgressBar.new(File.basename(value), File.size(value), STDOUT)
  when :finished
    @progress_bar.finish
    @progress_bar = nil
  when :interrupted
    if @progress_bar
      @progress_bar.halt
      @progress_bar = nil
    end
  when :progress
    @progress_bar.set(value)
  end
end

#handle_source_change(change, filename) ⇒ Object

Source change handler



245
246
247
 
# File 'lib/request_log_analyzer/controller.rb', line 245

def handle_source_change(change, filename)
  @aggregators.each { |agg| agg.source_change(change, File.expand_path(filename, Dir.pwd)) }
end

#install_signal_handlers ⇒ Object 



323
324
325
326
327
328
329
 
# File 'lib/request_log_analyzer/controller.rb', line 323

def install_signal_handlers
  Signal.trap("INT") do
    handle_progress(:interrupted)
    puts "Caught interrupt! Stopping parsing..."
    @interrupted = true
  end
end

#run! ⇒ Object

Runs RequestLogAnalyzer

  1. Call prepare on every aggregator

  2. Generate requests from source object

  3. Filter out unwanted requests

  4. Call aggregate for remaning requests on every aggregator

  5. Call finalize on every aggregator

  6. Call report on every aggregator

  7. Finalize Source



291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
 
# File 'lib/request_log_analyzer/controller.rb', line 291

def run!

  # @aggregators.each{|agg| p agg}

  @aggregators.each { |agg| agg.prepare }
  install_signal_handlers

  @source.each_request do |request|
    break if @interrupted
    aggregate_request(filter_request(request))
  end

  @aggregators.each { |agg| agg.finalize }

  @output.header
  @aggregators.each { |agg| agg.report(@output) }
  @output.footer

  @source.finalize

  if @output.io.kind_of?(File)
    puts
    puts "Report written to: " + File.expand_path(@output.io.path)
    puts "Need an expert to analyze your application?"
    puts "Mail to [email protected] or visit us at http://railsdoctors.com"
    puts "Thanks for using request-log-analyzer!"
    @output.io.close
  elsif @output.io.kind_of?(RequestLogAnalyzer::Mailer)
    @output.io.mail
  end
end