Class: WIKK::WebBrowser

Inherits:
Object
  • Object
show all
Defined in:
lib/wikk_webbrowser.rb

Overview

WIKK WebBrowser class under MIT Lic. github.com/wikarekare. Wrapper around ruby’s http classes

WIKK_WebBrowser.new.https_session(host: 'www.blah.com') do |session|
  response = get_page(query: ,'/')
end

Defined Under Namespace

Classes: Error

Constant Summary collapse

VERSION = 
'0.9.7'

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(host:, port: nil, use_ssl: false, cookies: {}, verify_cert: true, debug: false) ⇒ WIKK_WebBrowser

Create a WIKK_WebBrowser instance

Parameters:

  • host (String) —

    the host we want to connect to

  • port (Fixnum) (defaults to: nil) —

    Optional http server port

  • use_ssl (Boolean) (defaults to: false) —

    Use https, if true

  • verify_cert (Boolean) (defaults to: true) —

    Validate certificate if true (Nb lots of embedded devices have self signed certs, so verify will fail)



46
47
48
49
50
51
52
53
54
 
# File 'lib/wikk_webbrowser.rb', line 46

def initialize(host:, port: nil, use_ssl: false, cookies: {}, verify_cert: true, debug: false)
  @host = host # Need to do this, as passing nil is different to passing nothing to initialize!
  @cookies = cookies.nil? ? {} : cookies
  @debug = debug
  @use_ssl = use_ssl
  @port = port.nil? ? ( use_ssl ? 443 : 80 ) : port
  @verify_cert = verify_cert
  @response = nil
end

Instance Attribute Details

#cookies ⇒ Object

Returns the value of attribute cookies.



29
30
31
 
# File 'lib/wikk_webbrowser.rb', line 29

def cookies
  @cookies
end

#debug ⇒ Object

Returns the value of attribute debug.



32
33
34
 
# File 'lib/wikk_webbrowser.rb', line 32

def debug
  @debug
end

#host ⇒ Object (readonly)

Returns the value of attribute host.



27
28
29
 
# File 'lib/wikk_webbrowser.rb', line 27

def host
  @host
end

#page ⇒ Object (readonly)

Returns the value of attribute page.



30
31
32
 
# File 'lib/wikk_webbrowser.rb', line 30

def page
  @page
end

#port ⇒ Object

Returns the value of attribute port.



34
35
36
 
# File 'lib/wikk_webbrowser.rb', line 34

def port
  @port
end

#referer ⇒ Object

Returns the value of attribute referer.



31
32
33
 
# File 'lib/wikk_webbrowser.rb', line 31

def referer
  @referer
end

#response ⇒ Object

Returns the value of attribute response.



36
37
38
 
# File 'lib/wikk_webbrowser.rb', line 36

def response
  @response
end

#session ⇒ Object

Returns the value of attribute session.



28
29
30
 
# File 'lib/wikk_webbrowser.rb', line 28

def session
  @session
end

#use_ssl ⇒ Object

Returns the value of attribute use_ssl.



35
36
37
 
# File 'lib/wikk_webbrowser.rb', line 35

def use_ssl
  @use_ssl
end

#verify_cert ⇒ Object

Returns the value of attribute verify_cert.



33
34
35
 
# File 'lib/wikk_webbrowser.rb', line 33

def verify_cert
  @verify_cert
end

Class Method Details

.http_session(host:, port: nil, debug: false, cookies: {}) {|the| ... } ⇒ Object

Create a WIKK_WebBrowser instance, connect to the host via http, and yield the WIKK_WebBrowser instance. Automatically closes the http session on returning from the block passed to it.

Parameters:

  • host (String) —

    the host we want to connect to

  • port (Fixnum) (defaults to: nil) —

    (80) the port the remote web server is running on

  • block (Proc)

Yield Parameters:

  • the (WIKK_WebBrowser) —

    session descriptor for further calls.



67
68
69
70
71
72
 
# File 'lib/wikk_webbrowser.rb', line 67

def self.http_session(host:, port: nil, debug: false, cookies: {})
  wb = self.new(host: host, port: port, debug: debug, use_ssl: false, cookies: cookies)
  wb.http_session do
    yield wb
  end
end

.https_session(host:, port: nil, verify_cert: true, cookies: {}, debug: false) {|the| ... } ⇒ Object

Create a WIKK_WebBrowser instance, connect to the host via https, and yield the WIKK_WebBrowser instance.

Automatically closes the http session on returning from the block passed to it.

Parameters:

  • host (String) —

    the host we want to connect to

  • port (Fixnum) (defaults to: nil) —

    (443) the port the remote web server is running on

  • verify_cert (Boolean) (defaults to: true) —

    Validate certificate if true (Nb lots of embedded devices have self signed certs, so verify will fail)

  • block (Proc)

Yield Parameters:

  • the (WIKK_WebBrowser) —

    session descriptor for further calls.



81
82
83
84
85
86
 
# File 'lib/wikk_webbrowser.rb', line 81

def self.https_session(host:, port: nil, verify_cert: true, cookies: {}, debug: false)
  wb = self.new(host: host, port: port, cookies: cookies, use_ssl: true, verify_cert: verify_cert, debug: debug)
  wb.http_session do
    yield wb
  end
end

Instance Method Details

#add_cookies(cookies) ⇒ Object

Add additional cookies

Parameters:

  • cookies (Hash) —

    cookie_name => cookie_value



123
124
125
 
# File 'lib/wikk_webbrowser.rb', line 123

def add_cookies(cookies)
  cookies.each { |cookie_name, cookie_value| @cookies[cookie_name] = cookie_value }
end

#basic_authorization(user:, password:) ⇒ String

Web basic authentication (not exactly secure)

Parameters:

  • user (String) —

    Account name

  • password (String) —

    Accounts password

Returns:

  • (String) —

    Base64 encoded concatentation of user + ‘:’ + password



109
110
111
112
 
# File 'lib/wikk_webbrowser.rb', line 109

def basic_authorization(user:, password:)
  # req.basic_auth( user, password) if  user != nil
  'Basic ' + Base64.encode64( "#{user}:#{password}" )
end

#bearer_authorization(token:) ⇒ String

Dropbox style token authentication

Parameters:

  • token (String) —

    Token, as issued by dropbox

Returns:

  • (String) —

    Concatenation of ‘Bearer ’ + token



117
118
119
 
# File 'lib/wikk_webbrowser.rb', line 117

def bearer_authorization(token:)
  'Bearer ' + token
end

#cookies_to_s ⇒ Object

Convert @cookies to ; separated strings

Returns:

  • cookies string



141
142
143
 
# File 'lib/wikk_webbrowser.rb', line 141

def cookies_to_s
  @cookies.to_a.map { |v| v.join('=') }.join('; ')
end

#delete_req(query:, authorization: nil, form_values: nil, content_type: '"application/octet-stream"', data: nil, extra_headers: {}, extra_cookies: {}) ⇒ String Also known as: delete

send a DELETE query to the server and return the response.

Parameters:

  • query (String) —

    URL, less the ‘host/’ part

  • authorization (String) (defaults to: nil) —

    If present, add Authorization header, using this string

  • form_values (Hash{String=>Object-with-to_s}) (defaults to: nil) —

    The parameter passed to the web server eg. ?key1=value1&key2=value2…

  • content_type (String) (defaults to: '"application/octet-stream"') —

    Posted content type. Only meaningful if there is data

  • data (String) (defaults to: nil) —

    Text to add to body of DELETE to the web server. Not always supported at the server end.

  • extra_headers (Hash) (defaults to: {}) —

    Add these to standard headers

  • extra_cookies (Hash) (defaults to: {}) —

    Add these to standard cookies

Returns:

  • (String) —

    The Net::HTTPResponse.body text response from the web server



283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
 
# File 'lib/wikk_webbrowser.rb', line 283

def delete_req(query:, authorization: nil, form_values: nil, content_type: '"application/octet-stream"', data: nil, extra_headers: {}, extra_cookies: {})
  $stderr.puts 'Debugging On' if @debug

  query += form_values_to_s(form_values, query.index('?') != nil)
  url = URI.parse("#{@use_ssl ? 'https' : 'http'}://#{@host}/#{query.gsub(/^\//, '')}")
  req = Net::HTTP::Delete.new(query)

  header = { 'HOST' => @host }
  add_cookies(extra_cookies)
  header['Cookie'] = cookies_to_s if @cookies.length > 0
  header['Authorization'] = authorization if authorization != nil

  extra_headers.each do |k, v|
    header[k] = v
  end
  req.initialize_http_header( header )

  if data.nil?
    req.body = ''
  elsif data.instance_of?(Hash)
    if content_type =~ /application\/octet-stream/
      req.set_form_data(data, '&')
    else
      req.set_form_data.to_j
    end
  else
    req.body = data # If json as a string or raw string
  end

  begin
    @response = @session.request(req)
    save_cookies(@response)

    if @response.code.to_i >= 300
      raise "#{url} : #{@response.code} #{@response.message}"
    end

    return @response.body
  rescue StandardError => e
    puts "#{e}"
    return nil
  end
end

#extract_input_fields(body) ⇒ Hash

Extract form field values from the html body.

Parameters:

  • body (String) —

    The html response body

Returns:

  • (Hash) —

    Keys are the field names, values are the field values



454
455
456
457
458
459
460
 
# File 'lib/wikk_webbrowser.rb', line 454

def extract_input_fields(body)
  @inputs = {}
  doc = Nokogiri::HTML(body)
  doc.xpath('//form/input').each do |f|
    @inputs[f.get_attribute('name')] = f.get_attribute('value')
  end
end

#extract_link_fields(body) ⇒ Hash

Extract links from the html body.

Parameters:

  • body (String) —

    The html response body

Returns:

  • (Hash) —

    Keys are the link text, values are the html links



465
466
467
468
469
470
471
472
 
# File 'lib/wikk_webbrowser.rb', line 465

def extract_link_fields(body)
  @inputs = {}
  doc = Nokogiri::HTML(body)
  doc.xpath('//a').each do |f|
    return URI.parse( f.get_attribute('href') ).path if f.get_attribute('name') == 'URL$1'
  end
  return nil
end

#form_values_to_s(form_values = nil, has_q = false) ⇒ String

Take a hash of the params to the post and generate a safe URL string.

Parameters:

  • form_values (Hash) (defaults to: nil) —

    Keys are the field names, values are the field values

  • has_q (Boolean) (defaults to: false) —

    We have a leading ? for the html get, so don’t need to add one.

Returns:

  • (String) —

    The ‘safe’ text for fields the get or post query to the web server



478
479
480
481
482
483
484
485
486
487
488
489
 
# File 'lib/wikk_webbrowser.rb', line 478

def form_values_to_s(form_values = nil, has_q = false)
  return '' if form_values.nil? || form_values.length == 0

  s = (has_q == true ? '' : '?')
  first = true
  form_values.each do |key, value|
    s += '&' unless first
    s += "#{CGI.escape(key.to_s)}=#{CGI.escape(value.to_s)}"
    first = false
  end
  return s
end

#get_page(query:, form_values: nil, authorization: nil, extra_headers: {}, extra_cookies: {}) ⇒ String Also known as: get

send a GET query to the web server using an http get, and returns the response.

Cookies in the response get preserved in @cookies, so they will be sent along with subsequent calls
We are currently ignoring redirects from the PDU's we are querying.

Parameters:

  • query (String) —

    The URL after the host/ bit and not usually not including parameters, if form_values are passed in

  • form_values (Hash{String=>Object-with-to_s}) (defaults to: nil) —

    The parameter passed to the web server eg. ?key1=value1&key2=value2…

  • authorization (String) (defaults to: nil) —

    If present, add Authorization header, using this string

  • extra_headers (Hash) (defaults to: {}) —

    Add these to standard headers

  • extra_cookies (Hash) (defaults to: {}) —

    Add these to standard cookies

Returns:

  • (String) —

    The Net::HTTPResponse.body text response from the web server



162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
 
# File 'lib/wikk_webbrowser.rb', line 162

def get_page(query:, form_values: nil, authorization: nil, extra_headers: {}, extra_cookies: {})
  $stderr.puts 'Debugging On' if @debug
  query += form_values_to_s(form_values, query.index('?') != nil) # Should be using req.set_form_data, but it seems to by stripping the leading / and then the query fails.
  url = URI.parse("#{@use_ssl ? 'https' : 'http'}://#{@host}/#{query.gsub(/^\//, '')}")
  $stderr.puts url if @debug

  req = Net::HTTP::Get.new(url.request_uri)

  header = { 'HOST' => @host }
  header['Accept'] = '*/*'
  header['Accept-Encoding'] = 'gzip, deflate, br'
  header['Accept-Language'] = 'en-US,en;q=0.5'
  header['Connection'] = 'keep-alive'
  header['User-Agent'] = 'Mozilla/5.0'
  header['Content-Type'] = 'application/x-www-form-urlencoded'
  add_cookies(extra_cookies)
  header['Cookie'] = cookies_to_s if @cookies.length > 0
  header['DNT'] = '1'
  header['Authorization'] = authorization if authorization != nil

  extra_headers.each do |k, v|
    header[k] = v
  end

  req.initialize_http_header( header )

  @response = @session.request(req)
  save_cookies(@response)

  $stderr.puts @response.code.to_i if @debug

  if @response.code.to_i >= 300
    if @response.code.to_i == 302
      # ignore the redirects.
      # $stderr.puts "302"
      # @response.each {|key, val| $stderr.printf "%s = %s\n", key, val }  #Location seems to have cgi params removed. End up with .../cginame?&
      # $stderr.puts "Redirect to #{@response['location']}"   #Location seems to have cgi params removed. End up with .../cginame?&
      # $stderr.puts
      return
    elsif @response.code.to_i >= 400 && @response.code.to_i < 500
      return @response.body
    end

    raise Error.new(web_return_code: @response.code.to_i, message: "#{@response.code} #{@response.message} #{query} #{form_values} #{@response.body}")
  end

  return @response.body
end

#header_value(key:) ⇒ String

Get a header value, from the last response

Parameters:

  • key (String) —

    header key

Returns:

  • (String) —

    header value, for the given key.



149
150
151
 
# File 'lib/wikk_webbrowser.rb', line 149

def header_value(key:)
  @response.header[key]
end

#http_session ⇒ Object

Creating a session for http connection attached block would then call get or post NET::HTTP calls

Parameters:

  • port (Fixnum) —

    Optional http server port

  • use_ssl (Boolean) —

    Use https, if true

  • verify_cert (Boolean) —

    Validate certificate if true (Nb lots of embedded devices have self signed certs, so verify will fail)

  • block (Proc) 


94
95
96
97
98
99
100
101
102
103
 
# File 'lib/wikk_webbrowser.rb', line 94

def http_session
  @http = Net::HTTP.new(@host, @port)
  @http.set_debug_output($stdout) if @debug
  @http.use_ssl = @use_ssl
  @http.verify_mode = OpenSSL::SSL::VERIFY_NONE if ! @use_ssl || ! @verify_cert
  @http.start do |session| # ensure we close the session after the block
    @session = session
    yield
  end
end

#patch_req(query:, authorization: nil, content_type: '"application/octet-stream"', data: nil, extra_headers: {}, extra_cookies: {}) ⇒ String Also known as: patch

send a PATCH query to the server and return the response.

Parameters:

  • query (String) —

    URL, less the ‘host/’ part

  • authorization (String) (defaults to: nil) —

    If present, add Authorization header, using this string

  • content_type (String) (defaults to: '"application/octet-stream"') —

    Posted content type

  • data (String) (defaults to: nil) —

    Text to add to body of post to the web server

  • extra_headers (Hash) (defaults to: {}) —

    Add these to standard headers

  • extra_cookies (Hash) (defaults to: {}) —

    Add these to standard cookies

Returns:

  • (String) —

    The Net::HTTPResponse.body text response from the web server



398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
 
# File 'lib/wikk_webbrowser.rb', line 398

def patch_req(query:, authorization: nil, content_type: '"application/octet-stream"', data: nil, extra_headers: {}, extra_cookies: {})
  url = URI.parse("#{@use_ssl ? 'https' : 'http'}://#{@host}/#{query}")
  req = Net::HTTP::Patch.new(url.path)

  header = { 'HOST' => @host }
  header['Accept'] = '*/*'
  header['Accept-Encoding'] = 'gzip, deflate, br'
  header['Accept-Language'] = 'en-US,en;q=0.5'
  header['Connection'] = 'keep-alive'
  header['User-Agent'] = 'Mozilla/5.0'
  header['Content-Type'] = content_type
  add_cookies(extra_cookies)
  header['Cookie'] = cookies_to_s if @cookies.length > 0
  header['DNT'] = '1'
  header['Authorization'] = authorization if authorization != nil

  extra_headers.each do |k, v|
    header[k] = v
  end
  req.initialize_http_header( header )

  if data.nil?
    req.body = ''
  elsif data.instance_of?(Hash)
    if content_type =~ /application\/octet-stream/
      req.set_form_data(data, '&')
    else
      req.set_form_data.to_j
    end
  else
    req.body = data # If json as a string or raw string
  end

  @response = @session.request(req)
  save_cookies(@response)

  if @response.code.to_i >= 300
    if @response.code.to_i == 302
      # ignore the redirects.
      # puts "302"
      # @response.each {|key, val| printf "%s = %s\n", key, val }  #Location seems to have cgi params removed. End up with .../cginame?&
      # puts "Redirect of Post to #{@response['location']}" #Location seems to have cgi params removed. End up with .../cginame?&
      return
    end

    raise Error.new(web_return_code: @response.code, message: "#{@response.code} #{@response.message} #{query} #{data} #{@response.body}")
  end

  return @response.body
end

#post_page(query:, authorization: nil, content_type: 'application/x-www-form-urlencoded', data: nil, extra_headers: {}, extra_cookies: {}) ⇒ String Also known as: post

send a POST query to the server and return the response.

Parameters:

  • query (String) —

    URL, less the ‘host/’ part

  • authorization (String) (defaults to: nil) —

    If present, add Authorization header, using this string

  • content_type (String) (defaults to: 'application/x-www-form-urlencoded') —

    Posted content type

  • data (String) (defaults to: nil) —

    Text to add to body of post to the web server

  • extra_headers (Hash) (defaults to: {}) —

    Add these to standard headers

  • extra_cookies (Hash) (defaults to: {}) —

    Add these to standard cookies

Returns:

  • (String) —

    The Net::HTTPResponse.body text response from the web server



221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
 
# File 'lib/wikk_webbrowser.rb', line 221

def post_page(query:, authorization: nil, content_type: 'application/x-www-form-urlencoded', data: nil, extra_headers: {}, extra_cookies: {})
  url = URI.parse("#{@use_ssl ? 'https' : 'http'}://#{@host}/#{query}")
  req = Net::HTTP::Post.new(url.path)

  header = { 'HOST' => @host }
  header['Accept'] = '*/*'
  header['Accept-Encoding'] = 'gzip, deflate, br'
  header['Accept-Language'] = 'en-US,en;q=0.5'
  header['Connection'] = 'keep-alive'
  header['User-Agent'] = 'Mozilla/5.0'
  header['Content-Type'] = content_type
  add_cookies(extra_cookies)
  header['Cookie'] = cookies_to_s if @cookies.length > 0
  header['DNT'] = '1'
  header['Authorization'] = authorization if authorization != nil

  extra_headers.each do |k, v|
    header[k] = v
  end
  req.initialize_http_header( header )

  if data.nil?
    req.body = ''
  elsif data.instance_of?(Hash)
    if content_type =~ /application\/octet-stream/
      req.set_form_data(data, '&')
    else
      req.set_form_data.to_j
    end
  else
    req.body = data # If json as a string or raw string
  end

  @response = @session.request(req)
  save_cookies(@response)

  if @response.code.to_i >= 300
    if @response.code.to_i == 302
      # ignore the redirects.
      # puts "302"
      # @response.each {|key, val| printf "%s = %s\n", key, val }  #Location seems to have cgi params removed. End up with .../cginame?&
      # puts "Redirect of Post to #{@response['location']}" #Location seems to have cgi params removed. End up with .../cginame?&
      return
    end

    raise Error.new(web_return_code: @response.code, message: "#{@response.code} #{@response.message} #{query} #{data} #{@response.body}")
  end

  return @response.body
end

#put_req(query:, authorization: nil, content_type: '"application/octet-stream"', data: nil, extra_headers: {}, extra_cookies: {}) ⇒ String Also known as: put

send a PUT query to the server and return the response.

Parameters:

  • query (String) —

    URL, less the ‘host/’ part

  • authorization (String) (defaults to: nil) —

    If present, add Authorization header, using this string

  • content_type (String) (defaults to: '"application/octet-stream"') —

    Posted content type

  • data (String) (defaults to: nil) —

    Text to add to body of post to the web server

  • extra_headers (Hash) (defaults to: {}) —

    Add these to standard headers

  • extra_cookies (Hash) (defaults to: {}) —

    Add these to standard cookies

Returns:

  • (String) —

    The Net::HTTPResponse.body text response from the web server



337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
 
# File 'lib/wikk_webbrowser.rb', line 337

def put_req(query:, authorization: nil, content_type: '"application/octet-stream"', data: nil, extra_headers: {}, extra_cookies: {})
  url = URI.parse("#{@use_ssl ? 'https' : 'http'}://#{@host}/#{query}")
  req = Net::HTTP::Put.new(url.path)

  header = { 'HOST' => @host }
  header['Accept'] = '*/*'
  header['Accept-Encoding'] = 'gzip, deflate, br'
  header['Accept-Language'] = 'en-US,en;q=0.5'
  header['Connection'] = 'keep-alive'
  header['User-Agent'] = 'Mozilla/5.0'
  header['Content-Type'] = content_type
  add_cookies(extra_cookies)
  header['Cookie'] = cookies_to_s if @cookies.length > 0
  header['DNT'] = '1'
  header['Authorization'] = authorization if authorization != nil

  extra_headers.each do |k, v|
    header[k] = v
  end
  req.initialize_http_header( header )

  if data.nil?
    req.body = ''
  elsif data.instance_of?(Hash)
    if content_type =~ /application\/octet-stream/
      req.set_form_data(data, '&')
    else
      req.set_form_data.to_j
    end
  else
    req.body = data # If json as a string or raw string
  end

  @response = @session.request(req)
  save_cookies(@response)

  if @response.code.to_i >= 300
    if @response.code.to_i == 302
      # ignore the redirects.
      # puts "302"
      # @response.each {|key, val| printf "%s = %s\n", key, val }  #Location seems to have cgi params removed. End up with .../cginame?&
      # puts "Redirect of Post to #{@response['location']}" #Location seems to have cgi params removed. End up with .../cginame?&
      return
    end

    raise Error.new(web_return_code: @response.code, message: "#{@response.code} #{@response.message} #{query} #{data} #{@response.body}")
  end

  return @response.body
end

#save_cookies(response) ⇒ Object

Save cookies returned by last html get/post. Removes previous cookies.

Parameters:

  • response (Net::HTTPResponse) —

    result from HTTP calls



130
131
132
133
134
135
136
137
 
# File 'lib/wikk_webbrowser.rb', line 130

def save_cookies(response)
  if (cookie_lines = response.get_fields('set-cookie')) != nil
    cookie_lines.each do |cookie_line|
      cookies = cookie_line.split('; ').map { |v| v.split('=') }
      cookies.each { |c| @cookies[c[0]] = c[1] }
    end
  end
end