Class: Api

Inherits:
Object
  • Object
show all
Defined in:
lib/ocean/api.rb,
lib/ocean/api_remote_resource.rb

Overview

This class encapsulates all logic for calling other API services.

Defined Under Namespace

Classes: NoResponseError, RemoteResource, Response, TimeoutError

Constant Summary collapse

DEFAULT_ACTIONS =

These are the default controller actions. The purpose of this constant is to map action names to hyperlink and HTTP method (for authorisation purposes). Don’t be alarmed by the non-standard GET* - it’s purely symbolic and is never used as an actual HTTP method. We need it to differentiate between a GET of a member and a GET of a collection of members. The extra_actions keyword in ocean_resource_controller follows the same format.

{
  'show' =>    ['self', 'GET'],
  'index' =>   ['self', 'GET*'],
  'create' =>  ['self', 'POST'],
  'update' =>  ['self', 'PUT'],
  'destroy' => ['self', 'DELETE'],
  'connect' =>    ['connect', 'PUT'],
  'disconnect' => ['connect', 'DELETE']
}

Class Method Summary collapse

Class Method Details

.adorn_basename(basename, ocean_env: "dev", rails_env: "development", suffix_only: false) ⇒ Object

Adds environment info to the basename, so that testing and execution in various combinations of the Rails env and the Ocean environment can be done without collision.

The ocean_env will always be appended to the basename, since we never want to share queues between different Ocean environments.

If the ocean_env is ‘dev’ or ‘ci’, we must separate things as much as possible: therefore, we add the local IP number and the Rails environment.

We also add the same information if by any chance the Rails environment isn’t ‘production’. This is a precaution; in staging and prod apps should always run in Rails production mode, but if by mistake they don’t, we must prevent the production queues from being touched.

If suffix_only is true, the basename will be excluded from the returned string.



43
44
45
46
47
48
49
50
51
 
# File 'lib/ocean/api.rb', line 43

def self.adorn_basename(basename, ocean_env: "dev", rails_env: "development",
                        suffix_only: false)
  fullname = suffix_only ? "_#{ocean_env}" : "#{basename}_#{ocean_env}"
  # if rails_env != 'production' || ocean_env == 'dev' || ocean_env == 'ci'
  #   local_ip = UDPSocket.open {|s| s.connect("64.233.187.99", 1); s.addr.last}.gsub('.', '-')
  #   fullname += "_#{local_ip}_#{rails_env}"
  # end
  fullname
end

.async_job_body(href = nil, method = :get, body: {}, credentials: nil, token: nil, steps: nil, default_step_time: nil, default_poison_limit: nil, max_seconds_in_queue: nil, poison_email: nil) ⇒ Object

Constructs a hash suitable as the body of a POST to async_jobs. It lets you set all the allowed attributes, and it also provides a terse shortcut for one-step jobs (by far the most common ones). The following:

Api.async_body "/v1/foos", :put

is equivalent to

Api.async_body steps: [{"url" => "#{INTERNAL_OCEAN_API_URL}/v1/foos"}
                        "method" => "PUT",
                        "body" => {}}]

A URL not starting with a / will be internalized.



549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
 
# File 'lib/ocean/api.rb', line 549

def self.async_job_body(href=nil, method=:get, body: {},
                        credentials: nil,
                        token: nil,
                        steps: nil,
                        default_step_time: nil,
                        default_poison_limit: nil,
                        max_seconds_in_queue: nil,
                        poison_email: nil)
  h = {}
  unless steps
    method = method && method.to_s.upcase
    href = href && (href.first == "/" ? "#{INTERNAL_OCEAN_API_URL}#{href}" : Api.internalize_uri(href))
    steps = if href && method
              if ["POST","PUT"].include? method
                [{"url" => href, "method" => method, "body" => body}]
              else
                [{"url" => href, "method" => method}]
              end
            end
    steps ||= []
  end
  h['steps'] = steps
  h['credentials'] = credentials || Api.credentials
  h['token'] = token || Api.service_token
  h['default_step_time'] = default_step_time if default_step_time
  h['default_poison_limit'] = default_poison_limit if default_poison_limit
  h['max_seconds_in_queue'] = max_seconds_in_queue if max_seconds_in_queue
  h['poison_email'] = poison_email if poison_email
  h
end

.authenticate(username = API_USER, password = API_PASSWORD) ⇒ Object

Authenticates against the Auth service (which must be deployed and running) with a given username and password. If successful, the authentication token is returned. If the credentials match the service’s own, the token is also assigned to the instance variable @service_token. If not successful, nil is returned.



385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
 
# File 'lib/ocean/api.rb', line 385

def self.authenticate(username=API_USER, password=API_PASSWORD)
  attempts = 3
  loop do
    response = Typhoeus.post "#{INTERNAL_OCEAN_API_URL}/v1/authentications",
                 body: "", headers: {'X-API-Authenticate' => credentials(username, password)}
    case response.code
    when 100..199
      return nil if attempts < 1
      attempts -= 1
      next
    when 200..299
      token = JSON.parse(response.body)['authentication']['token']
      @service_token = token if username == API_USER && password == API_PASSWORD
      return token
    when 403
      # Does not authenticate. Don't repeat the request.
      return nil
    when 400
      # Malformed credentials. Don't repeat the request.
      return nil
    when 400..499
      # Client error. Don't repeat the request.
      return nil
    when 500
      # Error. Don't repeat.
      return nil if attempts < 1
      attempts -= 1
      next
    else
      # Should never end up here.
      raise "Authentication weirdness"
    end
  end
end

.authorization_string(extra_actions, controller, action, app = "*", context = "*", service = APP_NAME) ⇒ Object

Returns an authorisation string suitable for use in calls to Api.permitted?. The extra_actions arg holds the extra actions as defined in the Ocean controller; it must be included here so that actions can be mapped to the proper hyperlink and verb. The controller and action args are mandatory. The app and context args are optional and will default to “*”. The last arg, service, defaults to the name of the service itself.



478
479
480
481
482
483
 
# File 'lib/ocean/api.rb', line 478

def self.authorization_string(extra_actions, controller, action, app="*", context="*", service=APP_NAME)
  app = '*' if app.blank?
  context = '*' if context.blank?
  hyperlink, verb = Api.map_authorization(extra_actions, controller, action)
  "#{service}:#{controller}:#{hyperlink}:#{verb}:#{app}:#{context}"
end

.ban(path) ⇒ Object

Makes an internal BAN call to all Varnish instances. The call is made in parallel. Varnish will only accept BAN requests coming from the local network.



341
342
343
344
345
346
347
348
349
350
351
 
# File 'lib/ocean/api.rb', line 341

def self.ban(path)
  hydra = Typhoeus::Hydra.hydra
  escaped_path = escape(path)
  escaped_path = '/' + escaped_path if escaped_path[0] != '/'
  VARNISH_CACHES.each do |host|
    url = "http://#{host}#{escaped_path}"
    request = Typhoeus::Request.new(url, method: :ban, headers: {})
    hydra.queue request
  end
  hydra.run
end

.basename_suffix ⇒ Object

Like adorn_basename, but returns only the suffix. Uses OCEAN_ENV and Rails.env.



57
58
59
 
# File 'lib/ocean/api.rb', line 57

def self.basename_suffix
  adorn_basename '', suffix_only: true, ocean_env: OCEAN_ENV, rails_env: Rails.env
end

.credentials(username = nil, password = nil) ⇒ Object

Encodes a username and password for authentication in the format used for standard HTTP authentication. The encoding can be reversed and is intended only to lightly mask the credentials so that they’re not immediately apparent when reading logs.



426
427
428
429
430
431
 
# File 'lib/ocean/api.rb', line 426

def self.credentials(username=nil, password=nil)
  raise "Only specifying the username is not allowed" if username && !password
  username ||= API_USER
  password ||= API_PASSWORD
  ::Base64.strict_encode64 "#{username}:#{password}"
end

.decode_credentials(encoded) ⇒ Object

Takes encoded credentials (e.g. by Api.encode_credentials) and returns a two-element array where the first element is the username and the second is the password. If the encoded credentials are missing or can’t be decoded properly, [“”, “”] is returned. This allows you to write:

un, pw = Api.decode_credentials(creds)
raise "Please supply your username and password" if un.blank? || pw.blank?



443
444
445
446
447
 
# File 'lib/ocean/api.rb', line 443

def self.decode_credentials(encoded)
  return ["", ""] unless encoded
  username, password = ::Base64.decode64(encoded).split(':', 2)
  [username || "", password || ""]
end

.escape(path) ⇒ Object

This escapes BAN request paths, which is needed as they are regexes.



357
358
359
 
# File 'lib/ocean/api.rb', line 357

def self.escape(path)
  URI.escape(path, Regexp.new("[^/$\\-+_.!~*'()a-zA-Z0-9]"))
end

.internalize_uri(uri) ⇒ Object

Convert an external Ocean URI to an internal one. This is mainly useful in master and staging, as the https URIs aren’t available for CronJobs, etc.



13
14
15
 
# File 'lib/ocean/api.rb', line 13

def self.internalize_uri(uri) #, ocean_env=OCEAN_ENV)
  uri.sub(OCEAN_API_URL, INTERNAL_OCEAN_API_URL)
end

.map_authorization(extra_actions, controller, action) ⇒ Object

Returns the hyperlink and HTTP method to use for an action in a certain controller. First, the DEFAULT_ACTIONS are searched, then any extra actions defined for the controller. Raises an exception if the action can’t be found.



509
510
511
512
513
 
# File 'lib/ocean/api.rb', line 509

def self.map_authorization(extra_actions, controller, action)
  DEFAULT_ACTIONS[action] ||
  extra_actions[controller][action] ||
  raise("The #{controller} lacks an extra_action declaration for #{action}")
end

.permitted?(token, args = {}) ⇒ Boolean

Performs authorisation against the Auth service. The token must be a token received as a result of a prior authentication operation. The args should be in the form

query: "service:controller:hyperlink:verb:app:context"

e.g.

Api.permitted?(@service_token, query: "cms:texts:self:GET:*:*")

Api.authorization_string can be used to produce the query string.

Returns the HTTP response as-is, allowing the caller to examine the status code and messages, and also the body.

Returns:

  • (Boolean) 


465
466
467
468
 
# File 'lib/ocean/api.rb', line 465

def self.permitted?(token, args={})
  raise unless token
  Api.request "/v1/authentications/#{token}", :get, args: args
end

.purge(*args) ⇒ Object

Makes an internal PURGE call to all Varnish instances. The call is made in parallel. Varnish will only accept PURGE requests coming from the local network.



326
327
328
329
330
331
332
333
334
 
# File 'lib/ocean/api.rb', line 326

def self.purge(*args)
  hydra = Typhoeus::Hydra.hydra
  VARNISH_CACHES.each do |host|
    url = "http://#{host}#{path}"
    request = Typhoeus::Request.new(url, method: :purge, headers: {})
    hydra.queue request
  end
  hydra.run
end

.request(url, http_method, args: nil, headers: {}, body: nil, credentials: nil, x_api_token: , reauthentication: true, ssl_verifypeer: true, ssl_verifyhost: 2, retries: 0, backoff_time: 1, backoff_rate: 0.9, backoff_max: 30, x_metadata: , timeout: 0, &postprocessor) ⇒ Object

Api.request is designed to be the lowest common denominator for making any kind of HTTP request, except parallel requests which will have a similar method (to which BAN and PURGE requests to Varnish constitute a special case).

In its present form it assumes the request is a JSON one. Eventually, keyword args will provide abstract control of content type.

url is the URL to which the request will be made. http_method is the HTTP method to use (:post, :get, :head, :put, :delete, etc). args, if given, should be a hash of query arguments to add to the URL. headers, if given, is a hash of extra HTTP headers for the request. body, if given, is the body of the request (:post, :put) as a string. +credentials, if given, are the credentials to use when authenticating. x_api_token, if given, is a string which will be used as an X-API-Token header. reauthentication (true by default), controls whether 400 and 419 will trigger reauth. ssl_verifypeer (true by default), controls SSL peer verification. ssl_verifyhost (2 by default), controls SSL host verification. timeout sets the timeout of the request. It defaults to 0, meaning no timeout.

Automatic retries for GET requests are available:

retries, if given and > 0, specifies the number of retries to perform for GET requests. Defaults to 0, meaning no retries. backoff_time (default 1) the initial time to wait between retries. backoff_rate (default 0.9) the rate at which the time is increased. backoff_max (default 30) the maximum time to wait between retries.

The backoff time is increased after each wait period by the product of itself and backoff_rate. The backoff time is capped by backoff_max. The default time and rate settings will generate the progression 1, 1.9, 3.61, 6.859, 13.0321, 24.76099, 30, 30, 30, etc. To disable waiting between retries entirely, set backoff_time to zero.

If a postprocessor block is given, it will be called with the result of the response; whatever it returns will be returned as the result of the Api request.

Api.request can be called inside a simultaneously block.



180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
 
# File 'lib/ocean/api.rb', line 180

def self.request(url, http_method, args: nil, headers: {}, body: nil,
                 credentials: nil,
                 x_api_token: headers['X-API-Token'],
                 reauthentication: true,
                 ssl_verifypeer: true, ssl_verifyhost: 2,
                 retries: 0, backoff_time: 1, backoff_rate: 0.9, backoff_max: 30,
                 x_metadata: Thread.current[:metadata],
                 timeout: 0,
                 &postprocessor)
  # Set up the request
  headers['Accept'] = "application/json"
  headers['Content-Type'] = "application/json" if [:post, :put].include?(http_method)
  headers['User-Agent'] = "Ocean"
  x_api_token =  Api.authenticate(*Api.decode_credentials(credentials)) if x_api_token.blank? && credentials.present?
  headers['X-API-Token'] = x_api_token if x_api_token.present?
  headers['X-Metadata'] = x_metadata if x_metadata.present?

  retries = 0 unless ["GET", "HEAD"].include? http_method.to_s.upcase

  @hydra ||= Typhoeus::Hydra.hydra

  request = nil
  response = nil
  response_getter = lambda { response }

  url = url.first == "/" ? "#{INTERNAL_OCEAN_API_URL}#{url}" : Api.internalize_uri(url)

  # This is a Proc which when run queues the request and schedules retries
  enqueue_request = lambda do
    # First construct a request. It will not be sent yet.
    request = Typhoeus::Request.new(url,
                                    method: http_method,
                                    headers: headers,
                                    params: args,
                                    body: body,
                                    ssl_verifypeer: ssl_verifypeer,
                                    ssl_verifyhost: ssl_verifyhost,
                                    timeout: timeout)
    # Define a callback to process the response and do retries
    request.on_complete do |typhoeus_response|
      response = Response.new typhoeus_response
      case response.status
      when 100..199
        enqueue_request.call  # Ignore and retry
      when 200..299, 304
        # Success, call the post-processor if any. Any further Api.request
        # calls done by the post-processor will use the same response
        # accessors, which means the final result will be what the last
        # post-processor to finish returns.
        response = postprocessor.call(response) if postprocessor
      when 300..399
        nil  # Done, redirect
      when 400, 419
        if reauthentication && x_api_token.present?
          # Re-authenticate and retry
          if credentials
            x_api_token = Api.authenticate(*Api.decode_credentials(credentials))
            headers['X-API-Token'] = x_api_token
          else
            Api.reset_service_token
            headers['X-API-Token'] = Api.service_token
          end
          reauthentication = false   # This prevents us from ending up here twice
          enqueue_request.call
        else
          nil  # Done, fail
        end
      when 400..499
        nil  # Done, fail
      else
        # We got a 5xx. Retry if there are any retries left
        if retries > 0
          retries -= 1
          sleep backoff_time
          backoff_time = [backoff_time + backoff_time * backoff_rate, backoff_max].min
          enqueue_request.call
        else
          nil  # Done, don't retry
        end
      end
    end

    # Finally, queue the request (and its callback) for execution
    @hydra.queue request
    # Return nil, to emphasise that the side effects are what's important
    nil
  end

  # So create and enqueue the request
  enqueue_request.call
  # If doing parallel calls, return a lambda which returns the final response
  return response_getter if Api.simultaneously?
  # Run it now. Blocks until completed, possibly after any number of retries
  @hydra.run
  if response.is_a?(Response)
    # Raise any exceptions
    raise Api::TimeoutError, "Api.request timed out" if response.timed_out?
    raise Api::NoResponseError, "Api.request could not obtain a response" if response.status == 0
  end
  response
end

.reset_service_token ⇒ Object

Resets the service token, causing the next call to Api.service_token to re-authenticate.



374
375
376
 
# File 'lib/ocean/api.rb', line 374

def self.reset_service_token
  @service_token = nil
end

.run_async_job(href = nil, method = nil, job: nil, **keywords, &block) ⇒ Object

Takes the same args as .async_job_body, post an AsyncJob and returns the Response. The AsyncJob is always created as the service ApiUser; the job carries its own credentials and token, specified as keywords :credentials and :token. If you have precomputed the job description hash, pass it using the keyword :job as the only parameter.

A block may be given. It will be called with any TimeoutError or NoResponseError, which allows Api.run_async_job to be used very tersely in controllers, e.g.:

Api.run_async_job(...) do |e|
  render_api_error 422, e.message
  return
end



596
597
598
599
600
601
602
603
 
# File 'lib/ocean/api.rb', line 596

def self.run_async_job(href=nil, method=nil, job: nil, **keywords, &block)
  job ||= async_job_body(href, method, **keywords)
  Api.request "/v1/async_jobs", :post,
    body: job.to_json, x_api_token: Api.service_token
rescue TimeoutError, NoResponseError => e
  raise e unless block
  block.call e
end

.send_mail(from: "nobody@#{BASE_DOMAIN}", to: nil, subject: nil, plaintext: nil, html: nil, plaintext_url: nil, html_url: nil, substitutions: nil) ⇒ Object

Send an email asynchronously. The Mailer role is required.



519
520
521
522
523
524
525
526
527
528
529
530
531
 
# File 'lib/ocean/api.rb', line 519

def self.send_mail(from: "nobody@#{BASE_DOMAIN}", to: nil,
                   subject: nil,
                   plaintext: nil, html: nil,
                   plaintext_url: nil, html_url: nil, substitutions: nil)
  Api.request "/v1/mails", :post,
    x_api_token: Api.service_token, credentials: Api.credentials,
    body: {
      from: from, to: to, subject: subject,
      plaintext: plaintext, html: html,
      plaintext_url: plaintext_url, html_url: html_url,
      substitutions: substitutions
    }.to_json
end

.service_token ⇒ Object

This method returns the current token. If no current token has been obtained, authenticates.



366
367
368
 
# File 'lib/ocean/api.rb', line 366

def self.service_token
  @service_token ||= authenticate
end

.simultaneously(&block) ⇒ Object

Api.simultaneously is used for making requests in parallel. For example:

results = Api.simultaneously do |r|
  r << Api.request("http://foo.bar", :get, retries: 3)
  r << Api.request("http://foo.baz", :get, retries: 3)
  r << Api.request("http://foo.quux", :get, retries: 3)
end

The value returned is an array of responses from the Api.request calls.

TimeoutErrors and NoResponseError will not be raised in parallel mode.

Only Api.request is supported at the present time. Api::RemoteResource will follow, but requires a major internal reorganisation of Api::RemoteResource. It’s underway.



299
300
301
302
303
304
305
306
307
308
309
310
 
# File 'lib/ocean/api.rb', line 299

def self.simultaneously (&block)
  raise "block required" unless block
  @inside_simultaneously = true
  results = []
  @hydra = nil
  block.call(results)
  Typhoeus::Config.memoize = true
  @hydra.run if @hydra
  results.map(&:call)
ensure
  @inside_simultaneously = false
end

.simultaneously? ⇒ Boolean

Returns:

  • (Boolean) 


313
314
315
 
# File 'lib/ocean/api.rb', line 313

def self.simultaneously?
  @inside_simultaneously ||= false
end

.version_for(resource_name) ⇒ Object

When given a symbol or string naming a resource, returns a string such as v1 naming the latest version for the resource.



22
23
24
 
# File 'lib/ocean/api.rb', line 22

def self.version_for(resource_name)
  API_VERSIONS[resource_name.to_s] || API_VERSIONS['_default']
end