Class: Dynamoid::AdapterPlugin::AwsSdkV3

Inherits:
Object
  • Object
show all
Defined in:
lib/dynamoid/adapter_plugin/aws_sdk_v3.rb,
lib/dynamoid/adapter_plugin/aws_sdk_v3/scan.rb,
lib/dynamoid/adapter_plugin/aws_sdk_v3/query.rb,
lib/dynamoid/adapter_plugin/aws_sdk_v3/table.rb,
lib/dynamoid/adapter_plugin/aws_sdk_v3/create_table.rb,
lib/dynamoid/adapter_plugin/aws_sdk_v3/item_updater.rb,
lib/dynamoid/adapter_plugin/aws_sdk_v3/batch_get_item.rb,
lib/dynamoid/adapter_plugin/aws_sdk_v3/middleware/limit.rb,
lib/dynamoid/adapter_plugin/aws_sdk_v3/execute_statement.rb,
lib/dynamoid/adapter_plugin/aws_sdk_v3/middleware/backoff.rb,
lib/dynamoid/adapter_plugin/aws_sdk_v3/middleware/start_key.rb,
lib/dynamoid/adapter_plugin/aws_sdk_v3/until_past_table_status.rb

Overview

Note: Don’t use keyword arguments in public methods as far as method calls on adapter are delegated to the plugin.

There are breaking changes in Ruby related to delegating keyword arguments so we have decided just to avoid them when use delegation.

eregon.me/blog/2019/11/10/the-delegation-challenge-of-ruby27.html

Defined Under Namespace

Modules: Middleware Classes: BatchGetItem, CreateTable, ExecuteStatement, ItemUpdater, Query, Scan, Table, UntilPastTableStatus

Constant Summary collapse

EQ =
'EQ'
RANGE_MAP =
{
  range_greater_than: 'GT',
  range_less_than:    'LT',
  range_gte:          'GE',
  range_lte:          'LE',
  range_begins_with:  'BEGINS_WITH',
  range_between:      'BETWEEN',
  range_eq:           'EQ'
}.freeze
FIELD_MAP =
{
  eq:           'EQ',
  ne:           'NE',
  gt:           'GT',
  lt:           'LT',
  gte:          'GE',
  lte:          'LE',
  begins_with:  'BEGINS_WITH',
  between:      'BETWEEN',
  in:           'IN',
  contains:     'CONTAINS',
  not_contains: 'NOT_CONTAINS',
  null:         'NULL',
  not_null:     'NOT_NULL',
}.freeze
HASH_KEY =
'HASH'
RANGE_KEY =
'RANGE'
STRING_TYPE =
'S'
NUM_TYPE =
'N'
BINARY_TYPE =
'B'
TABLE_STATUSES =
{
  creating: 'CREATING',
  updating: 'UPDATING',
  deleting: 'DELETING',
  active: 'ACTIVE'
}.freeze
PARSE_TABLE_STATUS =
lambda { |resp, lookup = :table|
  # lookup is table for describe_table API
  # lookup is table_description for create_table API
  #   because Amazon, damnit.
  resp.send(lookup).table_status
}
BATCH_WRITE_ITEM_REQUESTS_LIMIT =
25
CONNECTION_CONFIG_OPTIONS =
%i[endpoint region http_continue_timeout http_idle_timeout http_open_timeout http_read_timeout].freeze

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Instance Attribute Details

#table_cacheObject (readonly)

Returns the value of attribute table_cache.



73
74
75
# File 'lib/dynamoid/adapter_plugin/aws_sdk_v3.rb', line 73

def table_cache
  @table_cache
end

Class Method Details

.attribute_value_list(operator, value) ⇒ Object

Build an array of values for Condition Is used in ScanFilter and QueryFilter docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Condition.html

Parameters:

  • operator (String)

    value of RANGE_MAP or FIELD_MAP hash, e.g. “EQ”, “LT” etc

  • value (Object)

    scalar value or array/set



80
81
82
83
84
85
86
87
88
89
90
91
92
# File 'lib/dynamoid/adapter_plugin/aws_sdk_v3.rb', line 80

def self.attribute_value_list(operator, value)
  # For BETWEEN and IN operators we should keep value as is (it should be already an array)
  # NULL and NOT_NULL require absence of attribute list
  # For all the other operators we wrap the value with array
  # https://docs.aws.amazon.com/en_us/amazondynamodb/latest/developerguide/LegacyConditionalParameters.Conditions.html
  if %w[BETWEEN IN].include?(operator)
    [value].flatten
  elsif %w[NULL NOT_NULL].include?(operator)
    nil
  else
    [value]
  end
end

Instance Method Details

#batch_delete_item(options) ⇒ Object

Delete many items at once from DynamoDB. More efficient than delete each item individually.

or

Dynamoid::AdapterPlugin::AwsSdkV3.batch_delete_item('table1' => [['hk1', 'rk2'], ['hk1', 'rk2']]]))

See:

TODO handle rejections because of internal processing failures

Examples:

Delete IDs 1 and 2 from the table testtable

Dynamoid::AdapterPlugin::AwsSdk.batch_delete_item('table1' => ['1', '2'])

Parameters:

  • options (Hash)

    the hash of tables and IDs to delete



245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
# File 'lib/dynamoid/adapter_plugin/aws_sdk_v3.rb', line 245

def batch_delete_item(options)
  requests = []

  options.each_pair do |table_name, ids|
    table = describe_table(table_name)

    ids.each_slice(BATCH_WRITE_ITEM_REQUESTS_LIMIT) do |sliced_ids|
      delete_requests = sliced_ids.map do |id|
        { delete_request: { key: key_stanza(table, *id) } }
      end

      requests << { table_name => delete_requests }
    end
  end

  requests.each do |items|
    client.batch_write_item(
      request_items: items,
      return_consumed_capacity: 'TOTAL',
      return_item_collection_metrics: 'SIZE'
    )
  end
rescue Aws::DynamoDB::Errors::ConditionalCheckFailedException => e
  raise Dynamoid::Errors::ConditionalCheckFailedException, e
end

#batch_get_item(table_names_with_ids, options = {}, &block) ⇒ Hash

Get many items at once from DynamoDB. More efficient than getting each item individually.

If optional block is passed ‘nil` will be returned and the block will be called for each read batch of items, meaning once per batch.

Block receives parameters:

  • hash with items like ‘{ table_name: [items]}`

  • and boolean flag is true if there are some unprocessed keys, otherwise false.

@todo: Provide support for passing options to underlying batch_get_item

Examples:

Retrieve IDs 1 and 2 from the table testtable

Dynamoid::AdapterPlugin::AwsSdkV3.batch_get_item('table1' => ['1', '2'])

Pass block to receive each batch

Dynamoid::AdapterPlugin::AwsSdkV3.batch_get_item('table1' => ids) do |hash, bool|
  puts hash['table1']

  if bool
    puts 'there are unprocessed keys'
  end
end

Parameters:

  • table_names_with_ids (Hash)

    the hash of tables and IDs to retrieve

  • options (Hash) (defaults to: {})

    to be passed to underlying BatchGet call

  • block (Proc)

    optional block can be passed to handle each batch of items

Returns:

Since:

  • 1.0.0



224
225
226
227
228
229
# File 'lib/dynamoid/adapter_plugin/aws_sdk_v3.rb', line 224

def batch_get_item(table_names_with_ids, options = {}, &block)
  tables_with_ids = table_names_with_ids.transform_keys do |name|
    describe_table(name)
  end
  BatchGetItem.new(client, tables_with_ids, options).call(&block)
end

#batch_write_item(table_name, objects, options = {}) {|true|false| ... } ⇒ Object

Puts multiple items in one table

If optional block is passed it will be called for each written batch of items, meaning once per batch. Block receives boolean flag which is true if there are some unprocessed items, otherwise false.

See:

Examples:

Saves several items to the table testtable

Dynamoid::AdapterPlugin::AwsSdkV3.batch_write_item('table1', [{ id: '1', name: 'a' }, { id: '2', name: 'b'}])

Pass block

Dynamoid::AdapterPlugin::AwsSdkV3.batch_write_item('table1', items) do |bool|
  if bool
    puts 'there are unprocessed items'
  end
end

Parameters:

  • table_name (String)

    the name of the table

  • objects (Array)

    to be processed

  • options (Hash) (defaults to: {})

    additional options

Yields:

  • (true|false)

    invokes an optional block with argument - whether there are unprocessed items



164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
# File 'lib/dynamoid/adapter_plugin/aws_sdk_v3.rb', line 164

def batch_write_item(table_name, objects, options = {})
  items = objects.map { |o| sanitize_item(o) }

  while items.present?
    batch = items.shift(BATCH_WRITE_ITEM_REQUESTS_LIMIT)
    requests = batch.map { |item| { put_request: { item: item } } }

    response = client.batch_write_item(
      {
        request_items: {
          table_name => requests
        },
        return_consumed_capacity: 'TOTAL',
        return_item_collection_metrics: 'SIZE'
      }.merge!(options)
    )

    yield(response.unprocessed_items.present?) if block_given?

    if response.unprocessed_items.present?
      items += response.unprocessed_items[table_name].map { |r| r.put_request.item }
    end
  end
rescue Aws::DynamoDB::Errors::ConditionalCheckFailedException => e
  raise Dynamoid::Errors::ConditionalCheckFailedException, e
end

#clientObject

Return the client object.

Since:

  • 1.0.0



137
138
139
# File 'lib/dynamoid/adapter_plugin/aws_sdk_v3.rb', line 137

def client
  @client
end

#connect!Aws::DynamoDB::Client

Establish the connection to DynamoDB.

Returns:

  • (Aws::DynamoDB::Client)

    the DynamoDB connection



97
98
99
100
# File 'lib/dynamoid/adapter_plugin/aws_sdk_v3.rb', line 97

def connect!
  @client = Aws::DynamoDB::Client.new(connection_config)
  @table_cache = {}
end

#connection_configObject



102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
# File 'lib/dynamoid/adapter_plugin/aws_sdk_v3.rb', line 102

def connection_config
  @connection_hash = {}

  (Dynamoid::Config.settings.compact.keys & CONNECTION_CONFIG_OPTIONS).each do |option|
    @connection_hash[option] = Dynamoid::Config.send(option)
  end

  # if credentials are passed, they already contain access key & secret key
  if Dynamoid::Config.credentials?
    @connection_hash[:credentials] = Dynamoid::Config.credentials
  else
    # otherwise, pass access key & secret key for credentials creation
    if Dynamoid::Config.access_key?
      @connection_hash[:access_key_id] = Dynamoid::Config.access_key
    end
    if Dynamoid::Config.secret_key?
      @connection_hash[:secret_access_key] = Dynamoid::Config.secret_key
    end
  end

  @connection_hash[:logger] = Dynamoid::Config.logger
  @connection_hash[:log_level] = :debug

  # https://github.com/aws/aws-sdk-ruby/blob/master/gems/aws-sdk-core/lib/aws-sdk-core/plugins/logging.rb
  # https://github.com/aws/aws-sdk-ruby/blob/master/gems/aws-sdk-core/lib/aws-sdk-core/log/formatter.rb
  if Dynamoid::Config.log_formatter
    @connection_hash[:log_formatter] = Dynamoid::Config.log_formatter
  end

  @connection_hash
end

#count(table_name) ⇒ Object



560
561
562
# File 'lib/dynamoid/adapter_plugin/aws_sdk_v3.rb', line 560

def count(table_name)
  describe_table(table_name, true).item_count
end

#create_table(table_name, key = :id, options = {}) ⇒ Object

Create a table on DynamoDB. This usually takes a long time to complete.

Parameters:

  • table_name (String)

    the name of the table to create

  • key (Symbol) (defaults to: :id)

    the table’s primary key (defaults to :id)

  • options (Hash) (defaults to: {})

    provide a range key here if the table has a composite key

Options Hash (options):

Since:

  • 1.0.0



281
282
283
284
285
286
287
288
# File 'lib/dynamoid/adapter_plugin/aws_sdk_v3.rb', line 281

def create_table(table_name, key = :id, options = {})
  Dynamoid.logger.info "Creating #{table_name} table. This could take a while."
  CreateTable.new(client, table_name, key, options).call
  true
rescue Aws::DynamoDB::Errors::ResourceInUseException => e
  Dynamoid.logger.error "Table #{table_name} cannot be created as it already exists"
  false
end

#create_table_synchronously(table_name, key = :id, options = {}) ⇒ Object

Create a table on DynamoDB synchronously. This usually takes a long time to complete. CreateTable is normally an asynchronous operation. You can optionally define secondary indexes on the new table,

as part of the CreateTable operation.

If you want to create multiple tables with secondary indexes on them,

you must create the tables sequentially.

Only one table with secondary indexes can be

in the CREATING state at any given time.

See: docs.aws.amazon.com/sdkforruby/api/Aws/DynamoDB/Client.html#create_table-instance_method

Parameters:

  • table_name (String)

    the name of the table to create

  • key (Symbol) (defaults to: :id)

    the table’s primary key (defaults to :id)

  • options (Hash) (defaults to: {})

    provide a range key here if the table has a composite key

Options Hash (options):

Since:

  • 1.2.0



320
321
322
# File 'lib/dynamoid/adapter_plugin/aws_sdk_v3.rb', line 320

def create_table_synchronously(table_name, key = :id, options = {})
  create_table(table_name, key, options.merge(sync: true))
end

#delete_item(table_name, key, options = {}) ⇒ Object

Removes an item from DynamoDB.

@todo: Provide support for various options docs.aws.amazon.com/sdkforruby/api/Aws/DynamoDB/Client.html#delete_item-instance_method

Parameters:

  • table_name (String)

    the name of the table

  • key (String)

    the hash key of the item to delete

  • options (Hash) (defaults to: {})

    provide a range key here if the table has a composite key

Since:

  • 1.0.0



333
334
335
336
337
338
339
340
341
342
343
344
345
# File 'lib/dynamoid/adapter_plugin/aws_sdk_v3.rb', line 333

def delete_item(table_name, key, options = {})
  options ||= {}
  range_key = options[:range_key]
  conditions = options[:conditions]
  table = describe_table(table_name)
  client.delete_item(
    table_name: table_name,
    key: key_stanza(table, key, range_key),
    expected: expected_stanza(conditions)
  )
rescue Aws::DynamoDB::Errors::ConditionalCheckFailedException => e
  raise Dynamoid::Errors::ConditionalCheckFailedException, e
end

#delete_table(table_name, options = {}) ⇒ Object

Deletes an entire table from DynamoDB.

Parameters:

  • table_name (String)

    the name of the table to destroy

  • options (Hash) (defaults to: {})

    a customizable set of options

Options Hash (options):

  • sync (Boolean)

    Wait for table status check to raise ResourceNotFoundException

Since:

  • 1.0.0



353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
# File 'lib/dynamoid/adapter_plugin/aws_sdk_v3.rb', line 353

def delete_table(table_name, options = {})
  resp = client.delete_table(table_name: table_name)

  if options[:sync]
    status = PARSE_TABLE_STATUS.call(resp, :table_description)
    if status == TABLE_STATUSES[:deleting]
      UntilPastTableStatus.new(client, table_name, :deleting).call
    end
  end

  table_cache.delete(table_name)
rescue Aws::DynamoDB::Errors::ResourceInUseException => e
  Dynamoid.logger.error "Table #{table_name} cannot be deleted as it is in use"
  raise e
end

#delete_table_synchronously(table_name, options = {}) ⇒ Object



369
370
371
# File 'lib/dynamoid/adapter_plugin/aws_sdk_v3.rb', line 369

def delete_table_synchronously(table_name, options = {})
  delete_table(table_name, options.merge(sync: true))
end

#execute(statement, parameters = [], options = {}) ⇒ [] | Array[Hash] | Enumerator::Lazy[Hash]

Run PartiQL query.

Dynamoid.adapter.execute("SELECT * FROM users WHERE id = ?", ["758"])

Parameters:

  • statement (String)

    PartiQL statement

  • parameters (Array) (defaults to: [])

    a list of bind parameters

  • options (Hash) (defaults to: {})
  • [Boolean] (Hash)

    a customizable set of options

Returns:

  • ([] | Array[Hash] | Enumerator::Lazy[Hash])

    items when used a SELECT statement and empty Array otherwise



574
575
576
577
578
579
580
581
582
583
584
# File 'lib/dynamoid/adapter_plugin/aws_sdk_v3.rb', line 574

def execute(statement, parameters = [], options = {})
  items = ExecuteStatement.new(client, statement, parameters, options).call

  if items.is_a?(Array)
    items
  else
    items.lazy.flat_map { |array| array }
  end
rescue Aws::DynamoDB::Errors::ConditionalCheckFailedException
  []
end

#get_item(table_name, key, options = {}) ⇒ Hash

Fetches an item from DynamoDB.

Parameters:

  • table_name (String)

    the name of the table

  • key (String)

    the hash key of the item to find

  • options (Hash) (defaults to: {})

    provide a range key here if the table has a composite key

Returns:

  • (Hash)

    a hash representing the raw item in DynamoDB

Since:

  • 1.0.0



386
387
388
389
390
391
392
393
394
395
396
397
398
# File 'lib/dynamoid/adapter_plugin/aws_sdk_v3.rb', line 386

def get_item(table_name, key, options = {})
  options = options.dup
  options ||= {}

  table = describe_table(table_name)
  range_key = options.delete(:range_key)
  consistent_read = options.delete(:consistent_read)

  item = client.get_item(table_name: table_name,
                         key: key_stanza(table, key, range_key),
                         consistent_read: consistent_read)[:item]
  item ? item_to_hash(item) : nil
end

#list_tablesObject

List all tables on DynamoDB.

Since:

  • 1.0.0



434
435
436
437
438
439
440
441
442
443
444
# File 'lib/dynamoid/adapter_plugin/aws_sdk_v3.rb', line 434

def list_tables
  [].tap do |result|
    start_table_name = nil
    loop do
      result_page = client.list_tables exclusive_start_table_name: start_table_name
      start_table_name = result_page.last_evaluated_table_name
      result.concat result_page.table_names
      break unless start_table_name
    end
  end
end

#put_item(table_name, object, options = {}) ⇒ Object

Parameters:

  • table_name (String)

    the name of the table

  • object (Object)

    a hash or Dynamoid object to persist

Since:

  • 1.0.0



454
455
456
457
458
459
460
461
462
463
464
465
466
467
# File 'lib/dynamoid/adapter_plugin/aws_sdk_v3.rb', line 454

def put_item(table_name, object, options = {})
  options ||= {}
  item = sanitize_item(object)

  client.put_item(
    {
      table_name: table_name,
      item: item,
      expected: expected_stanza(options)
    }.merge!(options)
  )
rescue Aws::DynamoDB::Errors::ConditionalCheckFailedException => e
  raise Dynamoid::Errors::ConditionalCheckFailedException, e
end

#query(table_name, options = {}) ⇒ Enumerable

Query the DynamoDB table. This employs DynamoDB’s indexes so is generally faster than scanning, but is only really useful for range queries, since it can only find by one hash key at once. Only provide one range key to the hash.

Parameters:

  • table_name (String)

    the name of the table

  • options (Hash) (defaults to: {})

    the options to query the table with

Options Hash (options):

  • :hash_value (String)

    the value of the hash key to find

  • :range_between (Number, Number)

    find the range key within this range

  • :range_greater_than (Number)

    find range keys greater than this

  • :range_less_than (Number)

    find range keys less than this

  • :range_gte (Number)

    find range keys greater than or equal to this

  • :range_lte (Number)

    find range keys less than or equal to this

Returns:

  • (Enumerable)

    matching items

Since:

  • 1.0.0



487
488
489
490
491
492
493
494
495
496
497
498
# File 'lib/dynamoid/adapter_plugin/aws_sdk_v3.rb', line 487

def query(table_name, options = {})
  Enumerator.new do |yielder|
    table = describe_table(table_name)

    Query.new(client, table, options).call.each do |page|
      yielder.yield(
        page.items.map { |item| item_to_hash(item) },
        last_evaluated_key: page.last_evaluated_key
      )
    end
  end
end

#query_count(table_name, options = {}) ⇒ Object



500
501
502
503
504
505
506
507
# File 'lib/dynamoid/adapter_plugin/aws_sdk_v3.rb', line 500

def query_count(table_name, options = {})
  table = describe_table(table_name)
  options[:select] = 'COUNT'

  Query.new(client, table, options).call
    .map(&:count)
    .reduce(:+)
end

#scan(table_name, conditions = {}, options = {}) ⇒ Enumerable

Scan the DynamoDB table. This is usually a very slow operation as it naively filters all data on the DynamoDB servers.

@todo: Provide support for various options docs.aws.amazon.com/sdkforruby/api/Aws/DynamoDB/Client.html#scan-instance_method

Parameters:

  • table_name (String)

    the name of the table

  • conditions (Hash) (defaults to: {})

    a hash of attributes: matching records will be returned by the scan

Returns:

  • (Enumerable)

    matching items

Since:

  • 1.0.0



520
521
522
523
524
525
526
527
528
529
530
531
# File 'lib/dynamoid/adapter_plugin/aws_sdk_v3.rb', line 520

def scan(table_name, conditions = {}, options = {})
  Enumerator.new do |yielder|
    table = describe_table(table_name)

    Scan.new(client, table, conditions, options).call.each do |page|
      yielder.yield(
        page.items.map { |item| item_to_hash(item) },
        last_evaluated_key: page.last_evaluated_key
      )
    end
  end
end

#scan_count(table_name, conditions = {}, options = {}) ⇒ Object



533
534
535
536
537
538
539
540
# File 'lib/dynamoid/adapter_plugin/aws_sdk_v3.rb', line 533

def scan_count(table_name, conditions = {}, options = {})
  table = describe_table(table_name)
  options[:select] = 'COUNT'

  Scan.new(client, table, conditions, options).call
    .map(&:count)
    .reduce(:+)
end

#truncate(table_name) ⇒ Object

Truncates all records in the given table

Parameters:

  • table_name (String)

    the name of the table

Since:

  • 1.0.0



548
549
550
551
552
553
554
555
556
557
558
# File 'lib/dynamoid/adapter_plugin/aws_sdk_v3.rb', line 548

def truncate(table_name)
  table = describe_table(table_name)
  hk    = table.hash_key
  rk    = table.range_key

  ids = scan(table_name, {}, {}).flat_map { |i| i }.map do |attributes|
    rk ? [attributes[hk], attributes[rk.to_sym]] : attributes[hk]
  end

  batch_delete_item(table_name => ids)
end

#update_item(table_name, key, options = {}) ⇒ Object

Edits an existing item’s attributes, or adds a new item to the table if it does not already exist. You can put, delete, or add attribute values

Parameters:

  • table_name (String)

    the name of the table

  • key (String)

    the hash key of the item to find

  • options (Hash) (defaults to: {})

    provide a range key here if the table has a composite key

Returns:

  • new attributes for the record



409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
# File 'lib/dynamoid/adapter_plugin/aws_sdk_v3.rb', line 409

def update_item(table_name, key, options = {})
  options = options.dup

  range_key = options.delete(:range_key)
  conditions = options.delete(:conditions)
  table = describe_table(table_name)

  item_updater = ItemUpdater.new(table, key, range_key)
  yield(item_updater)

  raise "non-empty options: #{options}" unless options.empty?

  result = client.update_item(table_name: table_name,
                              key: key_stanza(table, key, range_key),
                              attribute_updates: item_updater.attribute_updates,
                              expected: expected_stanza(conditions),
                              return_values: 'ALL_NEW')
  item_to_hash(result[:attributes])
rescue Aws::DynamoDB::Errors::ConditionalCheckFailedException => e
  raise Dynamoid::Errors::ConditionalCheckFailedException, e
end

#update_time_to_live(table_name, attribute) ⇒ Object



290
291
292
293
294
295
296
297
298
299
300
# File 'lib/dynamoid/adapter_plugin/aws_sdk_v3.rb', line 290

def update_time_to_live(table_name, attribute)
  request = {
    table_name: table_name,
    time_to_live_specification: {
      attribute_name: attribute,
      enabled: true,
    }
  }

  client.update_time_to_live(request)
end