Module: Google::Cloud::Dataflow

Defined in:
lib/google/cloud/dataflow.rb,
lib/google/cloud/dataflow/version.rb

Constant Summary collapse

VERSION = 
"0.6.2"

Class Method Summary collapse

Class Method Details

.configure {|::Google::Cloud.configure.dataflow| ... } ⇒ ::Google::Cloud::Config

Configure the google-cloud-dataflow library.

The following configuration parameters are supported:

  • credentials (type: String, Hash, Google::Auth::Credentials) - The path to the keyfile as a String, the contents of the keyfile as a Hash, or a Google::Auth::Credentials object.
  • lib_name (type: String) - The library name as recorded in instrumentation and logging.
  • lib_version (type: String) - The library version as recorded in instrumentation and logging.
  • interceptors (type: Array<GRPC::ClientInterceptor>) - An array of interceptors that are run before calls are executed.
  • timeout (type: Numeric) - Default timeout in seconds.
  • metadata (type: Hash{Symbol=>String}) - Additional headers to be sent with the call.
  • retry_policy (type: Hash) - The retry policy. The value is a hash with the following keys:
    • :initial_delay (type: Numeric) - The initial delay in seconds.
    • :max_delay (type: Numeric) - The max delay in seconds.
    • :multiplier (type: Numeric) - The incremental backoff multiplier.
    • :retry_codes (type: Array<String>) - The error codes that should trigger a retry.

Yields:

Returns:

  • (::Google::Cloud::Config) —

    The default configuration used by this library



499
500
501
502
503
 
# File 'lib/google/cloud/dataflow.rb', line 499

def self.configure
  yield ::Google::Cloud.configure.dataflow if block_given?

  ::Google::Cloud.configure.dataflow
end

.flex_templates_service(version: :v1beta3, transport: :grpc, &block) ⇒ ::Object

Create a new client object for FlexTemplatesService.

By default, this returns an instance of Google::Cloud::Dataflow::V1beta3::FlexTemplatesService::Client for a gRPC client for version V1beta3 of the API. However, you can specify a different API version by passing it in the version parameter. If the FlexTemplatesService service is supported by that API version, and the corresponding gem is available, the appropriate versioned client will be returned. You can also specify a different transport by passing :rest or :grpc in the transport parameter.

Raises an exception if the currently installed versioned client gem for the given API version does not support the given transport of the FlexTemplatesService service. You can determine whether the method will succeed by calling flex_templates_service_available?.

About FlexTemplatesService

Provides a service for Flex templates.

Parameters:

  • version (::String, ::Symbol) (defaults to: :v1beta3) —

    The API version to connect to. Optional. Defaults to :v1beta3.

  • transport (:grpc, :rest) (defaults to: :grpc) —

    The transport to use. Defaults to :grpc.

Returns:

  • (::Object) —

    A client object for the specified version.



428
429
430
431
432
433
434
435
436
437
438
 
# File 'lib/google/cloud/dataflow.rb', line 428

def self.flex_templates_service version: :v1beta3, transport: :grpc, &block
  require "google/cloud/dataflow/#{version.to_s.downcase}"

  package_name = Google::Cloud::Dataflow
                 .constants
                 .select { |sym| sym.to_s.downcase == version.to_s.downcase.tr("_", "") }
                 .first
  service_module = Google::Cloud::Dataflow.const_get(package_name).const_get(:FlexTemplatesService)
  service_module = service_module.const_get(:Rest) if transport == :rest
  service_module.const_get(:Client).new(&block)
end

.flex_templates_service_available?(version: :v1beta3, transport: :grpc) ⇒ boolean

Determines whether the FlexTemplatesService service is supported by the current client. If true, you can retrieve a client object by calling flex_templates_service. If false, that method will raise an exception. This could happen if the given API version does not exist or does not support the FlexTemplatesService service, or if the versioned client gem needs an update to support the FlexTemplatesService service.

Parameters:

  • version (::String, ::Symbol) (defaults to: :v1beta3) —

    The API version to connect to. Optional. Defaults to :v1beta3.

  • transport (:grpc, :rest) (defaults to: :grpc) —

    The transport to use. Defaults to :grpc.

Returns:

  • (boolean) —

    Whether the service is available.



452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
 
# File 'lib/google/cloud/dataflow.rb', line 452

def self.flex_templates_service_available? version: :v1beta3, transport: :grpc
  require "google/cloud/dataflow/#{version.to_s.downcase}"
  package_name = Google::Cloud::Dataflow
                 .constants
                 .select { |sym| sym.to_s.downcase == version.to_s.downcase.tr("_", "") }
                 .first
  return false unless package_name
  service_module = Google::Cloud::Dataflow.const_get package_name
  return false unless service_module.const_defined? :FlexTemplatesService
  service_module = service_module.const_get :FlexTemplatesService
  if transport == :rest
    return false unless service_module.const_defined? :Rest
    service_module = service_module.const_get :Rest
  end
  service_module.const_defined? :Client
rescue ::LoadError
  false
end

.jobs(version: :v1beta3, transport: :grpc, &block) ⇒ ::Object

Create a new client object for Jobs.

By default, this returns an instance of Google::Cloud::Dataflow::V1beta3::Jobs::Client for a gRPC client for version V1beta3 of the API. However, you can specify a different API version by passing it in the version parameter. If the Jobs service is supported by that API version, and the corresponding gem is available, the appropriate versioned client will be returned. You can also specify a different transport by passing :rest or :grpc in the transport parameter.

Raises an exception if the currently installed versioned client gem for the given API version does not support the given transport of the Jobs service. You can determine whether the method will succeed by calling jobs_available?.

About Jobs

Provides a method to create and modify Google Cloud Dataflow jobs. A Job is a multi-stage computation graph run by the Cloud Dataflow service.

Parameters:

  • version (::String, ::Symbol) (defaults to: :v1beta3) —

    The API version to connect to. Optional. Defaults to :v1beta3.

  • transport (:grpc, :rest) (defaults to: :grpc) —

    The transport to use. Defaults to :grpc.

Returns:

  • (::Object) —

    A client object for the specified version.



146
147
148
149
150
151
152
153
154
155
156
 
# File 'lib/google/cloud/dataflow.rb', line 146

def self.jobs version: :v1beta3, transport: :grpc, &block
  require "google/cloud/dataflow/#{version.to_s.downcase}"

  package_name = Google::Cloud::Dataflow
                 .constants
                 .select { |sym| sym.to_s.downcase == version.to_s.downcase.tr("_", "") }
                 .first
  service_module = Google::Cloud::Dataflow.const_get(package_name).const_get(:Jobs)
  service_module = service_module.const_get(:Rest) if transport == :rest
  service_module.const_get(:Client).new(&block)
end

.jobs_available?(version: :v1beta3, transport: :grpc) ⇒ boolean

Determines whether the Jobs service is supported by the current client. If true, you can retrieve a client object by calling jobs. If false, that method will raise an exception. This could happen if the given API version does not exist or does not support the Jobs service, or if the versioned client gem needs an update to support the Jobs service.

Parameters:

  • version (::String, ::Symbol) (defaults to: :v1beta3) —

    The API version to connect to. Optional. Defaults to :v1beta3.

  • transport (:grpc, :rest) (defaults to: :grpc) —

    The transport to use. Defaults to :grpc.

Returns:

  • (boolean) —

    Whether the service is available.



170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
 
# File 'lib/google/cloud/dataflow.rb', line 170

def self.jobs_available? version: :v1beta3, transport: :grpc
  require "google/cloud/dataflow/#{version.to_s.downcase}"
  package_name = Google::Cloud::Dataflow
                 .constants
                 .select { |sym| sym.to_s.downcase == version.to_s.downcase.tr("_", "") }
                 .first
  return false unless package_name
  service_module = Google::Cloud::Dataflow.const_get package_name
  return false unless service_module.const_defined? :Jobs
  service_module = service_module.const_get :Jobs
  if transport == :rest
    return false unless service_module.const_defined? :Rest
    service_module = service_module.const_get :Rest
  end
  service_module.const_defined? :Client
rescue ::LoadError
  false
end

.messages(version: :v1beta3, transport: :grpc, &block) ⇒ ::Object

Create a new client object for Messages.

By default, this returns an instance of Google::Cloud::Dataflow::V1beta3::Messages::Client for a gRPC client for version V1beta3 of the API. However, you can specify a different API version by passing it in the version parameter. If the Messages service is supported by that API version, and the corresponding gem is available, the appropriate versioned client will be returned. You can also specify a different transport by passing :rest or :grpc in the transport parameter.

Raises an exception if the currently installed versioned client gem for the given API version does not support the given transport of the Messages service. You can determine whether the method will succeed by calling messages_available?.

About Messages

The Dataflow Messages API is used for monitoring the progress of Dataflow jobs.

Parameters:

  • version (::String, ::Symbol) (defaults to: :v1beta3) —

    The API version to connect to. Optional. Defaults to :v1beta3.

  • transport (:grpc, :rest) (defaults to: :grpc) —

    The transport to use. Defaults to :grpc.

Returns:

  • (::Object) —

    A client object for the specified version.



217
218
219
220
221
222
223
224
225
226
227
 
# File 'lib/google/cloud/dataflow.rb', line 217

def self.messages version: :v1beta3, transport: :grpc, &block
  require "google/cloud/dataflow/#{version.to_s.downcase}"

  package_name = Google::Cloud::Dataflow
                 .constants
                 .select { |sym| sym.to_s.downcase == version.to_s.downcase.tr("_", "") }
                 .first
  service_module = Google::Cloud::Dataflow.const_get(package_name).const_get(:Messages)
  service_module = service_module.const_get(:Rest) if transport == :rest
  service_module.const_get(:Client).new(&block)
end

.messages_available?(version: :v1beta3, transport: :grpc) ⇒ boolean

Determines whether the Messages service is supported by the current client. If true, you can retrieve a client object by calling messages. If false, that method will raise an exception. This could happen if the given API version does not exist or does not support the Messages service, or if the versioned client gem needs an update to support the Messages service.

Parameters:

  • version (::String, ::Symbol) (defaults to: :v1beta3) —

    The API version to connect to. Optional. Defaults to :v1beta3.

  • transport (:grpc, :rest) (defaults to: :grpc) —

    The transport to use. Defaults to :grpc.

Returns:

  • (boolean) —

    Whether the service is available.



241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
 
# File 'lib/google/cloud/dataflow.rb', line 241

def self.messages_available? version: :v1beta3, transport: :grpc
  require "google/cloud/dataflow/#{version.to_s.downcase}"
  package_name = Google::Cloud::Dataflow
                 .constants
                 .select { |sym| sym.to_s.downcase == version.to_s.downcase.tr("_", "") }
                 .first
  return false unless package_name
  service_module = Google::Cloud::Dataflow.const_get package_name
  return false unless service_module.const_defined? :Messages
  service_module = service_module.const_get :Messages
  if transport == :rest
    return false unless service_module.const_defined? :Rest
    service_module = service_module.const_get :Rest
  end
  service_module.const_defined? :Client
rescue ::LoadError
  false
end

.metrics(version: :v1beta3, transport: :grpc, &block) ⇒ ::Object

Create a new client object for Metrics.

By default, this returns an instance of Google::Cloud::Dataflow::V1beta3::Metrics::Client for a gRPC client for version V1beta3 of the API. However, you can specify a different API version by passing it in the version parameter. If the Metrics service is supported by that API version, and the corresponding gem is available, the appropriate versioned client will be returned. You can also specify a different transport by passing :rest or :grpc in the transport parameter.

Raises an exception if the currently installed versioned client gem for the given API version does not support the given transport of the Metrics service. You can determine whether the method will succeed by calling metrics_available?.

About Metrics

The Dataflow Metrics API lets you monitor the progress of Dataflow jobs.

Parameters:

  • version (::String, ::Symbol) (defaults to: :v1beta3) —

    The API version to connect to. Optional. Defaults to :v1beta3.

  • transport (:grpc, :rest) (defaults to: :grpc) —

    The transport to use. Defaults to :grpc.

Returns:

  • (::Object) —

    A client object for the specified version.



288
289
290
291
292
293
294
295
296
297
298
 
# File 'lib/google/cloud/dataflow.rb', line 288

def self.metrics version: :v1beta3, transport: :grpc, &block
  require "google/cloud/dataflow/#{version.to_s.downcase}"

  package_name = Google::Cloud::Dataflow
                 .constants
                 .select { |sym| sym.to_s.downcase == version.to_s.downcase.tr("_", "") }
                 .first
  service_module = Google::Cloud::Dataflow.const_get(package_name).const_get(:Metrics)
  service_module = service_module.const_get(:Rest) if transport == :rest
  service_module.const_get(:Client).new(&block)
end

.metrics_available?(version: :v1beta3, transport: :grpc) ⇒ boolean

Determines whether the Metrics service is supported by the current client. If true, you can retrieve a client object by calling metrics. If false, that method will raise an exception. This could happen if the given API version does not exist or does not support the Metrics service, or if the versioned client gem needs an update to support the Metrics service.

Parameters:

  • version (::String, ::Symbol) (defaults to: :v1beta3) —

    The API version to connect to. Optional. Defaults to :v1beta3.

  • transport (:grpc, :rest) (defaults to: :grpc) —

    The transport to use. Defaults to :grpc.

Returns:

  • (boolean) —

    Whether the service is available.



312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
 
# File 'lib/google/cloud/dataflow.rb', line 312

def self.metrics_available? version: :v1beta3, transport: :grpc
  require "google/cloud/dataflow/#{version.to_s.downcase}"
  package_name = Google::Cloud::Dataflow
                 .constants
                 .select { |sym| sym.to_s.downcase == version.to_s.downcase.tr("_", "") }
                 .first
  return false unless package_name
  service_module = Google::Cloud::Dataflow.const_get package_name
  return false unless service_module.const_defined? :Metrics
  service_module = service_module.const_get :Metrics
  if transport == :rest
    return false unless service_module.const_defined? :Rest
    service_module = service_module.const_get :Rest
  end
  service_module.const_defined? :Client
rescue ::LoadError
  false
end

.snapshots(version: :v1beta3, transport: :grpc, &block) ⇒ ::Object

Create a new client object for Snapshots.

By default, this returns an instance of Google::Cloud::Dataflow::V1beta3::Snapshots::Client for a gRPC client for version V1beta3 of the API. However, you can specify a different API version by passing it in the version parameter. If the Snapshots service is supported by that API version, and the corresponding gem is available, the appropriate versioned client will be returned. You can also specify a different transport by passing :rest or :grpc in the transport parameter.

Raises an exception if the currently installed versioned client gem for the given API version does not support the given transport of the Snapshots service. You can determine whether the method will succeed by calling snapshots_available?.

About Snapshots

Provides methods to manage snapshots of Google Cloud Dataflow jobs.

Parameters:

  • version (::String, ::Symbol) (defaults to: :v1beta3) —

    The API version to connect to. Optional. Defaults to :v1beta3.

  • transport (:grpc, :rest) (defaults to: :grpc) —

    The transport to use. Defaults to :grpc.

Returns:

  • (::Object) —

    A client object for the specified version.



75
76
77
78
79
80
81
82
83
84
85
 
# File 'lib/google/cloud/dataflow.rb', line 75

def self.snapshots version: :v1beta3, transport: :grpc, &block
  require "google/cloud/dataflow/#{version.to_s.downcase}"

  package_name = Google::Cloud::Dataflow
                 .constants
                 .select { |sym| sym.to_s.downcase == version.to_s.downcase.tr("_", "") }
                 .first
  service_module = Google::Cloud::Dataflow.const_get(package_name).const_get(:Snapshots)
  service_module = service_module.const_get(:Rest) if transport == :rest
  service_module.const_get(:Client).new(&block)
end

.snapshots_available?(version: :v1beta3, transport: :grpc) ⇒ boolean

Determines whether the Snapshots service is supported by the current client. If true, you can retrieve a client object by calling snapshots. If false, that method will raise an exception. This could happen if the given API version does not exist or does not support the Snapshots service, or if the versioned client gem needs an update to support the Snapshots service.

Parameters:

  • version (::String, ::Symbol) (defaults to: :v1beta3) —

    The API version to connect to. Optional. Defaults to :v1beta3.

  • transport (:grpc, :rest) (defaults to: :grpc) —

    The transport to use. Defaults to :grpc.

Returns:

  • (boolean) —

    Whether the service is available.



99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
 
# File 'lib/google/cloud/dataflow.rb', line 99

def self.snapshots_available? version: :v1beta3, transport: :grpc
  require "google/cloud/dataflow/#{version.to_s.downcase}"
  package_name = Google::Cloud::Dataflow
                 .constants
                 .select { |sym| sym.to_s.downcase == version.to_s.downcase.tr("_", "") }
                 .first
  return false unless package_name
  service_module = Google::Cloud::Dataflow.const_get package_name
  return false unless service_module.const_defined? :Snapshots
  service_module = service_module.const_get :Snapshots
  if transport == :rest
    return false unless service_module.const_defined? :Rest
    service_module = service_module.const_get :Rest
  end
  service_module.const_defined? :Client
rescue ::LoadError
  false
end

.templates_service(version: :v1beta3, transport: :grpc, &block) ⇒ ::Object

Create a new client object for TemplatesService.

By default, this returns an instance of Google::Cloud::Dataflow::V1beta3::TemplatesService::Client for a gRPC client for version V1beta3 of the API. However, you can specify a different API version by passing it in the version parameter. If the TemplatesService service is supported by that API version, and the corresponding gem is available, the appropriate versioned client will be returned. You can also specify a different transport by passing :rest or :grpc in the transport parameter.

Raises an exception if the currently installed versioned client gem for the given API version does not support the given transport of the TemplatesService service. You can determine whether the method will succeed by calling templates_service_available?.

About TemplatesService

Provides a method to create Cloud Dataflow jobs from templates.

Parameters:

  • version (::String, ::Symbol) (defaults to: :v1beta3) —

    The API version to connect to. Optional. Defaults to :v1beta3.

  • transport (:grpc, :rest) (defaults to: :grpc) —

    The transport to use. Defaults to :grpc.

Returns:

  • (::Object) —

    A client object for the specified version.



358
359
360
361
362
363
364
365
366
367
368
 
# File 'lib/google/cloud/dataflow.rb', line 358

def self.templates_service version: :v1beta3, transport: :grpc, &block
  require "google/cloud/dataflow/#{version.to_s.downcase}"

  package_name = Google::Cloud::Dataflow
                 .constants
                 .select { |sym| sym.to_s.downcase == version.to_s.downcase.tr("_", "") }
                 .first
  service_module = Google::Cloud::Dataflow.const_get(package_name).const_get(:TemplatesService)
  service_module = service_module.const_get(:Rest) if transport == :rest
  service_module.const_get(:Client).new(&block)
end

.templates_service_available?(version: :v1beta3, transport: :grpc) ⇒ boolean

Determines whether the TemplatesService service is supported by the current client. If true, you can retrieve a client object by calling templates_service. If false, that method will raise an exception. This could happen if the given API version does not exist or does not support the TemplatesService service, or if the versioned client gem needs an update to support the TemplatesService service.

Parameters:

  • version (::String, ::Symbol) (defaults to: :v1beta3) —

    The API version to connect to. Optional. Defaults to :v1beta3.

  • transport (:grpc, :rest) (defaults to: :grpc) —

    The transport to use. Defaults to :grpc.

Returns:

  • (boolean) —

    Whether the service is available.



382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
 
# File 'lib/google/cloud/dataflow.rb', line 382

def self.templates_service_available? version: :v1beta3, transport: :grpc
  require "google/cloud/dataflow/#{version.to_s.downcase}"
  package_name = Google::Cloud::Dataflow
                 .constants
                 .select { |sym| sym.to_s.downcase == version.to_s.downcase.tr("_", "") }
                 .first
  return false unless package_name
  service_module = Google::Cloud::Dataflow.const_get package_name
  return false unless service_module.const_defined? :TemplatesService
  service_module = service_module.const_get :TemplatesService
  if transport == :rest
    return false unless service_module.const_defined? :Rest
    service_module = service_module.const_get :Rest
  end
  service_module.const_defined? :Client
rescue ::LoadError
  false
end