Module: Chainer

Defined in:
lib/chainer.rb,
lib/chainer/cuda.rb,
lib/chainer/link.rb,
lib/chainer/version.rb,
lib/chainer/function.rb,
lib/chainer/reporter.rb,
lib/chainer/variable.rb,
lib/chainer/optimizer.rb,
lib/chainer/parameter.rb,
lib/chainer/serializer.rb,
lib/chainer/utils/conv.rb,
lib/chainer/utils/math.rb,
lib/chainer/initializer.rb,
lib/chainer/utils/array.rb,
lib/chainer/configuration.rb,
lib/chainer/testing/array.rb,
lib/chainer/training/util.rb,
lib/chainer/variable_node.rb,
lib/chainer/datasets/cifar.rb,
lib/chainer/datasets/mnist.rb,
lib/chainer/gradient_check.rb,
lib/chainer/hyperparameter.rb,
lib/chainer/utils/variable.rb,
lib/chainer/dataset/convert.rb,
lib/chainer/gradient_method.rb,
lib/chainer/optimizers/adam.rb,
lib/chainer/dataset/download.rb,
lib/chainer/dataset/iterator.rb,
lib/chainer/training/trainer.rb,
lib/chainer/training/updater.rb,
lib/chainer/initializers/init.rb,
lib/chainer/utils/initializer.rb,
lib/chainer/training/extension.rb,
lib/chainer/initializers/normal.rb,
lib/chainer/serializers/marshal.rb,
lib/chainer/initializers/constant.rb,
lib/chainer/datasets/tuple_dataset.rb,
lib/chainer/links/model/classifier.rb,
lib/chainer/functions/math/identity.rb,
lib/chainer/functions/noise/dropout.rb,
lib/chainer/links/connection/linear.rb,
lib/chainer/optimizers/momentum_sgd.rb,
lib/chainer/functions/activation/relu.rb,
lib/chainer/functions/activation/tanh.rb,
lib/chainer/functions/math/basic_math.rb,
lib/chainer/iterators/serial_iterator.rb,
lib/chainer/training/standard_updater.rb,
lib/chainer/training/triggers/interval.rb,
lib/chainer/functions/connection/linear.rb,
lib/chainer/functions/activation/sigmoid.rb,
lib/chainer/functions/pooling/pooling_2d.rb,
lib/chainer/training/extensions/snapshot.rb,
lib/chainer/functions/evaluation/accuracy.rb,
lib/chainer/training/extensions/evaluator.rb,
lib/chainer/training/extensions/log_report.rb,
lib/chainer/functions/activation/leaky_relu.rb,
lib/chainer/links/connection/convolution_2d.rb,
lib/chainer/functions/activation/log_softmax.rb,
lib/chainer/functions/pooling/max_pooling_2d.rb,
lib/chainer/training/extensions/print_report.rb,
lib/chainer/training/extensions/progress_bar.rb,
lib/chainer/functions/loss/mean_squared_error.rb,
lib/chainer/functions/connection/convolution_2d.rb,
lib/chainer/functions/loss/softmax_cross_entropy.rb,
lib/chainer/functions/pooling/average_pooling_2d.rb,
lib/chainer/training/extensions/exponential_shift.rb,
lib/chainer/links/normalization/batch_normalization.rb,
lib/chainer/functions/normalization/batch_normalization.rb

Defined Under Namespace

Modules: Dataset, Datasets, Functions, Initializers, Iterators, Links, Optimizers, ReportService, Serializers, Testing, Training, Utils Classes: AbstractSerializer, Chain, ChainList, Configuration, Deserializer, DictSummary, Function, GradientMethod, Hyperparameter, HyperparameterProxy, Initializer, Link, Optimizer, Parameter, Reporter, Serializer, Summary, UpdateRule, Variable, VariableNode, WeightDecay

Constant Summary collapse

VERSION = 
"0.3.2"

Class Method Summary collapse

Class Method Details

._as_tuple(x) ⇒ Object 



60
61
62
63
64
65
66
 
# File 'lib/chainer/gradient_check.rb', line 60

def _as_tuple(x)
  if x.is_a? Array
    return x
  else
    return [x]
  end
end

._copy_arrays(xs) ⇒ Object 



2
3
4
5
 
# File 'lib/chainer/gradient_check.rb', line 2

def _copy_arrays(xs)
  xp = Chainer::get_array_module(*xs)
  xs.map{|x| (x.is_a? Numo::NArray) ? x.dup : x}
end

.check_backward(func, x_data, y_grad, params = [], eps: 0.001, atol: 1e-5, rtol: 1e-4, no_grads: nil, dtype: nil) ⇒ Object

Note:

func is called many times to get numerical gradients for all inputs. This function doesn’t work correctly when func behaves randomly as it gets different gradients.

Test backward procedure of a given function.

This function automatically check backward-process of given function. For example, when you have a Chainer::Function class MyFunc, that gets two arguments and returns one value, you can make its test like this:

def test_my_func(self):
  func = MyFunc()
  x1_data = Numo::NArray[...]
  x2_data = Numo::NArray[...]
  gy_data = Numo::NArray[...]
  check_backward(func, [x1_data, x2_data], gy_data)

This method creates Chainer::Variable objects with x_data and calls func with the Chainer::Variable s to get its result as Chainer::Variable. Then, it sets y_grad array to grad attribute of the result and calls backward method to get gradients of the inputs. To check correctness of the gradients, the function calls numerical_grad to calculate numerically the gradients and compares the types of gradients with Chainer::Testing.assert_allclose. If input objects (x1_data or/and x2_data in this example) represent integer variables, their gradients are ignored.

You can simplify a test when MyFunc gets only one argument:

check_backward(func, x1_data, gy_data)

If MyFunc is a loss function which returns a zero-dimensional array, pass nil to gy_data. In this case, it sets 1 to grad attribute of the result:

check_backward(my_loss_func, [x1_data, x2_data], nil)

If MyFunc returns multiple outputs, pass all gradients for outputs as a Array:

gy1_data = Numo::NArray[...]
gy2_data = Numo::NArray[...]
check_backward(func, x1_data, [gy1_data, gy2_data])

You can also test a Chainer::Link. To check gradients of parameters of the link, set a Array of the parameters to params arguments:

check_backward(my_link, [x1_data, x2_data], gy_data, [my_link.W, my_link.b])

Note that params are not Numo::NArray s, but Chainer::Variables s.

Function objects are acceptable as func argument:

check_backward(lambda{|x1, x1| f(x1, x2)}, [x1_data, x2_data], gy_data)

Parameters:

  • func (Method, Proc) —

    A function which gets Chainer::Variable s and returns Chainer::Variable s. func must returns a Array of Chainer::Variable s or one Chainer::Variable. You can use Chainer::Function object, Chainer::Link object or a function satisfying the condition.

  • x_data (Numo::NArray or Array<Numo::NArray>) —

    A set of Numo::NArray s to be passed to func. If x_data is one Numo::NArray object, it is treated as (x_data,).

  • y_grad (Numo::NArray or Array<Numo::NArray> or nil) —

    A set of Numo::NArray s representing gradients of return-values of func. If y_grad is one Numo::NArray object, it is treated as (y_grad,). If func is a loss-function, y_grad should be set to nil.

  • params (Chainer::Variable or Array<Chainder::Variable>) (defaults to: []) —

    A set of Chainer::Variable s whose gradients are checked. When func is a Chainer::Link object, set its parameters as params. If params is one Chainer::Variable object, it is treated as (params,).

  • eps (Float) (defaults to: 0.001) —

    Epsilon value to be passed to numerical_grad.

  • atol (Float) (defaults to: 1e-5) —

    Absolute tolerance to be passed to Chainer::Testing.assert_allclose.

  • rtol (Float) (defaults to: 1e-4) —

    Relative tolerance to be passed to Chainer::Testing.assert_allclose.

  • no_grads (Array<Boolean>) (defaults to: nil) —

    Flag to skip variable for gradient assertion. It should be same length as x_data.

  • dtype (Numo::NArray.class) (defaults to: nil) —

    x_data and y_grad are casted to this dtype when calculating numerical gradients. Only float types and nil are allowed.

See Also:



154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
 
# File 'lib/chainer/gradient_check.rb', line 154

def check_backward(func, x_data, y_grad, params=[], eps: 0.001, atol: 1e-5, rtol: 1e-4, no_grads: nil, dtype: nil)
  x_data = _as_tuple(x_data)
  if !y_grad.nil?
    y_grad = _as_tuple(y_grad)
  end

  params = _as_tuple(params)
  xs = x_data.map{|x| Chainer::Variable.new(x)}
  y = func.(*xs)
  y = _as_tuple(y)
  y = Chainer::Functions::Math::Identity.identity(*y)
  y = _as_tuple(y)

  if !y_grad.nil?
    if (y).size != (y_grad).size
      raise TypeError, "`y_grad` must have the same length of output values"
    end

    y.zip(y_grad).each do |iy, igy|
      iy.grad = igy
    end
  else
    if (y).size != 1
      raise TypeError, "When `y_grad` is `nil`, the function must return azero-dimentional array"
    end
    y_grad = [1]
  end
  # We only need to call `backward` for one result `Chainer::Variable`.
  # `Chainer::Variable.backward` method calls `Chainer::Function.backward` of its creator.
  y[0].backward()

  if dtype.nil?
    casted_xs = x_data.map{|x| Chainer::Variable.new(x)}
  else
    if (dtype != Numo::DFloat) and (dtype != Numo::SFloat)
      raise TypeError, "`dtype` is allowed only float type"
    end
    if (params).size > 0
      raise TypeError, "`dtype` is available only if `params` is empty"
    end
    casted_xs = x_data.map{|x|
                  if x.class == Numo::DFloat or x.class == Numo::SFloat
                    Chainer::Variable.new(dtype.cast(x))
                  else
                    Chainer::Variable.new(x)
                  end
                }
  end

  f = lambda do |_|
    ys = func.(*casted_xs)
    ys = _as_tuple(ys)
    return ys.map{|y| y.data}.to_a
  end

  if no_grads.nil?
    no_grads = xs.map{|x| (x.dtype != Numo::DFloat) and (x.dtype != Numo::SFloat)}
  else
    if no_grads.size != xs.size
      raise TypeError, "Length of no_grads param and xs should be same."
    end
  end

  no_grads.zip(xs, casted_xs).each do |skip, x, cx|
    if skip
      raise unless x.grad.nil?
      next
    end
    gx, = numerical_grad(f, [cx.data], y_grad, eps)
    Chainer::Testing.assert_allclose(x.grad, gx, atol: atol, rtol: rtol)
    if dtype.nil?
      raise unless gx.class == x.grad.class
    else
      if ((gx.class != Numo::DFloat) and (gx.class != Numo::SFloat)) and (gx.class != dtype)
         raise
      end
    end
  end

  params.each do |p|
    gp, = numerical_grad(f, [p.data], y_grad, eps)
    Chainer::Testing.assert_allclose(p.grad, gp, atol: atol, rtol: rtol)
    raise unless gp.dtype === p.grad.dtype
  end
end

.configuration ⇒ Object 



79
80
81
 
# File 'lib/chainer.rb', line 79

def self.configuration
  @configuration ||= Configuration.new
end

.configure {|configuration| ... } ⇒ Object

Yields:



75
76
77
 
# File 'lib/chainer.rb', line 75

def self.configure
  yield(configuration)
end

.get_array_module(*args) ⇒ Numo::NArray

TODO:

CUDA is not supported, yet.

Gets an appropriate one from Numo::NArray or Cumo::NArray.

This is almost equivalent to Chainer::get_array_module. The differences are that this function can be used even if CUDA is not available and that it will return their data arrays’ array module for Chainer::Variable arguments.

Parameters:

  • args (Array<Chainer::Variable> or Array<Numo::NArray> or Array<Cumo::NArray>) —

    Values to determine whether Numo or Cumo should be used.

Returns:

  • (Numo::NArray) —

    Cumo::NArray or Numo::NArray is returned based on the types of the arguments.



14
15
16
 
# File 'lib/chainer/cuda.rb', line 14

def get_array_module(*args)
  return Numo::NArray
end

.numerical_grad(f, inputs, grad_outputs, eps = 0.001) ⇒ Array

Computes numerical gradient by finite differences.

This function is used to implement gradient check. For usage example, see unit tests of Chainer::Functions.

Parameters:

  • f (function) —

    Ruby function with no arguments that runs forward computation and returns the result.

  • inputs (Array<Arrays>) —

    Array of arrays that should be treated as inputs. Each element of them is slightly modified to realize numerical gradient by finite differences.

  • grad_outputs (Array<Arrays>) —

    Array of arrays that are treated as output gradients.

  • eps (Float) (defaults to: 0.001) —

    Epsilon value of finite differences.

Returns:

  • (Array) —

    Numerical gradient arrays corresponding to inputs.



22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
 
# File 'lib/chainer/gradient_check.rb', line 22

def numerical_grad(f, inputs, grad_outputs, eps=0.001)
  raise unless eps > 0
  inputs = inputs.to_a
  grad_outputs = grad_outputs.to_a
  xp = Numo::NArray
  grads = inputs.map{|x| x.new_zeros()}

  if inputs[0].ndim < 2
    tmp = [[inputs[0], grads[0]]]
  else
    tmp = (0...inputs[0].shape[0]).map{|i|[inputs[0][i, false], grads[0][i, false]]}
  end

  tmp.each do |x, gx|
    x.each_with_index{|xx, *i|
      orig = x[*i]   # hold original value
      x[*i] = orig + eps
      ys1 = _copy_arrays(f.call(x))
      x[*i] = orig - eps
      ys2 = _copy_arrays(f.call(x))
      x[*i] = orig

      ys1.zip(ys2, grad_outputs).each do |y1, y2, gy|
        if !gy.nil?
          if  ((y1 - y2) * gy).is_a? Numo::NArray
            dot = ((y1 - y2) * gy).sum()
          else
            dot = ((y1 - y2) * gy).inject(:+)
          end
          gx[*i] += dot / (2*eps).to_f
        end
      end
    }
  end

  return grads
end