Class: Python::Pickle::Deserializer Private

Inherits:
Object
  • Object
show all
Defined in:
lib/python/pickle/deserializer.rb

Overview

This class is part of a private API. You should avoid using this class if possible, as it may be removed or be changed in the future.

Handles deserializing a stream of Python Pickle instructions.

Constant Summary collapse

OBJECT_CLASS =

This constant is part of a private API. You should avoid using this constant if possible, as it may be removed or be changed in the future.

The Python object class.

PyClass.new('__builtins__','object')

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(constants: nil, extensions: nil) ⇒ Deserializer

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Initializes the deserializer.

Parameters:

  • extensions (Hash{Integer => Object}) (defaults to: nil) —

    A Hash of registered extension IDs and their Objects.

  • constants (Hash{String => Hash{String => Class,Method}}) (defaults to: nil) —

    An optional mapping of custom Python constant names to Ruby classes or methods.



110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
 
# File 'lib/python/pickle/deserializer.rb', line 110

def initialize(constants: nil, extensions: nil)
  @meta_stack = []
  @stack = []
  @memo  = []

  @constants = {
    # Python 2.x
    'copy_reg' => {
      '_reconstructor' => method(:copyreg_reconstructor)
    },

    '__builtin__' => {
      'object'    => OBJECT_CLASS,
      'bytearray' => ByteArray
    },

    # Python 3.x
    'builtins' => {
      'object'    => OBJECT_CLASS,
      'bytearray' => ByteArray
    }
  }
  @constants.merge!(constants) if constants

  @extensions = {}
  @extensions.merge!(extensions) if extensions
end

Instance Attribute Details

#constants ⇒ Hash{String => Hash{String => Class,Method}} (readonly)

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Mapping of Python constants to Ruby classes and methods.

Returns:

  • (Hash{String => Hash{String => Class,Method}}) 


90
91
92
 
# File 'lib/python/pickle/deserializer.rb', line 90

def constants
  @constants
end

#extensions ⇒ Hash{Integer => Object} (readonly)

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Mapping of Python Pickle extension codes to Ruby objects.

Returns:

  • (Hash{Integer => Object}) 


95
96
97
 
# File 'lib/python/pickle/deserializer.rb', line 95

def extensions
  @extensions
end

#memo ⇒ Array (readonly)

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

The memo dictionary.

Returns:

  • (Array) 


85
86
87
 
# File 'lib/python/pickle/deserializer.rb', line 85

def memo
  @memo
end

#meta_stack ⇒ Array (readonly)

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

The meta-stack for saving/restoring #stack.

Returns:

  • (Array) 


75
76
77
 
# File 'lib/python/pickle/deserializer.rb', line 75

def meta_stack
  @meta_stack
end

#stack ⇒ Array (readonly)

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

The object stack.

Returns:

  • (Array) 


80
81
82
 
# File 'lib/python/pickle/deserializer.rb', line 80

def stack
  @stack
end

Instance Method Details

#copyreg_reconstructor(klass, super_class, init_arg) ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Implements Python's copyreg._reconstructor function for Python Pickle protocol 0 compatibility.

Parameters:

  • class (PyClass, Class) —

    The Python or Ruby class to be initialized.

  • super_class (PyClass) —

    The Python super-class of the class.

  • init_arg (Array, nil) —

    The argument(s) that will be passed to the class'es new method.



455
456
457
 
# File 'lib/python/pickle/deserializer.rb', line 455

def copyreg_reconstructor(klass,super_class,init_arg)
  klass.new(*init_arg)
end

#execute(instruction) ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Executes a Python Pickle instruction.



161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
 
# File 'lib/python/pickle/deserializer.rb', line 161

def execute(instruction)
  case instruction
  when Instructions::Proto,
       Instructions::Frame
    # no-op
  when Instructions::Get,
       Instructions::BinGet,
       Instructions::LongBinGet
    execute_get(instruction)
  when Instructions::MARK     then execute_mark
  when Instructions::POP_MARK then execute_pop_mark
  when Instructions::DUP      then execute_dup
  when Instructions::Put,
       Instructions::BinPut
    execute_put(instruction)
  when Instructions::POP     then execute_pop
  when Instructions::MEMOIZE then execute_memoize
  when Instructions::Ext1,
       Instructions::Ext2,
       Instructions::Ext4
    execute_ext(instruction)
  when Instructions::NONE     then execute_none
  when Instructions::NEWTRUE  then execute_newtrue
  when Instructions::NEWFALSE then execute_newfalse
  when Instructions::Float,
       Instructions::BinFloat,
       Instructions::Int,
       Instructions::BinInt1,
       Instructions::Long,
       Instructions::Long1,
       Instructions::Long4,
       Instructions::BinBytes,
       Instructions::ShortBinBytes,
       Instructions::BinBytes8,
       Instructions::String,
       Instructions::BinString,
       Instructions::ShortBinString,
       Instructions::BinUnicode,
       Instructions::ShortBinUnicode,
       Instructions::BinUnicode8
    @stack.push(instruction.value)
  when Instructions::ByteArray8   then execute_byte_array8(instruction)
  when Instructions::EMPTY_LIST   then execute_empty_list
  when Instructions::EMPTY_TUPLE  then execute_empty_tuple
  when Instructions::TUPLE        then execute_tuple
  when Instructions::EMPTY_DICT   then execute_empty_dict
  when Instructions::APPEND       then execute_append
  when Instructions::APPENDS      then execute_appends
  when Instructions::LIST         then execute_list
  when Instructions::TUPLE1       then execute_tuple1
  when Instructions::TUPLE2       then execute_tuple2
  when Instructions::TUPLE3       then execute_tuple3
  when Instructions::DICT         then execute_dict
  when Instructions::Global       then execute_global(instruction)
  when Instructions::STACK_GLOBAL then execute_stack_global
  when Instructions::NEWOBJ       then execute_newobj
  when Instructions::NEWOBJ_EX    then execute_newobj_ex
  when Instructions::REDUCE       then execute_reduce
  when Instructions::BUILD        then execute_build
  when Instructions::SETITEM      then execute_setitem
  when Instructions::SETITEMS     then execute_setitems
  when Instructions::STOP
    return :halt, @stack.pop
  else
    raise(NotImplementedError,"instruction is currently not fully supported: #{instruction.inspect}")
  end
end

#execute_append ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Executes an APPEND instruction.



367
368
369
370
371
372
373
374
375
376
 
# File 'lib/python/pickle/deserializer.rb', line 367

def execute_append
  item = @stack.pop
  list = @stack.last

  unless list.kind_of?(Array)
    raise(DeserializationError,"cannot append element #{item.inspect} onto a non-Array: #{list.inspect}")
  end

  list.push(item)
end

#execute_appends ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Executes an APPENDS instruction.



381
382
383
384
385
386
387
388
389
390
 
# File 'lib/python/pickle/deserializer.rb', line 381

def execute_appends
  items = pop_meta_stack
  list  = @stack.last

  unless list.kind_of?(Array)
    raise(DeserializationError,"cannot append elements #{items.inspect} onto a non-Array: #{list.inspect}")
  end

  list.concat(items)
end

#execute_build ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Executes a BUILD instruction.



549
550
551
552
553
554
555
556
557
558
559
560
 
# File 'lib/python/pickle/deserializer.rb', line 549

def execute_build
  arg    = @stack.pop
  object = @stack.last

  if object.respond_to?(:__setstate__)
    object.__setstate__(arg)
  elsif object.kind_of?(Hash)
    object.merge!(arg)
  else
    raise(DeserializationError,"cannot execute BUILD on an object that does not define a __setstate__ method or is not a Hash: #{object.inspect}")
  end
end

#execute_byte_array8(instruction) ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Executes a BYTEARRAY8 instruction.



331
332
333
 
# File 'lib/python/pickle/deserializer.rb', line 331

def execute_byte_array8(instruction)
  @stack.push(ByteArray.new(instruction.value))
end

#execute_dict ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Executes a DICT instruction.



430
431
432
433
434
435
436
437
438
439
440
 
# File 'lib/python/pickle/deserializer.rb', line 430

def execute_dict
  pairs    = pop_meta_stack
  new_dict = {}

  until pairs.empty?
    key, value = pairs.pop(2)
    new_dict[key] = value
  end

  @stack.push(new_dict)
end

#execute_dup ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Executes a DUP instruction.



258
259
260
 
# File 'lib/python/pickle/deserializer.rb', line 258

def execute_dup
  @stack.push(@stack.last)
end

#execute_empty_dict ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Executes an EMPTY_DICT instruction.



360
361
362
 
# File 'lib/python/pickle/deserializer.rb', line 360

def execute_empty_dict
  @stack.push({})
end

#execute_empty_list ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Executes the EMPTY_LIST instruction.



338
339
340
 
# File 'lib/python/pickle/deserializer.rb', line 338

def execute_empty_list
  @stack.push([])
end

#execute_empty_tuple ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Executes the EMPTY_TUPLE instruction.



345
346
347
 
# File 'lib/python/pickle/deserializer.rb', line 345

def execute_empty_tuple
  @stack.push(Tuple.new)
end

#execute_ext(instruction) ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Executes a EXT1, EXT2, or EXT4 instruction.

Parameters:

Raises:



298
299
300
301
302
303
304
305
 
# File 'lib/python/pickle/deserializer.rb', line 298

def execute_ext(instruction)
  ext_id = instruction.value
  object = @extensions.fetch(ext_id) do
    raise(DeserializationError,"unknown extension ID: #{ext_id.inspect}")
  end

  @stack.push(object)
end

#execute_get(instruction) ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Executes a GET, BINGET, or LONG_BINGET instruction.

Parameters:



235
236
237
238
239
 
# File 'lib/python/pickle/deserializer.rb', line 235

def execute_get(instruction)
  index = instruction.value

  @stack.push(@memo[index])
end

#execute_global(instruction) ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Executes a GLOBAL instruction.

Parameters:



485
486
487
488
489
490
491
 
# File 'lib/python/pickle/deserializer.rb', line 485

def execute_global(instruction)
  namespace = instruction.namespace
  name      = instruction.name
  constant  = resolve_constant(namespace,name)

  @stack.push(constant)
end

#execute_list ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Executes a LIST instruction.



395
396
397
398
 
# File 'lib/python/pickle/deserializer.rb', line 395

def execute_list
  elements = pop_meta_stack
  @stack.push(elements)
end

#execute_mark ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Executes a MARK instruction.



244
245
246
 
# File 'lib/python/pickle/deserializer.rb', line 244

def execute_mark
  push_meta_stack
end

#execute_memoize ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Executes the MEMOIZE instruction.



285
286
287
 
# File 'lib/python/pickle/deserializer.rb', line 285

def execute_memoize
  @memo.push(@stack.last)
end

#execute_newfalse ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Executes a NEWFALSE instruction.



324
325
326
 
# File 'lib/python/pickle/deserializer.rb', line 324

def execute_newfalse
  @stack.push(false)
end

#execute_newobj ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Executes a NEWOBJ instruction.



506
507
508
509
510
511
 
# File 'lib/python/pickle/deserializer.rb', line 506

def execute_newobj
  py_class, args = @stack.pop(2)
  py_object = py_class.new(*args)

  @stack.push(py_object)
end

#execute_newobj_ex ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Executes a NEWOBJ_EX instruction.



516
517
518
519
520
521
522
523
524
525
526
527
 
# File 'lib/python/pickle/deserializer.rb', line 516

def execute_newobj_ex
  py_class, args, kwargs = @stack.pop(3)
  py_object = if kwargs
                kwargs = kwargs.transform_keys(&:to_sym)

                py_class.new(*args,**kwargs)
              else
                py_class.new(*args)
              end

  @stack.push(py_object)
end

#execute_newtrue ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Executes a NEWTRUE instruction.



317
318
319
 
# File 'lib/python/pickle/deserializer.rb', line 317

def execute_newtrue
  @stack.push(true)
end

#execute_none ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Executes a NONE instruction.



310
311
312
 
# File 'lib/python/pickle/deserializer.rb', line 310

def execute_none
  @stack.push(nil)
end

#execute_pop ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Executes the POP instruction.



278
279
280
 
# File 'lib/python/pickle/deserializer.rb', line 278

def execute_pop
  @stack.pop
end

#execute_pop_mark ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Executes a POP_MARK instruction.



251
252
253
 
# File 'lib/python/pickle/deserializer.rb', line 251

def execute_pop_mark
  pop_meta_stack
end

#execute_put(instruction) ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Executes a PUT, BINPUT, or LONG_BINPUT instruction.

Parameters:



268
269
270
271
272
273
 
# File 'lib/python/pickle/deserializer.rb', line 268

def execute_put(instruction)
  index = instruction.value
  value = @stack.last

  @memo[index] = value
end

#execute_reduce ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Executes a REDUCE instruction.



532
533
534
535
536
537
538
539
540
541
542
543
544
 
# File 'lib/python/pickle/deserializer.rb', line 532

def execute_reduce
  callable, arg = @stack.pop(2)
  object = case callable
           when PyClass, Class
             callable.new(*arg)
           when Method
             callable.call(*arg)
           else
             raise(DeserializationError,"cannot execute REDUCE on a non-class: #{callable.inspect}")
           end

  @stack.push(object)
end

#execute_setitem ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Executes a SETITEM instruction.



565
566
567
568
569
570
571
572
573
574
 
# File 'lib/python/pickle/deserializer.rb', line 565

def execute_setitem
  key, value = @stack.pop(2)
  dict = @stack.last

  unless dict.kind_of?(Hash)
    raise(DeserializationError,"cannot set key (#{key.inspect}) and value (#{value.inspect}) into non-Hash: #{dict.inspect}")
  end

  dict[key] = value
end

#execute_setitems ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Executes a SETITEMS instruction.



579
580
581
582
583
584
585
586
587
588
589
590
591
 
# File 'lib/python/pickle/deserializer.rb', line 579

def execute_setitems
  pairs = pop_meta_stack
  dict  = @stack.last

  unless dict.kind_of?(Hash)
    raise(DeserializationError,"cannot set key value pairs (#{pairs.inspect}) into non-Hash: #{dict.inspect}")
  end

  until pairs.empty?
    key, value = pairs.pop(2)
    dict[key] = value
  end
end

#execute_stack_global ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Executes a STACK_GLOBAL instruction.



496
497
498
499
500
501
 
# File 'lib/python/pickle/deserializer.rb', line 496

def execute_stack_global
  namespace, name = @stack.pop(2)
  constant = resolve_constant(namespace,name)

  @stack.push(constant)
end

#execute_tuple ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Executes a TUPLE instruction.



352
353
354
355
 
# File 'lib/python/pickle/deserializer.rb', line 352

def execute_tuple
  items = Tuple.new(pop_meta_stack)
  @stack.push(items)
end

#execute_tuple1 ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Executes a TUPLE1 instruction.



403
404
405
406
407
 
# File 'lib/python/pickle/deserializer.rb', line 403

def execute_tuple1
  new_tuple = Tuple.new(@stack.pop(1))

  @stack.push(new_tuple)
end

#execute_tuple2 ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Executes a TUPLE2 instruction.



412
413
414
415
416
 
# File 'lib/python/pickle/deserializer.rb', line 412

def execute_tuple2
  new_tuple = Tuple.new(@stack.pop(2))

  @stack.push(new_tuple)
end

#execute_tuple3 ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Executes a TUPLE3 instruction.



421
422
423
424
425
 
# File 'lib/python/pickle/deserializer.rb', line 421

def execute_tuple3
  new_tuple = Tuple.new(@stack.pop(3))

  @stack.push(new_tuple)
end

#pop_meta_stack ⇒ Array

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Pops a previous stack off of #meta_stack and restores #stack.

Returns:

  • (Array) —

    The current #stack values will be returned.



152
153
154
155
156
 
# File 'lib/python/pickle/deserializer.rb', line 152

def pop_meta_stack
  items  = @stack
  @stack = (@meta_stack.pop || [])
  return items
end

#push_meta_stack ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Pushes the #stack onto the #meta_stack.



141
142
143
144
 
# File 'lib/python/pickle/deserializer.rb', line 141

def push_meta_stack
  @meta_stack.push(@stack)
  @stack = []
end

#resolve_constant(namespace, name) ⇒ Class, ...

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Resolves a constant that exists in a Python namespace.

Parameters:

  • namespace (String) —

    The namespace name.

  • name (String) —

    The name of the constant within the namespace.

Returns:

  • (Class, PyClass, Method, nil) —

    The resolved class or method.



471
472
473
474
475
476
477
 
# File 'lib/python/pickle/deserializer.rb', line 471

def resolve_constant(namespace,name)
  constant = if (mod = @constants[namespace])
               mod[name]
             end

  return constant || PyClass.new(namespace,name)
end