Class: Enumerator

Inherits:
Object show all
Includes:
Enumerable
Defined in:
enumerator.c,
enumerator.c

Overview

A class which allows both internal and external iteration.

An Enumerator can be created by the following methods.

  • Object#to_enum

  • Object#enum_for

  • Enumerator.new

Most methods have two forms: a block form where the contents are evaluated for each item in the enumeration, and a non-block form which returns a new Enumerator wrapping the iteration.

enumerator = %w(one two three).each
puts enumerator.class # => Enumerator

enumerator.each_with_object("foo") do |item, obj|
  puts "#{obj}: #{item}"
end

# foo: one
# foo: two
# foo: three

enum_with_obj = enumerator.each_with_object("foo")
puts enum_with_obj.class # => Enumerator

enum_with_obj.each do |item, obj|
  puts "#{obj}: #{item}"
end

# foo: one
# foo: two
# foo: three

This allows you to chain Enumerators together. For example, you can map a list’s elements to strings containing the index and the element as a string via:

puts %w[foo bar baz].map.with_index { |w, i| "#{i}:#{w}" }
# => ["0:foo", "1:bar", "2:baz"]

External Iteration

An Enumerator can also be used as an external iterator. For example, Enumerator#next returns the next value of the iterator or raises StopIteration if the Enumerator is at the end.

e = [1,2,3].each   # returns an enumerator object.
puts e.next   # => 1
puts e.next   # => 2
puts e.next   # => 3
puts e.next   # raises StopIteration

next, next_values, peek, and peek_values are the only methods which use external iteration (and Array#zip(Enumerable-not-Array) which uses next internally).

These methods do not affect other internal enumeration methods, unless the underlying iteration method itself has side-effect, e.g. IO#each_line.

FrozenError will be raised if these methods are called against a frozen enumerator. Since rewind and feed also change state for external iteration, these methods may raise FrozenError too.

External iteration differs significantly from internal iteration due to using a Fiber:

  • The Fiber adds some overhead compared to internal enumeration.

  • The stacktrace will only include the stack from the Enumerator, not above.

  • Fiber-local variables are not inherited inside the Enumerator Fiber, which instead starts with no Fiber-local variables.

  • Fiber storage variables are inherited and are designed to handle Enumerator Fibers. Assigning to a Fiber storage variable only affects the current Fiber, so if you want to change state in the caller Fiber of the Enumerator Fiber, you need to use an extra indirection (e.g., use some object in the Fiber storage variable and mutate some ivar of it).

Concretely:

Thread.current[:fiber_local] = 1
Fiber[:storage_var] = 1
e = Enumerator.new do |y|
  p Thread.current[:fiber_local] # for external iteration: nil, for internal iteration: 1
  p Fiber[:storage_var] # => 1, inherited
  Fiber[:storage_var] += 1
  y << 42
end

p e.next # => 42
p Fiber[:storage_var] # => 1 (it ran in a different Fiber)

e.each { p _1 }
p Fiber[:storage_var] # => 2 (it ran in the same Fiber/"stack" as the current Fiber)

Convert External Iteration to Internal Iteration

You can use an external iterator to implement an internal iterator as follows:

def ext_each(e)
  while true
    begin
      vs = e.next_values
    rescue StopIteration
      return $!.result
    end
    y = yield(*vs)
    e.feed y
  end
end

o = Object.new

def o.each
  puts yield
  puts yield(1)
  puts yield(1, 2)
  3
end

# use o.each as an internal iterator directly.
puts o.each {|*x| puts x; [:b, *x] }
# => [], [:b], [1], [:b, 1], [1, 2], [:b, 1, 2], 3

# convert o.each to an external iterator for
# implementing an internal iterator.
puts ext_each(o.to_enum) {|*x| puts x; [:b, *x] }
# => [], [:b], [1], [:b, 1], [1, 2], [:b, 1, 2], 3

Direct Known Subclasses

ArithmeticSequence, Chain, Lazy, Product

Defined Under Namespace

Classes: ArithmeticSequence, Chain, Generator, Lazy, Producer, Product, Yielder

Class Method Summary collapse

Instance Method Summary collapse

Methods included from Enumerable

#all?, #any?, #chain, #chunk, #chunk_while, #collect, #collect_concat, #compact, #count, #cycle, #detect, #drop, #drop_while, #each_cons, #each_entry, #each_slice, #entries, #filter, #filter_map, #find, #find_all, #find_index, #first, #flat_map, #grep, #grep_v, #group_by, #include?, #inject, #lazy, #map, #max, #max_by, #member?, #min, #min_by, #minmax, #minmax_by, #none?, #one?, #partition, #reduce, #reject, #reverse_each, #select, #slice_after, #slice_before, #slice_when, #sort, #sort_by, #sum, #take, #take_while, #tally, #to_a, #to_h, #uniq, #zip

Constructor Details

#new(size = nil) {|yielder| ... } ⇒ Object

Creates a new Enumerator object, which can be used as an Enumerable.

Iteration is defined by the given block, in which a “yielder” object, given as block parameter, can be used to yield a value by calling the yield method (aliased as <<):

fib = Enumerator.new do |y|
  a = b = 1
  loop do
    y << a
    a, b = b, a + b
  end
end

fib.take(10) # => [1, 1, 2, 3, 5, 8, 13, 21, 34, 55]

The optional parameter can be used to specify how to calculate the size in a lazy fashion (see Enumerator#size). It can either be a value or a callable object.

Yields:

  • (yielder) 


478
479
480
481
482
483
484
485
486
487
 
# File 'enumerator.c', line 478

static VALUE
enumerator_initialize(int argc, VALUE *argv, VALUE obj)
{
    VALUE iter = rb_block_proc();
    VALUE recv = generator_init(generator_allocate(rb_cGenerator), iter);
    VALUE arg0 = rb_check_arity(argc, 0, 1) ? argv[0] : Qnil;
    VALUE size = convert_to_feasible_size_value(arg0);

    return enumerator_init(obj, recv, sym_each, 0, 0, 0, size, false);
}

Class Method Details

.produce(initial = nil) {|prev| ... } ⇒ Object

Creates an infinite enumerator from any block, just called over and over. The result of the previous iteration is passed to the next one. If initial is provided, it is passed to the first iteration, and becomes the first element of the enumerator; if it is not provided, the first iteration receives nil, and its result becomes the first element of the iterator.

Raising StopIteration from the block stops an iteration.

Enumerator.produce(1, &:succ)   # => enumerator of 1, 2, 3, 4, ....

Enumerator.produce { rand(10) } # => infinite random number sequence

ancestors = Enumerator.produce(node) { |prev| node = prev.parent or raise StopIteration }
enclosing_section = ancestors.find { |n| n.type == :section }

Using ::produce together with Enumerable methods like Enumerable#detect, Enumerable#slice_after, Enumerable#take_while can provide Enumerator-based alternatives for while and until cycles:

# Find next Tuesday
require "date"
Enumerator.produce(Date.today, &:succ).detect(&:tuesday?)

# Simple lexer:
require "strscan"
scanner = StringScanner.new("7+38/6")
PATTERN = %r{\d+|[-/+*]}
Enumerator.produce { scanner.scan(PATTERN) }.slice_after { scanner.eos? }.first
# => ["7", "+", "38", "/", "6"]

Yields:

  • (prev) 


3078
3079
3080
3081
3082
3083
3084
3085
3086
3087
3088
3089
3090
3091
3092
 
# File 'enumerator.c', line 3078

static VALUE
enumerator_s_produce(int argc, VALUE *argv, VALUE klass)
{
    VALUE init, producer;

    if (!rb_block_given_p()) rb_raise(rb_eArgError, "no block given");

    if (rb_scan_args(argc, argv, "01", &init) == 0) {
        init = Qundef;
    }

    producer = producer_init(producer_allocate(rb_cEnumProducer), init, rb_block_proc());

    return rb_enumeratorize_with_size_kw(producer, sym_each, 0, 0, producer_size, RB_NO_KEYWORDS);
}

.product(*enums) ⇒ Object .product(*enums) {|elts| ... } ⇒ Object

Generates a new enumerator object that generates a Cartesian product of given enumerable objects. This is equivalent to Enumerator::Product.new.

e = Enumerator.product(1..3, [4, 5])
e.to_a #=> [[1, 4], [1, 5], [2, 4], [2, 5], [3, 4], [3, 5]]
e.size #=> 6

When a block is given, calls the block with each N-element array generated and returns nil.

Overloads:

  • .product(*enums) {|elts| ... } ⇒ Object

    Yields:

    • (elts) 


3722
3723
3724
3725
3726
3727
3728
3729
3730
3731
3732
3733
3734
3735
3736
3737
3738
3739
3740
3741
 
# File 'enumerator.c', line 3722

static VALUE
enumerator_s_product(int argc, VALUE *argv, VALUE klass)
{
    VALUE enums = Qnil, options = Qnil, block = Qnil;

    rb_scan_args(argc, argv, "*:&", &enums, &options, &block);

    if (!NIL_P(options) && !RHASH_EMPTY_P(options)) {
        rb_exc_raise(rb_keyword_error_new("unknown", rb_hash_keys(options)));
    }

    VALUE obj = enum_product_initialize(argc, argv, enum_product_allocate(rb_cEnumProduct));

    if (!NIL_P(block)) {
        enum_product_run(obj, block);
        return Qnil;
    }

    return obj;
}

Instance Method Details

#+(enum) ⇒ Object

Returns an enumerator object generated from this enumerator and a given enumerable.

e = (1..3).each + [4, 5]
e.to_a #=> [1, 2, 3, 4, 5]



3387
3388
3389
3390
3391
 
# File 'enumerator.c', line 3387

static VALUE
enumerator_plus(VALUE obj, VALUE eobj)
{
    return new_enum_chain(rb_ary_new_from_args(2, obj, eobj));
}

#each {|elm| ... } ⇒ Object #each ⇒ Enumerator #each(*appending_args) {|elm| ... } ⇒ Object #each(*appending_args) ⇒ Object

Iterates over the block according to how this Enumerator was constructed. If no block and no arguments are given, returns self.

Examples

"Hello, world!".scan(/\w+/)                     #=> ["Hello", "world"]
"Hello, world!".to_enum(:scan, /\w+/).to_a      #=> ["Hello", "world"]
"Hello, world!".to_enum(:scan).each(/\w+/).to_a #=> ["Hello", "world"]

obj = Object.new

def obj.each_arg(a, b=:b, *rest)
  yield a
  yield b
  yield rest
  :method_returned
end

enum = obj.to_enum :each_arg, :a, :x

enum.each.to_a                  #=> [:a, :x, []]
enum.each.equal?(enum)          #=> true
enum.each { |elm| elm }         #=> :method_returned

enum.each(:y, :z).to_a          #=> [:a, :x, [:y, :z]]
enum.each(:y, :z).equal?(enum)  #=> false
enum.each(:y, :z) { |elm| elm } #=> :method_returned

Overloads:



611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
 
# File 'enumerator.c', line 611

static VALUE
enumerator_each(int argc, VALUE *argv, VALUE obj)
{
    struct enumerator *e = enumerator_ptr(obj);

    if (argc > 0) {
        VALUE args = (e = enumerator_ptr(obj = rb_obj_dup(obj)))->args;
        if (args) {
#if SIZEOF_INT < SIZEOF_LONG
            /* check int range overflow */
            rb_long2int(RARRAY_LEN(args) + argc);
#endif
            args = rb_ary_dup(args);
            rb_ary_cat(args, argv, argc);
        }
        else {
            args = rb_ary_new4(argc, argv);
        }
        RB_OBJ_WRITE(obj, &e->args, args);
        e->size = Qnil;
        e->size_fn = 0;
    }
    if (!rb_block_given_p()) return obj;

    if (!lazy_precheck(e->procs)) return Qnil;

    return enumerator_block_call(obj, 0, obj);
}

#each_with_index {|(*args), idx| ... } ⇒ Object #each_with_index ⇒ Object

Same as Enumerator#with_index(0), i.e. there is no starting offset.

If no block is given, a new Enumerator is returned that includes the index.

Overloads:

  • #each_with_index {|(*args), idx| ... } ⇒ Object

    Yields:

    • ((*args), idx) 


695
696
697
698
699
 
# File 'enumerator.c', line 695

static VALUE
enumerator_each_with_index(VALUE obj)
{
    return enumerator_with_index(0, NULL, obj);
}

#each_with_object(obj) {|(*args), obj| ... } ⇒ Object #each_with_object(obj) ⇒ Object #with_object(obj) {|(*args), obj| ... } ⇒ Object #with_object(obj) ⇒ Object

Iterates the given block for each element with an arbitrary object, obj, and returns obj

If no block is given, returns a new Enumerator.

Example

to_three = Enumerator.new do |y|
  3.times do |x|
    y << x
  end
end

to_three_with_string = to_three.with_object("foo")
to_three_with_string.each do |x,string|
  puts "#{string}: #{x}"
end

# => foo: 0
# => foo: 1
# => foo: 2

Overloads:

  • #each_with_object(obj) {|(*args), obj| ... } ⇒ Object

    Yields:

    • ((*args), obj)
  • #with_object(obj) {|(*args), obj| ... } ⇒ Object

    Yields:

    • ((*args), obj) 


739
740
741
742
743
744
745
746
 
# File 'enumerator.c', line 739

static VALUE
enumerator_with_object(VALUE obj, VALUE memo)
{
    RETURN_SIZED_ENUMERATOR(obj, 1, &memo, enumerator_enum_size);
    enumerator_block_call(obj, enumerator_with_object_i, memo);

    return memo;
}

#obj ⇒ nil

Sets the value to be returned by the next yield inside e.

If the value is not set, the yield returns nil.

This value is cleared after being yielded.

# Array#map passes the array's elements to "yield" and collects the
# results of "yield" as an array.
# Following example shows that "next" returns the passed elements and
# values passed to "feed" are collected as an array which can be
# obtained by StopIteration#result.
e = [1,2,3].map
p e.next           #=> 1
e.feed "a"
p e.next           #=> 2
e.feed "b"
p e.next           #=> 3
e.feed "c"
begin
  e.next
rescue StopIteration
  p $!.result      #=> ["a", "b", "c"]
end

o = Object.new
def o.each
  x = yield         # (2) blocks
  p x               # (5) => "foo"
  x = yield         # (6) blocks
  p x               # (8) => nil
  x = yield         # (9) blocks
  p x               # not reached w/o another e.next
end

e = o.to_enum
e.next              # (1)
e.feed "foo"        # (3)
e.next              # (4)
e.next              # (7)
                    # (10)

Returns:

  • (nil) 


1053
1054
1055
1056
1057
1058
1059
1060
1061
1062
1063
1064
1065
1066
 
# File 'enumerator.c', line 1053

static VALUE
enumerator_feed(VALUE obj, VALUE v)
{
    struct enumerator *e = enumerator_ptr(obj);

    rb_check_frozen(obj);

    if (!UNDEF_P(e->feedvalue)) {
        rb_raise(rb_eTypeError, "feed value already set");
    }
    RB_OBJ_WRITE(obj, &e->feedvalue, v);

    return Qnil;
}

#initialize_copy(orig) ⇒ Object

:nodoc:



490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
 
# File 'enumerator.c', line 490

static VALUE
enumerator_init_copy(VALUE obj, VALUE orig)
{
    struct enumerator *ptr0, *ptr1;

    if (!OBJ_INIT_COPY(obj, orig)) return obj;
    ptr0 = enumerator_ptr(orig);
    if (ptr0->fib) {
        /* Fibers cannot be copied */
        rb_raise(rb_eTypeError, "can't copy execution context");
    }

    TypedData_Get_Struct(obj, struct enumerator, &enumerator_data_type, ptr1);

    if (!ptr1) {
        rb_raise(rb_eArgError, "unallocated enumerator");
    }

    RB_OBJ_WRITE(obj, &ptr1->obj, ptr0->obj);
    RB_OBJ_WRITE(obj, &ptr1->meth, ptr0->meth);
    RB_OBJ_WRITE(obj, &ptr1->args, ptr0->args);
    ptr1->fib  = 0;
    ptr1->lookahead  = Qundef;
    ptr1->feedvalue  = Qundef;
    RB_OBJ_WRITE(obj, &ptr1->size, ptr0->size);
    ptr1->size_fn  = ptr0->size_fn;

    return obj;
}

#inspect ⇒ String

Creates a printable version of e.

Returns:



1226
1227
1228
1229
1230
 
# File 'enumerator.c', line 1226

static VALUE
enumerator_inspect(VALUE obj)
{
    return rb_exec_recursive(inspect_enumerator, obj, 0);
}

#next ⇒ Object

Returns the next object in the enumerator, and move the internal position forward. When the position reached at the end, StopIteration is raised.

Example

a = [1,2,3]
e = a.to_enum
p e.next   #=> 1
p e.next   #=> 2
p e.next   #=> 3
p e.next   #raises StopIteration

See class-level notes about external iterators.

Returns:



919
920
921
922
923
924
 
# File 'enumerator.c', line 919

static VALUE
enumerator_next(VALUE obj)
{
    VALUE vs = enumerator_next_values(obj);
    return ary2sv(vs, 0);
}

#next_values ⇒ Array

Returns the next object as an array in the enumerator, and move the internal position forward. When the position reached at the end, StopIteration is raised.

See class-level notes about external iterators.

This method can be used to distinguish yield and yield nil.

Example

o = Object.new
def o.each
  yield
  yield 1
  yield 1, 2
  yield nil
  yield [1, 2]
end
e = o.to_enum
p e.next_values
p e.next_values
p e.next_values
p e.next_values
p e.next_values
e = o.to_enum
p e.next
p e.next
p e.next
p e.next
p e.next

## yield args       next_values      next
#  yield            []               nil
#  yield 1          [1]              1
#  yield 1, 2       [1, 2]           [1, 2]
#  yield nil        [nil]            nil
#  yield [1, 2]     [[1, 2]]         [1, 2]

Returns:



862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
 
# File 'enumerator.c', line 862

static VALUE
enumerator_next_values(VALUE obj)
{
    struct enumerator *e = enumerator_ptr(obj);
    VALUE vs;

    rb_check_frozen(obj);

    if (!UNDEF_P(e->lookahead)) {
        vs = e->lookahead;
        e->lookahead = Qundef;
        return vs;
    }

    return get_next_values(obj, e);
}

#peek ⇒ Object

Returns the next object in the enumerator, but doesn’t move the internal position forward. If the position is already at the end, StopIteration is raised.

See class-level notes about external iterators.

Example

a = [1,2,3]
e = a.to_enum
p e.next   #=> 1
p e.peek   #=> 2
p e.peek   #=> 2
p e.peek   #=> 2
p e.next   #=> 2
p e.next   #=> 3
p e.peek   #raises StopIteration

Returns:



1000
1001
1002
1003
1004
1005
 
# File 'enumerator.c', line 1000

static VALUE
enumerator_peek(VALUE obj)
{
    VALUE vs = enumerator_peek_values(obj);
    return ary2sv(vs, 1);
}

#peek_values ⇒ Array

Returns the next object as an array, similar to Enumerator#next_values, but doesn’t move the internal position forward. If the position is already at the end, StopIteration is raised.

See class-level notes about external iterators.

Example

o = Object.new
def o.each
  yield
  yield 1
  yield 1, 2
end
e = o.to_enum
p e.peek_values    #=> []
e.next
p e.peek_values    #=> [1]
p e.peek_values    #=> [1]
e.next
p e.peek_values    #=> [1, 2]
e.next
p e.peek_values    # raises StopIteration

Returns:



970
971
972
973
974
 
# File 'enumerator.c', line 970

static VALUE
enumerator_peek_values_m(VALUE obj)
{
    return rb_ary_dup(enumerator_peek_values(obj));
}

#rewind ⇒ Object

Rewinds the enumeration sequence to the beginning.

If the enclosed object responds to a “rewind” method, it is called.



1077
1078
1079
1080
1081
1082
1083
1084
1085
1086
1087
1088
1089
1090
1091
1092
 
# File 'enumerator.c', line 1077

static VALUE
enumerator_rewind(VALUE obj)
{
    struct enumerator *e = enumerator_ptr(obj);

    rb_check_frozen(obj);

    rb_check_funcall(e->obj, id_rewind, 0, 0);

    e->fib = 0;
    e->dst = Qnil;
    e->lookahead = Qundef;
    e->feedvalue = Qundef;
    e->stop_exc = Qfalse;
    return obj;
}

#size ⇒ Integer, ...

Returns the size of the enumerator, or nil if it can’t be calculated lazily.

(1..100).to_a.permutation(4).size # => 94109400
loop.size # => Float::INFINITY
(1..100).drop_while.size # => nil

Returns:



1243
1244
1245
1246
1247
1248
1249
1250
1251
1252
1253
1254
1255
1256
1257
1258
1259
1260
1261
1262
1263
1264
1265
1266
1267
1268
1269
1270
1271
1272
1273
1274
1275
1276
1277
1278
 
# File 'enumerator.c', line 1243

static VALUE
enumerator_size(VALUE obj)
{
    struct enumerator *e = enumerator_ptr(obj);
    int argc = 0;
    const VALUE *argv = NULL;
    VALUE size;

    if (e->procs) {
        struct generator *g = generator_ptr(e->obj);
        VALUE receiver = rb_check_funcall(g->obj, id_size, 0, 0);
        long i = 0;

        for (i = 0; i < RARRAY_LEN(e->procs); i++) {
            VALUE proc = RARRAY_AREF(e->procs, i);
            struct proc_entry *entry = proc_entry_ptr(proc);
            lazyenum_size_func *size_fn = entry->fn->size;
            if (!size_fn) {
                return Qnil;
            }
            receiver = (*size_fn)(proc, receiver);
        }
        return receiver;
    }

    if (e->size_fn) {
        return (*e->size_fn)(e->obj, e->args, obj);
    }
    if (e->args) {
        argc = (int)RARRAY_LEN(e->args);
        argv = RARRAY_CONST_PTR(e->args);
    }
    size = rb_check_funcall_kw(e->size, id_call, argc, argv, e->kw_splat);
    if (!UNDEF_P(size)) return size;
    return e->size;
}

#with_index(offset = 0) {|(*args), idx| ... } ⇒ Object #with_index(offset = 0) ⇒ Object

Iterates the given block for each element with an index, which starts from offset. If no block is given, returns a new Enumerator that includes the index, starting from offset

offset

the starting index to use

Overloads:

  • #with_index(offset = 0) {|(*args), idx| ... } ⇒ Object

    Yields:

    • ((*args), idx) 


674
675
676
677
678
679
680
681
682
683
 
# File 'enumerator.c', line 674

static VALUE
enumerator_with_index(int argc, VALUE *argv, VALUE obj)
{
    VALUE memo;

    rb_check_arity(argc, 0, 1);
    RETURN_SIZED_ENUMERATOR(obj, argc, argv, enumerator_enum_size);
    memo = (!argc || NIL_P(memo = argv[0])) ? INT2FIX(0) : rb_to_int(memo);
    return enumerator_block_call(obj, enumerator_with_index_i, (VALUE)MEMO_NEW(memo, 0, 0));
}

#each_with_object(obj) {|(*args), obj| ... } ⇒ Object #each_with_object(obj) ⇒ Object #with_object(obj) {|(*args), obj| ... } ⇒ Object #with_object(obj) ⇒ Object

Iterates the given block for each element with an arbitrary object, obj, and returns obj

If no block is given, returns a new Enumerator.

Example

to_three = Enumerator.new do |y|
  3.times do |x|
    y << x
  end
end

to_three_with_string = to_three.with_object("foo")
to_three_with_string.each do |x,string|
  puts "#{string}: #{x}"
end

# => foo: 0
# => foo: 1
# => foo: 2

Overloads:

  • #each_with_object(obj) {|(*args), obj| ... } ⇒ Object

    Yields:

    • ((*args), obj)
  • #with_object(obj) {|(*args), obj| ... } ⇒ Object

    Yields:

    • ((*args), obj) 


739
740
741
742
743
744
745
746
 
# File 'enumerator.c', line 739

static VALUE
enumerator_with_object(VALUE obj, VALUE memo)
{
    RETURN_SIZED_ENUMERATOR(obj, 1, &memo, enumerator_enum_size);
    enumerator_block_call(obj, enumerator_with_object_i, memo);

    return memo;
}