yield from - What Is It? Understanding Generators in Python

Have you ever seen the term yield from in some Python code and wondered what it was? In this article we will dive even deeper into generators and understand, with examples, what yield from means and how we can use it to make our code even more efficient.

Origin

#

In the article about generators we saw that they originated in the context of coroutines, but that they are applicable in many other contexts, so much so that you don’t even need to know about coroutines to use them. yield from was made official in PEP 380, still in a very technical way, aimed at that specific context. But let’s take some excerpts from that PEP and bring them closer to simple examples.

The abstract of PEP 380 reads:

A syntax is proposed for a generator to delegate part of its operations to another generator. This allows a section of code containing ‘yield’ to be factored out and placed in another generator. Additionally, the subgenerator is allowed to return with a value, and the value is made available to the delegating generator. The new syntax also opens up some opportunities for optimisation when one generator re-yields values produced by another.

The key point is in the first sentence: a generator delegating part of its operations to another generator.

Another quote comes from the language’s documentation on expressions, where we have:

When yield from <expr> is used, it treats the supplied expression as a subiterator. (…) the supplied expression must be an iterable.

That is, we can convert an iterable into an iterator and yield from it. Interesting. And, in case you need to refresh these concepts, see this article about iterators and iterables with many examples.

Let’s see how to apply what was presented in a few simple examples.

Simple examples

#

I recently wrote about infinite sequences and in that article we saw the islice function. Let’s use all the knowledge acquired there.

Consuming from two generators

#

Consider that we have two generators of infinite sequences, one for positive even integers and another for positive odd integers:

from itertools import islice


def positive_evens():
    value = 0
    while True:
        yield value
        value += 2
        
        
def positive_odds():
    value = 1
    while True:
        yield value
        value += 2

Now suppose we want a generator of positive integers, without distinguishing even from odd. We can reuse the generators we already created:

def positive_integers():
    for even, odd in zip(positive_evens(), positive_odds()):
        yield even
        yield odd

Let’s check the first 10 values generated:

gen_int = positive_integers()
tuple(islice(gen_int, 10))

(0, 1, 2, 3, 4, 5, 6, 7, 8, 9)

It worked… but how? Let’s start by looking at the documentation of the zip function:

help(zip)

Help on class zip in module builtins:

class zip(object)
 |  zip(*iterables) --> A zip object yielding tuples until an input is exhausted.
 |  
 |     >>> list(zip('abcdefg', range(3), range(4)))
 |     [('a', 0, 0), ('b', 1, 1), ('c', 2, 2)]
 |  
 |  The zip object yields n-length tuples, where n is the number of iterables
 |  passed as positional arguments to zip().  The i-th element in every tuple
 |  comes from the i-th iterable argument to zip().  This continues until the
 |  shortest argument is exhausted.
 |  
 |  Methods defined here:
 |  
 |  __getattribute__(self, name, /)
 |      Return getattr(self, name).
 |  
 |  __iter__(self, /)
 |      Implement iter(self).
 |  
 |  __next__(self, /)
 |      Implement next(self).
 |  
 |  __reduce__(...)
 |      Return state information for pickling.
 |  
 |  ----------------------------------------------------------------------
 |  Static methods defined here:
 |  
 |  __new__(*args, **kwargs) from builtins.type
 |      Create and return a new object.  See help(type) for accurate signature.

See that zip receives iterables and generates tuples until those iterables are consumed. Like this:

zip(positive_evens(), positive_odds())

<zip at 0x7f25b857f300>

next(zip(positive_evens(), positive_odds()))

(0, 1)

Let’s look at the first 5 pairs generated:

tuple(islice(zip(positive_evens(), positive_odds()), 5))

((0, 1), (2, 3), (4, 5), (6, 7), (8, 9))

So, basically, the for loop in the body of the generator goes through each item of each tuple, making it available (yield) to calls to the generator.

Now, if the zip function generates tuples, and tuples are iterables, we can pass these tuples to yield from according to the documentation seen earlier. So, we can redefine our generator as:

def positive_integers():
    for tup in zip(positive_evens(), positive_odds()):
        yield from tup    

gen_int = positive_integers()
tuple(islice(gen_int, 10))

(0, 1, 2, 3, 4, 5, 6, 7, 8, 9)

Quite simple and concise.

Consuming from another generator

#

Let’s now better understand the part of the PEP about delegating part of the operations to a generator. We will create a generator of positive integers divisible by 10. We know that to be divisible by 10, a number must end in 0 and is therefore an even number. So it makes sense to use the already created generator of even numbers and generate only those divisible by 10 on demand:

def positive_integers_divisible_by_ten():
    even = positive_evens()
    yield from (p for p in even if (p % 10 == 0))

Note that we created a generator expression and we are generating from it with yield from.

gen_ten = positive_integers_divisible_by_ten()
tuple(islice(gen_ten, 10))

(0, 10, 20, 30, 40, 50, 60, 70, 80, 90)

“Flattening” lists of lists

#

Consider that you have a list of lists containing numbers and would like to generate these numbers as if they belonged to a single list. This process is called flatten, meaning to flatten the sublists into the main list.

list_of_sublists = [[1, 2, 3], [4, 5, 6], [7], [8, 9]]

One way to write a generator would be:

def flatten(lst):
    for sublist in lst:
        for item in sublist:
            yield item

gen = flatten(list_of_sublists)
tuple(gen)

(1, 2, 3, 4, 5, 6, 7, 8, 9)

Now, each sublist is an iterable and, therefore, we can pass it directly to yield from:

def flatten(lst):
    for sublist in lst:
        yield from sublist


gen = flatten(list_of_sublists)
tuple(gen)

(1, 2, 3, 4, 5, 6, 7, 8, 9)

This flattening has many real applications and this approach is not necessarily the most efficient, read more here. But it is a good way to see more applications of yield from.

Conclusions

#

Did you enjoy learning more about generators? They are a very efficient way to handle many situations where it is not possible, or not healthy from a resource standpoint, to store all the values beforehand.

Did you like this article? It is part of the programming Drops, a set of shorter posts focused on fundamentals, covering some aspects of the Python language and of programming in general. You can read more of these articles by searching for the tag “drops” here on the site.

See you next time!