Do you know the difference between a “normal” function and a generator function
in Python? What is the difference between the return of a usual function and
the yield of a generator? In this article we answer these questions and dig
deeper into some aspects of the language.
If you prefer to watch a video, click the player below (in Portuguese). The complete article follows the video.
return - Regular functions #
Let’s start by declaring a simple function that receives a sequence of numbers, squares them, and returns those squares in a list. If you need to review the concept of an iterable, see this article, where we used this same function as an example.
def square_numbers(numbers_iterable):
result = []
for number in numbers_iterable:
result.append(number**2)
return resultNow let’s create a tuple of numbers and pass it to that function:
numbers = (1, 2, 3)
square_numbers(numbers)[1, 4, 9]
OK, the behavior we expected. We can check the type of square_numbers and of
square_numbers(numbers):
type(square_numbers)function
type(square_numbers(numbers))list
Notice that the object square_numbers is of type function, as expected. And
when the argument numbers was passed, the object became a list. After all,
the return of the function is a list. The
return
statement ends the function, delivering what was requested, the list.
yield - Generator functions #
Let’s start by seeing how the language’s own documentation defines generators:
A function which returns a generator iterator. It looks like a normal function except that it contains
yieldexpressions for producing a series of values usable in a for-loop or that can be retrieved one at a time with thenext()function. Usually refers to a generator function, but may refer to a generator iterator in some contexts. In cases where the intended meaning isn’t clear, using the full terms avoids ambiguity.
I believe that was not very enlightening, but let’s fix that. First, to better understand the idea of an iterator, read this article where I explain it in detail. It is important for what we will see here. The documentation also defines the term “generator iterator”:
An object created by a generator function. Each
yieldtemporarily suspends processing, remembering the location execution state (including local variables and pending try-statements). When the generator iterator resumes, it picks up where it left off (in contrast to functions which start fresh on every invocation).
In this last definition there are important aspects, especially the idea of
temporary suspension. Let’s create our generator by modifying our function.
Notice that we just removed the creation of the list and replaced the return
with yield number**2:
def squares_generator(numbers_iterable):
for number in numbers_iterable:
yield number**2Before interacting with this generator, let’s look at the word yield. Dictionaries give, among others, these meanings:
- to produce
- to provide
- to give a return, as an investment does
Combining these meanings with the definitions, we can think of a generator
as a construct that produces values on demand at each call of next. We will
now see that idea in action. Let’s start by evaluating the types:
type(squares_generator)function
squares_generator(numbers)<generator object squares_generator at 0x7f4e15114970>
type(squares_generator(numbers))generator
g = squares_generator(numbers)
g<generator object squares_generator at 0x7f4e15114a50>
Notice that the object squares_generator itself is of type function. However,
when we pass the argument, it becomes of type generator, and that object has a
reference in memory. Observe that no value was returned. According to the
documentation, the values will be generated one at a time on demand:
next(g)1
next(g)4
next(g)9
This is really the same behavior we saw in the article about
iterators. So, we should
expect a StopIteration when the whole iterable passed has been consumed:
next(g)---------------------------------------------------------------------------
StopIteration Traceback (most recent call last)
<ipython-input-13-e734f8aca5ac> in <module>
----> 1 next(g)
StopIteration:
Generator expressions #
The same generator function could be written in a single line using a generator expression:
(number**2 for number in numbers)<generator object <genexpr> at 0x7f4e14017ba0>
Note that its syntax is similar to a list comprehension, but with parentheses instead of brackets. It has the same behavior as the generator created earlier:
for square in (number**2 for number in numbers):
print(square)1
4
9
g = (number**2 for number in numbers)next(g)1
next(g)4
next(g)9
next(g)---------------------------------------------------------------------------
StopIteration Traceback (most recent call last)
<ipython-input-20-e734f8aca5ac> in <module>
----> 1 next(g)
StopIteration:
Practical applications of generators #
More important than knowing that generators exist is knowing how to apply them in real cases. PEP 255, which implements generators in the Python language, presents the rationale that led to the implementation, but it can be somewhat abstract for beginners because it deals with callbacks and coroutines. Here I will try to present uses that are more accessible, but no less important, than the ones presented in the PEP.
Let’s start by creating a list of names and a list of majors that stand for students at some institution:
names = ['Americium', 'Californium', 'Copernicium', 'Dysprosium', 'Einsteinium']
courses = ['Geography', 'Film', 'Astronomy', 'Languages - Greek', 'Physics']Imagine that you want to create a registry, unifying in a single data structure the students’ names and their majors, besides associating a sequential enrollment number with each student.
In the next cell we make two implementations. One is a traditional function, storing each student as a dictionary inside a list and returning that list. The other generates each student dictionary, that is, without storing it in any structure. If storage is desired, it must be done by whoever consumes the generator:
def student_registry(names, courses):
registry = []
for i, (name, course) in enumerate(zip(names, courses)):
student = {
'id': i,
'name': name,
'course': course,
}
registry.append(student)
return registry
def student_generator(names, courses):
for i, (name, course) in enumerate(zip(names, courses)):
student = {
'id': i,
'name': name,
'course': course,
}
yield studentLet’s start with the function, seeing the result of passing the variables
names and courses as arguments and seeing how long this processing takes. We
have already seen in this article that
IPython and the Jupyter Notebook have their own functions
(“magics” in
the documentation) that allow some interesting analyses. One of them is
%timeit,
which measures how long a given command takes. Since this article is being
written in a Jupyter Notebook, we can use this function.
student_registry(names, courses)[{'id': 0, 'name': 'Americium', 'course': 'Geography'},
{'id': 1, 'name': 'Californium', 'course': 'Film'},
{'id': 2, 'name': 'Copernicium', 'course': 'Astronomy'},
{'id': 3, 'name': 'Dysprosium', 'course': 'Languages - Greek'},
{'id': 4, 'name': 'Einsteinium', 'course': 'Physics'}]
%timeit student_registry(names, courses)1.02 µs ± 21.2 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)
Observe that it is a short time, on the order of 1 microsecond. Let’s now see the generator. First, let’s check that the generator returns no value at all, only an object in memory:
student_generator(names, courses)<generator object student_generator at 0x7f4e15114e40>
And that, if we want all the results at once, we need to consume the generator by putting it, for example, in a list:
list(student_generator(names, courses))[{'id': 0, 'name': 'Americium', 'course': 'Geography'},
{'id': 1, 'name': 'Californium', 'course': 'Film'},
{'id': 2, 'name': 'Copernicium', 'course': 'Astronomy'},
{'id': 3, 'name': 'Dysprosium', 'course': 'Languages - Greek'},
{'id': 4, 'name': 'Einsteinium', 'course': 'Physics'}]
But if the generator only creates a reference in memory, without actually creating a data structure, how long does that take?
%timeit student_generator(names, courses)161 ns ± 1.35 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)
Note that we dropped to the order of nanoseconds! After all, only a reference was created and each student record will be created on demand:
%timeit next(student_generator(names, courses))534 ns ± 31.5 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)
If only a reference was created and not a data structure, then this should also be reflected in the space occupied, and this is where it gets even more interesting. To analyze the space occupied by each object, we will use the pympler library:
from pympler.asizeof import asizeofasizeof(student_registry(names, courses))2400
asizeof(student_generator(names, courses))1416
The results are in bytes. Since the function returns a list with the records, in the second-to-last cell we are seeing the space occupied by that list, while in the last cell we see only the space of the generated reference.
In case that was not clear, let’s do an example that demands more resources. Modifying the logic of the previous function and generator a little, we can make them receive an integer that represents how many students we want to store or generate, respectively. And these students, in our example, will be generated randomly from the two lists created earlier.
import random
def random_student_registry(count):
registry = []
for i in range(count):
student = {
'id': i,
'name': random.choice(names),
'course': random.choice(courses),
}
registry.append(student)
return registry
def random_student_generator(count):
for i in range(count):
student = {
'id': i,
'name': random.choice(names),
'course': random.choice(courses),
}
yield studentLet’s see an example with 5 random students:
random_student_registry(5)[{'id': 0, 'name': 'Americium', 'course': 'Film'},
{'id': 1, 'name': 'Einsteinium', 'course': 'Languages - Greek'},
{'id': 2, 'name': 'Americium', 'course': 'Languages - Greek'},
{'id': 3, 'name': 'Copernicium', 'course': 'Astronomy'},
{'id': 4, 'name': 'Einsteinium', 'course': 'Languages - Greek'}]
random_student_generator(5)<generator object random_student_generator at 0x7f4dfeade430>
list(random_student_generator(5))[{'id': 0, 'name': 'Einsteinium', 'course': 'Geography'},
{'id': 1, 'name': 'Americium', 'course': 'Astronomy'},
{'id': 2, 'name': 'Californium', 'course': 'Languages - Greek'},
{'id': 3, 'name': 'Dysprosium', 'course': 'Geography'},
{'id': 4, 'name': 'Einsteinium', 'course': 'Astronomy'}]
Notice that the results are different and there may be repetition, since they are now random.
But what happens if we ask for, say… 1 million students?
%timeit random_student_registry(1_000_000)880 ms ± 7.31 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
%timeit random_student_generator(1_000_000)144 ns ± 4.69 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)
Notice that the function went from the microsecond range to the millisecond range. After all, it now has to build the whole list of records. However, the generator remains in the nanosecond range, because nothing was actually created or stored, only the generator object was stored in memory, and that takes little time.
And what about the space occupied by each object?
asizeof(random_student_registry(1_000_000))272698416
It is even hard to grasp this number in bytes, so let’s convert it to megabytes:
_ / 2**20 # conversion to megabytes260.0654754638672
A list with 1 million entries, each one a dictionary, occupying 260 MB. Did you imagine that number?
Let’s see the generator:
asizeof(random_student_generator(1_000_000))440
Notice that this is a negligible amount of space. After all, nothing was actually stored except the generator object. Now, if we actually consume this generator, storing each result in, for example, a list, we will get the same values as before:
%timeit list(random_student_generator(1_000_000))854 ms ± 21.1 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
asizeof(list(random_student_generator(1_000_000))) / 2**20259.63890075683594
Now, we do not necessarily need to consume the whole generator at once. That is the beauty of generators: we can consume on demand and, therefore, manage computational resources better, both execution time and storage.
Here on the site I will show uses in the future and link them here, but you can already imagine some: consuming a large database in small pieces; reading large files in parts; the possibility of creating virtually infinite series of values; and so on.
Conclusion #
That is it for this article. I hope you understood it and learned something new. We went through several concepts, such as iterables, iterators, functions, generators, and did a bit of analysis of time and space occupied. A lot of added value, I am sure.
Did you like this article? It is part of the programming Drops, a set of shorter posts focused on fundamentals, covering some aspects of the Python language and of programming in general. You can read more of these articles by browsing the "drops" tag here on the site.
Note
The time reported by %timeit may vary from machine to machine, so you will not
necessarily get the same values. However, the difference in orders of
magnitude should hold.
See you next time!
