Iterators and iterables in Python

Do you know what an iterable is? And an iterator? How do you recognize these structures in Python? Answering these questions is the goal of this article.

If you prefer to watch a video, click the player below (in Portuguese). The complete article follows the video.

Iterables

#

Let’s start by creating a function that receives a collection of values, squares them, and returns a list with those squares:

def square_all(iterable_of_numbers):
    result = []
    for number in iterable_of_numbers:
        result.append(number**2)
    return result

Note that the function parameter was named iterable_of_numbers. The goal is to show what can be iterable and how to recognize an iterable.

Let’s start with the language documentation. In its glossary, an iterable is defined as:

An object capable of returning its members one at a time. Examples of iterables include all sequence types (such as list, str, and tuple) and some non-sequence types like dict, file objects, and objects of any classes you define with an __iter__() method or with a __getitem__() method that implements Sequence semantics.

Iterables can be used in a for loop and in many other places where a sequence is needed (zip(), map(), …). When an iterable object is passed as an argument to the built-in function iter(), it returns an iterator for the object. This iterator is good for one pass over the set of values. When using iterables, it is usually not necessary to call iter() or deal with iterator objects yourself. The for statement does that automatically for you, creating a temporary unnamed variable to hold the iterator for the duration of the loop.

Nothing beats an example to understand these technicalities. Let’s start by creating a variable numbers that will be a tuple of integers, since the glossary says that sequence types are naturally iterable:

numbers = (1, 2, 3)

Note that the documentation says an iterable can be recognized by an __iter__() or __getitem__() method. To check the attributes and methods available for an object, there is the function dir:

dir(numbers)

['__add__',
 '__class__',
 '__contains__',
 '__delattr__',
 '__dir__',
 '__doc__',
 '__eq__',
 '__format__',
 '__ge__',
 '__getattribute__',
 '__getitem__',
 '__getnewargs__',
 '__gt__',
 '__hash__',
 '__init__',
 '__init_subclass__',
 '__iter__',
 '__le__',
 '__len__',
 '__lt__',
 '__mul__',
 '__ne__',
 '__new__',
 '__reduce__',
 '__reduce_ex__',
 '__repr__',
 '__rmul__',
 '__setattr__',
 '__sizeof__',
 '__str__',
 '__subclasshook__',
 'count',
 'index']

Note that these methods are present. If you have trouble finding them in the list, the following code confirms their presence (and also makes you curious to research set in Python 😄 )

set(('__iter__', '__getitem__')) & set(dir(numbers))

{'__getitem__', '__iter__'}

OK, numbers is an iterable. Following the glossary text, it is possible to go through each item using a for loop. Look again at the body of the function created at the start of the article and notice that this is exactly what happens.

So, it is possible to pass numbers as an argument to our function:

square_all(numbers)

[1, 4, 9]

As an exercise, make numbers a list and see that it works the same. Create your own exercises with the types mentioned in the glossary to understand the subject even better.

Another way to recognize numbers as an iterable is through a list comprehension, since a for loop also occurs in it:

[number**2 for number in numbers]

[1, 4, 9]

The glossary also says that an iterable can be passed to language functions such as map. The map function applies a function to all items of an iterable. Let’s create an anonymous function that also squares all the items:

map(lambda x : x**2, numbers)

<map at 0x7fe4d05d5d60>

Note that map returns an object in memory that is, in fact, an iterator, our next subject. But first, just to show that it worked, let’s pass the map to a list:

list(map(lambda x : x**2, numbers))

[1, 4, 9]

Iterators

#

Let’s consult the glossary again:

An object representing a stream of data. Repeated calls to the iterator’s __next__() method (or passing it to the built-in function next()) return successive items in the stream. When no more data are available a StopIteration exception is raised instead. At this point, the iterator object is exhausted and any further calls to its __next__() method just raise StopIteration again. Iterators are required to have an __iter__() method that returns the iterator object itself so every iterator is also iterable and may be used in most places where other iterables are accepted. One notable exception is code which attempts multiple iteration passes. A container object (such as a list) produces a fresh new iterator each time you pass it to the iter() function or use it in a for loop. Attempting this with an iterator will just return the same exhausted iterator object used in the previous iteration pass, making it appear like an empty container.

To understand, let’s create an iterator by passing our variable numbers to the iter() function and assigning the result to a new variable num_iter:

num_iter = iter(numbers)

Let’s check the type of the generated variable:

type(num_iter)

tuple_iterator

As the type itself shows, it is an iterator. Being an iterator, following the documentation, there must be an __iter__() method and also a __next__(). Let’s check:

dir(num_iter)

['__class__',
 '__delattr__',
 '__dir__',
 '__doc__',
 '__eq__',
 '__format__',
 '__ge__',
 '__getattribute__',
 '__gt__',
 '__hash__',
 '__init__',
 '__init_subclass__',
 '__iter__',
 '__le__',
 '__length_hint__',
 '__lt__',
 '__ne__',
 '__new__',
 '__next__',
 '__reduce__',
 '__reduce_ex__',
 '__repr__',
 '__setattr__',
 '__setstate__',
 '__sizeof__',
 '__str__',
 '__subclasshook__']

set(('__iter__', '__next__')) & set(dir(num_iter))

{'__iter__', '__next__'}

Now let’s understand what the documentation means by successive calls to next():

next(num_iter)

1

Note that only the first number was displayed. Let’s make three more calls:

next(num_iter)

2

next(num_iter)

3

next(num_iter)

---------------------------------------------------------------------------

StopIteration                             Traceback (most recent call last)

<ipython-input-16-1d816cd6b265> in <module>
----> 1 next(num_iter)


StopIteration: 

As described, when the iterator is exhausted, a StopIteration exception is raised, indicating that the whole iterator has been consumed.

Just to clear up a question that may be crossing your mind, an iterable is not necessarily an iterator. Example:

next(numbers)

---------------------------------------------------------------------------

TypeError                                 Traceback (most recent call last)

<ipython-input-17-2cf59712f126> in <module>
----> 1 next(numbers)


TypeError: 'tuple' object is not an iterator

See that numbers itself is not an iterator even though it is an iterable.

Let’s now check that the map object generated in the previous section is also an iterator. Recall:

map(lambda x : x**2, numbers)

<map at 0x7fe4d0507730>

Let’s assign it to a variable to make the object easier to handle:

num_map = map(lambda x : x**2, numbers)

Let’s see the type of the object and make successive calls to next:

type(num_map)

map

next(num_map)

1

next(num_map)

4

next(num_map)

9

next(num_map)

---------------------------------------------------------------------------

StopIteration                             Traceback (most recent call last)

<ipython-input-24-bc498b643d23> in <module>
----> 1 next(num_map)


StopIteration: 

See that it is the same behavior seen with the object created via iter().

Since the documentation says that an iterator is also an iterable, we can pass the map object as an argument to our function:

square_all(map(lambda x : x**2, numbers))

[1, 16, 81]

Since the anonymous function already squared the values, the for loop inside the square_all function is squaring them again. Hence the values obtained.

The for loop handles StopIteration automatically, which is why we don’t see the exception when the iterator is consumed.

When the iterator is passed to a container type, it is consumed entirely, also without raising the exception:

list(map(lambda x : x**2, numbers))

[1, 4, 9]

tuple(map(lambda x : x**2, numbers))

(1, 4, 9)

If the iterator has started to be consumed and is then passed to a container, only the rest of the iterator will be included. For example, let’s recreate our map iterator and call next once:

num_map = map(lambda x : x**2, numbers)

next(num_map)

1

Now, let’s pass the iterator to a list:

list(num_map)

[4, 9]

Note that only the last two items appear in the list. After all, the iterator keeps no record of past items. That is the meaning of the phrase “an object representing a stream of data” in the glossary. The iterator had already returned the item 1, the next ones in the stream were 4 and 9, and these were passed to the list when it consumed the iterator entirely.

If you want some historical context on iterators in the language, I recommend reading PEP 234.

Conclusion

#

As you may have noticed, iterators and iterables appear frequently in the language, and you may not even have realized you were using them at various points. Understanding these concepts is crucial for understanding more complex structures of the language such as, for example, generators, which will soon be the topic of an article here on the site.

Did you like this article? It is part of the Drops of programming, a set of shorter posts focused on fundamentals, covering some aspects of the Python language and of programming in general. You can read more of these articles by searching for the tag “drops” here on the site.

See you next time!