The any and all Functions in Python

I have already written here on the site about short-circuiting in the or and and operators in Python. In this article, we will verify that the concept of short-circuiting also applies to the all and any functions, available in a standard installation of the language. We will also see what each of these functions is useful for and what to watch out for when using them, based on a complexity analysis.

all

#

Let’s start by checking what the documentation says about the function:

help(all)

Help on built-in function all in module builtins:

all(iterable, /)
    Return True if bool(x) is True for all values x in the iterable.
    
    If the iterable is empty, return True.

As the documentation indicates, if every value x of an iterable passed to the function returns True in bool(x), the function returns True. An iterable, according to the documentation, is nothing more than an object that can return its members one at a time, such as sequence types like lists, strings, and tuples. Using a list as an example, we can verify that it is an iterable through a simple for loop:

example_list = [1, 2, 3, 4]
for number in example_list:
    print(number)

1
2
3
4

What for does is pass the iterable, in the example the list, to the iter() function, which returns an iterator, from which for requests each item successively through a next() call until the iterable is completely consumed. In the example, the list is consumed by passing each item successively to the print function.

If you are still in doubt, I strongly recommend reading this article about iterators and iterables, where the concept is explained in detail with several examples.

Let’s go to a simple example, passing some sequences to the all function:

all_true = (True, True, True)
all(all_true)

True

If at least one value is False, the function returns False:

some_true = (True, False, True)
all(some_true)

False

We have already seen in other articles that the interpreter considers some cases as False. Here are some examples:

print(bool(False))  # the False keyword itself
print(bool(None))   # the None keyword
print(bool(0))      # the integer 0 (zero)
print(bool(""))     # an empty string
print(bool(()))     # an empty tuple
print(bool([]))     # an empty list
print(bool({}))     # an empty dictionary

False
False
False
False
False
False
False

So we can build a more complete example in which the return of the all function is True:

all_true = (True, "Chico", 1, (1, 2), [1, 2], {'a': 1, 'b': 2})
all(all_true)

True

The following example returns False because there is an empty string among the values passed:

some_true = (True, "", 1, (1, 2), [1, 2], {'a': 1, 'b': 2})
all(some_true)

False

The next example returns False because there is an integer with value zero among the values passed:

some_true = (True, "Chico", 0, (1, 2), [1, 2], {'a': 1, 'b': 2})
all(some_true)

False

I hope these examples were enough to understand the function’s return. In the last two, since there is at least one element that returns False when passed as an argument to bool(), the return of all() will be False. Now, is it necessary to analyze every value passed in order to get the return? To evaluate this, let’s create the following function:

def executed_function(value):
    print("I was executed!")
    return value

Note that it is a very simple function: it receives a value, which can be of any type, prints the string “I was executed!”, and returns the same value passed. Let’s do a small test, passing the string “hi” as the function’s argument:

executed_function('hi')

I was executed!





'hi'

Note the message “I was executed!” and the return of the string passed to the function. But what is the purpose of this function?

Consider the code cell below. In it there is a list called test_list. Note that all of its items return True when passed as an argument to the bool function, except the third item, because it is the integer zero. This means that passing such a list to the all function should return False. But does the all function need to evaluate all the values?

To find out, let’s create a generator using a generator expression. Such an expression produces a generator that, instead of returning all the values at once, returns them on demand. For more details, see this article on the site about generators. See in the cells below how the generator_expression returns each value on demand from the next function:

test_list = [True, "Chico", 0, (1, 2), [1, 2], {'a': 1, 'b': 2}]
generator_expression = (executed_function(item) for item in test_list)
generator_expression

<generator object <genexpr> at 0x7f2b1d5eec10>

next(generator_expression)

I was executed!





True

next(generator_expression)

I was executed!





'Chico'

next(generator_expression)

I was executed!





0

I think you get the idea: we could keep requesting each item until the list is fully consumed. If you want all the remaining values at once, just pass the generator to a container such as, for example, a list:

list(generator_expression)

I was executed!
I was executed!
I was executed!





[(1, 2), [1, 2], {'a': 1, 'b': 2}]

Now the generator is exhausted, so a new next will raise a StopIteration exception:

next(generator_expression)

---------------------------------------------------------------------------

StopIteration                             Traceback (most recent call last)

<ipython-input-16-7ac23c2b0c8d> in <module>
----> 1 next(generator_expression)


StopIteration: 

Note that the behavior of a generator is that of an iterable, according to the concept at the beginning of the article. Since the generator was exhausted, let’s create it again and pass it to the all function:

generator_expression = (executed_function(item) for item in test_list)
all(generator_expression)

I was executed!
I was executed!
I was executed!





False

Note that the message “I was executed!” was displayed three times. That makes sense, because the first two items return True when passed as an argument to the bool function, while the third, the integer zero, returns False. Since the purpose of the all function is to return True only if all items return True, it is not necessary to evaluate the remaining items after finding a False case. Thus, we have shown that the all function exhibits short-circuit behavior.

OK, but what do we do with this information? What is the practical consequence? Let’s see what happens if we pass a large iterable, in this case a list of one million values:

giant_list_all_true = [True] * 1_000_000

All the values in this list are True, which means the return of the all function should be True. But, from what we saw earlier, the function goes through every value before giving its return. We saw in this article that IPython and Jupyter Notebook have their own functions (“magics” in the documentation) that allow some interesting analyses. One of them is %timeit, which measures how long a given command takes. Since this article is being written in a Jupyter Notebook, we can use this function. Let’s see the time when passing this list with one million True values:

%timeit all(giant_list_all_true)

3.72 ms ± 74.4 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

Let’s compare this figure with the time for lists that have at least one False value. We already know the return will be False, but how long will it take? Let’s start by generating a list, also with one million items, with a False value right at the beginning, at index 10:

giant_list_with_false_index_10 = giant_list_all_true.copy()
giant_list_with_false_index_10[10] = False

%timeit all(giant_list_with_false_index_10)

98.1 ns ± 0.44 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)

Note the huge difference! We went from milliseconds (10^-3 seconds) to nanoseconds (10^-9 seconds). A million-fold improvement! But that is because the value was right at the beginning. Let’s create another list of one million items, with a single False but in the last position:

giant_list_with_false_last_index = giant_list_all_true.copy()
giant_list_with_false_last_index[-1] = False

%timeit all(giant_list_with_false_last_index)

3.8 ms ± 123 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

Note that we are back in the millisecond range, a value almost equal to that of the list with all True values. That makes sense: each value was analyzed sequentially until the last one was reached.

Just to close the reasoning, let’s put a single False exactly in the middle of the list:

giant_list_with_false_middle_index = giant_list_all_true.copy()
giant_list_with_false_middle_index[499_999] = False

%timeit all(giant_list_with_false_middle_index)

1.83 ms ± 3.31 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

As expected, we got a value that is practically half of the previous one.

That is, we have a function with linear behavior with respect to a time complexity analysis. In the worst case, it will have to evaluate all the items to be able to return. Keep this in mind if you intend to use this function on a large amount of data, especially if there is reason to believe there will be few False values, since if they are at the end of the iterable, a long time will be needed. If you want to see the C source code of this function and check the implementation, click here.

any

#

Let’s follow the same script we used earlier for the all function, starting with the documentation:

help(any)

Help on built-in function any in module builtins:

any(iterable, /)
    Return True if bool(x) is True for any x in the iterable.
    
    If the iterable is empty, return False.

Again, this function receives an iterable. But, in this case, it returns True if at least one of the items returns True when passed as an argument to the bool function. Let’s see two examples. In the first, we pass only False values:

all_false = (False, False, False)
any(all_false)

False

As expected, we got False as the return. If at least one of the values passed is True, that is enough for us to get True as the return:

some_false = (False, True, False)
any(some_false)

True

As we have already seen, we do not necessarily need to pass True or False directly: we can pass other types that will be True or False when passed to the bool function. Let’s redefine our test_list variable below. Note that the first three items return False when passed to the bool function (see the examples of values considered False above if you have trouble understanding). Let’s pass this list to our generator expression:

test_list = [False, "", 0, (1, 2), [1, 2], {'a': 1, 'b': 2}]
generator_expression = (executed_function(item) for item in test_list)
generator_expression

<generator object <genexpr> at 0x7f2b1d5eef20>

And pass this expression to the any function:

any(generator_expression)

I was executed!
I was executed!
I was executed!
I was executed!





True

Note that the function was executed 4 times. Upon finding the first item that returns True, it is no longer necessary to keep consuming the generator, because a single True is enough for the function to return True. Again, short-circuit behavior.

So, as we did for all, let’s see what happens if we pass a large iterable, in this case a list of one million values:

giant_list_all_false = [False] * 1_000_000

All the values in this list are False, which means the return of the any function should be False. But, from what we saw earlier, the function goes through every value before giving its return. Let’s see the time when passing this list with one million False values:

%timeit any(giant_list_all_false)

3.18 ms ± 93.4 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

Let’s compare this figure with the time for lists that have at least one True value. We already know the return will be True, but how long will it take? Let’s start by generating a list, also with one million items, with a True value right at the beginning, at index 10:

giant_list_with_true_index_10 = giant_list_all_false.copy()
giant_list_with_true_index_10[10] = True

%timeit any(giant_list_with_true_index_10)

94.4 ns ± 0.355 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)

Note the huge difference! We went from milliseconds (10^-3 seconds) to nanoseconds (10^-9 seconds). A million-fold improvement! But that is because the value was right at the beginning. Let’s create another list of one million items, with a single True but in the last position:

giant_list_with_true_last_index = giant_list_all_false.copy()
giant_list_with_true_last_index[-1] = True

%timeit any(giant_list_with_true_last_index)

3.18 ms ± 41.6 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

Note that we are back in the millisecond range, a value almost equal to that of the list with all False values. That makes sense: each value was analyzed sequentially until the last one was reached.

Just to close the reasoning, let’s put a single True exactly in the middle of the list:

giant_list_with_true_middle_index = giant_list_all_false.copy()
giant_list_with_true_middle_index[499_999] = True

%timeit any(giant_list_with_true_middle_index)

1.54 ms ± 3.18 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

As expected, we got a value that is practically half of the previous one.

That is, just like all, any is also a function with linear behavior with respect to a time complexity analysis. In the worst case, it will have to evaluate all the items to be able to return. Keep this in mind if you intend to use this function on a large amount of data, especially if there is reason to believe there will be few True values, since if they are at the end of the iterable, a long time will be needed. If you want to see the C source code of this function and check the implementation, click here.

Conclusion

#

That’s it for this article. I hope you understood it and learned something new. We went through several concepts, such as iterables, iterators, and generator functions, and did a bit of time complexity analysis of algorithms. A lot of added value, I am sure.

Did you like this article? It is part of the programming Drops, a set of shorter posts focused on fundamentals, covering some aspects of the Python language and of programming in general. You can read more of these articles by browsing the tag “drops” here on the site.

See you next time!