Have you ever seen the term yield from in some Python code and wondered what it was? In this article we will delve even deeper into generators and understand, with examples, what yield from means and how we can use it to make our code even more efficient.
In the article about generators we saw that the origin of generators was in the context of coroutines, but that they are applicable in several other contexts, so that knowledge about coroutines is not necessary to apply them. The yield from was formalized in PEP 380, still in a very technical way, aimed at this specific context. But let’s take excerpts from this PEP and bring them closer to simple examples.
The PEP 380 abstract brings the following text:
A syntax is proposed for a generator to delegate part of its operations to another generator. This allows a section of code containing ‘yield’ to be factored out and placed in another generator. Additionally, the subgenerator is allowed to return with a value, and the value is made available to the delegating generator. The new syntax also opens up some opportunities for optimisation when one generator re-yields values produced by another.
The key point is the written in the first sentence: a generator delegating part of its operations to another generator.
Another quote is in the language expressions documentation, where we have:
When
yield fromis used, it treats the supplied expression as a subiterator. (…) the supplied expression must be an iterable.
That is, we can convert an iterable into an iterator and generate from it. Interesting. And, if you need to remember the concepts, see this article on iterators and iterables with several examples.
Let’s see how to apply the presented in some simple examples.
Simple examples
Recently I wrote about infinite sequences and in that article we saw the use of the islice method. Let’s use all the knowledge acquired here.
Consuming from two generators
Consider that we have two generators of infinite sequences, one for positive even integers and the other for positive odd integers:
from itertools import islice
def positive_even_numbers():
value = 0
while True:
yield value
value += 2
def positive_odd_numbers():
value = 1
while True:
yield value
value += 2Suppose now that we want a generator of positive integers, without discriminating whether it is even or odd. We can take advantage of the generators already created:
def positive_integers():
for pair, odd in zip(positive_even_numbers(), positive_odd_numbers()):
yield pair
yield oddWe can check the first 10 yielded values:
gen_int = positive_integers()
tuple(islice(gen_int, 10))(0, 1, 2, 3, 4, 5, 6, 7, 8, 9)
It worked… but how? Let’s start by looking at the documentation of the zip function:
help(zip)Help on class zip in module builtins:
class zip(object)
| zip(*iterables, strict=False) --> Yield tuples until an input is exhausted.
|
| >>> list(zip('abcdefg', range(3), range(4)))
| [('a', 0, 0), ('b', 1, 1), ('c', 2, 2)]
|
| The zip object yields n-length tuples, where n is the number of iterables
| passed as positional arguments to zip(). The i-th element in every tuple
| comes from the i-th iterable argument to zip(). This continues until the
| shortest argument is exhausted.
|
| If strict is true and one of the arguments is exhausted before the others,
| raise a ValueError.
|
| Methods defined here:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __iter__(self, /)
| Implement iter(self).
|
| __next__(self, /)
| Implement next(self).
|
| __reduce__(...)
| Return state information for pickling.
|
| __setstate__(...)
| Set state information for unpickling.
|
| ----------------------------------------------------------------------
| Static methods defined here:
|
| __new__(*args, **kwargs) from builtins.type
| Create and return a new object. See help(type) for accurate signature.
Note that zip receives iterables and yields tuples until these iterables are consumed. So:
zip(positive_even_numbers(), positive_odd_numbers())next(zip(positive_even_numbers(), positive_odd_numbers()))(0, 1)
The 5 first pairs yielded are:
tuple(islice(zip(positive_even_numbers(), positive_odd_numbers()), 5))((0, 1), (2, 3), (4, 5), (6, 7), (8, 9))
So, basically the for loop in the generator body goes through each item of each tuple, making it available (yield) for calls to the generator.
Moving on, if the zip function yields tuples, and tuples are iterables, we can pass these tuples to yield from according to the documentation seen earlier. Thus, we can redefine our generator as:
def positive_integers():
for tuple in zip(positive_even_numbers(), positive_odd_numbers()):
yield from tuplegen_int = positive_integers()
tuple(islice(gen_int, 10))(0, 1, 2, 3, 4, 5, 6, 7, 8, 9)
Simple and succinct.
Consuming from another generator
Moving forward, let’s better understand the part of the PEP that talks about delegating part of the operations to a generator. Let’s create a generator of positive integers divisible by 10. We know that to be divisible by 10, a number must end in 0 and, therefore, it is an even number. So it makes sense to use the already created generator of even numbers and generate only those divisible by 10 on demand:
def positive_integers_divisible_by_ten():
even = positive_even_numbers()
yield from (e for e in even if (e % 10 == 0))See that we created a generator expression and we are yielding from it with yield from.
gen_ten = positive_integers_divisible_by_ten()
tuple(islice(gen_ten, 10))(0, 10, 20, 30, 40, 50, 60, 70, 80, 90)
“Flattening” lists of lists
To illustrate the concept, let’s consider that you have a list of lists containing numbers and you would like to yield these numbers as if they belonged to only one list. This process is called flatten, something like flattening the sublists in the main list.
list_of_lists = [[1, 2, 3], [4, 5, 6], [7], [8, 9]]A way to write a generator would be:
def flatten(list_of_lists):
for list_ in list_of_lists:
for item in list_:
yield itemgen = flatten(list_of_lists)
tuple(gen)(1, 2, 3, 4, 5, 6, 7, 8, 9)
Now, each inner list is an iterable and, therefore, we can pass it directly to yield from:
def flatten(list_of_lists):
for list_ in list_of_lists:
yield from list_gen = flatten(list_of_lists)
tuple(gen)(1, 2, 3, 4, 5, 6, 7, 8, 9)
This flattening has several real applications and this approach is not necessarily the most efficient, read more here. But it is a good way to see more applications of yield from.
Conclusion
Did you enjoy learning more about generators? It is a very efficient way to deal with various situations where it is not possible, from a resource perspective, to store all values previously.
Did you like this article? It is part of the Python Drops, a set of shorter posts focused on fundamentals talking about some aspects of the Python language and programming in general. You can read more of these articles by searching for the tag “drops” here on the site. Until next time!
