This is a detailed tutorial of the Python Generators. Learn about yield statement, generator expressions and the send(), throw() and close() methods.
What are Python Generators?
These are a special kind of functions that instead of returning one value, at last, keeps the ability to return multiple values as the time passes and function processing continues. This concept is also known as lazy returning as the function keeps on returning the values as it further goes through the function code.
Let’s say there’s a function that is reading a very large text file. The file contains so many lines that even while ready, it is not possible to load the entire number of file lines at once in your computer’s memory. So, what if you can start the reading process of the file and keep on getting a limited number of lines, every few time units. For example, while the reading process continues, it keeps on printing every 1000 lines to the console every 2 seconds. This way, the function keeps on returning something after every 2 seconds until the reading process is completed for the entire file. Hence, such kind of function is a generator function that generates and return a number of values with time.
There are multiple ways to create generator objects. The first one and the most popular way is to use the statement
yield instead of the return statement in Python Functions.
Example 1. Simple Generator Function using
Given below is an example of the Generator function. It is also known as a generator object. The generator
threeDates() returns three dates, namely, today, yesterday and tomorrow in the
from datetime import date, timedelta def threeDates(): today = date.today() yield "Today: " + today.strftime("%d/%m/%Y") yesterday = today - timedelta(1) yield "Yesterday: " + yesterday.strftime("%d/%m/%Y") tomorrow = today + timedelta(1) yield "tomorrow: " + tomorrow.strftime("%d/%m/%Y") print(threeDates()) print(type(threeDates())) for date in threeDates(): print(date)
<generator object threeDates at 0x7f83f7f44a98> <class 'generator'> Today: 30/06/2020 Yesterday: 29/06/2020 tomorrow: 01/07/2020
The code is clear. Here we’ve used the
datetime module to fetch the current date and then by using the
timedelta object we’ve also calculated the dates for yesterday and tomorrow. Find out more about date and time in the following linked articles.
We’ve used the
yield statement three times to return these three different dates.
Now when you will call the function, it will not return the three dates, rather it will say, it’s a generator object at some memory location. You can also see this in the output of the above code and also we’ve used the
type() function to verify the data type of the generator function object.
Generator object is an iterable object. Therefore, to actually get or yield the values you must iterate over the generator object. In the above code, we’ve used the Python For loop for the purpose of yielding and printing the values from the generator function. You can also use the
next() function to fetch one yield value at a time and it is also illustrated in the following example.
Example 2. Infinite Yield Values and the usage of
For the sake of better understanding, we’ve illustrated another logic here. A Generator function may return an infinite number of values. In the following example, we’ve written a function that yields the multiples of a particular number that can be passed as the argument to the generator function.
def multiplesOf(number): count = 0 while True: yield number * count count += 1 generator1 = multiplesOf(5) #First Five Multiples of 5, starting from 0 print(next(generator1)) print(next(generator1)) print(next(generator1)) print(next(generator1)) print(next(generator1)) generator2 = multiplesOf(15) #First 10 multiples of 15 for i in range(0, 10): print(next(generator2))
0 5 10 15 20 0 15 30 45 60 75 90 105 120 135
Note. Do not apply the
next() function directly on the generator function, otherwise, you will get the first yield value always. You have to first create an instance of the generator object and then have to refer it by some variable, then you can pass the variable to the
next() function to get the values in a sequence.
You can always use the
next() function instead of the for loop to fetch the yield values whenever you are not sure about the exact number of values that the given generator function is yielding.
In case, there are no more yields left and you’re still applying the next() function to fetch the next yield value, it will raise the
As generators are capable of working with infinite sequences without much memory allocations, you can also use them as an alternative to traditional loop (Infinite For Loop or Infinite While Loop) logics.
Python Generators vs Normal Python Functions
Although you might have already pointed out the differences between the Python Generators and the normal functions, yet here we have described the differences in detail.
- Generators make use of the yield statements, usually a number of times. On the other hand, the normal functions may use one or more return statements but only once the return statement is executed that comes first in execution control.
- Normal functions return the value specified in the return statement at once after completing the execution of the function code that comes before the return statement in order. On the other hand, the generator functions return the generator object and it does not start the execution right away.
- Generator objects are iterable and the functions like next() can be used to fetch the yield values in the sequence. This is because the methods like
__next__()are implicitly implemented by the generators.
- Generators remember the state of the function’s local variables while the functions the supposed to forget everything after the return statement is executed.
- In generators, when the iteration is complete or there are no more values to yield,
StopIterationerror is raised automatically.
Python Generator Expressions
You can also create python generators using simple expressions. It is one of the easiest ways of creating generators without actually defining a function. This concept is quite similar to the concept of lambda functions.
The syntax to create generators using expressions is almost the same as the syntax of Python List comprehension. The only major difference that I observe here is that, in case of the list comprehension, we make use of the square brackets while in case of generator expressions, round parenthesis is used.
The technical difference between the list comprehension and the generator expressions is that when you define a list comprehension, the entire list items are generated at once after completing all the iterations in one go. On the other hand, in generator expressions, a single item is created in one go. This is also one of the reasons generator functions are considered more efficient as they retrieve or process the value as per the requirements or when demanded. Therefore, these are also memory efficient than the comprehension way outs.
The following code illustrates how you can a generator object using generator expressions and also shows the syntactical difference from the List Comprehension method.
#List Comprehension Method #List of squares of the numbers from 0 to 10 aList = [x*x for x in range(0, 11)] #Generator Expressions Method #Generator of squares of the numbers from 0 to 10 aGenerator = (x*x for x in range(0, 11)) print(aList) print(type(aList)) print(aGenerator) print(type(aGenerator))
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81, 100] <class 'list'> <generator object <genexpr> at 0x7f49545cdaf0> <class 'generator'>
As you can see in output also, the full list is generated with list comprehension while the generator expression has only created the generator object and the yield values will only be calculated or processed when required.
Advantages of Using Python Generators
Python Generators are quite useful for different kind of purposes. Here I have listed all those things that you should know about Generators that put them up in the list of your choices when you want to achieve some particular goals using Python.
Let’s say you want a normal function in python to first generate a sequence and then return it. The sequence contains a lot of items, let say, 10,000 items. There’s a logic written inside the function that generates these items. You function first have to do all the processing to generate these items and then have to store the sequence so generated in the memory to further return it. Hence, it consumes a lot of memory at once.
Now, we write the same logic but now inside a generated function. Let’s say we have used the yield keyword to return one item at a time of the sequence. This way, only the generator object will be created and the no memory will be required to store the 10,000 list items. Any item whenever will be required will be generated by using the generator object.
Hence, generator functions are memory efficient.
Infinite Sequence Possibilities
You simply can’t generate infinite sequences and return them with normal function. Even if you will try, the function will go into an infinite loop stage and will never return. But there’s no such issue with the Generator Functions. You can specify the logic to create the infinite sequence in the generator function and can yield one value at a time.
We’ve already discussed this concept in Example 2 where we have created a generated function that can generate infinite number of multiples of a particular number. Another example given below generates a sequence of odd numbers, starting from 1 upto infinity.
def oddNumbers(): count = 1 while True: yield count count += 1 print(oddNumbers()) gen = oddNumbers() #Printing First 5 odd numbers print(next(gen)) print(next(gen)) print(next(gen)) print(next(gen)) print(next(gen))
<generator object oddNumbers at 0x7fa70eee45c8> 1 2 3 4 5
In case you want to do multiple operations and that too on multiple series or sequences. Although it seems complex, yet using generators, the process can be pipelined. This can be better explained with the help of an example.
Let’s say we have a sequence of odd numbers from 1 to 1099. We want to find out the sum of the cubes of the odd numbers from 1 to 1099. The code makes use of the generators for the purpose is written below.
def oddNumbers(last): count = 1 while True: yield count count += 1 if count > last: break def cube(numbers): for number in numbers: yield number**3 print(sum(cube(oddNumbers(1099))))
Reduced Code Length & Easy Implementation
In the end, the Python Generators are easy to implement and reduces the overall length of the code. It automatically implements the class methods like
__next__() that are required to generate the sequences with normal classes.
I have defined one class and one generator function in the following code. Both generate a sequence containing cubes of numbers from 1 to 100.
class CubeClass: def __init__(this, max=0): this.count = 0 this.max = max def __iter__(this): return this def __next__(this): if this.count >= this.max: raise StopIteration result = 3 ** this.count this.count += 1 return result def cubeGenerator(max=0): count = 0 while count < max: yield 3 ** count count += 1 print(CubeClass(10)) print(cubeGenerator(10)) for cube in CubeClass(10): print(cube) for cube in cubeGenerator(10): print(cube)
As you can observe CubeClass code is quite lengthy and complex while the code of cubeGenerator is simple and can be written in just a few lines.
<__main__.CubeClass object at 0x7fd0b5ab6438> <generator object cubeGenerator at 0x7fd0bc1d4ca8> 1 3 9 27 81 243 729 2187 6561 19683 1 3 9 27 81 243 729 2187 6561 19683
Generator Object Methods
There are three advanced methods that you can use with Python Generator objects. These are listed below.
Let’s understand each of these methods with the help of illustrative examples.
generator.send() method is used to send a given value to the generator that can be used within the generator function body for the calculation of the next upcoming sequence item.
Have a look at the following example to understand the usage of the Python Generator send() method.
#Generates cube of the number sent using send() method def cube(): while True: number = yield yield number ** 3 gen = cube() next(gen) gen.send(3)
The above example is just to illustrate how you can make use of the send() method to send the yield values to the generator function logic. It is not a suitable practical coding example.
This is another useful method that you can use to run the exceptionally handled generator function logic. In other words. it can be used to raise a given exception manually within the generator function logic. Therefore, you can throw a particular exception in case you wish to execute the code written within the except block. Understand its usage with the help of the following example.
def getNumbers(): count = 0 try: while True: yield count count += 1 except: while ValueError: while True: yield count ** 3 count += 1 gen = getNumbers() #Getting simple numbers print(next(gen)) print(next(gen)) print(next(gen)) #Getting cubes gen.throw(ValueError) print(next(gen)) print(next(gen)) print(next(gen))
In the generator function written in the above code, we’ve written the logic to return simple numbers in the try block and the logic to return cubes of the sequence numbers in the except block. Initially, we used the generator object and the
next() function to print the first three sequence numbers and then by throwing the
ValueError, we started the sequence of the except block logic, i.e. yield the cubes of the sequence numbers.
0 1 2 27 64 125
This method is quite useful when you want multiple operations to be performed on a running sequence with different types of Exceptions.
The Python Generator close() method is used to stop the further iterations of a given sequence. The following example explains the usage of this method.
#Generates infinite numbers sequence, starting from 1 def numbers(): count = 1 while True: yield count count += 1 gen = numbers() #Starting Sequence print(next(gen)) print(next(gen)) print(next(gen)) #Stopping Further Sequence gen.close() #Trying to fetch the next seqeunce numbers #Will raise StopIteration Error print(next(gen))
As you can see in the output, the last line, the line no. 19 raises the StopIteration error because we have already stopped the further iteration using the
close() method applied on the generator object.
1 2 3 Traceback (most recent call last): File "<stdin>", line 19, in <module> print(next(gen)) StopIteration
I hope you found this guide useful. If so, do share it with others who are willing to learn Python and other programming languages. If you have any questions related to this article, feel free to ask us in the comments section.
And do not forget to subscribe to WTMatter!