The problem I encountered when working with generators
Working with generators through map, filter And allI ran into an empty array problem:
the problem is that when passing the result filter(…) into a function alland then when continuing to work with the generator obtained from the function filterfor example, converting it to tuple, to see which elements were in the array after passing through the filter, I received an empty one tuple.
Let's abstract from everything that we don't need and consider the problem itself.
Example 1.1
generator_reverse = (i for i in range(10, -1, -1))
print(all(generator_reverse))
print(tuple(generator_reverse))
input:
-> False
-> ()
I specifically flipped the elements in range to make it clearer.
Here all works as we need, since we bypass generator_reverse first. But then the problems begin. tuple(generator_reverse) returns an empty array.
And it seems like… if you think about it, everything is logical. At 4 o'clock in the morning, exhausted, I didn't think so 🙂
The generator went through all the elements in the function once all and on the last element, namely 0, completed the work, returning False, and when attempting the second traversal due to the conversion of the generator to tuple an exception is thrown immediately StopIterationand the output is an empty array.
Of course, if you don't write the fourth line with conversion to tupleyou won’t even notice it, unless, of course, you intend to continue working with this generator in the future.
Now I will still return the array to its normal step.
Example 1.2
generator = (i for i in range(10))
print(all(generator))
print(tuple(generator))
input:
-> False
-> (1, 2, 3, 4, 5, 6, 7, 8, 9)
all will call 1 time generator.next() and, having received 0, will immediately return False. generator saved the state while working with alland when converting it to tuple we receive all the items from the moment all finished (we lost the null element during conversion)
What if we transform generator V tuple before the call all?
Example 1.3
generator = (i for i in range(10))
print(tuple(generator))
print(all(generator))
print(tuple(generator))
input:
(0, 1, 2, 3, 4, 5, 6, 7, 8, 9)
True
()
On the first conversion everything is fine. We iterate over all elements of the generator and get the expected array. But in all will be transferred generator with the state on the last element. As a result, an exception is immediately thrown StopIteration And all not seeing a single element in the generator, it returns True, and in the subsequent conversion to tuple an exception is also thrown immediately StopIterationreturning us an empty array.
(I’m just showing what happens if you pass an empty array to the function all)
bool_ = all(tuple())
print(bool_)
input:
True
This prompted me to check the behavior of the generator regarding its dynamism.
Many people know that if you add elements to the array on which the generator is built (after the generator has been created), then they will appear in the generator, if, of course, the generator itself reaches them.
Example 2.1
list_ = list()
gen = (i for i in list_)
list_.append(1)
list_.append(2)
print(next(gen))
list_.append(3)
list_.append(4)
list_.append(5)
print(tuple(gen))
input:
1
(2, 3, 4, 5)
First we create a generator based on an empty list. Then we add 2 elements to the list. Call 1 time next(gen). Everything is as we expect. Although we added elements to the list after creating the generator.
Please note that at this stage we did not reach the end of the generator and did not throw an exception StopIteration.
After that, we add a few more elements to the list and call the generator to be converted to a tuple, thereby getting all the remaining elements, including those that we added after the call next(gen). Everything works as expected.
But.. if before adding elements to the array you already reach the end of the generator by calling StopIterationthen the generator washes its hands of this.
Example 2.2
list_ = list()
gen = (i for i in list_)
list_.append(1)
list_.append(2)
print(tuple(gen))
list_.append(3)
list_.append(4)
list_.append(5)
print(tuple(gen))
input:
(1, 2)
()
The same thing happens here, except that we still call StopIteration before adding all the elements to the list. Those elements that we managed to add before the exclusion StopIterationwere reflected in the generator, but all the others were not.
In addition to this, I asked the neuron about this:
This behavior can be explained by the fact that generators are iterators, which means that once they have reached the end of the sequence they are traversing, they cannot be reset or rewinded to the beginning. Thus, if the generator has already reached the end of its sequence, it will no longer produce values, even if new elements are added to the base sequence.
The answer is clear, and in general we could end here. But I still tried to get information about where exactly the progress goes if the generator is called after it has reached the end.
It is completely logical that an exception is thrown almost immediately StopIteration. But.. I wanted to know where exactly. Unfortunately the following example didn't give me any information
Example 2.3
def generator(list_):
print('start')
try:
for i in list_:
print('gen')
yield i
except StopIteration:
print('stopped')
def main():
nums = list()
gen = generator(nums)
nums.append(1)
next(gen)
nums.append(2)
for i in gen:
print(i)
nums.append(3)
print(tuple(gen))
next(gen) # raise StopIteration
nums.append(4)
nums.append(5)
main()
input:
start
gen
gen
2
()
Traceback (most recent call last):
in <module>
main()
in main
next(gen)
StopIteration
When converting gen V tuple after a full traversal, we get an empty array.
But.. none of the prints are inside our function generator did not work when converting.
Exactly the same as when calling next(gen). There are no prints.
Means. after the first call StopIteration the generator seems to store something like a flag end = True and when calling next returns immediately StopIteration? That's what I thought at first
Here's how the neuron explained it:
Your guess is that the generator stores some flag end=Truenot entirely accurate, but you are right that after the first call StopIteration the generator can no longer be used.
Any further method calls next() will cause the same exception to be thrown immediately StopIteration, without executing any code inside the generator function. This behavior is due to the fact that generators are disposable objects that cannot be reset or rewound. Once their iteration is complete, they are effectively dead and cannot be used again. This behavior is intentional and is a fundamental part of how generators work in Python.
All for nothing. we already knew that. I couldn't find a more accurate answer on the Internet.
Actually, I accidentally came across this while working on a telegram bot on aiogram.
I needed a generator that I would call one at a time, receiving messages that I need to send and processing the response from users. At the same time.. I needed the dynamism of the generator, which I showed in example 2.1since I assumed that the array on which the generator will be built will be replenished as the bot interacts with other users
Concluding my monologue, I would like to emphasize that generators are powerful and useful tools. However, like any other tool, they should be used with caution.
