Master Python Yield: Boost Performance with Efficient Generator Functions

The Python yield statement changes a function into a generator function. Calling it produces a generator object without running the function body immediately; execution resumes when the caller requests the next value. That lazy behavior is useful for processing sequences, streams and pipelines without eagerly constructing every result.

Generators are not a universal optimization. They shift work into iteration and require attention to exhaustion, resource lifetime, exceptions and the number of times data must be consumed. The examples below show the precise behaviors that distinguish yield from return.

A generator function versus return

A return statement ends a function call and produces its final result. A yield expression suspends execution, preserves local state and can produce multiple values over successive iterations.

def numbers():
    yield 1
    yield 2
    yield 3
for value in numbers():
    print(value)

Iteration and exhaustion

The iterator protocol calls next() to request a value. Once a generator is exhausted it raises StopIteration; it does not automatically restart. Create a new generator by calling the generator function again.

it = numbers()
print(next(it))  # 1
print(list(it))  # [2, 3]
print(list(it))  # []

Filter data lazily

A generator expression applies a selection as items are requested. It can reduce peak memory compared with a materialized list, but it still takes time to compute each item and may retain references to underlying resources.

values = range(10)
evens = (n for n in values if n % 2 == 0)
print(next(evens))  # 0
print(list(evens))  # [2, 4, 6, 8]

Yield from and delegation

yield from passes values from another iterable or generator to the caller. It can simplify composition and also forwards certain iterator operations, so understand the delegated generator’s lifecycle before using it with advanced send or throw behavior.

def combined():
    yield from range(3)
    yield from ("a", "b")
print(list(combined()))

Generators for file processing

A generator can process one input line at a time, but opening a file inside a generator changes when the file is actually opened and closed. Ensure the consumer exhausts the generator or explicitly closes resources. A with statement around iteration is often clearer.

with open("events.log", encoding="utf-8") as file:
    for line in file:
        if "ERROR" in line:
            print(line.rstrip())

Sending values back into a generator

yield is an expression that may receive a value from send(). An unstarted generator must normally be advanced with next() or send(None) before sending a non-None value. This advanced feature is useful for some coroutines, but ordinary function arguments are simpler for many tasks.

def echo():
    value = yield "ready"
    yield f"received: {value}"
g = echo()
print(next(g))
print(g.send("hello"))

Exception handling and closing

Errors inside generator code propagate when iteration reaches them. try/finally can release resources when the generator terminates or is closed. Do not assume a deferred generator computation succeeded merely because construction returned an object.

def sample():
    try:
        yield "one"
    finally:
        print("cleanup")
g = sample()
next(g)
g.close()

Fibonacci and infinite streams

Generators can describe potentially unbounded sequences. Consumers must set a limit or stop condition so they do not iterate forever. Remember that laziness alone is not a performance benchmark; profile the workload with realistic input sizes.

from itertools import islice
def fibonacci():
    a, b = 0, 1
    while True:
        yield a
        a, b = b, a + b
print(list(islice(fibonacci(), 6)))

Conclusion

Use yield when a caller benefits from receiving values incrementally and can handle iterator state explicitly. Choose lists when repeated indexing or reuse matters, and use controlled resource lifetimes and error reporting for streaming pipelines.