AI 服务暂时不可用，以下为来源正文，待恢复后补全翻译。
Iterators and Generators for Memory-Efficient Processing - MachineLearningMastery.com Iterators and Generators for Memory-Efficient Processing - MachineLearningMastery.com In this article, you will learn how Python’s iterators and generators can be used to build memory-efficient data processing pipelines that handle large files without loading them into memory all at once. Topics we will cover include: How the iterator protocol works and why it is the foundation of lazy, streaming data processing in Python. How to build a multi-stage pipeline using generator functions and generator expressions that process one record at a time. How to measure peak memory usage with tracemalloc to verify that a pipeline is genuinely lazy and identify where eager materialisation creeps back in. Iterators and Generators for Memory-Efficient Processing The script that summarizes the payments log ran fine for a year. It opened the file, built a list of parsed records, filtered the errors and printed a table. Then log rotation changed, a week of traffic landed in one 9 GB file, and the nightly job died with a MemoryError before it had summarized anything. Nothing in the parsing logic was wrong. The bug was the list. 1 2 3 4 5 with open("app.log") as f: records = [parse(line) for line in f] # every record exists at once errors = [r for r in records if r["level"] == "ERROR"] slow = [r for r in errors if int(r["latency_ms"]) > 1000] Nobody would flag any of those lines in review. Together they build three lists, and the first one has to hold every record in the file before the filter can start. If each step only needs one record at a time, why make every record exist at once? Python has had the tools to avoid that for over twenty years. Iterators, generator functions, and generator expressions let a pipeline pull one record through every stage and release it before the next one arrives. The trick is keeping the whole path lazy rather than sprinkling yield into one function. Lazy has to hold from the file object at the top to the batch written at the bottom, because a single eager step in between brings the list back. The tutorial stays in pure Python. Pandas chunking, Spark, Dask, and async streams all solve neighbouring problems. None of them is needed for a log file, an export, or a line-oriented dataset dump that fits on disk but not in RAM. Iterator vs Iterable: The Contract That Makes Streaming Work Python’s for loop hides a small contract, and the whole streaming story depends on it. An iterable is something you can ask for an iterator: a list, a string, a dictionary, an open file. The iterator is the thing that actually remembers where you are. When for line in f runs, Python calls iter(f) once, then calls next() on the result over and over until it raises StopIteration. StopIteration is the loop’s normal ending, not an error. The iterator protocol is just two methods. iter() hands back the iterator itself, and next() either produces the next item or raises StopIteration because there isn’t one. 1 2 3 4 5 6 numbers = iter([10, 20, 30]) next(numbers) # 10 next(numbers) # 20 next(numbers) # 30 next(numbers) # raises StopIteration The list still holds all three values. The iterator is not a copy of it but a cursor over it, and the cursor only moves forward. That one property is what makes streaming possible, because a cursor over a 9 GB file needs a small read buffer rather than 9 GB of strings. The forward-only cursor is also what catches people out. Once an iterator has raised StopIteration, the documentation is blunt: it must keep raising it on subsequent calls. An exhausted iterator does not rewind. When a generator appears to return nothing the second time a function loops over it, it wasn’t empty. The first loop consumed it, and the first loop is often a len() check, a debug print, or a list() call someone added while investigating. The fix is to iterate once, or to call the function that creates the generator again. Start With the File Object Before Writing a Generator Before writing any generator code, take the memory win the standard library already gives away. A file object is its own iterator, and the tutorial’s description of looping over it is that it is memory efficient, fast, and leads to simple code. f.read() pulls the entire file into one string, and f.readlines() builds a list with one string per line. for line in f reads a buffered chunk at a time and hands you one line. 1 2 3 4 5 6 7 8 9 10 # eager: the whole file becomes a list of strings first with open("app.log") as f: lines = f.readlines() for line in lines: handle(line) # lazy: one line in memory at a time with open("app.log") as f: for line in f: handle(line) For a log with a timestamp, a level, a service name, and a handful of key=value fields, that is already enough to parse and discard irrelevant lines without holding anything. The parser sees one line, decides whether the level or the service matters, and moves on. Notice what did not change: the algorithm. Line-by-line reading solves the memory problem only when the work is itself line-by-line. If the summary needs every record sorted by latency, or deduplicated across the whole week, a file iterator alone does not rescue it. Pretending otherwise is how the list comes back at step two. Turn Parsing Steps Into Generator Functions The file iterator handles reading, but parsing and filtering still live in one loop body, which is where scripts grow long and untestable. A generator function splits them into named stages that each stream. Any function containing yield becomes a generator function. Call it and nothing in the body runs yet. What comes back is a generator object, and the body advances one yield at a time as the caller asks for values. That is easy to forget the first time a print statement inside a generator never fires. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 import re LINE = re.compile(r"^(?P\S+) (?P[A-Z]+) (?P[\w-]+)(?P.*)$") def read_lines(paths): for path in paths: with open(path, encoding="utf-8") as f: yield from f def parse_records(lines): for line in lines: match = LINE.match(line) if match is None: continue # skip, count or log it; never stop the stream record = match.groupdict() record["fields"] = dict( item.split("=", 1) for item in record.pop("rest").split() if "=" in item ) yield record def only_level(records, level): for record in records: if record["level"] == level: yield record read_lines uses yield from to hand every line of each rotated file up to the caller. Three log files therefore behave like one stream, and each file is closed before the next one opens. parse_records turns a line into a dictionary and yields it, only_level passes through the ones that matter, and none of the three knows or cares how large the input is. What makes this work is suspension. When parse_records reaches yield, the language reference says all local state is retained: the current bindings of local variables, the instruction pointer, the internal evaluation stack. The function is paused mid-loop with match and record still in scope, and it resumes on the next request as if yield had been an ordinary call. Suspension at yield is why a generator holds one record at a time. Record size never enters into it. The function hasn’t reached the next line of the file, so the next record does not exist yet, in memory or anywhere else. When the loop runs out and the function returns, the generator raises StopIteration on the caller’s behalf, and the stages stacked above it drain out one after another. Each stage can also be tested on its own with a list of three lines. That is a better deal than testing a forty-line loop body by running it against production data. Keep the Middle of the Pipeline Lazy Wire the stages together and the pipeline becomes a chain of assignments that do no work until the end: 1 2 3 4 5 6 7 8 9 10 from itertools import islice lines = read_lines(["app.log.2", "app.log.1", "app.log"]) records = parse_records(lines) errors = only_level(records, "ERROR") slow = (r for r in errors if int(r["fields"].get("latency_ms", 0)) > 1000) rows = ((r["ts"], r["service"], r["fields"].get("request_id", "-")) for r in slow) for row in islice(rows, 5): print(row) The two parenthesized expressions are generator expressions, the one-line cousins of generator functions. Where a list comprehension in square brackets builds the whole list, the same expression in parentheses builds a generator that evaluates one element per request. They belong in the small transformations between named stages: a filter with one condition, a projection down to the fields the sink needs. One point about laziness is worth knowing precisely: the language reference states that the iterable in the leftmost for clause is evaluated immediately, and everything else lazily. So (r for r in errors if ...) evaluates the name errors right away, which is harmless because errors is already a generator. Write (r for r in load_everything() if ...) and load_everything() runs in full at the moment the expression is defined, not when the first value is requested. The generator expression did not make the source lazy, only the filtering. islice is the tool for looking without consuming. That five-row loop is the fastest sanity check available: five rows come through the whole chain, the loop stops, and the other 9 GB never gets touched. Run the five-row check before every other change to the pipeline. Most of itertools follows the same rule of returning an iterator rather than a list, so chain can glue two sources together and takewhile can cut a stream off at a condition without either one holding anything. Use them when they replace a stage you’d otherwise write by hand, and prefer a plain generator function with a good name whenever that reads better, which it often does. The traps sit at the other end of the chain, and most of them look like tidy code. list(rows) is the obvious one. sorted(rows, key=...) is the same trap with better manners, since a sort cannot return its first element until it has seen the last. A dictionary keyed by request_id, for counting or deduplication, grows with the number of distinct IDs. In a payments log that is close to the number of records, so it is the list from the opening in a different container. Then there are the library calls that materialise on your behalf: build a DataFrame from records, or call json.dumps(list(rows)), and the entire stream is sitting in memory under a friendlier name. Speed is a separate question. Generators aren’t faster than lists, and sum() over a generator does the same work as sum() over a list, sometimes a little more. The benefit is bounded memory and stages you can rearrange, which is a different promise from speed. The four stages of the pipeline and what each one holds at any moment. Every stage passes one record along and keeps nothing; only the sink materialises, and only a batch at a time. Alt text: left-to-right diagram with a log file feeding a file iterator, a parse generator, a filter and projection generator expression, and a bounded batch sink, each labelled with the memory it holds. Credit: original diagram based on the Python iterator, yield expression and itertools documentation. Batch Only at the Boundary Laziness is not a religion. Somewhere the rows have to land: a file, a database, a model that scores them, an API that accepts a POST. Most of those consumers want a group rather than a single row, and a bounded group is the correct thing to materialise. The rule is that the batch has a size you chose, and nothing upstream of it is ever whole. Since Python 3.12 the standard library has had itertools.batched(), which is this pattern done properly. It pulls just enough from the input to fill a tuple of n, [truncated for AI cost control]