翻訳待ち：Re-engineering Apache Airflow for speed and scale

AI サービスが一時的に利用できないため、復旧後に翻訳を補完します。

← Blog|AUG 31 2026 The Astro Runtime: Airflow re-engineered for speed and scale Ian BussPrincipal Software Engineer Michael ClaassenStaff Software Engineer Jed CunninghamPrincipal Software Engineer Neel DalsaniaStaff Software Engineer Julian LaNeveCTO Carter Page Executive Vice President, R&D 36 min read | Apache Airflow is the open-source standard for defining and running data workflows. Astronomer builds Astro, a managed Airflow platform for teams running those workflows in production. Airflow is no longer used only for scheduled batch pipelines. Teams now use it to coordinate thousands of data and AI workflows across an enterprise: loading warehouse tables, building dbt models, preparing training data, running model evaluations, and reacting to external events. These workflows can release tens of thousands of tasks at once, keep hundreds of thousands running, and still require a newly ready task to start in hundreds of milliseconds. Our customers hit ceilings in how fast Airflow starts a task and how many it keeps running, so we rebuilt the path behind both. Over the past several years, we have rebuilt Airflow's scheduling, execution, scaling, and recovery systems on Astro to perform beyond any current demands. And we did it in a way where Astro still works seamlessly with Airflow 3+ releases, meaning nothing needs to change with how pipelines are defined. In our load tests, Astro sustained 500,000 concurrent Airflow tasks in a single deployment and reached 228 milliseconds p95 task-start latency at 100,000 concurrent tasks—less than one-hundredth of the 23.582 seconds we measured when we pushed open-source Airflow to just half that load. At 300,000 concurrent tasks, p95 remained 294 milliseconds. We made these changes without replacing Airflow's workflow model. User Dags, task dependencies, task execution status, and operator code remain compatible with open-source Airflow. Astro just changes components under the hood to make task execution faster and more scalable while improving reliability: Coordination Airflow API server1-10 replicas Airflow triggererup to 8 replicas Astro Scheduler1-10 replicas228ms p95 task start latency Airflow Dag processor2 replicas with HA Execution— one executor per deployment Astro Executor500,000 concurrent tasks Airflow Celery executorautoscaled workers Airflow Kubernetes executorone pod per task Platform Astro HypervisorAirflow-aware scaling and healing Multi-Region DBRPO <15m · RTO <1h Astro Runtime image2 years of maintenance Heavy borders mark the systems Astronomer built; the rest is Apache Airflow. This is the control path around a task, not every service in a deployment. How Airflow got here Airflow began 10 years ago with a clear job: read workflows defined as Python code, decide which tasks were ready, and send them somewhere to run. A scheduler made those decisions, an executor handled the delivery, and a SQL database recorded Dag runs and task states. That design gave Airflow two useful properties. Dag authors could choose how tasks ran without changing their workflow code, and a failed scheduler could rebuild its view from the database. The database, rather than any scheduler process, held the lasting account of the work. As deployments added more Dags and tasks, the scheduler had to parse more Python files, create more Dag runs, check more dependencies, and queue more task instances. One scheduler also left the whole deployment dependent on one process. Airflow 2.0 and beyond addressed those limits with a few large changes, without replacing the database model. AIP-15 let several scheduler replicas run at once. Each could examine Dag runs in parallel, then use database row locks when moving task instances from scheduled into an executor. Airflow also changed to store serialized Dags in the database, so a scheduler could make decisions from a stored graph instead of importing every Python Dag file itself. Airflow 2.2 introduced ways to offload tasks that were waiting for some external system via the triggerer. Consider a daily ingestion Dag that cannot start until a partner uploads complete.json to an object-storage bucket. A sensor may spend hours checking for that file while doing almost no work, yet it can still occupy a worker slot. A deferrable operator stores what it needs to resume, releases the worker, and hands the wait to a triggerer. When the file arrives, the trigger fires and the scheduler checks the task again before it can run. Airflow 2.3 then introduced the separate Dag parser component, which separated out user code from the core scheduler component and loop. This was much better from a security/isolation perspective and also meant that Dag processing could scale separately from scheduling. Airflow 3 gave task execution a clear API contract. Under AIP-72, task processes no longer need direct access to the metadata database like they did in Airflow 2.x. The Task SDK and Execution API define what a task receives, including its identity, connections, and variables, and what it reports, including heartbeats, state changes, and XComs (used for cross-task communication). Workers and task code use that interface instead of reading and updating Airflow's internal tables. Modern Airflow divides the original system's work among Dag processing, active-active schedulers, triggerers, executors, API servers, and workers. This is the architecture we helped build in Airflow: durable database state, active-active scheduling, deferred execution, and a clear task API. At Astronomer, we led the HA scheduler, the triggerer and deferred tasks, and the Task SDK and Execution API, and we contributed to moving Dag processing out of the scheduler loop. As deployments grew and workflows carried more weight, we also saw where the task-start path needed a different design. The lifecycle of an Airflow task Here is what happens after one task finishes, first in open-source Airflow and then in Astro. The Dag is a basic three-task ETL. In our example Dag, when extract succeeds, Airflow records that result in the metadata database. The scheduler must notice the change, confirm that transform may run, check shared limits, queue it through the configured executor, and wait for a worker to start its process. Only then does transform begin running. These steps run in order. transform cannot skip any of them, so each adds its own wait to the total. 1. The upstream task finishes The dag_run state for example_etl is running, the task_instance state for extract is success, and the rows for transform and publish have a NULL state. Airflow reads the dependency from the serialized Dag and evaluates it against the task states for this run on a loop. The lasting record of transform is its task_instance row, not an in-memory object passed from one process to another. Different parts of Airflow update that row as they make decisions about it. 2. A scheduling pass must discover the change The standard scheduler works in batches. It queries for active Dag runs, calls Airflow's dependency logic, and returns later for another pass, on a loop. Until one of those passes examines our Dag run, transform remains NULL even though extract has already recorded success. Every tuning knob here trades one cost for another. Larger batches raise throughput but let one scheduler hoard runs; running the loop more often cuts the wait but issues more queries when nothing has changed, and at high load those passes compete with heartbeats, state updates, and API calls for the same database. Adding schedulers helps throughput and availability, but the replicas still coordinate through database locks. When a scheduler finds transform and its dependencies pass, it changes the row to scheduled. By this point, transform has waited for a scheduling pass and dependency checks. 3. Shared limits must clear before the task queues When a task is scheduled, that does not necessarily mean it's ready to run. Airflow has a robust queueing and pooling system, which lets a Dag author declare cross-Dag behavior for how tasks run. For example, an external service may impose API rate limits, so the author declares a pool that limits how many tasks interacting with that API can run. Before changing a task to queued, the scheduler must account for limits shared by many Dags: pool slots, global system concurrency, Dag concurrency, task concurrency, and executor capacity. Its critical section protects those shared limits while it selects a batch of tasks. More scheduler replicas can therefore increase both scheduling capacity and contention for the same database state. By this point, transform has waited for a scheduling pass, dependency checks, shared-limit checks, and a database update. 4. The executor must deliver the workload Airflow separates the decision to run a task from the system that starts it. Once the scheduler marks transform as queued, the configured executor takes over. LocalExecutor starts a process on the same machine, CeleryExecutor sends a workload to a pool of workers through a broker, usually Redis or RabbitMQ, or KubernetesExecutor asks Kubernetes to create a pod. We focus on CeleryExecutor because it has long been a common choice for distributed Airflow deployments and thus is the open-source execution path used in our benchmarks. Celery adds a delivery chain on top of the scheduling work: the scheduler serializes a workload, the broker stores it on the right queue, and a worker consumes it subject to its free concurrency. During a burst, every link in that chain handles more connections, deeper queues, and workers already holding prefetched work—and adding workers does not remove the broker round trip. By this point, transform has waited for a scheduling pass, dependency checks, shared-limit checks, a database update, and a broker round trip. 5. A worker must have a free slot and start the process Receiving the workload does not start the task. transform remains queued until the Celery worker has a concurrency slot and starts a child process. Under Airflow 3, that process uses the Task SDK and Execution API to report that it is running, send heartbeats, fetch the values it needs, and report its final state. If every matching worker is full, the workload waits in the broker. If no worker exists, the deployment must start a worker VM or pod before the process can run. Broker wait, worker capacity, and infrastructure startup therefore all count toward the delay after the scheduler queues the task. By this point, transform has waited for a scheduling pass, dependency checks, shared-limit checks, a database update, a broker round trip, and a worker slot. Only now does it become running. Capacity must already exist For there to be no infrastructure holdups, every step above assumes the schedulers, database, broker, and workers it needs are already running. That assumption has its own timing problem. Infrastructure metrics describe what a deployment is doing now, while Airflow's tables often show what it will need next. A trigger can wait for an external event while using little CPU, then make thousands of task instances runnable when the event arrives. A timetable can show that a large Dag run is due before its first task consumes any worker resource. A growing queue can appear in the database before a queue depth metric crosses a scaling threshold. If queue depth is the only scaling input, new workers begin to start only after the existing workers are busy and tasks are waiting. Schedulers, API servers, triggerers, and database proxies have the same timing problem: each sees one part of demand, but a task needs capacity across the whole path. How the waits add up At modest load, each stage can finish quickly. As task counts grow, the wait at every gate below grows with it. 1. extract finishes Airflow records the result. The row for transform already exists with a NULL state. Nothin [truncated for AI cost control]