How 7 engineers ran 150B records/day across ~10,000 DAGs
A four-year case study of building and operating an enterprise data platform from inside a business org — no IT support, no test-infra budget — and making an AI agent a first-class operator.
The Leverage Story
ConfirmedThe hook. Lead the writeup with these numbers.
A team of 4–7 engineers, embedded in the business org with no dedicated IT support, built and operated over 4 years a platform serving 14 orgs and 300 peak users. It runs ~10,000 DAGs producing ~9,500 tables from ~700 source systems, processes ~150 billion records/day and ~5 PB in a peak month, and absorbed a multi-million-dollar Oracle-to-BigQuery migration spanning 2,500 DAGs.
Leverage ratios
| Ratio | Value (at 7 engineers) | Why it matters |
|---|---|---|
| DAGs / engineer | ~1,440 | Industry typical: 30–100. Pipelines are not engineer-touched. |
| Users / engineer | 43:1 | Tier-1 support is automated into the product, not into team capacity. |
| Records / engineer / day | ~21B | Operational efficiency, not raw scale, is the story. |
| Deploys / engineer at peak | ~18 / day | Engineers don’t approve deploys. Governance is delegated to org QA roles. |
The Constraint Set
ConfirmedThe immovables. Every architecture decision answers a constraint here.
| Constraint | Implication |
|---|---|
| Business-org placement | No dedicated IT support. Infrastructure access was negotiated, not requisitioned. |
| Zero test-infra budget | No automated test suite. Compensating control: a full-volume UAT replica. |
| No network-layer control | IPv6-only databases on segregated networks → custom SSH tunneling for data access. |
| Cloud topology imposed | Metadata on managed Postgres, warehouse on BigQuery, legacy on Oracle — availability, not choice. |
| Team owns dev + support + releases | Self-service must be the default; capacity for individual requests is near-zero. |
The User Contract
ConfirmedThe headline abstraction. One sentence that earns the leverage ratio.
Analysts write SQL. They click a button. A governed pipeline — DAG, target schema, deployment, audit trail — is live in production in under a minute.
What users never write
- YAML — internal intermediate representation only
- DDL — inferred from SQL, generated, queued, approved, applied
- Python / DAG code — generated from templates at deploy time
- Deployment scripts — replaced by the in-platform CI/CD subsystem
What users do write
- SQL — source SELECT statements and transformation logic
- UI-declared metadata — DAG type, schedule, target, keys, validation thresholds
Layer 1 · SQL or Config, Never Code
CorrectedTakes analysts out of the engineering loop — plus the rule that makes the whole platform scale: one DAG = one table.
The discipline this forces: parse-time hygiene
At ~4,500 DAGs per environment, the scheduler re-parses the entire estate every few seconds. If any DAG file does real work at module-load, the scheduler falls behind and the platform degrades. The self-imposed rule: almost nothing runs at parse time.
- Connections, file handles, and clients are constructed inside
execute(), never at__init__ - Library imports must be lazy or cheap
- The DAG processor is tuned to keep parse cycles bounded
Layer 2 · Auto-DDL + Sub-Minute Deploy
ConfirmedTakes analysts out of the deployment loop.
Outcomes & Numbers
In progressWhat the leverage bought, in figures that survive anonymization.
| Metric | Figure |
|---|---|
| Users / engineer | 43:1 |
| Idea → production | <60s |
| Peak deploys / day | 128 |
| Data growth Y/Y | 14× |
| AI operator tools | 81 |
| At-scale processing | ~$7 / TB |
The Thesis
DraftWhat the platform proves about how enterprise data work should be organized.
Leverage at this ratio isn’t a tooling trick — it’s the compounding of four correct abstractions over four years. Build the boring layer right, and the impressive layer comes almost for free.
Take people out of the loop one loop at a time — the engineering loop, then the deployment loop, then the clicking loop. Each abstraction is only possible because the one beneath it held. The AI operator isn’t the story; it’s the dividend of work that was correct before the category existed.