system-type-data-pipeline Skill

Source

• Creator: microsoft
• Repository: microsoft/amplifier-bundle-systems-design
• Skill file: skills/system-type-data-pipeline/SKILL.md

What it does

Domain patterns for data pipeline architecture — batch processing, stream processing, ETL/ELT, DAG scheduling, data quality, schema evolution, backfill strategies, and failure modes. Use when designing or evaluating data pipelines, ETL systems, or streaming data infrastructure.

Skill instructions

System Type: Data Pipeline Patterns, failure modes, and anti-patterns for batch and streaming data pipelines. --- Core Patterns Batch Processing What it is. Process bounded datasets on a schedule — hourly, daily, or triggered. Read a full partition, transform, write output. The workhorse of data engineering. When to use. Reporting, analytics, ML feature generation, any workload where latency of minutes to hours is acceptable. When the source data naturally arrives in chunks (file drops, database snapshots, daily exports). When transformations are complex aggregations across the full dataset. When to avoid. When business requirements demand sub-minute freshness. When the dataset grows faster than the batch window can process it — you'll never catch up. When downstream consumers need continuous updates, not periodic dumps. Stream Processing What it is. Process unbounded data continuously as it arrives. Events flow through a topology of operators. State is maintained in-stream. When to us