The problem
Thousands of factory-floor sensors streamed readings into a pipeline that had grown ad hoc over several years. During peak load — usually a shift change across many sites at once — the pipeline silently dropped messages, and nobody found out until a customer asked why a machine's history had gaps.
What we did
- Ingestion. Rebuilt the pipeline around Kafka with proper backpressure handling, replacing a queue that had no way to signal it was falling behind.
- Platform. Moved processing onto Kubernetes with autoscaling tied to queue depth, and rewrote all infrastructure in Terraform so changes are reviewed in a pull request instead of made by hand.
- Observability. Built Grafana dashboards and alerts on the metrics that actually predicted an outage — queue lag and consumer errors — not just CPU and memory.
The outcome
Data loss during peak load is effectively gone, and the team now sees backpressure building minutes before it would have caused drops, not after a customer notices missing history. Every infrastructure change goes through code review before it reaches production.