Summary

During a routine deployment, we introduced a database migration that ended up degrading the performance of our primary database more than expected. This slowed down our ingestion pipeline and caused a growing backlog of incoming measurements.

As the backlog increased, it exposed a weakness in how our ingestion workers handle concurrency. Instead of recovering once the database performance stabilized, the system got stuck in a state where workers were effectively slowing each other down. That meant the backlog couldn’t clear itself, and data delays persisted longer than they should have.

Throughout the incident, no measurement data was lost. However, dashboards and API responses showed outdated values until the system fully recovered.

Root Cause

There were two things at play here.

The initial trigger was the database migration, which temporarily made queries slower and reduced ingestion throughput. That part alone would have been manageable.

The bigger issue was how our ingestion workers behaved under pressure. They were configured to handle many tasks concurrently using shared database and cache connections. Under normal conditions, this works well and is efficient. But once the backlog built up, tasks started competing for those shared resources. Instead of working through the queue faster, they got in each other’s way, turning a temporary slowdown into a sustained bottleneck.

Resolution

We first addressed the database performance issue to remove the initial trigger.

After that, we changed how ingestion workers run. Instead of using a shared, thread-based model, we moved to process-based workers where each one has its own dedicated connections. This removed the contention entirely, and the system caught up within minutes.

Follow-up Actions

We’re adding better visibility into this kind of situation, especially around queue depth and processing times, so we can react earlier if something similar happens again.

We’re also reviewing other parts of the system for similar concurrency patterns that could behave poorly under load.

Finally, we’ll introduce an additional staging step for database migrations to catch performance regressions before they reach production.

If anything was unclear during the incident or you have questions, feel free to reach out.

Datacake - undefined undefined

API outage