Datacake - Data processing delays – Incident details

The full post mortem can be found on our Engineering Blog: https://engineering.datacake.co/post-mortem-service-degradation-on-march-14-2024/

All queues have now been cleared with no data loss. We apologize for these delays and appreciate your understanding. In dedication to clarity and process improvement, a comprehensive follow-up about the incident, explaining its causes and the measures taken to prevent recurrence, will be provided shortly.

The queries in question have now stabilized, resulting in a consistent reduction of the queue. We'll continue to provide updates on the situation in this space.

We've identified the initial root cause of the delays as a few long-running migration queries. We are actively monitoring the situation and adjusting resources as necessary to expedite the process.

We are still in the process of examining an extended measurement queue. Please note that data transmission still encounters delays.

We've identified the root cause of the recent issue as an unexpected high load on one of our databases. We've enhanced the resources allocated to this database and are currently monitoring its performance closely.

Please note that due to a pending backlog, temporary data gaps may be evident in the chart visualizations. However, we're working to ensure data integrity as soon as possible.