Datacake - Notice history

On December 8, several of our services experienced a brief period of instability due to an issue originating from our database provider. The incident began at 11:42 UTC and services recovered by 11:51 UTC.

What happened
Our vendor identified the root cause as a failure in their backend node replacement service. This failure caused delays in deploying new nodes, which led to performance issues and affected tasks related to DNS updates, including node replacements.
The impact was limited to databases whose nodes had recently been replaced or newly created, for example after forking or scheduled maintenance.

Resolution
The vendor has identified and fixed the underlying issue in their node replacement service. Once the affected database recovered, all Datacake services returned to normal operation and have remained stable since.

Next steps
We are continuing to work with the vendor to ensure the issue is fully understood and that safeguards are in place to prevent a recurrence.

All systems continue to be stable, so we are marking this incident as resolved. Our database vendor has acknowledged the issue as originating on their end and is continuing to investigate the root cause.

We’ve identified the root cause as an issue with one of our database providers. The affected database has recovered, and all services are fully back online.

While we continue to monitor the situation, we’re working closely with the vendor to understand what happened and to prevent future occurrences. We excuse any inconvenience caused and appreciate your patience.

The issue causing incomplete historical data display for certain devices has been resolved. All data is now fully accessible across dashboards and the application.

What Happened:

We identified an issue where historical data from early to mid-September 2025 was not appearing in queries and visualizations for specific devices in certain retention plans. The root cause was traced to a bug in our time-series database software related to Bloom filters on compressed data chunks. These filters were incorrectly marking data as non-existent, causing the database to skip chunks that actually contained the requested data.

Resolution:

Our database vendor Timescale identified and fixed a software bug in their compression filtering logic. The fix has been applied to our database instance, and all queries now return complete and accurate results.

Data Safety:

All data remained safely stored throughout this incident. No data was lost or corrupted - it was simply not being returned in certain filtered queries due to the database software bug.

We appreciate Timescale's quick response in identifying and resolving this issue. We apologize for any confusion or inconvenience this may have caused and thank you for your patience.