Datacake Status - Incident history https://datacake-status.com Datacake Wed, 13 May 2026 01:00:00 +0000 Data processing delays Type: Incident Duration: 4 hours and 32 minutes Affected Components: MQTT Server, Outgoing Webhooks May 13, 01:00:00 GMT+0 - Investigating - Our system is experiencing processing delays. No data has been lost, and the system should be caught up shortly. May 13, 05:31:45 GMT+0 - Resolved - This incident has been resolved. Type: Incident

Duration: 4 hours and 32 minutes

Affected Components: ,

<p><small>May <var data-var='date'> 13</var>, <var data-var='time'>01:00:00</var> GMT+0</small><br><strong>Investigating</strong> - Our system is experiencing processing delays. No data has been lost, and the system should be caught up shortly..</p> <p><small>May <var data-var='date'> 13</var>, <var data-var='time'>05:31:45</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> ]]> Wed, 13 May 2026 01:00:00 +0000 https://datacake-status.com/incident/cmp3lfztl01xoqo9so968nohl https://datacake-status.com/incident/cmp3lfztl01xoqo9so968nohl Investigating intermittent time series data fetch errors Type: Incident Duration: 28 minutes Affected Components: GraphQL API, Reports, Web Application Apr 28, 11:35:14 GMT+0 - Investigating - We are currently investigating sporadic errors when fetching time series data. This may result in incomplete or delayed data being displayed in dashboards and via the API. Our team is actively looking into the root cause and will provide updates as we learn more. Apr 28, 12:03:34 GMT+0 - Resolved - After investigating the issue, we worked with our database operator to implement a fix. Data fetching has returned to normal and all systems are operating as expected. Type: Incident

Duration: 28 minutes

Affected Components: , ,

<p><small>Apr <var data-var='date'> 28</var>, <var data-var='time'>11:35:14</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating sporadic errors when fetching time series data. This may result in incomplete or delayed data being displayed in dashboards and via the API. Our team is actively looking into the root cause and will provide updates as we learn more..</p> <p><small>Apr <var data-var='date'> 28</var>, <var data-var='time'>12:03:34</var> GMT+0</small><br><strong>Resolved</strong> - After investigating the issue, we worked with our database operator to implement a fix. Data fetching has returned to normal and all systems are operating as expected..</p> ]]> Tue, 28 Apr 2026 11:35:14 +0000 https://datacake-status.com/incident/cmoijujoz00gp11fs07a63i3i https://datacake-status.com/incident/cmoijujoz00gp11fs07a63i3i API outage Type: Incident Duration: 5 hours and 16 minutes Mar 17, 19:30:32 GMT+0 - Resolved - All systems have fully recovered. The ingestion backlog has been cleared and incoming data is now being processed in real time again. Dashboards and API responses are up to date. We’ll publish a detailed postmortem shortly with more information on what happened and what we’re doing to prevent this in the future. Mar 17, 19:39:02 GMT+0 - Postmortem - ### Summary During a routine deployment, we introduced a database migration that ended up degrading the performance of our primary database more than expected. This slowed down our ingestion pipeline and caused a growing backlog of incoming measurements. As the backlog increased, it exposed a weakness in how our ingestion workers handle concurrency. Instead of recovering once the database performance stabilized, the system got stuck in a state where workers were effectively slowing each other down. That meant the backlog couldn’t clear itself, and data delays persisted longer than they should have. Throughout the incident, no measurement data was lost. However, dashboards and API responses showed outdated values until the system fully recovered. --- ### Root Cause There were two things at play here. The initial trigger was the database migration, which temporarily made queries slower and reduced ingestion throughput. That part alone would have been manageable. The bigger issue was how our ingestion workers behaved under pressure. They were configured to handle many tasks concurrently using shared database and cache connections. Under normal conditions, this works well and is efficient. But once the backlog built up, tasks started competing for those shared resources. Instead of working through the queue faster, they got in each other’s way, turning a temporary slowdown into a sustained bottleneck. --- ### Resolution We first addressed the database performance issue to remove the initial trigger. After that, we changed how ingestion workers run. Instead of using a shared, thread-based model, we moved to process-based workers where each one has its own dedicated connections. This removed the contention entirely, and the system caught up within minutes. --- ### Follow-up Actions We’re adding better visibility into this kind of situation, especially around queue depth and processing times, so we can react earlier if something similar happens again. We’re also reviewing other parts of the system for similar concurrency patterns that could behave poorly under load. Finally, we’ll introduce an additional staging step for database migrations to catch performance regressions before they reach production. --- If anything was unclear during the incident or you have questions, feel free to reach out. Mar 17, 14:14:05 GMT+0 - Investigating - We are currently investigating issues with the availability of our API. Mar 17, 14:22:06 GMT+0 - Identified - We have identified the issue to be caused by an upstream database. We are continuing to work on a fix for this incident. Mar 17, 14:39:51 GMT+0 - Monitoring - We identified a number of database queries that got stuck, which impacted overall performance. After clearing those, things are back to normal again. There is still a backlog of data being processed, so you might notice slight delays in data ingestion until the queue has fully caught up. We’re keeping a close eye on it and will share further updates if needed. Mar 17, 17:28:43 GMT+0 - Monitoring - There is still a backlog of data being processed, which is currently causing ingestion delays of around 30–40 minutes. We’re actively working on improving queue performance to reduce this delay and will continue to keep you updated as things progress. Type: Incident

Duration: 5 hours and 16 minutes

<p><small>Mar <var data-var='date'> 17</var>, <var data-var='time'>19:30:32</var> GMT+0</small><br><strong>Resolved</strong> - All systems have fully recovered. The ingestion backlog has been cleared and incoming data is now being processed in real time again. Dashboards and API responses are up to date. We’ll publish a detailed postmortem shortly with more information on what happened and what we’re doing to prevent this in the future..</p> <p><small>Mar <var data-var='date'> 17</var>, <var data-var='time'>19:39:02</var> GMT+0</small><br><strong>Postmortem</strong> - ### Summary During a routine deployment, we introduced a database migration that ended up degrading the performance of our primary database more than expected. This slowed down our ingestion pipeline and caused a growing backlog of incoming measurements. As the backlog increased, it exposed a weakness in how our ingestion workers handle concurrency. Instead of recovering once the database performance stabilized, the system got stuck in a state where workers were effectively slowing each other down. That meant the backlog couldn’t clear itself, and data delays persisted longer than they should have. Throughout the incident, no measurement data was lost. However, dashboards and API responses showed outdated values until the system fully recovered. --- ### Root Cause There were two things at play here. The initial trigger was the database migration, which temporarily made queries slower and reduced ingestion throughput. That part alone would have been manageable. The bigger issue was how our ingestion workers behaved under pressure. They were configured to handle many tasks concurrently using shared database and cache connections. Under normal conditions, this works well and is efficient. But once the backlog built up, tasks started competing for those shared resources. Instead of working through the queue faster, they got in each other’s way, turning a temporary slowdown into a sustained bottleneck. --- ### Resolution We first addressed the database performance issue to remove the initial trigger. After that, we changed how ingestion workers run. Instead of using a shared, thread-based model, we moved to process-based workers where each one has its own dedicated connections. This removed the contention entirely, and the system caught up within minutes. --- ### Follow-up Actions We’re adding better visibility into this kind of situation, especially around queue depth and processing times, so we can react earlier if something similar happens again. We’re also reviewing other parts of the system for similar concurrency patterns that could behave poorly under load. Finally, we’ll introduce an additional staging step for database migrations to catch performance regressions before they reach production. --- If anything was unclear during the incident or you have questions, feel free to reach out..</p> <p><small>Mar <var data-var='date'> 17</var>, <var data-var='time'>14:14:05</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating issues with the availability of our API..</p> <p><small>Mar <var data-var='date'> 17</var>, <var data-var='time'>14:22:06</var> GMT+0</small><br><strong>Identified</strong> - We have identified the issue to be caused by an upstream database. We are continuing to work on a fix for this incident..</p> <p><small>Mar <var data-var='date'> 17</var>, <var data-var='time'>14:39:51</var> GMT+0</small><br><strong>Monitoring</strong> - We identified a number of database queries that got stuck, which impacted overall performance. After clearing those, things are back to normal again. There is still a backlog of data being processed, so you might notice slight delays in data ingestion until the queue has fully caught up. We’re keeping a close eye on it and will share further updates if needed..</p> <p><small>Mar <var data-var='date'> 17</var>, <var data-var='time'>17:28:43</var> GMT+0</small><br><strong>Monitoring</strong> - There is still a backlog of data being processed, which is currently causing ingestion delays of around 30–40 minutes. We’re actively working on improving queue performance to reduce this delay and will continue to keep you updated as things progress..</p> ]]> Tue, 17 Mar 2026 14:14:05 +0000 https://datacake-status.com/incident/cmmup11h004vcwv6z360kksn8 https://datacake-status.com/incident/cmmup11h004vcwv6z360kksn8 API outage Type: Incident Duration: 5 minutes Feb 19, 03:23:12 GMT+0 - Investigating - We are currently investigating issues with the availability of our API. Feb 19, 03:28:21 GMT+0 - Resolved - We have identified and fixed the underlying issue, which was caused by a database having issues. We have replaced this database with another instance and the service is back online. To ensure such outages do not happen again, we will continue making our infrastructure more robust and fail-safe. Type: Incident

Duration: 5 minutes

<p><small>Feb <var data-var='date'> 19</var>, <var data-var='time'>03:23:12</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating issues with the availability of our API..</p> <p><small>Feb <var data-var='date'> 19</var>, <var data-var='time'>03:28:21</var> GMT+0</small><br><strong>Resolved</strong> - We have identified and fixed the underlying issue, which was caused by a database having issues. We have replaced this database with another instance and the service is back online. To ensure such outages do not happen again, we will continue making our infrastructure more robust and fail-safe..</p> ]]> Thu, 19 Feb 2026 03:23:12 +0000 https://datacake-status.com/incident/cmlswbuki05x9ldpd5avf1py4 https://datacake-status.com/incident/cmlswbuki05x9ldpd5avf1py4 API outage Type: Incident Duration: 5 minutes Feb 3, 03:18:52 GMT+0 - Investigating - We are currently investigating issues with the availability of our API. Feb 3, 03:24:01 GMT+0 - Resolved - We have identified and fixed the underlying issue, which was caused by a database having issues. We have replaced this database with another instance and the service is back online. To ensure such outages do not happen again, we will continue making our infrastructure more robust and fail-safe. Type: Incident

Duration: 5 minutes

<p><small>Feb <var data-var='date'> 3</var>, <var data-var='time'>03:18:52</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating issues with the availability of our API..</p> <p><small>Feb <var data-var='date'> 3</var>, <var data-var='time'>03:24:01</var> GMT+0</small><br><strong>Resolved</strong> - We have identified and fixed the underlying issue, which was caused by a database having issues. We have replaced this database with another instance and the service is back online. To ensure such outages do not happen again, we will continue making our infrastructure more robust and fail-safe..</p> ]]> Tue, 3 Feb 2026 03:18:52 +0000 https://datacake-status.com/incident/cml614mv303q33t49b5pd25jq https://datacake-status.com/incident/cml614mv303q33t49b5pd25jq API outage Type: Incident Duration: 11 minutes Feb 2, 03:20:16 GMT+0 - Investigating - We are currently investigating issues with the availability of our API. Feb 2, 03:31:00 GMT+0 - Resolved - We have identified and fixed the underlying issue, which was caused by a database having issues. We have replaced this database with another instance and the service is back online. To ensure such outages do not happen again, we will continue making our infrastructure more robust and fail-safe. Type: Incident

Duration: 11 minutes

<p><small>Feb <var data-var='date'> 2</var>, <var data-var='time'>03:20:16</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating issues with the availability of our API..</p> <p><small>Feb <var data-var='date'> 2</var>, <var data-var='time'>03:31:00</var> GMT+0</small><br><strong>Resolved</strong> - We have identified and fixed the underlying issue, which was caused by a database having issues. We have replaced this database with another instance and the service is back online. To ensure such outages do not happen again, we will continue making our infrastructure more robust and fail-safe..</p> ]]> Mon, 2 Feb 2026 03:20:16 +0000 https://datacake-status.com/incident/cml4lqkyc04sr4dfo1uxyuc31 https://datacake-status.com/incident/cml4lqkyc04sr4dfo1uxyuc31 API outage Type: Incident Duration: 5 minutes Jan 30, 03:23:13 GMT+0 - Investigating - We are currently investigating issues with the availability of our API. Jan 30, 03:28:20 GMT+0 - Resolved - We have identified and fixed the underlying issue, which was caused by a database having issues. We have replaced this database with another instance and the service is back online. To ensure such outages do not happen again, we will continue making our infrastructure more robust and fail-safe. Type: Incident

Duration: 5 minutes

<p><small>Jan <var data-var='date'> 30</var>, <var data-var='time'>03:23:13</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating issues with the availability of our API..</p> <p><small>Jan <var data-var='date'> 30</var>, <var data-var='time'>03:28:20</var> GMT+0</small><br><strong>Resolved</strong> - We have identified and fixed the underlying issue, which was caused by a database having issues. We have replaced this database with another instance and the service is back online. To ensure such outages do not happen again, we will continue making our infrastructure more robust and fail-safe..</p> ]]> Fri, 30 Jan 2026 03:23:13 +0000 https://datacake-status.com/incident/cml0bitpc0g0x7icl34wde15h https://datacake-status.com/incident/cml0bitpc0g0x7icl34wde15h API outage Type: Incident Duration: 5 minutes Jan 17, 03:18:55 GMT+0 - Investigating - We are currently investigating issues with the availability of our API. Jan 17, 03:24:02 GMT+0 - Resolved - We have identified and fixed the underlying issue, which was caused by a database having issues. We have replaced this database with another instance and the service is back online. To ensure such outages do not happen again, we will continue making our infrastructure more robust and fail-safe. Type: Incident

Duration: 5 minutes

<p><small>Jan <var data-var='date'> 17</var>, <var data-var='time'>03:18:55</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating issues with the availability of our API..</p> <p><small>Jan <var data-var='date'> 17</var>, <var data-var='time'>03:24:02</var> GMT+0</small><br><strong>Resolved</strong> - We have identified and fixed the underlying issue, which was caused by a database having issues. We have replaced this database with another instance and the service is back online. To ensure such outages do not happen again, we will continue making our infrastructure more robust and fail-safe..</p> ]]> Sat, 17 Jan 2026 03:18:55 +0000 https://datacake-status.com/incident/cmkhqn7yo02i1tvihrk62wzn4 https://datacake-status.com/incident/cmkhqn7yo02i1tvihrk62wzn4 Performance degradation Type: Incident Duration: 51 minutes Affected Components: GraphQL API Dec 8, 11:43:42 GMT+0 - Investigating - We are investigating increased latencies with the API. Dec 8, 11:58:14 GMT+0 - Monitoring - We’ve identified the root cause as an issue with one of our database providers. The affected database has recovered, and all services are fully back online. While we continue to monitor the situation, we’re working closely with the vendor to understand what happened and to prevent future occurrences. We excuse any inconvenience caused and appreciate your patience. Dec 8, 12:34:21 GMT+0 - Resolved - All systems continue to be stable, so we are marking this incident as resolved. Our database vendor has acknowledged the issue as originating on their end and is continuing to investigate the root cause. Dec 9, 12:51:39 GMT+0 - Postmortem - On December 8, several of our services experienced a brief period of instability due to an issue originating from our database provider. The incident began at 11:42 UTC and services recovered by 11:51 UTC. **What happened** Our vendor identified the root cause as a failure in their backend node replacement service. This failure caused delays in deploying new nodes, which led to performance issues and affected tasks related to DNS updates, including node replacements. The impact was limited to databases whose nodes had recently been replaced or newly created, for example after forking or scheduled maintenance. **Resolution** The vendor has identified and fixed the underlying issue in their node replacement service. Once the affected database recovered, all Datacake services returned to normal operation and have remained stable since. **Next steps** We are continuing to work with the vendor to ensure the issue is fully understood and that safeguards are in place to prevent a recurrence. Type: Incident

Duration: 51 minutes

Affected Components:

<p><small>Dec <var data-var='date'> 8</var>, <var data-var='time'>11:43:42</var> GMT+0</small><br><strong>Investigating</strong> - We are investigating increased latencies with the API..</p> <p><small>Dec <var data-var='date'> 8</var>, <var data-var='time'>11:58:14</var> GMT+0</small><br><strong>Monitoring</strong> - We’ve identified the root cause as an issue with one of our database providers. The affected database has recovered, and all services are fully back online. While we continue to monitor the situation, we’re working closely with the vendor to understand what happened and to prevent future occurrences. We excuse any inconvenience caused and appreciate your patience..</p> <p><small>Dec <var data-var='date'> 8</var>, <var data-var='time'>12:34:21</var> GMT+0</small><br><strong>Resolved</strong> - All systems continue to be stable, so we are marking this incident as resolved. Our database vendor has acknowledged the issue as originating on their end and is continuing to investigate the root cause..</p> <p><small>Dec <var data-var='date'> 9</var>, <var data-var='time'>12:51:39</var> GMT+0</small><br><strong>Postmortem</strong> - On December 8, several of our services experienced a brief period of instability due to an issue originating from our database provider. The incident began at 11:42 UTC and services recovered by 11:51 UTC. **What happened** Our vendor identified the root cause as a failure in their backend node replacement service. This failure caused delays in deploying new nodes, which led to performance issues and affected tasks related to DNS updates, including node replacements. The impact was limited to databases whose nodes had recently been replaced or newly created, for example after forking or scheduled maintenance. **Resolution** The vendor has identified and fixed the underlying issue in their node replacement service. Once the affected database recovered, all Datacake services returned to normal operation and have remained stable since. **Next steps** We are continuing to work with the vendor to ensure the issue is fully understood and that safeguards are in place to prevent a recurrence..</p> ]]> Mon, 8 Dec 2025 11:43:42 +0000 https://datacake-status.com/incident/cmix32baj0gpxkm589otmjj36 https://datacake-status.com/incident/cmix32baj0gpxkm589otmjj36 Scheduled maintenance for the MQTT broker Type: Maintenance Duration: 15 minutes Affected Components: MQTT Server Nov 24, 10:00:00 GMT+0 - Identified - We are planning for a scheduled maintenance during that time. No downtime is expected, though connections may re-connect automatically. Nov 24, 10:00:01 GMT+0 - Identified - Maintenance is now in progress Nov 24, 10:15:00 GMT+0 - Completed - Maintenance has completed successfully Type: Maintenance

Duration: 15 minutes

Affected Components:

<p><small>Nov <var data-var='date'> 24</var>, <var data-var='time'>10:00:00</var> GMT+0</small><br><strong>Identified</strong> - We are planning for a scheduled maintenance during that time. No downtime is expected, though connections may re-connect automatically..</p> <p><small>Nov <var data-var='date'> 24</var>, <var data-var='time'>10:00:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> <p><small>Nov <var data-var='date'> 24</var>, <var data-var='time'>10:15:00</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully.</p> ]]> Mon, 24 Nov 2025 10:00:00 +0000 https://datacake-status.com/maintenance/cmictc2qu080yjlj3lwo72cs0 https://datacake-status.com/maintenance/cmictc2qu080yjlj3lwo72cs0 Partial Outage - EU1 Region (The Things Industries / Datacake LNS) Type: Incident Duration: 18 hours and 15 minutes Affected Components: EU1 Oct 8, 13:22:45 GMT+0 - Monitoring - **The Things Industries (TTI)** has implemented a fix for the issue affecting the **EU1 region**. Both TTI and Datacake are now monitoring the situation to ensure full recovery of all affected LoRaWAN data flows. We will continue to observe system performance and provide further updates if necessary. Oct 9, 04:24:46 GMT+0 - Resolved - The incident affecting the EU1 region of The Things Industries (TTI) — and consequently the Datacake LoRaWAN Network Server (LNS) — has been resolved. TTI has confirmed that services in the EU1 region are fully operational again. LoRaWAN data ingestion and related functionality have returned to normal. We will continue to monitor system performance, but no further impact is expected. For reference, see TTI’s incident report: <https://status.thethings.industries/incidents/8bvwyj791rrm>. Oct 8, 10:10:00 GMT+0 - Investigating - We are currently observing a partial outage impacting the **EU1 region of The Things Industries (TTI)**. This affects both the **Datacake LoRaWAN Network Server (LNS)** and **customers who operate their own TTI instances in the EU1 region**. As a result, some LoRaWAN devices may experience delayed or missing uplinks until the issue is resolved. The incident originates from TTI’s infrastructure. For the latest updates, please refer to their official status page: <https://status.thethings.industries/incidents/8bvwyj791rrm>. We are monitoring the situation closely and will update this post as more information becomes available. Type: Incident

Duration: 18 hours and 15 minutes

Affected Components:

<p><small>Oct <var data-var='date'> 8</var>, <var data-var='time'>13:22:45</var> GMT+0</small><br><strong>Monitoring</strong> - **The Things Industries (TTI)** has implemented a fix for the issue affecting the **EU1 region**. Both TTI and Datacake are now monitoring the situation to ensure full recovery of all affected LoRaWAN data flows. We will continue to observe system performance and provide further updates if necessary..</p> <p><small>Oct <var data-var='date'> 9</var>, <var data-var='time'>04:24:46</var> GMT+0</small><br><strong>Resolved</strong> - The incident affecting the EU1 region of The Things Industries (TTI) — and consequently the Datacake LoRaWAN Network Server (LNS) — has been resolved. TTI has confirmed that services in the EU1 region are fully operational again. LoRaWAN data ingestion and related functionality have returned to normal. We will continue to monitor system performance, but no further impact is expected. For reference, see TTI’s incident report: <https://status.thethings.industries/incidents/8bvwyj791rrm>..</p> <p><small>Oct <var data-var='date'> 8</var>, <var data-var='time'>10:10:00</var> GMT+0</small><br><strong>Investigating</strong> - We are currently observing a partial outage impacting the **EU1 region of The Things Industries (TTI)**. This affects both the **Datacake LoRaWAN Network Server (LNS)** and **customers who operate their own TTI instances in the EU1 region**. As a result, some LoRaWAN devices may experience delayed or missing uplinks until the issue is resolved. The incident originates from TTI’s infrastructure. For the latest updates, please refer to their official status page: <https://status.thethings.industries/incidents/8bvwyj791rrm>. We are monitoring the situation closely and will update this post as more information becomes available..</p> ]]> Wed, 8 Oct 2025 10:10:00 +0000 https://datacake-status.com/incident/cmghuulb4014l6966eo2kbvtu https://datacake-status.com/incident/cmghuulb4014l6966eo2kbvtu Incomplete Data Display for Some Devices Type: Incident Duration: 3 days, 19 hours and 47 minutes Affected Components: Web Application, Reports Oct 6, 11:56:01 GMT+0 - Investigating - We are currently experiencing an issue where dashboards and other parts of the application may show incomplete historical data for certain devices in specific retention plans. Data from early to mid-September 2025 may not be visible in queries and visualizations for affected devices, though **all data remains safely stored and intact**. We have identified the root cause as a database indexing issue with our external database vendor and are working directly with their support team to resolve it. Oct 10, 07:43:16 GMT+0 - Resolved - The issue causing incomplete historical data display for certain devices has been resolved. All data is now fully accessible across dashboards and the application. **What Happened:** We identified an issue where historical data from early to mid-September 2025 was not appearing in queries and visualizations for specific devices in certain retention plans. The root cause was traced to a bug in our time-series database software related to Bloom filters on compressed data chunks. These filters were incorrectly marking data as non-existent, causing the database to skip chunks that actually contained the requested data. **Resolution:** Our database vendor Timescale identified and fixed a software bug in their compression filtering logic. The fix has been applied to our database instance, and all queries now return complete and accurate results. **Data Safety:** All data remained safely stored throughout this incident. No data was lost or corrupted - it was simply not being returned in certain filtered queries due to the database software bug. We appreciate Timescale's quick response in identifying and resolving this issue. We apologize for any confusion or inconvenience this may have caused and thank you for your patience. Type: Incident

Duration: 3 days, 19 hours and 47 minutes

Affected Components: ,

<p><small>Oct <var data-var='date'> 6</var>, <var data-var='time'>11:56:01</var> GMT+0</small><br><strong>Investigating</strong> - We are currently experiencing an issue where dashboards and other parts of the application may show incomplete historical data for certain devices in specific retention plans. Data from early to mid-September 2025 may not be visible in queries and visualizations for affected devices, though **all data remains safely stored and intact**. We have identified the root cause as a database indexing issue with our external database vendor and are working directly with their support team to resolve it..</p> <p><small>Oct <var data-var='date'> 10</var>, <var data-var='time'>07:43:16</var> GMT+0</small><br><strong>Resolved</strong> - The issue causing incomplete historical data display for certain devices has been resolved. All data is now fully accessible across dashboards and the application. **What Happened:** We identified an issue where historical data from early to mid-September 2025 was not appearing in queries and visualizations for specific devices in certain retention plans. The root cause was traced to a bug in our time-series database software related to Bloom filters on compressed data chunks. These filters were incorrectly marking data as non-existent, causing the database to skip chunks that actually contained the requested data. **Resolution:** Our database vendor Timescale identified and fixed a software bug in their compression filtering logic. The fix has been applied to our database instance, and all queries now return complete and accurate results. **Data Safety:** All data remained safely stored throughout this incident. No data was lost or corrupted - it was simply not being returned in certain filtered queries due to the database software bug. We appreciate Timescale's quick response in identifying and resolving this issue. We apologize for any confusion or inconvenience this may have caused and thank you for your patience..</p> ]]> Mon, 6 Oct 2025 11:56:01 +0000 https://datacake-status.com/incident/cmgf2rhf7002s10duqsmlslr1 https://datacake-status.com/incident/cmgf2rhf7002s10duqsmlslr1 API outage Type: Incident Duration: 11 minutes May 5, 00:59:00 GMT+0 - Investigating - We are currently investigating issues with the availability of our API. May 5, 01:09:38 GMT+0 - Resolved - We have identified and fixed the underlying issue, which was caused by a database having issues. We have replaced this database with another instance and the service is back online. To ensure such outages do not happen again, we will continue making our infrastructure more robust and fail-safe. Type: Incident

Duration: 11 minutes

<p><small>May <var data-var='date'> 5</var>, <var data-var='time'>00:59:00</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating issues with the availability of our API..</p> <p><small>May <var data-var='date'> 5</var>, <var data-var='time'>01:09:38</var> GMT+0</small><br><strong>Resolved</strong> - We have identified and fixed the underlying issue, which was caused by a database having issues. We have replaced this database with another instance and the service is back online. To ensure such outages do not happen again, we will continue making our infrastructure more robust and fail-safe..</p> ]]> Mon, 5 May 2025 00:59:00 +0000 https://datacake-status.com/incident/cmaadhczx01vjat0idxuf70nf https://datacake-status.com/incident/cmaadhczx01vjat0idxuf70nf Performance degradation Type: Incident Duration: 1 hour and 7 minutes Affected Components: Web Application Apr 14, 09:55:34 GMT+0 - Investigating - We are investigating sporadic failures when querying historic data. This can affect components such as device list, device history and dashboards. Apr 14, 11:02:52 GMT+0 - Resolved - All systems are operational again. The issue was traced to a database managed by an external service provider. We are continuing to work with them to identify the root cause. Type: Incident

Duration: 1 hour and 7 minutes

Affected Components:

<p><small>Apr <var data-var='date'> 14</var>, <var data-var='time'>09:55:34</var> GMT+0</small><br><strong>Investigating</strong> - We are investigating sporadic failures when querying historic data. This can affect components such as device list, device history and dashboards..</p> <p><small>Apr <var data-var='date'> 14</var>, <var data-var='time'>11:02:52</var> GMT+0</small><br><strong>Resolved</strong> - All systems are operational again. The issue was traced to a database managed by an external service provider. We are continuing to work with them to identify the root cause..</p> ]]> Mon, 14 Apr 2025 09:55:34 +0000 https://datacake-status.com/incident/cm9gweiil004gac1gmd33qzyx https://datacake-status.com/incident/cm9gweiil004gac1gmd33qzyx MQTT processing delays Type: Incident Duration: 6 hours and 42 minutes Affected Components: MQTT Server Apr 9, 05:24:20 GMT+0 - Investigating - Our MQTT broker is experiencing processing delays. No data has been lost, and the system should be caught up shortly. Apr 9, 12:06:42 GMT+0 - Resolved - Earlier this morning, we identified an incident where outgoing MQTT messages began queueing up, causing delays in message delivery. Our operations team immediately implemented countermeasures by adjusting infrastructure configurations, which successfully increased queue processing rates. The backlog has now been completely cleared, and after a thorough monitoring period, we can confirm the system is operating normally. All MQTT services have returned to optimal performance levels. We apologize for any inconvenience this may have caused. Our engineering team is actively working on improvements to our MQTT infrastructure to prevent similar incidents in the future. Type: Incident

Duration: 6 hours and 42 minutes

Affected Components:

<p><small>Apr <var data-var='date'> 9</var>, <var data-var='time'>05:24:20</var> GMT+0</small><br><strong>Investigating</strong> - Our MQTT broker is experiencing processing delays. No data has been lost, and the system should be caught up shortly..</p> <p><small>Apr <var data-var='date'> 9</var>, <var data-var='time'>12:06:42</var> GMT+0</small><br><strong>Resolved</strong> - Earlier this morning, we identified an incident where outgoing MQTT messages began queueing up, causing delays in message delivery. Our operations team immediately implemented countermeasures by adjusting infrastructure configurations, which successfully increased queue processing rates. The backlog has now been completely cleared, and after a thorough monitoring period, we can confirm the system is operating normally. All MQTT services have returned to optimal performance levels. We apologize for any inconvenience this may have caused. Our engineering team is actively working on improvements to our MQTT infrastructure to prevent similar incidents in the future..</p> ]]> Wed, 9 Apr 2025 05:24:20 +0000 https://datacake-status.com/incident/cm99hig0e008fpbnv1xpjmfdn https://datacake-status.com/incident/cm99hig0e008fpbnv1xpjmfdn Performance degradation Type: Incident Duration: 53 minutes Affected Components: Web Application, GraphQL API Oct 15, 08:24:41 GMT+0 - Investigating - We are investigating increased latencies with the API. Oct 15, 08:33:36 GMT+0 - Identified - We are currently experiencing an issue with one of our primary databases and are actively working to resolve it. Thank you for your patience. Oct 15, 08:52:13 GMT+0 - Monitoring - The issue with the database server has been resolved and services are being restored. We will continue to monitor the situation. Oct 15, 09:18:09 GMT+0 - Resolved - The issue has been resolved and all systems are operating normally. We are working with our managed database provider to identify the root cause. We apologize for any inconvenience caused. Type: Incident

Duration: 53 minutes

Affected Components: ,

<p><small>Oct <var data-var='date'> 15</var>, <var data-var='time'>08:24:41</var> GMT+0</small><br><strong>Investigating</strong> - We are investigating increased latencies with the API..</p> <p><small>Oct <var data-var='date'> 15</var>, <var data-var='time'>08:33:36</var> GMT+0</small><br><strong>Identified</strong> - We are currently experiencing an issue with one of our primary databases and are actively working to resolve it. Thank you for your patience..</p> <p><small>Oct <var data-var='date'> 15</var>, <var data-var='time'>08:52:13</var> GMT+0</small><br><strong>Monitoring</strong> - The issue with the database server has been resolved and services are being restored. We will continue to monitor the situation..</p> <p><small>Oct <var data-var='date'> 15</var>, <var data-var='time'>09:18:09</var> GMT+0</small><br><strong>Resolved</strong> - The issue has been resolved and all systems are operating normally. We are working with our managed database provider to identify the root cause. We apologize for any inconvenience caused..</p> ]]> Tue, 15 Oct 2024 08:24:41 +0000 https://datacake-status.com/incident/cm2a6ggi60014akfzeiv86bq5 https://datacake-status.com/incident/cm2a6ggi60014akfzeiv86bq5 Data processing delays Type: Incident Duration: 7 hours and 45 minutes Affected Components: Web Application, Reports, MQTT Server, Outgoing Webhooks Aug 30, 06:14:27 GMT+0 - Investigating - Our system is experiencing processing delays. No data has been lost, and the system should be caught up shortly. Aug 30, 06:36:15 GMT+0 - Identified - We have identified delays in data processing due to an overloaded queue and have already scaled up our processing power for faster performance. Our team is closely monitoring the queue to ensure it is cleared efficiently. Aug 30, 13:59:49 GMT+0 - Resolved - All queues have now finished processing and are operating in real-time. Thank you for your patience. Aug 30, 09:03:05 GMT+0 - Identified - The main affected queue has been successfully processed. However, due to the high volume of data, some secondary queues, such as MQTT and database insertion, are experiencing high load. Data delays are still occurring, and we are actively managing this to minimize impact. Aug 30, 11:59:49 GMT+0 - Monitoring - We are pleased to report that almost all important queues have caught up and are now processing data in real-time. The MQTT publish queue is still slightly behind, and we are working to resolve this. Type: Incident

Duration: 7 hours and 45 minutes

Affected Components: , , ,

<p><small>Aug <var data-var='date'> 30</var>, <var data-var='time'>06:14:27</var> GMT+0</small><br><strong>Investigating</strong> - Our system is experiencing processing delays. No data has been lost, and the system should be caught up shortly..</p> <p><small>Aug <var data-var='date'> 30</var>, <var data-var='time'>06:36:15</var> GMT+0</small><br><strong>Identified</strong> - We have identified delays in data processing due to an overloaded queue and have already scaled up our processing power for faster performance. Our team is closely monitoring the queue to ensure it is cleared efficiently..</p> <p><small>Aug <var data-var='date'> 30</var>, <var data-var='time'>13:59:49</var> GMT+0</small><br><strong>Resolved</strong> - All queues have now finished processing and are operating in real-time. Thank you for your patience..</p> <p><small>Aug <var data-var='date'> 30</var>, <var data-var='time'>09:03:05</var> GMT+0</small><br><strong>Identified</strong> - The main affected queue has been successfully processed. However, due to the high volume of data, some secondary queues, such as MQTT and database insertion, are experiencing high load. Data delays are still occurring, and we are actively managing this to minimize impact..</p> <p><small>Aug <var data-var='date'> 30</var>, <var data-var='time'>11:59:49</var> GMT+0</small><br><strong>Monitoring</strong> - We are pleased to report that almost all important queues have caught up and are now processing data in real-time. The MQTT publish queue is still slightly behind, and we are working to resolve this..</p> ]]> Fri, 30 Aug 2024 06:14:27 +0000 https://datacake-status.com/incident/cm0gbjsa100im578f9i0yywmd https://datacake-status.com/incident/cm0gbjsa100im578f9i0yywmd Data processing delays Type: Incident Duration: 22 hours and 15 minutes Affected Components: Web Application, Reports, MQTT Server, Outgoing Webhooks May 28, 08:47:00 GMT+0 - Resolved - Dear customers, We are informing you that some of our services on the Datacake IoT platform have recently experienced delays. Please be assured that there was no downtime and no data was lost. The protection and integrity of your data remain our top priority. The issue affected a subset of devices, causing delays of up to two hours for certain tasks. While some devices continued to operate at full performance, others were impacted by the delays. The cause was increased wait times for specific task processing, including Payload Decoder and API requests. Although the platform's core API remained stable, queues became overwhelmed, leading to longer processing times. To ensure the platform's stability, we have temporarily disabled some non-essential features and are monitoring the situation closely. We are taking measures to optimize internal processes and will keep you informed of any changes. Thank you for your understanding and patience during this time. Best regards, The Datacake Team May 27, 10:31:41 GMT+0 - Investigating - Our system is experiencing processing delays. No data has been lost, and the system should be caught up shortly. May 27, 12:16:09 GMT+0 - Identified - We are continuing to work on a fix for this incident. May 27, 14:06:07 GMT+0 - Investigating - We are currently investigating this incident. May 27, 17:17:31 GMT+0 - Identified - We are still working on a mitigation of the delayed processing of device data. May 27, 21:47:27 GMT+0 - Monitoring - Queue processing is largely complete, with only outgoing MQTT and webhooks remaining to be finished. Type: Incident

Duration: 22 hours and 15 minutes

Affected Components: , , ,

<p><small>May <var data-var='date'> 28</var>, <var data-var='time'>08:47:00</var> GMT+0</small><br><strong>Resolved</strong> - Dear customers, We are informing you that some of our services on the Datacake IoT platform have recently experienced delays. Please be assured that there was no downtime and no data was lost. The protection and integrity of your data remain our top priority. The issue affected a subset of devices, causing delays of up to two hours for certain tasks. While some devices continued to operate at full performance, others were impacted by the delays. The cause was increased wait times for specific task processing, including Payload Decoder and API requests. Although the platform's core API remained stable, queues became overwhelmed, leading to longer processing times. To ensure the platform's stability, we have temporarily disabled some non-essential features and are monitoring the situation closely. We are taking measures to optimize internal processes and will keep you informed of any changes. Thank you for your understanding and patience during this time. Best regards, The Datacake Team.</p> <p><small>May <var data-var='date'> 27</var>, <var data-var='time'>10:31:41</var> GMT+0</small><br><strong>Investigating</strong> - Our system is experiencing processing delays. No data has been lost, and the system should be caught up shortly..</p> <p><small>May <var data-var='date'> 27</var>, <var data-var='time'>12:16:09</var> GMT+0</small><br><strong>Identified</strong> - We are continuing to work on a fix for this incident..</p> <p><small>May <var data-var='date'> 27</var>, <var data-var='time'>14:06:07</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident..</p> <p><small>May <var data-var='date'> 27</var>, <var data-var='time'>17:17:31</var> GMT+0</small><br><strong>Identified</strong> - We are still working on a mitigation of the delayed processing of device data..</p> <p><small>May <var data-var='date'> 27</var>, <var data-var='time'>21:47:27</var> GMT+0</small><br><strong>Monitoring</strong> - Queue processing is largely complete, with only outgoing MQTT and webhooks remaining to be finished..</p> ]]> Mon, 27 May 2024 10:31:41 +0000 https://datacake-status.com/incident/clwotwnlg31492b5n6854y3rg5 https://datacake-status.com/incident/clwotwnlg31492b5n6854y3rg5 Data processing delays Type: Incident Duration: 1 day, 4 hours and 21 minutes Affected Components: Web Application, Reports, MQTT Server, Outgoing Webhooks May 23, 06:44:19 GMT+0 - Investigating - Our system is experiencing processing delays. No data has been lost, and the system should be caught up shortly. May 24, 11:05:23 GMT+0 - Resolved - All queues have been processed. We are still investigating the root cause for the unusual buildup and have already taken measures in the form of automated scaling to prevent such situations from happening again in the future. May 23, 13:48:21 GMT+0 - Monitoring - Most queues have been successfully processed. However, MQTT and outgoing Webhooks are still experiencing delays. Our teams are working to resolve these remaining issues as quickly as possible. May 23, 08:12:11 GMT+0 - Identified - We have identified the issue to be related to certain background workers not functioning as expected. We're actively working on implementing improvements to optimize their performance and resolve the issue. Type: Incident

Duration: 1 day, 4 hours and 21 minutes

Affected Components: , , ,

<p><small>May <var data-var='date'> 23</var>, <var data-var='time'>06:44:19</var> GMT+0</small><br><strong>Investigating</strong> - Our system is experiencing processing delays. No data has been lost, and the system should be caught up shortly..</p> <p><small>May <var data-var='date'> 24</var>, <var data-var='time'>11:05:23</var> GMT+0</small><br><strong>Resolved</strong> - All queues have been processed. We are still investigating the root cause for the unusual buildup and have already taken measures in the form of automated scaling to prevent such situations from happening again in the future..</p> <p><small>May <var data-var='date'> 23</var>, <var data-var='time'>13:48:21</var> GMT+0</small><br><strong>Monitoring</strong> - Most queues have been successfully processed. However, MQTT and outgoing Webhooks are still experiencing delays. Our teams are working to resolve these remaining issues as quickly as possible..</p> <p><small>May <var data-var='date'> 23</var>, <var data-var='time'>08:12:11</var> GMT+0</small><br><strong>Identified</strong> - We have identified the issue to be related to certain background workers not functioning as expected. We're actively working on implementing improvements to optimize their performance and resolve the issue..</p> ]]> Thu, 23 May 2024 06:44:19 +0000 https://datacake-status.com/incident/clwiw0vak83556baoepc0b382b https://datacake-status.com/incident/clwiw0vak83556baoepc0b382b Performance degradation Type: Incident Duration: 23 hours and 39 minutes Affected Components: Web Application Apr 29, 10:51:04 GMT+0 - Investigating - We are investigating increased latencies with the API. Apr 30, 10:30:26 GMT+0 - Resolved - Our team has successfully implemented optimizations and fine-tuned our infrastructure, resulting in improved performance and reduced latencies. We continue to closely monitor the situation to ensure stability. If you encounter any further issues or have questions, please don't hesitate to reach out to our support team for assistance. Apr 29, 11:46:37 GMT+0 - Monitoring - The situation has stabilized. We are still investigating a small percentage of requests with unusual latencies and / or timeouts. Type: Incident

Duration: 23 hours and 39 minutes

Affected Components:

<p><small>Apr <var data-var='date'> 29</var>, <var data-var='time'>10:51:04</var> GMT+0</small><br><strong>Investigating</strong> - We are investigating increased latencies with the API..</p> <p><small>Apr <var data-var='date'> 30</var>, <var data-var='time'>10:30:26</var> GMT+0</small><br><strong>Resolved</strong> - Our team has successfully implemented optimizations and fine-tuned our infrastructure, resulting in improved performance and reduced latencies. We continue to closely monitor the situation to ensure stability. If you encounter any further issues or have questions, please don't hesitate to reach out to our support team for assistance..</p> <p><small>Apr <var data-var='date'> 29</var>, <var data-var='time'>11:46:37</var> GMT+0</small><br><strong>Monitoring</strong> - The situation has stabilized. We are still investigating a small percentage of requests with unusual latencies and / or timeouts..</p> ]]> Mon, 29 Apr 2024 10:51:04 +0000 https://datacake-status.com/incident/clvku9r0f135665bnlmm1w6weol https://datacake-status.com/incident/clvku9r0f135665bnlmm1w6weol Data processing delays Type: Incident Duration: 10 hours and 24 minutes Affected Components: Web Application Mar 14, 22:02:43 GMT+0 - Resolved - All queues have now been cleared with no data loss. We apologize for these delays and appreciate your understanding. In dedication to clarity and process improvement, a comprehensive follow-up about the incident, explaining its causes and the measures taken to prevent recurrence, will be provided shortly. Mar 14, 11:44:27 GMT+0 - Investigating - We are currently investigating this incident. Mar 14, 15:16:30 GMT+0 - Monitoring - The queries in question have now stabilized, resulting in a consistent reduction of the queue. We'll continue to provide updates on the situation in this space. Mar 14, 11:38:58 GMT+0 - Investigating - Our system is experiencing processing delays with incoming data. We are currently investigating the cause of this delay. Mar 14, 12:03:28 GMT+0 - Identified - We've identified the root cause of the recent issue as an unexpected high load on one of our databases. We've enhanced the resources allocated to this database and are currently monitoring its performance closely. Please note that due to a pending backlog, temporary data gaps may be evident in the chart visualizations. However, we're working to ensure data integrity as soon as possible. Mar 14, 12:48:29 GMT+0 - Identified - We are still in the process of examining an extended measurement queue. Please note that data transmission still encounters delays. Mar 15, 14:24:15 GMT+0 - Resolved - The full post mortem can be found on our Engineering Blog: <https://engineering.datacake.co/post-mortem-service-degradation-on-march-14-2024/> Mar 14, 13:24:54 GMT+0 - Identified - We've identified the initial root cause of the delays as a few long-running migration queries. We are actively monitoring the situation and adjusting resources as necessary to expedite the process. Type: Incident

Duration: 10 hours and 24 minutes

Affected Components:

<p><small>Mar <var data-var='date'> 14</var>, <var data-var='time'>22:02:43</var> GMT+0</small><br><strong>Resolved</strong> - All queues have now been cleared with no data loss. We apologize for these delays and appreciate your understanding. In dedication to clarity and process improvement, a comprehensive follow-up about the incident, explaining its causes and the measures taken to prevent recurrence, will be provided shortly..</p> <p><small>Mar <var data-var='date'> 14</var>, <var data-var='time'>11:44:27</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident..</p> <p><small>Mar <var data-var='date'> 14</var>, <var data-var='time'>15:16:30</var> GMT+0</small><br><strong>Monitoring</strong> - The queries in question have now stabilized, resulting in a consistent reduction of the queue. We'll continue to provide updates on the situation in this space..</p> <p><small>Mar <var data-var='date'> 14</var>, <var data-var='time'>11:38:58</var> GMT+0</small><br><strong>Investigating</strong> - Our system is experiencing processing delays with incoming data. We are currently investigating the cause of this delay..</p> <p><small>Mar <var data-var='date'> 14</var>, <var data-var='time'>12:03:28</var> GMT+0</small><br><strong>Identified</strong> - We've identified the root cause of the recent issue as an unexpected high load on one of our databases. We've enhanced the resources allocated to this database and are currently monitoring its performance closely. Please note that due to a pending backlog, temporary data gaps may be evident in the chart visualizations. However, we're working to ensure data integrity as soon as possible..</p> <p><small>Mar <var data-var='date'> 14</var>, <var data-var='time'>12:48:29</var> GMT+0</small><br><strong>Identified</strong> - We are still in the process of examining an extended measurement queue. Please note that data transmission still encounters delays..</p> <p><small>Mar <var data-var='date'> 15</var>, <var data-var='time'>14:24:15</var> GMT+0</small><br><strong>Resolved</strong> - The full post mortem can be found on our Engineering Blog: <https://engineering.datacake.co/post-mortem-service-degradation-on-march-14-2024/>.</p> <p><small>Mar <var data-var='date'> 14</var>, <var data-var='time'>13:24:54</var> GMT+0</small><br><strong>Identified</strong> - We've identified the initial root cause of the delays as a few long-running migration queries. We are actively monitoring the situation and adjusting resources as necessary to expedite the process..</p> ]]> Thu, 14 Mar 2024 11:38:58 +0000 https://datacake-status.com/incident/cltr5q6b826301bbocvusf7emf https://datacake-status.com/incident/cltr5q6b826301bbocvusf7emf Delay in outgoing webhooks Type: Incident Duration: 3 hours and 2 minutes Affected Components: Outgoing Webhooks Jan 24, 08:40:00 GMT+0 - Investigating - We're addressing increased load on our systems responsible for processing outgoing webhook calls, which may result in some delays. We apologize for any inconvenience and are working to resolve this issue promptly. Jan 24, 11:22:52 GMT+0 - Identified - We have identified the root cause of the issue and are actively developing a solution. We appreciate your patience as we work to resolve this matter. Jan 24, 11:42:27 GMT+0 - Resolved - The issue has been mitigated by pinpointing webhooks that were timing out and contributing to the delays. We've temporarily increased the number of workers to manage the workload more effectively. Meanwhile, we are formulating a sustainable fix to prevent recurrence of this issue. Thank you for your understanding as we enhance our systems. Type: Incident

Duration: 3 hours and 2 minutes

Affected Components:

<p><small>Jan <var data-var='date'> 24</var>, <var data-var='time'>08:40:00</var> GMT+0</small><br><strong>Investigating</strong> - We're addressing increased load on our systems responsible for processing outgoing webhook calls, which may result in some delays. We apologize for any inconvenience and are working to resolve this issue promptly..</p> <p><small>Jan <var data-var='date'> 24</var>, <var data-var='time'>11:22:52</var> GMT+0</small><br><strong>Identified</strong> - We have identified the root cause of the issue and are actively developing a solution. We appreciate your patience as we work to resolve this matter..</p> <p><small>Jan <var data-var='date'> 24</var>, <var data-var='time'>11:42:27</var> GMT+0</small><br><strong>Resolved</strong> - The issue has been mitigated by pinpointing webhooks that were timing out and contributing to the delays. We've temporarily increased the number of workers to manage the workload more effectively. Meanwhile, we are formulating a sustainable fix to prevent recurrence of this issue. Thank you for your understanding as we enhance our systems..</p> ]]> Wed, 24 Jan 2024 08:40:00 +0000 https://datacake-status.com/incident/clrrnr3zs16433byn533l36xuj https://datacake-status.com/incident/clrrnr3zs16433byn533l36xuj Dashboard maintenance Type: Incident Duration: 3 hours and 11 minutes Affected Components: Web Application Jan 9, 09:37:32 GMT+0 - Monitoring - We have resolved the issues that were impacting a number of dashboards. The majority of dashboards are now displaying the most recent data, however, we are continuing to monitor them diligently to ensure there are no remaining bugs or issues with stale cache. Jan 9, 08:46:05 GMT+0 - Investigating - We acknowledge that certain dashboards are not displaying the latest data, and our team is actively working on a solution. Please note that this issue solely impacts the visualization of data on the dashboards; rest assured, all data recording remains accurate and uninterrupted. We will keep you informed of our progress on this matter. Jan 9, 11:56:54 GMT+0 - Resolved - This issue is now considered resolved as we have no further known problems to address. Should you continue to experience any issues with your dashboards, please do not hesitate to contact our support team for assistance. Type: Incident

Duration: 3 hours and 11 minutes

Affected Components:

<p><small>Jan <var data-var='date'> 9</var>, <var data-var='time'>09:37:32</var> GMT+0</small><br><strong>Monitoring</strong> - We have resolved the issues that were impacting a number of dashboards. The majority of dashboards are now displaying the most recent data, however, we are continuing to monitor them diligently to ensure there are no remaining bugs or issues with stale cache..</p> <p><small>Jan <var data-var='date'> 9</var>, <var data-var='time'>08:46:05</var> GMT+0</small><br><strong>Investigating</strong> - We acknowledge that certain dashboards are not displaying the latest data, and our team is actively working on a solution. Please note that this issue solely impacts the visualization of data on the dashboards; rest assured, all data recording remains accurate and uninterrupted. We will keep you informed of our progress on this matter..</p> <p><small>Jan <var data-var='date'> 9</var>, <var data-var='time'>11:56:54</var> GMT+0</small><br><strong>Resolved</strong> - This issue is now considered resolved as we have no further known problems to address. Should you continue to experience any issues with your dashboards, please do not hesitate to contact our support team for assistance..</p> ]]> Tue, 9 Jan 2024 08:46:05 +0000 https://datacake-status.com/incident/clr63xga516099bkonoin5qbps https://datacake-status.com/incident/clr63xga516099bkonoin5qbps Performance degradation Type: Incident Duration: 57 minutes Affected Components: Web Application Dec 18, 14:17:49 GMT+0 - Resolved - This incident has been resolved. Dec 18, 13:20:38 GMT+0 - Investigating - We are investigating increased latencies with the API. Dec 18, 13:40:05 GMT+0 - Monitoring - The monitored system are healthy again. There was a brief period (< 5 minutes) of increased latencies and occasional timeouts when trying to access the API. We will continue monitoring for any disruptions. Type: Incident

Duration: 57 minutes

Affected Components:

<p><small>Dec <var data-var='date'> 18</var>, <var data-var='time'>14:17:49</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> <p><small>Dec <var data-var='date'> 18</var>, <var data-var='time'>13:20:38</var> GMT+0</small><br><strong>Investigating</strong> - We are investigating increased latencies with the API..</p> <p><small>Dec <var data-var='date'> 18</var>, <var data-var='time'>13:40:05</var> GMT+0</small><br><strong>Monitoring</strong> - The monitored system are healthy again. There was a brief period (< 5 minutes) of increased latencies and occasional timeouts when trying to access the API. We will continue monitoring for any disruptions..</p> ]]> Mon, 18 Dec 2023 13:20:38 +0000 https://datacake-status.com/incident/clqay1sb215459biooj93dhxzk https://datacake-status.com/incident/clqay1sb215459biooj93dhxzk API outage Type: Incident Duration: 15 days, 13 hours and 15 minutes Dec 6, 00:07:17 GMT+0 - Investigating - We are currently investigating issues with the availability of our API. Dec 6, 06:20:31 GMT+0 - Resolved - We have identified and fixed the underlying issue, which was caused by a database having issues. We have replaced this database with another instance and the service is back online. To ensure such outages do not happen again, we will continue making our infrastructure more robust and fail-safe. Dec 6, 05:59:47 GMT+0 - Identified - We have identified the issue and are working on a solution. Type: Incident

Duration: 15 days, 13 hours and 15 minutes

<p><small>Dec <var data-var='date'> 6</var>, <var data-var='time'>00:07:17</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating issues with the availability of our API..</p> <p><small>Dec <var data-var='date'> 6</var>, <var data-var='time'>06:20:31</var> GMT+0</small><br><strong>Resolved</strong> - We have identified and fixed the underlying issue, which was caused by a database having issues. We have replaced this database with another instance and the service is back online. To ensure such outages do not happen again, we will continue making our infrastructure more robust and fail-safe..</p> <p><small>Dec <var data-var='date'> 6</var>, <var data-var='time'>05:59:47</var> GMT+0</small><br><strong>Identified</strong> - We have identified the issue and are working on a solution..</p> ]]> Wed, 6 Dec 2023 00:07:17 +0000 https://datacake-status.com/incident/clpt0famn33186hkoq8gho135t https://datacake-status.com/incident/clpt0famn33186hkoq8gho135t Delay in delivery of webhooks Type: Incident Duration: 14 hours and 37 minutes Affected Components: Web Application, MQTT Server, White Label, Reports Nov 28, 12:52:24 GMT+0 - Identified - We have identified the issue and are working on the implementation of a fix. Until then, outgoing webhooks might still be delivered late. Nov 29, 00:01:00 GMT+0 - Resolved - The issue has been resolved and all webhooks have been delivered. Sorry for any inconvenience this might have caused. Nov 28, 09:24:00 GMT+0 - Investigating - We are currently investigating an increased delay in the processing of webhooks, that can also cause other parts of the system to slow down. Type: Incident

Duration: 14 hours and 37 minutes

Affected Components: , , ,

<p><small>Nov <var data-var='date'> 28</var>, <var data-var='time'>12:52:24</var> GMT+0</small><br><strong>Identified</strong> - We have identified the issue and are working on the implementation of a fix. Until then, outgoing webhooks might still be delivered late..</p> <p><small>Nov <var data-var='date'> 29</var>, <var data-var='time'>00:01:00</var> GMT+0</small><br><strong>Resolved</strong> - The issue has been resolved and all webhooks have been delivered. Sorry for any inconvenience this might have caused..</p> <p><small>Nov <var data-var='date'> 28</var>, <var data-var='time'>09:24:00</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating an increased delay in the processing of webhooks, that can also cause other parts of the system to slow down..</p> ]]> Tue, 28 Nov 2023 09:24:00 +0000 https://datacake-status.com/incident/clpi94otv55920con4mery6bol https://datacake-status.com/incident/clpi94otv55920con4mery6bol