Datacake - Data processing delays – Incident details

All systems operational

Data processing delays

Resolved
Degraded performance
Started 7 months agoLasted 1 day

Affected

Web Application

Degraded performance from 6:44 AM to 1:48 PM, Operational from 1:48 PM to 11:05 AM

MQTT Server

Degraded performance from 6:44 AM to 11:05 AM

Reports

Degraded performance from 6:44 AM to 1:48 PM, Operational from 1:48 PM to 11:05 AM

Outgoing Webhooks

Degraded performance from 6:44 AM to 11:05 AM

Updates
  • Resolved
    Resolved

    All queues have been processed. We are still investigating the root cause for the unusual buildup and have already taken measures in the form of automated scaling to prevent such situations from happening again in the future.

  • Monitoring
    Monitoring

    Most queues have been successfully processed. However, MQTT and outgoing Webhooks are still experiencing delays. Our teams are working to resolve these remaining issues as quickly as possible.

  • Identified
    Identified

    We have identified the issue to be related to certain background workers not functioning as expected. We're actively working on implementing improvements to optimize their performance and resolve the issue.

  • Investigating
    Investigating

    Our system is experiencing processing delays. No data has been lost, and the system should be caught up shortly.