Datacake - Data processing delays – Incident details

All systems operational

Data processing delays

Resolved
Degraded performance
Started almost 2 years agoLasted about 8 hours

Affected

Web Application

Degraded performance from 6:14 AM to 11:59 AM, Operational from 11:59 AM to 1:59 PM

MQTT Server

Degraded performance from 6:14 AM to 1:59 PM

Reports

Degraded performance from 6:14 AM to 11:59 AM, Operational from 11:59 AM to 1:59 PM

Outgoing Webhooks

Operational from 6:14 AM to 9:03 AM, Degraded performance from 9:03 AM to 11:59 AM, Operational from 11:59 AM to 1:59 PM

Updates
  • Resolved
    UTC
    Resolved

    All queues have now finished processing and are operating in real-time. Thank you for your patience.

  • Monitoring
    UTC
    Monitoring

    We are pleased to report that almost all important queues have caught up and are now processing data in real-time. The MQTT publish queue is still slightly behind, and we are working to resolve this.

  • Update
    UTC
    Update

    The main affected queue has been successfully processed. However, due to the high volume of data, some secondary queues, such as MQTT and database insertion, are experiencing high load. Data delays are still occurring, and we are actively managing this to minimize impact.

  • Identified
    UTC
    Identified

    We have identified delays in data processing due to an overloaded queue and have already scaled up our processing power for faster performance. Our team is closely monitoring the queue to ensure it is cleared efficiently.

  • Investigating
    UTC
    Investigating

    Our system is experiencing processing delays. No data has been lost, and the system should be caught up shortly.