Datacake - Data processing delays – Incident details

Data processing delays

Resolved
Degraded performance
Started 1 day agoLasted about 5 hours

Affected

GraphQL API

Degraded performance from 9:21 AM to 1:51 PM

MQTT Server

Degraded performance from 9:21 AM to 1:51 PM

Outgoing Webhooks

Degraded performance from 9:21 AM to 1:51 PM

Updates
  • Resolved
    UTC
    Resolved

    The processing queue has now fully cleared, and all system metrics look healthy.

    No data was lost during the incident. We apologize for the inconvenience and will publish a detailed technical postmortem soon.

  • Monitoring
    UTC
    Monitoring

    We have identified and resolved the root cause of the delays. An internal metrics exporter was placing additional load on the processing queue.

    We have implemented optimizations and are already seeing a significant performance improvement. The remaining backlog is still being processed, and we will provide another update once the queue has fully cleared.

    No data has been lost.

  • Identified
    UTC
    Identified

    We have identified the cause of the delays and implemented corrective measures. The fix has improved processing, but the existing queue backlog is still being cleared. Delays in data ingestion therefore continue at this time.

  • Investigating
    UTC
    Investigating

    Our system is experiencing processing delays. No data has been lost, and the system should be caught up shortly.