Battleground Live - Send Pipeline Latency – Incident details

All systems operational

Send Pipeline Latency

Resolved
Degraded performance
Started about 2 months agoLasted about 2 hours

Affected

Third-Party Providers

Operational from 8:11 PM to 8:11 PM, Degraded performance from 8:11 PM to 10:09 PM

Courier (Touchbase SMS and Push)

Operational from 8:11 PM to 8:11 PM, Degraded performance from 8:11 PM to 10:09 PM

Updates
  • Resolved
    UTC
    Resolved

    Send pipeline has resolved and enqueued messages are now sending properly.

  • Monitoring
    UTC
    Monitoring

    Messages are being processed while our pipeline continues to process enqueued messages.

  • Update
    UTC
    Update

    The team has identified that the most recent deployment was not the cause of the regression. The team is still working on identifying the root cause. Messages will not be dropped, and they will be processed once the fix is out.

  • Update
    UTC
    Update

    The rollback deploy will land in about 30 minutes.

  • Identified
    UTC
    Identified

    The team has identified a regression causing send requests to be stuck in a queued state. We have initiated a rollback to quickly resolve the issue. More updates to follow