Status: Resolved
New call and chat processing paused overnight on 26 August 2026, after a routine infrastructure change unintentionally disabled the trigger that starts processing once a call is uploaded. We identified and fixed this the next morning, then worked through the resulting backlog, which was fully cleared by the morning of 28 August. We later found the same fault had also affected a separate part of our infrastructure serving a subset of other customers, and corrected that within a few days. No call data was lost at any point; the impact was to processing time, not to the calls themselves.
New call uploads stopped being picked up for processing from the evening of 26 August until the fault was identified the next morning. Uploads themselves continued to be received and safely queued throughout, they were delayed rather than lost. Once the fault was fixed, a large backlog of delayed calls needed to be worked through, which meant scoring and alerts were also delayed for affected calls until the backlog cleared. Everything queued was fully reprocessed, with no data lost or duplicated.
Separately, we later identified that the same underlying fault had also affected a different part of our infrastructure, used by a subset of other customers, and corrected it within a few days of the main incident being resolved.
26 August, evening: A routine infrastructure change, made as part of planned migration work, unintentionally disabled the trigger that starts call processing after upload. No error was raised at the time.
27 August, 09:35: The issue was reported and the incident declared, after new conversations stopped appearing in the processing pipeline.
27 August, morning: The root cause was confirmed and the trigger was corrected; new uploads resumed processing.
27 August, later morning: The scale of the backlog became clear, and we began proactively notifying affected customers while backlog recovery got under way.
27–28 August: The backlog of delayed calls was worked through in stages, alongside the resulting delay to scores and alerts.
28 August, 08:26: All processing confirmed back to normal and the incident resolved.
Early September: The same fault was found to have affected a separate part of our infrastructure and was corrected.
New calls reach our platform through an automated pipeline that starts as soon as a file is uploaded. As part of planned infrastructure work, a routine change to how that trigger is configured unintentionally replaced the existing one, without raising any error at the time of the change. As a result, uploads kept arriving but stopped being picked up for processing.
This wasn't caught immediately for two reasons: our regular automated checks test a different part of the upload path to the one that broke, and our existing monitoring for this type of failure didn't flag it either. Both are being addressed (see below).
Once identified, the trigger was corrected the same morning and new uploads resumed processing immediately. Clearing the backlog of delayed calls took longer, since it involved working through a very large volume of queued conversations in stages, with checks along the way to confirm nothing was being duplicated. The backlog was fully cleared, and everything confirmed back to normal, by the following morning.
We've replaced the upload trigger with a configuration that can't be silently overwritten by future infrastructure changes. We're also making backlog recovery safer and faster for large volumes, so clearing a backlog of this size doesn't require repeated manual intervention; broadening our automated checks to cover the part of the upload path that broke here; and adding direct alerting for this specific failure mode, so a stall in processing is caught automatically rather than depending on a customer report.