Status: Resolved
New call and chat data stopped flowing into the platform for a period between 28 and 30 July 2026, meaning recently completed conversations weren't appearing for review, scoring or alerting during that window. This is fully resolved: the underlying fault has been fixed, every delayed conversation was reprocessed, and further safeguards are being put in place. No conversation data was lost or duplicated at any point, only its arrival was delayed.
During this period, new calls and chats were not being ingested into the platform, so recently completed conversations weren't available for review, scoring or alerting until ingestion resumed. Anything already ingested before the incident was unaffected. Once the fault was resolved, every delayed conversation was reprocessed in full; our ingestion process is designed to avoid duplicates, so nothing was lost or counted twice.
28 July, evening: The underlying fault began, though it wasn't yet noticed.
29 July, 06:19: The pause in ingestion was identified internally and the incident was declared.
29 July, morning: Investigation ruled out a database outage or a recent change as the cause; the database itself remained healthy and reachable throughout.
29 July, 08:41: The affected service component was restarted, restoring connectivity, and reprocessing of delayed conversations began.
29–30 July: Delayed conversations were reprocessed workspace by workspace.
30 July, 11:39: All delayed conversations confirmed reprocessed.
31 July: Incident formally closed after confirming no further reports.
New calls and chats reach our platform through an integration layer that manages the connection between external sources and our processing pipeline. This layer relies on a small pool of database connections to track which conversations it has already picked up. A design limitation meant this tracking check read a growing internal record in full on every run, and once call volume crossed a threshold, this exhausted the available database connections. Once exhausted, the integration layer could not complete its own internal operations and stopped processing entirely, rather than slowing down gradually. This wasn't caused by a change we made or by any fault in the underlying database, which stayed healthy and reachable throughout; it was a design limit being reached as volume grew.
We restarted the affected service to release the exhausted connections, then reprocessed the delayed conversations for every affected workspace, confirming directly with the most affected workspace exactly how much needed to catch up rather than assuming the backlog had cleared.
We've redesigned the tracking mechanism that caused the growing read load, increased database connection capacity, and upgraded the underlying database. We're also moving this integration layer onto its own dedicated database to remove shared-infrastructure risk, and adding direct monitoring for this specific failure mode, since it doesn't surface as a typical error and previously depended on being noticed rather than being alerted on. Further safeguards have also been added to prevent a slow-running check from hanging, along with a nightly reconciliation check to catch anything missed.
