Resolved -
This incident has been resolved.
Feb 5, 07:12 UTC
Update -
We are pleased to confirm that Voyc's call processing is fully operational this morning!
Thanks to the dedication of the team, we managed to process all backlogged calls before 7 a.m. UTC today 3 Feb, meaning all calls from Friday 31 Jan are now available on the platform. The team is continuing to monitor things closely.
Yesterday, the team identified and fixed core functionality outages as a result of the transcriber changes related to the scorecard, report, alert, and call summary functionality. The transcriber processing capacity was tripled to process the backlog of uploaded and unprocessed calls, allowing us to process over 190 000 conversations overnight.
Thank you once again for your patience and understanding throughout this process.
Feb 3, 08:59 UTC
Update -
Call processing continues to be stable and we have now fully processed all calls from Friday 31 January. We will send out a detailed update around 09:00 UTC / 11:00 GMT+2
Feb 3, 06:38 UTC
Update -
As of 23:30 UTC, things have been stable with call processing and transcription running at the highest possible rates. We're continuing to monitor the rate and stability of processing and will provide a detailed update, including and ETC for backlog processing, a little later in the morning.
Thank you for your patience while the team pulls out all the stops to get us back up and running smoothly.
Feb 3, 04:58 UTC
Monitoring -
Calls have started processing and flowing onto the platform with all subprocesses appearing to be working as expected
We will send out a detailed update during the course of the morning tomorrow (Monday 3 February)
Feb 2, 21:58 UTC
Update -
All processes and subprocesses are now working as expected after thorough testing.
We are currently restarting the transcribers to begin processing the backlog.
Feb 2, 21:41 UTC
Update -
Alerts and Scoring are now working as expected after thorough testing.
The team is currently finalising work on remaining subprocesses such as speaker identification
Feb 2, 18:02 UTC
Update -
Issue Summary
We have identified an issue preventing call transcriptions from being written to the database. While calls are still being received, their transcriptions are currently delayed.
Actions we are taking and Next Steps
Our team has been working around the clock to resolve the issue, and we’ve made significant progress:
- Transcription is now fully operational
- We are systematically restoring all remaining sub-processes, including alerts, scoring, and speaker identification.
We understand how important it is for you to have access to your data, and we want to assure you that every effort is being made to bring all services back online as quickly and smoothly as possible.
- Conversations from Friday will likely begin appearing later today, with more being processed continuously.
- New calls can continue to be uploaded to Voyc, and while they won’t be available immediately, they will be processed as soon as our systems are fully restored.
Our team is actively monitoring every step of the recovery to ensure stability, accuracy, and reliability.
Thank you for your patience - we’re committed to getting everything back to full speed, and we won’t stop until it’s done.
Feb 2, 16:30 UTC
Update -
Issue Summary
We have identified an issue preventing call transcriptions from being written to the database. While calls are still being received, their transcriptions are currently delayed.
Current Impact
- Call recordings are still being received.
- Transcriptions are delayed, with a ~22-hour backlog that will take time to clear.
Actions We Are Taking
- Implementing a more scalable storage solution that will resolve the issue
- Processing all outstanding calls
Next Update
We will provide another update by Saturday at 10 pm GMT, or sooner if there are significant developments.
Jan 31, 15:07 UTC
Identified -
The issue has been identified and a fix is being implemented.
Jan 31, 07:53 UTC
Investigating -
We are currently experiencing processing delays due to an unexpected increase in call volumes. Our team is actively working to resolve the issue. We apologise for any inconvenience and appreciate your patience.
Jan 30, 16:24 UTC