This incident is resolved and our engineering team will be continuing to closely monitor global ingest to ensure service continuity.
The incident today occurred during a global rollout of a new feature for the Streaming Network, for which maintenance has quietly been going on for over the last 48 hours. The rollout of this feature was verified in our testing and staging environments and had been performed so far without any impact on stream uptime. Unfortunately, one minor step in our deployment process caused an unexpected configuration change in our ingest server infrastructure, which triggered a software bug leading to unavailability of our ingest clusters. Our engineers quickly mitigated the issue by correcting the configuring change, and will be deploying a permanent fix for the software bug shortly. We will be working to understand the procedural failure that lead to this issue impacting our Streaming Network and improving our processes to prevent such an issue from happening again.
We apologize for the inconvenience this incident had on your streams and thank you for your patience as we worked to restore service. If you have any questions or concerns, please contact support (help@oscillicious.com).
Posted Jul 15, 2022 - 15:06 EDT
Update
Service has been restored to all users and global traffic levels are normalizing.
Posted Jul 15, 2022 - 14:30 EDT
Monitoring
At 2:15 PM EST (18:15 UTC), our engineers identified a software bug causing a temporary disruption of our global ingest servers, during routine maintenance. For a period of approximately 8 minutes, listeners were unable to connect to the Streaming Network. At this time, service has been fully restored. We have implemented a fix and are monitoring closely to prevent a recurrence. We will be performing an investigation to understand the procedural error that lead to this incident. We apologize for the inconvenience caused by this and will improve our upgrade for the future. If you have any questions, please contact support.