Beginning at approximately 3 AM EST until 10 AM EST on Dec 16, 2023, a subset of streams had erroneously limited listener capacity across our edge servers serving the Miami region of our Streaming Network. New listeners connecting to high capacity streams in this region experienced a "403 Over Capacity" error message. This issue has been resolved and we are monitoring to ensure it does not recur.
The root cause was a bug in a software update deployed to the Streaming Network. Streaming regions serving the West Coast US and East Coast US were also effected to a much lesser extent, with a much smaller portion of new listeners encountering this error but successfully connecting on retries. Since Radio Mast Stream Monitoring rotates through different edge servers and monitors from different cities, affected broadcasters would have received multiple downtime alerts overnight, showing streams going up and down, as different edge servers were tested.
We apologize for the impact this had on affected streams and we will be working on additional monitoring and testing processes to prevent a bug like this from entering production again. If you have any questions or concerns, please contact support: https://radiomast.io/support