Incident Report: 20170225
| Incident Report | |
|---|---|
| Streaming services were restarted, then failed to come back up | |
| Summary | |
| Severity | Critical |
| Impact | High (Dead air for around 15 minutes) |
| Event Start | 25/02/2017 18:45 |
| Event End | 25/02/2017 19:10 |
| Recurrence Mitigation | Don't restart streams in termtime. |
| Contacts | |
| Recovery Leader | Charles Pigott <charles.pigott@ury.org.uk> |
| Other Attendees | Chris Taylor <christhebaron@ury.org.uk>
Brooke Hatton <brooke@ury.org.uk> Anthony Williams <anthony@ury.org.uk> |
During setup for the OB for ERN2017, it was found that the OB line was not actually getting selected when switched to, only continuing to broadcast jukebox. This was suspected to be because the jack audio inputs were not being mapped to the correct output, but qjackctl on dolby was unable to be accessed to confirm this (as it requires horrible X-forwarding to an office PC and the process for which is unclear). A possible cause for this is when listeners to the `/OB-Line` stream were "moved" within icecast to `/jukebox` at an earlier point in the week while testing.
At 18:54, having run out of further options and running out of time (the OB was scheduled to start at 19:00), both liquidsoap and icecast services were restarted (until that point had been running solidly since September) which then failed to come back properly. At this point station output went offline.
At 19:01, GodEmperor Anthony was phoned.
At 19:08, the stream was brought back up (by running `startAudio.sh`), and all was well again. (Total dead air: 18:53:39 - 19:09:00)
At 19:22, output was successfully switched to OB (they were running late anyway), which then proceeded without incident.
Causes
Output failing to switch to OB lead to some bad decisions. Original cause was due to the "OB listener" being "moved" to the wrong stream earlier and the audio path getting unconfigured.
Work Required
Investigate a better method of accessing jack config (jack_lsb?) Clear up the process for restarting the streaming services, and the possible (likely) results of doing so.