13
edits
Changes
Jump to navigation
Jump to search
Incident Report: 20170225 (view source)
Revision as of 13:44, 26 February 2017
, 13:44, 26 February 2017Created page with "{{Incident |brief=Streaming services were restarted, then failed to come back up |severity=Critical |impact=High (Dead air for around 15 minutes) |start=25/02/2017 18:..."
{{Incident
|brief=Streaming services were restarted, then failed to come back up
|severity=Critical
|impact=High (Dead air for around 15 minutes)
|start=25/02/2017 18:45
|end=25/02/2017 19:10
|mitigation=Don't restart streams in termtime.
|leader=[[Charles Pigott]] <charles.pigott@ury.org.uk>
|others=[[Chris Taylor]] <christhebaron@ury.org.uk>
[[Brooke Hatton]] <brooke@ury.org.uk>
[[Anthony Williams]] <anthony@ury.org.uk>
}}
During setup for the OB for ERN2017, it was found that the OB line was not actually getting selected when switched to, only continuing to broadcast jukebox. This was suspected to be because the jack audio inputs were not being mapped to the correct output, but qjackctl on dolby was unable to be accessed to confirm this (as it requires horrible X-forwarding to an office PC and the process for which is unclear). A possible cause for this is when listeners to the `/OB-Line` stream were "moved" within icecast to `/jukebox` at an earlier point in the week while testing.
At 18:54, having run out of further options and running out of time (the OB was scheduled to start at 19:00), both liquidsoap and icecast services were restarted (until that point had been running solidly since September) which then failed to come back properly. At this point station output went offline.
At 19:01, GodEmperor Anthony was phoned.
At 19:08, the stream was brought back up (by running `startAudio.sh`), and all was well again.
At 19:22, output was successfully switched to OB (they were running late anyway), which then proceeded without incident.
== Causes ==
Output failing to switch to OB lead to some bad decisions. Original cause as yet unclear and investigations ongoing.
== Work Required ==
Investigate a better method of accessing jack config (jack_lsb?)
Clear up the process for restarting the streaming services, and the possible (likely) results of doing so.
[[Category:Incident Reports]]
|brief=Streaming services were restarted, then failed to come back up
|severity=Critical
|impact=High (Dead air for around 15 minutes)
|start=25/02/2017 18:45
|end=25/02/2017 19:10
|mitigation=Don't restart streams in termtime.
|leader=[[Charles Pigott]] <charles.pigott@ury.org.uk>
|others=[[Chris Taylor]] <christhebaron@ury.org.uk>
[[Brooke Hatton]] <brooke@ury.org.uk>
[[Anthony Williams]] <anthony@ury.org.uk>
}}
During setup for the OB for ERN2017, it was found that the OB line was not actually getting selected when switched to, only continuing to broadcast jukebox. This was suspected to be because the jack audio inputs were not being mapped to the correct output, but qjackctl on dolby was unable to be accessed to confirm this (as it requires horrible X-forwarding to an office PC and the process for which is unclear). A possible cause for this is when listeners to the `/OB-Line` stream were "moved" within icecast to `/jukebox` at an earlier point in the week while testing.
At 18:54, having run out of further options and running out of time (the OB was scheduled to start at 19:00), both liquidsoap and icecast services were restarted (until that point had been running solidly since September) which then failed to come back properly. At this point station output went offline.
At 19:01, GodEmperor Anthony was phoned.
At 19:08, the stream was brought back up (by running `startAudio.sh`), and all was well again.
At 19:22, output was successfully switched to OB (they were running late anyway), which then proceeded without incident.
== Causes ==
Output failing to switch to OB lead to some bad decisions. Original cause as yet unclear and investigations ongoing.
== Work Required ==
Investigate a better method of accessing jack config (jack_lsb?)
Clear up the process for restarting the streaming services, and the possible (likely) results of doing so.
[[Category:Incident Reports]]