249
edits
Changes
Jump to navigation
Jump to search
Line 22:
Line 22: − + − (BOLLACKOVS HERE PLS) + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Incident Report: 20191029 (view source)
Revision as of 14:10, 5 November 2019
, 14:10, 5 November 2019no edit summary
Turns out during this thoughtful TX stare someone came in and thought "Why the fuck is Red on thats not right!" and switched to Jukebox, which suddenly worked as normal. MM is very confused at this and declares the incident over at 12:26, unsure if the incident was over (But the dead air was gone and MM had shit to do so eh, good enough).
Turns out during this thoughtful TX stare someone came in and thought "Why the fuck is Red on thats not right!" and switched to Jukebox, which suddenly worked as normal. MM is very confused at this and declares the incident over at 12:26, unsure if the incident was over (But the dead air was gone and MM had shit to do so eh, good enough).
=MPs Investigation=
== What Went Well ==
After the incident was declared, partial recovery only took five minutes and a complete recovery took 20.
== What Did Not Go Well ==
The dead air was unnoticed for over an hour, due to a failure in the dead air alarm
== How We Got Lucky ==
* The Chief Engineer was in the station when he thought to check AM and noticed the dead air - had it been a presenter who may not have checked, the dead air may have been far longer
* Jukebox/Liquidsoap/whatever broke unbroke itself
=MP's Investigation=
After running out of his seminar on East, and getting stopped dead in his tracks by First Bus, MP started pulling all the logs he could - namely the HQ audio logs and the Liquidsoap logs. From this he was able to establish the following timeline (all times GMT):
== Timeline ==
Dead air begins: 11:02:30
MM reports dead air in Slack: 12:06
Dead air ends with MM's presPC bodge: 12:11:20
Jukebox back on air via selector: 12:26:12
Total dead air: 1:08:50
== Significant realisations ==
* Jukebox continued to tracklist even throughout the dead air
* The selector logs showed nothing out of the ordinary - at 00:02:12 output was switched to Jukebox, where it stayed until 12:11:20 when MM switched to Red to end the dead air
* The dead air was heard on both AM and online - it appears clearly in the HQ log
* The dead air alarm... did not alarm about the dead air
* When MM switched output to Red, it worked without any problems
== Conclusions ==
The most likely culprit is Liquidsoap hitting one of its numerous bugs, getting stuck, and failing to output audio until the switch to Red unstuck'd it. The dead air alarm is part of <code>sel.liq</code>, so it not going off implies that the whole script broke, which would make sense considering Jukebox continued to tracklist (implying that it thought it was on air).
Other potential culprits are the Jack connection from Liquidsoap to the sound card, or the sound card itself, or TX itself, but all of these are unlikely:
* Jack is relatively stable software
* While Focusrite does not officially have Linux drivers, their cards are actually often more stable on Linux than on Windows
* TX is hard to break, plus, the dead air was on both AM and online, which makes a physical signal path break unlikely
= Further Work =
* Create a more reliable dead air detector
** in progress - MP working on DEARIE ME (Disappointingly Empty Airwave Remediation and Incident Escalation Management Engine), a dead air alarm listening to the AM feed
* Upgrade liquidsoap in the hopes of fixing bugs
** Will need to wait, as this is a risky process
[[Category:Incident Reports]]
[[Category:Incident Reports]]