Incident Report: 20191130
| Incident Report | |
|---|---|
| Jukebox just gives up. | |
| Summary | |
| Severity | Moderate |
| Impact | Moderate (Approx 20mins of dead air) |
| Event Start | 30/11/2019 03:50 |
| Event End | 30/11/2019 05:02 |
| Recurrence Mitigation | All necessary changes implemented |
| Contacts | |
| Recovery Leader | Michael Grace (MG) |
| Other Attendees | Ben Allen (BA) |
Summary
Jukebox just couldn't deal with playing music anymore. Liquidsoap seems to have turned to those solid soap bars you get, which always get covered in dirt and never used. Often they smell nice though.
Timeline
03:49:50 - Jukebox Begins Playing You Really Got Me - The Kinks
03:50:53 - Dead Air Starts
03:51 - First Liquidsoap Email, Jukebox has switched to Jukebox. Dearie-Me message posted on #studio-faults. MG is on the bus from Flares, swears out loud, and is angry in slack
03:54 - Second Liquidsoap Email, Jukebox has switched to Jukebox. MG listens to all the feeds on audio.ury.org.uk to find we’re not broadcasting on AM, online or on the jukebox feed
03:56 - MG is on dolby through csteach1 and attempts to restart the jukebox.liq file. It doesn’t do anything. MG also tries ‘sel 1’ then ‘sel 3’. Also, doesn’t work.
03:58 - The sad realisation hits that MG isn’t going home, and will have to run to the studio from the library bus stop. Gets really strange looks from Vanbrugh porters when MG picks up the key card.
04:06:54 - MG, out of breath, starts breathing heavily down the microphone in red. DEAD AIR ENDS. Total Dead Air: 16:01. The jukebox fader is also outputting silence.
04:07:09 - MG plays Flagship News In, giving him time to schedule a few things in BAPS. Messages Slack to say he’s doing a show.
04:11:12 - MG plays Flagship News Out and starts Pyramid Song by Radiohead. Can now start troubleshooting given music is cued in BAPS. MG attempts to restart the liquidsoap service to no result. MG opens the liquidsoap logs to see what’s happened.
04:16:00 - Pyramid song ends and MG geeks out about it’s time signature and rhythm. Attempts to sing on air but realises immediately its a big mistake.
04:16:35 - Venusian 1 - King Gizzard and the Lizard Wizard starts. MG still looking through logs.
04:19:45 - MG starts another link, saying he’s signing into Google Drive so he can save screenshots of the logs. MG contemplates staying in the studio all night.
04:20:20 - Come Together - The Beatles starts. MG tries putting his screenshot into Google Drive, but the quality isn’t good enough to bother. Takes pictures on phone instead.
04:24:29 - Mr Brightside - The Killers starts. MG attempts to run some of the files in the liquidsoap/scripts directory on dolby, starting with jackStart.sh.
04:28:44 - Smells Like Teen Spirit - Nirvana starts. During this song, BA, who can’t sleep, looks at his phone to find what’s happened. He starts listening to URY, thinking its working because it’s playing music.
04:33:34 - MG admits on air he needs to go to the toilet. BA realises that actually MG is presenting, not Jukebox. BA messages Slack saying he’s on his way.
04:44:45 - Bohemian Rhapsody starts because it’s a long song. MG goes to the Vanbrugh toilets.
04:36 - MG returns and replies to BA on Slack saying he’s presenting. Asks for a jumper because he’s cold in only a short sleeved shirt.
04:37 - BA arrives in studio out of breath.
04:39:35 - Bohemian Rhapsody ends. MG and BA do the first joint link. Towards the end of the link, the chair MG is sitting on starts going down. It was weird.
04:40:54 - Fire - Kasabian starts. MG briefs BA, and they attempt jackConnect.sh and jackStart.sh. These also doesn’t work.
04:45:00 - MG and BA give their opinions on the URY christmas decorations. BA complains about how engineering need to fix the headphones in the studio. MG and BA discuss when the watershed ends, which looking up afterwards is 5:30am. MG introduces U Can’t Touch This, which is from 1990, and MG tries to make a comment about touching jukebox, but fails miserably.
04:48:10 - U Can’t Touch This - MC Hammer starts. During this song, we run startAudio.sh, and Jukebox is now playing through the fader. We assume this means it is working properly now.
04:52:24 - MG and BA discuss their nights out. They want to go and do the 5am news now.
04:54:43 - Free Bird - Lynyrd Skynyrd starts. They are monitoring the Jukebox fader during this, hoping all is good.
04:58:59 - Final link before the news.
04:59:45 - Flagship news in starts. MG and BA sign off to do the news.
05:02:00 - Flagship news out, and switch to Jukebox. End of Situation. MG and BA monitor for about another 25 minutes. MG runs to get a 05:35 bus to Goodricke and goes to bed at about 06:30.
What Went Well
Michael and Ben pulled off a good radio show whilst trying to fix the problem.
What Did Not Go Well
After the jukebox software died, there was nothing to replace it, as even Monty Python is in the jukebox script for when MyRadio gives up.
How We Got Lucky
MG had been on a night out, and was on the bus back, so the response time was very quick.
Investigation
Despite the Dead Air starting at 03:50, the Jukebox logs go until 03:56 where shutdown starts at 03:56:53, and the log ends at 03:56:59. There is nothing unusual in the logs beforehand, except at 03:56:14 at "#get called when not #is_ready" message appeared, which isn't supposed to happen, but frankly is normal with Jukebox.
At the point of Dead Air starting, another one of those error messages appeared. After this, the log restarts at 03:55:58, which MG thinks is when he attempted to restart jukebox on the bus.
Differences to last time this happened
Last time, on 29/10/2019, jukebox was still running, just not outputting on AM or online. This was rectified by resetting the selector. This time, the jukebox software had completely crashed.
The last event brought the introduction of DEARIE-ME, which was where the first notifications came from for this incident.
Conclusions
We believe what happened was a bug in liquidsoap related to the #get and #is_ready error message caused jukebox to crash for the first time, despite many of these errors. In attempting to reload everything, Jack basically said "I've already got a Jukebox feed, I don't care about a new one" as the jack wasn't released when Jukebox crashed.
Further Work
The issue with the error message has been there all term, and we're still trying to work out why is sometimes gets called, why the liquidsoap clocks get out of sync giving us brief moments of silence. The investigations must continue.