Incident Report: 20200304: Difference between revisions
Created page with "{{Incident |brief=Dante is hard, mmkay? |severity=High |impact=High (Approx 15mins of dead air during a live show, also server reboot) |start=04/03/2020 21:17 |end=0..." |
No edit summary |
||
| Line 33: | Line 33: | ||
= Timeline = | = Timeline = | ||
19:00-21:00 MS and MP investigating CPU issues on Dolby. | |||
21:00-21:10 sporadic bits of dead air on AM & online (some presenter error, some potentially Dante related) | |||
21:17:00 MG reports “big issue” in Slack | |||
21:18:35 Dante Controller reports “Device 001DC10208C1 (device name not known) is now a grandmaster” - the RedNet 3 has dropped off | |||
21:18:36 MP screams “DANTE!” | |||
21:19-21:20 MP turns the RedNet 3 off and on again | |||
21:20 Dante starts flapping clock, but seemingly stabilises | |||
21:28 The AM feed becomes noticeably sosig, slowly degrades quality | |||
21:29-21:32 The AM goes to silence with occasional bursts of techno. At the same time, Dante Controller reports various clock switches and devices muting (which is Dante-speak for “oman i am no good with audio pls to halp”). The clock master cycles between Phil, Office, and Studio Red several times. MP says in Slack at 21:31 “[Dante] is not in a good place right now.” | |||
Relevant log entry: 2020-03-04 21:30:53 GMT 1583357453638 Information "uryStores" "Timed out 3 times sending message 'UpdateRxChannels' to uryStores, giving up." | |||
21:33:34 MP, MS, BA set Studio Blue to clock master. Dante sort-of stabilises except not really | |||
21:36:21 IL reports that the clock coax between uryStores and Scarlett is hot to the touch. | |||
Around this time (exact timing TBC) IL attempts switching out the coax, but fails when he realises that the replacement cable is a piece of shit. | |||
21:38:25 we go to DEAD AIR, and at 21:39:14 Dearie-Me triggers. | |||
Around 21:37-39 MP uses RedNet Control to switch the RedNet 3’s clock input to ADAT, believing this will help things, and at 21:38:49 Dante Controller reports that uryStores is clock master. This does not make the Scarlett happy. | |||
21:39:56 Studio Blue is switched to Clock Master, confirmed by MP in Slack at 21:40:07. Still dead air. | |||
21:42 MP tries an audio pipeline restart (aka startAudio.sh). This does not help. | |||
21:44 MP makes the call to reboot Dolby. | |||
21:46:42 Dolby boots to Linux. | |||
21:47:04 MP SSHes into Dolby and tries starting audio, failing. | |||
21:49:41 DEAD AIR ENDS - goes to horrible sosig Flagship News (from the TX OB) | |||
21:50:17 MP runs startAudio.sh | |||
21:50:50 uryStores becomes clock master, at 21:51:09 Dante finishes re-synchronising | |||
21:52:00 horrible sosig Flagship News ends and AM has off-air loop | |||
21:54-21:56 MP realises that SelectorListener isn’t running, and runs some commands to try and start it. | |||
21:58:59 MP declares the incident over | |||