Difference between revisions of "Incident Report: 20191029"

From URY Wiki
Jump to navigation Jump to search
(2 intermediate revisions by the same user not shown)
Line 5: Line 5:
 
   |start=29/10/2019 11:02
 
   |start=29/10/2019 11:02
 
   |end=29/10/2019 12:26
 
   |end=29/10/2019 12:26
−
   |mitigation=Make sure the dead air alarm actually alerts when dead air happens
+
   |mitigation=Stop the self-DDoS, restart samba, cry
 
   |leader=Marks Polakovs (MP)
 
   |leader=Marks Polakovs (MP)
 
   |others=Morgan McKay(MM)
 
   |others=Morgan McKay(MM)
 
}}
 
}}
−
 
−
=Summary=
 
−
An (As of yet) not pinpointed issue caused Jukebox to fail to output to TX and Online for over an hour, despite the jukebox service still playing out music on Icecast and through the studio faders.
 
−
 
−
=Rough Sequence of Events=
 
−
At around 11:02 Jukebox stops playing out audio to streaming and TX. Nobody notices as nobody listens to us, and the dead air detector sends no emails (Not that it could do much other than "We detected a silence, jukebox has been switched to jukebox!")
 
−
 
−
Having finished his morning lectures at around 12:00, MM enters the station and has a quick check of Jukebox via a desks AM feed. MM notices that no sound is coming out. Checking the feed from the AM2, Yamaha, and Icecast show the same. In a nervous panic MM alerts the committee, switching to red and playing out the Jukebox feed via the Pres Pc (Through Icecast, as Jukebox still was playing audio itself) at 12:11. At the time MM didnt know online was off as well as AM so ran into TX and stared at it. The transmitter was sitting there all happy and smug, no reflected power to speak of, just the ominous "Audio break" LED latched on. Having no idea what to check, MM comes back to the studio after a good 10 minute stare and contemplate to find that Jukebox is back on air playing as normal.
 
−
 
−
Turns out during this thoughtful TX stare someone came in and thought "Why the fuck is Red on thats not right!" and switched to Jukebox, which suddenly worked as normal. MM is very confused at this and declares the incident over.
 
−
 
−
=MPs Investigation=
 
−
(BOLLACKOVS HERE PLS)
 

Revision as of 13:19, 5 November 2019

Incident Report
Jukebox forgets how an audio happens
Summary
Severity Moderate
Impact High (Approx 1hr of dead air)
Event Start 29/10/2019 11:02
Event End 29/10/2019 12:26
Recurrence Mitigation Stop the self-DDoS, restart samba, cry
Contacts
Recovery Leader Marks Polakovs (MP)
Other Attendees Morgan McKay(MM)
Retrieved from ‘https://ury.org.uk/wiki/mediawiki/index.php?title=Incident_Report:_20191029&oldid=866’