Incident Report: 20191029: Difference between revisions
No edit summary |
|||
| (2 intermediate revisions by the same user not shown) | |||
| Line 20: | Line 20: | ||
Turns out during this thoughtful TX stare someone came in and thought "Why the fuck is Red on thats not right!" and switched to Jukebox, which suddenly worked as normal. MM is very confused at this and declares the incident over at 12:26, unsure if the incident was over (But the dead air was gone and MM had shit to do so eh, good enough). | Turns out during this thoughtful TX stare someone came in and thought "Why the fuck is Red on thats not right!" and switched to Jukebox, which suddenly worked as normal. MM is very confused at this and declares the incident over at 12:26, unsure if the incident was over (But the dead air was gone and MM had shit to do so eh, good enough). | ||
== What Went Well == | === What Went Well === | ||
After the incident was declared, partial recovery only took five minutes and a complete recovery took 20. | After the incident was declared, partial recovery only took five minutes and a complete recovery took 20. | ||
== What Did Not Go Well == | === What Did Not Go Well === | ||
The dead air was unnoticed for over an hour, due to a failure in the dead air alarm | The dead air was unnoticed for over an hour, due to a failure in the dead air alarm | ||
== How We Got Lucky == | === How We Got Lucky === | ||
* The Chief Engineer was in the station when he thought to check AM and noticed the dead air - had it been a presenter who may not have checked, the dead air may have been far longer | * The Chief Engineer was in the station when he thought to check AM and noticed the dead air - had it been a presenter who may not have checked, the dead air may have been far longer | ||
| Line 36: | Line 36: | ||
After running out of his seminar on East, and getting stopped dead in his tracks by First Bus, MP started pulling all the logs he could - namely the HQ audio logs and the Liquidsoap logs. From this he was able to establish the following timeline (all times GMT): | After running out of his seminar on East, and getting stopped dead in his tracks by First Bus, MP started pulling all the logs he could - namely the HQ audio logs and the Liquidsoap logs. From this he was able to establish the following timeline (all times GMT): | ||
== Timeline == | === Timeline === | ||
Dead air begins: 11:02:30 | Dead air begins: 11:02:30 | ||
| Line 73: | Line 73: | ||
** Done - MP created DEARIE ME (Disappointingly Empty Airwave Remediation and Incident Escalation Management Engine), a dead air alarm listening to the AM feed | ** Done - MP created DEARIE ME (Disappointingly Empty Airwave Remediation and Incident Escalation Management Engine), a dead air alarm listening to the AM feed | ||
* Upgrade liquidsoap in the hopes of fixing bugs | * Upgrade liquidsoap in the hopes of fixing bugs | ||
** | ** Done as of 20191209 - whether this will actually fix anything or not is a different matter entirely | ||
[[Category:Incident Reports]] | [[Category:Incident Reports]] | ||