Incident Report: 20191029 - Revision history

11090 at 14:54, 11 December 2019

2019-12-11T14:54:43Z

← Older revision		Revision as of 15:54, 11 December 2019
Line 73:		Line 73:
	** Done - MP created DEARIE ME (Disappointingly Empty Airwave Remediation and Incident Escalation Management Engine), a dead air alarm listening to the AM feed		** Done - MP created DEARIE ME (Disappointingly Empty Airwave Remediation and Incident Escalation Management Engine), a dead air alarm listening to the AM feed
	* Upgrade liquidsoap in the hopes of fixing bugs		* Upgrade liquidsoap in the hopes of fixing bugs
	** ~~Will need to wait,~~ as this is a ~~risky process~~		** Done as of 20191209 - whether this will actually fix anything or not is a different matter entirely

	[[Category:Incident Reports]]		[[Category:Incident Reports]]

11090: /* MP's Investigation */

2019-11-23T23:33:22Z

MP's Investigation

← Older revision		Revision as of 00:33, 24 November 2019
Line 36:		Line 36:
	After running out of his seminar on East, and getting stopped dead in his tracks by First Bus, MP started pulling all the logs he could - namely the HQ audio logs and the Liquidsoap logs. From this he was able to establish the following timeline (all times GMT):		After running out of his seminar on East, and getting stopped dead in his tracks by First Bus, MP started pulling all the logs he could - namely the HQ audio logs and the Liquidsoap logs. From this he was able to establish the following timeline (all times GMT):

	== Timeline ==		=== Timeline ===

	Dead air begins: 11:02:30		Dead air begins: 11:02:30

11090 at 23:32, 23 November 2019

2019-11-23T23:32:24Z

← Older revision		Revision as of 00:32, 24 November 2019
Line 20:		Line 20:
	Turns out during this thoughtful TX stare someone came in and thought "Why the fuck is Red on thats not right!" and switched to Jukebox, which suddenly worked as normal. MM is very confused at this and declares the incident over at 12:26, unsure if the incident was over (But the dead air was gone and MM had shit to do so eh, good enough).		Turns out during this thoughtful TX stare someone came in and thought "Why the fuck is Red on thats not right!" and switched to Jukebox, which suddenly worked as normal. MM is very confused at this and declares the incident over at 12:26, unsure if the incident was over (But the dead air was gone and MM had shit to do so eh, good enough).

	== What Went Well ==		=== What Went Well ===

	After the incident was declared, partial recovery only took five minutes and a complete recovery took 20.		After the incident was declared, partial recovery only took five minutes and a complete recovery took 20.

	== What Did Not Go Well ==		=== What Did Not Go Well ===

	The dead air was unnoticed for over an hour, due to a failure in the dead air alarm		The dead air was unnoticed for over an hour, due to a failure in the dead air alarm

	== How We Got Lucky ==		=== How We Got Lucky ===

	* The Chief Engineer was in the station when he thought to check AM and noticed the dead air - had it been a presenter who may not have checked, the dead air may have been far longer		* The Chief Engineer was in the station when he thought to check AM and noticed the dead air - had it been a presenter who may not have checked, the dead air may have been far longer

11090: /* Further Work */

2019-11-07T15:24:12Z

Further Work

← Older revision		Revision as of 16:24, 7 November 2019
Line 71:		Line 71:

	* Create a more reliable dead air detector		* Create a more reliable dead air detector
	** ~~in progress~~ - MP ~~working on~~ DEARIE ME (Disappointingly Empty Airwave Remediation and Incident Escalation Management Engine), a dead air alarm listening to the AM feed		** Done - MP created DEARIE ME (Disappointingly Empty Airwave Remediation and Incident Escalation Management Engine), a dead air alarm listening to the AM feed
	* Upgrade liquidsoap in the hopes of fixing bugs		* Upgrade liquidsoap in the hopes of fixing bugs
	** Will need to wait, as this is a risky process		** Will need to wait, as this is a risky process

	[[Category:Incident Reports]]		[[Category:Incident Reports]]

11090: it's not tho

2019-11-06T11:11:12Z

it's not tho

← Older revision		Revision as of 12:11, 6 November 2019
Line 9:		Line 9:
	\|others=Morgan McKay(MM)		\|others=Morgan McKay(MM)
	}}		}}

	~~Report under construction.~~

	=Summary=		=Summary=

11090: /* Conclusions */

2019-11-05T13:11:19Z

Conclusions

← Older revision		Revision as of 14:11, 5 November 2019
Line 67:		Line 67:
	* While Focusrite does not officially have Linux drivers, their cards are actually often more stable on Linux than on Windows		* While Focusrite does not officially have Linux drivers, their cards are actually often more stable on Linux than on Windows
	* TX is hard to break, plus, the dead air was on both AM and online, which makes a physical signal path break unlikely		* TX is hard to break, plus, the dead air was on both AM and online, which makes a physical signal path break unlikely

			We cannot conclusively point the finger at liquidsoap, but it is more likely than not to be the culprit.

	= Further Work =		= Further Work =

11090 at 13:10, 5 November 2019

2019-11-05T13:10:49Z

@@ Line 22: / Line 22: @@
 Turns out during this thoughtful TX stare someone came in and thought "Why the fuck is Red on thats not right!" and switched to Jukebox, which suddenly worked as normal. MM is very confused at this and declares the incident over at 12:26, unsure if the incident was over (But the dead air was gone and MM had shit to do so eh, good enough).
-=MPs Investigation=
+== What Went Well ==
 [[Category:Incident Reports]]

10749 at 12:47, 5 November 2019

2019-11-05T12:47:50Z

← Older revision		Revision as of 13:47, 5 November 2019
Line 24:		Line 24:
	=MPs Investigation=		=MPs Investigation=
	(BOLLACKOVS HERE PLS)		(BOLLACKOVS HERE PLS)



			[[Category:Incident Reports]]

10749 at 12:37, 5 November 2019

2019-11-05T12:37:31Z

← Older revision		Revision as of 13:37, 5 November 2019
Line 20:		Line 20:
	Having finished his morning lectures at around 12:00, MM enters the station and has a quick check of Jukebox via a desks AM feed. MM notices that no sound is coming out. Checking the feed from the AM2, Yamaha, and Icecast show the same. In a nervous panic MM alerts the committee, switching to red and playing out the Jukebox feed via the Pres Pc (Through Icecast, as Jukebox still was playing audio itself) at 12:11. At the time MM didnt know online was off as well as AM so ran into TX and stared at it. The transmitter was sitting there all happy and smug, no reflected power to speak of, just the ominous "Audio break" LED latched on. Having no idea what to check, MM comes back to the studio after a good 10 minute stare and contemplate to find that Jukebox is back on air playing as normal.		Having finished his morning lectures at around 12:00, MM enters the station and has a quick check of Jukebox via a desks AM feed. MM notices that no sound is coming out. Checking the feed from the AM2, Yamaha, and Icecast show the same. In a nervous panic MM alerts the committee, switching to red and playing out the Jukebox feed via the Pres Pc (Through Icecast, as Jukebox still was playing audio itself) at 12:11. At the time MM didnt know online was off as well as AM so ran into TX and stared at it. The transmitter was sitting there all happy and smug, no reflected power to speak of, just the ominous "Audio break" LED latched on. Having no idea what to check, MM comes back to the studio after a good 10 minute stare and contemplate to find that Jukebox is back on air playing as normal.

	Turns out during this thoughtful TX stare someone came in and thought "Why the fuck is Red on thats not right!" and switched to Jukebox, which suddenly worked as normal. MM is very confused at this and declares the incident over.		Turns out during this thoughtful TX stare someone came in and thought "Why the fuck is Red on thats not right!" and switched to Jukebox, which suddenly worked as normal. MM is very confused at this and declares the incident over at 12:26, unsure if the incident was over (But the dead air was gone and MM had shit to do so eh, good enough).

	=MPs Investigation=		=MPs Investigation=
	(BOLLACKOVS HERE PLS)		(BOLLACKOVS HERE PLS)

10749: /* Summary */

2019-11-05T12:33:50Z

Summary

← Older revision		Revision as of 13:33, 5 November 2019
Line 13:		Line 13:

	=Summary=		=Summary=
	An (As of yet) not pinpointed issue caused Jukebox to fail to output to TX and Online for over an hour, despite the jukebox service still playing out music on Icecast and through the studio faders.		An (As of yet) not pinpointed issue caused Jukebox to fail to output to TX and Online for over an hour, despite the jukebox service still playing out music on Icecast and through the studio faders. Also the dead air alarm didnt tell anyone, the bastard.

	=Rough Sequence of Events=		=Rough Sequence of Events=