Incident Report: 20181114 - Revision history

10418: /* Causes */

2018-11-14T23:41:53Z

Causes

← Older revision		Revision as of 00:41, 15 November 2018
Line 18:		Line 18:
	Interestingly, Rednet (Dante) and BAPS continued to function properly, so Jordan and Tom Burrows (who just happened to be in the station) did an unplanned Chat and Such, and were later joined by Jacob Dicker.		Interestingly, Rednet (Dante) and BAPS continued to function properly, so Jordan and Tom Burrows (who just happened to be in the station) did an unplanned Chat and Such, and were later joined by Jacob Dicker.

	Approximately 30 seconds before the resumption of full services (ITS turned themselves back off and on again) Isaac was able to manually re-start ~~Jutebox~~ and the crisis was over.		Approximately 30 seconds before the resumption of full services (ITS turned themselves back off and on again) Isaac was able to manually re-start Jukebox and the crisis was over.

	== Work Required ==		== Work Required ==

10418: /* Work Required */

2018-11-14T23:41:44Z

Work Required

← Older revision		Revision as of 00:41, 15 November 2018
Line 25:		Line 25:
	Honestly, we got lucky. If this had happened at 4am the dead-air time might have been measured in hours rather than minutes.		Honestly, we got lucky. If this had happened at 4am the dead-air time might have been measured in hours rather than minutes.

	Also, don't walk into Studio ~~Read~~ and proclaim everything is broken.		Also, don't walk into Studio Red and proclaim everything is broken.

	[[Category:Incident Reports]]		[[Category:Incident Reports]]

10418: /* Work Required */

2018-11-14T23:41:39Z

Work Required

← Older revision		Revision as of 00:41, 15 November 2018
Line 21:		Line 21:

	== Work Required ==		== Work Required ==
			Investigate reducing dependency on ITS systems (NTP/DNS).

			Honestly, we got lucky. If this had happened at 4am the dead-air time might have been measured in hours rather than minutes.

			Also, don't walk into Studio Read and proclaim everything is broken.

	[[Category:Incident Reports]]		[[Category:Incident Reports]]

10418: /* Causes */

2018-11-14T23:40:30Z

Causes

← Older revision		Revision as of 00:40, 15 November 2018
Line 18:		Line 18:
	Interestingly, Rednet (Dante) and BAPS continued to function properly, so Jordan and Tom Burrows (who just happened to be in the station) did an unplanned Chat and Such, and were later joined by Jacob Dicker.		Interestingly, Rednet (Dante) and BAPS continued to function properly, so Jordan and Tom Burrows (who just happened to be in the station) did an unplanned Chat and Such, and were later joined by Jacob Dicker.

	ITS turned themselves back off and on again, and ~~our stuff started working again~~.		Approximately 30 seconds before the resumption of full services (ITS turned themselves back off and on again) Isaac was able to manually re-start Jutebox and the crisis was over.

	== Work Required ==		== Work Required ==

10418 at 23:39, 14 November 2018

2018-11-14T23:39:50Z

← Older revision		Revision as of 00:39, 15 November 2018
Line 14:		Line 14:
	This page is under development because this literally just happened.		This page is under development because this literally just happened.
	== Causes ==		== Causes ==
	Basically, ITS broke so anything at URY that's reliant on DNS/Nameservers also broke, so we lost Jukebox, MyRadio and even the studio selector. However, it couldn't have happened at a better time with Head of Computing Jordan Cameron and Assistant Head of Computing Isaac Lowe both in the station at the same time.		Basically, ITS broke so anything at URY that's reliant on their DNS/Nameservers also broke, so we lost Jukebox, MyRadio and even the studio selector. However, it couldn't have happened at a better time with Head of Computing Jordan Cameron and Assistant Head of Computing Isaac Lowe both in the station at the same time.

	Interestingly, Rednet (Dante) and BAPS continued to function properly, so Jordan and Tom Burrows (who just happened to be in the station) did an unplanned Chat and Such, and were later joined by Jacob Dicker.		Interestingly, Rednet (Dante) and BAPS continued to function properly, so Jordan and Tom Burrows (who just happened to be in the station) did an unplanned Chat and Such, and were later joined by Jacob Dicker.

			ITS turned themselves back off and on again, and our stuff started working again.

	== Work Required ==		== Work Required ==

10418 at 23:39, 14 November 2018

2018-11-14T23:39:15Z

← Older revision		Revision as of 00:39, 15 November 2018
Line 12:		Line 12:
	(Total dead air: 16:28:44 - 16:30:36)		(Total dead air: 16:28:44 - 16:30:36)

			This page is under development because this literally just happened.
	== Causes ==		== Causes ==
			Basically, ITS broke so anything at URY that's reliant on DNS/Nameservers also broke, so we lost Jukebox, MyRadio and even the studio selector. However, it couldn't have happened at a better time with Head of Computing Jordan Cameron and Assistant Head of Computing Isaac Lowe both in the station at the same time.

			Interestingly, Rednet (Dante) and BAPS continued to function properly, so Jordan and Tom Burrows (who just happened to be in the station) did an unplanned Chat and Such, and were later joined by Jacob Dicker.

	== Work Required ==		== Work Required ==

10418 at 18:27, 14 November 2018

2018-11-14T18:27:31Z

← Older revision		Revision as of 19:27, 14 November 2018
Line 1:		Line 1:
	{{Incident		{{Incident
	\|brief=ITS broke		\|brief=ITS broke
	\|severity=~~Critical~~		\|severity=Moderate
	\|impact=High (Dead air for around 2 minutes)		\|impact=High (Dead air for around 2 minutes)
	\|start=25/02/2017 16:29		\|start=25/02/2017 16:29
Line 9:		Line 9:
	\|others=[[Jordan Cameron]]		\|others=[[Jordan Cameron]]
	}}		}}

	~~ITS had a major system error. This broke our stuff. More to follow.~~

	(Total dead air: 16:28:44 - 16:30:36)		(Total dead air: 16:28:44 - 16:30:36)

	~~At 19:22, output was successfully switched to OB (they were running late anyway), which then proceeded without incident.~~

	== Causes ==		== Causes ==
	~~Namesevers.~~

	== Work Required ==		== Work Required ==
	~~Namesevers.~~

	[[Category:Incident Reports]]		[[Category:Incident Reports]]

10418 at 18:26, 14 November 2018

2018-11-14T18:26:11Z

← Older revision		Revision as of 19:26, 14 November 2018
Line 5:		Line 5:
	\|start=25/02/2017 16:29		\|start=25/02/2017 16:29
	\|end=25/02/2017 17:30		\|end=25/02/2017 17:30
	\|mitigation=~~ITS Broke~~		\|mitigation=Fix Nameservers
	\|leader=[[Isaac Lowe]]		\|leader=[[Isaac Lowe]]
	\|others=[[Jordan Cameron]]		\|others=[[Jordan Cameron]]

10418: Created page with "{{Incident |brief=ITS broke |severity=Critical |impact=High (Dead air for around 2 minutes) |start=25/02/2017 16:29 |end=25/02/2017 17:30 |mitigation=ITS Broke |..."

2018-11-14T18:25:50Z

Created page with "{{Incident |brief=ITS broke |severity=Critical |impact=High (Dead air for around 2 minutes) |start=25/02/2017 16:29 |end=25/02/2017 17:30 |mitigation=ITS Broke |..."

New page

{{Incident
|brief=ITS broke
|severity=Critical
|impact=High (Dead air for around 2 minutes)
|start=25/02/2017 16:29
|end=25/02/2017 17:30
|mitigation=ITS Broke
|leader=[[Isaac Lowe]]
|others=[[Jordan Cameron]]
}}

ITS had a major system error. This broke our stuff. More to follow.

(Total dead air: 16:28:44 - 16:30:36)

At 19:22, output was successfully switched to OB (they were running late anyway), which then proceeded without incident.

== Causes ==
Namesevers.

== Work Required ==
Namesevers.

[[Category:Incident Reports]]