Incident Report: 20200511 - Revision history

11090 at 09:47, 7 July 2020

2020-07-07T09:47:55Z

← Older revision		Revision as of 10:47, 7 July 2020
Line 7:		Line 7:
	\|mitigation=Reduce dependency on uplink		\|mitigation=Reduce dependency on uplink
	\|leader=Connor Sanders (CS)		\|leader=Connor Sanders (CS)
	\|others=Isaac Lowe (IL)		\|others=Isaac Lowe (IL), Marks Polakovs (MP)
	}}		}}

Line 22:		Line 22:
	* Reduce dependency on upstream services		* Reduce dependency on upstream services
	:* Investigate a local caching DNS resolver?		:* Investigate a local caching DNS resolver?
			:* '''MP - done-ish, running unbound on uryfw0 and many (but not all boxes use it)'''
	* Ask ITS nicely to tell us when they take down our campus uplink		* Ask ITS nicely to tell us when they take down our campus uplink
			:* '''MP - done'''
	* Ask ITS nicely to make it reboot at xx:30 instead of xx:00		* Ask ITS nicely to make it reboot at xx:30 instead of xx:00
			:* '''MP - done'''
	* Improve documentation and logging of the new WebStudio services, to make future troubleshooting easier		* Improve documentation and logging of the new WebStudio services, to make future troubleshooting easier
	* Figure out why Dearie-Me didn't fire - possibly needs a recalibrate		* Figure out why Dearie-Me didn't fire - possibly needs a recalibrate

2020-05-11T11:37:05Z

Reoccurrence mitigation

← Older revision		Revision as of 12:37, 11 May 2020
Line 22:		Line 22:
	* Reduce dependency on upstream services		* Reduce dependency on upstream services
	:* Investigate a local caching DNS resolver?		:* Investigate a local caching DNS resolver?
	* Ask ITS ~~kindly~~ to tell us when they take down our campus uplink		* Ask ITS nicely to tell us when they take down our campus uplink
	* Ask ITS ~~kindly~~ to make it reboot at xx:30 instead of xx:00		* Ask ITS nicely to make it reboot at xx:30 instead of xx:00
	* Improve documentation and logging of the new WebStudio services, to make future troubleshooting easier		* Improve documentation and logging of the new WebStudio services, to make future troubleshooting easier
	* Figure out why Dearie-Me didn't fire - possibly needs a recalibrate		* Figure out why Dearie-Me didn't fire - possibly needs a recalibrate

2020-05-11T11:36:33Z

← Older revision		Revision as of 12:36, 11 May 2020
Line 4:		Line 4:
	\|impact=High (Dead air for around 45 minutes)		\|impact=High (Dead air for around 45 minutes)
	\|start=2020-05-11 05:00		\|start=2020-05-11 05:00
	\|end=~~202~~-05-11 05:46		\|end=2020-05-11 05:46
	\|mitigation=Reduce dependency on uplink		\|mitigation=Reduce dependency on uplink
	\|leader=Connor Sanders (CS)		\|leader=Connor Sanders (CS)

2020-05-11T11:35:22Z

2020-05-11T11:33:57Z

Reoccurrence mitigation

← Older revision		Revision as of 12:33, 11 May 2020
Line 25:		Line 25:
	* Ask ITS kindly to make it reboot at xx:30 instead of xx:00		* Ask ITS kindly to make it reboot at xx:30 instead of xx:00
	* Improve documentation and logging of the new WebStudio services, to make future troubleshooting easier		* Improve documentation and logging of the new WebStudio services, to make future troubleshooting easier
			* Figure out why Dearie-Me didn't fire - possibly needs a recalibrate
	* Reduce log spamminess of Dearie-Me, it filled up its journald buffer quite quickly		* Reduce log spamminess of Dearie-Me, it filled up its journald buffer quite quickly

2020-05-11T11:31:18Z

← Older revision		Revision as of 12:31, 11 May 2020
Line 9:		Line 9:
	\|others=Isaac Lowe (IL)		\|others=Isaac Lowe (IL)
	}}		}}

	~~HH:MM:SS~~
	~~Dead air start: 05:02:06.500~~
	~~Dead air ends: 05:45:42.000~~
	~~TOTAL: 00:43:35.500~~

	== Summary ==		== Summary ==
Line 31:		Line 26:
	* Improve documentation and logging of the new WebStudio services, to make future troubleshooting easier		* Improve documentation and logging of the new WebStudio services, to make future troubleshooting easier
	* Reduce log spamminess of Dearie-Me, it filled up its journald buffer quite quickly		* Reduce log spamminess of Dearie-Me, it filled up its journald buffer quite quickly

			== Timings ==

			HH:MM:SS
			Dead air start: 05:02:06.500
			Dead air ends: 05:45:42.000
			TOTAL: 00:43:35.500

	[[Category:Incident Reports]]		[[Category:Incident Reports]]

2020-05-11T11:30:47Z

← Older revision		Revision as of 12:30, 11 May 2020
Line 1:		Line 1:
	{{Incident		{{Incident
	\|brief=Switch reboot took down DNS		\|brief=Switch reboot took down DNS, which brooke'd selector
	\|severity=Moderate		\|severity=Moderate
	\|impact=High (Dead air for around 45 minutes)		\|impact=High (Dead air for around 45 minutes)

2020-05-11T11:30:07Z

2020-05-11T11:25:47Z

← Older revision		Revision as of 12:25, 11 May 2020
Line 3:		Line 3:
	\|severity=Moderate		\|severity=Moderate
	\|impact=High (Dead air for around 45 minutes)		\|impact=High (Dead air for around 45 minutes)
	\|start=~~25/02/2017 16~~:29		\|start=2020-05-11 05:00
	\|end=~~25/02/2017 17~~:30		\|end=202-05-11 05:46
	\|mitigation=Reduce dependency on uplink		\|mitigation=Reduce dependency on uplink
	\|leader=Connor Sanders (CS)		\|leader=Connor Sanders (CS)

2020-05-11T11:24:58Z

← Older revision		Revision as of 12:24, 11 May 2020
Line 2:		Line 2:
	\|brief=Switch reboot took down DNS		\|brief=Switch reboot took down DNS
	\|severity=Moderate		\|severity=Moderate
	\|impact=High (Dead air for around 2 minutes)		\|impact=High (Dead air for around 45 minutes)
	\|start=25/02/2017 16:29		\|start=25/02/2017 16:29
	\|end=25/02/2017 17:30		\|end=25/02/2017 17:30