Incident Report: 20131217 - Revision history

7449 at 09:48, 20 January 2014

2014-01-20T09:48:41Z

← Older revision		Revision as of 10:48, 20 January 2014
Line 1:		Line 1:
	[[Category:Incident Reports]]		[[Category:Incident Reports]]
	{{Incident		{{Incident
	\|brief=Some planned IT ~~Service~~ maintenance that went as planned had an unplanned impact on one of our servers.		\|brief=Some planned IT Services maintenance that went as planned had an unplanned impact on one of our servers.
	\|severity=Low		\|severity=Low
	\|impact=None (Only Production in term-time)		\|impact=None (Only Production in term-time)

7449 at 20:02, 19 December 2013

2013-12-19T20:02:50Z

← Older revision		Revision as of 21:02, 19 December 2013
Line 2:		Line 2:
	{{Incident		{{Incident
	\|brief=Some planned IT Service maintenance that went as planned had an unplanned impact on one of our servers.		\|brief=Some planned IT Service maintenance that went as planned had an unplanned impact on one of our servers.
	\|severity=~~Medium~~		\|severity=Low
	\|impact=~~Low~~ (~~service users have perished~~)		\|impact=None (Only Production in term-time)
	\|start=17/12/2013 07:30 (est)		\|start=17/12/2013 07:30 (est)
	\|end=19/12/2013 11:45		\|end=19/12/2013 11:45

7449 at 17:25, 19 December 2013

2013-12-19T17:25:28Z

← Older revision		Revision as of 18:25, 19 December 2013
Line 1:		Line 1:
			[[Category:Incident Reports]]
	{{Incident		{{Incident
	\|brief=Some planned IT Service maintenance that went as planned had an unplanned impact on one of our servers.		\|brief=Some planned IT Service maintenance that went as planned had an unplanned impact on one of our servers.
Line 20:		Line 21:
	* Xymon does not currently have any monitoring enabled for uryrrod. This should monitor at least system availability.		* Xymon does not currently have any monitoring enabled for uryrrod. This should monitor at least system availability.
	* Make further use of the Snapshot capabilities of VMWare. Once the server was restored, a snapshot was taken to enable easy restoration in the event of recurrence, but this should possibly become a more regular process.		* Make further use of the Snapshot capabilities of VMWare. Once the server was restored, a snapshot was taken to enable easy restoration in the event of recurrence, but this should possibly become a more regular process.

	~~[[Category:Incident Reports]]~~

7449 at 17:25, 19 December 2013

2013-12-19T17:25:12Z

← Older revision		Revision as of 18:25, 19 December 2013
Line 21:		Line 21:
	* Make further use of the Snapshot capabilities of VMWare. Once the server was restored, a snapshot was taken to enable easy restoration in the event of recurrence, but this should possibly become a more regular process.		* Make further use of the Snapshot capabilities of VMWare. Once the server was restored, a snapshot was taken to enable easy restoration in the event of recurrence, but this should possibly become a more regular process.

	[[Category: Incident Reports]]		[[Category:Incident Reports]]

7449 at 17:20, 19 December 2013

2013-12-19T17:20:58Z

← Older revision		Revision as of 18:20, 19 December 2013
Line 20:		Line 20:
	* Xymon does not currently have any monitoring enabled for uryrrod. This should monitor at least system availability.		* Xymon does not currently have any monitoring enabled for uryrrod. This should monitor at least system availability.
	* Make further use of the Snapshot capabilities of VMWare. Once the server was restored, a snapshot was taken to enable easy restoration in the event of recurrence, but this should possibly become a more regular process.		* Make further use of the Snapshot capabilities of VMWare. Once the server was restored, a snapshot was taken to enable easy restoration in the event of recurrence, but this should possibly become a more regular process.

			[[Category: Incident Reports]]

7449: Created page with "{{Incident |brief=Some planned IT Service maintenance that went as planned had an unplanned impact on one of our servers. |severity=Medium |impact=Low (service users hav..."

2013-12-19T17:20:40Z

Created page with "{{Incident |brief=Some planned IT Service maintenance that went as planned had an unplanned impact on one of our servers. |severity=Medium |impact=Low (service users hav..."

New page

{{Incident
|brief=Some planned IT Service maintenance that went as planned had an unplanned impact on one of our servers.
|severity=Medium
|impact=Low (service users have perished)
|start=17/12/2013 07:30 (est)
|end=19/12/2013 11:45
|mitigation=Some action taken, further action to be considered.
|leader=[[Lloyd Wallis]] <lpw@ury.org.uk>
|others=[[Gavin Atkinson]] <gavin@ury.org.uk>
}}

On Tuesday, 17th December 2013, IT Services did some long planned work on some of the filestores that back various campus services, including RentedFS and FlexFS, which back parts of their virtualisation infrastructure. Our understanding of the maintenance was that the systems would be moved from the legacy csrv.york.ac.uk domain to its.york.ac.uk.

However, unbeknownst to us, the filestores were upgraded at the same time during this period, briefly taking them offline. As such, I/O to VMs running in the IT Services Cloud had their requests queued for some time. Our virtual server, uryrrod.york.ac.uk, failed to recover when its virtual had disk was reconnected, as the queued I/O had already timed out so the machine fell over.

This was discovered on the Thursday morning, two days after the incident. The server was rebooted, but it failed to come back up cleanly - we had ourselves a corrupted journal! Several minutes of manual fsck'ing later, the system came back up cleanly and mixclouder operation was restored.

== Proposed Actions ==
* IT Services increase the IO timeout on their VMs for this reason. We should investigate doing the same.
* Xymon does not currently have any monitoring enabled for uryrrod. This should monitor at least system availability.
* Make further use of the Snapshot capabilities of VMWare. Once the server was restored, a snapshot was taken to enable easy restoration in the event of recurrence, but this should possibly become a more regular process.