Changes

'''''The below email was a post-mortem of a Campus North power outage at 01:42am, Sunday, 1st December 2013. Assistant Head of Computing Anthony Williams and Live-Next-To-The-Station-Man Lloyd Wallis were on site to respond. This is how it went.'''''

'''''The below email was a post-mortem of a Campus North power outage at 01:42am, Sunday, 1st December 2013. Assistant Head of Computing Anthony Williams and Live-Next-To-The-Station-Man Lloyd Wallis were on site to respond. This is how it went.'''''

{{Taxobox

{{Incident

| name = Sweetbay Magnolia

  |brief=A displeased 12kV power cable shorted, taking out parts of the local power grid. URY's servers were unhappy with this series of events, so massaging was needed.

| image = Sweetbay1082.jpg

  |severity=Critical

| regnum = [[Plant]]ae

  |impact=Medium (main outage at night, residual issues were for minor services)

| divisio = [[Angiospermae]]

  |start=01/12/2013 01:42

| unranked_classis = [[Magnoliid]]s

  |end=01/12/2013 07:11

| ordo = [[Magnoliales]]

  |mitigation=Steps taken, but further action needed.

| familia = [[Magnoliaceae]]

  |leader=[[Anthony Williams]] <anthony@ury.org.uk>

| genus = ''[[Magnolia]]''

  |others=[[Lloyd Wallis]] <lpw@ury.org.uk>

| species = '''''M. virginiana'''''

| binomial_authority = [[Carolus Linnaeus|L.]]

}}

}}

== Event Start: 01:42 ==

== Event Start: 01:42 ==

At first, we didn’t think there was a substantial issue - the power in James college was knocked out for about a second, but their UPS coped so we assumed ours would too. Some amusement was poked at YSTV for neglecting to put their web/db server on said UPS. Once we confirmed our website was inaccessible, we made our way over to investigate, but assumed bits of Vanbrugh network were just rebooting.

At first, we didn’t think there was a substantial issue - the power in James college was knocked out for about a second, but their UPS coped so we assumed ours would too. Some amusement was poked at YSTV for neglecting to put their web/db server on said UPS. Once we confirmed our website was inaccessible, we made our way over to investigate, but assumed v1rs0 was just rebooting.

We reached URY in a state of running on UPS power, so started a graceful shutdown of all servers using ACPI (i.e. the power button). We left logger2 running at this stage because of the uptime, and logger2 is a hardy-ish device that’s seen a lot anyway. Everything but that was powered down by 01:52.

We reached URY in a state of running on UPS power, so started a graceful shutdown of all servers using ACPI (i.e. the power button). We left logger2 running at this stage because of the uptime, and logger2 is a hardy-ish device that’s seen a lot anyway. Everything but that was powered down by 01:52.

== Problem #1: Stores Distribution Board ==

== Problem #1: Stores Distribution Board ==

The B16 breaker that estates told me they would upgrade to C16 last year tripped from the sudden desire of a large quantity of batteries to charge up. Turning the power back on caused the breaker to trip again.

The B16 breaker that estates told me they would upgrade to C16 last year tripped from the sudden desire of a large quantity of batteries to charge up. Turning the power back on caused a happy bright flash, some dimmed lights, then the breaker re-tripping.

Luckily, switching off the compressors and distribution amplifier for the audio path was sufficient, and the UPS then began to charge at 05:44, and enabled server power at 05:49.

Luckily, switching off the compressors and distribution amplifier for the audio path was sufficient, and the UPS then began to charge at 05:44, and enabled server power at 05:49.

== Problem #2: logger2 ==

== Problem #2: logger2 ==

As is traditional, logger2 decided not to play ball when we powered it up. It made strange noises, so we left it off overnight. Confirmation that logger1 and uryblue were operating left us happy to power the transmitter on by 06:30. It did start up later in the day, so was probably just being fussy because it was a bit cold.

As is traditional, logger2 decided not to play ball when we powered it up. It made strange noises and smelled a bit, so we left it off overnight. Confirmation that logger1 and uryblue were operating left us happy to power the transmitter on by 06:30. It did start up later in the day, so was probably just being fussy because it was a bit cold.

== Problem #3: IT Services Systems ==

== Problem #3: IT Services Systems ==