249
edits
Changes
Jump to navigation
Jump to search
Incident Report: 20230109 (view source)
Revision as of 17:48, 9 January 2023
, 17:48, 9 January 2023Created page with "{{Incident |brief=ITS replaced some networking gear and broke some routes. Sad panda. |severity=High |impact=Medium (anything URY-related unavailable for around ~8 hours..."
{{Incident
|brief=ITS replaced some networking gear and broke some routes. Sad panda.
|severity=High
|impact=Medium (anything URY-related unavailable for around ~8 hours)
|start=2023-01-09 09:30
|end=2023-01-09 17:19
|leader=Joseph Sisson (JS)
|others=Michael Grace (MG), Marks Polakovs (MP)
}}
== Chronicle of Events ==
''(All times GMT)''
Early in the morning of the first day of the first week of term, IT Services (as we later found out) replaced some networking equipment in the Vanbrugh area, and may have incorrectly set some static routes. The first we found out about this is at 9:30, when all of our monitoring pinged that URY had dropped off the face of the internet.
Much scrambling and some reboots later, we narrowed down the state of the network to:
* Anything with a 144.32.64.160/27 IP (so all of URY) could send packets ''out'' of URY, but no packets would make it ''in''.
:* With the exception of uryfw0 which has a separate IP (144.32.109.64).
:* Traceroutes showed the packets getting into a loop somewhere in Berrick Saul.
* This meant that the website and online streams were down.
* We could carry on broadcasting on AM, but FM failed
:* MP later diagnosed the FM issue to be a dependency on audio.ury.org.uk for the backup feed, and restored FM at 14:39.
Cue an ITS ticket, and at around 17:14 service was restored.
== Lessons Learned ==
TODO
[[Category:Incident Reports]]
|brief=ITS replaced some networking gear and broke some routes. Sad panda.
|severity=High
|impact=Medium (anything URY-related unavailable for around ~8 hours)
|start=2023-01-09 09:30
|end=2023-01-09 17:19
|leader=Joseph Sisson (JS)
|others=Michael Grace (MG), Marks Polakovs (MP)
}}
== Chronicle of Events ==
''(All times GMT)''
Early in the morning of the first day of the first week of term, IT Services (as we later found out) replaced some networking equipment in the Vanbrugh area, and may have incorrectly set some static routes. The first we found out about this is at 9:30, when all of our monitoring pinged that URY had dropped off the face of the internet.
Much scrambling and some reboots later, we narrowed down the state of the network to:
* Anything with a 144.32.64.160/27 IP (so all of URY) could send packets ''out'' of URY, but no packets would make it ''in''.
:* With the exception of uryfw0 which has a separate IP (144.32.109.64).
:* Traceroutes showed the packets getting into a loop somewhere in Berrick Saul.
* This meant that the website and online streams were down.
* We could carry on broadcasting on AM, but FM failed
:* MP later diagnosed the FM issue to be a dependency on audio.ury.org.uk for the backup feed, and restored FM at 14:39.
Cue an ITS ticket, and at around 17:14 service was restored.
== Lessons Learned ==
TODO
[[Category:Incident Reports]]