Incident Report: 20230109: Difference between revisions
No edit summary |
|||
| (2 intermediate revisions by 2 users not shown) | |||
| Line 2: | Line 2: | ||
|brief=ITS replaced some networking gear and broke some routes. Sad panda. | |brief=ITS replaced some networking gear and broke some routes. Sad panda. | ||
|severity=High | |severity=High | ||
|impact= | |impact=High (anything URY-related unavailable for around ~8 hours) | ||
|start=2023-01-09 09:30 | |start=2023-01-09 09:30 | ||
|end=2023-01-09 17:19 | |end=2023-01-09 17:19 | ||
| Line 24: | Line 24: | ||
::* You really don't want to know how... no, seriously, it's horrible... | ::* You really don't want to know how... no, seriously, it's horrible... | ||
An ITS ticket was filed at 12:12, and at around 17:14 service was restored. | |||
== Lessons Learned == | == Lessons Learned == | ||
* We need to be more happy to assume, if we haven't touched the network and something's happened, it could be an ITS issue. Look at things like traceroutes earlier. | |||
* ITS don't really know about us - like they'd have just assumed uryfw0 is just 144.32.109.64 and not a gateway for all of 144.32.64.160/27, so be happy to remind them of this. | |||
* Phone ITS earlier if it's a big problem. | |||
* Don't get distracted by things that are merely the result of the problem - i.e. our DNS is often external to URY (i.e. wogan or ITS nameservers), so it can't resolve DNS. But it doesn't mean the key problem is DNS - the key problem is that no traffic reached us. What didn't help was the confusion about guest PC being able to access the internet - this is because it has a web proxy (because IRN). | |||
[[Category:Incident Reports]] | [[Category:Incident Reports]] | ||