Incident Report: 20200627: Difference between revisions
No edit summary |
No edit summary |
||
| (2 intermediate revisions by the same user not shown) | |||
| Line 15: | Line 15: | ||
At 13:58:29 the Stores UPS logged “UPS: The output power is turned off.” For, seemingly, no apparent reason. It tried to send us emails to warn us, but, considering it had just turned off power to the email server, that didn’t go well. At this point all of URY was down. '''OUTAGE BEGINS.''' | At 13:58:29 the Stores UPS logged “UPS: The output power is turned off.” For, seemingly, no apparent reason. It tried to send us emails to warn us, but, considering it had just turned off power to the email server, that didn’t go well. At this point all of URY was down. '''OUTAGE BEGINS.''' | ||
: Note: IL asked Danny to check his email (he uses a non-URY email for computing emails) for UPS alerts. None. Although that makes sense, as it had just killed the gateway it tried sending us the emails through. | |||
At 14:01 HS asked in Slack “Is it just me or is the website down?”, and at 14:03 MG confirmed with an @channel that we had dropped off completely. '''INCIDENT BEGINS.''' | At 14:01 HS asked in Slack “Is it just me or is the website down?”, and at 14:03 MG confirmed with an @channel that we had dropped off completely. '''INCIDENT BEGINS.''' | ||
| Line 62: | Line 64: | ||
* Replace freshping with something better - MP done with uptimerobot | * Replace freshping with something better - MP done with uptimerobot | ||
* Get a spare stick of raid RAM | * Get a spare stick of raid RAM | ||
* Figure out what the hell went wrong with the UPS - it shouldn't really be in the business of killing power willy-nilly | |||
[[Category:Incident Reports]] | |||