290
edits
Changes
Jump to navigation
Jump to search
Line 2:
Line 2: − + Line 33:
Line 33: − * MyRadio currently runs as a super database user. Line 41:
Line 40: − * MyRadio should not be configured to run as a database super user.
Incident Report: 20140202 (view source)
Revision as of 12:40, 3 February 2014
, 12:40, 3 February 2014no edit summary
|brief=A segmentation fault on our web server caused cascading failures on all URY Computing Services
|brief=A segmentation fault on our web server caused cascading failures on all URY Computing Services
|severity=Critical
|severity=Critical
|impact=High (Complete failure of URY Computing Services for 37+ minutes)
|impact=High (Complete failure of many URY Computing Services for 37+ minutes)
|start=02/02/2014 23:00
|start=02/02/2014 23:00
|end=02/02/2014 23:49
|end=02/02/2014 23:49
* The segmentation fault that causes MyRadio to fail is still under investigation to identify the root cause.
* The segmentation fault that causes MyRadio to fail is still under investigation to identify the root cause.
* The stage of processing that MyRadio appears to be in at the time of segfault means that a database connection is opened, but does not get cleanly closed due to the crash. This left idle connections on the database which over time cause other systems to fail.
* The stage of processing that MyRadio appears to be in at the time of segfault means that a database connection is opened, but does not get cleanly closed due to the crash. This left idle connections on the database which over time cause other systems to fail.
== Work Required ==
== Work Required ==
* Monitoring of system failures of this nature needs to be reviewed and improved, including automated reconnection of IRC bots and email reporting.
* Monitoring of system failures of this nature needs to be reviewed and improved, including automated reconnection of IRC bots and email reporting.
* A behaviour change of URY members is required to ensure that problems are reported through the correct channels. Lloyd Wallis is not a correct channel for reporting problems.
* A behaviour change of URY members is required to ensure that problems are reported through the correct channels. Lloyd Wallis is not a correct channel for reporting problems.
[[Category:Incident Reports]]
[[Category:Incident Reports]]