249
edits
Changes
Jump to navigation
Jump to search
Line 59:
Line 59: − + − − 22:59 - AW notices the stuck(?) offsite backup job − − 23:04 - AW notes that the RAID controllers are chugging as well − + − − 23:25 - AW notes that the incremental backup job... is not incremental... and has been sending the entire music dataset to moyles for over 12 hours
Incident Report: 20191104 (view source)
Revision as of 15:49, 5 November 2019
, 15:49, 5 November 2019→Timeline
22:36 - MP adds debug logging to the MyRadio upload controller, showing that it is hanging on a call to <code>move_uploaded_file</code>, a built-in PHP function. This leads him to point the finger to the music store.
22:36 - MP adds debug logging to the MyRadio upload controller, showing that it is hanging on a call to <code>move_uploaded_file</code>, a built-in PHP function. This leads him to point the finger to the music store.
22:55 - AW logs on to backup0 to troubleshoot the pools, sees the degraded performance, but notes that zpool reports it is running fine.
22:55 - AW logs on to backup0 to troubleshoot the pools, sees the degraded performance, but notes that zpool reports it is running fine. AW notices the stuck(?) offsite backup job, notes that the RAID controllers are chugging as well
23:10 - MP and AW decide to restart Samba, but have to wait until midnight as there is a show on air
23:10 - MP and AW decide to restart Samba, but have to wait until midnight as there is a show on air
23:13 - AW notices that a Samba log is generating 100KB/s, and a 5MB log size limit means that it is rotating the log every few seconds. He also notes that the ARC cache is very low and that swap is not enabled. Around this time, a very frustrated MP goes for a run to let off steam.
23:13-23:25 - AW notices that a Samba log is generating 100KB/s, and a 5MB log size limit means that it is rotating the log every few seconds. He also notes that the ARC cache is very low and that swap is not enabled. AW notes that the incremental backup job... is not incremental... and has been sending the entire music dataset to moyles for over 12 hours. Around this time, a very frustrated MP goes for a run to let off steam.
23:35 - IL logs on and reports that the backup was ''intentionally'' not an incremental one (for reasons that the author (MP) does not yet understand). At 23:38 he kills it.
23:35 - IL logs on and reports that the backup was ''intentionally'' not an incremental one (for reasons that the author (MP) does not yet understand). At 23:38 he kills it.