Incident Report: 20191104: Difference between revisions

Line 59:

22:36 - MP adds debug logging to the MyRadio upload controller, showing that it is hanging on a call to <code>move_uploaded_file</code>, a built-in PHP function. This leads him to point the finger to the music store.

22:55 - AW logs on to backup0 to troubleshoot the pools, sees the degraded performance, but notes that zpool reports it is running fine.

22:55 - AW logs on to backup0 to troubleshoot the pools, sees the degraded performance, but notes that zpool reports it is running fine. AW notices the stuck(?) offsite backup job, notes that the RAID controllers are chugging as well

~~22:59 -~~ AW notices the stuck(?) offsite backup job

~~23:04 - AW~~ notes that the RAID controllers are chugging as well

23:10 - MP and AW decide to restart Samba, but have to wait until midnight as there is a show on air

23:13 - AW notices that a Samba log is generating 100KB/s, and a 5MB log size limit means that it is rotating the log every few seconds. He also notes that the ARC cache is very low and that swap is not enabled. ~~Around this time, a very frustrated MP goes for a run to let off steam.~~

23:13-23:25 - AW notices that a Samba log is generating 100KB/s, and a 5MB log size limit means that it is rotating the log every few seconds. He also notes that the ARC cache is very low and that swap is not enabled. AW notes that the incremental backup job... is not incremental... and has been sending the entire music dataset to moyles for over 12 hours. Around this time, a very frustrated MP goes for a run to let off steam.

~~23:25 -~~ AW notes that the incremental backup job... is not incremental... and has been sending the entire music dataset to moyles for over 12 hours

23:35 - IL logs on and reports that the backup was ''intentionally'' not an incremental one (for reasons that the author (MP) does not yet understand). At 23:38 he kills it.

@@ Line 59: / Line 59: @@
 :36 - MP adds debug logging to the MyRadio upload controller, showing that it is hanging on a call to <code>move_uploaded_file</code>, a built-in PHP function. This leads him to point the finger to the music store.
-:55 - AW logs on to backup0 to troubleshoot the pools, sees the degraded performance, but notes that zpool reports it is running fine.
+:55 - AW logs on to backup0 to troubleshoot the pools, sees the degraded performance, but notes that zpool reports it is running fine. AW notices the stuck(?) offsite backup job, notes that the RAID controllers are chugging as well
-:59 - AW notices the stuck(?) offsite backup job
-:04 - AW notes that the RAID controllers are chugging as well
 :10 - MP and AW decide to restart Samba, but have to wait until midnight as there is a show on air
-:13 - AW notices that a Samba log is generating 100KB/s, and a 5MB log size limit means that it is rotating the log every few seconds. He also notes that the ARC cache is very low and that swap is not enabled. Around this time, a very frustrated MP goes for a run to let off steam.
+:13-23:25 - AW notices that a Samba log is generating 100KB/s, and a 5MB log size limit means that it is rotating the log every few seconds. He also notes that the ARC cache is very low and that swap is not enabled. AW notes that the incremental backup job... is not incremental... and has been sending the entire music dataset to moyles for over 12 hours. Around this time, a very frustrated MP goes for a run to let off steam.
-:25 - AW notes that the incremental backup job... is not incremental... and has been sending the entire music dataset to moyles for over 12 hours
 :35 - IL logs on and reports that the backup was ''intentionally'' not an incremental one (for reasons that the author (MP) does not yet understand). At 23:38 he kills it.