Difference between pages "Incident Report: 20140116" and "In all their"

From URY Wiki
(Difference between pages)
Jump to navigation Jump to search
(Created page with "{{Incident |brief=A user error resulted in a total loss of the source code for the iTones Liquidsoap service. |severity=Critical |impact=Medium (Several minutes of silen...")
 
 
Line 1: Line 1:
−
{{Incident
+
[[Category:In-jokes]]
−
  |brief=A user error resulted in a total loss of the source code for the iTones Liquidsoap service.
+
[[Category:Computing]]
−
  |severity=Critical
+
'''(In all their...)''' is a [[Computing]] Team [[in-joke]], originating from the early-2000s "URY International News Headquarters (European Branch) Provision of Information Resources cupboard" cooling documentation.
−
  |impact=Medium (Several minutes of silence on output, Campus Playout unavailable for several hours)
 
−
  |start=16/01/2014 09:22
 
−
  |end=16/01/2014 10:49
 
−
  |mitigation=Multiple actions taken to prevent recurrence.
 
−
  |leader=[[Lloyd Wallis]] <lpw@ury.org.uk>
 
−
  |others=[[Andrew Durant]] <aj@ury.org.uk>, [[Stephen Clarke]] <sc1152@ury.org.uk>
 
−
}}
 
  
−
During a routine procedure of taking an existing production configuration directory and creating a git repository, a failure of [[Lloyd Wallis]] to invoke `git commit` before `git pull` resulted in a file with the same name in the remote repository overwriting the local /etc/liquidsoap/jukebox.liq file that runs our production jukebox system.
+
The document[https://docs.google.com/document/d/1azS7jXB6AHYz8niVD6kfz2h1Mhp_1bnufSsLs3obXyU/edit] is a comprehensive user manual for the cooling apparatus installed into the then-server-cupboard. Whilst these days we refer to the location as [[The Hub]], since our servers are now in [[Stores]], the cooling system is still somewhat in place (with the exception of Estates completely replacing it in Summer 2013). The documentation is, to this day, completely valid, making it the longest-standing single piece of documentation that is still accurate to this day.
  
−
This issue was then compounded in two ways - since the code was in the process of being added to the repository, it was not yet under version control, and the [[jukebox]] server was not backed up. This meant there was no way to restore the file from a recent state.
+
The full guide include a full rationale of the system, during which whenever the Computing Team (in all their wisdom) are mentioned, the phrase "in all their wisdom" is added in parentheses. Likewise with the (then Technical) Engineering Team (in all their glory), News Team (in all their usefulness) and Speech Team (in all their... erm... something... not sure what... ask Leo). Oh, and apprehension (whoever she was).
−
 
 
−
During this incident, the file was somewhat reconstructed using an old version of the code from June 2012, and several pastebin snippets found in IRC logs. As far as it is known, the system is now operational back with its original functionality, however for reasons unknown running jukebox as a service no longer works.
 
−
 
 
−
This incident brought an important issue to light: the lack of backups of our jukebox server. This issue has now been rectified, and urybackup0:/pool0/jukebox now has daily backups using the same script as our other servers. The Liquidsoap script is now in a GitHub repository under version control, which means the issue will not recur as long as the version control process is adhered to.
 
−
 
 
−
[[Category:Incident Reports]]
 

Latest revision as of 23:28, 29 December 2013

(In all their...) is a Computing Team in-joke, originating from the early-2000s "URY International News Headquarters (European Branch) Provision of Information Resources cupboard" cooling documentation.

The document[1] is a comprehensive user manual for the cooling apparatus installed into the then-server-cupboard. Whilst these days we refer to the location as The Hub, since our servers are now in Stores, the cooling system is still somewhat in place (with the exception of Estates completely replacing it in Summer 2013). The documentation is, to this day, completely valid, making it the longest-standing single piece of documentation that is still accurate to this day.

The full guide include a full rationale of the system, during which whenever the Computing Team (in all their wisdom) are mentioned, the phrase "in all their wisdom" is added in parentheses. Likewise with the (then Technical) Engineering Team (in all their glory), News Team (in all their usefulness) and Speech Team (in all their... erm... something... not sure what... ask Leo). Oh, and apprehension (whoever she was).

Retrieved from ‘https://ury.org.uk/wiki/mediawiki/index.php?title=Incident_Report:_20140116&oldid=405’