Difference between pages "Incident Report: 20140202" and "Student Radio Awards"

From URY Wiki
(Difference between pages)
Jump to navigation Jump to search
 
 
Line 1: Line 1:
−
{{Incident
+
The Student Radio Awards is an annual event hosted by the [[Student Radio Association]]. Every year, URY enters itself for several different awards, sometimes winning.
−
  |brief=A segmentation fault on our web server caused cascading failures on all URY Computing Services
 
−
  |severity=Critical
 
−
  |impact=High (Complete failure of many URY Computing Services for 37+ minutes)
 
−
  |start=02/02/2014 23:00
 
−
  |end=02/02/2014 23:49
 
−
  |mitigation=Multiple actions required to prevent recurrence.
 
−
  |leader=[[Anthony Williams]] <anthony@ury.org.uk>
 
−
  |others=[[Andrew Durant]] <aj@ury.org.uk>
 
−
}}
 
  
−
An at-the-time regular failure of PHP5 APC package caused many of URY's Computing Services to be unavailable for a period of time on Sunday 2nd of February.
+
== Trivia ==
 +
* URY regularly loses out to Best Station because [[URN]] claim to be better. The majority of the committee justify this with Nottingham being significantly larger, URN having a lot more money, and a part time staff member to run it.
 +
* URY's very own [[Coco Cole]] was one of the Voices of the Awards in 2012.  A small amount of gratification was to be had when she eventually had to read out URY's Best Technical Achievement submission name.
 +
* In 2013, URY was joint first in Most Nominations with Fly FM, and Most Awarded with Fly FM and URN.
 +
* Harry Whittaker was the first person in living Student Radio Association history that has received 3 nominations in a single year.
  
−
Due to a system issue under active investigation, PHP's APC module, which underpins the [[MyRadio]] caching system, the ability to service some PHP requests currently regularly fails due to a segmentation fault. This issue breaks member-facing services, not public facing, and due to monitoring systems is usually restored within 5 minutes of failing. Investigation currently involves increasing levels of debug compile options being enabled to locate the root cause of the segmentation fault. It was also at the time not clear whether or not this was related to another issue, where some Apache modules would fail after a log rotate due to not updating file pointers correctly.
+
== Award History ==
  
−
From approximately 18:25 until 18:35 on 2nd February, The University of York's connection to Janet briefly failed for reasons currently unknown (src: "''gavinatkinson doesn't actually know what happened with the offsite glitch''"). Several monitoring IRC robots that often provide useful information, including ''xymon-bsod'', our service monitoring bot, dropped off from Freenode and failed to reconnect once access was restored.
+
{| class="wikitable"
 +
|+[[2014 SRAs|2014]]
 +
|-
 +
! Nominations
 +
! Winner
 +
|-
 +
| Best Journalistic Programming
 +
| SCRATCH RADIO (Gold), '''River Safety: York’s Rising Problem - URY (Silver)''', URN (Bronze)
 +
|-
 +
| Best Specialist Music Programming
 +
| ???: Dance Revolution (Gold), LSRFM: Mim Shaikh Presents The Shaikh World Show (Silver), '''URY: Coco Electro (Bronze)'''
 +
|}
  
−
Due to xymon-bsod being offline, none of us were aware of the failure of myradio_daemon, one of our backend services, and the increasing load averages on our web server, which would have notified us when the initial segmentation fault appeared at 23:00 and load gradually increased.
+
{| class="wikitable"
 +
|+[[2013 SRAs|2013]]
 +
|-
 +
! Nominations
 +
! Winner
 +
|-
 +
| Best Technical Achievement (Talkback)
 +
| 1449AM URYB's UK Radioplayer Console & Unified Station Management Application (Gold), URY: Show Planner (Silver), Shock Radio: New Broadcast Architecture (Bronze)
 +
|-
 +
| [[Best Technical Achievement (Show Planner)]]
 +
| 1449AM URYB's UK Radioplayer Console & Unified Station Management Application (Gold), '''URY: Show Planner (Silver)''', Shock Radio: New Broadcast Architecture (Bronze)
 +
|-
 +
| Best Specialist Music Programming (Bos Tones)
 +
| Will Metcalf (Gold), Monster FM: Roya's Celtic Show (Silver), Fly FM: The Urban Swift with Temi Jonah (Bronze)
 +
|-
 +
| Best Specialist Music Programming (Ellie Wright's World Music)
 +
| Will Metcalf (Gold), Monster FM: Roya's Celtic Show (Silver), Fly FM: The Urban Swift with Temi Jonah (Bronze)
 +
|-
 +
| Best Entertainment Show (The Harry Whittaker Show)
 +
| The Morning Show with Giles Gear (Gold), Wake and Bake with Laurence and Gareth (Silver), '''The Harry Whittaker Show - ((URY)) (Bronze)'''
 +
|-
 +
| Best Male (Harry Whittaker)
 +
| Pure FM: Glen Scott (Gold), Radio LaB: Danny Fullbrook (Silver), Spark FM: Jonny Chambers (Bronze)
 +
|-
 +
| Best Newcomer (Harry Whittaker)
 +
| Ben Sheppard (Gold), '''Harry Whittaker - ((URY)) (Silver)''', Dom Stirling (Bronze)
 +
|-
 +
| [[Best Speech Programming (The New World Order: Part 2: 1651)]]
 +
| '''URY: The New World Order: Part 2: 1651 (Gold)''', URY: Trimble (Silver), A Seaside Town (Bronze)
 +
|-
 +
| [[Best Speech Programming (Trimble)]]
 +
| URY: The New World Order: Part 2: 1651 (Gold), '''URY: Trimble (Silver)''', A Seaside Town (Bronze)
 +
|-
 +
|}
  
−
At 23:12, Pyramid, the framework that our website is based on, started to report failures with some of its backend requests which are based on the MyRadio API or certain parts of the database. At this point, xymon-bsod would have likely picked up HTTP response alerts too.
+
{| class="wikitable"
 +
|+[[2012 SRAs|2012]]
 +
|-
 +
! Nominations
 +
! Winner
 +
|-
 +
| [http://www.studentradioawards.co.uk/content/Awards/2012/Winners/pdf_w3s1i1.pdf Best Station]
 +
| URN (Gold), '''URY (Silver)''', 107 Spark FM (Bronze)
 +
|-
 +
| Best Specialist Music Programming
 +
| ???: Dance Revolution (Gold), LSRFM: Mim Shaikh Presents The Shaikh World Show (Silver), '''URY: Coco Electro (Bronze)'''
 +
|}
  
−
At 23:27, Pyramid started completely timing out processing new requests. At this point, Apache's wsgi handlers also started queueing up waiting for responses from Pyramid, retrying several times before themselves timing out.
+
[[Category: Awards]] [[Category: History]]
−
 
 
−
At 23:28, the first user report was received, via a private Facebook message to [[Lloyd Wallis]]. Lloyd was unavailable at the time and the message went unnoticed. Four more users reported the failure in this way over the next few minutes.
 
−
 
 
−
At 23:33, our PostgreSQL database reported that service had reached the maximum number of allowed active connections and stopped serving new requests. At this time, services such as our Jukebox Scheduler, Tracklisting and BAPS all stopped working, significantly hampering broadcast capabilities.
 
−
 
 
−
At 23:34 the scale of the outage finally led to phone calls and other notifications drawing attention of the issue to the actual URY Computing Team. At this point the load on our web server was over 90. The database is diagnosed as the cause of many issues and a restart was requested at 23:38, but was delayed waiting for the idle connections to terminate. The restart of this service was completed at 23:42 and services required for broadcast recovered.
 
−
 
 
−
Apache was then stopped at 23:48 to allow the web server to calm down and recover to a regular load average. It was restarted at 23:49, at which point all URY Computing Services were once again operating normally.
 
−
 
 
−
== Causes ==
 
−
* The segmentation fault that causes MyRadio to fail is still under investigation to identify the root cause.
 
−
* The stage of processing that MyRadio appears to be in at the time of segfault means that a database connection is opened, but does not get cleanly closed due to the crash. This left idle connections on the database which over time cause other systems to fail.
 
−
 
 
−
== Work Required ==
 
−
* A review of the MyRadio code is required to ensure that any steps possible are taken to cleanly terminate database connections after a failure.
 
−
* A review of PostgreSQL is required to see if it can have reduced idle timeouts or better handling of broken connections.
 
−
* The root cause of the APC Segmentation Faults needs to be discovered and rectified, replacing APC with another solution if necessary.
 
−
* Monitoring of system failures of this nature needs to be reviewed and improved, including automated reconnection of IRC bots and email reporting.
 
−
* A behaviour change of URY members is required to ensure that problems are reported through the correct channels. Lloyd Wallis is not a correct channel for reporting problems.
 
−
 
 
−
[[Category:Incident Reports]]
 

Revision as of 10:21, 1 October 2016

The Student Radio Awards is an annual event hosted by the Student Radio Association. Every year, URY enters itself for several different awards, sometimes winning.

Trivia

  • URY regularly loses out to Best Station because URN claim to be better. The majority of the committee justify this with Nottingham being significantly larger, URN having a lot more money, and a part time staff member to run it.
  • URY's very own Coco Cole was one of the Voices of the Awards in 2012. A small amount of gratification was to be had when she eventually had to read out URY's Best Technical Achievement submission name.
  • In 2013, URY was joint first in Most Nominations with Fly FM, and Most Awarded with Fly FM and URN.
  • Harry Whittaker was the first person in living Student Radio Association history that has received 3 nominations in a single year.

Award History

2014
Nominations Winner
Best Journalistic Programming SCRATCH RADIO (Gold), River Safety: York’s Rising Problem - URY (Silver), URN (Bronze)
Best Specialist Music Programming ???: Dance Revolution (Gold), LSRFM: Mim Shaikh Presents The Shaikh World Show (Silver), URY: Coco Electro (Bronze)
2013
Nominations Winner
Best Technical Achievement (Talkback) 1449AM URYB's UK Radioplayer Console & Unified Station Management Application (Gold), URY: Show Planner (Silver), Shock Radio: New Broadcast Architecture (Bronze)
Best Technical Achievement (Show Planner) 1449AM URYB's UK Radioplayer Console & Unified Station Management Application (Gold), URY: Show Planner (Silver), Shock Radio: New Broadcast Architecture (Bronze)
Best Specialist Music Programming (Bos Tones) Will Metcalf (Gold), Monster FM: Roya's Celtic Show (Silver), Fly FM: The Urban Swift with Temi Jonah (Bronze)
Best Specialist Music Programming (Ellie Wright's World Music) Will Metcalf (Gold), Monster FM: Roya's Celtic Show (Silver), Fly FM: The Urban Swift with Temi Jonah (Bronze)
Best Entertainment Show (The Harry Whittaker Show) The Morning Show with Giles Gear (Gold), Wake and Bake with Laurence and Gareth (Silver), The Harry Whittaker Show - ((URY)) (Bronze)
Best Male (Harry Whittaker) Pure FM: Glen Scott (Gold), Radio LaB: Danny Fullbrook (Silver), Spark FM: Jonny Chambers (Bronze)
Best Newcomer (Harry Whittaker) Ben Sheppard (Gold), Harry Whittaker - ((URY)) (Silver), Dom Stirling (Bronze)
Best Speech Programming (The New World Order: Part 2: 1651) URY: The New World Order: Part 2: 1651 (Gold), URY: Trimble (Silver), A Seaside Town (Bronze)
Best Speech Programming (Trimble) URY: The New World Order: Part 2: 1651 (Gold), URY: Trimble (Silver), A Seaside Town (Bronze)
2012
Nominations Winner
Best Station URN (Gold), URY (Silver), 107 Spark FM (Bronze)
Best Specialist Music Programming ???: Dance Revolution (Gold), LSRFM: Mim Shaikh Presents The Shaikh World Show (Silver), URY: Coco Electro (Bronze)
Retrieved from ‘https://ury.org.uk/wiki/mediawiki/index.php?title=Incident_Report:_20140202&oldid=624’