249
edits
Changes
mLine 1:
Line 1: − {{Incident+ − |brief=We DDoS'd ourselves − |severity=Moderate − |impact=Medium (Manual uploaded non-functional, one canceled show as a result - only a few seconds of dead air though) − |start=04/11/2019 09:00 − |end=05/11/2019 02:00 − |mitigation=Stop the self-DDoS, restart samba, cry − |leader=Marks Polakovs (MP) − |others=Matthew Stratford (MS), Isaac Lowe (IL), Anthony Williams (AW) − }} − =URY is FUCKED (Frustratingly, URY Can't Keep Electronics Deferential)=+ − '''This is still under construction because this literally just happened. When finished, I'll move this page into mainspace.'''+ − __TOC__+ − = Summary = − A combination of a regularly scheduled offsite backup job and some strange behaviour by the studio PCs caused urybackup0 (specifically the music store) to have severely degraded performance. This in turn meant that music uploads to the central library, which normally take less than a minute, would take far longer, sometimes reaching 10 minutes and timing out. One show had to be canceled as a result because the host was not able to get all his music onto the system in time. The incident was not resolved, but mitigated by restarting Samba and the studio PCs.+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + − + − * Effective collaboration by the incident team, especially considering MP didn't really know what he was doing around backup0+ + − === What Did Not Go Well ===+ + + + − * It took a long time to get the incident declared, by which point it had forced one show to cancel+ − * We had limited insight into what was going on with samba/zfs, limited to htop and tcpdumps+ − * Took some time to find the people who knew what they were doing − * Red herrings abound (FFMPEG, RAID issues, hammering, the backup job - any or all of these could have been a red herring) − + + − * After the samba and studioPCs restart, performance brought itself back down to acceptable levels+ + − + − + − * Continue investigating - no real conclusion as to the causes yet − * We Do Not Patch Shit Without Telling Anyone, and anyone who does that will be slapped around a bit with a trout − − − − (all times GMT) − − 9:00 - A regularly scheduled offsite backup job, which should be nightly and incremental... was not either of those things, for reasons unknown to man. It starts copying the entirety of the music store from backup0 to moyles (in IT Services TFTA data-center on East), which starts hammering the ZFS pools on backup0. − − 20:48 - A member reports in Slack that they are unable to upload music. With just over an hour to go to their show, and 12 songs left to upload, they are forced to cancel. − − 21:32 - MP alerts #computing in Slack to a serious incident. '''INCIDENT BEGINS.''' − − 21:35 - MS reports that he had done some package updates on dolby(?) over the weekend, including ffmpeg, which may be causing stuck uploads. (This is eventually shown to be a red herring, but is important to include in this report.) − − ''Between these entries, the computing team are sidetracked by an unrelated issue with MyRadio emails'' − − 22:30 - MS reports that ffmpeg conversions are working manually. − − 22:36 - MP adds debug logging to the MyRadio upload controller, showing that it is hanging on a call to <code>move_uploaded_file</code>, a built-in PHP function. This leads him to point the finger to the music store. − − 22:55 - AW logs on to backup0 to troubleshoot the pools, sees the degraded performance, but notes that zpool reports it is running fine. − − 22:59 - AW notices the stuck(?) offsite backup job − − 23:04 - AW notes that the RAID controllers are chugging as well − − 23:10 - MP and AW decide to restart Samba, but have to wait until midnight as there is a show on air − − 23:13 - AW notices that a Samba log is generating 100KB/s, and a 5MB log size limit means that it is rotating the log every few seconds. He also notes that the ARC cache is very low and that swap is not enabled. Around this time, a very frustrated MP goes for a run to let off steam. − − 23:25 - AW notes that the incremental backup job... is not incremental... and has been sending the entire music dataset to moyles for over 12 hours − − 23:35 - IL logs on and reports that the backup was ''intentionally'' not an incremental one (for reasons that the author (MP) does not yet understand). At 23:38 he kills it. − − ''23:41 - MP gets lost in Alcuin.'' − − 23:49 - MS and IL report that the Windows 10 photos app is seemingly hammering backup0's shares − − 00:05 - MP returns and stops samba. Performance on backup0 immediately jumps tenfold. However, MP screws up and doesn't realise that samba also handles his credentials (from the Windows domain). − − 00:10 - MS logs on with his local account and restarts samba. Performance degrades immediately. MP notes that Xymon does not monitor samba on backup0 at all. − − 00:14 - AW notes that the picture hammering noted earlier is likely one of the studio PCs. − − 00:38-00:43 - MP remotes into wogan and restarts all the studio presenter and guest PCs. At 00:56 he reports that performance has not, in fact, improved substantially. − − 01:06 - MS tests song uploading and notes that, while it is still slow, it has improved dramatically. '''INCIDENT ENDS''' − − 01:26 - MP and the gang head to bed.
11090 moved page Contract to Presenters' Contract: proper title
== Presenters' Contract ==
I, The Undersigned, agree to abide by URY's station rules and regulations as set out below, and the Ofcom Programming Code (accessible at www.ofcom.org.uk). I have fully read and understood these rules and regulations, and I understand that if I break any of the rules or regulations stated by Ofcom or its successor, I will be solely liable for any resulting fines or actions that may be levied against University Radio York.
I also understand that should I break any of URY’s rules or regulations, I am liable to be subject to disciplinary action as decided by URY’s Programme Controller. I will also abide by the YUSU media charter (available from the Student Centre). I have never been convicted of any criminal offence related to radio broadcasting.
In the event of The Undersigned ceasing to be a member or honorary member of URY, this contract will expire before the expiry date at the discretion of the Programme Controller.
=== URY Rules And Regulations ===
# You must have passed a URY training session, paid £7 membership to URY and have permission to broadcast from the Programme Controller in order to be allowed to operate broadcast equipment.
# If you operate broadcast equipment, you are responsible for its correct usage and for upholding URY’s stated rules and regulations at all times.
#
## You are responsible for the content of all broadcasts in which you operate the broadcast equipment, including all technical and audible contributions that you, co-presenters or guests make.
## An individual may appear as a guest presenter on air twice in an academic year without being a URY member. After those two appearances the individual must pay membership, register on MyRadio and become studio trained before they may appear on another URY show.
# You must do all shows that you have signed-up for, unless the Programme Controller states otherwise.
# If you are going to be late for your show, you must contact the Programme Controller as soon as possible. If you believe that you will be unable to do your show, you must inform the Programme Controller at least 48 hours before the occasion in question.
# If you have to cancel your show suddenly due to unforeseen circumstances, you should contact the Programme Controller as soon as possible.
# Anyone who runs broadcast equipment on-air must sign in using SIS (Studio Information System). This person must also log in any guests present in the studio.
# You must broadcast the station name and frequency at least once an hour during your show (e.g. play the News-in Jingle) unless the Programme Controller states otherwise.
# You must broadcast the news every hour. You must also broadcast all or any SBN/IRN adverts that follow provided the feed is available. In the event that there is a technical fault, you must inform the Programme Controller as soon as possible.
#
## You must play all copyrighted music content on your show through BAPS (i.e. you may not use digital music players, including MP3 players, laptops and mobile phones, streaming services such as Spotify or YouTube, or any physical media such as compact discs), except with special permission from the Programme Controller.
## All copyrighted music content that is not played through BAPS (subject to the previous clause) must be logged manually, as instructed by the Programme Controller. BAPS will do this automatically.
# You must follow our PPL music licence laws at all times. The laws are outlined as follows:
## During any 3 hour period:
### You must play no more than 3 tracks from any album (and no more than 2 consecutively).
### You must play no more than 4 tracks from any particular artist or compilation (and no more than 3 consecutively).
### You must only play a track once in any 1-hour period.
### You must not use tracks as beds at any time.
## For a single hour show, you can ensure you do not infringe on PPL laws by abiding by these guidelines:
### Play no more than 1 track from any album by a single artist.
### Play no more than 2 tracks from any particular artist or compilation (and no more than 2 consecutively).
### A Track may only be played once in any 1-hour period.
### You must not use tracks as beds at any time
# You must show impartiality on all political, sectarian, or otherwise contentious issues. If you intend to or predict that you might broadcast potentially controversial output, consult the Programme Controller.
#
## You must not swear on-air, nor express racist, sexist, homophobic, or otherwise offensive opinions in any broadcast. Use of gratuitous swearing is always unacceptable.
## Songs that contain swearing may be played on shows between the hours of 9pm and 5am with advance permission from the Programme Controller. A standard or otherwise pre-agreed disclaimer is to be played at the beginning of each hour of the show. It is unacceptable for presenters and their guests to swear at any time.
# You must not mention that we can be heard off campus on 1350AM.
# You must not broadcast under the influence of drugs or alcohol, nor allow anyone on the show to do so.
# You must play all adverts scheduled during your show. The advert schedule can be found on the notice board in Studio 1, and must be signed when the relevant adverts have been played.
# You must leave the studio clean and tidy. You may not remove any URY property without permission from a URY officer. You may not remove or copy any URY music. Such activities are illegal and will be dealt with accordingly.
# If you are the last person to leave URY, you must pull down all blinds, close all windows and lock the front door. Station key cards must be returned to the porter. Should there be any problem with station key cards or any aspect of URY security, please contact the Programme Controller. In an emergency, contact Security Services on (01904) 434444.
# You must abide by the Ofcom Programming Code at all times.
=== What Went Well ===
=== Disciplinary Actions ===
==== Warnings ====
There are 2 kinds of warnings that the Programme Controller can issue on any occasion, should they see fit. Inappropriate behaviour, misuse of URY property, breach of URY rules and breach of Ofcom rules are the most sensitive occasions in which warnings will be considered.
;Informal Warning
:Any member of URY may be issued with an informal warning in the event that he or she commits an offence that the Programme Controller considers noteworthy, relatively minor, inappropriate or unacceptable. If you receive an informal warning and repeat the infraction warned against, you will receive a formal warning.
;Formal Warning
:Any member of URY may be issued with a formal warning at the discretion of the Programme Controller. The person in question need not have received any informal warnings nor committed previous offences. In the event that he or she breaches URY rules, Ofcom rules or behaves otherwise inappropriately, the Programme Controller may decide to contact the member in question and/or issue a formal warning. If you receive a formal warning and repeat the infraction warned against, your show can be suspended or taken permanently off-air.
==== Suspension ====
The Programme Controller can suspend a presenter on any occasion for as long as they see fit.
=== How We Got Lucky ===
==== Expulsion ====
The Programme Controller can decide to cancel a show or ban a presenter from doing a show on any occasion. Presenters may be banned from using URY facilities and equipment, or from URY as a whole.
==== Appeals Procedure ====
If your show is suspended it will be retained virtually on the system for one week, though you will be unable to present this show. Any appeals against the decision can be made to the Station Management within this time. If no appeal is made, the show will be removed permanently from the schedule. If an appeal is made, the presenter will be invited to meet with the Station Manager, Assistant Station Manager, Programme Controller and Training Coordinator to make their case. All decisions will be made on a case-by-case basis.
= Next Steps =
=== Updates to the Contract ===
The URY Committee reserve the right to update this contract at any time following the procedures laid out in the [[Constitution]] and will notify members when this will take place.
= Timeline =