Difference between pages "HowTo Do an OB From Your Laptop" and "Incident Report: 20191029"

From URY Wiki
(Difference between pages)
Jump to navigation Jump to search
(Created page with "Usually for an OB you'll be using the OB PC in The Rhino. Sometimes, however, The Rhino is totally overkill for your use case - sometimes a single laptop with a mini-mixer and...")
 
 
Line 1: Line 1:
−
Usually for an OB you'll be using the OB PC in The Rhino. Sometimes, however, The Rhino is totally overkill for your use case - sometimes a single laptop with a mini-mixer and one or two microphones is perfectly fine. This guide will tell you how to set up your laptop for it, and how to conduct the OB.
+
{{Incident
 +
  |brief=Jukebox forgets how an audio happens. and suddenly remembers after scaring the Chief Engineer
 +
  |severity=Moderate
 +
  |impact=High (Approx 1hr of dead air)
 +
  |start=29/10/2019 11:02
 +
  |end=29/10/2019 12:26
 +
  |mitigation=Make sure the dead air alarm actually alerts when dead air happens
 +
  |leader=Marks Polakovs (MP)
 +
  |others=Morgan McKay(MM)
 +
}}
  
−
== Things You Will Need ==
+
=Summary=
 +
An (As of yet) not pinpointed issue caused Jukebox to fail to output to TX and Online for over an hour, despite the jukebox service still playing out music on Icecast and through the studio faders. Also the dead air alarm didnt tell anyone, the bastard.
  
−
* A laptop (duh)
+
=Rough Sequence of Events=
−
''Note that this guide is written for Windows PCs - it can probably be adapted to work on macOS or Linux but you're on your own there.''
+
At around 11:02 Jukebox stops playing out audio to streaming and TX. Nobody notices as nobody listens to us, and the dead air detector sends no emails (Not that it could do much other than "We detected a silence, jukebox has been switched to jukebox!")
−
* An audio interface (unless you're using a mixer with a built-in one)
 
−
* A mixer (not always necessary)
 
  
−
== Setting Up The Laptop ==
+
Having finished his morning lectures at around 12:00, MM enters the station and has a quick check of Jukebox via a desks AM feed. MM notices that no sound is coming out. Checking the feed from the AM2, Yamaha, and Icecast show the same. In a nervous panic MM alerts the committee, switching to red and playing out the Jukebox feed via the Pres Pc (Through Icecast, as Jukebox still was playing audio itself) at 12:11. At the time MM didnt know online was off as well as AM so ran into TX and stared at it. The transmitter was sitting there all happy and smug, no reflected power to speak of, just the ominous "Audio break" LED latched on. Having no idea what to check, MM comes back to the studio after a good 10 minute stare and contemplate to find that Jukebox is back on air playing as normal.
  
−
This is a bit of a long section, but you only need to do it once.
+
Turns out during this thoughtful TX stare someone came in and thought "Why the fuck is Red on thats not right!" and switched to Jukebox, which suddenly worked as normal. MM is very confused at this and declares the incident over at 12:26, unsure if the incident was over (But the dead air was gone and MM had shit to do so eh, good enough).
  
−
To do an OB from your laptop, you'll need a couple pieces of software. The first is butt - it stands for "Broadcast Using This Tool" and its acronym makes it hilariously un-Google-able. Track it down and install it.
+
=== What Went Well ===
  
−
You'll also need OpenVPN to access the broadcast servers (because, for obvious reasons, Computing don't like it when anyone on campus can stream to them). Look for the Community Edition, not the new "OpenVPN Connect" app which is hopelessly crap and probably won't work. The Community Edition isn't much better, but at least it works, which is a major upgrade.
+
After the incident was declared, partial recovery only took five minutes and a complete recovery took 20.
  
−
To set it up you'll also need some files from Computing (namely, the profile, CA certificate, and client certificate and key). Ask your nearest friendly neighbourhood compuneer for these. If they stare at you blankly, point them to [https://ury.org.uk/ceedox/computing:wisdom:outside_broadcasts this Ceedox page] (you may not be able to access it, but they should).
+
=== What Did Not Go Well ===
  
−
Once you have everything you need, crack open the OpenVPN app, right-click on its taskbar icon, choose "Import File", and browse to the <code>.ovpn</code> file your friendly neighbourhood compuneer gave you. Now, right-click it again and choose "Connect". If everything went according to plan, you should see a success message and you should still have an internet connection. If it didn't go according to plan, ask your friendly neighbourhood compuneer for help.
+
The dead air was unnoticed for over an hour, due to a failure in the dead air alarm
  
−
Now it's time to crack open butt (ha!) and set up the profile. Click the settings button, and under "Servers" click "New". Fill it in with the following details:
+
=== How We Got Lucky ===
  
−
Name: ''whatever''
+
* The Chief Engineer was in the station when he thought to check AM and noticed the dead air - had it been a presenter who may not have checked, the dead air may have been far longer
−
Type: IceCast
+
* Jukebox/Liquidsoap/whatever broke unbroke itself
−
Address: <code>dolby.ury.york.ac.uk</code>
 
−
Port: 7070
 
−
Password: ''ask your friendly neighbourhood compuneer''
 
−
IceCast mountpoint: <code>OB-line</code> '''(caps matter!)'''
 
−
IceCast user: <code>source</code>
 
  
−
Save it all. While you're at it, head over to the Audio tab and set streaming bitrate to at least 192K.
+
=MP's Investigation=
 +
After running out of his seminar on East, and getting stopped dead in his tracks by First Bus, MP started pulling all the logs he could - namely the HQ audio logs and the Liquidsoap logs. From this he was able to establish the following timeline (all times GMT):
  
−
== Streaming ==
+
=== Timeline ===
  
−
Now you're ready to start the stream, whether just to test it all works, or to actually go live! To fire it up, check you're connected to the VPN, and click the play icon in butt (not the record icon - common mistake). After a few seconds the "Stream Time" should start ticking up. To check that everything has gone to plan, go to [https://ury.org.uk/timelord] and check that the OB box at the bottom has lit up. If it has, you're good to go! When the time comes for your OB, just switch the studio (or, more likely, ask Computing to do it) and you're live!
+
Dead air begins: 11:02:30
 +
 
 +
MM reports dead air in Slack: 12:06
 +
 
 +
Dead air ends with MM's presPC bodge: 12:11:20
 +
 
 +
Jukebox back on air via selector: 12:26:12
 +
 
 +
Total dead air: 1:08:50
 +
 
 +
== Significant realisations ==
 +
 
 +
* Jukebox continued to tracklist even throughout the dead air
 +
* The selector logs showed nothing out of the ordinary - at 00:02:12 output was switched to Jukebox, where it stayed until 12:11:20 when MM switched to Red to end the dead air
 +
* The dead air was heard on both AM and online - it appears clearly in the HQ log
 +
* The dead air alarm... did not alarm about the dead air
 +
* When MM switched output to Red, it worked without any problems
 +
 
 +
== Conclusions ==
 +
 
 +
The most likely culprit is Liquidsoap hitting one of its numerous bugs, getting stuck, and failing to output audio until the switch to Red unstuck'd it. The dead air alarm is part of <code>sel.liq</code>, so it not going off implies that the whole script broke, which would make sense considering Jukebox continued to tracklist (implying that it thought it was on air).
 +
 
 +
Other potential culprits are the Jack connection from Liquidsoap to the sound card, or the sound card itself, or TX itself, but all of these are unlikely:
 +
 
 +
* Jack is relatively stable software
 +
* While Focusrite does not officially have Linux drivers, their cards are actually often more stable on Linux than on Windows
 +
* TX is hard to break, plus, the dead air was on both AM and online, which makes a physical signal path break unlikely
 +
 
 +
We cannot conclusively point the finger at liquidsoap, but it is more likely than not to be the culprit.
 +
 
 +
= Further Work =
 +
 
 +
* Create a more reliable dead air detector
 +
** Done - MP created DEARIE ME (Disappointingly Empty Airwave Remediation and Incident Escalation Management Engine), a dead air alarm listening to the AM feed
 +
* Upgrade liquidsoap in the hopes of fixing bugs
 +
** Will need to wait, as this is a risky process
 +
 
 +
[[Category:Incident Reports]]

Revision as of 00:33, 24 November 2019

Incident Report
Jukebox forgets how an audio happens. and suddenly remembers after scaring the Chief Engineer
Summary
Severity Moderate
Impact High (Approx 1hr of dead air)
Event Start 29/10/2019 11:02
Event End 29/10/2019 12:26
Recurrence Mitigation Make sure the dead air alarm actually alerts when dead air happens
Contacts
Recovery Leader Marks Polakovs (MP)
Other Attendees Morgan McKay(MM)


Summary

An (As of yet) not pinpointed issue caused Jukebox to fail to output to TX and Online for over an hour, despite the jukebox service still playing out music on Icecast and through the studio faders. Also the dead air alarm didnt tell anyone, the bastard.

Rough Sequence of Events

At around 11:02 Jukebox stops playing out audio to streaming and TX. Nobody notices as nobody listens to us, and the dead air detector sends no emails (Not that it could do much other than "We detected a silence, jukebox has been switched to jukebox!")

Having finished his morning lectures at around 12:00, MM enters the station and has a quick check of Jukebox via a desks AM feed. MM notices that no sound is coming out. Checking the feed from the AM2, Yamaha, and Icecast show the same. In a nervous panic MM alerts the committee, switching to red and playing out the Jukebox feed via the Pres Pc (Through Icecast, as Jukebox still was playing audio itself) at 12:11. At the time MM didnt know online was off as well as AM so ran into TX and stared at it. The transmitter was sitting there all happy and smug, no reflected power to speak of, just the ominous "Audio break" LED latched on. Having no idea what to check, MM comes back to the studio after a good 10 minute stare and contemplate to find that Jukebox is back on air playing as normal.

Turns out during this thoughtful TX stare someone came in and thought "Why the fuck is Red on thats not right!" and switched to Jukebox, which suddenly worked as normal. MM is very confused at this and declares the incident over at 12:26, unsure if the incident was over (But the dead air was gone and MM had shit to do so eh, good enough).

What Went Well

After the incident was declared, partial recovery only took five minutes and a complete recovery took 20.

What Did Not Go Well

The dead air was unnoticed for over an hour, due to a failure in the dead air alarm

How We Got Lucky

  • The Chief Engineer was in the station when he thought to check AM and noticed the dead air - had it been a presenter who may not have checked, the dead air may have been far longer
  • Jukebox/Liquidsoap/whatever broke unbroke itself

MP's Investigation

After running out of his seminar on East, and getting stopped dead in his tracks by First Bus, MP started pulling all the logs he could - namely the HQ audio logs and the Liquidsoap logs. From this he was able to establish the following timeline (all times GMT):

Timeline

Dead air begins: 11:02:30

MM reports dead air in Slack: 12:06

Dead air ends with MM's presPC bodge: 12:11:20

Jukebox back on air via selector: 12:26:12

Total dead air: 1:08:50

Significant realisations

  • Jukebox continued to tracklist even throughout the dead air
  • The selector logs showed nothing out of the ordinary - at 00:02:12 output was switched to Jukebox, where it stayed until 12:11:20 when MM switched to Red to end the dead air
  • The dead air was heard on both AM and online - it appears clearly in the HQ log
  • The dead air alarm... did not alarm about the dead air
  • When MM switched output to Red, it worked without any problems

Conclusions

The most likely culprit is Liquidsoap hitting one of its numerous bugs, getting stuck, and failing to output audio until the switch to Red unstuck'd it. The dead air alarm is part of sel.liq, so it not going off implies that the whole script broke, which would make sense considering Jukebox continued to tracklist (implying that it thought it was on air).

Other potential culprits are the Jack connection from Liquidsoap to the sound card, or the sound card itself, or TX itself, but all of these are unlikely:

  • Jack is relatively stable software
  • While Focusrite does not officially have Linux drivers, their cards are actually often more stable on Linux than on Windows
  • TX is hard to break, plus, the dead air was on both AM and online, which makes a physical signal path break unlikely

We cannot conclusively point the finger at liquidsoap, but it is more likely than not to be the culprit.

Further Work

  • Create a more reliable dead air detector
    • Done - MP created DEARIE ME (Disappointingly Empty Airwave Remediation and Incident Escalation Management Engine), a dead air alarm listening to the AM feed
  • Upgrade liquidsoap in the hopes of fixing bugs
    • Will need to wait, as this is a risky process
Retrieved from ‘https://ury.org.uk/wiki/mediawiki/index.php?title=HowTo_Do_an_OB_From_Your_Laptop&oldid=909’