Cold-Starting URY Systems

PLEASE EVACUATE THE BUILDING SAFELY AND UPDATE SOCIAL MEDIA BEFORE ATTEMPTING TO RESTORE SERVICE

This is a ops-critical document. A printed copy is available in the Server Cupboard and should be updated whenever this online version is.

So, you've done a full shutdown. Or, there was a power cut or zombie apocalypse that interrupted the ability of our physical servers to operate. The good news is that you now thing you're ready to turn things back on.

Remember - during any power failure it is advised to immediately switch off the transmitter. See Shutting Down URY In A Hurry.

Before You Start - Is It Safe Checklist

• Is power back on yet? Has it been stable for a few minutes?
• Has the Head of Computing or Station Manager given consent to restoring service?
• Does information from Estates, YUSU or other relevant sources suggest all is okay?
• If you are going to re-start AM Transmission, have you got consent from the Chief Engineer to power on the transmitter audio path?
• Do you have at least two technical team members on site (ideally one engineer)?

Great - lets give this a go.

Network Infrastructure

We got all these servers right? Well they ain't no good until there's a network. You do this stage in The Hub.

• urysw4 should come up on its own, as it has PoE [???] - check the injector is on
• Power on urysw3 (The HP ProCurve 2626 [The top one])
• Power on urysw1 (The Netgear GS748T [The bottom one])

Have both of these switched on? Is urysw3 blinking happily? Is urysw1 looking like nothing much is happening? Perfect.

Stores Power

Our power supply gets a little upset very easily. If this outage was caused by a power cut, chances are you'll want to use this section to restore power to the Server Cupboard circuit.

• Ensure the Transmitter is switched OFF (if it wasn't already, you aren't very good at reading this guide)
• Ensure the AM compressor is switched OFF
• Ensure the output compressor is switched OFF
• Ensure the AM Receiver is switched OFF

If you don't do this, then the initial inrush of power from turning on a rack full of equipment will overload the B16 breaker. It will make a noise as things try to turn on, give you a little fright, then promptly trip again. Possibly with a bright flash of light for dramatic effect.

• [Two People Required] Turn on the breaker labelled "metal clad sockets"
• If it did not do so automatically, switch on the UPS
• Turn the AM Receiver back on and ensure it is still tuned to 1350AM
• Turn the lower output compressor back ON (third box from the top)

The UPS will now begin charging. If it is fully depleted, it will be around 5 minutes before it will enable output power to the servers. Depending on the BIOS configuration, some may then start to automatically boot. Avoid this, if at all possible.

Critical Servers

We identify critical servers as those that enable us to broadcast on AM. URY Policy states that we must have two operating loggers before restoration of AM service. You'll also want the jukebox to play some noise.

• Power on uryfw0
• No really, turn on uryfw0. Are you sure it’s on yet? Since this is the gateway for all URY systems, other servers may have trouble bringing up interfaces if it is not up.
• Power on uryred
• Power on uryblue
• On both the loggers, run sudo service loggerng status and start it if it fails to auto-start
• Power on dolby
• If you don't get audio, run cd /usr/local/etc/liquidsoap/scripts && sudo ./startAudio.sh

The station should start outputting the world's most annoying loop, featuring a happy instrumental tune and someone telling you that we're off air right now. We're most definitely not.

• If it tries to play a jingle, this might fail spectacularly and go to a loop of Monty Python's Intermission, featuring Alex Boyall giving a grammatically incorrect technical difficulties message.

AM Broadcast

You can't start this section until at least 5-10 minutes after following Critical Servers. It might be worth skipping to Core Computing Services and coming back in a bit. It also requires permission from the Chief Engineer.

• Two persons must separately check and verify that at least two logger services are operating correctly and have recorded the last 5 minutes of station output (or in the case of an AM logger, several minutes of static).
• Switch on the AM compressor (second box from the top)
• Switch on the AM Transmitter

If any of the following tests fail, switch off the transmitter immediately and follow Transmitter Troubleshooting

• Are all four power indicators on the left lit?
• Is the fan on the rear of the unit spinning?
• Is the forward power meter registering approximately 20W?
• Is the reflected power meter registering a negligible level (2-3W is okay, slightly more if it's damp outside)?
• Is the AM receiver showing 3 signal bars on its display?

• After 5 minutes, two people should then check the AM loggers.

Core Computing Services

Core Computing Services are defined as those which must be operational for URY to broadcast anything other than iTones (or, at this point, Intermission).

• Power on urybackup0 and urysteve
• Wait for urysteve to finish booting
• Power on ury
• Ensure selector is powered in The Hub
• Fade up jukebox in Studio Red, then switch to S1 then back to S3
  • This ensures selector state is up to date
  • Jukebox should now be playing actual music. You might need to restart it if it's stuck on techlude - on Dolby, cd /usr/local/etc/liquidsoap/scripts && sudo systemctl stop ury-jack && sudo ./startAudio.sh && sel 8 && sel 3

That's the absolute basics. Now verify the following are accessible and functioning:

• http://ury.org.uk/
• https://ury.org.uk/myradio/
• https://ury.org.uk/roundcube/ (including sending test emails both internally and externally)
  • It is possible that mta.york.ac.uk is not yet back online, or that it is but refuses to route mail. You can test this with good ol' telnet mta.york.ac.uk 25
• http://ury.org.uk/live/
• live-high, live-mobile, live-high-ogg, jukebox streams are visible at https://audio.ury.org.uk/status
• BAPS (all the presenter and guest PCs)
• Timelord (might need a reboot on eccleston/tennant/smith)
• myradio_daemon

We're now at a point where shows can go on and things will mostly be okay.

Dante

• Open Dante Controller on one of the studio or production PCs (you might need Wogan up for this), and ensure that everything is happy and you see lots of green check marks. If you see any angry red X-es, mouse over them to check which box they're complaining about, and ensure it's powered on
• Check that you can hear
  • Studio Red in Blue, and vice versa
  • Jukebox and News (beeeeeeeep) in both studios
  • AM on Phil in the office

Additional Servers

Power on other systems:

• urybsod
  • Don't forget to remount urybackup0:/pool0/backup to /mnt/pool0
  • Use https://urybsod.york.ac.uk/xymon/ to monitor other services
  • https://ury.org.uk/loggerng/ should now be available
• uryrrod
  • Only provides mixclouder service and webcams, will have no immediate noticable impact
  • Note that in the event of a full power outage, the ITS Cloud may not be immediately available. Patience.
• wogan
  • The Windows PCs may be a bit unhappy if it isn't around
  • See above about the ITS Cloud.
• urystv
• moyles - you'll need ITS to do this

Nearly Done

Everything should be bright and cheery again now. You should now complete a full incident report and make it available online in Category:Incident Reports.

Ideally, you'd also act on any recommendations this review brings up to make things run better in future.