249
edits
Changes
Jump to navigation
Jump to search
Line 36:
Line 36: − A number of related and unrelated issues all happened at the same time:+ − − * Dolby CPU issues - potentially related, potentially not − − ::* Bug in MyRadio causing the sso_session to grow quite a bit with lots of empty sessions − ::* Bug in Campus Playout putting it into a loop − ::* Other things? − * Initial dead air - UNKNOWN CAUSE, thonking in progress (probably rednet) − * RedNet 3 failure − :* Known hardware/firmware issue, has happened before − * About 9 minutes later, the grandmaster starts cycling - for, as far as we know so far, no good reason − * Scarlett go fucky − :* Probably caused by ADAT clock switch and double-clocking − ::* Caused by human error - mis-understanding of clocking − * Hot coax − :* Potential short or other electrical fault in the RedNet or Scarlett − * Full recovery delayed due to issues with SelectorListener
no edit summary
= Analysis =
= Analysis =
The most likely culprit in this scenario is some kind of hardware issue, potentially electrical fault in the RedNet 3. This would explain why it was cycling between link DOWN and UP on the network - it's possible that shifting logic levels inside confused the ethernet controller and/or sent invalid Layer 1 signals. It's possible that during the clocking storm (21:29-21:32) the RedNet was disappearing from the network, triggering a Dante clock master election, then re-appearing, establishing itself as master, before leaving again. (This does not explain, however, why the master would cycle between Phil, Office, and StudioRed, as one would hope Dante can at least pick one as master and stick with it.) This would also explain the hot cable - normally a word clock signal is a mere 5V, not even close to sufficient for heating it up. The noise heard during the clocking storm also sounds digital, almost like dropping packets. Anecdotally, we've had issues with RedNet power supplies in the past.
:* Caused by
= Actions =
= Actions =