Changes

   |end=02/02/2014 23:49

   |end=02/02/2014 23:49

   |mitigation=Multiple actions required to prevent recurrence.

   |mitigation=Multiple actions required to prevent recurrence.

   |leader=[[Anthony Williams]] <lpw@ury.org.uk>

   |leader=[[Anthony Williams]] <anthony@ury.org.uk>

   |others=[[Andrew Durant]] <aj@ury.org.uk>

   |others=[[Andrew Durant]] <aj@ury.org.uk>

}}

}}

At 23:27, Pyramid started completely timing out processing new requests. At this point, Apache's wsgi handlers also started queueing up waiting for responses from Pyramid, retrying several times before themselves timing out.

At 23:27, Pyramid started completely timing out processing new requests. At this point, Apache's wsgi handlers also started queueing up waiting for responses from Pyramid, retrying several times before themselves timing out.

At 23:28, the first user report was received, via a private Facebook message to [[Lloyd Wallis]]. Lloyd was unavailable at the time and the message went unnoticed. Five more users reported the failure in this way over the next few minutes.

At 23:28, the first user report was received, via a private Facebook message to [[Lloyd Wallis]]. Lloyd was unavailable at the time and the message went unnoticed. Four more users reported the failure in this way over the next few minutes.

At 23:33, our PostgreSQL database reported that service had reached the maximum number of allowed active connections and stopped serving new requests. At this time, services such as our Jukebox Scheduler, Tracklisting and BAPS all stopped working, significantly hampering broadcast capabilities.

At 23:33, our PostgreSQL database reported that service had reached the maximum number of allowed active connections and stopped serving new requests. At this time, services such as our Jukebox Scheduler, Tracklisting and BAPS all stopped working, significantly hampering broadcast capabilities.