On the 25th November 2021, the Walton Institute in Waterford suffered an uplanned and total power loss in their Datacenter in Carriganore. The exact cause is still being investigated. This talk will cover what was observed before and during the outage, the downtime and damaged it caused. It will also detail the actions that were taken to firstly keep everybody safe, secondly, to minimise the damage to hardware and services, thirdly the process of temporarily maintaining service continuity under adverse conditions and finally the recovery process over the next few months after the outage.

The Research Infrastructure and Testbed Services team of the Walton Institute are still picking up the pieces and some of the recovery processes are still ongoing. We would like to share some lessons learned, some mistakes to be avoided and interesting pictures of the event that even we had not seen before.