Fram compute nodes down

All nodes are back in production and the cooling issue has been resolved.

-Infra Team

De-escalating incident as most of the system is up. We are keeping the system capacity at 60-70% until service from the vendor has been completed. We expect the service to be completed tomorrow.

The cooling system went down last night. This triggered auto-poweroff of compute nodes.
The cooling system is running again now, and we have started compute nodes again. But we have reduced the load with 30-40 %, awaiting service from the vendor. This can lead to longer wait time in the job queue.

Most compute nodes went down at around 22:42 yesterday. We are investigating the cause.