It appears that the cooling on Fram has failed. The result is that many compute nodes are unavailable. We are investigating.
[15-06-2022 – 15:10] – Cooling fixed and computenodes are back in production. Sorry for the inconvenience this has caused.
Category: Hardware
Fram downtime 11th – 12th May 2022
[UPDATE, 2022-05-13 08:00] The maintenance stop is over. There may still be some file system issues. Please report via regular support channels
[UPDATE, 2022-05-11 08:00] The maintenance stop has now started.
The will be a maintenance of the storage system on the Fram supercomputer. The cluster will be unavailable from the 11th of May at 08:00 until the 12th of May at 20:00.
Fram downtime 23rd – 24th February
[Update, 2022-02-24 22:30]: The maintenance is over and Fram is in production again. Thank you for your patience!
[Update, 2022-02-24 20:30]: The maintenance is taking a little longer than planned. We plan to get back into production at 22:00.
[Update, 2022-02-23 12:00]: The maintenance stop has now begun.
Fram supercomputer will be unavailable due to maintenance on the cooling system from February 23rd 12:00 until 24th 20:00
If time allows it we will also upgrade whole or parts/components of the storage system, including file system clients (compute nodes)
Betzy unstability/hw maintenance
We are currently conducting various hardware maintennace on Betzy, including reseating infiniband cables. This may cause instabillity and crashed jobs in other parts of the system not directly connected to the cable being reseated.
We apologize for any inconvenience and lost jobs.
[FINISHED] Short Downtime for Betzy Wednesday 15th December 09:00-15:00
[UPDATE, 2021-12-15 15:00] Betzy is back in prodcution again.
[UPDATE, 2021-12-15 09:00] The downtime has now started.
There will be a short downtime for Betzy next Wednesday 15th from 09:00 until 15:00 to fix remaining hardware issues.
[Finished] Betzy downtime EXTENDED
We regret to inform you that the downtime for Betzy has been extended due to a failed hw component not being delivered in time. We hope to have the component delivered during Thursday and subsequently have the system online again in the afternoon of Thursday 9th.
Betzy downtime in progress
The downtime has started and will continue until wednesday 8th December evening or until upgrades are done.
Downtime Betzy, 6th December 08:00 until 9th December 20:00
There will be a scheduled downtime for Betzy lasting three days starting on Monday 6th December at 08:00. Downtime will last until Thursday 9th, 20:00.
During the downtime we will conduct:
- Full upgrade of the Lustre filesystem (both servers and clients)
- Full upgrade of the infiniband firmware
- Full upgrade of the Mellanox infiniband drivers
- minor updates to other parts of the system (Slurm, configs, etc)
Please be aware that this does also affect the storage services recently moved from NIRD to Betzy.
We apologize for the inconvenience
Update 08.12.2021 18:00 : Betzy downtime is over, and system is open for users. All planned update is performed .
[FINISHED] Short 1 hour downtime on Fram, 24th November 12:00
[Update, 2021-11-24 14:15] Now the NIRD mounts are working again.
[Update, 2021-11-24 13:30] We are back in production and jobs are running som normal again. We are missing the NIRD mounts on two of the login nodes, but are working on fixing that.
[Update, 2021-11-24 12:00] The maintenance has started now.
There will be a short 1 hour downtime for Fram on 24th November, starting at 12:00.
During downtime we will update the firmware on interconnect infiniband switches
[SOLVED] Betzy Downtime 7th June 15:00-20:00
[UPDATE, 2021-06-08 08:00] Betzy is now up and in production again.
[UPDATE] Unfortunately, the downtime is taking longer than anticipated, and will not be finished tonight. We plan on getting Betzy up again at around 08:00 tomorrow morning.
Campusservice at NTNU will conduct maintenance on the High Voltage circuits for Non-redundant power on 7th of June 2021, between 15:00 and 20:00. All compute nodes and login nodes will be shut down during this time, and no jobs will be running during this period. Submitted jobs estimated to run into the downtime reservation will be held in queue.