FRAM power outage

Dear Fram users. Unfortunately, there has been a short power outage in Tromsø causing a shutdown of compute nodes on Fram. We are working on bringing them back to production as soon as possible.

Sorry for the inconvenience this has caused.

[2022-05-03 – 10:45] – Fram is back in production.

[2022-05-04 – 13:20] – As a result of the power outage we have some problems with FRAM file system. Slowness/lagging. We are currently working on fixing this and are sorry for the inconvenience this is causing.

[2022-05-06 – 13:25] Fram filesystem is still priodically slow for some users. We assure you that we are continuously working to resolve this issue, but it is hard to debug due to the inconcistancy of the problem.

Best regards

Infra team

Fram downtime 11th – 12th May 2022

[UPDATE, 2022-05-13 08:00] The maintenance stop is over. There may still be some file system issues. Please report via regular support channels

[UPDATE, 2022-05-11 08:00] The maintenance stop has now started.

The will be a maintenance of the storage system on the Fram supercomputer. The cluster will be unavailable from the 11th of May at 08:00 until the 12th of May at 20:00.

Maintenance Stops on Saga, Fram and Betzy

[Update, 2022-04-30 11:10] The Fram and Saga maintenance is now over, and jobs are running again.

[Update, 2022-04-29 08:00] The Fram and Saga maintenances have now started.

[Update, 2022-04-28 12:56] The Betzy maintenance is now over, and jobs are starting again.

[Update, 2022-04-28 08:00] The Betzy maintenance has now started.

There will unfortunately be maintenance stops on all NRIS clusters next week, for an important security update. The maintenance stops will be

  • Betzy: Thursday, April 28. at 08:00
  • Fram and Saga: Friday, April 29. at 08:00

We expect the stops will last a couple of hours. We have set up maintenance reservations on all nodes on the clusters, so jobs that would have run into the reservation will be left pending in the job queue until after the maintenance stop.

We are sorry for the inconvenience this creates. We had hoped to be able to apply the security update with jobs running, but that turned out not to be possible.

Maintenance NIRD – TOS

We need to conduct some work on the filesystem controllers for NIRD – TOS. Unfortunately this results in a short unavailability (downtime) period.

All services connected to- and/or utilizing TOS (Tromsø) part of NIRD will be affected. Exported NFS services mounted on FRAM will unfortunately NOT be available either.

The maintenance is set for Thursday 07.04.22 from 09:00-11:00 AM

We are sorry for any inconveniences that may occur. Opslog is updated as soon as the system is back in production.

UPDATE 07-04-2022 – 11:25 … we are still working on the issue and starting to bring the file system up, we hope to back in production soon

UPDATE 07-04-2022 – 12:25 … we are struggling and fighting with the file system, doing our best, we are very sorry for the troubles the issue is causing you

UPDATE 07-04-2022 – 15:35 … the file system is back up and running

Fram downtime 23rd – 24th February

[Update, 2022-02-24 22:30]: The maintenance is over and Fram is in production again. Thank you for your patience!

[Update, 2022-02-24 20:30]: The maintenance is taking a little longer than planned. We plan to get back into production at 22:00.

[Update, 2022-02-23 12:00]: The maintenance stop has now begun.

Fram supercomputer will be unavailable due to maintenance on the cooling system from February 23rd 12:00 until 24th 20:00

If time allows it we will also upgrade whole or parts/components of the storage system, including file system clients (compute nodes)

FRAM – connectivity blip 09.02.2022

Dear Fram Users,

We need to conduct some test of core switches (together with the vendor) tomorrow 09.02.2022 between 10 – 11 am. The overall connectivity should not be affected but you might encounter a short blip (seconds) in your connection towards Fram.

The outcome of the test should ensure uplinks of the core switches are fully redundant.

We are very sorry for any inconvenience this might cause to you.

NRIS infrastructure team