[Updated] Fram Downtime on Wednesday 20th January 2021 from 12:00-15:00

Update: The file system servers have now been fixed, and we are back online again. Thank you for your patience.

We have an ongoing performance issue with Fram filesystem. We need to shut down file servers to get this fixed, and therefore need to have three hours downtime:

Wednesday 20th January between 12:00 and 15:00, Fram will be unavailable

Fram: compute nodes are down

Dear Fram users,

We have problem with Fram compute nodes, there are about 870 nodes is down due to unknown reason, we are working on the issue, and will keep you updated.

Update 2020-12-22, 20:05: Most of the compute nodes have now been brought back online. There are still a few nodes that needs more checking before being made available for jobs.

Update 2020-12-22, 18:04: The cooling system has been stable for the last hour after making some adjustments together with the vendor. We are slowly bringing up the nodes.

Update 2020-12-22, 16:01: In order to keep the cooling as stable as possible, we have decided to take down all high memory nodes. This way we can keep some of the normal compute nodes up for the time being. We are also working together with the vendor to make adjustments on the cooling system to ensure continued stability.

We are very sorry about the inconvenience.

Update 2020-12-22, 13:41: We have identified the cause to be the cooling system and are working on mitigating the issues. Most of the compute nodes must remain down while doing so, unfortunately.

Update 2020-12-24 10:30: Compute nodes shutdown again due to electrical problems in machine room, problem has been resolved according to machine room service department, we are working to take up all nodes.

Update 2020-12-24 12:10: Most of the compute nodes on Fram is back online.

Downtime for NIRD-TOS (including toolkit) and Fram 2nd November- 6th November

We will have downtime the following week to try again to replace all internal cables in NIRD-TOS and Fram storage systems.

NIRD-TOS (Including the toolkit) will be down from 08:00 Monday 2nd November to wednesday 4th 12:00

Fram will be down from Wednesday 4th 08:00 until Friday 6th 12:00

There is still a chance that the downtime will not happen, but proper notification will be given in the opslog. Unfortunately the current situation with Covid-19 makes it difficult to make detailed plans.

We apologize for any inconvenience.

Downtime for NIRD_TOS and Fram is postponed

The downtime for NIRD-TOS on 26th October until 29th October is cancelled and the downtime for Fram from 28th October until 29th of October is cancelled.

New dates for the downtime will be announced monday 26th or tuesday 27th.

During the downtime we will replace all internal cables between disk controllers and disk enclosures. The firmware upgrade two weeks ago helped a lot, but we are still seeing ccommunication errors so the decision is to remove all cables and replace them.

Downtime for Fram and NIRD on 12th October – 15th October

There will be a scheduled downtime for Fram and NIRD during week 42.

The downtime will start on Monday 12th at 08:00 and last until Thursday 15th at 15:00. Parts of the systems may be available during this period

The Downtime on monday is primarily for testing UPS and power facility in Fram datacenter, but we will at the same time start installing new firmware for all storage systems in Tromsø and NIRD Trondheim.

Fram file system issues

11:30 15-09-2020 [Update 7]: Quick heads-up: We are trying to put one of the storage servers back into production. This could result in some users/jobs experiencing some short hangs. If you are in doubt about the behaviour of your jobs, please, do not hesitate to contact us at support@metacenter.no.

14:30 14-09-2020 [Update 6]: Most compute nodes are running now with the old lustre client. So, what regards the most recent issues, it should be safe to submit jobs. Unfortunately, this also means that the «hung io-wait issue» may happen again. Just contact us via support@metacenter.no in case you continue to have file system issues.

12:15 14-09-2020 [Update 5]: We found the reason for the behaviour many users have reported (problems with the module system, crashes, etc). It seems the new file system client causes this. So, the only immediate “solution” is to go back to the old version of the client. This may cause other issues, however, they are less severe than what we see now. We will inform here if it is safe to submit jobs.

10:30 14-09-2020 [Update 4]: Over the weekend, on the majority of compute nodes the lustre client for the parallel file system was updated. However, users are still reporting issues, particularly, when loading modules. It seems that the module system is not configured correctly on the updated nodes. We are looking into fixing the issue and keep you up-to-date here.

Sorry for the inconvenience!

15:00 11-09-2020[Update 3]: We are currently upgrading lustre filesystem clients to mitigate a «hung io-wait issue». We are also at reduced capacity performance-wice as one of eight io-servers are down. Full production is to be expected from Monday morning. A small hang is expected when io-server i phased in. We expect hung io-wait to go away during next two weeks as clients are upgraded

20:50 10-09-2020[Update 2] : Sorry to inform that we are still having some issues and vendor has been contacted

13:15 10-09-2020[Update 1] : The file system is partially back in operation. Which means you may use Fram but the performance will be sub-optimal. Some jobs may be affected when we try to bring back a object storage latter today.

08:15 10-09-2020 : We are experiencing some issues with the Fram file system and working on fix. Sorry for the inconvenience.