[Updated] Fram Downtime on Wednesday 20th January 2021 from 12:00-15:00

Update: The file system servers have now been fixed, and we are back online again. Thank you for your patience.

We have an ongoing performance issue with Fram filesystem. We need to shut down file servers to get this fixed, and therefore need to have three hours downtime:

Wednesday 20th January between 12:00 and 15:00, Fram will be unavailable

NIRD-TOS maintenance is active

Dear NIRD and NIRD Toolkit user: NIRD-TOS is currently down and will remain unavailable until Wednesday 12:00. We are replacing all cables during the next coupe of days.

Note that NIRD-home is also not available during that time.

All remote mounts on Fram, Saga and Betzy using NIRD-TOS will be unavailable until downtime is over

Downtime for NIRD-TOS (including toolkit) and Fram 2nd November- 6th November

We will have downtime the following week to try again to replace all internal cables in NIRD-TOS and Fram storage systems.

NIRD-TOS (Including the toolkit) will be down from 08:00 Monday 2nd November to wednesday 4th 12:00

Fram will be down from Wednesday 4th 08:00 until Friday 6th 12:00

There is still a chance that the downtime will not happen, but proper notification will be given in the opslog. Unfortunately the current situation with Covid-19 makes it difficult to make detailed plans.

We apologize for any inconvenience.

Downtime for NIRD_TOS and Fram is postponed

The downtime for NIRD-TOS on 26th October until 29th October is cancelled and the downtime for Fram from 28th October until 29th of October is cancelled.

New dates for the downtime will be announced monday 26th or tuesday 27th.

During the downtime we will replace all internal cables between disk controllers and disk enclosures. The firmware upgrade two weeks ago helped a lot, but we are still seeing ccommunication errors so the decision is to remove all cables and replace them.

Downtime for Fram and NIRD on 12th October – 15th October

There will be a scheduled downtime for Fram and NIRD during week 42.

The downtime will start on Monday 12th at 08:00 and last until Thursday 15th at 15:00. Parts of the systems may be available during this period

The Downtime on monday is primarily for testing UPS and power facility in Fram datacenter, but we will at the same time start installing new firmware for all storage systems in Tromsø and NIRD Trondheim.

Maintenance on NIRD, NIRD Toolkit and Fram , 20th April -24th April

23 April – 18:50 NIRD and the NIRD toolkit services are now back into production

24th April: Fram is back in production.

WARNING: MAINTENANCE IS CURRENTLY ONGOING!

Dear NIRD, NIRD Toolkit, and Fram User,

We will have a four day long scheduled maintenance on NIRD, NIRD Toolkit and Fram starting on the 20th of April, 09:00 AM.

Running HPC jobs and logging in to Saga is NOT affected.
NIRD connectivity, and backup of files, from Saga IS affected

During the maintenance we will:

  • carry out software and firmware updates on all systems

Files stored on NIRD will be unavailable during the time of the maintenance and therefore so will be the services. This will of course affect the NIRD file systems available on Fram and Saga too.

Login services to NIRD, NIRD-toolkit and Fram will be disabled during the maintenance

Please note that backups taken from the Fram and Saga HPC clusters will also be affected and will be unavailable during this period.

Please accept our apologies for the inconvenience this downtime is causing.

Metacenter Operations

NIRD and NIRD Toolkit scheduled maintenance

Update:

  • 2020-01-23 17:30: Services are now progressively restarted.
  • 2020-01-22 21:49: We have detected file system level corruption and to avoid data corruption we had to unmount and rescan all the file systems (about 18PB) on NIRD.
    We are currently working on starting back the services on NIRD Toolkit.
  • 2020-01-22 11:11: Software and firmware is now upgraded on NIRD Toolkit.
    Most of the fileset changes are also carried out. We are currently working on the last bits. Will keep you updated.
  • 2020-01-20 08:58: Maintenance has started. NIRD file systems are unmounted from Fram until maintenance is finished.

Dear NIRD and NIRD Toolkit User,

We will have a three day long scheduled maintenance on NIRD and NIRD Toolkit starting on the 20th of January, 09:00 AM.

During the maintenance we will:

  • carry out software and firmware updates,
  • change geo-locality for some of the projects,
  • replace synchronization mechanisms,
  • depending on part delivery times from disk vendor – expand the storage and quotas.

Files stored on NIRD will be unavailable during the time of the maintenance and therefore so will be the services. This will of course affect the NIRD file systems available on Fram too.

Please note that backups taken from the Fram and Saga HPC clusters will also be affected and will be unavailable during this period.

Please accept out apologies for the inconvenience this downtime is causing.

Metacenter Operations