Fram, NIRD and Service Platform scheduled downtime starting on 21st November 2018

UPDATE:

  • 2018-11-23 09:45:
    • We plan to re-open access to Fram during today. Will keep you updated.
    • We are currently running benchmarks and tests on the upgraded system.
    • Cooling units are stress-tested now to pinpoint any outstanding issues.
    • OpenMPI is upgraded now.
  • 2018-11-22 22:48:
    • Lustre servers are upgraded on Fram and several tests, benchmarks were run to fine tune parameters.
    • First step of the CDU maintenance is carried out now.
  • 2018-11-22 11:33:
    • NIRD Service Platform is up again.
    • Access to NIRD is reopened. Please note that we have now four login nodes and SSH fingerprint is changed.
  • 2018-11-21 18:41:
    • Needed hardware replacement for NIRD was carried out and all firmware upgrades are finalized on both Tromsø and Trondheim site storage systems.
    • NIRD file systems are started back now and we plan to reopen access tomorrow before noon.
    • Firmware is updated now on the Fram storage system.
    • Several other updates, including the OpenFabrics stack and Lustre, were done in parallel.
  • 2018-11-21 08:00: Maintenance has started.

Dear Fram, NIRD and Service Platform User,

On the 21st and 22nd of November we will have a scheduled maintenance on Fram, NIRD and Service Platform.

This will be a comprehensive maintenance on the national HPC and research data infrastructure, ongoing on multiple levels and sites. Due to it’s complexity and amount of work involved, some parts of the infrastructure might require downtime extension for the 23rd of November, too.

The work will include, but will not be limited to:

– firmware upgrades on disks, enclosures, chassis, etc.

– operative system upgrades

– queue system upgrade

– file system upgrades

– kernel upgrades

– upgrade of OpenMPI

– upgrades on the OpenFabrics stack

– maintenance on the cooling system units

 

Our aim is to enhance the stability and security of the infrastructure, eliminate bugs and enhance performance, while having the shortest downtime possible.

We understand that system unavailability has big impact on your daily work and such we try to bring back our systems functional as soon as possible.

 

Thank you for your consideration!

Metacenter Operations

Fram: scheduled downtime on the 28th of August

UPDATE

2018-08-28 18:02 Fram is up and jobs running again.

We will have a one day scheduled downtime on Fram on the 28th of August starting from 08:00 AM.

Jobs not being able to finish before the maintenance window, will be left pending in the queue with a Reason “ReqNodeNotAvail” and will be started when the maintenance is over.

We will keep you updated via OpsLog/Twitter.

Thank you for your consideration!
Metacenter Operations

Planned maintenance on 12th of June

Update

  • 2018-06-12 17:16 Access it NIRD is reopened now.
  • 2018-06-12 16:05 Service are started and back in production on NIRD Service Platform.
  • 2018-06-12 15:55 Queue reservation is now removed and jobs are running on part of Fram. Rest of the nodes will be added back to the queue as soon as they are updated.
  • 2018-06-12 14:55 Access is re-opened to Fram. Queue reservation is still in place.
  • 2018-06-12 08:34 Maintenance has started.

Dear Fram and NIRD user,

We will have a one day planned maintenance on 12th of June starting from 08:30 AM.
Fram, NIRD and the Service Platform will be affected. One storage enclosure must be replaced, needing downtime for the file systems served from NIRD.

There is a system reservation in place on Fram starting on 12.06.2018 08:45 AM. Jobs not being able to finish before the maintenance window, will be left pending in the queue with a Reason “ReqNodeNotAvail” and will be started when the maintenance is over.

We will keep you updated via OpsLog/Twitter.

Thank you for your consideration!
Metacenter Operations

2 days downtime starting on 25th of April

Update:

  • 2018-04-30 14:46 File system issues are solved now on Fram and access is reopened. Jobs are temporarily on hold due to some troubles with the cooling system in the server room. As soon as that is sorted out, jobs will be permitted again.
  • 2018-04-30 10:15 We are still struggling with the /cluster file system. The problem is escalated to the Vendor. At the moment we do not have a time estimate when Fram is back online, but there is work in progress to fix this as soon as possible, hopefully during the day.
  • 2018-04-27 18:44 Unfortunately there are still problems taking up the Lustre file system on Fram. Issue is caused by an incompatibility hitting routing between IB networks/fabrics on the Lustre object storage servers. The vendor is now planning and working to carry out an emergency update on the system. We are sorry for the trouble.
  • 2018-04-27 16:49 Access to NIRD is reopened now.
  • 2018-04-26 22:50 We are having problems on taking up the Lustre file system on Fram. The issue is reported to the vendor. Additionally, there are some minor issues which must be addressed on NIRD before opening it for production, but we expect reopening the access to both Fram and NIRD during tomorrow.

 

Dear Fram and NIRD user,

A two day downtime is scheduled for week 17. The scheduled maintenance will start on Wednesday, 25th of April, at 09:00 AM and will affect Fram, NIRD and the Service Platform.

During this time we will:
1. Extend NIRD storage space with ~1.1PB.
– The new hardware will be coupled to NIRD and extra disks loaded to the system during these two days.
– Please note that the above advertised storage will not be available at once. Storage space is gradually added as soon as loaded disks are formatted and available to the file system.
– One of our top priorities is to address the inode shortage on $HOME areas.
2. Address file system related bugs on NIRD by upgrading the afferent software and tune some parameters on the servers.
3. Fix broken hardware on Fram.
4. Apply any outstanding patches to both Fram and NIRD.
5. Carry out maintenance work on the cooling system for Fram.

There is a job reservation in place on Fram starting on 08:45 AM 25th of April.  Jobs that cannot complete before that time, will be left pending in the queue with a Reason “ReqNodeNotAvail” and an estimated start time of 2154.  They will be started when the maintenance is over.

We will keep you updated via OpsLog/Twitter.

Thank you for your consideration!
Metacenter Operations

Fram Downtime February 8th 2018 07:00 – POSTPONED

Due to delay in the work done by the power company the power outage will be postoned, no new time is currently scheduled  We are sorry for this and any trouble this may cause for you.  We estimate the downtime to be no longer than 4 hours.

A new system reservation will be made when the new outtage is planned. We will need to re-queue any running jobs that is not finished by the time for the outage.

on behalf of the Fram HPCstaff

Steinar Trædal-Henden

Fram Downtime, February the 7th 2018 08:45 – POSTPONED

We need to schedule a downtime of Fram, due to work on the electricity circuits powering the datacenter.  We are sorry for this very short notice, and any trouble this may cause for you.  We estimate the downtime to be no longer than 4 hours.

A system reservation is registered in the queue system to avoid starting jobs which can not finish by 08:15, 7th of February. Jobs with longer walltime, that are already started and are not finished by the start of the downtime will be re-queued.

 

on behalf of the Fram HPCstaff

Steinar Trædal-Henden