Unstable NIRD connection

We are experiencing unstable network connection between NIRD and Fram. On Fram login nodes, some of the project area might be missing.

Please try to check both login nodes (login-1-1,login-1-2) for your project area. Currently all project areas are mounted on login-1-2.

We are working on this issue and will keep you updated.

We apologize for the inconvenience it may caused you.

DATA LOSS in NS2345K

Update 25.01.2019 11:20 Original Recover process is terminated due to limit on filesystem space. To make project data accessible, we have the following steps and procedures:

  1. Project users are able to get access to the NS2345K project snapshot in read-only mode so that they can get to the necessary data. Snapshot location:
    /projects/NS2345K/.snapshots/Tuesday-15-Jan-2019
  2. The project users will also get access to a space created for project NS2345 on the Fram side, namely /cluster/NS2345K. This is a temporary space where project users can work and store new data. This space has a nightly backup. This space is also accessible from NIRD (at moment only from login4.nird.sigma2.no) at:
    /projects/NS2345K/FRAM
  3. In the meantime, we are working on recovering NS2345K project space.

We will keep you updated.

Update 15:25 Recovery process is still scanning through the missing inodes from the snapshots and dispatching the operation to related nodes. When this is done, files will start to recover.

Update 11:50 We have locked the project to avoid interference with the restoration process.

We have lost some of the data in project NS2345K. We are in the process of recovering the lost data, affected files will hopefully be gradually recovered.

Fram longer queue times

We will have a four hour maintenance on the cooling system for Fram on 16th of May from 09:00. To limit cooling requirements, only half of the compute nodes will be operational during this time.
This might lead to longer queue times.

Thank you for your understanding!
Metacenter Operations