Log over changes and events on national e-infrastructure
We have the pleasure to announce that the Saga HPC cluster is now opened for production for existing pilot users.
Candidate projects for migration from the Abel cluster will be contacted directly.
Link to the Saga documentation page.
Metacenter Operations
With two nodes short of a fully 1404 node capacity, we are now back in full production. A few jobs were lost due to missing lustre filesystem on a few nodes, wich again was due to a faulty interconnect/infiniband cable.
Thank you for your patience.
We are currently experiencing a network error on VIlje, causing around 100 nodes to be unavailable until further notice. Some jobs may be lost.
We apologize for the inconvenience.
Update:
Dear NIRD and Service Platform users,
We have a planned downtime on the 22nd of August, to replace some defective hardware. Systems will be taken offline starting from 08:00AM.
Engineer from storage vendor will assist us from the very first hour.
We expect the maintenance to finish in one and a half day.
NIRD projects will still be accessible during the maintenance
from login-trd.nird.sigma2.no but in read-only mode.
Will keep you updated here.
Sorry for the short notice.
Metacenter Operations
There is a power outage in Tromsø affecting parts of campus right now. Stallo is down, Fram is still up.
Dear Fram cluster users:
We have a problem with the cooling system in the Fram machine room,
due to this, we have to reduce the load on the cluster by reserving the entire cluster, which means no job will run.
We are sorry for the inconvenience, and we will keep you updated.
Update 2019.07.11 08:00: Fram should be fully operational again, we are monitoring the machine and releasing compute nodes back to production.
Update 14:55: Some of the nodes are crashed, which means it’s possible that some of the jobs get killed
Update 2019.07.03 10:55: To keep the machine room temperature reasonably low with only one working CDU, we have kept 495 nodes in maintenance state while 197 nodes are in downstate, we will monitor the power consumption in the machine room and release more nodes accordingly.
Update 2019.07.08 11:55: Fram is expected to be back to full its full capacity on Wednesday, 2019.07.10.
Dear NIRD and Service Platform User,
We have a planned downtime on the 26th of June, Wednesday next week, to replace some defective hardware. Systems will taken offline starting from 08:00AM.
Engineer from storage vendor will assist us from the very first hour.
We expect the maintenance to finish in one day.
Will keep you updated here.
Metacenter Operations
Dear Fram User,
Once again has the Fram metadata server crashed and likely had impact on running jobs.
We are in contact with the storage vendor for patching the file system.
Apologies for the inconvenience!
Dear Fram User,
As of today we have adjusted the queue system policies to facilitate code development and testing on Fram and meanwhile limit possible misuse of devel queue.
devel is now adjusted to allow:
We have additionally introduced a short queue with following settings:
We will continue to monitor and improve the queue system. Please stay tuned.
You may find more information here.
Metacenter Operations
Dear Fram User,
This morning around 09:05, once again has the Fram metadata server crashed and likely had impact on running jobs.
A mitigating patch was delivered by the vendor yesterday and we used this opportunity to apply it on our metadata servers.
We will keep the system closely monitored and cooperate with the vendor on further stabilizing the system.
Apologies for any inconvenience this may have caused!