Vilje filesystem has been fixed with good help from DDN and we are now open for business.
Please be aware that some files may have been lost.
Always back up your files.
Vilje is back in (near) full production
With two nodes short of a fully 1404 node capacity, we are now back in full production. A few jobs were lost due to missing lustre filesystem on a few nodes, wich again was due to a faulty interconnect/infiniband cable.
Thank you for your patience.
Vilje partially down
We are currently experiencing a network error on VIlje, causing around 100 nodes to be unavailable until further notice. Some jobs may be lost.
We apologize for the inconvenience.
Vilje is online
The infiniband error was due to a controller module with bad connection. This has been corrected.
The queueing system is back online. Also: 19 additional nodes has been recovered.
Three jobs were lost. We apologize for the inconvenience.
Vilje infiniband problems
We are currently experiencing infiniband problems on VIlje. The queueing system is unavailable until further notice.
Some jobs may have been lost.
Vilje is back online
Vilje is online.
The outage was caused by the loss of infiniband connectivity/loss of two infiniband switches.
36 nodes will remain out of production.
There may still be dns issues with connectivity from innside the cluster to outside (i.e: licence server lookups). Please report any issues to: support@metacenter.no
550 nodes down on Fram
550 nodes went down at 00:00 Monday morning. We are investigating the issue and will bring nodes back online as soon as possible