Update: The file system servers have now been fixed, and we are back online again. Thank you for your patience.
We have an ongoing performance issue with Fram filesystem. We need to shut down file servers to get this fixed, and therefore need to have three hours downtime:
Wednesday 20th January between 12:00 and 15:00, Fram will be unavailable
As previously announced, Saga will be down in the coming week, from 7th December 08:00 until 11th December 16:00.
The downtime is allocated for expanding the storage. When we come back we will have ca 4 Petabyte in addition to the already existing 1 PetaByte.
Update: Saga is back online and running jobs again. The new storage is not online yet, but all the hardware has been mounted.
As Betzy now has started production, the time has come to stop production on Vilje. On 1st of December no more jobs will be accepted into the queueing system and running jobs will be terminated. During the next few days after 1st December, the /work file system will be shut down and disconnected. ALL DATA on /work will be lost if you do not take proper measures and make sure you copy data out of Vilje before 1st of December.
We will try to have the /home filesystem still operational after 1st of december, but there is no guarantee this will be possible. There is also still a small possibility that Vilje could be available for running jobs after 1st of December, although highly unlikely.
On behalf of all previous and present NTNU staff:
Thanks for all the computing
We are going to expand the storage on Saga. This will happen during week 50, between 7th and 11th December. Hopefully this will give oss a few Petabytes extra and enough storage for the lifetime of the system.
On 1st of December 2020 between 07:45 and 16:00 there will be a power outage on Fram compute nodes due to scheduled maintenance on UPS and backup power equipment.
We have finished the recablingof both Fram and NIRD-TOS storage systems. All cabling is now balanced and all cables are same vendor (instead of a mix between four different vendors). Huge thanks to all who participated in the downtime and a huge thank to all of you waiting patiently to start running jobs and transfer data again.
Cabling before replacement and cleanup:
And after replacement and cleeanup:
As of today, Wednesday 4th, November at 08:00, Fram is down for maintenance. We will do the same exercise as on NIRD-TOS, namely change all internal cables on the storage system.
17:20 NIRD-TOS and services are now up.
Dear NIRD and NIRD Toolkit user: NIRD-TOS is currently down and will remain unavailable until Wednesday 12:00. We are replacing all cables during the next coupe of days.
Note that NIRD-home is also not available during that time.
All remote mounts on Fram, Saga and Betzy using NIRD-TOS will be unavailable until downtime is over
We will have downtime the following week to try again to replace all internal cables in NIRD-TOS and Fram storage systems.
NIRD-TOS (Including the toolkit) will be down from 08:00 Monday 2nd November to wednesday 4th 12:00
Fram will be down from Wednesday 4th 08:00 until Friday 6th 12:00
There is still a chance that the downtime will not happen, but proper notification will be given in the opslog. Unfortunately the current situation with Covid-19 makes it difficult to make detailed plans.
We apologize for any inconvenience.
The downtime for NIRD-TOS on 26th October until 29th October is cancelled and the downtime for Fram from 28th October until 29th of October is cancelled.
New dates for the downtime will be announced monday 26th or tuesday 27th.
During the downtime we will replace all internal cables between disk controllers and disk enclosures. The firmware upgrade two weeks ago helped a lot, but we are still seeing ccommunication errors so the decision is to remove all cables and replace them.