We have finished the recablingof both Fram and NIRD-TOS storage systems. All cabling is now balanced and all cables are same vendor (instead of a mix between four different vendors). Huge thanks to all who participated in the downtime and a huge thank to all of you waiting patiently to start running jobs and transfer data again.
We will have downtime the following week to try again to replace all internal cables in NIRD-TOS and Fram storage systems.
NIRD-TOS (Including the toolkit) will be down from 08:00 Monday 2nd November to wednesday 4th 12:00
Fram will be down from Wednesday 4th 08:00 until Friday 6th 12:00
There is still a chance that the downtime will not happen, but proper notification will be given in the opslog. Unfortunately the current situation with Covid-19 makes it difficult to make detailed plans.
The downtime for NIRD-TOS on 26th October until 29th October is cancelled and the downtime for Fram from 28th October until 29th of October is cancelled.
New dates for the downtime will be announced monday 26th or tuesday 27th.
During the downtime we will replace all internal cables between disk controllers and disk enclosures. The firmware upgrade two weeks ago helped a lot, but we are still seeing ccommunication errors so the decision is to remove all cables and replace them.
The Fram firmware upgrade is finished. We have opened up the Fram login nodes for business, but NIRD project directories and other NIRD directories are not mounted since we are waiting for NIRD-TOS firmware upgrade to finish tomorrow afternoon.
This week we are executing a series of firmware upgrade on storage for Fram, NIRD-TOS and NIRD-TRD. As of 08:00 Monday morning until Thursday 15:00 all three systems will experience shorter or longer stops to apply the update.
We are experiencing problems with the file systems in NIRD-TOS again today. This affects all or some services dependent on NIRD-TOS, including Serviceplatform and data accessibility for some data. We are working with the vendor to rectify the issue.