The electical work have met unforseen problems, one aggregate will not power on. They need to continue the work further into this evening.
The electricians did not manage to finalize their work on the electrical power infrastructure in the previouse time slot (January 10-11th.)
So we need to power off the machine again on January 31st.. All jobs with walltime beyond the start time of the poweroff will be held pending in the queue until the system is up and running again.
2019-11-01 14:37 Update: stallo is back online and in production again!
Due to work on the electrical power infrastructure in the building housing stallo, we need to power off the machine in the given periode. All jobs with walltime beyond the start time of the poweroff will be held pending in the queue until the system is up and running again.
2018-23-11 15:24: Fram is now available again for use. The service was a success and we were able to complete everything on our list, as previously announced via email, nird and nird service platform is already up and running.
2018-11-14, 09:54 Vilje is experiencing a failure with the Lustre filesystem. The system will be down until further notice.
- 2018-11-23 09:45:
- We plan to re-open access to Fram during today. Will keep you updated.
- We are currently running benchmarks and tests on the upgraded system.
- Cooling units are stress-tested now to pinpoint any outstanding issues.
- OpenMPI is upgraded now.
- 2018-11-22 22:48:
- Lustre servers are upgraded on Fram and several tests, benchmarks were run to fine tune parameters.
- First step of the CDU maintenance is carried out now.
- 2018-11-22 11:33:
- NIRD Service Platform is up again.
- Access to NIRD is reopened. Please note that we have now four login nodes and SSH fingerprint is changed.
- 2018-11-21 18:41:
- Needed hardware replacement for NIRD was carried out and all firmware upgrades are finalized on both Tromsø and Trondheim site storage systems.
- NIRD file systems are started back now and we plan to reopen access tomorrow before noon.
- Firmware is updated now on the Fram storage system.
- Several other updates, including the OpenFabrics stack and Lustre, were done in parallel.
- 2018-11-21 08:00: Maintenance has started.
Dear Fram, NIRD and Service Platform User,
On the 21st and 22nd of November we will have a scheduled maintenance on Fram, NIRD and Service Platform.
This will be a comprehensive maintenance on the national HPC and research data infrastructure, ongoing on multiple levels and sites. Due to it’s complexity and amount of work involved, some parts of the infrastructure might require downtime extension for the 23rd of November, too.
The work will include, but will not be limited to:
– firmware upgrades on disks, enclosures, chassis, etc.
– operative system upgrades
– queue system upgrade
– file system upgrades
– kernel upgrades
– upgrade of OpenMPI
– upgrades on the OpenFabrics stack
– maintenance on the cooling system units
Our aim is to enhance the stability and security of the infrastructure, eliminate bugs and enhance performance, while having the shortest downtime possible.
We understand that system unavailability has big impact on your daily work and such we try to bring back our systems functional as soon as possible.
Thank you for your consideration!
2018-11-12:11:55: Login node and services are back into production.
2018-11-12 10:20: Disk pool raid sets were rebuilt until Saturday, but a set of drives failed once again. A new rebuild was ongoing and we had to reset IO card and power cycle the storage today. At this point all is up and functional on the storage side and file system is up. We are currently switching back geo-replication and expect to reopen access around 12:00 PM today. Will keep you posted.
2018-11-09 13:59: The firmware is now applied without any problem. However we still need to wait for a rebuild to finish. The time estimate for the rebuild is 12 hours left. We will open the system for regular use as soon as we can.
2018-11-09 12:45: Most of the rebuilds are ready and we are currently patching the firmware on the disk enclosures. If all goes well, we expect to have NIRD up and functional during the day today. Will keep you updated.
2018-11-08 13:27: The firmware update is running. We have to wait for rebuild of broken drives before we can upgrade the enclosures and finnish up the emergency maintenance. We don’t expect the rebuild to be finished before tomorrow (friday november 9th). Hence the system in whole will not be available before tomorrow.
We are very sorry any inconvenience this may cause.
2018.09.03 13:19 we just lost cooling on all compute nodes in Fram. We are working to get the system back online.
We experience some trouble with the filesystem on fram at the moment. We are working to find the cause of the problem and fix it.
UPDATE, 13:21 The FS is stable again now.
We apologize for the long downtime and appreciate your patience.