Sigma2 router upgrade in Tromsø

Uninett will conduct a firmware upgrade of one of the routers in Tromsø this Friday, 5th November between 11:00 and 15:00. This will not affect internal networks on Fram, NIRD or NIRD Toolkit or any production on the systems, but external network may briefly disconnect or stall

If the upgrade is successful, the other router will be upgraded next week.

[Finished] NIRD Service Platform Maintenance, 22-23 September

Update 2021-09-23: The maintenance is now finished on both sites. Services should be back in production.

Dear users,

We’ll have scheduled maintenance on the NIRD Service platform on 22 and 23 September in order to perform upgrades on the clusters.

In addition to project deployments running on the service platform, the following services are affected during the maintenance:

  • NIRD Toolkit
  • NIRD Archive
  • EasyDMP

The service platform consists of two sites, one in Tromsø and the other in Trondheim. This maintenance will be performed on one site at a time, planned as follows:

22 september: Tromsø
Services running on TOS-SP will be offline. NIRD will be accessible from login-trd.nird.sigma2.no.

23 september: Trondheim

Services running on TRD-SP will be offline. NIRD will be accessible from login-tos.nird.sigma2.no.

To check what site your project is running on, you may log in on the NIRD login-nodes and run the following command: (ssh login.nird.sigma2.no)

readlink /projects/<project number>

Make sure to write the project number in all uppercase.
This will then output the full path to the volume, starting with either “trd” for Trondheim or “tos” for Tromsø.

Example:

[user@login0-nird-trd ~]$ readlink /projects/NS9999K
/tos-project3/NS9999K

The output indicates that this project have it’s primary site in Tromsø (tos-project).

If you have any questions, please do not hesitate to contact us.

Service not activated in NIRD Service Platform

We regret to inform you that, due to a recent change made by Feide in response to the new national security directives in the sector, you might no longer be able to launch services on the NIRD Toolkit.  The reason is that, from now on, your institution shall approve the services requiring Feide login. If a service is not approved, you cannot access it with your Feide account. Unfortunately, the approval cannot be exercised when the services are deployed dynamically, and on-demand like in the NIRD Toolkit.  

What can I do? 

If you are experiencing a problem with using the NIRD Toolkit, we advise you to email the Feide administrator at your institution with us in CC (sigma2@uninett.no).  

If this takes time and you have an urgent need to use the NIRD Toolkit, there is a workaround (a little cumbersome but only temporary) to mitigate the problem, described here: 

Deploy a service through the NIRD Toolkit – Service not activated 

More information about the changes 
 

You can read more about the changes Feide has made in this article on www.feide.no (in Norwegian).  

We are currently working with Feide to resolve the issue. The solution shall allow automatic approval of all the services deployed through the NIRD Toolkit if the NIRD Toolkit service itself is approved. In the meantime, some organisations have already dealt with this problem by choosing the “Opt-in” option and therefore by approving all Feide Services. This is the temporary solution suggested by Feide and we will contact your organization’s Feide administrator to inform them about this option. 

Please note that Sigma2 was not notified of the changes, and therefore we could not inform you beforehand. Apologies for the inconvenience this may have caused!  

This post will be used to provide updates as we have more information available.

Apologies for the inconvenience this may have caused! 

Downtime for NIRD-TOS (including toolkit) and Fram 2nd November- 6th November

We will have downtime the following week to try again to replace all internal cables in NIRD-TOS and Fram storage systems.

NIRD-TOS (Including the toolkit) will be down from 08:00 Monday 2nd November to wednesday 4th 12:00

Fram will be down from Wednesday 4th 08:00 until Friday 6th 12:00

There is still a chance that the downtime will not happen, but proper notification will be given in the opslog. Unfortunately the current situation with Covid-19 makes it difficult to make detailed plans.

We apologize for any inconvenience.

Downtime for NIRD_TOS and Fram is postponed

The downtime for NIRD-TOS on 26th October until 29th October is cancelled and the downtime for Fram from 28th October until 29th of October is cancelled.

New dates for the downtime will be announced monday 26th or tuesday 27th.

During the downtime we will replace all internal cables between disk controllers and disk enclosures. The firmware upgrade two weeks ago helped a lot, but we are still seeing ccommunication errors so the decision is to remove all cables and replace them.

Downtime 20th – 24th of April is over. Services are back in production

All services on Fram and NIRD are now be back in production, except for slurmbrowser and desktop.fram.sigma2.no.

Here is a list of what has been done during the last four days:

  • Firmware upgrade on NIRD in Trondheim and Tromsø
  • Firmware upgrade on NIRD Toolkit
  • Firmware upgrade on Fram storage and Fram nodes, switches m.m
  • Software/OS upgrade on NIRD Trondheim and Tromsø
  • Software/OS upgrade on NIRD Toolkit
  • Software/OS upgrade on Fram nodes

In total, including vendors, ca 15 people were involved in the upgrade.

We thank you for your patience.

tos-project3 on NIRD is read only

Due to underlying hardware issues, tos-project3 filesystem is set to READ-ONLY while we investigate the issue.

These are the projects affected:

NN9999K
NS1002K
NS4704K
NS9001K
NS9012K
NS9014K
NS9033K
NS9054K
NS9063K
NS9066K
NS9114K
NS9191K
NS9320K
NS9404K
NS9518K
NS9602K
NS9615K
NS9641K
NS9672K
NS0000K
NS1004K
NS9000K
NS9003K
NS9013K
NS9021K
NS9035K
NS9060K
NS9064K
NS9081K
NS9133K
NS9305K
NS9357K
NS9478K
NS9560K
NS9603K
NS9616K
NS9655K
NS9999K