[RESOLVED] Saga downtime. 7th December-11th December. Adding 4PB storage.

As previously announced, Saga will be down in the coming week, from 7th December 08:00 until 11th December 16:00.

The downtime is allocated for expanding the storage. When we come back we will have ca 4 Petabyte in addition to the already existing 1 PetaByte.

Update: Saga is back online and running jobs again. The new storage is not online yet, but all the hardware has been mounted.

Stallo – file system problem

Dear Stallo Users,

UPDATE – 27.11/16:20 – we have opened the machine for you guys but there might be some instabilities on global file system as we have also lost one object storage server. The issue is being investigated and we are waiting for some spare parts.

We have some major problems with Lustre file system at the moment. One of the main storage coolers is down. We are kicking out all users now and hope to get the machine back to an operational state ASAP.

Thank you for your patience.

HPC staff (UiT)

Stallo down

As published on Twitter; Stallo is now down due to necessary power infrastructure work in the hosting building at UiT. Due to this, and an overall work-load and risk consideration done by the operational team for Stallo, the oldest nodes – c1-c19; i.e. the 16 core nodes, will not be turned on again tomorrow. This includes the high mem partition on Stallo. Thus Stallo will be reduced by approximately 4800 cores from now. Expected downtime is 24 hrs from 12 today. from noon tomorrow only the 20 core nodes will be up, a total of approx 10k cores running out dec. 20.

Vilje final shutdown and end of life.

As Betzy now has started production, the time has come to stop production on Vilje. On 1st of December no more jobs will be accepted into the queueing system and running jobs will be terminated. During the next few days after 1st December, the /work file system will be shut down and disconnected. ALL DATA on /work will be lost if you do not take proper measures and make sure you copy data out of Vilje before 1st of December.

We will try to have the /home filesystem still operational after 1st of december, but there is no guarantee this will be possible. There is also still a small possibility that Vilje could be available for running jobs after 1st of December, although highly unlikely.

On behalf of all previous and present NTNU staff:

Thanks for all the computing

Betzy pre-production

Dear HPC User,

We are pleased to announce that Betzy is opened for pre-production Friday 20 November.

Being close to the weekend, Betzy is opened stepwise. First to prior pilot projects and then for general access Tuesday 24 November.

It has been a long journey, but we are happy to see good performance and stability on the system.

Please note, that during the coming days, changes will be made to the queue system setup, which could necessitate the cancelling of running jobs.

Finally, support will be also offered only from 24 November.

Thank you for your patience and we wish you happy computing!

Best regards,

Lorand Szentannai, on behalf of the preparations team

Updated information about Betzy production

Dear HPC User,

As mentioned previous week, the validation benchmarks have been stable, and we were ready to run and evaluate the site acceptance test. Unfortunately, the interconnect stability issues reoccured once again. 

We and the vendor have been running extensive tests since. The R&D department from the vendor of the interconnect released a new firmware yesterday afternoon, which was applied already yesterday evening and stress-tests immediately started. In order to be sure that the problem is resolved, several days of testing is needed.

Therefore, we have to postpone the production yet again with a week. Current production estimate is end of week 47.

We can assure you that we are very eager to have the system 100% stabilized and in production and everybody involved in the project (be it from Sigma2, the Metacenter, or vendor) is working intensively with this.

Thank you for your understanding!

Best regards,

Lorand Szentannai, on behalf of the preparations team