Betzy file system issue

We are experiencing some issues with the Betzy file system. Please check back here for updates.

UPDATE 10-08-2021: We have finally resolved the problem with fileserver, one of the storage controller is rebooted.

UPDATE 08-08-2021: Filesystem is again having problem, we have discovered that one of the fileserver missed network connection, we are woking on the issue.

UPDATE 07-08-2021: There is again a problem with the login nodes.

UPDATE 06-08-2021: We may have identified the root cause of the problem (high usage on the storage servers, particularly one storage server was 95% used). We have cleaned up data older than 21 days under /cluster/user/work. Usage is now down to about 70 %. The file system seems responsive for the moment and we do not see new errors in internal logs. We are monitoring the file system during the coming days before we consider it resolved. Again, we are sorry for the inconvenience this has caused for the last ~ 2 weeks.

UPDATE 04-08-2021: The problem is still not solved, and the performance and accessibility of the file system varies a lot. Also some jobs crash. We are in contact with the vendors about this.

UPDATE 2 02-08-2021: Login very slow/hanging again. Possibly other activities affected as well.

UPDATE 02-08-2021: Login to all frontends was working smoothly this morning. We are still watching the file system in case the problem reoccurs.

UPDATE 28-07-2021: we are still investigating the issue(s), the system might be unstable until the beginning of the next week.

We are very sorry for the inconvenience this might cause to you.