(Update at 19:50, August 12)
The file system consistency checks (fsck) on the volumes of the affected instances were completed at approximately 19:30 on August 12.
All recovery work related to this incident has now been completed, and the mdxII system is fully operational.
We sincerely apologize for the prolonged inconvenience and concern this may have caused.
—-
(Update at 18:00, August 12)
Investigation and initial response to this incident have been completed.
[Summary of the Incident]
On August 12, following hardware replacement work, it was confirmed that this issue began occurring at approximately 14:24.
As a result, for instances that were running at 14:24, the following conditions occurred:
Unable to read or write files
Unable to access instances via SSH
Additionally, it was not possible to launch new instances during this period.
We sincerely apologize for the inconvenience caused.
[Current Status]
As of 17:10, August 12, the restart of affected instances has been completed, and the issue has been resolved.
Launching new instances is also now possible.
[Next Steps]
We are currently performing file system integrity checks (fsck) on the volumes of the affected instances.
This work is expected to be completed around 19:30 today, and we will provide an update once it is finished.
We apologize for the extended disruption and appreciate your patience and understanding as we continue to address the matter.
—-
As of August 12, 4:30 PM, we have confirmed that file system I/O errors are occurring on some instances in the mdxII OpenStack environment.
Similar issues have been reported in multiple projects, and we are currently investigating the cause.
If you encounter any irregularities or errors in file read/write operations on your instances, please contact our emergency support desk immediately:
mdxII Emergency Support Desk
mdx2-support@hpc.cmc.osaka-u.ac.jp
Please note that we may conduct emergency maintenance if deemed necessary.
We sincerely apologize for the inconvenience this may cause and appreciate your understanding and cooperation.