Since the system failure that occurred on October 5, we have confirmed an issue affecting some virtual machines (instances), where write operations result in errors, causing the entire virtual machine to enter a paused state and become inaccessible for connection and operation.
（Closed）mdx II System: Restart of Running Instances Due to System Issue (Oct. 5)
The issue occurs when writing data to an additional storage volume (Cinder volume) attached to a virtual machine. Virtual machines that do not have an additional volume attached are not affected by this issue.
When a virtual machine enters a paused state, the “Power State” displayed in the user portal will show either “Paused” or “Suspended.” On the other hand, no similar write errors have been observed at this time on boot disks (boot volumes), which contain the operating system, or on Lustre storage accessed directly from virtual machines.
For information regarding Cinder volumes, please refer to Section 4.1.11.1, “Using Cinder volumes,” in the User’s Manual (PDF). At this time, we have not confirmed any occurrences of this issue among users who are not using Cinder volumes.
User’s Manual (PDF): Section 4.1.11.1 Using Cinder volumes
If you are unsure whether your environment is affected, please contact the mdx II Operations Office.
Current Measures and Request to Users
We have temporarily suspended the creation of new additional volumes until this issue is resolved.
We also ask users to refrain from using existing additional volumes for the time being.
As a temporary mitigation measure until full recovery is achieved, we have introduced a mechanism that checks the status of virtual machines every 10 minutes and automatically reboots any virtual machine detected in the “Paused” state. While virtual machines can continue to be used, please note that running processes may be interrupted when an automatic restart occurs.
Future Actions
To resolve this issue, we plan to restart virtual machines that are using additional volumes.
The schedule for this work will be announced as soon as it is finalized.
We sincerely apologize for the inconvenience and trouble this issue has caused.
We appreciate your understanding and cooperation.