Drives fail the same way every time: quietly, then at the worst moment. Back up first, ask questions second.
Before you start
Back up anything you can't lose. Do that before you run any diagnostics, because tests read the disk heavily. You'll need root access and your Main IP.
Steps
- Look for signs. Check the system log for disk errors:
sudo dmesg | grep -i -E 'i/o error|ata|nvme|sector'. - Check SMART health. Install
smartmontoolsif needed, then runsudo smartctl -a /dev/sda(use/dev/nvme0for NVMe). Look for SMART overall-health self-assessment test result and for reallocated, pending, or uncorrectable sectors above zero. - Check RAID status. For software RAID run
cat /proc/mdstat. AU_or_Umeans a member is missing. - Copy your data off now if SMART or the logs show errors. Use
rsyncto another server. Open a ticket. Set Department to Technical Support. Include:
- your Main IP
- the drive device name
- the output of
smartctl -aandcat /proc/mdstat - the error lines from step 1
- Wait for our reply. We'll tell you whether the drive can be replaced on your server and what happens to your data.
- After any hardware work, check that everything is mounted and healthy:
df -handcat /proc/mdstat.
If it didn't work
- If the server halts at boot for a file system check, let it finish. Don't power it off mid-check.
- On clearance servers, we do not swap disks or other components. If yours is a clearance server, ask in the ticket about your options, such as moving to a standard configuration.
- On managed servers, we monitor the hardware too and may contact you first. Reply to that ticket instead of opening a new one.