A full disk breaks everything politely at first, then all at once. Five minutes of detective work usually finds the culprit.
Steps
Connect by SSH as root or a user with sudo. If you can't log in because the disk is full, use the IPMI console instead (see our article on reaching IPMI).
- See which filesystem is full:
df -h. Note the one at or near 100% in the Use% column. - Check for inode exhaustion too:
df -i. Millions of tiny files can fill a disk that looks half empty. - Find the biggest top-level directories:
sudo du -xh --max-depth=1 / | sort -h | tail -15. - Drill into the biggest one the same way, for example
sudo du -xh --max-depth=1 /var | sort -h | tail -15. Repeat until you find the files. Check the usual suspects:
- logs:
sudo du -sh /var/log - systemd journal:
journalctl --disk-usage - mail queue and spool:
sudo du -sh /var/spool - old backups and core dumps
- logs:
- Trim the journal if it is large:
sudo journalctl --vacuum-size=500M. - Remove or compress old logs and backups you no longer need. Do not delete a log file that a running service has open; truncate it instead with
sudo truncate -s 0 /var/log/example.log. - Check that the space is free:
df -h. If you deleted a file and the space didn't come back, a process still holds it open; find it withsudo lsof +L1, then restart that service. - Prevent a repeat. Make sure
logrotateis running and set up monitoring for disk use.
If it didn't work
- If nothing obvious is large, but the disk is still full, the data may be on a different mount point; compare
df -hwithlsblk. - If a drive is reporting errors rather than filling up, see our article on failing drives.
- Need more capacity? Open a ticket in the portal, set Department to Technical Support, and include your Main IP, the output of
df -h, and what you've already cleared.