« Photography: Fractal Food: Self-Similarity on the Supermarket Shelf | Main | Linux: /var-acious Dictionary Spam »
Wednesday, March 23, 2005
Linux: Advance to the Rear
After yet another disastrous ext3 filesystem journal failure today, I decided to pack it in and revert the mission-critical /server filesystem to ext2. I accomplished this by halting the system, rebooting specifying a kernel boot option of "single", then converting the ext3 filesystem (/dev/sda9) to ext2 with:
umount /dev/sda9
/sbin/tunefs -O ^has_journal /dev/sda9
/sbin/e2fsck /dev/sda9
vi /etc/fstab
# Change filesystem type for
# /dev/sda9 from ext3 to ext2
/sbin/reboot
So far, I've only done this on the main server. If all goes well, I'll propagate it to the backup server in a few days. Of course, when the server farm is fully functional, there won't be any distinction between main and backup servers, but while running down these kinds of bugs, it's nice to be able to draw a line between them.
Update: Despite this change to the /server filesystem, the server died again around 05:00 UTC on March 24th with the same "journal commit error". Enough--I converted all the filesystems back to ext2 to see if that will put an end to this nonsense. The root filesystem was already ext2 due to reports of diasters attempting to recover ext3 roots, so the only filesystem which was a bit tricky to convert was /usr. I booted from the Fedora Core 3 Rescue CD-ROM and converted /usr and edited the filesystem table in its rescue mount tree location of /mnt/sysimage/etc/fstab, then rebooted into normal mode--so far, so good.
Posted at March 23, 2005 02:32