OpenBSD File System Corruption - Partitions Not Mounting
I came into work one morning and found out that my database server running OpenBSD with MySQL was no longer running and would not boot up. With a little bit of research I determined that the server went down two days before (over the weekend) and that the file system appeared to be corrupted on two partitions. A co-worker found out that the server’s primary channel on the motherboard was damage. So switched the hard drive over to the secondary channel. This I thought was odd but appeared to be true. I looked at the server and tried to get it to boot up myself. No luck as I was receiving two similar error messages as shown below. I apologize for it not being exact I was in a hurry and panicking to get the server back up and running.
Error Messages - (approximate message)
/dev/rwd0e (/var/): incorrect block count I=36484 (6 should be 0) (corr.) 15025024 DUP I=3515543.
/dev/rwd0g (/usr/): incorrect block count I=36484 (6 should be 0) (corr.) 15025024 DUP I=3515543.
Run fsck_ffs manually.
Solution
After moving the hard drive over to the secondary channel I was prompted at boot “Enter pathname of shell or RETURN for Sh:”. Now due to the fact that my CLI was in the damanged partition (/usr/local/bin/bash) I entered at the prompt “/bin/csh” without quotes and pressed enter. Now that I was into the system I ran “fsck_ffs /dev/rwd0e” and “fsck_ffs /dev/rwd0g” without quotes multiple times. It took me roughly 4 hours to resolve the problem. There was a lot of damage but nothing major or anything that was important. The database data was not affected except for a one log file.
As for how to use fsck_ffs after you run the command I just answered each questions it prompted me for. It’s quite interactive. I do ad might that I wasn’t sure what to say on some of them and wish I had a better understand of it. If anyone knows of good material that explains the fsck_ffs better and how the whole BSD file system works, please let me know.
Thanks to users in the IRC channels for OpenBSD
Saying all this I have to say thanks to all the people on IRC EFNet and Freenode in the channel #OpenBSD. Thanks to a few comments I was able to understand how to use fsck_ffs. Prior to this event I never have worked with fsck_ffs before. Again thanks everyone!
