Fixing A Broken Sudoers File on an Amazon’s EC2
I was working on getting our CI server set up and wanted to grant Jenkins the ability to run a command as root. Pretty standard stuff really. I ran sudo visudo and discovered that it had dropped me into Nano. Being an editor snob, this was unacceptible. I immediately quit out of the weak little thing.
The next few moments were where I made my fatal mistake. I ran my own, personally preferred, more powerful, edtior (as root) on /etc/sudoers, made a few changes and exited my editor. I tested the changes and low-and behold I was met with the message:
"parse error in /etc/sudoers"
‘Ok’ I thought. I tried to edit /etc/sudoers again.
"parse error in /etc/sudoers"
Oh, er…well, now I can no longer use sudo. Ok, how about becoming root. Oh, this is an Ubuntu instance. root has no password that I can use. Can’t ssh into the box as root either.
Damn. I’m stuck.
The solutions I found on Google all talked about booting the machine from a Live CD, mounting the disk with the sudoers file and fixing it. This isn’t possible on EC2. What was possible though is attaching the root volume (an EBS instance) to another EC2 instance and then editing the file. After some reading and expirementation, I ended up doing the following:
1. I wrote down the following:
- instance ID of the EC2 instance
- a list of each mounted EBS volume and the device it is mounted as.
2. Shutdown (but do not termiante) the EC2 instance.
3. Detach the EBS volume that has the bad sudoers file.
4. Attach and then Mount the EBS volume on another EC2 instance.
- Attaching the volume is done in the AWS Management Console
- Once attached, I noted the device name (in my case sdh) and mounted it from within the running instance:
sudo mkdir /mnt/othersudo mount /dev/xvdh /mnt/other
5. Edit the sudoers file, fixing the syntax error.
6. Unmount and then Detach the EBS volume from the alternate EC2 instance.
sudo umount /mnt/other
7. Re-attach the EBS volume to the original EC2 instance.
8. Start the original EC2 instance and confirm that the sudoer’s file is valid.
Once I had figured out these steps it took me about 30 minutes of time to go through the steps and I had a working instance again.
Conclusion
The moral of the story is not to repeat any of this, rather it is to always use visudo. Here’s how to do so using a custom editor:
@EDITOR=my-editor sudo -E visudo@