How to recover from chmod 0 /bin/chmod

Recently, user garryjp posted a couple of Linux certification exam questions both here at LQ and on StackExchange. One of them, asked at SE, catched my attention.
Quote:

How would you recover from Code:sudo chmod 0000 /bin/chmodon a running system? Describe all possible alternative options you would use to recover from this command.

There were quite a few solutions suggested in the comments. Not all of them where following the requirement "on a running system" though. So with all due respect to the guys at SE who commented on this, I'll try to recap them here. The actual chmod command can be /bin/chmod on some (older) systems, or /usr/bin/chmod on others, so I'll be referring to it as $CHMOD from now on. You may assign
Code:CHMOD=/usr/bin/chmodbefore trying out the code bits below.

So here it goes.

1. Fix file permissions directly.
   • Another language interface to chmod(2) syscall
     The actual task of changing file permissions is being done by system call chmod(2). The command chmod(1) is just a command line interface to it. So we'll just use another interface to that syscall.
     
     The most straightforward solution is to write, compile and execute a small C program using examples in the chmod(3p) manpage.
     
     The easiest solution probably is this Perl one-liner though:
     Code:perl -e "chmod 0755,"'$CHMOD'"Most languages, both scripting and compiled, include some means of changing file metadata, either in their core parts or in a library/module. Perl has two advantages going for it. 1) chmod is part of the core language, so the syntax is very concise; 2) its ubiquity: Perl is installed on pretty much every Linux system.
     
     PHP may compete on the first account, but not on the second:
     Code:php -r "chmod('$CHMOD',0755);"Python may compete on the second, but not on the first. Besides, you'll need to figure out if the system in question has python2 or python3 or both and adjust accordingly:
     Code:python2 -c "import os;os.chmod('$CHMOD',0755)"Code:python3 -c "import os;os.chmod('$CHMOD,0o755)"Ruby?
     Code:ruby -r fileutils -e "FileUtils.chmod 0755,'$CHMOD'"Tcl? Well, Tcl is rather verbose comparing to others. Besides, it doesn't provide for one-liners, so you'll need to run tclsh interactively, or write a small script:
     Code:#!/usr/bin/tclsh
     file attributes /usr/bin/chmod -permissions 0755Lua is an interesting case. It's conceived as an embedded interpreter for other applications, so it doesn't include much by default. There are external packages like lua-fs that provide required functionality. Luckily, rpm includes Lua interpreter together with everything needed. I tend to have rpm installed even on Debian-based systems for purposes of inspecting, unpacking and converting RPM packages.
     Code:rpm -E "%{lua:posix.chmod('$CHMOD','755')}"
   • ACL
     Code:getfacl /usr/bin/chown|setfacl --set-file=- $CHMODor just
     Code:setfacl -m u::rx $CHMODAlso see below NFS ACLs.
   • Use package manager facilities
     Some package management systems provide means of either restoring file metadata to the state they were at the installation time:
     Code:rpm --restore $(rpm -qf $CHMOD)or overriding them:
     Code:dpkg-statoverride --update --add root root 755 $CHMOD
     dpkg-statoverride --remove $CHMOD
   • Edit inode
     This one is a heavyweight ;). Well, for completeness sake. Things we'll need to know are 1) FS type and device of the FS where $CHMOD is located (df -T $CHMOD) and 2) inode number of $CHMOD (ls -i /usr/bin/chmod). Then it's the task of a bin/hex editor or a FS debugging tool. E.g.
     
     debugfs for ext2/3/4
     Don't confuse this ext2/3/4 debugger with in-kernel pseudo-FS also named debugfs!
     
     Actually, I couldn't make it work for me. Neither this:
     Code:debugfs -wR "sif $CHMOD mode 755" $(df $CHMOD|sed '$!d;s/ .*//')nor that:
     Code:debugfs -wR "sif <$(stat -c%i $CHMOD)> mode 755" $(df $CHMOD|sed '$!d;s/ .*//')Any thoughts?
     
     xfs_db for xfs
     Well, this one doesn't work on a running system because the FS must be remounted read-only, and it won't allow me to do this saying "mount point is busy". I didn't pursue it any further.
     Code:fs=$($CHMOD|sed '$!d;s/ .*//')
     mount -o remount,ro $fs
     xfs_db -i -x \
     -c 'type inode' \
     -c "inode $(stat -c%i $CHMOD)" \
     -c 'write mode 755' $fs
   • Do it remotely
     The first I can think of is
     Code:echo chmod 755 $CHMOD|sftp -b - root@affected_hostThis obviously won't work if root is not allowed to ssh, but I'm pretty sure there are many other ways to accomplish this remotely either from the command line or e.g. via an administrative web interface.
2. Copy content to another file setting the permissions
   Any tool capable of moving files around and setting permissions in the process can be abused for the side effect.
   
   The most obvious candidate is install:
   Code:install $CHMOD /tmp/Or just cp over another file that happens to have the right permissions:
   Code:cp /bin/chown /tmp/chmod
   cp $CHMOD /tmp/There are many others as well. E.g. rsync
   Code:rsync --chmod=755 $CHMOD /tmp/Or even tar:
   Code:tar -C "${CHMOD%/*}" -c --mode=755 "${CHMOD##*/}"|tar xMore adventurous among us may even attempt writing the copy back over the original file:
   Code:tar -cP --mode=755 $CHMOD|tar xOr how about git?
   Code:mkdir repo
   cd repo
   git init
   cp $CHMOD .
   sudo git update-index --add --chmod=+x chmod
   git checkout chmodCopy to a filesystem that makes the problem disappear
   A variation of the previous. We just need a filesystem that enforces certain permissions for all files. FAT is an obvious candidate, but many others will do as well.
   Code:mkdosfs -C /tmp/fat.img 180
   mount /tmp/fat.img /mnt
   cp $CHMOD /mntMany filesystem types have mount options to set permissions for all files. For some they only work as umask by combining the assigned value with permission bits for the file to make effective permissions. But for some, mostly for such filesystems as FAT that don't keep permission bits for every file, those options set effective permissions itself. There are even means to enforce file permissions for some types of network FS like CIFS or DAVFS. Interestingly, NFSv4 has its own set of ACLs, completely independent of POSIX ACLs, so if we've copied the file to an NFS share, we can then
   Code:nfs4_setfacl -a A::OWNER@:RX copied_file
3. Restore/recreate the file
   E.g. by reinstalling the package
   Code:dnf reinstall $(rpm -qf $CHMOD)Code:apt reinstall $(dpkg-query -S bin/chmod|cut -d: -f1)But anything goes as long as you'll get an intact copy of the damaged file: backup, ZFS/btrfs snapshots, LVM2 snapshots, VM snapshots if your system is being run inside a VM, an ISO image of the install disk, and so on.
4. Use a working chmod from another place
   • On-system
     I tend to always have busybox installed, just in case. So
     Code:busybox chmod 755 $CHMODOn Debian-based systems at least, another possibility is extracting chmod from an initrd image.
     Code:mkdir /tmp/initrd
     unmkinitramfs /boot/initrd.img /initrd
     /tmp/initrd/usr/bin/chmod 755 $CHMODUnfortunately, initrd images created with dracut don't include chmod, so if your distro uses dracut (most RPM-based distros, e.g. Fedora, RHEL, CentOS, Oracle, SLE, OpenSUSE) this won't work for you. OTOH, dracut includes the lsinitrd script which I like better than unmkinitramfs. So I can imagine something like this on a Debian-based distro:
     Code:apt install dracut-core
     lsinitrd --unpack /boot/initrd.img usr/chmod
   • Reboot to a rescue environment
     The title says it all. E.g., on a recent Debian-based system
     Code:sudo apt install grml-rescueboot
     sudo update-grml-rescueboot
     sudo update-grub
     sudo rebootOlder Debian systems don't have the update-grml-rescueboot command, so you'll need to download a Grml ISO image manually and put it to /boot/grml/. After reboot into the rescue environment it's a matter of mounting the right FS, so the df command from the previous entry will be a useful prerequisite for this one as well.
   • Copy a working chmod from another system
     It even needn't to be the same release as on the affected host. Basically, chmod depends only on glibc, and all we have to check is 1) the architecture is compatible (a 64-bit binary cannot be executed on a 32-bit system), and 2) the chmod we'll copy wasn't linked against a newer glibc than what we have on the affected host. An older chmod will do. Actually, I tried this by copying the chmod binary from a 32-bit CentOS 6 to a 64-bit CentOS 8. Worked like a charm. It goes without saying that I already have had the 32-bit glibc in place on the CentOS 8 system:
     Code:dnf install glibc32Before attempting this, check the shared object dependencies of both chmod versions with ldd -v. And this bring us to the last item.
5. Run through dynamic linker
   chmod (as almost everything nowadays) is dynamically linked. In the output of ldd you'll see a line with ld-linux.so (or ld-lsb.so or ld.so). This is the dynamic linker. Now, recall how we run a script. There are two ways to do it.
   
   One is to put shebang as the first line, make the file executable and run it by its name. So, taking a shell script as an example,
   Code:sed -i '1i #!/bin/sh' myscript
   chmod +x myscript
   ./myscriptAnother way is to call the script as an argument to the script intepreter
   Code:sh myscriptIn this case setting the executable bit for myscript is unnecessary.
   
   For dynamically linked ELF binaries, there is a certain similarity here. Obviously, we don't use shebang lines for them, but we're supposed to make them executable and then run by name. OTOH, we also can call them as arguments to dynamic linker, even if they don't have the executable bit set. Which gives
   Code:$(ldd $CHMOD|awk '/\/ld-/,$0=$1') $CHMOD 755 $CHMOD

Comments are welcome. Have I forgotten anything?