Tumbleweed update "oops" problem ... or something else
by rnturn from LinuxQuestions.org on (#4YZ7W)
I got a notice in this morning's email (via a job running in cron) that there were updates available. (Sigh... seems like only yesterday.)
I logged into the box sending the alert and ran "zypper dup" first thing and noticed no errors or warnings while it ran (well, nothing except some odd messages about symlinks being wrong for some reason for items I'd never have any need to be changing). The updates took the system to version ID "20200201" and required a reboot. Shouldn't be a probl... oh, wait... the NFS clients that typically auto remount after the boot failed to remount this time. Issuing "service --status-all" on the updated system showed that the "nfs-server" service was, indeed, down. So was NTP. I was able to restart NTP from the command line. Restarting NFS was a little different. After issuing "service nfs-server start" and seeing no error messages I figured that the problem has been resolved. However, the NFS clients were still unable to remount. "/var/log/messages" didn't contain any blatant errors that mentioned NFS.
Going back to "/var/log/messages" to check if I missed something, I ran across this message while scrolling back in time closer to the post-zypper reboot:
Code:... nscd[1377]: 1377 checking for monitored file `/etc/services': No such file or directoryWell, that surely can't be. Except I checked and, sure enough, no "/etc/services" file.
I restored it using a copy from another Tumbleweed system (rather than spend the afternoon going through old backups) and restarted the "nfs-server" service and this time it started successfully, clients could re-mount, etc. so everything's back to normal. (At least what passes for normal around here.)
NFS was working on the system (the one where "/etc/services" disappeared) right up until the moment it was rebooted. The previous reboot was back on Jan 21 (another zypper-related requirement). There is no occurrence of that "No such file" message in the pre-Jan_21 version of "/var/log/messages". The only time it's appeared was right after today's zypper+reboot. So it must have disappeared sometime after NFS started up on Jan 21.
Nothing that explicitly mentions "/etc/services" appears in "/var/log/zypper.log".
It's nice that nscd(8) views "/etc/services" as a "monitored file" but it'd be nice if there was a way to a.) control the frequency of the file checks (i.e., daily) and, more importantly, b.) be notified about any missing file (or files should that ever occur). In this case it might have been a problem as the server with the missing file is the Postfix/IMAP server so it might have been unable to communicate that there was a problem had one occurred. Or is that not true?
Any ideas as to what may have gone wrong and could result in this critical file disappearing from the system? Modification to "/etc/services" that failed in a big way, perhaps? Or just Tumbleweed being Tumbleweed?
TIA for any insights, conspiracy theories, etc....
Addendum: While typing up this thread, I noticed that Postfix had not started at the last reboot either. Starting it after the "/etc/service" restore worked fine.
I logged into the box sending the alert and ran "zypper dup" first thing and noticed no errors or warnings while it ran (well, nothing except some odd messages about symlinks being wrong for some reason for items I'd never have any need to be changing). The updates took the system to version ID "20200201" and required a reboot. Shouldn't be a probl... oh, wait... the NFS clients that typically auto remount after the boot failed to remount this time. Issuing "service --status-all" on the updated system showed that the "nfs-server" service was, indeed, down. So was NTP. I was able to restart NTP from the command line. Restarting NFS was a little different. After issuing "service nfs-server start" and seeing no error messages I figured that the problem has been resolved. However, the NFS clients were still unable to remount. "/var/log/messages" didn't contain any blatant errors that mentioned NFS.
Going back to "/var/log/messages" to check if I missed something, I ran across this message while scrolling back in time closer to the post-zypper reboot:
Code:... nscd[1377]: 1377 checking for monitored file `/etc/services': No such file or directoryWell, that surely can't be. Except I checked and, sure enough, no "/etc/services" file.
I restored it using a copy from another Tumbleweed system (rather than spend the afternoon going through old backups) and restarted the "nfs-server" service and this time it started successfully, clients could re-mount, etc. so everything's back to normal. (At least what passes for normal around here.)
NFS was working on the system (the one where "/etc/services" disappeared) right up until the moment it was rebooted. The previous reboot was back on Jan 21 (another zypper-related requirement). There is no occurrence of that "No such file" message in the pre-Jan_21 version of "/var/log/messages". The only time it's appeared was right after today's zypper+reboot. So it must have disappeared sometime after NFS started up on Jan 21.
Nothing that explicitly mentions "/etc/services" appears in "/var/log/zypper.log".
It's nice that nscd(8) views "/etc/services" as a "monitored file" but it'd be nice if there was a way to a.) control the frequency of the file checks (i.e., daily) and, more importantly, b.) be notified about any missing file (or files should that ever occur). In this case it might have been a problem as the server with the missing file is the Postfix/IMAP server so it might have been unable to communicate that there was a problem had one occurred. Or is that not true?
Any ideas as to what may have gone wrong and could result in this critical file disappearing from the system? Modification to "/etc/services" that failed in a big way, perhaps? Or just Tumbleweed being Tumbleweed?
TIA for any insights, conspiracy theories, etc....
Addendum: While typing up this thread, I noticed that Postfix had not started at the last reboot either. Starting it after the "/etc/service" restore worked fine.