Mariadb won't start now it's a slave
by glennbtn from LinuxQuestions.org on (#542CS)
Hi All
I have setup a master slave using this document https://dzone.com/articles/how-to-se...ve-replication
Once completed the databases synced fine with no issues. I need to do a reboot of the slave after a couple of days and since then mariadb wont start and nothing shows in the log as it's not even being written to. If I run systemctl status mariadb.service this is what I get
Failed to dump process list, ignoring: Input/output error
* mariadb.service - MariaDB 10.1.44 database server
Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Fri 2020-05-29 10:04:06 BST; 12min ago
Docs: man:mysqld(8)
https://mariadb.com/kb/en/library/systemd/
Process: 8944 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=exited, status=1/FAILURE)
Process: 8860 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= || VAR=`/usr/bin/galera_recovery`; [ $? -eq 0 ] && systemctl set-environment _WSREP_START_POSITION=$VAR || exit 1 (code=exited, sta
Process: 8858 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
Process: 8857 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
Main PID: 8944 (code=exited, status=1/FAILURE)
Status: "MariaDB server is down"
Tasks: 0 (limit: 4915)
CGroup: /system.slice/mariadb.service
May 29 10:04:06 stanslave3 mysqld[8944]: 2020-05-29 10:04:06 140379451227264 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
May 29 10:04:06 stanslave3 mysqld[8944]: 2020-05-29 10:04:06 140379451227264 [Note] Plugin 'FEEDBACK' is disabled.
May 29 10:04:06 stanslave3 mysqld[8944]: 2020-05-29 10:04:06 140379451227264 [ERROR] Could not open mysql.plugin table. Some plugins may be not loaded
May 29 10:04:06 stanslave3 mysqld[8944]: 2020-05-29 10:04:06 140379451227264 [ERROR] Unknown/unsupported storage engine: InnoDB
May 29 10:04:06 stanslave3 mysqld[8944]: 2020-05-29 10:04:06 140379451227264 [ERROR] Aborting
May 29 10:04:06 stanslave3 systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
May 29 10:04:06 stanslave3 systemd[1]: mariadb.service: Failed to kill control group /system.slice/mariadb.service, ignoring: Input/output error
May 29 10:04:06 stanslave3 systemd[1]: mariadb.service: Failed to kill control group /system.slice/mariadb.service, ignoring: Input/output error
May 29 10:04:06 stanslave3 systemd[1]: mariadb.service: Failed with result 'exit-code'.
May 29 10:04:06 stanslave3 systemd[1]: Failed to start MariaDB 10.1.44 database server.
journalctl -xe displays
May 29 10:21:57 stanslave3 mysqld[9067]: 2020-05-29 10:21:57 140434246253696 [Warning] Can't create test file /var/lib/mysql/stanslave3.lower-test
May 29 10:21:57 stanslave3 mysqld[9067]: 2020-05-29 10:21:57 140434246253696 [ERROR] mysqld: File '/var/lib/mysql/aria_log_control' not found (Errcode: 13 "Permission denied")
May 29 10:21:57 stanslave3 mysqld[9067]: 2020-05-29 10:21:57 140434246253696 [ERROR] mysqld: Got error 'Can't open file' when trying to use aria control file '/var/lib/mysql/aria_log_control'
May 29 10:21:57 stanslave3 mysqld[9067]: 2020-05-29 10:21:57 140434246253696 [ERROR] Plugin 'Aria' init function returned error.
May 29 10:21:57 stanslave3 mysqld[9067]: 2020-05-29 10:21:57 140434246253696 [ERROR] Plugin 'Aria' registration as a STORAGE ENGINE failed.
May 29 10:21:57 stanslave3 mysqld[9067]: 2020-05-29 10:21:57 140434246253696 [Note] InnoDB: innodb_empty_free_list_algorithm has been changed to legacy because of small buffer pool size. In order to use backoff, incre
May 29 10:21:57 stanslave3 mysqld[9067]: 2020-05-29 10:21:57 140434246253696 [Note] InnoDB: Using mutexes to ref count buffer pool pages
May 29 10:21:57 stanslave3 mysqld[9067]: 2020-05-29 10:21:57 140434246253696 [Note] InnoDB: The InnoDB memory heap is disabled
May 29 10:21:57 stanslave3 mysqld[9067]: 2020-05-29 10:21:57 140434246253696 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
May 29 10:21:57 stanslave3 mysqld[9067]: 2020-05-29 10:21:57 140434246253696 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
May 29 10:21:57 stanslave3 mysqld[9067]: 2020-05-29 10:21:57 140434246253696 [Note] InnoDB: Compressed tables use zlib 1.2.11
May 29 10:21:57 stanslave3 mysqld[9067]: 2020-05-29 10:21:57 140434246253696 [Note] InnoDB: Using Linux native AIO
May 29 10:21:57 stanslave3 mysqld[9067]: 2020-05-29 10:21:57 140434246253696 [Note] InnoDB: Using SSE crc32 instructions
May 29 10:21:57 stanslave3 mysqld[9067]: 2020-05-29 10:21:57 140434246253696 [Note] InnoDB: Initializing buffer pool, size = 128.0M
May 29 10:21:57 stanslave3 mysqld[9067]: 2020-05-29 10:21:57 140434246253696 [Note] InnoDB: Completed initialization of buffer pool
May 29 10:21:57 stanslave3 mysqld[9067]: 2020-05-29 10:21:57 140434246253696 [ERROR] InnoDB: ./ibdata1 can't be opened in read-write mode
May 29 10:21:57 stanslave3 mysqld[9067]: 2020-05-29 10:21:57 140434246253696 [ERROR] InnoDB: The system tablespace must be writable!
May 29 10:21:57 stanslave3 mysqld[9067]: 2020-05-29 10:21:57 140434246253696 [ERROR] Plugin 'InnoDB' init function returned error.
May 29 10:21:57 stanslave3 mysqld[9067]: 2020-05-29 10:21:57 140434246253696 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
May 29 10:21:57 stanslave3 mysqld[9067]: 2020-05-29 10:21:57 140434246253696 [Note] Plugin 'FEEDBACK' is disabled.
May 29 10:21:57 stanslave3 mysqld[9067]: 2020-05-29 10:21:57 140434246253696 [ERROR] Could not open mysql.plugin table. Some plugins may be not loaded
May 29 10:21:57 stanslave3 mysqld[9067]: 2020-05-29 10:21:57 140434246253696 [ERROR] Unknown/unsupported storage engine: InnoDB
May 29 10:21:57 stanslave3 mysqld[9067]: 2020-05-29 10:21:57 140434246253696 [ERROR] Aborting
May 29 10:21:57 stanslave3 systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
May 29 10:21:57 stanslave3 systemd[1]: mariadb.service: Failed to kill control group /system.slice/mariadb.service, ignoring: Input/output error
May 29 10:21:57 stanslave3 systemd[1]: mariadb.service: Failed to kill control group /system.slice/mariadb.service, ignoring: Input/output error
May 29 10:21:57 stanslave3 systemd[1]: mariadb.service: Failed with result 'exit-code'.
May 29 10:21:57 stanslave3 systemd[1]: Failed to start MariaDB 10.1.44 database server.
-- Subject: Unit mariadb.service has failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Unit mariadb.service has failed.
--
-- The result is RESULT.
May 29 10:22:01 stanslave3 cron[154]: (*system*) WRONG FILE OWNER (/etc/crontab)
May 29 10:22:01 stanslave3 cron[154]: (*system*sendmail) WRONG FILE OWNER (/etc/cron.d/sendmail)
May 29 10:22:01 stanslave3 cron[154]: (*system*popularity-contest) WRONG FILE OWNER (/etc/cron.d/popularity-contest)
May 29 10:22:01 stanslave3 cron[154]: (root) WRONG FILE OWNER (crontabs/root)
Can anyone advise, thanks
I have setup a master slave using this document https://dzone.com/articles/how-to-se...ve-replication
Once completed the databases synced fine with no issues. I need to do a reboot of the slave after a couple of days and since then mariadb wont start and nothing shows in the log as it's not even being written to. If I run systemctl status mariadb.service this is what I get
Failed to dump process list, ignoring: Input/output error
* mariadb.service - MariaDB 10.1.44 database server
Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Fri 2020-05-29 10:04:06 BST; 12min ago
Docs: man:mysqld(8)
https://mariadb.com/kb/en/library/systemd/
Process: 8944 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=exited, status=1/FAILURE)
Process: 8860 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= || VAR=`/usr/bin/galera_recovery`; [ $? -eq 0 ] && systemctl set-environment _WSREP_START_POSITION=$VAR || exit 1 (code=exited, sta
Process: 8858 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
Process: 8857 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
Main PID: 8944 (code=exited, status=1/FAILURE)
Status: "MariaDB server is down"
Tasks: 0 (limit: 4915)
CGroup: /system.slice/mariadb.service
May 29 10:04:06 stanslave3 mysqld[8944]: 2020-05-29 10:04:06 140379451227264 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
May 29 10:04:06 stanslave3 mysqld[8944]: 2020-05-29 10:04:06 140379451227264 [Note] Plugin 'FEEDBACK' is disabled.
May 29 10:04:06 stanslave3 mysqld[8944]: 2020-05-29 10:04:06 140379451227264 [ERROR] Could not open mysql.plugin table. Some plugins may be not loaded
May 29 10:04:06 stanslave3 mysqld[8944]: 2020-05-29 10:04:06 140379451227264 [ERROR] Unknown/unsupported storage engine: InnoDB
May 29 10:04:06 stanslave3 mysqld[8944]: 2020-05-29 10:04:06 140379451227264 [ERROR] Aborting
May 29 10:04:06 stanslave3 systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
May 29 10:04:06 stanslave3 systemd[1]: mariadb.service: Failed to kill control group /system.slice/mariadb.service, ignoring: Input/output error
May 29 10:04:06 stanslave3 systemd[1]: mariadb.service: Failed to kill control group /system.slice/mariadb.service, ignoring: Input/output error
May 29 10:04:06 stanslave3 systemd[1]: mariadb.service: Failed with result 'exit-code'.
May 29 10:04:06 stanslave3 systemd[1]: Failed to start MariaDB 10.1.44 database server.
journalctl -xe displays
May 29 10:21:57 stanslave3 mysqld[9067]: 2020-05-29 10:21:57 140434246253696 [Warning] Can't create test file /var/lib/mysql/stanslave3.lower-test
May 29 10:21:57 stanslave3 mysqld[9067]: 2020-05-29 10:21:57 140434246253696 [ERROR] mysqld: File '/var/lib/mysql/aria_log_control' not found (Errcode: 13 "Permission denied")
May 29 10:21:57 stanslave3 mysqld[9067]: 2020-05-29 10:21:57 140434246253696 [ERROR] mysqld: Got error 'Can't open file' when trying to use aria control file '/var/lib/mysql/aria_log_control'
May 29 10:21:57 stanslave3 mysqld[9067]: 2020-05-29 10:21:57 140434246253696 [ERROR] Plugin 'Aria' init function returned error.
May 29 10:21:57 stanslave3 mysqld[9067]: 2020-05-29 10:21:57 140434246253696 [ERROR] Plugin 'Aria' registration as a STORAGE ENGINE failed.
May 29 10:21:57 stanslave3 mysqld[9067]: 2020-05-29 10:21:57 140434246253696 [Note] InnoDB: innodb_empty_free_list_algorithm has been changed to legacy because of small buffer pool size. In order to use backoff, incre
May 29 10:21:57 stanslave3 mysqld[9067]: 2020-05-29 10:21:57 140434246253696 [Note] InnoDB: Using mutexes to ref count buffer pool pages
May 29 10:21:57 stanslave3 mysqld[9067]: 2020-05-29 10:21:57 140434246253696 [Note] InnoDB: The InnoDB memory heap is disabled
May 29 10:21:57 stanslave3 mysqld[9067]: 2020-05-29 10:21:57 140434246253696 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
May 29 10:21:57 stanslave3 mysqld[9067]: 2020-05-29 10:21:57 140434246253696 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
May 29 10:21:57 stanslave3 mysqld[9067]: 2020-05-29 10:21:57 140434246253696 [Note] InnoDB: Compressed tables use zlib 1.2.11
May 29 10:21:57 stanslave3 mysqld[9067]: 2020-05-29 10:21:57 140434246253696 [Note] InnoDB: Using Linux native AIO
May 29 10:21:57 stanslave3 mysqld[9067]: 2020-05-29 10:21:57 140434246253696 [Note] InnoDB: Using SSE crc32 instructions
May 29 10:21:57 stanslave3 mysqld[9067]: 2020-05-29 10:21:57 140434246253696 [Note] InnoDB: Initializing buffer pool, size = 128.0M
May 29 10:21:57 stanslave3 mysqld[9067]: 2020-05-29 10:21:57 140434246253696 [Note] InnoDB: Completed initialization of buffer pool
May 29 10:21:57 stanslave3 mysqld[9067]: 2020-05-29 10:21:57 140434246253696 [ERROR] InnoDB: ./ibdata1 can't be opened in read-write mode
May 29 10:21:57 stanslave3 mysqld[9067]: 2020-05-29 10:21:57 140434246253696 [ERROR] InnoDB: The system tablespace must be writable!
May 29 10:21:57 stanslave3 mysqld[9067]: 2020-05-29 10:21:57 140434246253696 [ERROR] Plugin 'InnoDB' init function returned error.
May 29 10:21:57 stanslave3 mysqld[9067]: 2020-05-29 10:21:57 140434246253696 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
May 29 10:21:57 stanslave3 mysqld[9067]: 2020-05-29 10:21:57 140434246253696 [Note] Plugin 'FEEDBACK' is disabled.
May 29 10:21:57 stanslave3 mysqld[9067]: 2020-05-29 10:21:57 140434246253696 [ERROR] Could not open mysql.plugin table. Some plugins may be not loaded
May 29 10:21:57 stanslave3 mysqld[9067]: 2020-05-29 10:21:57 140434246253696 [ERROR] Unknown/unsupported storage engine: InnoDB
May 29 10:21:57 stanslave3 mysqld[9067]: 2020-05-29 10:21:57 140434246253696 [ERROR] Aborting
May 29 10:21:57 stanslave3 systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
May 29 10:21:57 stanslave3 systemd[1]: mariadb.service: Failed to kill control group /system.slice/mariadb.service, ignoring: Input/output error
May 29 10:21:57 stanslave3 systemd[1]: mariadb.service: Failed to kill control group /system.slice/mariadb.service, ignoring: Input/output error
May 29 10:21:57 stanslave3 systemd[1]: mariadb.service: Failed with result 'exit-code'.
May 29 10:21:57 stanslave3 systemd[1]: Failed to start MariaDB 10.1.44 database server.
-- Subject: Unit mariadb.service has failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Unit mariadb.service has failed.
--
-- The result is RESULT.
May 29 10:22:01 stanslave3 cron[154]: (*system*) WRONG FILE OWNER (/etc/crontab)
May 29 10:22:01 stanslave3 cron[154]: (*system*sendmail) WRONG FILE OWNER (/etc/cron.d/sendmail)
May 29 10:22:01 stanslave3 cron[154]: (*system*popularity-contest) WRONG FILE OWNER (/etc/cron.d/popularity-contest)
May 29 10:22:01 stanslave3 cron[154]: (root) WRONG FILE OWNER (crontabs/root)
Can anyone advise, thanks