Mariadb crashed. Does not restart but produces no errors.
Any help or insights would be appreciated.
I’ve had openemr 7.0.2 up and running for several months.
I’ve been doing a DIY backup by stopping the the server and copying the files in /var/lib/mysql somewhere safe.
I decided to swtich over to the recommended backup method using mysqldump.
It wasn’t installed so I did
apt install mariadb-client-10.3
which completed without a problem.
I was trying to get the syntax correct for mysqldump (it never did run) eg
mysqldump --databases openemr --opt --quote-names --hex-blob -u drb -pXXXXX | gzip > $dest/oemr.sql.$tstamp.gz
when the sever crashed.
What’s very perplexing is that after restarting mariadb, the status says it’s running but it’s not.
[user@u2004 /var/www/drb/test]systemctl restart mariadb
[user@u2004 /var/www/drb/test]systemctl status mariadb
● mariadb.service - LSB: Start and stop the mysql database server daemon
Loaded: loaded (/etc/init.d/mariadb; generated)
Drop-In: /etc/systemd/system/mariadb.service.d
└─migrated-from-my.cnf-settings.conf
Active: active (exited) since Mon 2024-10-21 11:09:50 PDT; 11s ago
Docs: man:systemd-sysv-generator(8)
Process: 19284 ExecStart=/etc/init.d/mariadb start (code=exited, status=0/SUCCESS)
Oct 21 11:09:50 u2004 systemd[1]: Starting LSB: Start and stop the mysql database server daemon...
Oct 21 11:09:50 u2004 systemd[1]: Started LSB: Start and stop the mysql database server daemon.
===
[user@u2004 /var/www/drb/test]mariadb
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)
=== but there is not an mariadb in the process table
[user@u2004 /var/www/drb/test]ps -ef | grep maria
root 19303 3140 0 11:10 pts/3 00:00:00 grep --color=auto maria
[user@u2004 /var/www/drb/test]ps -ef | grep mysql
root 19305 3140 0 11:10 pts/3 00:00:00 grep --color=auto mysql
=== journalctl looks fine
[user@u2004 /var/www/drb/test] journalctl -n 10 -u mariadb.service
-- Logs begin at Sun 2022-01-23 19:03:49 PST, end at Mon 2024-10-21 11:09:50 PDT. --
Oct 21 06:46:34 u2004 systemd[1]: Stopping LSB: Start and stop the mysql database server daemon...
Oct 21 06:46:34 u2004 systemd[1]: mariadb.service: Succeeded.
Oct 21 06:46:34 u2004 systemd[1]: Stopped LSB: Start and stop the mysql database server daemon.
Oct 21 06:57:16 u2004 systemd[1]: Starting LSB: Start and stop the mysql database server daemon...
Oct 21 06:57:16 u2004 systemd[1]: Started LSB: Start and stop the mysql database server daemon.
Oct 21 07:12:13 u2004 systemd[1]: Stopping LSB: Start and stop the mysql database server daemon...
Oct 21 07:12:13 u2004 systemd[1]: mariadb.service: Succeeded.
Oct 21 07:12:13 u2004 systemd[1]: Stopped LSB: Start and stop the mysql database server daemon.
Oct 21 11:09:50 u2004 systemd[1]: Starting LSB: Start and stop the mysql database server daemon...
Oct 21 11:09:50 u2004 systemd[1]: Started LSB: Start and stop the mysql database server daemon.
=== nothing bad in syslog
grep mariadb /var/log/syslog
Oct 21 05:18:07 u2004 systemd[1]: mariadb.service: Succeeded.
Oct 21 05:56:55 u2004 systemd[1]: mariadb.service: Succeeded.
Oct 21 06:02:52 u2004 systemd[1]: mariadb.service: Succeeded.
Oct 21 06:20:11 u2004 systemd[1]: mariadb.service: Succeeded.
Oct 21 06:31:36 u2004 systemd[1]: mariadb.service: Succeeded.
Oct 21 06:46:34 u2004 systemd[1]: mariadb.service: Succeeded.
Oct 21 07:12:13 u2004 systemd[1]: mariadb.service: Succeeded.
===
There are no new log entries in /var/log/mysql since the crash.
The only thing I can think of is that in installing mysqldump something went astray but I’m in way over my head.
I have spend some hours searching for an answer so I thought I’d ask here in case someone else has run into this.
Thanks.
mariadb Ver 15.1 Distrib 10.3.39-MariaDB, for debian-linux-gnu (x86_64) using readline 5.2
mysqldump Ver 10.19 Distrib 10.3.39-MariaDB, for debian-linux-gnu (x86_64)
Ubuntu 20.04.6 LTS
Actually, if running on MariaDB, the backup utility would be mariadb-dump, but Ubuntu and most other linux distros make mysqldump a sym link to that executable to be helpful.
Now, if you don’t remember exactly what package operations may have happened during the installation of the mariadb-client-10.3 , it may be helpful to examine the apt activity logs under /var/log/apt, and see exactly what packages were installed/upgraded/removed when that was done.
Also, verify that your MariaDB daemon configuration actually has error logging turned on, and that it’s directing that output to the expected /var/log/mysql/error.log destination.
If MariaDB isn’t starting (which it obviously isn’t) then there is definitely an error, but if the error log isn’t enabled/configured, then you’re not going to have anything recorded for troubleshooting purposes.
Get the error logging working, then try stopping and restarting the mariadb service, so you can get some useful output to work with.
One other thing, depending on how the mariadb daemon was set up and secured, you’re probably going to need to run the mariadb client either via sudo, e.g. “sudo mariadb -u root” , or supply your username to connect, e.g. “mariadb -u drb -p” , and then authenticate.
Turns out that the mariadb-client install was the problem.
The /etc/init.d/mariadb startup file must have be altered because it was looking for files eg
/usr/sbin/mariadbd-safe
and
/usr/sbin/mariadbd
that do not exist on my machine.
And it was failing silently when it did not find them.
The mariadb site suggests
sudo apt-get install --reinstall mariadb-server
I’ll just think about that a bit first… I’d rather not start all over.
=Sidebar
I’ve had a mysql server running on another machine for 15+ years and never had a problem except when it would be unhappy after a power outage.
Lulled me into not setting up a cron backup since I was just checking the waters in openemr.
On a normal installation of mariadb, the ‘mariadbd’ executable will be in /usr/sbin, whereas the ‘mariadbd-safe’ executable should be under /usr/bin.
If your host really does not have a /usr/sbin/mariadbd executable, then something has gone horribly wrong during the software installations to leave it in that state.
The re-installation of mariadb server really isn’t a big deal, but obviously, since you don’t have any proper SQL data backups, you’re going to want to make sure you’ve got the contents of your mariadb data directory backed up somewhere safe, just in case (that is, assuming that you’ve got anything of value that you want to preserve in your openemr database).
If the preservation of your existing database isn’t a priority, you might be better off installing a newer version of MariaDB 10, like 10.6, or maybe even MariaDB 11, and then getting a clean start with a fresh install of OpenEMR.
It’s helpful to know that there should be a file /usr/sbin/mariadbd. Mine’s gone but I wondered if it was just that the init.d file (which calls /usr/sbin/mariadbd) was corrupted. So now I know it’s a systemwide problem and re-installing the server is a must.
I’m not yet using openemr but I did move almost 20,000 patients over and recreating the steps will take a bit of time.
OK, so it sounds like you’d probably rather preserve your existing database.
(Duh, that’s why you’re setting up backup in the first place, right?)
If I were in that situation, here’s what I’d do (and I might be wrong, smarter people will probably drop in and give much better advice):
Assuming your existing SQL data files are intact, go ahead and back them up, to external media, or at the very least another location on your server if space permits.
Since your previously running MariaDB was 10.3, go ahead and do the removal/re-install of that version, get your DB service running again. Make sure your existing openemr database is intact, and then get a good backup via mariadb-backup. Since you don’t have a database dump as a restore point, the short term focus needs to be on bringing your database back online using the existing contents of your /var/lib/mysql directory. At that point, you can either leave it as is, or upgrade to a newer version of MariaDB, either way, you’ve at least got good backups going forward. This is a textbook example of why you always have to have backups, it’s like a fire extinguisher, even if you’ve never had cause to use it in the past, you still keep one ready to go, because when you need it, you need it, and sometimes there’s no solution besides restoring from backup.
My previous experience with mysql was using only myisam data files which could just be copied as a backup and later restored back to the data directory if needed. So I regarded the mysqldump as an inferior backup method.
The innodb doesn’t seem to like that method unless the server is shutdown first — and since in this instance it crashed I’m concerned that when I try to restart there will be some inconsistencies.
Lesson learned: mysqldump in the future.
From what I’ve been reading, I might want to use the same root user/password in new mariadb as in the original. I think I recall it but won’t know for sure until I try.
I have copies of all the mysql/mariadb config /etc files in case I need them
Thanks for sticking with me: as you suggest I was thinking that if I just get it running I can do a mariadbdump and then update… hearing it from you confirms my inclination.