Apache ram problem
by Tasslehoff from LinuxQuestions.org on (#5RTWH)
Hi, I have a strange behavior from a CentOS 7 vm which is running a pretty basic lamp configuration:
Basically on the vm runs a single TYPO3 site used as a content crawler for another site running on other servers, the problem I'm experiencing is frequent OOMs with Apache using all the vm ram and not releasing any of this ram, even if there are no more http requests.
This is an example of this vm state every morning.
Code:[root@drakaris ~]#free -m
total used free shared buff/cache available
Mem: 7820 7489 132 37 198 79
Swap: 1023 752 271
[root@drakaris ~]#ps_mem
Private + Shared = RAM used Program
8.0 KiB + 25.0 KiB = 33.0 KiB agetty (2)
4.0 KiB + 43.5 KiB = 47.5 KiB systemd-udevd
4.0 KiB + 97.5 KiB = 101.5 KiB dhclient
92.0 KiB + 65.0 KiB = 157.0 KiB gssproxy
132.0 KiB + 36.0 KiB = 168.0 KiB irqbalance
164.0 KiB + 43.0 KiB = 207.0 KiB crond
140.0 KiB + 86.5 KiB = 226.5 KiB master
144.0 KiB + 85.5 KiB = 229.5 KiB qmgr
188.0 KiB + 57.5 KiB = 245.5 KiB auditd
196.0 KiB + 61.5 KiB = 257.5 KiB lvmetad
256.0 KiB + 79.5 KiB = 335.5 KiB chronyd
588.0 KiB + 63.5 KiB = 651.5 KiB systemd-logind
620.0 KiB + 90.5 KiB = 710.5 KiB dbus-daemon
612.0 KiB + 137.0 KiB = 749.0 KiB su
704.0 KiB + 67.5 KiB = 771.5 KiB cvfwd
548.0 KiB + 256.0 KiB = 804.0 KiB cvlaunchd
620.0 KiB + 261.5 KiB = 881.5 KiB rsyslogd
588.0 KiB + 315.5 KiB = 903.5 KiB tuned
932.0 KiB + 111.5 KiB = 1.0 MiB polkitd
1.1 MiB + 124.5 KiB = 1.2 MiB pickup
1.1 MiB + 184.5 KiB = 1.3 MiB sudo
1.5 MiB + 104.0 KiB = 1.6 MiB bash (2)
1.0 MiB + 592.5 KiB = 1.6 MiB zabbix_agentd (6)
148.0 KiB + 1.5 MiB = 1.7 MiB php-fpm (6)
1.3 MiB + 570.5 KiB = 1.9 MiB ClMgrS
664.0 KiB + 1.7 MiB = 2.3 MiB sshd (3)
3.0 MiB + 229.0 KiB = 3.3 MiB systemd-journald
2.3 MiB + 1.2 MiB = 3.5 MiB collectd
3.5 MiB + 124.0 KiB = 3.7 MiB systemd
5.6 MiB + 731.0 KiB = 6.4 MiB cvd
80.4 MiB + 151.5 KiB = 80.5 MiB mysqld
7.0 GiB + 17.9 MiB = 7.0 GiB httpd (4)
---------------------------------
7.1 GiB
=================================
[root@drakaris ~]#smem -pt
PID User Command Swap USS PSS RSS
560 root /usr/lib/systemd/systemd-ud 0.23% 0.00% 0.00% 0.00%
1029 root /sbin/dhclient -1 -q -lf /v 0.20% 0.00% 0.00% 0.00%
1519 root /sbin/agetty --noclear tty1 0.01% 0.00% 0.00% 0.00%
1521 root /sbin/agetty --keep-baud 11 0.01% 0.00% 0.00% 0.00%
... [SNIP]
16500 root sudo su - 0.00% 0.01% 0.02% 0.02%
1452 root /opt/commvault/Base/ClMgrS 1.25% 0.02% 0.02% 0.03%
517 root /usr/lib/systemd/systemd-jo 0.59% 0.04% 0.04% 0.05%
1503 root /usr/sbin/collectd 0.22% 0.03% 0.04% 0.06%
1 root /usr/lib/systemd/systemd -- 0.09% 0.05% 0.05% 0.05%
1449 root /opt/commvault/Base/cvd 13.52% 0.07% 0.08% 0.09%
18027 root python /bin/smem -pt 0.00% 0.13% 0.13% 0.14%
16878 mysql /usr/sbin/mysqld --daemoniz 30.91% 1.03% 1.03% 1.03%
13401 apache /usr/sbin/httpd -DFOREGROUN 0.97% 38.66% 38.76% 38.88%
10914 apache /usr/sbin/httpd -DFOREGROUN 18.10% 52.77% 52.87% 52.99%
-------------------------------------------------------------------------------
50 9 82.97% 92.94% 93.20% 93.66%If a simply do an Apache restart the ram is released and everything came back to a normal idle state.
I thought it could be some memory leak from TYPO3 but I can't check it because I have no access to it, but I would like to find why Apache keeps all this ram occupied apparently for no reason, and in case I'll find some clue about TYPO3 I will submit them to the project owner for a solution.
Do you think it's possible to dump some data from those Apache processes to find out why it's using all this ram for all this time?
Thanks in advance for any suggestion
Tas
- Apache 2.4.6 (MPM worker)
- MySQL 5.7 (Percona Server 5.7)
- php 7.2 (with php-fpm)
Basically on the vm runs a single TYPO3 site used as a content crawler for another site running on other servers, the problem I'm experiencing is frequent OOMs with Apache using all the vm ram and not releasing any of this ram, even if there are no more http requests.
This is an example of this vm state every morning.
Code:[root@drakaris ~]#free -m
total used free shared buff/cache available
Mem: 7820 7489 132 37 198 79
Swap: 1023 752 271
[root@drakaris ~]#ps_mem
Private + Shared = RAM used Program
8.0 KiB + 25.0 KiB = 33.0 KiB agetty (2)
4.0 KiB + 43.5 KiB = 47.5 KiB systemd-udevd
4.0 KiB + 97.5 KiB = 101.5 KiB dhclient
92.0 KiB + 65.0 KiB = 157.0 KiB gssproxy
132.0 KiB + 36.0 KiB = 168.0 KiB irqbalance
164.0 KiB + 43.0 KiB = 207.0 KiB crond
140.0 KiB + 86.5 KiB = 226.5 KiB master
144.0 KiB + 85.5 KiB = 229.5 KiB qmgr
188.0 KiB + 57.5 KiB = 245.5 KiB auditd
196.0 KiB + 61.5 KiB = 257.5 KiB lvmetad
256.0 KiB + 79.5 KiB = 335.5 KiB chronyd
588.0 KiB + 63.5 KiB = 651.5 KiB systemd-logind
620.0 KiB + 90.5 KiB = 710.5 KiB dbus-daemon
612.0 KiB + 137.0 KiB = 749.0 KiB su
704.0 KiB + 67.5 KiB = 771.5 KiB cvfwd
548.0 KiB + 256.0 KiB = 804.0 KiB cvlaunchd
620.0 KiB + 261.5 KiB = 881.5 KiB rsyslogd
588.0 KiB + 315.5 KiB = 903.5 KiB tuned
932.0 KiB + 111.5 KiB = 1.0 MiB polkitd
1.1 MiB + 124.5 KiB = 1.2 MiB pickup
1.1 MiB + 184.5 KiB = 1.3 MiB sudo
1.5 MiB + 104.0 KiB = 1.6 MiB bash (2)
1.0 MiB + 592.5 KiB = 1.6 MiB zabbix_agentd (6)
148.0 KiB + 1.5 MiB = 1.7 MiB php-fpm (6)
1.3 MiB + 570.5 KiB = 1.9 MiB ClMgrS
664.0 KiB + 1.7 MiB = 2.3 MiB sshd (3)
3.0 MiB + 229.0 KiB = 3.3 MiB systemd-journald
2.3 MiB + 1.2 MiB = 3.5 MiB collectd
3.5 MiB + 124.0 KiB = 3.7 MiB systemd
5.6 MiB + 731.0 KiB = 6.4 MiB cvd
80.4 MiB + 151.5 KiB = 80.5 MiB mysqld
7.0 GiB + 17.9 MiB = 7.0 GiB httpd (4)
---------------------------------
7.1 GiB
=================================
[root@drakaris ~]#smem -pt
PID User Command Swap USS PSS RSS
560 root /usr/lib/systemd/systemd-ud 0.23% 0.00% 0.00% 0.00%
1029 root /sbin/dhclient -1 -q -lf /v 0.20% 0.00% 0.00% 0.00%
1519 root /sbin/agetty --noclear tty1 0.01% 0.00% 0.00% 0.00%
1521 root /sbin/agetty --keep-baud 11 0.01% 0.00% 0.00% 0.00%
... [SNIP]
16500 root sudo su - 0.00% 0.01% 0.02% 0.02%
1452 root /opt/commvault/Base/ClMgrS 1.25% 0.02% 0.02% 0.03%
517 root /usr/lib/systemd/systemd-jo 0.59% 0.04% 0.04% 0.05%
1503 root /usr/sbin/collectd 0.22% 0.03% 0.04% 0.06%
1 root /usr/lib/systemd/systemd -- 0.09% 0.05% 0.05% 0.05%
1449 root /opt/commvault/Base/cvd 13.52% 0.07% 0.08% 0.09%
18027 root python /bin/smem -pt 0.00% 0.13% 0.13% 0.14%
16878 mysql /usr/sbin/mysqld --daemoniz 30.91% 1.03% 1.03% 1.03%
13401 apache /usr/sbin/httpd -DFOREGROUN 0.97% 38.66% 38.76% 38.88%
10914 apache /usr/sbin/httpd -DFOREGROUN 18.10% 52.77% 52.87% 52.99%
-------------------------------------------------------------------------------
50 9 82.97% 92.94% 93.20% 93.66%If a simply do an Apache restart the ram is released and everything came back to a normal idle state.
I thought it could be some memory leak from TYPO3 but I can't check it because I have no access to it, but I would like to find why Apache keeps all this ram occupied apparently for no reason, and in case I'll find some clue about TYPO3 I will submit them to the project owner for a solution.
Do you think it's possible to dump some data from those Apache processes to find out why it's using all this ram for all this time?
Thanks in advance for any suggestion
Tas