Not reporting memory use properly?
by Sirkent from LinuxQuestions.org on (#503XN)
I'm assuming this is kernel related for now, but the actual cause of the issue could be any number of things.
So - I have a Proxmox (Debian) server running a container. That container runs Ubuntu 16.04. I have 100Gb of RAM allocated to that container. It's a web application server, running Apache, PHP-FPM, Memcached, Redis and Puppeteer with headless Chrome.
It seems that since I've started using Puppeteer/headless chrome more, I am seeing some sort of memory leak - or at least memory being used up and then not freed by the kernel? This could just be a coincidence. The memory in use by the server slowly grows and grows. I cannot determine what is using memory. Stopping or killing services does not account for the used memory. Rebooting the container returns everything to normal, before it slowly grows again, over the space of several weeks.
This is not just cached memory increasing. Actual 'used' memory slowly increases until most of the memory on the server is in use, but there is no clear cause as to what is using it.
Code:$ free -m
total used free shared buff/cache available
Mem: 102400 40598 44825 4131 16976 61801So, according to free, 40598 is used and 16976 is cache, so 23622 is in actual use.
If I run ps_mem (https://github.com/pixelb/ps_mem) then this returns a total of 4.1Gb used across my processes. Running top/htop does not show me what is using the memory.
Code:$ sudo ps_mem
Private + Shared = RAM used Program
292.0 KiB + 77.5 KiB = 369.5 KiB autossh (3)
432.0 KiB + 186.5 KiB = 618.5 KiB agetty (3)
464.0 KiB + 390.5 KiB = 854.5 KiB pickup
516.0 KiB + 364.5 KiB = 880.5 KiB master
628.0 KiB + 257.5 KiB = 885.5 KiB sh (7)
856.0 KiB + 52.5 KiB = 908.5 KiB systemd-logind
580.0 KiB + 373.5 KiB = 953.5 KiB qmgr
956.0 KiB + 95.5 KiB = 1.0 MiB sudo
908.0 KiB + 177.0 KiB = 1.1 MiB sftp-server (4)
840.0 KiB + 282.5 KiB = 1.1 MiB dbus-daemon
1.3 MiB + 110.5 KiB = 1.4 MiB ntpd
1.1 MiB + 420.0 KiB = 1.5 MiB cron (4)
1.4 MiB + 172.5 KiB = 1.6 MiB pgbouncer
1.7 MiB + 198.5 KiB = 1.9 MiB atop
1.0 MiB + 1.1 MiB = 2.1 MiB polkitd
2.1 MiB + 30.5 KiB = 2.1 MiB rsyslogd
1.2 MiB + 1.0 MiB = 2.2 MiB accounts-daemon
2.1 MiB + 411.5 KiB = 2.6 MiB smtpd
2.6 MiB + 99.5 KiB = 2.7 MiB bash
2.2 MiB + 559.0 KiB = 2.8 MiB (sd-pam) (2)
1.7 MiB + 1.3 MiB = 3.0 MiB zabbix_agentd (6)
2.2 MiB + 2.1 MiB = 4.3 MiB systemd (3)
6.0 MiB + 948.5 KiB = 7.0 MiB ssh (3)
2.5 MiB + 5.0 MiB = 7.6 MiB sshd (9)
13.0 MiB + 859.5 KiB = 13.8 MiB unattended-upgr
13.4 MiB + 1.1 MiB = 14.6 MiB supervisord
68.5 MiB + 24.5 KiB = 68.5 MiB redis-server
23.2 MiB + 56.3 MiB = 79.4 MiB systemd-journald
65.7 MiB + 25.5 MiB = 91.2 MiB apache2 (5)
111.6 MiB + 34.3 MiB = 146.0 MiB phantomjs (4)
149.2 MiB + 10.3 MiB = 159.5 MiB php7.3 (6)
523.3 MiB + 154.5 KiB = 523.4 MiB memcached
3.1 GiB + 75.7 MiB = 3.1 GiB php-fpm7.3 (14)
---------------------------------
4.2 GiB
=================================In a few weeks, I can expect 'used' to more than double and buff/cache to stay reasonably close to what it is now. How can I discover what's actually using up my memory?
Any help would be thoroughly appreciated!
So - I have a Proxmox (Debian) server running a container. That container runs Ubuntu 16.04. I have 100Gb of RAM allocated to that container. It's a web application server, running Apache, PHP-FPM, Memcached, Redis and Puppeteer with headless Chrome.
It seems that since I've started using Puppeteer/headless chrome more, I am seeing some sort of memory leak - or at least memory being used up and then not freed by the kernel? This could just be a coincidence. The memory in use by the server slowly grows and grows. I cannot determine what is using memory. Stopping or killing services does not account for the used memory. Rebooting the container returns everything to normal, before it slowly grows again, over the space of several weeks.
This is not just cached memory increasing. Actual 'used' memory slowly increases until most of the memory on the server is in use, but there is no clear cause as to what is using it.
Code:$ free -m
total used free shared buff/cache available
Mem: 102400 40598 44825 4131 16976 61801So, according to free, 40598 is used and 16976 is cache, so 23622 is in actual use.
If I run ps_mem (https://github.com/pixelb/ps_mem) then this returns a total of 4.1Gb used across my processes. Running top/htop does not show me what is using the memory.
Code:$ sudo ps_mem
Private + Shared = RAM used Program
292.0 KiB + 77.5 KiB = 369.5 KiB autossh (3)
432.0 KiB + 186.5 KiB = 618.5 KiB agetty (3)
464.0 KiB + 390.5 KiB = 854.5 KiB pickup
516.0 KiB + 364.5 KiB = 880.5 KiB master
628.0 KiB + 257.5 KiB = 885.5 KiB sh (7)
856.0 KiB + 52.5 KiB = 908.5 KiB systemd-logind
580.0 KiB + 373.5 KiB = 953.5 KiB qmgr
956.0 KiB + 95.5 KiB = 1.0 MiB sudo
908.0 KiB + 177.0 KiB = 1.1 MiB sftp-server (4)
840.0 KiB + 282.5 KiB = 1.1 MiB dbus-daemon
1.3 MiB + 110.5 KiB = 1.4 MiB ntpd
1.1 MiB + 420.0 KiB = 1.5 MiB cron (4)
1.4 MiB + 172.5 KiB = 1.6 MiB pgbouncer
1.7 MiB + 198.5 KiB = 1.9 MiB atop
1.0 MiB + 1.1 MiB = 2.1 MiB polkitd
2.1 MiB + 30.5 KiB = 2.1 MiB rsyslogd
1.2 MiB + 1.0 MiB = 2.2 MiB accounts-daemon
2.1 MiB + 411.5 KiB = 2.6 MiB smtpd
2.6 MiB + 99.5 KiB = 2.7 MiB bash
2.2 MiB + 559.0 KiB = 2.8 MiB (sd-pam) (2)
1.7 MiB + 1.3 MiB = 3.0 MiB zabbix_agentd (6)
2.2 MiB + 2.1 MiB = 4.3 MiB systemd (3)
6.0 MiB + 948.5 KiB = 7.0 MiB ssh (3)
2.5 MiB + 5.0 MiB = 7.6 MiB sshd (9)
13.0 MiB + 859.5 KiB = 13.8 MiB unattended-upgr
13.4 MiB + 1.1 MiB = 14.6 MiB supervisord
68.5 MiB + 24.5 KiB = 68.5 MiB redis-server
23.2 MiB + 56.3 MiB = 79.4 MiB systemd-journald
65.7 MiB + 25.5 MiB = 91.2 MiB apache2 (5)
111.6 MiB + 34.3 MiB = 146.0 MiB phantomjs (4)
149.2 MiB + 10.3 MiB = 159.5 MiB php7.3 (6)
523.3 MiB + 154.5 KiB = 523.4 MiB memcached
3.1 GiB + 75.7 MiB = 3.1 GiB php-fpm7.3 (14)
---------------------------------
4.2 GiB
=================================In a few weeks, I can expect 'used' to more than double and buff/cache to stay reasonably close to what it is now. How can I discover what's actually using up my memory?
Any help would be thoroughly appreciated!