Article 6GHW3 Decreasing available memory in Linux while memory consumption of running processes stays about the same

Decreasing available memory in Linux while memory consumption of running processes stays about the same

by
arivo
from LinuxQuestions.org on (#6GHW3)
Hello,

in our production systems we are using Linux (kernel 5.10.55-051055-generic) on Intel NUCs (4GB RAM, 2GB swap) together where our Services run in Docker container. Those services are mainly small services communicating with zmq (over tcp) but also a service running CNNs using OpenVino on the integrated Intel GPU.

We observe a problem that we run out of memory over time. The problem is that the memory consumption of our services does no increase over time. This was first checked via docker stats and then in more detail using /sys/fs/cgroup/memory/docker/<container_id>/memory.usage_in_bytes and memory.stat. Also in the log output of the out of memory killer the memory usage is less than the available memory.

We further investigated our problem using procrank (https://github.com/csimmonds/procrank_linux) and /proc/<process_id>/smaps
Here we got the following result:
Output of procrank (here just summary, see procrank_before.txt)
Code: Pss Uss
1232388K 1207288K TOTAL

RAM: 3672156K total, 136880K free, 15776K buffers, 404284K cached, 199164K shmem, 219112K slabOutput of free:
Code: total used free shared buff/cache available
Mem: 3672156 3022628 138180 199164 511348 236164
Swap: 1951740 1951716 24So we have about 1.2 GB usage of RAM and nearly 2GB swapped, where according to smaps about 1.8 GB are from processes. This is about 3 GB, which makes sense for our services.
The sum of Pss, buffers, cached, shmem (shared memory) and slab is about 2 GB. As far as i know this should mean that the rest of our 4 GB should be available. However only 138 MB are free and 236 MB available. What causes this difference?

After a reboot we have (full output in procrank_after.txt)
Code:
Pss Uss
2838033K 2760564K TOTAL

RAM: 3672152K total, 114896K free, 34960K buffers, 725004K cached, 291704K shmem, 178684K slabOutput of free:
Code:
total used free shared buff/cache available
Mem: 3672152 2698696 115636 291704 857820 446580
Swap: 1951740 62976 1888764There we have about 3 GB usage by our services (in RAM and swap) and everything sums up as expected.

What can cause the "missing" memory which is increasing over time (the system was running for a few weeks before the restart). What can we do to further investigate this problem?

Thanks for your Help
Attached Files
txt.gifprocrank_before.txt (7.2 KB)
txt.gifprocrank_after.txt (6.7 KB)
External Content
Source RSS or Atom Feed
Feed Location https://feeds.feedburner.com/linuxquestions/latest
Feed Title LinuxQuestions.org
Feed Link https://www.linuxquestions.org/questions/
Reply 0 comments