Article 543Y9 missing memory

missing memory

by
axet
from LinuxQuestions.org on (#543Y9)
Hello!

I'm trying to understand where my free memory goes. No, this is not about usual caches and buffers.

To clear the process I'm using clear linux machine after reboot, steps:

1) reboot into rescue mode
2) su user

Memory usage about 200M showed in htop and /proc/meminfo

Then I start a wine game:

3) startx ~/bin/game

Play it for a few moments and exit. Memory usage 1.6G after exiting the game and it stays used.

I'm trying to understand where this memory goes. But I do not see any trails. I able to clam back this memory by issuing:

echo 2 > /proc/sys/vm/drop_caches

But this does not explain who using it. I did compare /proc/meminfo before and after drop_caches. I see only significant change in MemAvailable (1.2G change) but everything else stays almost the same (nor slab, nor buffers, nor cached).

I did ran slabtop - same, nothing calming to have 1.2G of memory. Any suggestion how I can see kernel memory usage by module / type?

EDIT: simple script shows what is going on:

Code:#!/bin/bash

free -m

echo
echo "----- meminfo"
cat /proc/meminfo

echo
echo "----- ps_mem (https://github.com/pixelb/ps_mem)"
sudo ~/bin/ps_mem.py

echo
echo "----- dmesg info"
dmesg | grep Memory: | grep -E -o '[0-9]+K (kernel code|data|rwdata|rodata|init)' && awk '{print $2/1024 "K " $1 }' /proc/modules | sort -hr | head

echo
echo "----- slabtop"
sudo slabtop -s c -o | head -201) fresh reboot (200M used)

Code: total used free shared buff/cache available
Mem: 5944 189 5541 1 214 5522
Swap: 3073 0 3073

----- meminfo
MemTotal: 6087576 kB
MemFree: 5673864 kB
MemAvailable: 5654600 kB
Buffers: 36144 kB
Cached: 146924 kB
SwapCached: 0 kB
Active: 128120 kB
Inactive: 91760 kB
Active(anon): 37308 kB
Inactive(anon): 816 kB
Active(file): 90812 kB
Inactive(file): 90944 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 3147660 kB
SwapFree: 3147660 kB
Dirty: 120 kB
Writeback: 0 kB
AnonPages: 34836 kB
Mapped: 55800 kB
Shmem: 1308 kB
KReclaimable: 36492 kB
Slab: 110228 kB
SReclaimable: 36492 kB
SUnreclaim: 73736 kB
KernelStack: 2944 kB
PageTables: 1392 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 6191448 kB
Committed_AS: 197488 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 27132 kB
VmallocChunk: 0 kB
Percpu: 8640 kB
HardwareCorrupted: 0 kB
AnonHugePages: 2048 kB
ShmemHugePages: 0 kB
ShmemPmdMapped: 0 kB
FileHugePages: 0 kB
FilePmdMapped: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
Hugetlb: 0 kB
DirectMap4k: 136616 kB
DirectMap2M: 3989504 kB
DirectMap1G: 3145728 kB

----- ps_mem
Private + Shared = RAM used Program

144.0 KiB + 63.5 KiB = 207.5 KiB swapspace
356.0 KiB + 173.5 KiB = 529.5 KiB rtkit-daemon
788.0 KiB + 369.5 KiB = 1.1 MiB su
896.0 KiB + 536.5 KiB = 1.4 MiB systemd-sulogin-shell
1.1 MiB + 367.5 KiB = 1.4 MiB sudo
1.1 MiB + 760.5 KiB = 1.9 MiB systemd-timesyncd
1.3 MiB + 836.5 KiB = 2.1 MiB systemd-logind
1.3 MiB + 871.0 KiB = 2.2 MiB dbus-daemon (2)
1.1 MiB + 1.2 MiB = 2.2 MiB alsactl
1.9 MiB + 540.5 KiB = 2.4 MiB bluetoothd
4.0 MiB + 235.5 KiB = 4.2 MiB systemd-udevd
2.6 MiB + 1.6 MiB = 4.2 MiB bash (3)
3.1 MiB + 1.3 MiB = 4.4 MiB polkitd
4.1 MiB + 733.5 KiB = 4.9 MiB wpa_supplicant
2.6 MiB + 2.3 MiB = 4.9 MiB dhclient (2)
4.0 MiB + 5.0 MiB = 9.0 MiB systemd (3)
9.5 MiB + 1.4 MiB = 10.9 MiB NetworkManager
11.7 MiB + 723.5 KiB = 12.4 MiB systemd-journald
17.3 MiB + 1.8 MiB = 19.1 MiB pulseaudio
---------------------------------
89.6 MiB
=================================

----- dmesg info
10243K kernel code
1210K rwdata
3984K rodata
1664K init
4892K amdgpu
1440K btrfs
856K mac80211
816K cfg80211
776K kvm
748K ext4
648K bluetooth
572K drm
360K aesni_intel
312K iwlmvm

----- slabtop
Active / Total Objects (% used) : 376531 / 383503 (98,2%)
Active / Total Slabs (% used) : 8438 / 8438 (100,0%)
Active / Total Caches (% used) : 94 / 142 (66,2%)
Active / Total Size (% used) : 92820,45K / 95443,88K (97,3%)
Minimum / Average / Maximum Object : 0,01K / 0,25K / 14,56K

OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
21978 21568 98% 0,59K 814 27 13024K inode_cache
37170 34457 92% 0,19K 885 42 7080K dentry
3520 3520 100% 2,00K 220 16 7040K kmalloc-2k
1288 1272 98% 4,00K 161 8 5152K kmalloc-4k
4770 3996 83% 1,06K 159 30 5088K ext4_inode_cache
20826 20670 99% 0,20K 534 39 4272K vm_area_struct
32960 32960 100% 0,12K 1030 32 4120K kernfs_node_cache
3488 3488 100% 1,00K 109 32 3488K kmalloc-1k
13024 12848 98% 0,25K 407 32 3256K filp
4776 4615 96% 0,66K 199 24 3184K proc_inode_cache
4844 4130 85% 0,57K 173 28 2768K radix_tree_node
680 680 100% 3,75K 85 8 2720K task_struct
5120 5120 100% 0,50K 160 32 2560K kmalloc-5122) after exiting a game (1.7G used)

Code: total used free shared buff/cache available
Mem: 5944 1760 2541 1 1642 3944
Swap: 3073 0 3073

----- meminfo
MemTotal: 6087576 kB
MemFree: 2602836 kB
MemAvailable: 4039216 kB
Buffers: 73444 kB
Cached: 1558660 kB
SwapCached: 0 kB
Active: 695112 kB
Inactive: 978076 kB
Active(anon): 792 kB
Inactive(anon): 41620 kB
Active(file): 694320 kB
Inactive(file): 936456 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 3147660 kB
SwapFree: 3147660 kB
Dirty: 8 kB
Writeback: 0 kB
AnonPages: 37080 kB
Mapped: 58864 kB
Shmem: 1312 kB
KReclaimable: 49740 kB
Slab: 131000 kB
SReclaimable: 49740 kB
SUnreclaim: 81260 kB
KernelStack: 3040 kB
PageTables: 1528 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 6191448 kB
Committed_AS: 241408 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 27264 kB
VmallocChunk: 0 kB
Percpu: 8640 kB
HardwareCorrupted: 0 kB
AnonHugePages: 2048 kB
ShmemHugePages: 0 kB
ShmemPmdMapped: 0 kB
FileHugePages: 0 kB
FilePmdMapped: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
Hugetlb: 0 kB
DirectMap4k: 1897896 kB
DirectMap2M: 4325376 kB
DirectMap1G: 1048576 kB

----- ps_mem
Private + Shared = RAM used Program

144.0 KiB + 61.5 KiB = 205.5 KiB swapspace
356.0 KiB + 166.5 KiB = 522.5 KiB rtkit-daemon
788.0 KiB + 355.5 KiB = 1.1 MiB su
896.0 KiB + 508.5 KiB = 1.4 MiB systemd-sulogin-shell
1.1 MiB + 333.5 KiB = 1.4 MiB sudo
1.1 MiB + 728.5 KiB = 1.8 MiB systemd-timesyncd
1.3 MiB + 804.5 KiB = 2.1 MiB systemd-logind
1.4 MiB + 857.0 KiB = 2.2 MiB dbus-daemon (2)
1.1 MiB + 1.2 MiB = 2.2 MiB alsactl
1.8 MiB + 505.5 KiB = 2.3 MiB bluetoothd
3.0 MiB + 1.1 MiB = 4.0 MiB polkitd
4.0 MiB + 219.5 KiB = 4.2 MiB systemd-udevd
2.7 MiB + 1.6 MiB = 4.3 MiB bash (3)
4.1 MiB + 681.5 KiB = 4.8 MiB wpa_supplicant
2.6 MiB + 2.2 MiB = 4.8 MiB dhclient (2)
4.7 MiB + 1.4 MiB = 6.1 MiB udisksd
4.0 MiB + 4.9 MiB = 8.9 MiB systemd (3)
9.4 MiB + 1.1 MiB = 10.6 MiB NetworkManager
11.7 MiB + 693.5 KiB = 12.4 MiB systemd-journald
17.5 MiB + 1.7 MiB = 19.2 MiB pulseaudio
---------------------------------
94.5 MiB
=================================

----- dmesg info
10243K kernel code
1210K rwdata
3984K rodata
1664K init
4892K amdgpu
1440K btrfs
856K mac80211
816K cfg80211
776K kvm
748K ext4
648K bluetooth
572K drm
360K aesni_intel
312K iwlmvm

----- slabtop
Active / Total Objects (% used) : 439459 / 453680 (96,9%)
Active / Total Slabs (% used) : 10158 / 10158 (100,0%)
Active / Total Caches (% used) : 95 / 142 (66,9%)
Active / Total Size (% used) : 110972,32K / 115627,59K (96,0%)
Minimum / Average / Maximum Object : 0,01K / 0,25K / 14,56K

OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
21952 20652 94% 0,57K 784 28 12544K radix_tree_node
18090 17093 94% 0,59K 670 27 10720K inode_cache
8040 6970 86% 1,06K 268 30 8576K ext4_inode_cache
37968 32553 85% 0,19K 904 42 7232K dentry
3584 3584 100% 2,00K 224 16 7168K kmalloc-2k
1408 1398 99% 4,00K 176 8 5632K kmalloc-4k
23790 23678 99% 0,20K 610 39 4880K vm_area_struct
35488 35209 99% 0,12K 1109 32 4436K kernfs_node_cache
4352 4129 94% 1,00K 136 32 4352K kmalloc-1k
5568 5411 97% 0,66K 232 24 3712K proc_inode_cache
14208 14032 98% 0,25K 444 32 3552K filp
776 776 100% 3,75K 97 8 3104K task_struct
5728 5728 100% 0,50K 179 32 2864K kmalloc-5123) echo 1 > /proc/sys/vm/drop_caches # free pagecache (about the same 1.6G usage)

Code: total used free shared buff/cache available
Mem: 5944 1759 4085 1 100 4003
Swap: 3073 0 3073

----- meminfo
MemTotal: 6087576 kB
MemFree: 4183300 kB
MemAvailable: 4099392 kB
Buffers: 728 kB
Cached: 62388 kB
SwapCached: 0 kB
Active: 37504 kB
Inactive: 66616 kB
Active(anon): 904 kB
Inactive(anon): 41596 kB
Active(file): 36600 kB
Inactive(file): 25020 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 3147660 kB
SwapFree: 3147660 kB
Dirty: 8 kB
Writeback: 0 kB
AnonPages: 37152 kB
Mapped: 58720 kB
Shmem: 1312 kB
KReclaimable: 40164 kB
Slab: 121416 kB
SReclaimable: 40164 kB
SUnreclaim: 81252 kB
KernelStack: 3040 kB
PageTables: 1548 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 6191448 kB
Committed_AS: 246760 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 27232 kB
VmallocChunk: 0 kB
Percpu: 8640 kB
HardwareCorrupted: 0 kB
AnonHugePages: 2048 kB
ShmemHugePages: 0 kB
ShmemPmdMapped: 0 kB
FileHugePages: 0 kB
FilePmdMapped: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
Hugetlb: 0 kB
DirectMap4k: 1897896 kB
DirectMap2M: 4325376 kB
DirectMap1G: 1048576 kB

----- ps_mem
Private + Shared = RAM used Program

144.0 KiB + 61.5 KiB = 205.5 KiB swapspace
356.0 KiB + 166.5 KiB = 522.5 KiB rtkit-daemon
788.0 KiB + 360.5 KiB = 1.1 MiB su
896.0 KiB + 507.5 KiB = 1.4 MiB systemd-sulogin-shell
1.1 MiB + 335.5 KiB = 1.4 MiB sudo
1.1 MiB + 728.5 KiB = 1.8 MiB systemd-timesyncd
1.3 MiB + 805.5 KiB = 2.1 MiB systemd-logind
1.4 MiB + 857.0 KiB = 2.2 MiB dbus-daemon (2)
1.1 MiB + 1.2 MiB = 2.2 MiB alsactl
1.8 MiB + 506.5 KiB = 2.3 MiB bluetoothd
3.0 MiB + 1.1 MiB = 4.1 MiB polkitd
4.0 MiB + 218.5 KiB = 4.2 MiB systemd-udevd
2.7 MiB + 1.6 MiB = 4.2 MiB bash (3)
4.1 MiB + 690.5 KiB = 4.8 MiB wpa_supplicant
2.6 MiB + 2.2 MiB = 4.8 MiB dhclient (2)
4.7 MiB + 1.4 MiB = 6.1 MiB udisksd
4.0 MiB + 4.9 MiB = 8.9 MiB systemd (3)
9.4 MiB + 1.1 MiB = 10.6 MiB NetworkManager
11.7 MiB + 693.5 KiB = 12.4 MiB systemd-journald
17.5 MiB + 1.7 MiB = 19.2 MiB pulseaudio
---------------------------------
94.5 MiB
=================================

----- dmesg info
10243K kernel code
1210K rwdata
3984K rodata
1664K init
4892K amdgpu
1440K btrfs
856K mac80211
816K cfg80211
776K kvm
748K ext4
648K bluetooth
572K drm
360K aesni_intel
312K iwlmvm

----- slabtop
Active / Total Objects (% used) : 411472 / 427217 (96,3%)
Active / Total Slabs (% used) : 9322 / 9322 (100,0%)
Active / Total Caches (% used) : 95 / 142 (66,9%)
Active / Total Size (% used) : 100732,35K / 105776,20K (95,2%)
Minimum / Average / Maximum Object : 0,01K / 0,25K / 14,56K

OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
18090 17093 94% 0,59K 670 27 10720K inode_cache
8040 6976 86% 1,06K 268 30 8576K ext4_inode_cache
37968 32553 85% 0,19K 904 42 7232K dentry
3584 3584 100% 2,00K 224 16 7168K kmalloc-2k
1408 1377 97% 4,00K 176 8 5632K kmalloc-4k
22776 22632 99% 0,20K 584 39 4672K vm_area_struct
35456 35207 99% 0,12K 1108 32 4432K kernfs_node_cache
4352 4126 94% 1,00K 136 32 4352K kmalloc-1k
6916 5324 76% 0,57K 247 28 3952K radix_tree_node
5568 5411 97% 0,66K 232 24 3712K proc_inode_cache
14080 13904 98% 0,25K 440 32 3520K filp
776 776 100% 3,75K 97 8 3104K task_struct
5728 5728 100% 0,50K 179 32 2864K kmalloc-5124) echo 2 > /proc/sys/vm/drop_caches # free reclaimable slab objects (includes dentries and inodes) (back to normal 200M usage)

Code: total used free shared buff/cache available
Mem: 5944 204 5642 1 98 5559
Swap: 3073 0 3073

----- meminfo
MemTotal: 6087576 kB
MemFree: 5777624 kB
MemAvailable: 5692820 kB
Buffers: 3928 kB
Cached: 62752 kB
SwapCached: 0 kB
Active: 40840 kB
Inactive: 62992 kB
Active(anon): 928 kB
Inactive(anon): 37520 kB
Active(file): 39912 kB
Inactive(file): 25472 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 3147660 kB
SwapFree: 3147660 kB
Dirty: 8 kB
Writeback: 0 kB
AnonPages: 37164 kB
Mapped: 58752 kB
Shmem: 1312 kB
KReclaimable: 34616 kB
Slab: 115652 kB
SReclaimable: 34616 kB
SUnreclaim: 81036 kB
KernelStack: 3056 kB
PageTables: 1580 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 6191448 kB
Committed_AS: 243244 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 27248 kB
VmallocChunk: 0 kB
Percpu: 8640 kB
HardwareCorrupted: 0 kB
AnonHugePages: 2048 kB
ShmemHugePages: 0 kB
ShmemPmdMapped: 0 kB
FileHugePages: 0 kB
FilePmdMapped: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
Hugetlb: 0 kB
DirectMap4k: 1897896 kB
DirectMap2M: 4325376 kB
DirectMap1G: 1048576 kB

----- ps_mem
Private + Shared = RAM used Program

144.0 KiB + 60.5 KiB = 204.5 KiB swapspace
356.0 KiB + 166.5 KiB = 522.5 KiB rtkit-daemon
788.0 KiB + 360.5 KiB = 1.1 MiB su
896.0 KiB + 507.5 KiB = 1.4 MiB systemd-sulogin-shell
1.1 MiB + 349.5 KiB = 1.4 MiB sudo
1.1 MiB + 728.5 KiB = 1.8 MiB systemd-timesyncd
1.3 MiB + 804.5 KiB = 2.1 MiB systemd-logind
1.4 MiB + 857.0 KiB = 2.2 MiB dbus-daemon (2)
1.1 MiB + 1.2 MiB = 2.2 MiB alsactl
1.8 MiB + 506.5 KiB = 2.3 MiB bluetoothd
3.0 MiB + 1.1 MiB = 4.1 MiB polkitd
4.0 MiB + 218.5 KiB = 4.2 MiB systemd-udevd
2.6 MiB + 1.6 MiB = 4.2 MiB bash (3)
4.1 MiB + 682.5 KiB = 4.8 MiB wpa_supplicant
2.6 MiB + 2.2 MiB = 4.8 MiB dhclient (2)
4.7 MiB + 1.4 MiB = 6.1 MiB udisksd
4.0 MiB + 4.9 MiB = 8.9 MiB systemd (3)
9.4 MiB + 1.1 MiB = 10.6 MiB NetworkManager
11.7 MiB + 693.5 KiB = 12.4 MiB systemd-journald
17.5 MiB + 1.7 MiB = 19.2 MiB pulseaudio
---------------------------------
94.5 MiB
=================================

----- dmesg info
10243K kernel code
1210K rwdata
3984K rodata
1664K init
4892K amdgpu
1440K btrfs
856K mac80211
816K cfg80211
776K kvm
748K ext4
648K bluetooth
572K drm
360K aesni_intel
312K iwlmvm

----- slabtop
Active / Total Objects (% used) : 406745 / 418837 (97,1%)
Active / Total Slabs (% used) : 9088 / 9088 (100,0%)
Active / Total Caches (% used) : 95 / 142 (66,9%)
Active / Total Size (% used) : 96245,45K / 100299,09K (96,0%)
Minimum / Average / Maximum Object : 0,01K / 0,24K / 14,56K

OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
18090 17093 94% 0,59K 670 27 10720K inode_cache
3584 3584 100% 2,00K 224 16 7168K kmalloc-2k
35826 31569 88% 0,19K 853 42 6824K dentry
1408 1377 97% 4,00K 176 8 5632K kmalloc-4k
22776 22776 100% 0,20K 584 39 4672K vm_area_struct
35456 35207 99% 0,12K 1108 32 4432K kernfs_node_cache
4352 4126 94% 1,00K 136 32 4352K kmalloc-1k
3840 3184 82% 1,06K 128 30 4096K ext4_inode_cache
5568 5411 97% 0,66K 232 24 3712K proc_inode_cache
13984 13752 98% 0,25K 437 32 3496K filp
5852 4758 81% 0,57K 209 28 3344K radix_tree_node
776 776 100% 3,75K 97 8 3104K task_struct
5760 5760 100% 0,50K 180 32 2880K kmalloc-512

You can see 1.6G memory coming back from no where, after clearing kernel slabs.

* https://www.kernel.org/doc/Documentation/sysctl/vm.txt
* https://code.woboq.org/linux/linux/f..._caches.c.htmllatest?d=yIl2AUoC8zA latest?i=JLNp6MGvtSQ:hY3bMRHjp_k:F7zBnMy latest?i=JLNp6MGvtSQ:hY3bMRHjp_k:V_sGLiP latest?d=qj6IDK7rITs latest?i=JLNp6MGvtSQ:hY3bMRHjp_k:gIN9vFwJLNp6MGvtSQ
External Content
Source RSS or Atom Feed
Feed Location https://feeds.feedburner.com/linuxquestions/latest
Feed Title LinuxQuestions.org
Feed Link https://www.linuxquestions.org/questions/
Reply 0 comments