Copy file to usb key and dirty memory
by ctrlaltca from LinuxQuestions.org on (#6QN13)
Usb keys are usually slow devices.
In order to boost their performance the Linux kernel caches changes in (so called "dirty") memory, until an explicit fsync() call is made or some memory limits are reached.
The memory limits are based on a % of available RAM, making it hard to choose a default value that fits all kind of uses.
This can cause some weird effects, eg. system stall (https://lwn.net/Articles/572911/) or the UI reporting wrong transfer speed / completion status (https://bugs.kde.org/show_bug.cgi?id=281270), or even slower transfer speed.
Eg. on my system the default memory limit is 2GB, so if I copy a few GBs of files on the USB key i will see 2GB of data being copied immediately, than the transfer will stall at almost zero B/s until the cache get freed.
Dolphin (KDE's file manager) seems particularly bad at this:
* it never calls fsync() or fdatasync(), causing the transfer to only start when other external factors occurs (the user press "eject" or the memory limits kicks in);
* it always try to pump new data in the cache, keeping it at the memory limit and causing an additional slowdown (my guess is context switching but i may be wrong).
The matter is widely discussed online and the solutions I found online are:
* tune the values of dirty memory depending on your RAM (eg. https://unix.stackexchange.com/quest...workaround-fix), or
* use the BDI interface introduced in Linux 6.2 to limit the writeback cache only for usb disk devices (eg. https://unix.stackexchange.com/quest...ge-stall-issue)
In the Arch world the solution is this package that implements the second method: https://github.com/biglinux/usb-dirty-pages-udev
Trying to use the udev rule on Slackware64-current didn't work for me, so after some debug here's the rule I'm using:
Code:# cat /etc/udev/rules.d/60-usb-dirty-pages-udev.rules
ACTION=="add", ENV{DEVNAME}=="/dev/sd[a-z]", SUBSYSTEM=="block", ENV{ID_TYPE}=="disk", RUN+="/usr/bin/bash -c 'echo 1 > /sys/block/%k/bdi/strict_limit; echo 16777216 > /sys/block/%k/bdi/max_bytes'"I'd like to ask if anybody else ever dealt with this issue and what other solutions they found.
In order to boost their performance the Linux kernel caches changes in (so called "dirty") memory, until an explicit fsync() call is made or some memory limits are reached.
The memory limits are based on a % of available RAM, making it hard to choose a default value that fits all kind of uses.
This can cause some weird effects, eg. system stall (https://lwn.net/Articles/572911/) or the UI reporting wrong transfer speed / completion status (https://bugs.kde.org/show_bug.cgi?id=281270), or even slower transfer speed.
Eg. on my system the default memory limit is 2GB, so if I copy a few GBs of files on the USB key i will see 2GB of data being copied immediately, than the transfer will stall at almost zero B/s until the cache get freed.
Dolphin (KDE's file manager) seems particularly bad at this:
* it never calls fsync() or fdatasync(), causing the transfer to only start when other external factors occurs (the user press "eject" or the memory limits kicks in);
* it always try to pump new data in the cache, keeping it at the memory limit and causing an additional slowdown (my guess is context switching but i may be wrong).
The matter is widely discussed online and the solutions I found online are:
* tune the values of dirty memory depending on your RAM (eg. https://unix.stackexchange.com/quest...workaround-fix), or
* use the BDI interface introduced in Linux 6.2 to limit the writeback cache only for usb disk devices (eg. https://unix.stackexchange.com/quest...ge-stall-issue)
In the Arch world the solution is this package that implements the second method: https://github.com/biglinux/usb-dirty-pages-udev
Trying to use the udev rule on Slackware64-current didn't work for me, so after some debug here's the rule I'm using:
Code:# cat /etc/udev/rules.d/60-usb-dirty-pages-udev.rules
ACTION=="add", ENV{DEVNAME}=="/dev/sd[a-z]", SUBSYSTEM=="block", ENV{ID_TYPE}=="disk", RUN+="/usr/bin/bash -c 'echo 1 > /sys/block/%k/bdi/strict_limit; echo 16777216 > /sys/block/%k/bdi/max_bytes'"I'd like to ask if anybody else ever dealt with this issue and what other solutions they found.