LWN.net

The fifthand final article in Adrian Ratiu's serieson eBPF delves into userspace tracing. "In our previous parts we focused on tracing the Linux kernel, for which the eBPF-based projects are, in our humble opinion, the most safe, widely available and useful methods (eBPF is fully upstreamed in Linux, guarantees a stable ABI, comes enabled by default in almost all distributions and integrates with all other tracing mechanisms). It has really become a no-brainer choice for kernel work. However, up until now, talking in-depth about userspace tracing was deliberately avoided because it merits special treatment, hence this full part 5 article devoted to it."

The planning process for the 2019 Linux Kernel and Maintainer's Summits(Lisbon, Portugal, September 9 to 12) hasbegun. If you have a topic that you would like to see discussed at eitherevent, now is the time to send in a proposal to theksummit-discuss list; click below for the details.

<p>Trond Myklebust and Bruce Fields led a session on some topics of interestin the NFS world at the 2019 Linux Storage, Filesystem, andMemory-Management Summit. Myklebust discussed the intersection of NFS andcontainers, as well adding TLS support to NFS. Fields also had somecontainer changes to discuss, along with a grab bag of other areas thatneed attention.

Security updates have been issued by CentOS (flatpak, ghostscript, and python-jinja2), Debian (cups-filters, imagemagick, qt4-x11, and samba), Fedora (httpd and wpa_supplicant), openSUSE (freeradius-server, nmap, python-Jinja2, signing-party, and webkit2gtk3), Red Hat (java-1.7.1-ibm and java-1.8.0-ibm), Scientific Linux (python-jinja2), SUSE (cf-cli, java-1_8_0-openjdk, and libxslt), and Ubuntu (isc-dhcp, openjdk-8, openjdk-lts, samba, and VCFtools).

<p>In the filesystems track at the 2019 Linux Storage, Filesystem, andMemory-Management Summit, Ted Ts'o led a discussion about an inode flag toindicate DAX files, which is meant to be applied to files that should bedirectly accessed without going through the page cache. XFS has such aflag, but ext4 and other filesystems do not. The semantics of what the flag would mean arenot clear to Ts'o (and probably others), so the intent of the discussionwas to try to nail those down.

Security updates have been issued by Debian (atftp, ghostscript, openjdk-7, and postgresql-9.4), Fedora (java-11-openjdk, mosquitto, and php), Mageia (bash, binutils, clamav, cronie, jasper, kernel, mxml, openexr, openssh, python, qt4, svgsalamander, sysstat, tar, and tcpreplay), openSUSE (openssl, python3, sqlite3, webkit2gtk3, and wireshark), Red Hat (bind, flatpak, freeradius:3.0, java-1.8.0-openjdk, python-jinja2, rh-ror42-rubygem-actionpack, rh-ror50-rubygem-actionpack, rh-ruby23-ruby, rh-ruby24-ruby, rh-ruby25-ruby, and thunderbird), SUSE (389-ds, bzip2, ImageMagick, jakarta-commons-fileupload, java-1_8_0-openjdk, pacemaker, python-Django1, samba, and sqlite3), and Ubuntu (postgresql-10, postgresql-11, postgresql-9.5).

The latest set of stable kernel updates consists of5.1.1,5.0.15,4.19.42,4.14.118, and4.9.175.Each contains the usual set of important fixes.

When he released the 5.1 kernel, LinusTorvalds noted that he had a family event happening in the middle of the5.2 merge window and that he would be offline for a few days in the middle.He appears to be trying to make up for lost time before it happens: over8,300 non-merge changesets have found their way into the mainline in thefirst four days. As always, there is a widevariety of work happening all over the kernel tree.

Security updates have been issued by Debian (bind9, postgresql-9.6, qemu, and symfony), Fedora (kernel, kernel-tools, mod_cluster, rubygem-actioncable, rubygem-actionmailer, rubygem-actionpack, rubygem-actionview, rubygem-activejob, rubygem-activemodel, rubygem-activerecord, rubygem-activestorage, rubygem-activesupport, rubygem-rails, and rubygem-railties), openSUSE (wireshark), Red Hat (freeradius), Scientific Linux (freeradius), and Ubuntu (bind9 and wpa).

Eric Rescorla has written adetailed summary of how the Firefox add-on fiasco happened, how it wasaddressed, and how a repeat might be avoided. "First, we should havea much better way of tracking the status of everything in Firefox that is apotential time bomb and making sure that we don’t find ourselves in asituation where one goes off unexpectedly. We’re still working out thedetails here, but at minimum we need to inventory everything of thisnature."

The 2019 Linux Storage, Filesystem, and Memory-Management Summit differedsomewhat from its predecessors in that it contained a fourth trackdedicated to the BPF virtual machine. LWN was unable to attend most ofthose sessions, but a couple of BPF-related talks were a part of thebroader program. Among those was a plenary talk by Dave Miller, describedas "a wholistic view" of why BPF is successful, its current state, andwhere things are going.

Security updates have been issued by Fedora (drupal7, exiv2, filezilla, and libfilezilla), openSUSE (gnutls, GraphicsMagick, hostinfo, supportutils, and ovmf), Scientific Linux (flatpak and ghostscript), SUSE (mutt and samba), and Ubuntu (Monit).

The LWN.net Weekly Edition for May 9, 2019 is available.

<p>The idea of "inheritance" is something that most students learn about earlyon when they are studying object-oriented programming (OOP). But one ofthe seminal books about OOP recommends favoring "composition" overinheritance. Ariel Ortiz came to PyCon in Cleveland, Ohio to describe the composition pattern and to explain the tradeoffs between usingit and inheritance.

Computer memory architecture is growing more complex over time, withdifferent types of memory attached to a CPU via a number of paths. Thekernel development community is duly working to make this memory availableto user space in an equally diverse set of ways. Two sessions at the 2019Linux Storage, Filesystem, and Memory-Management Summit presented possiblemechanisms and APIs to allow programs to work with the types of memory theyneed.

Christoph Lameter has spent years improving Linux for high-performancecomputing tasks. During the memory-management track of the 2019 LinuxStorage, Filesystem, and Memory-Management Summit, he talked about theproblem of keeping up with a 400Gb/s network interface. At that speed,there simply is no time for the system to get its work done. Some ways ofimproving the situation are in sight, but it's a hard problem overall and,despite some progress, the situation is getting worse.

The mmap_sem lock used in the memory-management subsystem has beena known scalability problem for years, but it has proved difficult toremove. During a session in the memory-management track of the 2019 LinuxStorage, Filesystem, and Memory-Management Summit, Laurent Dufour andMatthew Wilcox discussed a possible solution: replacing the red-black treecurrently used to track virtual memory areas (VMAs) with a new datastructure called a "maple tree".

For years, kernel developers have been told to avoid allocating large chunks ofphysically contiguous memory; as the system runs and memory becomesfragmented, satisfying such allocations becomes increasingly difficult.But, as Zi Yan pointed out in a memory-management track session at the 2019Linux Storage, Filesystem, and Memory-Management Summit, there are timeswhen contiguous memory is useful. In this session, the memory-managementdevelopers discussed ways to make such allocations more likely to succeed.

kmalloc() is one of the kernel's fundamental memory-allocationprimitives for relatively small objects. Most of the time, developersdon't worry about the alignment of memory returned fromkmalloc(), and things generally just work. But, Vlastimil Babkasaid during a plenary session at the 2019 Linux Storage,Filesystem, and Memory-Management Summit, every now and thenkmalloc() will do something surprising. He proposed tighteningthe guarantees around object alignment in the hope of generating fewersurprises in the future.

At the 2018 Linux Storage, Filesystem, and Memory-Management Summit(LSFMM), Boaz Harrosh introduced the ZUFSfilesystem. At this year's event, he was back to talk about what itwould take to merge ZUFS into the mainline. ZUFS, which Harrosh pronouncedas both "zoo-eff-ess"and "zoofs", has been running in production for his employer's (NetApp's) customers for some time now, so he wondered if it was something that couldgo upstream.

Stable kernels 5.0.14, 4.19.41, 4.14.117, and 4.9.174 have been released. As usual there areimportant fixes and users should upgrade.

Security updates have been issued by Arch Linux (dovecot, kernel, linux-zen, munin, nautilus, perl-email-address, and tcpreplay), Debian (atftp), Fedora (perl-YAML and teeworlds), Mageia (java-1.8.0-openjdk, ldb, libsolv, and putty/filezilla/wxgtk), openSUSE (freeradius-server, libjpeg-turbo, pacemaker, rubygem-actionpack-5_1, wpa_supplicant, and yubico-piv-tool), Red Hat (chromium-browser, container-tools:rhel8, edk2, firefox, flatpak, ghostscript, httpd:2.4, mod_auth_mellon, openwsman, python-jinja2, python27:2.7, python3, python36:3.6, redhat-virtualization-host, systemd, and wget), SUSE (freeradius-server), and Ubuntu (ghostscript and wpa).

One fixture of the memory-management track at the Linux Storage,Filesystem, and Memory-Management Summit is a discussion with subsystemmaintainer Andrew Mortonon how the development process is going. The 2019 version indicated thatthe memory-management developers are mostly happy with how the process isworking, but there are still things that they would like to see changed. Whilesome of the issues are old and intractable, others may be amenable toshort-term improvement.

The problems associated with the kernel's internal get_user_pages()function have been a topic of discussion at the Linux Storage, Filesystem,and Memory-Management Summit for a few years. At the 2019 event, Jan Kara began a plenary session by saying that it would be "like lastyear's session". It turned out rather differently, though, perhaps due tothe plenary setting; this discussion (along with the related session thatfollowed) turned out to be one of the most heated at the entire conference.

Red Hat has announced therelease of Red Hat Enterprise Linux 8. "Modern IT is hybridIT. But turning a sprawling ecosystem—from traditional datacenters topublic cloud services—into a true hybrid environment requires a fewthings. Scaling as needed. Moving workloads seamlessly. Developing andmanaging applications that run anywhere. There's an operating system thatmakes those things possible. And now it gives you predictive analytics andremediation." See the releasenotes for more information.

Memory control groups exist to track and limit the amount of memory used bysets of processes. Normally, one would not expect that memory used by onegroup would be charged to another but, as Shakeel Butt described in amemory-management track session at the 2019 Linux Storage, Filesystem, andMemory-Management Summit, that does happen in a number of differentsituations. It's often a problem, but occasionally it's also a usefulfeature.

Security updates have been issued by Debian (389-ds-base, firefox-esr, and symfony), Fedora (poppler), SUSE (audit, ovmf, and webkit2gtk3), and Ubuntu (aria2, FFmpeg, gnome-shell, and sudo).

Control groups are a useful mechanism for managing resource usage in thesystem, but what happens when the control groups themselves become aresource problem? In a plenary session at the 2019 Linux Storage,Filesystem, and Memory-Management Summit, Roman Gushchin described problemshe has been facing with deleted control groups that take their time beforeactually going away. Some of these problems have been fixed, but the issuehas not been truly resolved.

Shakeel Butt started his 2019 Linux Storage, Filesystem, andMemory-Management Summit session by noting that memory makes up a big partof the total cost of equipping a data center. As a result, data-centeroperators try to make the best use of memory they can, generallyovercommitting it significantly. In this session, Butt described a schemein use at Google to try to improve memory utilization; while the need forthe described functionality was generally agreed upon, the developers inthe room were not entirely happy with the solution presented.

The performance of the fget() function in the kernel was the topicof a discussion led by Dave Watson at the 2019 Linux Storage, Filesystem,and Memory-Management Summit (LSFMM).fget() is used to take a reference to afile (i.e. bump a reference count), based on its file descriptor, and toreturn the struct file pointer for it; references are dropped withfput(). Some recent profiling at Watson'semployer, Facebook, found the function to be taking a sizable portion ofthe CPU time for some applications, so he wanted to talk about some of thethings he has tried to make that situation better.

In a combined filesystem and storage session at the 2019 Linux Storage, Filesystem, and Memory-Management Summit (LSFMM), Dennis Zhou wanted to talkabout discard, which is the process of sending commands(e.g. TRIM) to block devices to indicateblocks that are no longer in use. Discard is a "serious black box", hesaid; it is a third way to interact with a drive, but Linux developers haveno real insight into what its actual effects will be. That can lead toperformance and other problems.

While persistent memory is normally valued for its persistence, there isalso a subcurrent of interest in using it in settings where persistence isnot important. In particular, the fact that this memory is relativelyinexpensive makes it appealing to use instead of ordinary RAM inbudget-conscious settings. At the 2019 Linux Storage, Filesystem, andMemory-Management Summit, two sessions in the memory-management tracklooked at how the kernel's NUMA mechanism could be pressed into service tomanage non-persistent uses of persistent memory.

Sometimes, the kernel's no-regression rule may not have the desiredresult. Andrea Arcangeli led a session at the 2019 Linux Storage,Filesystem, and Memory-Management Summit to make the point that the recentreversion of a fix after a performance regression was reported has led toworse performance overall — with, as is his wont, a lot of technicalinformation to back up that point. With a wider understanding of what is atstake here, he hopes, the reversion can itself be reverted.

Security updates have been issued by Debian (jquery, librecad, and phpbb3), Fedora (bubblewrap, java-11-openjdk, libvirt, openssh, and pacemaker), Mageia (virtualbox), openSUSE (chromium, ImageMagick, and java-11-openjdk), and SUSE (openssl-1_1).

There is a new Firefox browser release available; its main claim to fame isthat it has a fix for the certificate issue that disabled all extensions.

Linus has released the 5.1 kernel, right onschedule. Some of the significant changes in the release include BPF spinlocks, more year-2038 preparation, the TEO CPU-idle governor, The io_uring fast asynchronous I/O mechanism,initial support for pidfds (file descriptors that refer to a process), theSafeSetID security module, and much more.See the KernelNewbies 5.1page for lots of details.

The5.0.13 and4.19.40stable kernel updates have been released; they were originally due onMay 6, but went out ahead of schedule.

Compound pages are created by the kernel asa way of combining a number of small pages into a single, larger unit.Such pages are implemented as a single "head page" at the beginning,followed by a number of "tail pages". Matthew Wilcox has concluded thatit would be beneficial to minimize the use of tail pages in the kernel; heran a session during the memory-management track at the 2019 Linux Storage,Filesystem, and Memory-Management Summit to explore how that could be done.The discussion ranged widely, veering into the representation of DMA I/Ooperations, but few hard conclusions were reached.

Among the many other things crammed into the page structurethat is usedto represent a page of memory in the kernel is a set of flags to track thestate of the page. These flags have been in short supply for some time;LWN looked at the problem nearly ten yearsago. Jérôme Glisse ran a session during the memory-management track of the2019 Linux Storage, Filesystem, and Memory-Management Summit to exploreways of making some flags available for new uses. While there may be someeasily available bits in the field that holds the page flags, obtaining asignificant number of them may be tricky.

The5.0.12,4.19.39,4.14.116, and4.9.173stable kernel updates have been released; each contains another set ofimportant fixes.

The expiration of the extension signing certificate has evidently caused allextensions to be disabled, leading to a fair amount of discomfort onthe net. There is evidently afix being rolled out, but it requires that the "Studies" mechanism beenabled in the privacy preferences. Meanwhile, the best short-termapproach seems to be to avoid restarting Firefox if possible.

The Alliance for Open Media developedthe AV1 patent-free video codec and sponsors the development of dav1d, a referenceoptimized decoder for AV1. The 0.3.0release of dav1d is now available. "This third release continues to increase the ARM and SSSE3 speed, with more optimizations, as announced, and we get between 12 and 25% speed increases on those CPUs, depending on the samples.However, more surprisingly, we got a speedup on AVX-2 CPU, by optimizing the MSAC (entropy decoding), while we did not find a good solution in the past. This brings 4-5% speed improvements, which is quite huge, knowing the maturity of the AVX-2 code."

Security updates have been issued by Debian (linux-4.9 and otrs2), Fedora (gradle, java-1.8.0-openjdk, jetty, kernel, ruby, and runc), openSUSE (dovecot23, jasper, libsoup, ntfs-3g_ntfsprogs, and webkit2gtk3), SUSE (openssl), and Ubuntu (python-gnupg).

Version 9.1 of the GCC compiler suite is out. "In this release C++17 support is no longer marked experimental. TheC++ front-end implements the full C++17 language (already previous GCCmajor version implemented that) and the C++ standard library support isalmost complete. The C++ front-end and library also have numerous furtherC++2a draft features. GCC has a new front-end for the D language.GCC 9.1 has newly partial OpenMP 5.0 support and almostcomplete OpenACC 2.5 support." See this page for anextensive list of changes.

The userfaultfd()system call allows one process to handle page faults for another — in userspace. Its original use case was to support transparent containermigration, but other uses have developed over the years. At the 2019 LinuxStorage, Filesystem, and Memory-Management Summit, Andrea Arcangelidescribed a scheme to add write-protectionsupport to userfaultfd(). After a year of lost time fightingspeculative-execution problems, Arcangeli is about ready to move this featureinto the mainline.

Since its inception, the DAX mechanism (which provides for direct access tofiles stored on persistent memory) has been seen as somewhat experimentaland incomplete. At the 2019 Linux Storage, Filesystem, andMemory-Management Summit, Dan Williams ran a session where he said thatperhaps the time has come to end that experiment. Some of theunimplemented DAX features may never actually need to be implemented, andit might just be possible to declare DAX finished. But first there are afew more details to take care of.

Stable kernels 5.0.11, 4.19.38, 4.14.115, and 4.9.172 have been released. They all containimportant fixes and users should upgrade.

Security updates have been issued by Debian (proftpd-dfsg and signing-party), Fedora (php-horde-horde and php-horde-turba), and Ubuntu (php5).

Version 1.0.0 of the GNU Guix package manager has been released. "This 1.0 release is a major milestone for Guix. It represents 7 yearsof hard work with more than 40,000 commits by 260 people, 19 releases,and an equally amazing amount of work on documentation, translation,artwork, web design, mentoring, outreach, and many other activities thattogether have made it a thriving project." See thisblog entry for more information.

The LWN.net Weekly Edition for May 2, 2019 is available.