LWN.net

The LWN.net Weekly Edition for May 9, 2019 is available.

<p>The idea of "inheritance" is something that most students learn about earlyon when they are studying object-oriented programming (OOP). But one ofthe seminal books about OOP recommends favoring "composition" overinheritance. Ariel Ortiz came to PyCon in Cleveland, Ohio to describe the composition pattern and to explain the tradeoffs between usingit and inheritance.

Computer memory architecture is growing more complex over time, withdifferent types of memory attached to a CPU via a number of paths. Thekernel development community is duly working to make this memory availableto user space in an equally diverse set of ways. Two sessions at the 2019Linux Storage, Filesystem, and Memory-Management Summit presented possiblemechanisms and APIs to allow programs to work with the types of memory theyneed.

Christoph Lameter has spent years improving Linux for high-performancecomputing tasks. During the memory-management track of the 2019 LinuxStorage, Filesystem, and Memory-Management Summit, he talked about theproblem of keeping up with a 400Gb/s network interface. At that speed,there simply is no time for the system to get its work done. Some ways ofimproving the situation are in sight, but it's a hard problem overall and,despite some progress, the situation is getting worse.

The mmap_sem lock used in the memory-management subsystem has beena known scalability problem for years, but it has proved difficult toremove. During a session in the memory-management track of the 2019 LinuxStorage, Filesystem, and Memory-Management Summit, Laurent Dufour andMatthew Wilcox discussed a possible solution: replacing the red-black treecurrently used to track virtual memory areas (VMAs) with a new datastructure called a "maple tree".

For years, kernel developers have been told to avoid allocating large chunks ofphysically contiguous memory; as the system runs and memory becomesfragmented, satisfying such allocations becomes increasingly difficult.But, as Zi Yan pointed out in a memory-management track session at the 2019Linux Storage, Filesystem, and Memory-Management Summit, there are timeswhen contiguous memory is useful. In this session, the memory-managementdevelopers discussed ways to make such allocations more likely to succeed.

kmalloc() is one of the kernel's fundamental memory-allocationprimitives for relatively small objects. Most of the time, developersdon't worry about the alignment of memory returned fromkmalloc(), and things generally just work. But, Vlastimil Babkasaid during a plenary session at the 2019 Linux Storage,Filesystem, and Memory-Management Summit, every now and thenkmalloc() will do something surprising. He proposed tighteningthe guarantees around object alignment in the hope of generating fewersurprises in the future.

At the 2018 Linux Storage, Filesystem, and Memory-Management Summit(LSFMM), Boaz Harrosh introduced the ZUFSfilesystem. At this year's event, he was back to talk about what itwould take to merge ZUFS into the mainline. ZUFS, which Harrosh pronouncedas both "zoo-eff-ess"and "zoofs", has been running in production for his employer's (NetApp's) customers for some time now, so he wondered if it was something that couldgo upstream.

Stable kernels 5.0.14, 4.19.41, 4.14.117, and 4.9.174 have been released. As usual there areimportant fixes and users should upgrade.

Security updates have been issued by Arch Linux (dovecot, kernel, linux-zen, munin, nautilus, perl-email-address, and tcpreplay), Debian (atftp), Fedora (perl-YAML and teeworlds), Mageia (java-1.8.0-openjdk, ldb, libsolv, and putty/filezilla/wxgtk), openSUSE (freeradius-server, libjpeg-turbo, pacemaker, rubygem-actionpack-5_1, wpa_supplicant, and yubico-piv-tool), Red Hat (chromium-browser, container-tools:rhel8, edk2, firefox, flatpak, ghostscript, httpd:2.4, mod_auth_mellon, openwsman, python-jinja2, python27:2.7, python3, python36:3.6, redhat-virtualization-host, systemd, and wget), SUSE (freeradius-server), and Ubuntu (ghostscript and wpa).

One fixture of the memory-management track at the Linux Storage,Filesystem, and Memory-Management Summit is a discussion with subsystemmaintainer Andrew Mortonon how the development process is going. The 2019 version indicated thatthe memory-management developers are mostly happy with how the process isworking, but there are still things that they would like to see changed. Whilesome of the issues are old and intractable, others may be amenable toshort-term improvement.

The problems associated with the kernel's internal get_user_pages()function have been a topic of discussion at the Linux Storage, Filesystem,and Memory-Management Summit for a few years. At the 2019 event, Jan Kara began a plenary session by saying that it would be "like lastyear's session". It turned out rather differently, though, perhaps due tothe plenary setting; this discussion (along with the related session thatfollowed) turned out to be one of the most heated at the entire conference.

Red Hat has announced therelease of Red Hat Enterprise Linux 8. "Modern IT is hybridIT. But turning a sprawling ecosystem—from traditional datacenters topublic cloud services—into a true hybrid environment requires a fewthings. Scaling as needed. Moving workloads seamlessly. Developing andmanaging applications that run anywhere. There's an operating system thatmakes those things possible. And now it gives you predictive analytics andremediation." See the releasenotes for more information.

Memory control groups exist to track and limit the amount of memory used bysets of processes. Normally, one would not expect that memory used by onegroup would be charged to another but, as Shakeel Butt described in amemory-management track session at the 2019 Linux Storage, Filesystem, andMemory-Management Summit, that does happen in a number of differentsituations. It's often a problem, but occasionally it's also a usefulfeature.

Security updates have been issued by Debian (389-ds-base, firefox-esr, and symfony), Fedora (poppler), SUSE (audit, ovmf, and webkit2gtk3), and Ubuntu (aria2, FFmpeg, gnome-shell, and sudo).

Control groups are a useful mechanism for managing resource usage in thesystem, but what happens when the control groups themselves become aresource problem? In a plenary session at the 2019 Linux Storage,Filesystem, and Memory-Management Summit, Roman Gushchin described problemshe has been facing with deleted control groups that take their time beforeactually going away. Some of these problems have been fixed, but the issuehas not been truly resolved.

Shakeel Butt started his 2019 Linux Storage, Filesystem, andMemory-Management Summit session by noting that memory makes up a big partof the total cost of equipping a data center. As a result, data-centeroperators try to make the best use of memory they can, generallyovercommitting it significantly. In this session, Butt described a schemein use at Google to try to improve memory utilization; while the need forthe described functionality was generally agreed upon, the developers inthe room were not entirely happy with the solution presented.

The performance of the fget() function in the kernel was the topicof a discussion led by Dave Watson at the 2019 Linux Storage, Filesystem,and Memory-Management Summit (LSFMM).fget() is used to take a reference to afile (i.e. bump a reference count), based on its file descriptor, and toreturn the struct file pointer for it; references are dropped withfput(). Some recent profiling at Watson'semployer, Facebook, found the function to be taking a sizable portion ofthe CPU time for some applications, so he wanted to talk about some of thethings he has tried to make that situation better.

In a combined filesystem and storage session at the 2019 Linux Storage, Filesystem, and Memory-Management Summit (LSFMM), Dennis Zhou wanted to talkabout discard, which is the process of sending commands(e.g. TRIM) to block devices to indicateblocks that are no longer in use. Discard is a "serious black box", hesaid; it is a third way to interact with a drive, but Linux developers haveno real insight into what its actual effects will be. That can lead toperformance and other problems.

While persistent memory is normally valued for its persistence, there isalso a subcurrent of interest in using it in settings where persistence isnot important. In particular, the fact that this memory is relativelyinexpensive makes it appealing to use instead of ordinary RAM inbudget-conscious settings. At the 2019 Linux Storage, Filesystem, andMemory-Management Summit, two sessions in the memory-management tracklooked at how the kernel's NUMA mechanism could be pressed into service tomanage non-persistent uses of persistent memory.

Sometimes, the kernel's no-regression rule may not have the desiredresult. Andrea Arcangeli led a session at the 2019 Linux Storage,Filesystem, and Memory-Management Summit to make the point that the recentreversion of a fix after a performance regression was reported has led toworse performance overall — with, as is his wont, a lot of technicalinformation to back up that point. With a wider understanding of what is atstake here, he hopes, the reversion can itself be reverted.

Security updates have been issued by Debian (jquery, librecad, and phpbb3), Fedora (bubblewrap, java-11-openjdk, libvirt, openssh, and pacemaker), Mageia (virtualbox), openSUSE (chromium, ImageMagick, and java-11-openjdk), and SUSE (openssl-1_1).

There is a new Firefox browser release available; its main claim to fame isthat it has a fix for the certificate issue that disabled all extensions.

Linus has released the 5.1 kernel, right onschedule. Some of the significant changes in the release include BPF spinlocks, more year-2038 preparation, the TEO CPU-idle governor, The io_uring fast asynchronous I/O mechanism,initial support for pidfds (file descriptors that refer to a process), theSafeSetID security module, and much more.See the KernelNewbies 5.1page for lots of details.

The5.0.13 and4.19.40stable kernel updates have been released; they were originally due onMay 6, but went out ahead of schedule.

Compound pages are created by the kernel asa way of combining a number of small pages into a single, larger unit.Such pages are implemented as a single "head page" at the beginning,followed by a number of "tail pages". Matthew Wilcox has concluded thatit would be beneficial to minimize the use of tail pages in the kernel; heran a session during the memory-management track at the 2019 Linux Storage,Filesystem, and Memory-Management Summit to explore how that could be done.The discussion ranged widely, veering into the representation of DMA I/Ooperations, but few hard conclusions were reached.

Among the many other things crammed into the page structurethat is usedto represent a page of memory in the kernel is a set of flags to track thestate of the page. These flags have been in short supply for some time;LWN looked at the problem nearly ten yearsago. Jérôme Glisse ran a session during the memory-management track of the2019 Linux Storage, Filesystem, and Memory-Management Summit to exploreways of making some flags available for new uses. While there may be someeasily available bits in the field that holds the page flags, obtaining asignificant number of them may be tricky.

The5.0.12,4.19.39,4.14.116, and4.9.173stable kernel updates have been released; each contains another set ofimportant fixes.

The expiration of the extension signing certificate has evidently caused allextensions to be disabled, leading to a fair amount of discomfort onthe net. There is evidently afix being rolled out, but it requires that the "Studies" mechanism beenabled in the privacy preferences. Meanwhile, the best short-termapproach seems to be to avoid restarting Firefox if possible.

The Alliance for Open Media developedthe AV1 patent-free video codec and sponsors the development of dav1d, a referenceoptimized decoder for AV1. The 0.3.0release of dav1d is now available. "This third release continues to increase the ARM and SSSE3 speed, with more optimizations, as announced, and we get between 12 and 25% speed increases on those CPUs, depending on the samples.However, more surprisingly, we got a speedup on AVX-2 CPU, by optimizing the MSAC (entropy decoding), while we did not find a good solution in the past. This brings 4-5% speed improvements, which is quite huge, knowing the maturity of the AVX-2 code."

Security updates have been issued by Debian (linux-4.9 and otrs2), Fedora (gradle, java-1.8.0-openjdk, jetty, kernel, ruby, and runc), openSUSE (dovecot23, jasper, libsoup, ntfs-3g_ntfsprogs, and webkit2gtk3), SUSE (openssl), and Ubuntu (python-gnupg).

Version 9.1 of the GCC compiler suite is out. "In this release C++17 support is no longer marked experimental. TheC++ front-end implements the full C++17 language (already previous GCCmajor version implemented that) and the C++ standard library support isalmost complete. The C++ front-end and library also have numerous furtherC++2a draft features. GCC has a new front-end for the D language.GCC 9.1 has newly partial OpenMP 5.0 support and almostcomplete OpenACC 2.5 support." See this page for anextensive list of changes.

The userfaultfd()system call allows one process to handle page faults for another — in userspace. Its original use case was to support transparent containermigration, but other uses have developed over the years. At the 2019 LinuxStorage, Filesystem, and Memory-Management Summit, Andrea Arcangelidescribed a scheme to add write-protectionsupport to userfaultfd(). After a year of lost time fightingspeculative-execution problems, Arcangeli is about ready to move this featureinto the mainline.

Since its inception, the DAX mechanism (which provides for direct access tofiles stored on persistent memory) has been seen as somewhat experimentaland incomplete. At the 2019 Linux Storage, Filesystem, andMemory-Management Summit, Dan Williams ran a session where he said thatperhaps the time has come to end that experiment. Some of theunimplemented DAX features may never actually need to be implemented, andit might just be possible to declare DAX finished. But first there are afew more details to take care of.

Stable kernels 5.0.11, 4.19.38, 4.14.115, and 4.9.172 have been released. They all containimportant fixes and users should upgrade.

Security updates have been issued by Debian (proftpd-dfsg and signing-party), Fedora (php-horde-horde and php-horde-turba), and Ubuntu (php5).

Version 1.0.0 of the GNU Guix package manager has been released. "This 1.0 release is a major milestone for Guix. It represents 7 yearsof hard work with more than 40,000 commits by 260 people, 19 releases,and an equally amazing amount of work on documentation, translation,artwork, web design, mentoring, outreach, and many other activities thattogether have made it a thriving project." See thisblog entry for more information.

The LWN.net Weekly Edition for May 2, 2019 is available.

James Bottomley began his talk at the 2019 Linux Storage, Filesystem, andMemory-Management Summit (LSFMM) by noting that the main opposition to his ideaswas not present at the summit, which was likely to mean the ideas got a mucheasier reception than they would have otherwise. In particular, PeterZijlstra and Ingo Molnar expressed some strong reservations to the workthat Bottomley's colleague Mike Rapoport postedrecently; none of those three were in attendance at LSFMM. The idea is touse address spaces to reduce the attack surface available to virtualmachines (VMs) and containers such that kernel bugs of various sorts haveless reach on multi-tenant systems.

The Android system is designed to provide a responsive user experience onsystems that, in a relative sense at least, have limited amounts ofCPU and memory. Doing so requires a number of techniques, includingregular use of a low-memory process killer, that are not seen elsewhere. In amemory-management-track session at the 2019 Linux Storage, Filesystem, andMemory-Management Summit, Suren Baghdasaryan covered a number of issuesrelated to how Android ensures that interactive processes have enoughmemory to get their jobs done.

"People think that memory encryption sounds really cool; it will make mysystem more secure so I want it". At least, that is how Dave Hansencharacterized the situation at the beginning of a session on the topicduring the memory-management track at the 2019 Linux Storage, Filesystem,and Memory-Management Summit. This session, also led by Kirill Shutemov,covered a number of aspects of the memory-encryption problem on Intelprocessors and beyond. One clear outcome of the discussion was also raisedby Hansen at the beginning: users of memory encryption need to think hardabout where that extra security is actually coming from.

The splice()system call is, at its core, a write operation; it attempts to implementzero-copy I/O by moving pages from a pipe to a file. At the 2019 LinuxStorage, Filesystem, and Memory-Management Summit, Miklos Szeredi describeda nascent idea for rsplice() — a "reverse splice" system call.There were not a lot of definitive outcomes from this discussion, but onething was clear: rsplice() needs a much better description (andsome code posted) before the development community can begin to form anopinion on it.

Security updates have been issued by Fedora (libmediainfo, php-horde-horde, and php-horde-turba), SUSE (hostinfo, supportutils, libjpeg-turbo, and openssl), and Ubuntu (dovecot, libpng1.6, and memcached).

The 2019 version of the Linux Storage, Filesystem, and Memory-ManagementSummit opened with a plenary talk by Brendan Gregg on observing the stateof Linux systems using BPF. It is, he said, an exciting time; theBPF-based "superpowers" being added to the kernel are growing in capability andmaturity. It is now possible to ask many questions about what is happeningin a production Linux system without the need for kernel modifications oreven basic debugging information.

Fedora Magazine has announced therelease of Fedora 30. "Fedora Editions are targeted outputs geared toward specific “showcase” uses. Since we first started using this concept in the Fedora 21 release, the needs of the community have continued to evolve. As part of Fedora 30, we’re combining cloud and server into the Fedora Server edition. We’re bringing in Fedora CoreOS to replace Fedora Atomic Host as our container-focused deliverable in the Fedora 30 timeframe — stay tuned for that. The Fedora Workstation edition continues to focus on delivering the latest in open source desktop tools.Of course, we produce more than just the editions. Fedora Spins and Labs target a variety of audiences and use cases, including the Internet of Things. And, we haven’t forgotten our alternate architectures, ARM AArch64, Power, and S390x."

Security updates have been issued by CentOS (kernel, openwsman, and ovmf), Debian (gst-plugins-base1.0 and libvirt), Fedora (libX11, poppler, python-urllib3, samba, and wpewebkit), openSUSE (GraphicsMagick), SUSE (atftp, glibc, libssh2_org, and wpa_supplicant), and Ubuntu (wavpack).

Determining the license that any given package uses can be difficult, but it isessential in order to properly comply with that license and, thus, thedeveloper's wishes. There is an enormous amount of "open source" software available these days thatis not clearly licensed, which is where the ClearlyDefined project comesin. The project is collecting a curated list of packages,source location, and license information; some of that collection can beautomated, but ClearlyDefined is targeting the community to providecuration in the form of cleanups and additions.

The Apache Software Foundation (ASF) and GitHub have announced [ASF,GitHub]that all ASF projects using Git have moved to GitHub and the ASF Git service hasbeen decommissioned. (Thanks to Paul Wise)Update: ASF has anotherannouncement with more details. An older git service was decommissioned,but ASF projects are still available on https://gitbox.apache.org/. "Asstated above, our GitHub integration is an augmentation of our existingservice. It is available to all committers on git-based projects to makeuse of, should they so wish. All new git repositories will automatically beavailable on both GitHub and Gitbox." (Thanks to Lars Francke)

Security updates have been issued by Arch Linux (chromium, libpng, and openssh), Debian (checkstyle, evolution, gst-plugins-base0.10, gst-plugins-base1.0, imagemagick, libpng1.6, monit, and systemd), Fedora (aria2, php-symfony, php-symfony3, php-symfony4, and python-jinja2), openSUSE (ceph, libssh2_org, libvirt, php7, python3, samba, wget, and xerces-c), Red Hat (rh-python35-python), Slackware (bind), SUSE (libssh2_org), and Ubuntu (evince, gst-plugins-base0.10, gst-plugins-base1.0, and mysql-5.7).

Linus has released the 5.1-rc7 kernelprepatch for testing. "But it's all pretty tiny. Plus about 30% ofthe patches are marked for stable, so on the whole it really does feel like5.1 is on target for a regular release next weekend."