LWN.net

Stable kernels 5.0.14, 4.19.41, 4.14.117, and 4.9.174 have been released. As usual there areimportant fixes and users should upgrade.

Security updates have been issued by Arch Linux (dovecot, kernel, linux-zen, munin, nautilus, perl-email-address, and tcpreplay), Debian (atftp), Fedora (perl-YAML and teeworlds), Mageia (java-1.8.0-openjdk, ldb, libsolv, and putty/filezilla/wxgtk), openSUSE (freeradius-server, libjpeg-turbo, pacemaker, rubygem-actionpack-5_1, wpa_supplicant, and yubico-piv-tool), Red Hat (chromium-browser, container-tools:rhel8, edk2, firefox, flatpak, ghostscript, httpd:2.4, mod_auth_mellon, openwsman, python-jinja2, python27:2.7, python3, python36:3.6, redhat-virtualization-host, systemd, and wget), SUSE (freeradius-server), and Ubuntu (ghostscript and wpa).

One fixture of the memory-management track at the Linux Storage,Filesystem, and Memory-Management Summit is a discussion with subsystemmaintainer Andrew Mortonon how the development process is going. The 2019 version indicated thatthe memory-management developers are mostly happy with how the process isworking, but there are still things that they would like to see changed. Whilesome of the issues are old and intractable, others may be amenable toshort-term improvement.

The problems associated with the kernel's internal get_user_pages()function have been a topic of discussion at the Linux Storage, Filesystem,and Memory-Management Summit for a few years. At the 2019 event, Jan Kara began a plenary session by saying that it would be "like lastyear's session". It turned out rather differently, though, perhaps due tothe plenary setting; this discussion (along with the related session thatfollowed) turned out to be one of the most heated at the entire conference.

Red Hat has announced therelease of Red Hat Enterprise Linux 8. "Modern IT is hybridIT. But turning a sprawling ecosystem—from traditional datacenters topublic cloud services—into a true hybrid environment requires a fewthings. Scaling as needed. Moving workloads seamlessly. Developing andmanaging applications that run anywhere. There's an operating system thatmakes those things possible. And now it gives you predictive analytics andremediation." See the releasenotes for more information.

Memory control groups exist to track and limit the amount of memory used bysets of processes. Normally, one would not expect that memory used by onegroup would be charged to another but, as Shakeel Butt described in amemory-management track session at the 2019 Linux Storage, Filesystem, andMemory-Management Summit, that does happen in a number of differentsituations. It's often a problem, but occasionally it's also a usefulfeature.

Security updates have been issued by Debian (389-ds-base, firefox-esr, and symfony), Fedora (poppler), SUSE (audit, ovmf, and webkit2gtk3), and Ubuntu (aria2, FFmpeg, gnome-shell, and sudo).

Control groups are a useful mechanism for managing resource usage in thesystem, but what happens when the control groups themselves become aresource problem? In a plenary session at the 2019 Linux Storage,Filesystem, and Memory-Management Summit, Roman Gushchin described problemshe has been facing with deleted control groups that take their time beforeactually going away. Some of these problems have been fixed, but the issuehas not been truly resolved.

Shakeel Butt started his 2019 Linux Storage, Filesystem, andMemory-Management Summit session by noting that memory makes up a big partof the total cost of equipping a data center. As a result, data-centeroperators try to make the best use of memory they can, generallyovercommitting it significantly. In this session, Butt described a schemein use at Google to try to improve memory utilization; while the need forthe described functionality was generally agreed upon, the developers inthe room were not entirely happy with the solution presented.

The performance of the fget() function in the kernel was the topicof a discussion led by Dave Watson at the 2019 Linux Storage, Filesystem,and Memory-Management Summit (LSFMM).fget() is used to take a reference to afile (i.e. bump a reference count), based on its file descriptor, and toreturn the struct file pointer for it; references are dropped withfput(). Some recent profiling at Watson'semployer, Facebook, found the function to be taking a sizable portion ofthe CPU time for some applications, so he wanted to talk about some of thethings he has tried to make that situation better.

In a combined filesystem and storage session at the 2019 Linux Storage, Filesystem, and Memory-Management Summit (LSFMM), Dennis Zhou wanted to talkabout discard, which is the process of sending commands(e.g. TRIM) to block devices to indicateblocks that are no longer in use. Discard is a "serious black box", hesaid; it is a third way to interact with a drive, but Linux developers haveno real insight into what its actual effects will be. That can lead toperformance and other problems.

While persistent memory is normally valued for its persistence, there isalso a subcurrent of interest in using it in settings where persistence isnot important. In particular, the fact that this memory is relativelyinexpensive makes it appealing to use instead of ordinary RAM inbudget-conscious settings. At the 2019 Linux Storage, Filesystem, andMemory-Management Summit, two sessions in the memory-management tracklooked at how the kernel's NUMA mechanism could be pressed into service tomanage non-persistent uses of persistent memory.

Sometimes, the kernel's no-regression rule may not have the desiredresult. Andrea Arcangeli led a session at the 2019 Linux Storage,Filesystem, and Memory-Management Summit to make the point that the recentreversion of a fix after a performance regression was reported has led toworse performance overall — with, as is his wont, a lot of technicalinformation to back up that point. With a wider understanding of what is atstake here, he hopes, the reversion can itself be reverted.

Security updates have been issued by Debian (jquery, librecad, and phpbb3), Fedora (bubblewrap, java-11-openjdk, libvirt, openssh, and pacemaker), Mageia (virtualbox), openSUSE (chromium, ImageMagick, and java-11-openjdk), and SUSE (openssl-1_1).

There is a new Firefox browser release available; its main claim to fame isthat it has a fix for the certificate issue that disabled all extensions.

Linus has released the 5.1 kernel, right onschedule. Some of the significant changes in the release include BPF spinlocks, more year-2038 preparation, the TEO CPU-idle governor, The io_uring fast asynchronous I/O mechanism,initial support for pidfds (file descriptors that refer to a process), theSafeSetID security module, and much more.See the KernelNewbies 5.1page for lots of details.

The5.0.13 and4.19.40stable kernel updates have been released; they were originally due onMay 6, but went out ahead of schedule.

Compound pages are created by the kernel asa way of combining a number of small pages into a single, larger unit.Such pages are implemented as a single "head page" at the beginning,followed by a number of "tail pages". Matthew Wilcox has concluded thatit would be beneficial to minimize the use of tail pages in the kernel; heran a session during the memory-management track at the 2019 Linux Storage,Filesystem, and Memory-Management Summit to explore how that could be done.The discussion ranged widely, veering into the representation of DMA I/Ooperations, but few hard conclusions were reached.

Among the many other things crammed into the page structurethat is usedto represent a page of memory in the kernel is a set of flags to track thestate of the page. These flags have been in short supply for some time;LWN looked at the problem nearly ten yearsago. Jérôme Glisse ran a session during the memory-management track of the2019 Linux Storage, Filesystem, and Memory-Management Summit to exploreways of making some flags available for new uses. While there may be someeasily available bits in the field that holds the page flags, obtaining asignificant number of them may be tricky.

The5.0.12,4.19.39,4.14.116, and4.9.173stable kernel updates have been released; each contains another set ofimportant fixes.

The expiration of the extension signing certificate has evidently caused allextensions to be disabled, leading to a fair amount of discomfort onthe net. There is evidently afix being rolled out, but it requires that the "Studies" mechanism beenabled in the privacy preferences. Meanwhile, the best short-termapproach seems to be to avoid restarting Firefox if possible.

The Alliance for Open Media developedthe AV1 patent-free video codec and sponsors the development of dav1d, a referenceoptimized decoder for AV1. The 0.3.0release of dav1d is now available. "This third release continues to increase the ARM and SSSE3 speed, with more optimizations, as announced, and we get between 12 and 25% speed increases on those CPUs, depending on the samples.However, more surprisingly, we got a speedup on AVX-2 CPU, by optimizing the MSAC (entropy decoding), while we did not find a good solution in the past. This brings 4-5% speed improvements, which is quite huge, knowing the maturity of the AVX-2 code."

Security updates have been issued by Debian (linux-4.9 and otrs2), Fedora (gradle, java-1.8.0-openjdk, jetty, kernel, ruby, and runc), openSUSE (dovecot23, jasper, libsoup, ntfs-3g_ntfsprogs, and webkit2gtk3), SUSE (openssl), and Ubuntu (python-gnupg).

Version 9.1 of the GCC compiler suite is out. "In this release C++17 support is no longer marked experimental. TheC++ front-end implements the full C++17 language (already previous GCCmajor version implemented that) and the C++ standard library support isalmost complete. The C++ front-end and library also have numerous furtherC++2a draft features. GCC has a new front-end for the D language.GCC 9.1 has newly partial OpenMP 5.0 support and almostcomplete OpenACC 2.5 support." See this page for anextensive list of changes.

The userfaultfd()system call allows one process to handle page faults for another — in userspace. Its original use case was to support transparent containermigration, but other uses have developed over the years. At the 2019 LinuxStorage, Filesystem, and Memory-Management Summit, Andrea Arcangelidescribed a scheme to add write-protectionsupport to userfaultfd(). After a year of lost time fightingspeculative-execution problems, Arcangeli is about ready to move this featureinto the mainline.

Since its inception, the DAX mechanism (which provides for direct access tofiles stored on persistent memory) has been seen as somewhat experimentaland incomplete. At the 2019 Linux Storage, Filesystem, andMemory-Management Summit, Dan Williams ran a session where he said thatperhaps the time has come to end that experiment. Some of theunimplemented DAX features may never actually need to be implemented, andit might just be possible to declare DAX finished. But first there are afew more details to take care of.

Stable kernels 5.0.11, 4.19.38, 4.14.115, and 4.9.172 have been released. They all containimportant fixes and users should upgrade.

Security updates have been issued by Debian (proftpd-dfsg and signing-party), Fedora (php-horde-horde and php-horde-turba), and Ubuntu (php5).

Version 1.0.0 of the GNU Guix package manager has been released. "This 1.0 release is a major milestone for Guix. It represents 7 yearsof hard work with more than 40,000 commits by 260 people, 19 releases,and an equally amazing amount of work on documentation, translation,artwork, web design, mentoring, outreach, and many other activities thattogether have made it a thriving project." See thisblog entry for more information.

The LWN.net Weekly Edition for May 2, 2019 is available.

James Bottomley began his talk at the 2019 Linux Storage, Filesystem, andMemory-Management Summit (LSFMM) by noting that the main opposition to his ideaswas not present at the summit, which was likely to mean the ideas got a mucheasier reception than they would have otherwise. In particular, PeterZijlstra and Ingo Molnar expressed some strong reservations to the workthat Bottomley's colleague Mike Rapoport postedrecently; none of those three were in attendance at LSFMM. The idea is touse address spaces to reduce the attack surface available to virtualmachines (VMs) and containers such that kernel bugs of various sorts haveless reach on multi-tenant systems.

The Android system is designed to provide a responsive user experience onsystems that, in a relative sense at least, have limited amounts ofCPU and memory. Doing so requires a number of techniques, includingregular use of a low-memory process killer, that are not seen elsewhere. In amemory-management-track session at the 2019 Linux Storage, Filesystem, andMemory-Management Summit, Suren Baghdasaryan covered a number of issuesrelated to how Android ensures that interactive processes have enoughmemory to get their jobs done.

"People think that memory encryption sounds really cool; it will make mysystem more secure so I want it". At least, that is how Dave Hansencharacterized the situation at the beginning of a session on the topicduring the memory-management track at the 2019 Linux Storage, Filesystem,and Memory-Management Summit. This session, also led by Kirill Shutemov,covered a number of aspects of the memory-encryption problem on Intelprocessors and beyond. One clear outcome of the discussion was also raisedby Hansen at the beginning: users of memory encryption need to think hardabout where that extra security is actually coming from.

The splice()system call is, at its core, a write operation; it attempts to implementzero-copy I/O by moving pages from a pipe to a file. At the 2019 LinuxStorage, Filesystem, and Memory-Management Summit, Miklos Szeredi describeda nascent idea for rsplice() — a "reverse splice" system call.There were not a lot of definitive outcomes from this discussion, but onething was clear: rsplice() needs a much better description (andsome code posted) before the development community can begin to form anopinion on it.

Security updates have been issued by Fedora (libmediainfo, php-horde-horde, and php-horde-turba), SUSE (hostinfo, supportutils, libjpeg-turbo, and openssl), and Ubuntu (dovecot, libpng1.6, and memcached).

The 2019 version of the Linux Storage, Filesystem, and Memory-ManagementSummit opened with a plenary talk by Brendan Gregg on observing the stateof Linux systems using BPF. It is, he said, an exciting time; theBPF-based "superpowers" being added to the kernel are growing in capability andmaturity. It is now possible to ask many questions about what is happeningin a production Linux system without the need for kernel modifications oreven basic debugging information.

Fedora Magazine has announced therelease of Fedora 30. "Fedora Editions are targeted outputs geared toward specific “showcase” uses. Since we first started using this concept in the Fedora 21 release, the needs of the community have continued to evolve. As part of Fedora 30, we’re combining cloud and server into the Fedora Server edition. We’re bringing in Fedora CoreOS to replace Fedora Atomic Host as our container-focused deliverable in the Fedora 30 timeframe — stay tuned for that. The Fedora Workstation edition continues to focus on delivering the latest in open source desktop tools.Of course, we produce more than just the editions. Fedora Spins and Labs target a variety of audiences and use cases, including the Internet of Things. And, we haven’t forgotten our alternate architectures, ARM AArch64, Power, and S390x."

Security updates have been issued by CentOS (kernel, openwsman, and ovmf), Debian (gst-plugins-base1.0 and libvirt), Fedora (libX11, poppler, python-urllib3, samba, and wpewebkit), openSUSE (GraphicsMagick), SUSE (atftp, glibc, libssh2_org, and wpa_supplicant), and Ubuntu (wavpack).

Determining the license that any given package uses can be difficult, but it isessential in order to properly comply with that license and, thus, thedeveloper's wishes. There is an enormous amount of "open source" software available these days thatis not clearly licensed, which is where the ClearlyDefined project comesin. The project is collecting a curated list of packages,source location, and license information; some of that collection can beautomated, but ClearlyDefined is targeting the community to providecuration in the form of cleanups and additions.

The Apache Software Foundation (ASF) and GitHub have announced [ASF,GitHub]that all ASF projects using Git have moved to GitHub and the ASF Git service hasbeen decommissioned. (Thanks to Paul Wise)Update: ASF has anotherannouncement with more details. An older git service was decommissioned,but ASF projects are still available on https://gitbox.apache.org/. "Asstated above, our GitHub integration is an augmentation of our existingservice. It is available to all committers on git-based projects to makeuse of, should they so wish. All new git repositories will automatically beavailable on both GitHub and Gitbox." (Thanks to Lars Francke)

Security updates have been issued by Arch Linux (chromium, libpng, and openssh), Debian (checkstyle, evolution, gst-plugins-base0.10, gst-plugins-base1.0, imagemagick, libpng1.6, monit, and systemd), Fedora (aria2, php-symfony, php-symfony3, php-symfony4, and python-jinja2), openSUSE (ceph, libssh2_org, libvirt, php7, python3, samba, wget, and xerces-c), Red Hat (rh-python35-python), Slackware (bind), SUSE (libssh2_org), and Ubuntu (evince, gst-plugins-base0.10, gst-plugins-base1.0, and mysql-5.7).

Linus has released the 5.1-rc7 kernelprepatch for testing. "But it's all pretty tiny. Plus about 30% ofthe patches are marked for stable, so on the whole it really does feel like5.1 is on target for a regular release next weekend."

The5.0.10,4.19.37,4.14.114,4.9.171,4.4.179, and3.18.139stable kernel updates have all been released; each contains a moderatelylarge set of important fixes.

Adrian Ratiu continues his series on eBPF with part 3, which looks at various ways to write and build eBPF programs. It starts by looking at using "restricted C" with the LLVM eBPF compiler, moves into looking at the BPF Compiler Collection (BCC), then bpftrace, and finally the IOVisor cloud-based eBPF tools."Not everyone has kernel sources at hand, especially in production, and it's also a bad idea in general to tie eBPF-based tools to a specific kernel source revision. Designing and implementing the interactions between eBPF program's backends, frontends, loaders and data structures can be very complex, error-prone and time consuming, especially in C which is considered a dangerous low-level [language]. In addition to these risks developers are also in a constant danger of re-inventing the wheel for common problems, with endless design variations and implementations. To alleviate all these pains is why the BCC project exists: it provides an easy-to-use framework for writing, loading and running eBPF programs, by writing simple python or lua scripts in addition to the 'restricted C' as exemplified above."

The recently discovered vulnerability inThunderbolt has restarted discussions about protecting the kernelagainst untrusted, hotpluggable hardware. That vulnerability, known as Thunderclap, allows a hostile externaldevice to exploit Input-OutputMemory Management Unit (IOMMU) mapping limitations and access systemmemory it was not intended to. Thunderclap can be exploited byUSB-C-connected devices; while we have seen USB attacks in the past, thisvulnerability is different in that PCI devices, often considered astrusted, can be a source of attacks too. One way of stopping those attackswould be to make sure that the IOMMU is used correctly and restricts the deviceto accessing the memory that was allocated for it. Lu Baolu has postedan implementation of that approach in the form of bounce buffers foruntrusted devices.

Security updates have been issued by Debian (gpac and mercurial), Fedora (kernel-headers and kernel-tools), openSUSE (GraphicsMagick, kauth, lxc, lxcfs, python, qemu, and xmltooling), SUSE (freeradius-server, ImageMagick, libvirt, samba, and wireshark), and Ubuntu (bind9).

Over at Opensource.com, Jason Brock tries out Linux graphics tools, with an eye toward their ability to replace the proprietary tools he uses on a day-to-day basis. Overall, the tools held their own for a variety of tasks (e.g. logo and ad design, publication layout), though the lack of a certain type of tool brought the overall grade down to a B+: "The lack of available wireframing and prototyping applications really brought down the average, but I'd still call it a successful exercise. As I mentioned at the beginning, design is a craft and it relies on collaboration. All of the tools I looked at—Inkscape, LibreDraw, GIMP, and Scribus—can run just as well on Windows or MacOS as they do on any Linux distribution. The ability to create robust artwork and share editable files with stakeholders and colleagues on the platform of their choice means that a serious argument could be made that these tools are even more versatile than their proprietary counterparts."

The release of the 5.1-rc6 kernel prepatchon April 21 indicates that the 5.1 development cycle is getting closeto its conclusion. So naturally the time has come to put together somestatistics describing where the changes merged for 5.1 came from. It is,for the most part, a fairly typical development cycle.

Security updates have been issued by Debian (putty and systemd), Fedora (kernel, kernel-headers, and kernel-tools), Gentoo (ming and qemu), openSUSE (openexr and slurm), SUSE (ImageMagick, jasper, ntfs-3g_ntfsprogs, openssh, and webkit2gtk3), and Ubuntu (php5 and tcpflow).

The LWN.net Weekly Edition for April 25, 2019 is available.