LWN.net

The Fedora 28 release has been announced."The headline feature for Fedora 28 Server is the inclusion of thenew Modular repository. This lets you select between different versions ofsoftware like NodeJS or Django, so you can chose the stack you need foryour software." Some users will also appreciate that proprietaryblobs (such as the NVIDIA drivers) are now easier to obtain and install.

Matthew "Willy" Wilcox has been doing a fair amount of work in thememory-management area recently. He showed up at the 2018 Linux Storage,Filesystem, and Memory-Management Summit with a list of discussion topicsrelated to that work; it was enough to fill a plenary session with somespillover into the memory-management track the next day. Some of histopics were fairly straightforward; others look to be somewhat moreinvolved.

The kernel's memory-management subsystem has to manage a great deal ofconcurrency; that leads to an ongoing series of locking challenges thatsometimes seem intractable. Two recurring locking issues — the LRU locksand the mmap_sem lock — were the topic of sessions held during thememory-management track of the 2018 Linux Storage, Filesystem, andMemory-Management Summit. In both cases, it quickly became clear that,while some interesting ideas are being pursued, easysolutions are not on offer.

When kernel code needs to work directly with user-space pages, it oftencalls get_user_pages()(or one of several variants) to fault those pages into RAM and pin themthere. This function is not entirely easy to use, though, and recentchanges have made it harder to use safely. Jan Kara and Dan Williams led aplenary session at the 2018 Linux Storage, Filesystem, andMemory-Management Summit to discuss potential solutions, but it is notentirely clear that any were found.

Stable kernels 4.16.6, 4.14.38, 4.9.97, 4.4.130, and 3.18.107 have been released. They all containimportant fixes throughout the tree and users should upgrade.

Security updates have been issued by Arch Linux (drupal), Debian (chromium-browser, gunicorn, libvorbis, openjdk-8, roundcube, sdl-image1.2, slurm-llnl, and tor), Fedora (boost, cups-filters, ghostscript, gsoap, memcached, mod_http2, and qpdf), openSUSE (Chromium and mysql-community-server), and Red Hat (glusterfs, OpenShift Container Platform 3.1, OpenShift Container Platform 3.2, OpenShift Container Platform 3.3, OpenShift Container Platform 3.4, OpenShift Container Platform 3.5, OpenShift Container Platform 3.6, OpenShift Container Platform 3.7, OpenShift Container Platform 3.8, OpenShift Container Platform 3.9, and openvswitch).

The 4.17-rc3 kernel prepatch is out."And by now, I think we've fixed all the nastiest fall-out from themerge window. In particular, the PTI large-page fallout that hit somepeople with particular configurations should all be good."

The memory-management subsystem is maintained by a small but dedicatedgroup of developers. How healthy is that development community? MichalHocko raised that question during the memory-management track at the 2018Linux Storage, Filesystem, and Memory-Management Summit. Hocko is worried,but it appears that his concerns are not universally felt.

The non-uniform memory architecture (NUMA) was designed around the ideathat there are two types of memory on complex systems: local (faster) andremote (slower). During the memory-management track of the 2018 LinuxStorage, Filesystem, and Memory-Management Summit, Anshuman Khandualasserted that the situation has since become rather more complicated.Perhaps, he said, the time has come to rethink how we view NUMA systems.

Storage devices are in a period of extensive change. As theyget faster and become byte-addressable by the CPU, they tend to lookincreasingly like ordinary memory. But they aren't memory, so it stillisn't clear what the best model for accessing them should be. AdamManzanares led a session during the memory-management track of the 2018Linux Storage, Filesystem, and Memory-Management Summit, where his proposalof a new access mechanism ran into some skepticism.

Heterogeneous memory management (HMM) is a relatively new kernel subsystemthat allows the system to manage peripherals (such as graphics processors)that have their own memory-management units. In two sessions during thememory-management track of the 2018 Linux Storage,Filesystem, and Memory-Management Summit, HMM creator Jérôme Glisseprovided an update on the status of this subsystem and where it is going,along with a more detailed look at the memory-management unit (MMU)notifiers mechanism on which it depends.

At the 2018 Linux Storage, Filesystem, and Memory Management Summit, TedTs'o introduced an integrity feature akin to dm-verity that targets Android,at least to start with. It is meant to protect the integrity of files onthe system so that any tampering would be detectable. Theinitial use case would be for a certain special type of Android file, but othersystems may find uses for it as well.

Security updates have been issued by Debian (wordpress), Fedora (boost), openSUSE (perl and zsh), Oracle (kernel), Red Hat (apr), and Slackware (openvpn).

Ubuntu 18.04, a long-term-support release, is out."Codenamed 'Bionic Beaver', 18.04 LTS continues Ubuntu's proud traditionof integrating the latest and greatest open source technologies into ahigh-quality, easy-to-use Linux distribution. The team has been hard atwork through this cycle, introducing new features and fixing bugs."It features a 4.15 kernel, a new GNOME-based desktop environment, andmore. See therelease notes and this overview for details.

Christian Schaller looksforward to the Fedora 28 release (which will evidently be the first on-time Fedora release ever)."The Spectre/Meltdown situation did hammer home to a lot of peoplethe need to have firmware updates easily available and easy to update. Wecreated the Linux Vendor Firmware service for Fedora Workstation users withthat in mind and it was great to see the service paying off for many Linuxusers, not only on Fedora, but also on other distributions who startedusing the service we provided. I would like to call out to Dell who was acritical partner for the Linux Vendor Firmware effort from day 1 and thustheir users got the most benefit from it when Spectre and Meltdownhit. Spectre and Meltdown also helped get a lot of other vendors off thefence or to accelerate their efforts to support LVFS and Richard Hughes andPeter Jones have been working closely with a lot of new vendors during thiscycle to get support for their hardware and devices into LVFS."

The page structure is one of themost complex in the kernel due to the need to cram the maximum amount ofinformation into as little space as possible. Each field is so heavilyoverloaded that developers prefer to avoid making changes to structpage if they can avoid it. That didn't deter Jérôme Glisse from proposing a significant changeduring two plenary sessions atthe 2018 Linux Storage, Filesystem, and Memory-Management Summit, though.There are some interesting benefits on offer, but getting there will not bea simple task.

Stable kernels 4.16.5 and 4.14.37 have been released. They both containimportant fixes and users should upgrade.

Security updates have been issued by Debian (drupal7, gcc-4.9-backport, ghostscript, and openslp-dfsg), Fedora (anki, composer, perl, and perl-Module-CoreList), Red Hat (kernel and rh-mysql56-mysql), and SUSE (kernel, kvm, and zsh).

The LWN.net Weekly Edition for April 26, 2018 is available.

Once a niche feature, memory encryption is becoming mainstream with supportin both Intel and AMD processors, Kirill Shutemov said at the beginning ofhis session during the memory-management track of the 2018 Linux Storage,Filesystem, and Memory-Management Summit. Memory encryption can harden thesystem against attack, but it also presents some interesting challenges forthe kernel.

Security updates have been issued by Debian (lucene-solr and psensor), Oracle (librelp and PackageKit), Red Hat (kernel, librelp, and PackageKit), Scientific Linux (librelp), and Ubuntu (mysql-5.5 and packagekit).

After a session at last year's LinuxStorage, Filesystem, and Memory Management Summit (LSFMM), Jeff Layton was able tomake some improvements to block-layer errorhandling. Those changes, which added a newerrseq_t type to hold an error number and sequence number, seemedto help and were well received—except by the PostgreSQLdevelopers. So Layton led a session at the 2018 LSFMM to discuss waysto improve things further; it would be followed later in the week with asession by one of the PostgreSQL developers to look at the specifics of theproblem from their perspective.

Using the kernel thread (kthread) freezer has been a longtime problemfor a variety of reasons. It is meant as a way to suspend kthreads on theway toward system suspend, but in practice has proved problematic to thepoint that it came up at both the 2015 and2016 Kernel Summits (as well as on themailing lists over the years); the intent is to tryto remove the kthread freezer entirely. To that end, Luis Rodriguez led adiscussion in the filesystem track of the 2018 Linux Storage, Filesystem,and Memory-Management Summit on the problems and possible solutions.

Dave Hansen did much of the work to get kernel page-table isolation(PTI) into the kernel in response to the Meltdown CPU vulnerability. In thememory-management track of the2018 Linux Storage, Filesystem, and Memory-Management Summit, he ran adiscussion on how PTI came about, what the costs are, and what can be doneto minimize its performance impact.

Ever since kernel page-table isolation(PTI) was introduced as a mitigation for the Meltdown CPU vulnerability, users have worried about how it affects theperformance of their systems. Most of that concern has been directedtoward its impact on computing performance, but I/O performance alsomatters. At the 2018 Linux Storage, Filesystem, and Memory-ManagementSummit, Ming Lei presented some preliminary work he has done to try toquantify how severely PTI affects block I/O operations.

Stable kernels 4.16.4, 4.14.36, 4.9.96, 4.4.129, and 3.18.106 have been released. All of themcontain important fixes and users should update.

Security updates have been issued by Arch Linux (roundcubemail, xfig, and zsh), Debian (linux-tools), Fedora (java-1.8.0-openjdk and mingw-libid3tag), Gentoo (chromium), openSUSE (hdf5, ocaml, PackageKit, phpMyAdmin, salt, and virtualbox), Oracle (patch), Red Hat (java-1.6.0-sun, java-1.7.0-oracle, java-1.8.0-oracle, patch, and python-paramiko), Scientific Linux (patch), SUSE (kernel and PackageKit), and Ubuntu (linux, linux-aws, linux-kvm, linux-raspi2, linux-snapdragon, linux, linux-raspi2, linux-azure, linux-euclid, linux-hwe, linux-gcp, linux-oem, linux-lts-xenial, linux-aws, and mysql-5.5, mysql-5.7).

It is a good thing that strong coffee was served at the 2018 Linux Storage,Filesystem, and Memory-Management Summit; full awareness was required fromthe first session, in which Josef Bacik discussed some issues that havearisen in the interaction between filesystems and the memory-managementsubsystem. Filesystems cache a lot of data from files, but also a lot ofmetadata about those files. It turns out, though, that management of thecached metadata does not work as well as one might like.

At the 2018 Legal andLicensing Workshop (LLW), which is a yearly gathering of lawyers and technical folks organized by the Free Software FoundationEurope (FSFE), attendees got more details on a recent hearing in a German GPLenforcement case. Marcus von Welser is a lawyer who represented thedefendant, Geniatech, in a case that was brought by PatrickMcHardy. In the presentation, von Welser was joined by Armijn Hemel, who helped Geniatech in its compliance efforts. The hearingwas of interest for a number of reasons, not least because McHardywithdrew his request for an injunction once it became clear that the judgewas leaning infavor of the defendants—effectively stopping this case dead in its tracks.

Daniel Vetter looks atsome kernel-development statistics, with a focus on patches written bythe maintainers who commit them. "Naively extrapolating the relative trend predicts that around the year 2025 large numbers of kernel maintainers will do nothing else than be the bottleneck, preventing everyone else from getting their work merged and not contributing anything of their own. The kernel community imploding under its own bureaucratic weight being the likely outcome of that.This is a huge contrast to the 'everything is getting better, bigger, andthe kernel community is very healthy' fanfare touted at keynotes and theyearly kernel report. In my opinion, the kernel community is very much notlooking like it is coping with its growth well and an overall healthycommunity."

Each kernel development cycle includes a vast number of changes that arenot intended to change visible behavior and which, as a result, gounnoticed by most users and developers. One such change in 4.17 is arewiring of how system-call implementations are invoked within the kernel.The change is interesting, though, and provides an opportunity to look atthe macro magic that handles system-call definitions.

Security updates have been issued by Debian (gunicorn, libreoffice, libsdl2-image, ruby1.8, and ruby1.9.1), Fedora (java-1.8.0-openjdk, jgraphx, memcached, nghttp2, perl, perl-Module-CoreList, and roundcubemail), Gentoo (clamav, librelp, mbedtls, quagga, tenshi, and unadf), Mageia (freeplane, libcdio, libtiff, thunderbird, and zsh), openSUSE (cfitsio, chromium, mbedtls, and nextcloud), and Red Hat (chromium-browser, kernel, and rh-perl524-perl).

The 4.17-rc2 kernel prepatch is out."We've still got some known fallout from the merge window, but itshouldn't affect most normal configurations, so go out and test."

The first article in this series describedthe interface to the "rhashtable"resizable hash-table abstraction in Linux 4.15. While a knowledge ofthe interface can result in successful use of rhashtables, it oftenhelps to understand what is going on "under the hood", particularly whenthose details leak out through the interface, as is occasionally thecase with rhashtable. The centerpiece for understanding theimplementation is knowing exactly how the table is resized. So thisfollow-on article will explain that operation; it will also present theconfiguration parameters that were skimmed over last time and discusshow they affect the implementation.

Version 4.0 of the FFmpegmultimedia toolkit is out. There is a long list of new filters, formats,and more; see the announcement for details.

The furor over the Meltdown and Spectre vulnerabilities has calmed a bit —for now, at least — but that does not mean that developers have stoppedworrying about them. Spectre variant 1 (the bounds-check bypassvulnerability) has been of particular concern because, while the kernel isthought to contain numerous vulnerable spots, nobody really knows how tofind them all. As a result, the defenses that have been developed forvariant 1 have only been deployed in a few places. Recently, though,Dan Carpenter has enhanced the smatch tool to enable it to find possiblyvulnerable code in the kernel.

Greg Kroah-Hartman has released stable kernel 4.9.95 with important fixes throughout thetree. Users should update.

Security updates have been issued by Debian (libreoffice and mysql-5.5), Fedora (corosync), Oracle (java-1.8.0-openjdk), Red Hat (java-1.8.0-openjdk), Scientific Linux (java-1.8.0-openjdk), and Ubuntu (openssl).

In the performance-conscious world of high-speed networking, anything thatcan be done to avoid copying packet data is welcome. The MSG_ZEROCOPY feature added in 4.14enables zero-copy transmission of data, but does not address the receiveside of the equation. It now appears that the 4.18 kernel will include a zero-copy receive mechanism by Eric Dumazetto close that gap, at least for some relatively specialized applications.

Stable kernels 4.16.3, 4.15.18, and 4.14.35 have been released. This is the last4.15.y kernel and users should move to 4.16.y.

Security updates have been issued by Debian (opencv and wireshark), Fedora (corosync and pcs), Oracle (firefox, kernel, libvncserver, and libvorbis), Slackware (gd), SUSE (kernel), and Ubuntu (apache2).

The LWN.net Weekly Edition for April 19, 2018 is available.

It is normally the grumpy editor's job to lookat accounting software; he does so with an eye toward getting the business off of theproprietary QuickBooks application and moving to something free. It may bethat Beancount deserves a look ofthat nature before too long but, in the meantime, a slightly less grumpyeditor has been messing with this text-based accounting tool for a varietyof much smaller projects. It is an interesting system, with a lot ofcapabilities, but its reliance on hand-rolling for various piecesmay scare some folks off.

The release of pip 10.0 has been announced. Some highlights of thisrelease include the removal of Python 2.6 support, limited PEP 518 support (withmore to come), a new "pip config" command, and other improvements.

The new PyPI has been launched. Browsertraffic and API calls (including "pip install") have been redirected fromthe old pypi.python.org to the new site. The old PyPI will shut down onApril 30. LWN covered the new PyPI last week.

Developers of database management systems are, by necessity, concernedabout getting data safely to persistent storage. So when the PostgreSQLcommunity found out that the way the kernel handles I/O errors could resultin data being lost without any errors being reported to user space, a fairamount of unhappiness resulted. The problem, which is exacerbated by theway PostgreSQL performs buffered I/O, turns out not to be unique to Linux,and will not be easy to solve even there.

Security updates have been issued by Debian (freeplane and jruby), Fedora (kernel and python-bleach), Gentoo (evince, gdk-pixbuf, and ncurses), openSUSE (kernel), Oracle (gcc, glibc, kernel, krb5, ntp, openssh, openssl, policycoreutils, qemu-kvm, and xdg-user-dirs), Red Hat (corosync, glusterfs, kernel, and kernel-rt), SUSE (openssl), and Ubuntu (openssl and perl).

Security updates have been issued by Debian (corosync, linux-tools, qemu, qemu-kvm, and r-cran-readxl), openSUSE (evince, memcached, nodejs4, ntp, pdns-recursor, python-gunicorn, python3-gunicorn, and python3), and Ubuntu (ruby1.9.1, ruby2.0, ruby2.3).

Microsoft has issued apress release describing the security dangers involved with theInternet of things ("a weaponized stove, baby monitors that spy, thecontents of your refrigerator being held for ransom") and introducing"Microsoft Azure Sphere" as a combination of hardware and software toaddress the problem. "Unlike the RTOSes common to MCUs today, ourdefense-in-depth IoT OS offers multiple layers of security. It combinessecurity innovations pioneered in Windows, a security monitor, and a customLinux kernel to create a highly-secured software environment and atrustworthy platform for new IoT experiences."

Alpine Linux-based postmarketOS is touch-optimized and pre-configured forinstallation on smartphones and other mobile devices. The postmarketOSblog introducespostmarketOS-lowlevel which is a community project aimed at creatingfree bootloaders and cellular modem firmware, currently focused on MediaTekphones. "But before we get started, please keep in mind that theseare moon shots. So while there is some little progress, it's mostly aboutletting fellow hackers know what we've tried and what we're up to, in thehopes of attracting more interested talent to our cause. After all, ourphilosophy is to keep the community informed and engaged during thedevelopment phase!"