LWN.net

The 4.17-rc2 kernel prepatch is out."We've still got some known fallout from the merge window, but itshouldn't affect most normal configurations, so go out and test."

The first article in this series describedthe interface to the "rhashtable"resizable hash-table abstraction in Linux 4.15. While a knowledge ofthe interface can result in successful use of rhashtables, it oftenhelps to understand what is going on "under the hood", particularly whenthose details leak out through the interface, as is occasionally thecase with rhashtable. The centerpiece for understanding theimplementation is knowing exactly how the table is resized. So thisfollow-on article will explain that operation; it will also present theconfiguration parameters that were skimmed over last time and discusshow they affect the implementation.

Version 4.0 of the FFmpegmultimedia toolkit is out. There is a long list of new filters, formats,and more; see the announcement for details.

The furor over the Meltdown and Spectre vulnerabilities has calmed a bit —for now, at least — but that does not mean that developers have stoppedworrying about them. Spectre variant 1 (the bounds-check bypassvulnerability) has been of particular concern because, while the kernel isthought to contain numerous vulnerable spots, nobody really knows how tofind them all. As a result, the defenses that have been developed forvariant 1 have only been deployed in a few places. Recently, though,Dan Carpenter has enhanced the smatch tool to enable it to find possiblyvulnerable code in the kernel.

Greg Kroah-Hartman has released stable kernel 4.9.95 with important fixes throughout thetree. Users should update.

Security updates have been issued by Debian (libreoffice and mysql-5.5), Fedora (corosync), Oracle (java-1.8.0-openjdk), Red Hat (java-1.8.0-openjdk), Scientific Linux (java-1.8.0-openjdk), and Ubuntu (openssl).

In the performance-conscious world of high-speed networking, anything thatcan be done to avoid copying packet data is welcome. The MSG_ZEROCOPY feature added in 4.14enables zero-copy transmission of data, but does not address the receiveside of the equation. It now appears that the 4.18 kernel will include a zero-copy receive mechanism by Eric Dumazetto close that gap, at least for some relatively specialized applications.

Stable kernels 4.16.3, 4.15.18, and 4.14.35 have been released. This is the last4.15.y kernel and users should move to 4.16.y.

Security updates have been issued by Debian (opencv and wireshark), Fedora (corosync and pcs), Oracle (firefox, kernel, libvncserver, and libvorbis), Slackware (gd), SUSE (kernel), and Ubuntu (apache2).

The LWN.net Weekly Edition for April 19, 2018 is available.

It is normally the grumpy editor's job to lookat accounting software; he does so with an eye toward getting the business off of theproprietary QuickBooks application and moving to something free. It may bethat Beancount deserves a look ofthat nature before too long but, in the meantime, a slightly less grumpyeditor has been messing with this text-based accounting tool for a varietyof much smaller projects. It is an interesting system, with a lot ofcapabilities, but its reliance on hand-rolling for various piecesmay scare some folks off.

The release of pip 10.0 has been announced. Some highlights of thisrelease include the removal of Python 2.6 support, limited PEP 518 support (withmore to come), a new "pip config" command, and other improvements.

The new PyPI has been launched. Browsertraffic and API calls (including "pip install") have been redirected fromthe old pypi.python.org to the new site. The old PyPI will shut down onApril 30. LWN covered the new PyPI last week.

Developers of database management systems are, by necessity, concernedabout getting data safely to persistent storage. So when the PostgreSQLcommunity found out that the way the kernel handles I/O errors could resultin data being lost without any errors being reported to user space, a fairamount of unhappiness resulted. The problem, which is exacerbated by theway PostgreSQL performs buffered I/O, turns out not to be unique to Linux,and will not be easy to solve even there.

Security updates have been issued by Debian (freeplane and jruby), Fedora (kernel and python-bleach), Gentoo (evince, gdk-pixbuf, and ncurses), openSUSE (kernel), Oracle (gcc, glibc, kernel, krb5, ntp, openssh, openssl, policycoreutils, qemu-kvm, and xdg-user-dirs), Red Hat (corosync, glusterfs, kernel, and kernel-rt), SUSE (openssl), and Ubuntu (openssl and perl).

Security updates have been issued by Debian (corosync, linux-tools, qemu, qemu-kvm, and r-cran-readxl), openSUSE (evince, memcached, nodejs4, ntp, pdns-recursor, python-gunicorn, python3-gunicorn, and python3), and Ubuntu (ruby1.9.1, ruby2.0, ruby2.3).

Microsoft has issued apress release describing the security dangers involved with theInternet of things ("a weaponized stove, baby monitors that spy, thecontents of your refrigerator being held for ransom") and introducing"Microsoft Azure Sphere" as a combination of hardware and software toaddress the problem. "Unlike the RTOSes common to MCUs today, ourdefense-in-depth IoT OS offers multiple layers of security. It combinessecurity innovations pioneered in Windows, a security monitor, and a customLinux kernel to create a highly-secured software environment and atrustworthy platform for new IoT experiences."

Alpine Linux-based postmarketOS is touch-optimized and pre-configured forinstallation on smartphones and other mobile devices. The postmarketOSblog introducespostmarketOS-lowlevel which is a community project aimed at creatingfree bootloaders and cellular modem firmware, currently focused on MediaTekphones. "But before we get started, please keep in mind that theseare moon shots. So while there is some little progress, it's mostly aboutletting fellow hackers know what we've tried and what we're up to, in thehopes of attracting more interested talent to our cause. After all, ourphilosophy is to keep the community informed and engaged during thedevelopment phase!"

Security updates have been issued by Arch Linux (lib32-openssl and zsh), Debian (patch, perl, ruby-loofah, squirrelmail, tiff, and tiff3), Fedora (gnupg2), Gentoo (go), Mageia (firefox, flash-player-plugin, nxagent, puppet, python-paramiko, samba, and thunderbird), Red Hat (flash-plugin), Scientific Linux (python-paramiko), and Ubuntu (patch, perl, and ruby).

Version 1.10 of the Subversion version-control system is out.Improvements include a new interactive resolver for merge conflicts, betterpath-based authorization, LZ4 compression, and more; see therelease notes for details.

By the time the 4.17 merge window was closed and 4.17-rc1 was released, 11,769 non-merge changesets had been pulled into themainline repository. 4.17 thus looks to be a typically busy developmentcycle, with a merge window only slightly more busy than 4.16 had.Some 6,000 of those changes were pulled after last week's summary was written. There was alot of the usual maintenance work in those patches (over 10% of thosechanges were to device-tree files, for example), but also some moresignificant changes.

Linus has released 4.17-rc1 and closed themerge window for this release. "This does not seem to be shaping upto be a particularly big release, and there seems to be nothingparticularly special about it. The most special thing that happened ispurely numerology: we've passed the six million git objects mark, and thatis reason enough to call the next kernel 5.0. Except I probably won't,because I don't want to be too predictable."

A comparison of the feature sets for a handful of terminal emulators wasthe subject of a recent article; here I follow that up byexamining the performance of those terminals. This might seem like alesser concern, but as it turns out, terminals exhibit surprisinglyhigh latency for such fundamental programs. I also examine what istraditionally considered "speed" (but is really scroll bandwidth) andmemory usage, with the understanding that the impact of memory useis less than it was when I looked at this a decade ago (inFrench).Subscribers can read on for part 2 from guest author Antoine Beaupré.

The stable kernel train just keeps on rolling; Greg Kroah-Hartman has announcedthe release of the 4.9.94, 4.4.128, and 3.18.105 stable kernels. All contain a largenumber of fixes throughout the tree and users should upgrade.

The rhashtable data structure is a generic resizable hash-tableimplementation in the Linux kernel, which LWN first introduced as "relativistichash tables" back in 2014. I thought at the time that it might be fun to makeuse of rhashtables, but didn't, until an opportunity arose through my work onthe Lustre filesystem. Lustre is a cluster filesystem that is currently indrivers/staging while the code is revised to meet upstreamrequirements. One of those requirements is to avoid duplicatingsimilar functionality where possible. As Lustre contains a resizablehash table, it really needs to be converted to use rhashtables instead — atlast I have my opportunity.Subscribers can read on for a look at the rhashtable API by guest authorNeil Brown.

Security updates have been issued by Arch Linux (apache), openSUSE (libvirt, openssl, policycoreutils, and zziplib), Oracle (firefox and python-paramiko), and Red Hat (python-paramiko).

Greg Kroah-Hartman has released three new stable kernels: 4.16.2, 4.15.17, and 4.14.34. Users of those kernel series shouldupgrade.

Security updates have been issued by Debian (poppler), Fedora (koji and libofx), Gentoo (adobe-flash), Oracle (kernel), Red Hat (qemu-kvm-rhev and sensu), and Scientific Linux (firefox).

The LWN.net Weekly Edition for April 12, 2018 is available.

A "simple" utility to make a system beep is hardly the first place one wouldcheck for security flaws, but the strange case of the "Holey Beep"should perhaps lead to some rethinking. A Debian advisory for the beep utility, which was followedby another for Debian LTS, led to aseemingly satirical site publicizingthe bug (and giving it the "Holey Beep" name). But that site also exploitsa new flaw in the GNUpatch program—and the increased scrutiny on beep hasled to more problems being found.

The Python Package Index (PyPI) isthe principal repository of libraries for the Python programming language,serving more than 170 million downloads each week. Fifteen years after PyPIlaunched, a new edition is in beta at pypi.org, with features like bettersearch, a refreshed layout, and Markdown README files(and with some old features removed, like viewing GPG package signatures). StartingApril 16, users visiting the site or running pip install willbe seamlessly redirected to the new site. Two weeks after that, the legacy site isexpected to be shut down and the team will turn toward newfeatures; in the meantime, it is worth a look at what the new PyPI bringsto the table.

Security updates have been issued by Debian (pcs), Fedora (drupal7), openSUSE (git and mercurial), Red Hat (firefox and qemu-kvm-rhev), SUSE (libvirt and xen), and Ubuntu (patch).

Car manufacturers, like most companies, navigate a narrow lane between thebenefits of using free and open-source software and the perceived or realimportance of hiding their trade secrets. Many are usingfree software in some of the myriad software components that make up amodern car, and even work in consortia to develop free software. At therecent LibrePlanetconference, free-software advocate Jeremiah Foster covered progress in theautomotive sector and made an impassioned case for more free software in theirembedded systems.Subscribers can read on for a report on the talk by guest author Andy Oram.

Red Hat has announcedthe general availability of Red Hat Enterprise Linux 7.5. This versionfeatures enhanced hybrid cloud security and compliance, improved storageperformance and efficiency, simplified management, and production-readyLinux containers. RHEL 7.5 is available for x86, IBM Power, IBM z Systems, and 64-bit Arm. This release also brings support for single-host KVM virtualization and Open Container Initiative (OCI)-formatted runtime environment and base image to IBM z Systems.

The 3.18.104 kernel has been released witha single bugfix. If you had build errors in 3.18.103 then this update isfor you, otherwise there is no need to upgrade.

Security updates have been issued by CentOS (libvorbis and thunderbird), Debian (pjproject), Fedora (compat-openssl10, java-1.8.0-openjdk-aarch32, libid3tag, python-pip, python3, and python3-docs), Gentoo (ZendFramework), Oracle (thunderbird), Red Hat (ansible, gcc, glibc, golang, kernel, kernel-alt, kernel-rt, krb5, kubernetes, libvncserver, libvorbis, ntp, openssh, openssl, pcs, policycoreutils, qemu-kvm, and xdg-user-dirs), SUSE (openssl and openssl1), and Ubuntu (python-crypto, ubuntu-release-upgrader, and wayland).

Jim Gettys refutesthe claim that the early designers of Internet software were notconcerned about security. "Government export controls crippledInternet security and the design of Internet protocols from the verybeginning: we continue to pay the price to this day".

Several security vulnerabilities were found in Etherpad and version1.6.4 has been released with fixes. The vulnerabilities includearbitrary code execution and information disclosure. Site admins are urgedto update Etherpad to 1.6.4 as soon as possible.

Security updates have been issued by Arch Linux (openssl and zziplib), Debian (ldap-account-manager, ming, python-crypto, sam2p, sdl-image1.2, and squirrelmail), Fedora (bchunk, koji, libidn, librelp, nodejs, and php), Gentoo (curl, dhcp, libvirt, mailx, poppler, qemu, and spice-vdagent), Mageia (389-ds-base, aubio, cfitsio, libvncserver, nmap, and ntp), openSUSE (GraphicsMagick, ImageMagick, spice-gtk, and wireshark), Oracle (kubernetes), Slackware (patch), and SUSE (apache2 and openssl).

The Linux network stack does not lack for features; it also performs wellenough for most uses. At the highest network speeds, though, any overheadat all is too much; that has driven the most demanding users towardspecialized, user-space networking implementations that can outperform thekernel for highly constrained tasks. The express data path (XDP)development effort is an attempt to win those users back, with some apparentsuccess so far. With the posting of the AF_XDP patch set by Björn Töpel,another piece of the XDP puzzle is coming into focus.

The4.16.1,4.15.16,4.14.33,4.9.93,4.4.127, and3.18.103stable kernels have all been released; each contains a fairly long list ofimportant fixes.

As the 4.17 merge window opened, it seemedpossible that the kernel lockdown patch set could be merged at last.That was before the linux-kernel mailing list got its hands on the issue.What resulted was not one of the kernel community's finest moments. But itdid result in a couple of evident conclusions: kernel lockdown will almostcertainly not bemerged for 4.17, but something that looks very much like it is highlylikely to be accepted in a subsequent merge window.

Security updates have been issued by Debian (sharutils), Fedora (firefox, httpd, and mod_http2), openSUSE (docker-distribution, graphite2, libidn, and postgresql94), Oracle (libvorbis and thunderbird), Red Hat (libvorbis, python-paramiko, and thunderbird), Scientific Linux (libvorbis and thunderbird), SUSE (apache2), and Ubuntu (firefox, linux-lts-xenial, linux-aws, and ruby1.9.1, ruby2.0, ruby2.3).

As of this writing, 5,392 non-merge changesets have been pulled into themainline repository for the 4.17 release. The 4.17 merge window is thusoff to a good start, but it is far from complete. The changes pulled thusfar cover a wide part of the core kernel as well as the networking, driver,and filesystem subsystems.

Security updates have been issued by Arch Linux (drupal), Debian (openjdk-7), Fedora (exempi, gd, and tomcat), SUSE (python-paramiko), and Ubuntu (kernel, libvncserver, linux, linux-aws, linux-kvm, linux-raspi2, linux-snapdragon, linux-lts-trusty, and linux-raspi2).

The LWN.net Weekly Edition for April 5, 2018 is available.

It has been known for quite some time that Python 2 will reach its endof life in 2020—after being extended by five years from its original 2015expiry. After that, there will be no support, bug fixes, or security patches forPython 2, at least from the Python Software Foundation and the coredevelopers. Some distributions will need to continue to support the finalPython 2 release, however, since their support windows extend pastthat date; the enterprise and long-term support distributions willlikely be supporting it well into the 2020s and possibly beyond. But evenshorter-support-cycle distributions need to consider their plan for asweeping change of this sort—in less than two years.

Security updates have been issued by Debian (apache2, ldap-account-manager, and openjdk-7), Fedora (libuv and nodejs), Gentoo (glibc and libxslt), Mageia (acpica-tools, openssl, and php), SUSE (clamav, coreutils, and libvirt), and Ubuntu (kernel, libraw, linux-hwe, linux-gcp, linux-oem, and python-crypto).

The Linux Foundation and Nitrokey have announceda program whereby anybody who appears in the kernel's MAINTAINERS file orwho has a kernel.org email address can obtain a free Nitrokey Start crypto card. Theintent, of course, is that kernel developers will use these devices tosafeguard their GnuPG keys and, as a result, improve the security of thekernel development process as a whole. "A digital smartcard tokenlike Nitrokey Start contains a cryptographic chip that is capable ofstoring private keys and performing crypto operations directly on the tokenitself. Because the key contents never leave the device, the operatingsystem of the computer into which the token is plugged in is not able toretrieve the private keys themselves, therefore significantly limiting theways in which the keys can be leaked or stolen."See this LWN article for a look at crypto cards.

One of the trickiest aspects to concurrency in the kernel is waiting for aspecific event to take place. There is a wide variety of possible events,including a process exiting, the last reference to a data structure goingaway, a device completing an operation, or a timeout occurring.Waiting is surprisingly hard to get right — race conditions abound to trapthe unwary — so the kernel hasaccumulated a large set of wait_event_*() macros to make the task easier. Anattempt to add a new one, though, has led to the generalization of specifictypes of waits for 4.17.