LWN.net

Shawn Pearce, a longtime contributor to the Git community (and beyond), haspassed away. The thread on the Git mailing list makes it clear that hewill be missed by many people.

PostgreSQL developer Robert Haas describesa new storage module that is under development. "We are workingto build a new table storage format for PostgreSQL, which we’re callingzheap. In a zheap, whenever possible, we handle an UPDATE by moving the oldrow version to an undo log, and putting the new row version in the placepreviously occupied by the old one. If the transaction aborts, we retrievethe old row version from undo and put it back in the original location; ifa concurrent transaction needs to see the old row version, it can find itin undo. [...] This means that there is no need for VACUUM, or any similarprocess, to scan the table looking for dead rows."

Worth a read: this blogposting from Leonardo Chiariglione, the founder and chair of MPEG, onhow (in his view) the group is being destroyed by free codecs and patent trolls."Good stories have an end, so the MPEG business model could not lastforever. Over the years proprietary and 'royalty free' products haveemerged but have not been able to dent the success of MPEG standards. Moreimportantly IP holders – often companies not interested in exploiting MPEGstandards, so called Non Practicing Entities (NPE) – have become more andmore aggressive in extracting value from their IP." (Thanks to Paul Wise).

Security updates have been issued by Arch Linux (curl, lib32-curl, lib32-libcurl-compat, lib32-libcurl-gnutls, libcurl-compat, libcurl-gnutls, and rsync), Debian (curl), Fedora (clamav and java-1.8.0-openjdk), openSUSE (apache2), Oracle (kernel), and Ubuntu (linux-kvm and thunderbird).

On his blog, embedded developer Karim Yaghmour has written about his ten-day trip to Shenzen, China, which is known as the "Silicon Valley of hardware". His lengthy trip report covers much that would be of use to others who are thinking of making the trip, but also serves as an interesting travelogue even for those who are likely to never go. "The map didn't disappoint and I was able to find a large number of kiosks selling some of the items I was interested in. Obviously many kiosks also had items that I had seen on Amazon or elsewhere as well. I was mostly focusing on things I hadn't seen before. After a few hours of walking floors upon floors of shops, I was ready to start focusing on other aspects of my research: hard to source and/or evaluate components, tools and expanding my knowledge of what was available in the hardware space. Hint: TEGES' [The Essential Guide to Electronics in Shenzhen] advice about having comfortable shoes and comfortable clothing is completely warranted.Finding tools was relatively easy. TEGES indicates the building and floor to go to, and you'll find most anything you can think of from rework stations, to pick-and-place machines, and including things like oscilloscopes, stereo microscopes, multimeters, screwdrivers, etc. In the process I saw some tools which I couldn't immediately figure out the purpose for, but later found out their uses on some other visits. Satisfied with a first glance at the tools, I set out to look for one specific component I was having a hard time with. That proved a lot more difficult than anticipated. Actually I should qualify that. It was trivial to find tons of it, just not something that matched exactly what I needed. I used TEGES to identify one part of the market that seemed most likely to have what I was looking for, but again, I could find lots of it, just not what I needed."

The TCP protocol has become so ubiquitous that, to many people, the terms"TCP/IP" and "networking" are nearly synonymous. The fact that introducingnew protocols (or even modifying existing protocols) has become nearly impossible tends toreinforce that situation. That is not stopping people from trying, though.At linux.conf.au 2018, Jana Iyengar, a developer at Google, discussed thecurrent state of the QUIC protocol which, he said, is now used for about 7% of the traffic on theInternet as a whole.

Security updates have been issued by Arch Linux (glibc, lib32-glibc, and zziplib), Debian (clamav, ffmpeg, thunderbird, tiff, tiff3, and wireshark), Fedora (firefox, mingw-libtasn1, and webkitgtk4), Gentoo (fossil), Mageia (webkit2), openSUSE (chromium, clamav, and thunderbird), and SUSE (clamav and kernel).

Here's anopensource.com article on the virtues of CopperheadOS."Unlike other custom ROMs that strive to add lots of newfunctionality, Copperhead runs a pretty vanilla version of AOSP. Also,while the first thing you usually do when playing with a custom ROM is toadd root access to the device, not only does Copperhead prevent that, italso requires that you have a device that has verified boot, so there's nounlocking the bootloader. This is to prevent malicious code from gettingaccess to the handset."

Linus has released the 4.15 kernel."After a release cycle that was unusual in so many (bad) ways, thislast week was really pleasant. Quiet and small, and no last-minutepanics, just small fixes for various issues. I never got a feelingthat I'd need to extend things by yet another week, and 4.15 looksfine to me."Some of the more significant features in this release include:the long-awaited CPU controller for theversion-2 control-group interface,significant live-patching improvements,initial support for the RISC-V architecture,support for AMD's secure encrypted virtualization feature, andthe MAP_SYNC mechanism for workingwith nonvolatile memory.This release also, of course, includes mitigations for the Meltdown and Spectre variant-2vulnerabilities though, as Linus points out in the announcement, thework of dealing with these issues is not yet done.

The Linux Foundation has announced a new project, called LinuxBoot, that is working on replacements for much of the firmware used to boot our systems. The project is based on work by Google and others to use Linux (and Go programs) to replace most of the UEFI boot firmware. "Firmware has always had a simple purpose: to boot the OS. Achieving that has become much more difficult due to increasing complexity of both hardware and deployment. Firmware often must set up many components in the system, interface with more varieties of boot media, including high-speed storage and networking interfaces, and support advanced protocols and security features.LinuxBoot addresses the often slow, often error-prone, obscured code that executes these steps with a Linux kernel. The result is a system that boots in a fraction of the time of a typical system, and with greater reliability."

Security updates have been issued by CentOS (389-ds-base, dhcp, kernel, and nautilus), Debian (curl, openssh, and wireshark), Fedora (clamav, firefox, java-9-openjdk, and poco), Gentoo (clamav), openSUSE (curl, libevent, mupdf, mysql-community-server, newsbeuter, php5, redis, and tre), Oracle (389-ds-base, dhcp, kernel, and nautilus), Slackware (mozilla), and Ubuntu (kernel and linux-hwe, linux-azure, linux-gcp, linux-oem).

GCC 7.3 is out. This is mainly a bug-fix release, but it does also containthe "retpoline" support needed to build the kernel (and perhaps other code)with resistance to the Spectre variant-2 vulnerability.

Here's a34c3 conference report in CSO suggesting that the BSDs are losingdevelopers. "von Sprundel says he easily found around 115 kernel bugs across thethree BSDs, including 30 for FreeBSD, 25 for OpenBSD, and 60 forNetBSD. Many of these bugs he called 'low-hanging fruit.' He promptlyreported all the bugs, but six months later, at the time of his talk, manyremained unpatched.'By and large, most security flaws in the Linux kernel don't have a longlifetime. They get found pretty fast,' von Sprundel says. 'On the BSD side,that isn't always true. I found a bunch of bugs that have been around avery long time.' Many of them have been present in code for a decade ormore."

Security updates have been issued by CentOS (firefox), Debian (firefox-esr, gcab, and poppler), Fedora (clamav and firefox), Mageia (bind, firefox, glibc, graphicsmagick, squid, systemd, and virtualbox), openSUSE (firefox, GraphicsMagick, libexif, and libvpx), Red Hat (389-ds-base, dhcp, kernel, kernel-alt, kernel-rt, and nautilus), Scientific Linux (389-ds-base, dhcp, kernel, and nautilus), Slackware (curl), SUSE (kernel and webkit2gtk3), and Ubuntu (firefox, libtasn1-6, and mysql-5.5).

The LWN.net Weekly Edition for January 25, 2018 is available.

2017 was a big year for the Prometheus project, as it publishedits 2.0 release in November. The new release ships numerousbug fixes, new features, and, notably, a new storage engine that brings majorperformance improvements. This comes at the cost of incompatible changes tothe storage and configuration-file formats. An overview ofPrometheus and its new release was presented to the Kubernetes community in a talkheld during KubeCon+ CloudNativeCon. This article covers what changed in this new releaseand what is brewing next in the Prometheus community; it is a companion tothis article, which provided a generalintroduction to monitoring with Prometheus.

Richard Fontana exploresthe intersection of containers and copyleft licensing onopensource.com."One imperfect way of framing the question is whether GPL-licensedcode, when combined in some sense with proprietary code, forms a singlemodified work such that the proprietary code could be interpreted as beingsubject to the terms of the GPL. While we haven’t yet seen much of thatconcern directed to Linux containers, we expect more questions to be raisedas adoption of containers continues to grow. But it’s fairlystraightforward to show that containers do not raise new or concerning GPLscope issues."

Security updates have been issued by openSUSE (clamav-database and virtualbox), Oracle (firefox and kernel), Red Hat (firefox), Scientific Linux (firefox), and Ubuntu (gcab).

Firefox 58 has been released. "With this release, we’re building on the great foundation provided by our all-new Firefox Quantum browser. We're optimizing the performance gains we released in 57 by improving the way we render graphics and cache JavaScript. We also made functional and privacy improvements to Firefox Screenshots. On Firefox for Android, we’ve added support for Progressive Web Apps (PWAs) so you can add websites to your home screen and use them like native apps."

Sometimes, a data structure proves to be inadequate for its intended task.Other times, though, the problem may be somewhere else — in the API used toaccess it, for example. Matthew Wilcox's presentation during the 2018linux.conf.au Kernel miniconf made the case that, for the kernel'svenerable radix tree data structure, thelatter situation holds. His response is a new approach to an old datastructure that he is calling the "XArray".

The Free Software Foundation blog has a guestpost from GNU MediaGoblin founder Christopher Lemmer Webber announcing that ActivityPub has been made anofficial W3C recommended standard. "ActivityPub is a protocol for building decentralized social networking applications. It provides both a server-to-server protocol (i.e. federation) and a client-to-server protocol (for desktop and mobile applications to connect to your server). You can use the server-to-server protocol or the client-to-server protocol on their own, but one nice feature is that the designs for both are very similar. Chances are, if you've implemented support for one, you can get support for the other with very little extra effort! We've worked hard to make ActivityPub easy to understand."

Stable kernels 4.14.15, 4.9.78, and 4.4.113 have been released. They all containimportant fixes and users should upgrade.

Security updates have been issued by Debian (smarty3), Fedora (bind, bind-dyndb-ldap, dnsperf, glibc, kernel, libtasn1, libvpx, mariadb, python-bottle, ruby, and sox), Red Hat (rh-eclipse46-jackson-databind), SUSE (kernel), and Ubuntu (kernel, linux, linux-aws, linux-euclid, linux-hwe, linux-azure, linux-gcp, linux-oem, linux-lts-trusty, linux-lts-xenial, linux-aws, and rsync).

The Qubes project has described anew, not-yet-implemented design intended to address a number of problemsthat this high-security distribution project has encountered."One possible solution to these problems is actually to 'move Qubesto the cloud.' Readers who are allergic to the notion of having theirprivate computations running in the (untrusted) cloud should not give upreading just yet. Rest assured that we will also discuss other solutionsnot involving the cloud. The beauty of Qubes Air, we believe, lies in thefact that all these solutions are largely isomorphic, from both anarchitecture and code point of view."

By now, almost everybody has probably seen the press coverage of Linus Torvalds's remarks about one of thepatches addressing Spectre variant 2. Less noted, but much moreinformative, is David Woodhouse's responseon why those patches are the way they are. "That's why my initialidea, as implemented in this RFC patchset, was to stick with IBRS onSkylake, and use retpoline everywhere else. I'll give you 'garbagepatches', but they weren't being 'just mindlessly sent around'. If we'regoing to drop IBRS support and accept the caveats, then let's do it as aconscious decision having seen what it would look like, not just drop itquietly because poor Davey is too scared that Linus might shout at himagain."

BPF is an increasingly capable tool for instrumenting and tracing theoperation of the kernel; it has enabled the creation of the growing set ofBCC tools. Unfortunately, BCC has no support for a cross-developmentworkflow where the development machine and the target machine running thedeveloped code are different. Cross-development is favored byembedded-systems kernel developers who tend to develop on an x86 host andthen flash and test their code on SoCs (System on Chips) based on the ARMarchitecture. In this article, I introduce BPFd, a project to enable crossdevelopment using BPF and BCC.

The minor release of openSUSE Leap 42.2 will reachits end-of-life (EOL) on January 26. "The major release of the Leap 42 series has so far provided a support life cycle of 27 months and is expected to last until early 2019; when openSUSE Leap 42.3 will reach its EOL. That gives the major version of Leap 42 more than 36 months of life-cycle support. However, the EOL for the Leap 42 series is dependent on the release of the next major version, which will be openSUSE Leap 15 and it’s expected to be released later this Spring."

The Document Liberation Project has announced five new or improvedlibraries to export EPUB3 and import AbiWord, MS Publisher, PageMaker andQuarkXPress files. "The libraries have been originallydeveloped for the LibreOffice 6.0 major release, but can be used by anyother software thanks to the OSI (Open Source Initiative) compliantlicense."

Security updates have been issued by CentOS (bind), Debian (openocd), Mageia (unbound), Oracle (bind and microcode_ctl), Red Hat (bind, java-1.6.0-sun, libvirt, and qemu-kvm), Scientific Linux (bind), SUSE (kernel and perl-XML-LibXML), and Ubuntu (gimp, intel-microcode, mysql-5.5, mysql-5.7, and openssh).

Back in mid-1997, your editor (Jonathan Corbet) and Liz Coolbaugh wereengaged in a long-running discussion on how to trade our nice, stable,reliably paying jobs for a life of uncertainty, poverty, andaround-the-clock work. Not that we thought of it in those terms,naturally. We eventually settled on joining Red Hat's nascent "supportpartner" program; while we were waiting for it to get started, we decidedto start a weekly newsletter as a side project — not big andprofessional like the real press — to establish ourselves in the community.Thus began an amazing journey that has just completed its 20th year.

As might have been expected from watching the commit stream, the 4.15kernel is not ready for release, so we'll get 4.15-rc9 instead.Linus said: "I really really wanted to just release 4.15 today, but things haven'tcalmed down enough for me to feel comfy about it, and Davem tells mehe still has some networking fixes pending. Laura Abbott found andfixed a very subtle boot bug introduced this development cycle onlyyesterday, and it just didn't feel right to say that we're done."

Security updates have been issued by Debian (bind9, couchdb, lucene-solr, mysql-5.5, openocd, and php5), Mageia (gdk-pixbuf2.0, golang, and mariadb), openSUSE (curl, gd, ImageMagick, lxterminal, ncurses, newsbeuter, perl-XML-LibXML, and xmltooling), Oracle (kernel), and SUSE (xmltooling).

Wired recommendsswitching to F-Droid for Android apps."A polluted ocean of apps is plaguing Android, an operating systembuilt upon Free and Open-Source Software (FOSS) but now barely resemblingthose venerable roots. Today, the average Android device is not onlysusceptible to malware and trackers, it’s also heavily locked down andloaded with proprietary components—characteristics that are hardly thecalling cards of the FOSS movement. Though Android bears the moniker of open-source, the chain of trust between developers, distributors, and end-users is broken."

The OpenSSL project has announceda number of changes to how the project is developed. These includeshutting down the openssl-dev mailing list in favor of discussing allpatches on GitHub and the addition of a new, read-only (for the world)openssl-project list. "We are changing our release schedule so thatunless there are extenuating circumstances, security releases will go outon a Tuesday, with the pre-notification being the previous Tuesday. Wedon’t see a need to have people ready to sacrifice their weekend every timea new CVE comes out."

Here's abrief update from Greg Kroah-Hartman on the kernel's handling of theMeltdown and Spectre vulnerabilities. "This shows that my kernel isproperly mitigating the Meltdown problem by implementing PTI (Page TableIsolation), and that my system is still vulnerable to the Spectre variant1, but is trying really hard to resolve the variant 2, but is not quitethere (because I did not build my kernel with a compiler to properlysupport the retpoline feature)."

Linux’s deadline scheduler is a global early deadline first scheduler forsporadic tasks with constrained deadlines. These terms were defined in the first part of this series. In thisinstallment, the details of the Linux deadline scheduler and how it can beused will be examined.

Security updates have been issued by Arch Linux (bind, irssi, nrpe, perl-xml-libxml, and transmission-cli), CentOS (java-1.8.0-openjdk), Debian (awstats, libgd2, mysql-5.5, rsync, smarty3, and transmission), Fedora (keycloak-httpd-client-install and rootsh), and Red Hat (java-1.7.0-oracle and java-1.8.0-oracle).

Git v2.16.0 is now available. "It is comprised of 509 non-mergecommits since v2.15.0, contributed by 91 people, 26 of which are newfaces." The release notes are included in the link below.

Version 3.0 of theWine Windows emulation layer has been released. "This releaserepresents a year of development effort and over 6,000 individualchanges." Most of the improvements seem to be around Direct3Dgraphics, but it also now possible to package up Wine as an Android app;see the release notes fordetails.

This is the second article of a series discussing various methods ofreducing the size of the Linux kernel to make it suitable for smallenvironments.The first articleprovided a short rationale for this topic, and covered the link-timegarbage collection, also called the ld --gc-sections method. We've seenthat, though it is pretty straightforward, link-time garbage collection hasissues of its own when applied to the kernel, making achieving optimalresults more difficult than it is worth. In this article we'll have a look at what thecompiler itself can do using link-time optimization.

Security updates have been issued by CentOS (linux-firmware and microcode_ctl), Fedora (icecat and transmission), Oracle (java-1.8.0-openjdk and microcode_ctl), Red Hat (java-1.8.0-openjdk), Scientific Linux (java-1.8.0-openjdk), Slackware (bind), SUSE (kernel), and Ubuntu (eglibc).

The LWN.net Weekly Edition for January 18, 2018 is available.

Prometheus is a monitoring toolbuilt from scratch by SoundCloud in 2012. It works by pulling metrics frommonitored services and storing them in a time series database (TSDB). Ithas a powerful query language to inspect that database, create alerts, andplot basic graphs. Those graphs can then be used to detect anomalies ortrends for (possibly automated) resource provisioning. Prometheus also hasextensive service discovery features and supports high availabilityconfigurations.That's what the brochure says, anyway; let's see how it works in the handsof an old grumpy system administrator. I'll be drawing comparisonswith Munin and Nagios frequently because those are the tools I haveused for over a decade in monitoring Unix clusters.

Greg Kroah-Hartman has released stable kernels 4.14.14, 4.9.77, 4.4.112, and 3.18.92. All of them contain important fixesand users should upgrade.

Security updates have been issued by Debian (bind9, wordpress, and xbmc), Fedora (awstats, docker, gifsicle, irssi, microcode_ctl, mupdf, nasm, osc, osc-source_validator, and php), Gentoo (newsbeuter, poppler, and rsync), Mageia (gifsicle), Red Hat (linux-firmware and microcode_ctl), Scientific Linux (linux-firmware and microcode_ctl), SUSE (kernel and openssl), and Ubuntu (bind9, eglibc, glibc, and transmission).

Many techniques in software security are complicated and require a deepunderstanding of the internal workings of the computer and the software undertest. Some techniques, though, are conceptually simple and do not rely onknowledge of the underlying software. Fuzzing is a useful example: running aprogram with a wide variety of junk input and seeing if it does anythingabnormal or interesting, like crashing. Though it might seem unsophisticated,fuzzing is extremely helpful in finding the parsing and input processingproblems that are often the beginning of a security vulnerability.

Alison Chaiken looksin detail at how the kernel boots on opensource.com."Besides starting buggy spyware, what function does early bootfirmware serve? The job of a bootloader is to make available to a newlypowered processor the resources it needs to run a general-purpose operatingsystem like Linux. At power-on, there not only is no virtual memory, but noDRAM until its controller is brought up."

The deadline scheduler enables the user to specify a realtime task'srequirements using well-defined realtime abstractions, allowing the system to makethe best scheduling decisions, guaranteeing the scheduling of realtimetasks even in higher-load systems.This article, the first in a series of two, provides an introduction torealtime scheduling (deadline scheduling in particular) and some of the theory behind it.

Security updates have been issued by Debian (ca-certificates, gdk-pixbuf, and graphicsmagick), Fedora (qtpass), openSUSE (python-openpyxl and syncthing), Slackware (kernel), and Ubuntu (gdk-pixbuf).