The internet runs on free open-source software. Who pays to fix it?

Right now, Volkan Yazici is working 22 hour days for free.

Yazici is a member of the Log4J project, an open-source tool used widely to record activity inside various types of software. It helps run huge swaths of the internet, including applications ranging from iCloud to Twitter, and he and his colleagues are now desperately trying to deal with a massive vulnerability that has put billions of machines at risk.

The vulnerability in Log4J is extremely easy to exploit. After sending a malicious string of characters to a vulnerable machine, hackers can execute any code they want. Some of the earliest attacks were kids pasting the malicious code in Minecraft servers. Hackers, including some linked to China and Iran, are now seeking to exploit the vulnerability in any machine they can find that's running the flawed code.

And there's no clear end in sight. The Log4J issue amounts to a long-term security crisis expected to last months or years. Jen Easterly, director of the US Cybersecurity and Infrastructure Security Agency, has said this is one of the most serious flaws" she's ever seen.

To support MIT Technology Review's journalism, please consider becoming a subscriber.

For something so important, you might expect that the world's biggest tech firms and governments would have contracted hundreds of highly paid experts to quickly patch the flaw.

The truth is different: Log4J, which has long been a critical piece of core internet infrastructure, was founded as a volunteer project and is still run largely for free, even though many million- and billion-dollar companies rely on it and profit from it every single day. Yazici and his team are trying to fix it for next to nothing.

This strange situation is routine in the world of open-source software, programs that allow anyone to inspect, modify, and use their code. It's a decades-old idea that has become critical to the functioning of the internet. When it goes right, open-source is a collaborative triumph. When it goes wrong, it's a far-reaching danger.

Open-source runs the internet and, by extension, the economy," says Filippo Valsorda, a developer who works on open-source projects at Google. And yet, he explains, it is extremely common even for core infrastructure projects to have a small team of maintainers, or even a single maintainer that is not paid to work on that project."

No recognition

The team is working around the clock," Yazici told me by email when I first reached out to him. And my 6 a.m. to 4 a.m. (no, there is no typo in time) shift has just ended."

In the middle of his long days, Yazici took time to point a finger at critics, tweeting that Log4j maintainers have been working sleeplessly on mitigation measures; fixes, docs, CVE, replies to inquiries, etc. Yet nothing is stopping people to bash us, for work we aren't paid for, for a feature we all dislike yet needed to keep due to backward compatibility concerns."

Before the Log4J vulnerability made this obscure but ubiquitous software into headline news, project lead Ralph Goers had a grand total of three minor sponsors backing his work. Goers, who works on Log4J on top of a full-time job, is in charge of fixing the flawed code and extinguishing the fire that's causing millions of dollars in damage. It's an enormous workload for a spare-time pursuit.

The underfunding of open-source software is a systemic risk to the United States, to critical infrastructure, to banking, to finance," says Chris Wysopal, chief technology officer at the security firm Veracode. The open-source ecosystem is up there in importance to critical infrastructure with Linux, Windows, and the fundamental internet protocols. These are the top systemic risks to the internet."

How has it come to this? The answer comes in the form of another question: Why would tech companies pay for something they get for free? But the immense importance of open-source software means that the status quo is increasingly seen as untenable.

Volunteerism is unsustainable for critical infrastructure because volunteers are well within their rights to only work on the fun or interesting parts of the job,' Valsorda says. An open-source project also needs careful testing, release engineering, issue triage, security reviews, code review of contributions-and a maintainer may find some or none of these aspects motivating in themselves."

As pressure and critics pile on the Log4J team, old questions of fairness are being asked about the open-source world.

Fairness is a problem," says Ceki Gulcu, who founded Log4 . There's this weird imbalance, where you profit from something but you don't give anything back."

The public is also almost completely ignorant of the immense role-and risk-of the free-labor-powered open-source software that runs the internet. OpenSSL powers encryption, for example, and Linux is behind the most widely used operating systems on the planet, including Android.

Gulcu points to the problems of recruitment and retention on open-source projects. It's not easy to attract and keep talent on even big projects when the compensation ranges from a fraction of what a company might pay all the way down to zero. And that can have knock-on effects for national security.

In 2018, the developer behind a popular open-source project called ua-parser-js quit, unwilling to work for free anymore. The software is used by big tech firms including Google, Amazon, and Facebook. The person who took control of ua-parser-js then hijacked the software and added malicious code to the project to steal cryptocurrency. The US Department of Homeland Security eventually issued a warning to users about the hacker at work. Despite the many thousands of developers using the software, that project had raised a paltry $41.61 in funds. The original developer, who had freely given up control to the anonymous successor, called the situation insane."

It is not as though top-tier software developers always dedicate years of free labor and get nothing in return, however. Gulcu, for instance, parlayed his free work on Log4J into multiple lucrative software development jobs in the finance industry.

It's actually pretty typical for open-source work to help build a portfolio that then leads to paid jobs. In some ways the structure resembles unpaid internships in other industries-a system increasingly seen as unethical, exploitative, and unfairly advantageous to people who can afford to take on heaps of uncompensated work at the expense of those who cannot. In this way, the underfunding of open-source work may perpetuate more than just technical issues.

How to fix the status quo

The problems with this situation are at last gaining recognition.

Tech companies, enterprises, anyone writing software is dependent on open-source," says Wysopal. Now there is a recognition at the highest levels of government that this is a big risk."

Easterly and other experts say that tech companies need to improve transparency. Adopting a Software Bill of Materials, as mandated by a 2021 executive order on cybersecurity from President Joe Biden, would help both developers and users better understand what is actually vulnerable to hacking when software flaws are discovered.

Valsorda, who has managed to turn his own open-source work into a high-profile career, says that formalizing and professionalizing the relationship between developers and the big companies using their work could help. He advocates turning open-source work from a hobbyist pursuit into a professional career path so that critical infrastructure isn't dependent on the spare time of a developer who already has a full-time job. And he argues that companies should develop systems to pay the people who maintain open-source projects their fair market value.

Some companies have already recognized the need. Google recently pledged $100 million to support open-source development and to fix vulnerabilities.

Wysopal says more has to be done to understand the health of open-source projects-Was the last update a week ago or two years ago?-and then to systematically support good projects while killing the ones that can't be secured. Another Google project, the Open Source Technology Improvement Fund, aims to audit and improve critical open-source projects.

The fallout from the Log4J vulnerabilities is a perfect example of a larger problem, though. The flaws are in the design of the software, and so to find it, you need someone who really understands the design. Current bug bounty" models, which pay outsiders to take a look at software and find flaws, don't do enough to help here, because outsiders simply don't have the financial incentive to develop that kind of deep understanding.

This is absolutely a market failure," says Wysopal. We're taking the good part of shared code, and we're making someone else take the fall for the bad part. There has to be more funding for finding and fixing."