This shouldn't have happened: A vulnerability postmortem (Project Zero blog)

Over on the Project Zero blog, Tavis Ormandy has a lengthy postmortem on a vulnerability that he found in the Network Security Services (NSS) cryptography library. The vulnerability is a bog-standard buffer overflow that has existed in the library since 2012 despite various kinds of static analysis, testing, and fuzzing that Mozilla and others have applied to it over the years. He found it with a new fuzzing technique:

To solve this problem, I've been experimenting with monitoring the call stack during execution.

The naive implementation is too slow to be practical, but after a lot of optimization I had come up with a library that was fast enough to be integrated into coverage-guided fuzzing, and was testing how it performed with NSS and other libraries.

[...] The fuzzer I was using is able to isolate and extract interesting new ASN.1 OIDs, SEQUENCEs, INTEGERs, and so on. Once extracted, it can then randomly combine or insert them into template data. This isn't really a new idea, but is a new implementation. I'm planning to open source this code in the future.