Valen creator drives 'Golden Spike' to connect new languages with Rust

Programming language generics can now ride in style from one language's compiler to another and back again, thanks to experimental work from Mojo core contributor Evan Ovadia. This former Google engineer has endeavored to turn Rust into a library that other languages can build from, demonstrating a way to pass generic data types and functions between Rust and another language, an action the C ABI doesn't support. Technically, this is the most fascinating thing I've read in months," enthused the host of the Let's Get Rusty YouTube channel, explaining the work, which he characterized as Rust++." That's a geeky but hard-won victory to be sure. Ovadia describes his work in an explanatory blog post as the Golden Spike" that intertwines two different languages, referring to the final spike hammered into the United States' first transcontinental railroad line. Instead of being stripped down into raw primitives just to be crammed into the coarse wagons of C ABI, libraries of rich capability can be sped to foreign ecosystems on sturdy, high-speed rails. Ditching the C ABI at last The work came about in Ovadia's effort to create his own language, Valen (an offshoot of an earlier effort called Vale), which he envisions as a speedier, more dev-friendly version of Rust. The problem with creating a new language from scratch, he reasoned, was that he couldn't use his favorite Rust libraries (notably Rust's bitchin' wgpu graphics library). Today, a common way to use a library from one language in a program written in another is through the C ABI (application binary interface), a de facto interoperability layer that defines conventions for passing data and function calls between compiled code. (Rust has its own ABI, though it does not offer stability guarantees.) With the C ABI, if a Rust library exposes its functions as extern "C" and exposes its types as repr(C), then they can be accessed from another language using a Foreign Function Interface (FFI). All well and good, but the problem Ovadia had was that the C ABI did not understand the concept of generics (or placeholders for data types to be provided later), because C itself doesn't understand the concept of generics. But Rust uses generics quite a bit, and so does Valen, though not in the same way. So the central question, the central mystery to solve, is how do we do cross-language generics?" Ovadia wrote. Cross-language generics at last Getting two compilers to collaborate is super tricky, as compilers tend to be isolated entities, solitary kings of their own individual worlds. So Ovadia needed to come up with a way for Valen to call a Rust generic function, but also have a Rust generic function that calls back into Valen for needed values. After much rumination, Ovadia arrived at a two-part solution to cross-language generics: first, run the Rust compiler as a library. Surprisingly, Rust already possesses this capability (rustc_driver), which would allow Valen to embed rustc directly into its own memory space. The second step would be to patch Rust so that when it encounters non-Rust Valen types, it hands control over to the Valen compiler to handle its portion, and waits for the compiled result. Ovadia admits that patching the Rust compiler was the nuclear option." Honestly, this was the riskiest part of the whole endeavor," he wrote. The patch itself is only about 100 lines of code, but it is hacky and assumes Rust is using LLVM for machine code generation, which isn't always the case. It works, but it's definitely not upstreamable," Ovadia wrote. Nonetheless, Ovadia's cross-language bidirectional generics are a genuine step forward in programming language interoperability. Ovadia has his hands full with Valen at this point, though his cross-language generics work opens the possibility of using Rust to bootstrap other new languages that can be built from Rust libraries. Given a few modifications to Rust itself, the reliance on the creaky C ABI may one day be a thing of the past. On the Reddit and Lobste.rs programmer forums, observers praised Ovadia for tackling such a difficult problem, even as they fretted about the details. I appreciate the move to replace C as the universal FFI language,'" one commenter wrote. "It is a very bold choice and would require effort." Another reader commented that Rust may not be the ideal language for interoperability, given the limitations of its own type system. However, imagine you could build bindings on top of Ada! Now that would be amazing," they wrote. (R)