The Linux kernel's binfmt_misc functionality allows non-native/custom files to be run directly such as for Windows EXE files to be handled by Wine or JAR files to be handled by the Java runtime. For the upcoming Linux 7.3 merge window, the miscellaneous binary formats support is being extended so a binary type can be matched programmatically and the interpreter determined on a per-exec basis. BPF programs can also then be attached to dynamically choosing an execution environment on a per-binary basis.

Christian Brauner sent out his main set of VFS pull requests already in advance of the Linux 7.3 merge window opening. The Linux 7.3 cycle should kick off tomorrow, 17 August, assuming today's on-time release of the Linux 7.3 kernel. Among the many interesting VFS changes slated for Linux 7.3 is the extending of the binfmt_misc code to opening up a lot of new possibilities.

There's work for NixOS and other platforms that may want to pursue relocatable binaries with the interpreter / dynamic loader determined programmatically, such as relative to the binary file. With BPF programs now able to be attached to binfmt_misc entries also opens the door for other innovative possibilities.

Brauner explained in the binfmt_misc pull request:

"This contains a bunch of work for binfmt_misc. It fixes a bunch of old bugs, reworks the locking, and then extends the format registry so a binary type can be matched programmatically and its interpreter computed per exec instead of being a fixed string recorded at registration time.

This allows nixos and other to e.g., implement relocatable binaries meaning the interpreter/dynamic loader can be determined programatically, say found relative to the binary. The mechanism is flexible and can support other policies...Add the ability to attach bpf programs to binfmt_misc entries so it's possible to dynamically choose the execution environment such as the loader or interpreter on a per binary basis."

There are also two dispatch modes added as part of this binfmt_misc overhaul:

"So far the chosen interpreter owns the whole process identity (argv[0], /proc/pid/cmdline, /proc/self/exe all name interpreter information). So relocatable find the dynamic linker instead. Also a binary passed to execveat() as an inaccessible O_CLOEXEC fd cannot run at all and gdb trips because AT_ENTRY and AT_PHDR do not match the exe file. So PIE symbols are unrelocated.

This adds transparent dispatch which allows the interpreter to load the binary through AT_EXECFD and leaves the argument vector exactly as the caller built it and labels mm->exe_file and comm with the binary. It also raises the AT_FLAGS_TRANSPARENT_INTERP aux vector bit. The interpreter keeps control of mapping the binary.

The second mode is loader substitution. This allows a binary to be executed natively and only the interpreter to be changed."

Assuming no objections from Linus Torvalds during the Linux 7.3 merge window over this code, it will be interesting to see how this new binfmt_misc functionality is leveraged by different Linux distributions/environments moving forward.