Virginia Tech's "Popcorn Linux" For Distributed Thread Execution Seeking Feedback, Possible Upstreaming

Popcorn Linux has been a multi-year effort out of Virginia Tech's Software and Systems Research Group for distributed thread execution across systems and even potentially different ISAs/accelerators given today's heterogeneous hardware.

Popcorn Linux is principally based around the Popcorn Compiler, a modified version of LLVM, and modifications to the Linux kernel and a support run-time library to allow for execution with a shared code-base on distributed hosts.

According today's request for comments, "Popcorn allows applications to start execution on a particular host and
migrate, at run-time, to a remote host. Multi-threaded applications may migrate any particular thread to any remote host. Unlike userspace checkpoint-restart solutions (e.g., CRIU), Popcorn enables seamless and dynamic migration across hosts during execution (no user interaction), and ensures coherent virtual memory across hosts for concurrent thread execution."

As for the kernel changes for this Popcorn approach to distributed thread execution, "Popcorn Linux implements a software-based distributed shared memory by extending Linux's virtual memory subsystem and it enables processes on different machines to observe a common and coherent virtual address space. Coherency of virtual memory pages of different hosts is ensured using a reader-replicate/writer-invalidate, page-level consistency protocol."

While a "request for comments" on Popcorn Linux was sent out on Wednesday to the Linux Kernel Mailing List, it's still a long ways out from potentially being upstreamed: the current code is based on Linux 5.2 rather than Git head, right now only the x86 support is presented as part of the RFC, some security features like L1 Terminal Fault mitigations have to be disabled for the moment, and there are known security concerns to the point of recommending Popcorn Linux currently not be loaded onto public Internet-connected systems.

But in any case, the RFC is out there so we'll see if upstream developers would ultimately be interested in such a solution. Those wanting to learn more about Virginia Tech's research in this area can visit PopcornLinux.org and the GitHub repository.