Appendix A: Paravirtualization from Xen to Android¶

The rest of this book treats the Android Emulator as a thing in itself: a forked QEMU, virtio devices, and a per-host CPU accelerator. This appendix steps back to ask where that recipe came from. It traces a 2003 research idea about cooperative virtual machines. The idea reshaped Linux and spread across operating systems. It now lets a Debian virtual machine run inside the phone in your pocket.

The appendix also shows why the emulator, Cuttlefish, and Android's on-device virtualization framework are all variations on the same theme. It reflects developments through mid-2026.

A.1 The core idea¶

In full virtualization, the guest operating system believes it owns real hardware. The hypervisor has to trap and emulate every privileged operation the guest issues. In paravirtualization (PV), the guest knows it is virtualized and cooperates. It does not issue privileged instructions that the hypervisor must trap. It makes explicit, well-defined calls into the hypervisor instead. These calls are hypercalls.

The cost is that the guest must be modified. The benefit is speed: far fewer expensive transitions between guest and hypervisor, especially for memory management and I/O. Almost everything that follows is the story of how the industry negotiated that single trade-off. The answer evolved from "modify the whole kernel" to "keep the kernel native, cooperate only where it pays."

A.2 The arc at a glance¶

The word "paravirtualization" did not mean the same thing in 2025 that it meant in 2003. It started as a demand to port an entire guest kernel, and it ended as a narrow contract for device I/O. The timeline below shows that shift in three eras. First, classic full-kernel PV gave way to a long hybrid era. In that era, hardware took over CPU and memory virtualization, and PV survived only as the I/O path (virtio). Then that same hybrid recipe arrived on the phone.

flowchart TB
    E1["2003 · Classic Xen paravirtualization<br/>entire guest kernel modified"]
    E2["2006-07 · Hardware assist arrives (KVM)<br/>CPU and memory PV now optional"]
    E3["2007-08 · virtio standardizes PV I/O<br/>PV narrows to device drivers"]
    E4["2012 · Hybrid era: PVHVM / PVH<br/>HW virtualization + virtio wins"]
    E5["2022-23 · Android AVF + pKVM ship<br/>HW virt + virtio, on the phone"]
    E6["2025 · Linux Terminal · SESIP L5<br/>confidential on-device VMs"]
    E1 --> E2 --> E3 --> E4 --> E5 --> E6

    classDef pv fill:#FAECE7,stroke:#993C1D,color:#712B13;
    classDef hybrid fill:#E1F5EE,stroke:#0F6E56,color:#085041;
    classDef android fill:#EEEDFE,stroke:#534AB7,color:#3C3489;
    class E1 pv;
    class E2,E3,E4 hybrid;
    class E5,E6 android;

Figure A-1: Two decades of paravirtualization, grouped into three eras

A.3 Origins: Xen and x86 paravirtualization¶

Paravirtualization as a practical x86 technique was born from the XenoServers project at the University of Cambridge Computer Laboratory. The project began in 1999 under Ian Pratt. The motivating question was deceptively simple, and Pratt pursued it with his PhD student Keir Fraser. Rather than fool a guest kernel into thinking it runs on bare metal, what if you told it that it was virtualized? What if you gave it a cleaner, virtualization-friendly interface to work against?

The team collaborated with researchers at Intel and Microsoft. It took Linux and Windows XP and removed the instructions and behaviors that were slow or impossible to virtualize on the x86 of the era. It replaced them with hypercalls. The result was Xen. Its first public release came in October 2003. The landmark SOSP (Symposium on Operating Systems Principles) 2003 paper Xen and the Art of Virtualization accompanied it.

The context made it matter. The x86 of that era had no hardware virtualization support, and several privileged instructions did not trap cleanly. So full virtualization on x86 required heavy techniques such as binary translation. Paravirtualization sidestepped all of that by changing the guest.

Xen introduced vocabulary still in use: a privileged control domain (Dom0) that owns the hardware and management stack, and unprivileged guest domains (DomU). The model proved commercially decisive. Amazon Web Services ran on Xen from its beginnings, so PV was one of the quiet engines of the early public cloud.

A.4 Into Linux: pvops, lguest, KVM, and virtio¶

For years, Xen users had to maintain a heavily patched, "Xen-ified" Linux kernel out of tree. The cleanup came through paravirt-ops (pvops). It is an abstraction layer merged into mainline Linux. It lets a single kernel binary detect at runtime whether it runs natively or under a hypervisor (Xen, KVM, VMware's VMI, and others). The binary then selects the appropriate low-level operations. With pvops, upstream Linux gained Xen DomU (guest) support in kernel 2.6.23 (2007) and Dom0 (control domain) support in 2.6.37 (2011).

Meanwhile, two developments reframed the landscape. KVM (Kernel-based Virtual Machine), developed at Qumranet, was merged into Linux 2.6.20 in 2007. It turned the Linux kernel itself into a hypervisor. It relied on the brand-new hardware virtualization extensions (Intel VT-x, AMD-V). This was a different bet from Xen's: let the hardware handle CPU and memory virtualization, and use Linux as the host.

Alongside it came lguest, a deliberately tiny "Linux-on-Linux" hypervisor by Rusty Russell (merged in 2007). It was meant as a teaching tool, but its real legacy was what came with it.

That legacy was virtio. Russell, then at IBM, observed that the Linux kernel already supported at least eight distinct virtualization systems. They were Xen, KVM, VMware's VMI, IBM's System p and System z, User Mode Linux, lguest, and the legacy iSeries. Each of them shipped its own block, network, and console drivers.

His 2008 paper, virtio: towards a de-facto standard for virtual I/O devices, proposed a common front-end driver framework. The framework had a clean ring-buffer transport called vring. This transport is a descriptor array, plus an "available" ring the guest fills and a "used" ring the host fills. Any hypervisor that implemented the vring transport could immediately reuse Linux's well-maintained virtio drivers instead of inventing its own.

This is the pivotal conceptual move in the whole story. Classic Xen PV asked you to paravirtualize the entire kernel. virtio paravirtualized only the part that benefited most, device I/O. It used a matched pair of front-end drivers (in the guest) and back-end drivers (in the host) that talk over shared memory. It became the de-facto standard for KVM's paravirtualized devices. Today almost every cloud VM uses it for disk and network, whether the user realizes it or not.

A.5 Inside virtio: the virtqueue¶

Look inside virtio, because the same data structure underlies every paravirtualized device in this appendix (in the cloud, in Android, and in the emulator). The contract has two halves: a small configuration interface for discovery and feature negotiation, and one or more virtqueues that carry the actual data. A device can have several; the simplest virtio-net has two, one for transmit and one for receive.

What made virtio portable rather than merely tidy was a deliberate separation of three concerns. Earlier paravirtual device models had fused them. Russell's 2008 paper draws the line explicitly between three parts. The driver is a Linux-internal abstraction: an operations structure that a generic block or net driver calls. The transport is the ring mechanism that actually moves buffers. Configuration is how a device is discovered and how its features are negotiated.

If a design fuses those, as Xen's early model did, a driver is welded to one hypervisor's probing scheme. If a design separates them, the same driver rides any transport that implements the ring. That decoupling is why a single set of Linux drivers could serve the eight different virtualization systems that Linux already carried in 2008. These were Xen, KVM, VMware's VMI, IBM's System p and System z, User Mode Linux, lguest, and the legacy iSeries. Each of them had shipped its own duplicate block, network, and console drivers that no one enjoyed maintaining.

A virtqueue in its original "split" layout is three shared data structures the guest allocates, each writable by only one side. The descriptor table holds the buffers. Each descriptor has an address, a length, some flags, and an optional link to a next descriptor. This link lets one request chain several memory segments.

The available ring is where the driver publishes the heads of the descriptor chains it offers. The used ring is where the device publishes the chains it finishes with. Because each ring has exactly one writer, the two sides never contend for a lock on the shared structures.

flowchart TB
    DRV["Driver (guest) · virtio front-end"]
    subgraph VQ["virtqueue · shared guest memory"]
        DT["Descriptor table<br/>addr · len · flags · next"]
        AR["Available ring<br/>driver to device"]
        UR["Used ring<br/>device to driver"]
    end
    DEV["Device / back-end · QEMU · vhost(-user)"]
    DRV -- "publish + kick" --> VQ
    VQ -- "device reads" --> DEV
    DEV -- "writes completion" --> VQ
    VQ -- "interrupt" --> DRV

Figure A-2: The split virtqueue: three single-writer rings in shared memory

The cycle is simple. The driver writes a buffer into the descriptor table and puts its index into the available ring. Then it rings a doorbell to notify the device. The doorbell is a "kick", a single MMIO write. The device reads the available ring and does the work, for example it copies a packet or services a disk request. Then it writes the index into the used ring and raises an interrupt.

The crucial detail is that both notifications can be suppressed. Under load, a busy device stops asking to be interrupted, and a busy driver stops sending kicks. So a stream of requests costs almost no transitions into the hypervisor. That batching is the entire performance argument for paravirtualization, in concrete form.

Concretely, the guest-facing API is tiny. A driver calls add_buf to post a scatter-gather buffer, with readable parts first and writable parts after. It calls kick to ring the doorbell, often once after it queues many buffers. It calls get_buf to reap completions. Completions need not return in the order the driver offered them.

Two further calls, enable_cb and disable_cb, switch completion callbacks on and off. They are the software equivalent of masking a device's interrupt.

A single block read shows the shape. The driver chains three descriptors. The first is a read-only sixteen-byte header that carries the request type and sector number. The second is a writable data buffer that the device fills. The third is a one-byte writable status field.

The driver publishes the head of that chain in the available ring and kicks. The device reads the request and fills the buffer. It writes 0 into the status byte for success and returns the head in the used ring.

flowchart TB
    subgraph DESC["descriptor table — one block-read request, chained by next"]
        direction LR
        H["Header<br/>read-only · type, sector"]
        D["Data buffer<br/>write-only · device fills"]
        S["Status byte<br/>write-only · ok / fail"]
        H --> D --> S
    end
    AR["Available ring<br/>driver publishes head"]
    UR["Used ring<br/>device returns id + len"]
    AR -- "head index" --> H
    S -- "on completion" --> UR

Figure A-3: A block read as a chained descriptor

One small field in that used ring carries an outsized lesson. Each completion records which chain the device consumed. It also records how many bytes the device actually wrote. The paper is careful to note that this length must come from a trusted source. Picture a guest that has a buffer for a 1514-byte packet. The sender, malicious or merely buggy, copies in only part of the packet.

Without a trusted byte count, the receiver would treat whatever stale data already sat in the buffer as the rest of the packet. It could then leak that data onward. The used ring records the real length centrally. So no driver has to zero or sanitize its own buffers.

This is a miniature of the threat model that later sections carry to its conclusion. As soon as a virtqueue can face an untrusted peer, the discipline of who-writes-what stops being an optimization and becomes a security boundary. The peer can be another guest, or a host that a protected VM does not trust.

Refinements layer on top. Indirect descriptors let a chain live in a side table. This enlarges the effective ring capacity. The newer packed virtqueue was added in the virtio 1.1 standard. It collapses the three rings into one ring with driver and device wrap counters. This layout is friendlier to CPU caches, and to real hardware that fetches the structures over PCIe.

The transport is also deliberately separable. The same virtqueues ride over virtio-pci on an ordinary VM. They ride over virtio-mmio on lightweight hosts like the emulator's ranchu board, or over virtio-ccw on IBM mainframes.

The first of these was the pragmatic key to adoption. Full-virtualization hosts already emulate a PCI bus, and guests already know how to bind PCI drivers. So virtio dressed a virtqueue as a PCI device, under the vendor ID 0x1AF4 that Qumranet, KVM's originator, donated to the project. This let it slot into otherwise unmodified guests with almost no new host plumbing.

The back-end is separable too, and that is where modern performance lives. The device implementation need not sit in the VMM. vhost moves the data plane into the host kernel (vhost-net, vhost-scsi), so packets never traverse userspace. vhost-user moves it into a different userspace process, such as a DPDK switch or an SPDK storage target. That process maps the guest's virtqueues over a socket.

The paper's own future-work section foresaw the first move. It sketched a /dev/vring file descriptor to lift the ring out of userspace entirely. This was the seed that grew into vhost.

At the far end of the spectrum, vDPA (virtio data-path acceleration) lets a physical NIC speak the virtqueue protocol directly. So a guest's stock virtio driver talks to real silicon with no software back-end in the path at all. The paper already glimpsed this ending. It observed that a virtqueue behaves like a high-speed physical device. Software uses DMA to move data to and from such a device, and addresses it as seldom as possible. So hardware that speaks virtqueue natively is less a reinvention than a homecoming.

The breadth follows from the same minimalism. The framework standardizes only the ring discipline and a per-device feature handshake, so new device classes slot in without new plumbing. Beyond block and net there is virtio-scsi, virtio-gpu, virtio-fs (host directory sharing), virtio-rng, virtio-balloon (memory reclaim), virtio-snd, and virtio-vsock. The last one is the socket transport that carries messages between a protected VM and its host, as the Android section below shows.

A.6 The hardware-assist turn: from PV to PVHVM to PVH¶

When Intel VT-x and AMD-V arrived (2005–2006), they made it cheap to trap privileged CPU and memory operations. That undercut the rationale for full-kernel paravirtualization. An unmodified guest could now run efficiently as a hardware-assisted (HVM) guest. But it stayed painfully slow to emulate I/O devices register by register. The industry converged on a hybrid sweet spot.

Xen's taxonomy captured the spectrum with four modes. PV is a fully paravirtualized guest with paravirtualized devices; HVM is a hardware-assisted, unmodified guest with emulated devices. PVHVM uses hardware virtualization for CPU and memory, but paravirtualized virtio-style drivers for I/O. It is the practical default. PVH is a leaner mode that Mukesh Rathor of Oracle proposed at XenSummit 2012. It is a paravirtualized guest that hardware virtualization accelerates, and it sheds emulated-hardware baggage.

The lesson carried forward: pure PV faded, but paravirtualized I/O won permanently. "Paravirtualization" in modern practice almost always means virtio devices on top of a hardware-virtualized guest — not a rewritten kernel.

A.7 Beyond Linux and Xen¶

The cooperative-guest idea spread well past Xen. VMware introduced VMI (Virtual Machine Interface), a paravirtualization interface that it presented at the Ottawa Linux Symposium in 2006. It also introduced paravirtualized device drivers (VMware Tools) that long paralleled virtio's goals; VMI was later deprecated as hardware assist matured. IBM, for its part, shipped paravirtualized guests for decades on System p, System z, and the legacy iSeries. This mainframe virtualization heritage predates x86 entirely. The most consequential non-Linux adopter, though, was Microsoft, and it earns the section that follows.

A.8 Paravirtualization in Windows: Hyper-V enlightenments¶

Linux is not the only system that learned to cooperate with a hypervisor. Microsoft built paravirtualization into Windows under the name enlightenments. An "enlightened" guest detects that it runs on the Hyper-V hypervisor, then stops pretending that it owns hardware and asks the hypervisor directly. The interface is published as the Hyper-V Top-Level Functional Specification (TLFS). A guest finds it through CPUID and maps a hypercall page through synthetic model-specific registers. From then on, it issues hypercalls exactly as a Xen PV guest or an Android pKVM guest does.

The CPU-level enlightenments target the operations that are most expensive to trap and emulate. The guest uses a paravirtual APIC with exit-less end-of-interrupt signaling, instead of a real local APIC. It issues single hypercalls (HvFlushVirtualAddressSpace, HvCallSendSyntheticClusterIpi) in place of the slow broadcast of TLB shoot-downs and inter-processor interrupts. It reads time from an enlightened reference-TSC page and does not trap to a timer. It uses paravirtual spinlocks, so a virtual CPU that waits on a lock can yield instead of using up a time slice. A synthetic interrupt controller (SynIC) and synthetic timers round out the set.

For I/O, Hyper-V uses the same split-driver shape as virtio, with its own vocabulary. The privileged root partition (the host) runs Virtualization Service Providers (VSPs). Each guest child partition runs the matching Virtualization Service Clients (VSCs). The two halves of a synthetic device communicate over VMBus. A synthetic device is, for example, a synthetic NIC (NetVSP / netvsc) or a SCSI controller (StorVSP / storvsc).

VMBus is a purely synthetic channel. It exists as no virtual hardware at all, and it is established through hypercalls. Each VMBus channel is a pair of ring buffers in shared memory. Requests and responses are matched by transaction ID. It is, structurally, the front-end/back-end pattern this appendix keeps returning to.

flowchart TB
    subgraph CHILD["Child partition (guest)"]
        GOS["Enlightened guest OS<br/>Windows/Linux · hypercall-aware"]
        VSC["VSC client drivers<br/>netvsc · storvsc"]
    end
    subgraph ROOT["Root partition (host)"]
        VSP["VSP providers<br/>NetVSP · StorVSP"]
        PDD["Physical device drivers<br/>real NIC / disk"]
    end
    HV["Hyper-V hypervisor<br/>enlightenments · SynIC · synthetic timers"]
    HW["x86-64 + VT-x / AMD-V"]
    VSC <-- "VMBus" --> VSP
    VSP --> PDD
    GOS -- "hypercall" --> HV
    ROOT -- "hypercall" --> HV
    HV --> HW

Figure A-4: Hyper-V enlightened I/O over VMBus

The guest-side pieces ship as Integration Services. These are the VSC drivers plus helpers for time sync, heartbeat, and graceful shutdown. They are pre-installed in modern Windows. The equivalent code went into the mainline Linux kernel in 2.6.32, so a Linux guest runs enlightened on Hyper-V out of the box.

The relationship is symmetric across vendors, too: KVM can emulate the Hyper-V interface. So a Windows guest on Linux uses these same enlightenments for CPU and timing. It uses signed virtio drivers (the virtio-win package) for its disks and network. Those Windows drivers are nearly as old as virtio itself. Qumranet shipped beta virtio-PCI drivers for Windows guests in the framework's very first year. A Windows VM in the cloud is typically paravirtualized twice over: Hyper-V enlightenments above, virtio beneath.

There is a final twist that mirrors pKVM on Android. On a modern PC, Hyper-V often runs even when you never created a VM. Virtualization-Based Security, Credential Guard, Windows Sandbox, and WSL2 all switch the hypervisor on. At that point the ordinary Windows kernel is itself relocated into the root partition and runs as a guest of Hyper-V. As with Xen's Dom0 and Android's deprivileged host kernel, the "host" operating system quietly becomes an enlightened guest of the hypervisor beneath it.

A.9 The mobile lineage before Android adopted virtualization¶

Mobile virtualization is older than most people assume. It was paravirtualized from the start, because early phone CPUs lacked hardware virtualization. The standout was the OKL4 Microvisor from Open Kernel Labs. It was a microkernel-derived Type-1 hypervisor purpose-built for phones.

OK Labs systematically paravirtualized guest operating systems. It supported a remarkable range for the time: multiple Linux distributions, Android, Symbian, Windows Mobile, and assorted RTOSes. Its pitch was hardware consolidation. It collapsed the application processor and the baseband/radio stack onto a single core, to hit "smartphone experience at feature-phone prices." The 2010 "One Core" Android platform is an example. OKL4-family technology shipped in very large volumes, particularly in mobile baseband processors.

OK Labs shares its roots with something profound. It was spun out of NICTA, the Australian national research institute that also produced seL4. seL4 was the first microkernel to earn a machine-checked formal proof of its functional correctness. This is a security pedigree that still echoes through today's mobile hypervisor work.

Samsung, separately, ran a Xen-on-ARM project to bring Xen's PV model to ARM devices. So when Android finally embraced on-device virtualization, it did not invent mobile virtualization. It industrialized it, with modern hardware support and an open, standardized stack.

A.10 Android's framework: AVF, pKVM, crosvm, and Microdroid¶

The driving question was security, not server consolidation. How do you run high-value, sensitive code so that even a fully compromised Android kernel cannot read or tamper with it? Trusted Execution Environments (TEEs) existed, but they were fragmented across vendors, hard to target, and limited. Google's answer was the Android Virtualization Framework (AVF). Google developed it publicly from around 2020, and it first shipped in Android 13.

pKVM (protected KVM) is the hypervisor at the heart of AVF. It is an extension of mainline Linux KVM on ARM64. Its design is itself a paravirtualization story. During boot, the Linux kernel hands control of the EL2 exception level to the small pKVM core. After that, Linux runs as a deprivileged host at EL1 that can no longer freely access guest memory.

Context switching and inter-guest communication happen through hypercalls to pKVM at EL2. When memory is donated to a protected VM (a "pVM"), it is unmapped from the host entirely. The only shared pages are those that the guest explicitly shares back, notably for virtio devices. This keeps the trusted computing base tiny and auditable, and it deprivileges the very kernel it boots from.

crosvm is the Virtual Machine Manager. It is a Rust-based VMM that Google adapted from ChromeOS (its name comes from "Chrome OS Virtual Machine Monitor"). It allocates VM memory, creates virtual CPU threads, and implements the virtio back-end drivers. It relies heavily on paravirtualized devices, and it abstracts across multiple hypervisor backends. It is also used in Cuttlefish (the AOSP virtual device) and Google Play Games on Windows.

Microdroid is a stripped-down "mini-Android" OS image meant to run inside a pVM. It has no system server, no HALs, and no GUI. Its purpose is not to be a general OS, but to give developers a familiar environment (Bionic, Binder IPC, keystore). Developers use it to offload a portion of an app into an isolated VM. The VM has stronger confidentiality and integrity guarantees than Android proper can offer. By Android 14, a Microdroid VM booted twice as fast as on Android 13, and it used half the memory.

VirtualizationService manages the lifecycle of these VMs. It exposes an AIDL (Android Interface Definition Language) API, and it manages one crosvm process per VM. Inter-VM and VM-to-Android communication runs over vsock, the virtio socket interface. Each VM is addressed by a 32-bit context ID. pvmfw (protected VM firmware) is the first code a pVM runs. It bootstraps verified boot and derives the VM's unique secret via DICE (Device Identifier Composition Engine).

flowchart TB
    subgraph HOST["Host Android (EL1)"]
        VS["VirtualizationService<br/>manages pVM lifecycle"]
        CV["crosvm (VMM)<br/>virtio back-ends"]
    end
    subgraph PVM["Protected VM (pVM)"]
        GOS["Microdroid / Debian<br/>guest OS payload"]
        VF["virtio front-end<br/>virtio-blk, -net, vsock"]
    end
    PK["pKVM hypervisor (EL2)<br/>minimal trusted base · isolates pVM memory from host"]
    HW["ARM64 + virtualization extensions"]
    CV <-- "virtio / vsock" --> VF
    HOST -- "hypercall" --> PK
    PVM -- "hypercall" --> PK
    PK --> HW

Figure A-5: The AVF stack: host and protected VM as peers on pKVM

The host and the protected VM are peers. Both are guests of pKVM, not host-over-guest. The only memory they share is the virtio/vsock channel. pKVM unmaps the rest of the pVM's pages from the host, so even a compromised Android kernel cannot read into the VM. The guest-side virtio front-end and crosvm's back-end are the same split-driver pattern from the general virtio model. Here the pattern spans a security boundary, not only a performance one.

So the architecture is hardware-assisted virtualization (pKVM) plus paravirtualized I/O and transport (virtio, vsock). This is the exact hybrid that the broader industry converged on, now hardened for a hostile-host threat model. The classic "rewrite the whole guest kernel" form of PV is nowhere in sight; the surviving, dominant form of paravirtualization is.

A.11 From platform plumbing to a user-facing product¶

For its first couple of years, AVF was invisible to ordinary users. It was infrastructure for things like isolated compilation (Compilation OS) and protected workloads. That changed with the Linux Terminal app, which turned the framework into something people could touch. The app first shipped to Pixel devices in the March 2025 Pixel Drop, and it was gated behind Developer Options.

The Terminal app boots a full Debian virtual machine on the phone via AVF/KVM. On first run it downloads a roughly 500 MB image. It is, in effect, a real hardware-virtualized Linux box that runs alongside Android. It is sandboxed so it cannot disturb the host. It has access to Debian's tens of thousands of packages, far beyond what userspace tools like Termux offer.

Progress through Android 16 and its quarterly releases was quick. The original 16 GB storage cap was removed. Dynamic storage ballooning followed in Android 16 QPR1 (Quarterly Platform Release 1). The VM's disk inflates and deflates on demand, and this retired the manual resize slider. The releases also added tab support for multiple terminal sessions, and graphical-app support with GPU acceleration on newer Pixel builds. This is enough to run GIMP, LibreOffice, Chromium, and full desktop environments like XFCE via Wayland/Weston.

Google frames the goal less as "a second desktop OS on your phone". It frames the goal more as a developer and power-user environment, and as a step toward Android–ChromeOS convergence and desktop-class windowing. Caveats remain. It is still experimental, and the rich graphical experience is largely Pixel-only so far. Some vendors (notably Samsung, as of this writing) ship devices with AVF disabled. A quick getprop ro.boot.hypervisor.vm.supported reports whether a given device supports it.

A.12 Not just pKVM: Gunyah and vendor flexibility¶

AVF was deliberately built to abstract over more than one hypervisor. Gunyah is an alternative backend. It is an open-source Type-1 hypervisor that Qualcomm developed in Sydney, the same city that produced OK Labs and seL4, not coincidentally. Like pKVM, it provides strong VM isolation with ARM's stage-2 MMU and GIC (interrupt controller) virtualization. Like the rest of this lineage, it handles devices through paravirtualization via inter-VM communication. It keeps a small, auditable trusted base suited to battery-constrained devices.

crosvm can target Gunyah as well as KVM/pKVM. Qualcomm ships Gunyah-based virtualization in its mobile and embedded platforms. AVF's vendor-module mechanism further lets partners customize the hypervisor for their own needs.

A.13 The inverse case: the Android Emulator¶

In all earlier sections, virtual machines run on Android. The Android Emulator inverts the picture. Here Android is the guest and your development machine is the host. Yet the recipe is identical: hardware-assisted CPU virtualization plus paravirtualized virtio devices. Despite the name, the modern emulator is not pure emulation; that is only its slow fallback.

Under the hood it is a downstream fork of QEMU (Chapter 4). Its current virtual board is ranchu, successor to the original goldfish board. The change that matters for this appendix is what ranchu did to devices. It brought standard virtio to the board (Chapter 6).

Storage arrives as virtio-blk, and the system image appears inside the guest as /dev/block/vda. Networking is virtio too. Graphics and input can run over virtio or the older goldfish transports, depending on the system image.

ranchu is in fact a deliberate mixture rather than a clean sweep. It retains a handful of Android-specific goldfish devices that it inherited from the old board, alongside its virtio devices. These are the framebuffer, battery, audio, and an input/events device. Above all, it retains the goldfish pipe. The pipe is a fast paravirtual host-to-guest transport used for jobs like GPU streaming and sensor injection. The emulator guest is, in other words, a paravirtualized guest in exactly the virtio sense that the rest of this story describes.

flowchart TB
    subgraph GUEST["Android guest — ranchu board"]
        VF["virtio front-end<br/>virtio-blk, -net, -gpu, -input"]
        GP["goldfish pipe<br/>legacy host-guest transport"]
    end
    QEMU["QEMU (Android fork)<br/>virtio device back-ends · Qt UI · ADB"]
    ACC["Hardware hypervisor (accelerator)<br/>KVM · HVF · WHPX · AEHD"]
    HW["Host OS + CPU<br/>Intel VT-x / AMD-V · ARM64 virt exts"]
    GUEST -- "virtio / pipe" --> QEMU
    QEMU --> ACC
    ACC --> HW

Figure A-6: The Android Emulator virtualization stack

The "hardware acceleration" half is pluggable per host operating system (Chapter 5), which is where the emulator's history rhymes with the rest of the field. On Linux it uses KVM. On macOS it uses Apple's Hypervisor.framework (HVF), on both Intel and Apple-silicon Macs. On Windows it uses either the Windows Hypervisor Platform (WHPX, layered on Hyper-V and now the recommended option) or the Android Emulator Hypervisor Driver (AEHD).

That last option closes a loop. For years the Windows accelerator was Intel's HAXM (Hardware Accelerated Execution Manager). Intel discontinued it in January 2023, and recent emulator builds dropped it entirely. Its replacement, AEHD, was formerly the "Android Emulator Hypervisor Driver for AMD Processors", and before that GVM. It is literally Linux KVM ported into a Windows kernel driver. The same KVM code that turned Linux into a hypervisor in 2007 now accelerates the Android Emulator on Windows.

So the emulator is the PVHVM hybrid, hardware CPU virtualization plus virtio devices, on a developer's desktop. It brought that hybrid to Android tooling years before AVF brought it on-device. It is also easily confused with its two siblings, so they are worth separating.

flowchart TB
    subgraph EMU["Android Emulator · ranchu (QEMU fork)"]
        E1["QEMU device backends"]
        E2["KVM · HVF · WHPX · AEHD"]
        E3["App developers"]
    end
    subgraph CF["Cuttlefish · AOSP virtual device"]
        C1["crosvm (default)<br/>QEMU still optional"]
        C2["KVM acceleration"]
        C3["System developers"]
    end
    subgraph AVF["AVF + pKVM · on-device pVMs"]
        A1["crosvm VMM"]
        A2["pKVM hypervisor (EL2)"]
        A3["On-device confidentiality"]
    end
    FOUND["Shared virtio foundation"]
    EMU --> FOUND
    CF --> FOUND
    AVF --> FOUND

Figure A-7: Three faces of Android virtualization

They share DNA but serve different masters. The Emulator (QEMU/ranchu) is for app developers in Android Studio. Cuttlefish is the high-fidelity AOSP virtual device for people who work on the system itself (Chapter 26). It now defaults to crosvm, though it can still run on QEMU. AVF/pKVM runs protected VMs on real devices. All three rely on the same paravirtualized-I/O foundation; only the trust model and the audience change.

A.14 Paravirtualizing the GPU: virtio-gpu, rutabaga, and Magma¶

If block and network devices were the easy wins for paravirtualized I/O, the GPU is the hard frontier. It is also the most revealing test of the whole approach. Enormous, latency-sensitive, vendor- and version-specific command streams (Vulkan, GLES) drive a modern GPU. Graphics also wants zero-copy sharing of buffers with the host's compositor.

You cannot economically trap and emulate GPU registers, and even plain virtio data buffers are not enough. What has to cross the boundary is graphics API work. So the GPU pushed virtio one level higher.

The virtio-gpu device standardizes the unglamorous parts: resource creation, mapping of shared "blob" memory, and fence signaling. It then carries a negotiable command protocol on top. The pivotal step was the context type feature (merged in Linux 5.16, which shipped in early 2022). It let a single virtio-gpu device offer several command dialects at once. It also let the guest choose which dialect to speak.

Several dialects coexist. The original virgl path has the guest's Mesa driver emit a Gallium intermediate stream. The host's virglrenderer translates that stream back into OpenGL. Venus is a thin Vulkan-on-Vulkan passthrough, also carried by virglrenderer.

Google's gfxstream forwards GLES and Vulkan calls to the host with minimal translation (Chapters 13 and 14). A cross-domain context carries Wayland buffers, so guest windows can appear directly on the host desktop. Resource handling, dma_buf/dma_fence synchronization, and the ring itself stay common; only the command dialect changes from one context to the next.

Something on the host has to receive whichever context the guest selected and route it to the right renderer. That something is rutabaga_gfx, the "Rutabaga Virtual Graphics Interface." It is a small, cross-platform Rust library. It sits atop gfxstream and virglrenderer, implements the cross-domain Wayland path itself, and exposes a C FFI so non-Rust VMMs can embed it. It originated inside crosvm, and it is now developed in the open as a standalone, BSD-licensed project.

Rutabaga is, in effect, what virtio did for block and network, applied to the GPU. It is a stable virtio-gpu front-end with pluggable rendering back-ends behind a single seam. QEMU now ships it as a backend (virtio-gpu-rutabaga). The same isolation instinct that this appendix showed earlier reappears here. A vhost-user-style vhost-device-gpu can run the entire graphics stack in a separate, memory-safe process, to shrink the attack surface that a GPU back-end exposes.

flowchart TB
    GUEST["Guest — virtio-gpu driver<br/>Mesa: virgl · Venus · gfxstream guest"]
    DEV["virtio-gpu device (crosvm · QEMU)<br/>accepts the guest's chosen context type"]
    RUT["rutabaga VGI<br/>dispatch by context type"]
    GFX["gfxstream<br/>GLES/Vulkan streaming"]
    VIRGL["virglrenderer<br/>virgl GL · Venus Vulkan"]
    CD["cross-domain<br/>Wayland passthrough"]
    HOST["host GPU + vendor driver<br/>Vulkan · GL · Metal · D3D"]
    GUEST -- "commands, resources, fences" --> DEV
    DEV --> RUT
    RUT --> GFX
    RUT --> VIRGL
    RUT --> CD
    GFX --> HOST
    VIRGL --> HOST
    CD --> HOST

Figure A-8: GPU paravirtualization through virtio-gpu and rutabaga

This is the plumbing that quietly unifies this appendix's separate GPU stories. crosvm's virtio-gpu device renders through rutabaga. So Android's protected and standard VMs and the Cuttlefish virtual device both use it. gfxstream is built on the Android Emulator's AEMU base. It is the very same host renderer that the desktop emulator relies on.

It even closes the loop with the emulator's past. In 2018, the old goldfish GPU-streaming pipe gave way to a port of gfxstream onto virtio-gpu. At first this port was a symbol-compatible drop-in for virglrenderer. This is why the accelerated graphics in three setups are all variations on one paravirtualized-GPU stack. The setups are the on-device Linux Terminal in Android 16, a Cuttlefish instance in a datacenter, and a developer's emulator on macOS.

What remains unstandardized is the contract itself. gfxstream, virgl, and Venus are competing answers to one question. The Magma effort aims to define a single GPU system-call and protocol layer. The rutabaga_gfx repository now anchors the Magma project, which takes its name and shape from Fuchsia's Magma graphics interface. The layer is deliberately optimized for remoting and microkernels. An OS-specific virtgpu piece handles the paravirtualization.

If it succeeds, it would be the GPU's "virtio moment," the very move this appendix traces again and again. That move collapses a thicket of bespoke cooperative interfaces into one negotiated standard. Here it applies to the hardest device on the bus. Its form aims squarely at the protected-VM hypervisors and microkernels (pKVM, Gunyah, seL4) that the earlier sections followed. Fittingly, the project even ships a no-VM test path: the Kumquat server, which runs the virtio-gpu/gfxstream protocol over a plain socket. It is the GPU analog of lifting a virtio back-end out into its own process.

A.15 Trajectory: confidential computing on devices¶

The direction points squarely at confidential computing on consumer devices. In August 2025, pKVM became the first software security system designed for large-scale consumer deployment to achieve SESIP Level 5 certification. Dekra evaluated it, and the evaluation included the highest tier of vulnerability analysis under Common Criteria. The motivating use case that Google names is on-device AI that works on highly personal data, with strong privacy and integrity guarantees. In this use case, a model or an agent runs inside a protected VM. The main OS cannot inspect that VM, even if that OS is compromised.

That closes a long arc. Paravirtualization began in 2003 as a performance hack to make x86 virtualization viable before the hardware could help. Hardware assist then absorbed the CPU-and-memory half of the problem. The cooperative-I/O half, virtio, quietly became universal infrastructure.

Today that same cooperative model is the substrate for the most security-sensitive workloads on a phone. It sits on hardware virtualization and a minimal hypervisor. The phrase "modified guest" narrowed from "an entire ported kernel" to "a guest that shares a virtqueue". But the bet that Pratt and Fraser made in Cambridge is now more central than ever. They bet that cooperation between guest and hypervisor beats an arms race of trap-and-emulate.

A.16 The timeline in dates¶

The diagram in §A.2 shows this arc visually; the table records the precise milestones.

Year	Milestone
1999	XenoServers project begins at Cambridge (Pratt, Fraser)
2003	Xen 1.0 released; Xen and the Art of Virtualization (SOSP)
2005–06	Intel VT-x ships (2005), AMD-V follows (2006); VMware VMI paravirt interface (Ottawa Linux Symposium, 2006)
2007	KVM merged into Linux 2.6.20; lguest and Xen DomU support land in 2.6.23
2008	virtio merged in Linux 2.6.24; virtio "de-facto standard" paper
2008	Microsoft Hyper-V ships; enlightened I/O over VMBus (VSP/VSC)
~2010	OK Labs "One Core" paravirtualizes Android via OKL4; AWS scaling on Xen
2011	Xen Dom0 support upstream in Linux 2.6.37
2012	PVH mode proposed at XenSummit (Oracle)
~2015	Android Emulator's ranchu board moves to virtio devices (qemu2)
2016	virtio-gpu gains 3D via virgl; gfxstream ported onto virtio-gpu by 2018
2019	virtio 1.1 adds packed virtqueues (cache- and hardware-friendly)
2022	virtio-gpu context types (Linux 5.16); rutabaga multiplexes virgl/Venus/gfxstream
~2020	Google develops pKVM / AVF in the open
2022–23	AVF + pKVM first ship (Android 13); Microdroid 2× faster in Android 14
2023	Intel discontinues HAXM; emulator moves to AEHD / WHPX on Windows
2024	Qualcomm open-sources Gunyah as an AVF backend
Mar 2025	Linux Terminal app (Debian VM) ships in the Pixel Drop
2025	Android 16: GUI/GPU support, storage ballooning; pKVM earns SESIP Level 5
2025	rutabaga_gfx developed in the open; Magma effort to standardize GPU paravirtualization

A.17 Sources and further reading¶

Xen Project — Paravirtualization (PV) and The Paravirtualization Spectrum (PV, PVHVM, PVH; background on the 2003 SOSP paper Xen and the Art of Virtualization).
XenServer — The Birth of Xen (Cambridge origins; Pratt and Fraser).
Xen (Wikipedia) — release history, Dom0/DomU, and the PV/HVM/PVHVM/PVH modes.
Rusty Russell — virtio: towards a de-facto standard for virtual I/O devices (2008; also at ACM). Its ring-buffer transport drew on Van Jacobson's network channels (LWN, 2006).
IBM Developer — Virtio: An I/O virtualization framework for Linux.
virtio internals — Red Hat, Virtqueues and the virtio ring and Packed virtqueue; OASIS VIRTIO 1.1 specification; QEMU vhost-user protocol.
Hyper-V paravirtualization — Linux kernel VMBus documentation; QEMU Hyper-V enlightenments; Microsoft Hyper-V architecture (root/child partitions, VSP/VSC).
Shaken Fist — Virtualization history.
Android Open Source Project — AVF overview, architecture, Microdroid, VirtualizationService.
Android Developers Blog — Virtual Machine as a Core Android Primitive (2023).
Google Security Blog — Android's pKVM Achieves SESIP Level 5 (2025).
Relaxed virtual memory in Armv8-A (arXiv:2203.00642) — pKVM EL2 boot and handover details.
Esper — virtualization in Android 13 and on Pixel 6.
Linux Terminal and Android 16 — It's FOSS, Android Authority, Chrome Unboxed.
Qualcomm — Gunyah Hypervisor Software; quic/gunyah-hypervisor (GitHub).
OKL4 Microvisor — overview; "One Core" (2010) and seL4 / NICTA verification.
Android Studio — Configure hardware acceleration for the Android Emulator (HAXM deprecation; AEHD and WHPX).
The ranchu virtual board (android-emulator-dev); Linaro — Run Android using QEMU (goldfish/ranchu and virtio).
GPU paravirtualization — rutabaga_gfx (Rutabaga VGI; gfxstream/virglrenderer/cross-domain dispatch, Kumquat, Magma). Also virtio-gpu context types (Linux 5.16); QEMU virtio-gpu backends (virgl, rutabaga, Venus, vhost-user-gpu); Fuchsia Magma.