Summary of "Bhyve guests with hardware accelerated graphics by Michael Chiu"
Overview
Michael Chiu demonstrated that GPU passthrough to bhyve (the FreeBSD hypervisor) is possible in practice despite bhyve documentation stating VGA passthrough is unsupported. The talk focused on practical, home-lab–scale techniques rather than deep internals.
Motivation / Use Cases
Two main classes of GPU use in VMs:
- Application-level acceleration (e.g., VirtualGL, CUDA)
- Easier to set up; does not require hardware-level passthrough.
- Full hardware passthrough / vGPU
- Required when a VM must own the GPU or present a full display to the guest.
Key Findings and Demo Results
- Successfully passed an NVIDIA RTX 2070 to a bhyve guest and produced output on a monitor.
- AMD (open-source amdgpu DRM) worked more smoothly: guest console and display came up without special bus-id fiddling.
- NVIDIA cards were more problematic:
- Many GPUs won’t re-initialize after being used by the host or a previous guest.
- Proprietary drivers complicate debugging; some GPUs must remain uninitialized by host/BIOS/console to be passable.
- Basic GL benchmark (glmark) showed passthrough performance close to bare-metal but somewhat lower; results depend on CPU/VM resource allocation and driver/OpenGL coordination.
- Passing USB controllers/devices to guests was unstable (repeated attach/detach events observed in FreeBSD guests).
Hardware and Host Setup Used
- Motherboard: Gigabyte desktop board with five PCIe slots (chosen to simplify multi-GPU experiments).
- Host GPU (console): NVIDIA GTX 1050 Ti — used to keep the host console off other cards.
- Guest GPUs tried: NVIDIA RTX 2070, AMD RX 550/580 series.
- Guests installed on SSDs; VMs created via bhyve.
Prerequisites, Tips and Gotchas (Practical Guide Summary)
- GPU must be a PCI device and must not have been initialized by the host (BIOS/console). Use a separate host GPU for console if necessary.
- Integrated laptop/hybrid GPUs are often not passable because firmware or host initialization touches them.
- Configure standard PCI passthrough steps for bhyve (device assignment, IOMMU / VT-d).
- For FreeBSD guests with X:
- Use a UEFI loader to enable vt console.
- Pass the correct bus-id to Xorg when required.
- NVIDIA driver module loading can be tricky:
- Proprietary modules are large; FreeBSD loader has default buffer limits (e.g., ~64 MB) that can break loading.
- Workarounds: careful use of kldload/kldunload and loader.conf manipulations so the guest can bind the GPU driver.
- Some GPUs won’t reset cleanly. Attempts to force power-state changes or resets may fail — often requiring a host reboot to reuse the GPU.
- USB passthrough in his tests was unstable; expect repeated attach/detach behavior.
Driver / OS Behavior Comparisons
- AMD (amdgpu, open-source DRM)
- Generally reliable in bhyve guests; console and display usually work with little extra handling.
- NVIDIA (proprietary drivers)
- Fragile with re-initialization and reuse; Windows guests were especially problematic and often failed to boot with the passed GPU.
- Intel (i915 / GVT-g)
- Promising for multi-vGPU (vGPU) use — i915 DRM already provides much of the needed support.
- Linux/KVM reference pattern
- Typical Linux workflows rely on unbind/reattach sequences (unbind framebuffer drivers, detach PCI device, load vfio modules, pass device to VM; reverse on stop).
- This pattern suggests FreeBSD/bhyve may need similar unbind/attach handling to reliably reuse GPUs.
Troubleshooting / Debugging Insights
- When a GPU cannot be re-initialized consider three root causes:
- Hypervisor passthrough code (bhyve) issues.
- Vendor-imposed hardware/firmware limitations.
- Guest OS failing to properly tear down the GPU.
- Studying Linux/KVM unbind/detach/rebind flows can guide what might be missing in FreeBSD drivers or bhyve workflows.
- FreeBSD loader memory tunables can affect loading of large kernel modules; adjust loader settings when loading many/large modules.
Limitations and Open Problems
- Windows guests: could not get Windows 10 guests to boot with a passed GPU in his environment.
- Reusing the same GPU across reboots/VM restarts is often unreliable with NVIDIA hardware.
- Simultaneous sharing of a single discrete GPU across multiple guests is generally unavailable for consumer NVIDIA/AMD cards:
- Intel integrated GVT-g supports multi-vGPU capability.
- Dedicated enterprise vGPU products exist but are expensive.
- Limited Linux testing in his work due to config issues and crashes; Linux behavior may differ.
Practical Recommendations
- Use a dedicated host GPU for console so other cards remain pristine for passthrough.
- Prefer GPUs with open-source driver stacks (AMD, Intel) for easier debugging and more reliable reuse.
- Examine and replicate Linux/KVM unbind/attach sequences when trying to enable GPU reuse on FreeBSD/bhyve.
- Expect trial-and-error with module loading, PCI bus assignments, and guest bootloader (UEFI) choices.
Referenced Projects and Technologies
- bhyve (FreeBSD hypervisor)
- FreeBSD (loader, vt console)
- NVIDIA proprietary driver; CUDA; VirtualGL
- AMD DRM / amdgpu (open-source)
- Intel i915 / GVT-g (virtual GPU)
- Linux KVM and VFIO-style unbind/reattach patterns
- glmark (GL benchmark) for performance testing
Main Speaker / Sources
- Presenter: Michael Chiu
- Implicit references: bhyve documentation/wiki, FreeBSD, Linux/KVM behavior, and NVIDIA/AMD/Intel driver ecosystems.
Category
Technology
Share this summary
Is the summary off?
If you think the summary is inaccurate, you can reprocess it with the latest model.
Preparing reprocess...