NVIDIA Unlocks Revolutionary DMA-BUF Support: A Game-Changer for Linux 6.19!
NVIDIA has been hard at work, not only enhancing peer-to-peer (P2P) DMA for block devices in Linux 6.19 but also spearheading a groundbreaking initiative. They've successfully implemented DMA-BUF support for VFIO PCI devices, opening doors to a world of exciting possibilities. This new functionality is now an integral part of Linux 6.19, thanks to the VFIO pull request.
The credit for this achievement goes to NVIDIA's Leon Romanovsky and Jason Gunthorpe, along with Intel's Vivek Kasireddy. Their efforts have brought DMA-BUF support to VFIO PCI devices, as detailed in this week's VFIO pull request. But here's where it gets technical:
'Introducing dma-buf support for vfio-pci devices, enabling MMIO regions to be exposed through dma-buf objects with managed lifecycles. This facilitates low-level interactions, such as vfio-pci-based SPDK drivers directly communicating with dma-buf-capable RDMA devices for seamless peer-to-peer operations.'
The patch series further explains the significance of this development:
'This update empowers the VFIO PCI subsystem to export MMIO regions from PCI device BARs as dma-buf objects, ensuring secure sharing of non-struct page memory with controlled lifetime management. RDMA and other subsystems can now import dma-buf FDs and integrate them into memory regions for PCI P2P operations.'
One practical application is highlighted: 'For SPDK, this means an NVMe device can be controlled by SPDK via VFIO while interacting with an RDMA device. The RDMA device can directly access the NVMe CMB or manipulate the NVMe device's doorbell using PCI P2P.'
But the potential doesn't stop there. This mechanism can accommodate various other scenarios with VFIO. The dma-buf method is versatile, benefiting iommufd for generic and safe P2P mappings.
Moreover, the patch series isn't just about SPDK. It introduces a capability that proves invaluable when a buffer in device memory (like VRAM) needs to be shared between any two dGPU devices or instances, as long as they are P2P DMA compatible.
And this is the part most people miss: the series includes a substantial reorganization of the PCI P2PDMA subsystem, separating core P2P functions from memory allocation features. This modular approach is tailored for VFIO use cases that don't require struct page support.
Linus Torvalds officially merged these VFIO changes into Linux 6.19 on Friday, marking a significant milestone in the Linux kernel's evolution.
