dma_alloc_coherent() and physical addresses

The DMA-API-HOWTO document says that dma_alloc_coherent() returns two values: the virtual address which you can use to access it from the CPU and dma_handle which you pass to the card.

Shouldn't the virtual address and the dma_handle map to the same physical address? That is not what I see.

Is it expected that the dma_handle is the same as its physical address?

I am using a x86_64 system with the IOMMU enabled. The OS is Ubuntu 18.04, and the kernel is 4.15.0-47-generic.

My code:

ret = dma_set_mask_and_coherent(&pdev->dev, DMA_BIT_MASK(64));
dmabuffp = dma_alloc_coherent (dev_ptr, (size_t)4*1024*1024, &dma_hndl, GFP_KERNEL | GFP_DMA);
printk(KERN_INFO "init: dma_hndl = 0x%llX\n", dma_hndl);<br />
printk(KERN_INFO "init: dmabuffp VA = 0x%llX\n", (uint64_t)dmabuffp);<br />
printk(KERN_INFO "init: dma_hndl VA = 0x%llX\n", (uint64_t)bus_to_virt(dma_hndl));
printk(KERN_INFO "init: dmabuffp PA = 0x%010llX\n", (uint64_t)virt_to_phys(dmabuffp));<br />
printk(KERN_INFO "init: dma_hndl PA = 0x%010llX\n",(uint64_t)virt_to_phys(bus_to_virt(dma_hndl)));

After the module loads, dmesg shows this:

init: dma_hndl = 0xFFC00000<br />
init: dmabuffp VA = 0xFFFF888E9B000000<br />
init: dma_hndl VA = 0xFFFF88873FC00000<br />
init: dmabuffp PA = 0x085B000000<br />
init: dma_hndl PA = 0x00FFC00000

The device can write and read using dma_hndl, but the value it writes is not at *buffptr.

What am I missing and/or doing wrong?

2 Answers

Additional testing shows that dma_hndl does map to the same physical address as dmabuffp. Apparently virt_to_phys(bus_to_virt(dma_hndl)) does not resolve as expected and instead is circular, returning the original dma_hndl value instead of the physical address that is mapped by the IOMMU.

The dma_addr_t is an I/O virtual address (IOVA), it's the address the device uses to access the buffer returned by dma_alloc_coherent.

void * dma_alloc_coherent(struct device *dev, size_t size, dma_addr_t *dma_handle, gfp_t flag)

The reason dma_addr_t is not same as virt_to_phys(dmabuffp) is because IOMMU is enabled.

Only the device uses the dma_addr_t for DMA access. The processor accesses need to use the returned void*, or some sort of virt_to_phys() version of that to allow userspace to mmap it through devmem.

Without an IOMMU, the dma_addr_t is simply a virt_to_phy()/virt_to_bus() translation of the void* buffer, so it can be used for processor access directly but it is incorrect usage of the DMA API.