Introduce an IOVA allocator instead of mapping pages 1:1. Mapping pages 1:1
obviously reduces the overhead of IOVA allocation, but instead you have the
problem of doubly mapped pages, and making sure a page is only unmapped once
the last user is gone. My initial attempt, modeled after apldart(4), calls
the allocator for each segment. Unfortunately this introduces a performance
penalty which reduces performance from around 700 Mbit/s to about 20 Mbit/s,
or even less, in a simple single stream tcpbench scenario. Most mbufs from
userland seem to have at least 3 segments. Calculating the needed IOVA space
upfront reduces this penalty. IOVA allocation overhead could be reduced once
and for all if it is possible to reserve IOVA during bus_dmamap_create(9), as
it is only called upon creation and basically never for each DMA cycle. This
needs some more thought.
With this we now put the pressure on the PTED pools instead. Additionally, but
not part of this diff, percpu pools for the PTEDs seem to reduce the overhead
for that single stream tcpbench scenario to 0.3%. Right now this means we're
hitting a different bottleneck, not related to the IOMMU. The next bottleneck
will be discovered once forwarding is unlocked. Though it should be possible
to benchmark the current implementation, and different designs, using a cycles
counter.
With IOVA allocation it's not easily possible to correlate memory passed to
bus_dmamem_map(9) with memory passed to bus_dmamap_load(9). So far my code
try to use the same cachability attributes as the kenrel uses for its userland
mappings. For the devices we support, there seems to be no need so far. If
this ever gives us any trouble in the feature, I'll have a look and fix it.
While drivers should call bus_dmamap_unload(9) before bus_dmamap_destroy(9),
the API explicitly states that bus_dmamap_destroy(9) should unload the map
if it is still loaded. Hence we need to do exactly that. I actually have
found one network driver which behaves that way, and the developer intends
to change the network driver's behaviour.