Add per-CPU caches to the pmemrange allocator.
authormpi <mpi@openbsd.org>
Wed, 17 Apr 2024 13:12:58 +0000 (13:12 +0000)
committermpi <mpi@openbsd.org>
Wed, 17 Apr 2024 13:12:58 +0000 (13:12 +0000)
commit52feabc5fb30a7383597a410b90913a34940c81d
treea3351eb02f0df7a1a9918b6988c47f430fa97c2e
parent4ce91cbef917ccbe2f5040f3d676d36e329953a6
Add per-CPU caches to the pmemrange allocator.

The caches are used primarily to reduce contention on uvm_lock_fpageq() during
concurrent page faults.  For the moment only uvm_pagealloc() tries to get a
page from the current CPU's cache.  So on some architectures the caches are
also used by the pmap layer.

Each cache is composed of two magazines, design is borrowed from jeff bonwick
vmem's paper and the implementation is similar to the one of pool_cache from
dlg@.  However there is no depot layer and magazines are refilled directly by
the pmemrange allocator.

Tested by robert@, claudio@ and Laurence Tratt.

ok kettenis@
sys/arch/amd64/include/cpu.h
sys/arch/arm64/include/cpu.h
sys/arch/i386/include/cpu.h
sys/uvm/uvm_page.c
sys/uvm/uvm_pdaemon.c
sys/uvm/uvm_percpu.h [new file with mode: 0644]
sys/uvm/uvm_pmemrange.c
sys/uvm/uvm_pmemrange.h
sys/uvm/uvmexp.h
usr.bin/systat/uvm.c
usr.bin/vmstat/vmstat.c