Read on Twitter
Are memory-mapped files (mmap) faster than system call-based I/O? A thread. (1/11)
The blog post “Why mmap is faster than system calls” by Alexandra Fedorova (2019) argues that “[…] barring few exceptions, mmap is 2–6 times faster than system calls.” ( https://sasha-f.medium.com/why-mmap-is-faster-than-system-calls-24718e75ab37) (2/11)
The author shows that mmap() is more efficient because it eliminates the protection domain crossing from system calls and — more importantly — kernel/userspace data copies that I/O system calls such as read() imply. (3/11)
So should you be converting your application to use mmap? Not before you consider the alternatives. (4/11)
Linux more I/O access than just traditional read()/write() and mmap(). There’s direct I/O (DIO) and asynchronous direct I/O (AIO/DIO), which are covered by @avikivity’s excellent blog post comparing the I/O access methods (2017): https://www.scylladb.com/2017/10/05/io-access-methods-scylla/ (5/11)
The DIO and AIO/DIO interfaces use system calls, but they let callers specify a buffer for read/write operation, eliminating data copies. The system calls incur some copying, but there's an io_uring interface by @axboe to address them ( https://kernel.dk/io_uring.pdf ). (6/11)
So what are the trade-offs between the different I/O access methods? (7/11)
Traditional I/O and mmap() let the kernel control caching (via the page cache), whereas DIO and AIO/DIO give caching control to the userspace. Depending on your application, controlling caching can be critical for performance (for example, in database systems). (8/11)
AIO/DIO (and io_uring) are asynchronous interfaces, which allows I/O to be performed in the background. In contrast, mmap() is a blocking interface (when page fault occurs), which forces userspace to maintain a thread pool, which can be inefficient. (9/11)
On the flip side, the AIO/DIO and io_uring interfaces are more complex to program (because userspace needs to manage much more). There are also some other differences, and I encourage everyone to read @avikivity’s blog post. (10/11)
In summary, mmap() is an excellent I/O access interface if letting the kernel control caching is not an issue, and you can mitigate the occasional blocking from page faults. However, if you are looking for control and performance, AIO/DIO and io_uring are the way to go. (11/11)
For those that are new to Linux's AIO/DIO, @littledan's tutorial is an excellent starting point: https://github.com/littledan/linux-aio
