The idea is essentially to have a custom directory that writes immutable segments, this means only 2 file descriptors (one for writing and one for reading) are required per-index, which reduces the amount used in multi-tenant situations.
The writers have more stable performance since they depend less on the file cache to buffer writes and reads, the AIO implementation also leverages asynchronous background writing which makes CPU usage a bit more stable:
-
8 Threads, 2GB RAM, 2GB Swap, 5GB Produced index compressed.
-
Green: AIO Writer
-
Blue: MMapDirectory
Writes are messured in negative MiB/s for some reason, but just ignore the -
These unfinished segments as such can then be exported to a new file as the finished 'segment' as such, with each file re-organised to be contiguous.
