Consider using direct I/O

Question

Consider using direct I/O

Closed this issue 5 years ago · 3 comments

Since caching individual trie nodes is inefficient, there may be advantages to bypassing the OS page cache and using our own caches exclusively.

Answer 1 · 2020-07-08T09:44:28.000Z

OS cache works pretty well as long it is configured properly. I.e. F_RDAHEAD is disabled. The upside is that the user does not have to configure the memory budget. The OS can use all available memory for cache.

Also, as mentioned in the README, having no cache is one of the design goals of this library. With blockchain data it pretty much always makes more sense to cache at the higher level. E.g. trie key-values, rather than nodes.

Answer 2 · 2020-07-08T13:32:22.000Z

@arkpar it was those higher-level caches I was referring to, sorry. I was wondering if the OS-level cache was wasteful.

Answer 3 · 2020-07-08T13:54:59.000Z

The OS manages the FS cache size. It will be freed if the memory is required for other applications. And since al read from disk are done to memory pages anyway, it does not cost much for the OS to keep them around.