Panic When Trying to Insert Many Keys into Vaults

[HenriqueNogara]

Description

The code attached to this issue causes the following error:

thread '<unnamed>' panicked at 'Error setting memory protection to NoAccess', /home/henrique/.cargo/git/checkouts/stronghold.rs-fe28308a6634692d/aea8a9d/engine/runtime/src/boxed.rs:289:9

Here is my OS info:

OS: Ubuntu 20.04.3 LTS
OS Type: 64 bit
Gnome Version: 3.36.8
Windowing System: X11

The error is always after 8173 keys

code: https://gist.github.com/HenriqueNogara/f2caa968abf03ef860ead62d31a20cea

[felsweg-iota]

system configuration:

OS: Ubuntu 20.04.3 LTS
OS Type: 64 bit
Gnome Version: 3.36.8
Windowing System: X11

This seems to be a system's restriction. libsodium explicitly states, that some systems enforce a limit on possible memory protections:

Note: Many systems place limits on the amount of memory that may be locked by a process. Care should be taken to raise those limits (e.g. Unix ulimits) where neccessary. sodium_mlock() will return -1 when any limit is reached.

Taking a deep dive into the memory protection API concerning linux, non-priviledged processes are constraint in number of protected memory regions. I could successfully recreate the error on my system (same config).
Since libsodium matches the produced error stored in errno, I could verify that return errno is ENOMEN, that is described as

the caller had a nonzero RLIMIT_MEMLOCK soft resource limit, but tried to lock more memory than the limit permitted.

Here is a minimal variation of the test case mentioned above

use engine::vault::RecordHint;
use stronghold_utils::random as rand;

#[actix::test]
async fn test_maximum_vault_entries() {
    let stronghold = Stronghold::init_stronghold_system(b"client_path".to_vec(), vec![])
        .await
        .unwrap();

    for i in 0..usize::MAX {
        let location = Location::Generic {
            vault_path: rand::bytestring(32),
            record_path: rand::bytestring(32),
        };

        let hint = RecordHint::new(rand::bytestring(24)).unwrap();
        let payload = rand::bytestring(56);

        if let Err(error) = stronghold.write_to_vault(location, payload, hint, vec![]).await {
            panic!("Failed writing vault entry #{}, failed with error: {}", i, error);
        }
    }
}

This code example just writes random bytes into the vault at random locations.

And here is modification of the source creating the error, to verify the returned error

use errno::errno; // errno = "0.2.8"
use libc::{EAGAIN, EINVAL, ENOMEM, EPERM}; // libc = "*"

fn mprotect<T>(ptr: *mut T, prot: Prot) {
    if !match prot {
        Prot::NoAccess => unsafe {
            let error = sodium_mprotect_noaccess(ptr as *mut _);

            if error != 0 {
                let errno_t = errno();

                let _a = ENOMEM;
                let _b = EAGAIN;
                let _c = EINVAL;
                let _d = EPERM;

                match errno_t.0 {
                    ENOMEM => {
                        //  the caller had a nonzero RLIMIT_MEMLOCK soft resource limit, but tried to lock more memory than the limit permitted.
                        let _n = errno_t;
                        0;
                    }
                    EAGAIN => {
                        // Some or all of the specified address range could not be locked.
                        let _n = errno_t;
                        1;
                    }
                    EINVAL => {
                        // (Not on Linux) addr was not a multiple of the page size.
                        let _n = errno_t;
                        2;
                    }
                    EPERM => {
                        // The caller is not privileged, but needs privilege (CAP_IPC_LOCK) to perform the requested operation.
                        let _n = errno_t;
                        3;
                    }
                    _ => {}
                };

                false;
            }
            true
        },
        Prot::ReadOnly => unsafe { sodium_mprotect_readonly(ptr as *mut _) == 0 },
        Prot::ReadWrite => unsafe { sodium_mprotect_readwrite(ptr as *mut _) == 0 },
    } {
        panic!("Error setting memory protection to {:?}", prot);
    }
}

Note: The (underlying) system call returns 0 in case of success, or -1 if there was a failure. errno will be filled with the appropriate error.

Conclusion

There are some opportunities to counter the restriction:

• running the implementing application in privileged mode. this is DEFINITELY NOT ADVISABLE
• increasing the soft memory allocation limitation. This might be possible, but is currently out of scope, as Stronghold relies strongly on sensible defaults given by the operating system. We might consider lifting this restriction.

[elenaf9]

The problem here is actually not the amount of keys that are inserted, but instead the amount of different vaults that are used. A bit of background knowledge:
The libsodium memory protection is not used for every record in a vault. Instead, it is used to protect a key k for each vault. This key k in return is then used to encrypt every record in the vault in memory. Only when the record is used, it is decrypted with k and then protected by the libsodium memory protection API. After it was used it re-encrypted with k again.

You can test this by changing your example to only write to a limited amount of different vaults, which should then not panic:

match vault
      .insert(
          Location::generic((i % 100).to_string(), i.to_string()), // only use 100 different vauls
          private_key.as_ref(),
          default_hint(),
          &[],
      )
      .await
  {..}

I don't know the details about you specific use-case, but would a sufficient solution be to only use a limited number of different vaults?

[PhilippGackstatter]

@HenriqueNogara has confirmed this works.

We currently use a client per identity, but a different vault per key. However, this is planned to be changed with the upcoming refactor of the interface of stronghold itself, which will also affect us.

Overall, having a limit of ~8000 identities per stronghold (in-memory at the same time, I assume) seems okay. One can always open another stronghold, perhaps unload clients, or change their OS settings, which is acceptable, I think.

[felsweg-iota]

With the new runtime system this problem should now appear only on edge cases.