use `ulimit` syscall to read the value and then set it automatically depending on what is available 👍

[bee-san]

I think the best way would be to use the ulimit syscall to read the value and then set it automatically depending on what is available 👍

And then, notify the user about this "The current FD limit is too small, increase is blah blah".

Originally posted by @SakiiR in #21 (comment)

[issue-label-bot]

Issue-Label Bot is automatically applying the label feature_request to this issue, with a confidence of 0.91. Please mark this comment with 👍 or 👎 to give our bot feedback!

Links: app homepage, dashboard and code for this bot.

[SakiiR]

Here is the documentation:

https://www.man7.org/linux/man-pages/man2/getrlimit.2.html

I am trying to get a working rust code compiling on both Linux and OSx

[SakiiR]

Here it is:

use rlimit::Resource;
use rlimit::getrlimit;

fn main() {
    let (x, _) = getrlimit(Resource::NOFILE).unwrap();

    println!("File descriptor limit: {}", x);
}

Don't forget to import the crate ;)

 [dependencies]

 rlimit = "0.3.0"

[SakiiR]

Also !

There is a typo in the option name:

The default is 5000. Try -B 2500 ( it's -b not -B)

[bee-san]

Wow, awesome! I'm working on getting Homebrew on Mac to work :P I'll quickly add this too and try to update everything. The plan at the moment is to kind of cheat, and make it so Homebrew / AUR installs Rust and then cargo install rustscan since I can update cargo very easily (at least until I get CI that can automatically do this for me), but Homebrew doesn't seem to like that 😝

@all-contributors please add SakiiR for code, bugs

[allcontributors]

@brandonskerritt

I could not determine your intention.

Basic usage: @all-contributors please add @Someone for code, doc and infra

For other usages see the documentation

[bee-san]

🙄
@all-contributors please add @SakiiR for code and bugs

[allcontributors]

@brandonskerritt

I've put up a pull request to add @SakiiR! 🎉

[bee-san]

Give a flag that automatically ups the ulimit (so the user can say -u 6000) to auto-update the ulimit (which also updates the batch size). This way, users may find the "perfect" ulimit and make an alias for it.

[bee-san]

The code for this is written, just need to test it :)
See branch 1.2 for the code

[bee-san]

For reference, this is how it works in 1.2

    if !(ulimit_arg == "None") {
        let limit = ulimit_arg.parse::<u64>().unwrap();
        if !quiet{
            println!("Automatically upping ulimit to {}", ulimit_arg);
        }
        setrlimit(Resource::NOFILE, limit, limit);
    }

    let (x, _) = getrlimit(Resource::NOFILE).unwrap(); 

    // if maximum limit is lower than batch size
    // automatically re-adjust the batch size
    if x < batch_size.into() {
        if !quiet{
            println!("{}", "WARNING: Your file description limit is lower than selected batch size. Please considering upping this (how to is on the README). NOTE: this may be dangerous and may cause harm to sensitive servers. Automatically reducing Batch Size to match your limit, this process isn't harmful but reduces speed.".red());

            // if the OS supports high file limits like 8000
            // but the user selected a batch size higher than this
            // reduce to a lower number
            // basically, ubuntu is 8000 
            // but i can only get it to work on < 5k in testing
            // 5k is default, so 3000 seems safe
            if x > 8000{
                batch_size = 3000
            }
            else {
                // TODO this is a joke please fix I can't get type hinting to work and it won't compile
                let ten: u64 = 100;
                batch_size = x - ten;
            }
        }
    }
    // else if the ulimit is higher than batch size
    // tell the user they can increase batch size
    // if the user set ulimit arg they probably know what they are doing so don't print this
    else if x + 2 > batch_size.into() && (ulimit_arg == "None"){
        if !quiet{
            // TODO this is a joke please fix I can't get type hinting to work and it won't compile
            let one: u64 = 1;
            println!("Your file description limit is higher than the batch size. You can potentially increase the speed by increasing the batch size, but this may cause harm to sensitive servers. Your limit is {}, try batch size {}.", x, x - one);
        }
    }```

**TL;DR**

1. If the user set the ulimit arg (to auto up the arg) it will do that
2. If the file descriptor limit (ulimit) is lower than the batch size, reduce batch size. This algorithm is used:

if ulimt > 8000:
batch_size = 3000

This is because on Ubuntu, batch size is 8800 -- however, I have yet to be able to get this to work on Ubuntu with a batch size over 5k. I feel like for most systems, 3k would be a nice relatively safe limit while maintaining speed.

else
new batch size = batch size - 100

3. if the ulimit is higher than the batch size, tell the user they can potentially increase batch size to gain more speed.

The default batch size is still 4500.