davisking/dlib

[Bug]: Windows build doesn't set parallel build thread count properly

NeedsMoar opened this issue · 5 comments

NeedsMoar commented

What Operating System(s) are you seeing this problem on?

Windows

dlib version

19.24.2

Python version

3.11.7

Compiler

MSVC 19.39.33523

Expected Behavior

Build should set appropriate parallel thread count

Current Behavior

Build only uses 2 threads on 32/64 core machine with 512GB of ram.

Steps to Reproduce

pip install dlib
observe very long build time

Anything else?

A fix for this should be a quick conditional in the num_available_cpu_cores(ram_per_build_process_in_gb) function in setup.py if the os is Windows, something like:

try: 
   if platform.system() == "Windows":
      import win32.win32api as w32api
      memstatus = w32api.GlobalMemoryStatusEx()
      mem_bytes = memstatus['AvailPhys']
      mem_gib = mem_bytes/(1024.**3)
    else:
      #snip current sysconf stuff
   num_cores = multiprocessing.cpu_count() 
   #snip calculations

The windows functions should be standard with win python installs but I didn't submit a pull request since I haven't tested this inside that file, I just know the given functions to determine memory work.

The AvailPhys field returned doesn't include pagefile or virtual memory so there shouldn't be any chance of hitting either of those.
Right now it fails and returns 2 because os.sysconf isn't defined on Windows so it throws the exception and uses the default value.

davisking commented

Yeah, that's a good idea. Send me a PR that works for you on your windows machine :)

NeedsMoar commented

Sorry, missed this. I'll whip one up for you soonish, shouldn't take long since that's pretty much all of it up there.

  • ❤️1
dlib-issue-bot commented

Warning: this issue has been inactive for 35 days and will be automatically closed on 2024-07-03 if there is no further activity.

If you are waiting for a response but haven't received one it's possible your question is somehow inappropriate. E.g. it is off topic, you didn't follow the issue submission instructions, or your question is easily answerable by reading the FAQ, dlib's official compilation instructions, dlib's API documentation, or a Google search.

dlib-issue-bot commented

Warning: this issue has been inactive for 42 days and will be automatically closed on 2024-07-03 if there is no further activity.

If you are waiting for a response but haven't received one it's possible your question is somehow inappropriate. E.g. it is off topic, you didn't follow the issue submission instructions, or your question is easily answerable by reading the FAQ, dlib's official compilation instructions, dlib's API documentation, or a Google search.

dlib-issue-bot commented

Notice: this issue has been closed because it has been inactive for 45 days. You may reopen this issue if it has been closed in error.