NixOS/nix

Large files within source tree causes build to run out of memory

gulrotkake opened this issue · 18 comments

gulrotkake commented

If the size of a file or a sufficient number of files in a nix-build exceeds the amount of memory available, Nix runs out of memory and gives up. For example, given this simple build:

{ stdenv, pkgs, ... } :

stdenv.mkDerivation rec {
  name = "huge";
  src = ./.;
  buildInputs = [ ];
  buildPhase =
    ''
        md5sum huge.bin > md5.result
    '';

  installPhase =
   ''
     mkdir $out
     mv md5.result $out
  '';
}

where huge.bin is a very huge file, nix will first warn:

warning: dumping very large path (> 256 MiB); this may run out of memory

before eventually running out of memory.

The above was tested with

nix-build (Nix) 1.8pre3834_9b146a5
edolstra commented

As a workaround, you could use fetchurl to get the file, e.g. fetchurl { src = file:///bla/huge; sha256 = ...; }.

shlevy commented

Bonus feature: hole detection to avoid spurious reads.

edolstra commented

I don't think there are many realistic use cases where a Nix source file contains holes...

domenkozar commented

It would be nice if it printed out the file name

  • 👍12
sjmackenzie commented

Agreed, the file name would be most handy.

sjmackenzie commented

We have 50 instances of src = ./. each time the ./. points to the root of a rust project which consists of 4 files, one of which is in a folder called src. We have no instances of a large file/binary called with ./., just many (and growing quickly) instances of (./.).

Ericson2314 commented

Why does it even read the files into memory at once? I assume the hash can be computed with constant space.

  • 👍2
sjmackenzie commented

For the record, we found the cause: it was a .target folder generated by the rustc compiler. Deleting that folder straightened nix again.

ip1981 commented

As a workaround, you could use fetchurl to get the file, e.g. fetchurl { src = file:///bla/huge; sha256 = ...; }.

There might be permission problem.

fisx commented

Until this is fixed in Nix see https://nixos.org/wiki/How_to_add_files_to_the_nix-store#Large_files

dead link, and it looks like this page does not exist on https://nixos.wiki/?

dpc commented

Is there any way to make src = ./. except for ./target;?

  • 👍1
sjmackenzie commented

take a look at cleanSource and filterSource

dpc commented 
src = builtins.filterSource (p: t: lib.cleanSourceFilter p t && baseNameOf p != "target") ./.;

I think this is it, in case anyone lands here from google search.

  • 👍6
  • 😄1
  • 🎉2
  • ❤️3
langston-barrett commented

I am also experiencing this problem while trying to install Mathematica.

Here's the old Wiki page on archive.org in case anyone else is searching for it: https://web.archive.org/web/20160829175307/https://nixos.org/wiki/How_to_add_files_to_the_nix-store

  • 👍1
bgamari commented

This is addressed by #619.

  • 👍1
Pitometsu commented

Looks as a solution for me: https://stackoverflow.com/a/39493330/3195266

  • 👍1