Regression: poetry2nix can not be used in flake.nix to pull packages from private GitLab registries

mkPoetryApplication fails to build a package that depends on a private dependency (error message below),
when used inside a flake. It keeps on erroring with HTTP 401. It seems be caused by a recent switch
to fetchPypiLegacy from upstream nixpkgs instead of a vendored one (details below).

I have tried each of the following, in combination and independently.

Added ENVRC=<path/to/netrc> in my systemd config. I am not running NixOS, so I did this
through an override and I have confirmed that the daemon process did have the env variable
by inspecting running strings /proc/<proc-id>/environ, after I reloaded the daemon
and restarted the service.
Added extra-sandbox-paths = <path/to/netrc> in /etc/nix/nix.conf
Added netrc-file = <path/to/netrc> in /etc/nix/nix.conf

None of the above has worked. This was on d11c01e (tip of master currently).

In all cases, I can also confirm that <path/to/netrc> is accessible and readable by
the user group of the nix builders. My local user is also part of trusted-users. Finally,
I can also confirm the netrc file is correct, as it can be used with poetry (just poetry)
to pull the exact same private packages.

After a little investigation, I found that this seems to be a regression caused by a recent switch
to fetchPypiLegacy from upstream nixpkgs instead of a vendored one.

I then switched to 54083f9 (the last commit prior to the switch in 7f304a8), ran my build in flake again,
and it worked: I can pull the private package. When I did this, I also did all three
modifications I listed above, though I suspect the netrc-file was unecessary.

Delving deeper to find the root cause, it seems to be this the fact that nixpkgs
does not pass NETRC as an item in impureEnvVars.
This is different from poetry2nix, where there was a specific change that allows
NETRC to be passed in pure eval mode, the
one that flake uses (but not nix-build).

If this is indeed the root cause, it means that poetry2nix can only pull private
packages when not used inside a flake. As seen in both fetchFromPypiLegacy implementation,
NETRC can also be injected if present in builtins.nixPath. Sadly, adding to that
list is only possible when one uses nix-build and not flakes.

P.S. I feel like it's not 100% clear if this issue should be here or in nixpkgs, so feel
free to close this if you feel it's more appropriate there.

Additional context

Here is a redacted snippet of the error (<hash> may be different in each line):

error: builder for '/nix/store/<hash>-<pkg>-<version>.tar.gz.drv' failed with exit code 1;
       last 25 log lines:
       > calling 'envBuildTargetHook' function hook 'addPythonPath' /nix/store/<hash>-make-symlinks-relative.sh
       > calling 'envBuildTargetHook' function hook 'sysconfigdataHook' /nix/store/<hash>-make-symlinks-relative.sh
       > calling 'envBuildTargetHook' function hook 'addPythonPath' /nix/store/<hash>-move-docs.sh
       > calling 'envBuildTargetHook' function hook 'sysconfigdataHook' /nix/store/<hash>-move-docs.sh
       > calling 'envBuildTargetHook' function hook 'addPythonPath' /nix/store/<hash>-move-lib64.sh
       > calling 'envBuildTargetHook' function hook 'sysconfigdataHook' /nix/store/<hash>-move-lib64.sh
       > calling 'envBuildTargetHook' function hook 'addPythonPath' /nix/store/<hash>-move-sbin.sh
       > calling 'envBuildTargetHook' function hook 'sysconfigdataHook' /nix/store/<hash>-move-sbin.sh
       > calling 'envBuildTargetHook' function hook 'addPythonPath' /nix/store/<hash>-move-systemd-user-units.sh
       > calling 'envBuildTargetHook' function hook 'sysconfigdataHook' /nix/store/<hash>-move-systemd-user-units.sh
       > calling 'envBuildTargetHook' function hook 'addPythonPath' /nix/store/<hash>-multiple-outputs.sh
       > calling 'envBuildTargetHook' function hook 'sysconfigdataHook' /nix/store/<hash>-multiple-outputs.sh
       > calling 'envBuildTargetHook' function hook 'addPythonPath' /nix/store/<hash>-patch-shebangs.sh
       > calling 'envBuildTargetHook' function hook 'sysconfigdataHook' /nix/store/<hash>-patch-shebangs.sh
       > calling 'envBuildTargetHook' function hook 'addPythonPath' /nix/store/<hash>-prune-libtool-files.sh
       > calling 'envBuildTargetHook' function hook 'sysconfigdataHook' /nix/store/<hash>-prune-libtool-files.sh
       > calling 'envBuildTargetHook' function hook 'addPythonPath' /nix/store/<hash>-reproducible-builds.sh
       > calling 'envBuildTargetHook' function hook 'sysconfigdataHook' /nix/store/<hash>-reproducible-builds.sh
       > calling 'envBuildTargetHook' function hook 'addPythonPath' /nix/store/<hash>-set-source-date-epoch-to-latest.sh
       > calling 'envBuildTargetHook' function hook 'sysconfigdataHook' /nix/store/<hash>-set-source-date-epoch-to-latest.sh
       > calling 'envBuildTargetHook' function hook 'addPythonPath' /nix/store/<hash>-strip.sh
       > calling 'envBuildTargetHook' function hook 'sysconfigdataHook' /nix/store/<hash>-strip.sh
       > Reading index <private-pkg-url>
       > Got exception' HTTP Error 401: Unauthorized ', trying next package index
       > Could not fetch package '<private-pkg>' file '<private-pkg-source-tarball>' from any mirrors: ['<private-package-url>']

The command I ran was nix flake build .#local. And the relevant snippet of my flake looks like this:

{
  description = "Nix flake for local project";

  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
    flake-utils.url = "github:numtide/flake-utils";
    poetry2nix = {
      url = "github:nix-community/poetry2nix";
      inputs.nixpkgs.follows = "nixpkgs";
      inputs.flake-utils.follows = "flake-utils";
    };
  };

  outputs =
    {
      self,
      nixpkgs,
      flake-utils,
      poetry2nix,
    }:
    flake-utils.lib.eachDefaultSystem (
      system:
      let
        pkgs = import nixpkgs {
          inherit system;
          overlays = [ poetry2nix.overlays.default ];
        };
        overrides = pkgs.poetry2nix.overrides.withDefaults (
          _final: prev: { mypy = prev.mypy.override { preferWheel = true; }; }
        );
        projectDir = self;
        python = pkgs.python311; # NOTE: Keep in-sync with pyproject.toml.
        app = pkgs.poetry2nix.mkPoetryApplication { inherit overrides projectDir python; };
      in
      {
        packages =
          {
            local = app;
          };
      }
    );
}

Also experiencing this - see #1740

Quoting myself from #1740 (comment):

I believe NixOS/nixpkgs#333679 is the best solution.
We want the improved security from TLS when transferring credentials, but when we're not we want long-term reproducibility.

Thanks for getting back on this @adisbladis.

I am not quite sure how adding the cacert dependency affects this particular issue though (for the record, I did not encounter the TLS issue).

In the fetchPypiLegacy.py file in nixpkgs, NETRC is passed through netrc_file. And as far as I understand, netrc_file is nonempty only if there is a NETRC entry in the nix search path, something not possible in flakes.

In other words, when using flakes, NETRC will never be passed.

Right, sorry for the misunderstanding. Given the previous reference I assumed that this was the same issue (I hadn't yet given the issue body a proper read).

No worries ~ and please do let me know if I can help with anything 🙂.

This is different from poetry2nix, where there was a specific change that allows
NETRC to be passed in pure eval mode, the

When migrating this code to nixpkgs this was missed because it shouldn't have been changed and merged into the vendor directory in this repository. This is updated from https://github.com/nix-community/pyproject.nix and I don't check the commit logs of the vendor directory in between updates.

I wasn't even aware that this change had been made.

Can you test NixOS/nixpkgs#334059 and see if it fixes things for you?

I just tested NixOS/nixpkgs#334059. It still didn't go all the way through, but this time it progressed further 👍.

The error is in fact what is described in #1740. So now I can reproduce that:

<omitted-stack-trace>
...
       > urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1000)>

The TLS certificate for the machine that hosts the package is indeed self-signed, so I am not 100% sure NixOS/nixpkgs#333679 would fix it.

In my non-Nix workflow, I have added the self-signed certificate to my local system trust. And I can see now why pinning to 54083f9 worked for me without the SSL error: SSL verification was also disabled there.

Coming to your earlier comment (which now makes more sense), given that cacert is now added as a dependency that works for public CAs, can a similar change be made for private, self-signed CAs?

The TLS certificate for the machine that hosts the package is indeed self-signed, so I am not 100% sure NixOS/nixpkgs#333679 would fix it.

Ahh! We'd have to try with both NixOS/nixpkgs#334059 & NixOS/nixpkgs#333679 combined to cover your full use case.
I have pushed a combination of these PRs to https://github.com/adisbladis/nixpkgs/tree/poetry2nix-private-pypi-regression for testing.

You can create a custom cacert package using:

cacert.override { 
  extraCertificateFiles = [ ./path/to/cert.pem ];  # Add extra certs to the store
  extraCertificateStrings = [ ];  # Same as above, but using inline strings
}

You'd use it like this:

overrides = pkgs.poetry2nix.overrides.withDefaults (
  _final: prev: { 
    custom-pkg = custom-pkg.overridePythonAttrs(oldAttrs: {
      src = oldAttrs.src.override {
        cacert = cacert.override {
          extraCertificateStrings = [ ];
        };
      };
    });
  }
);

@adisbladis That did the trick 👍. I can now pull private dependencies that are hosted on a server whose SSL certificate is self-signed 🎉 ~ thank you!

I did that with a minor change: instead of cacert = pkgs.cacert.override ... it's cacert = cacert.override .... Otherwise, it's as you've shown.

Nixpkgs PRs have been merged & backported to 24.05.