`apache-flink = "^1.19.1"` does not work on MacOS
Opened this issue · 1 comments
usmcamp0811 commented
Describe the issue
I have a Nix Flink package that works 100% correctly on all things Linux. One of its dependencies is a Python environment with apache-flink
. When my buddy who has a Mac runs the package he gets the attached error..
Skimming through the error it looks like pyarrow
might be the problem which I believe is a dep of apache-beam
. But any way I pulled out the Python environment and made a colorful package to attempt to isolate the problem.
# has `apache-flink` no work on Mac
nix run gitlab:usmcamp0811/dotfiles/5fb41f9a60c833656996e4853ebfca3d4d17fc10#macossucksballs
# no `apache-flink` works on Mac
nix run gitlab:usmcamp0811/dotfiles/1c47084208caa0a31ba472e5941a125831ff24e1#macossucksballs
# no `apache-flink` but has `pyarrow` no work on Mac
nix run --refresh gitlab:usmcamp0811/dotfiles/5fb41f9a60c833656996e4853ebfca3d4d17fc10#macossucksballs
It seems to be a problem with PyArrow actually. I got him to run
nix run --refresh gitlab:usmcamp0811/dotfiles/5fb41f9a60c833656996e4853ebfca3d4d17fc10#macossucksballs
error:
… while calling the 'derivationStrict' builtin
at /derivation-internal.nix:9:12:
8|
9| strict = derivationStrict drvAttrs;
| ^
10|
… while evaluating derivation 'python3-3.11.9-env'
whose name attribute is located at /nix/store/sj9yrq21wbbfr5715hys3laa2qd6x471-source/pkgs/stdenv/generic/make-derivation.nix:333:7
… while evaluating attribute 'passAsFile' of derivation 'python3-3.11.9-env'
at /nix/store/sj9yrq21wbbfr5715hys3laa2qd6x471-source/pkgs/build-support/trivial-builders/default.nix:69:9:
68| inherit buildCommand name;
69| passAsFile = [ "buildCommand" ]
| ^
70| ++ (derivationArgs.passAsFile or [ ]);
error: Missing suitable source/wheel file entry for pyarrow
Additional context
default.nix
/shell.nix
/flake.nix
:
{ lib, inputs, pkgs, ... }:
let
src = ./.;
pypkgs-build-requirements = {
avro = [ "setuptools" ];
avro-python3 =
[ "setuptools" "python-snappy" "zstandard" "isort" "pycodestyle" ];
apache-flink = [ "setuptools" "pyarrow" ];
mocker = [ "setuptools" ];
apache-flink-libraries = [ "setuptools" ];
};
p2n-overrides = pkgs.poetry2nix.defaultPoetryOverrides.extend (self: super:
builtins.mapAttrs (package: build-requirements:
super."${package}".overridePythonAttrs (oldAttrs: {
buildInputs = (oldAttrs.buildInputs or [ ])
++ (builtins.map (req: super."${req}") build-requirements);
})) pypkgs-build-requirements);
python-env = pkgs.poetry2nix.mkPoetryEnv {
projectDir = src;
python = pkgs.python311;
overrides = p2n-overrides;
preferWheels = true; # Prefer wheels to speed up the build process
};
in python-env
pyproject.toml
:
[tool.poetry]
name = "example-flink-job"
version = "0.1.0"
description = ""
authors = ["Matt Camp <matt@aicampground.com>"]
readme = "README.md"
[tool.poetry.dependencies]
python = "^3.11"
setuptools = "^65.5.0" # Add this line
pyarrow = "^17.0.0"
[tool.poetry.group.dev.dependencies]
pytest-mock = "^3.14.0"
pytest = "^8.1.1"
[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
usmcamp0811 commented
I think this is related to #1724