`yaml2attrs`/`fromYAML` results in IFD causing `nix flake check` to fail
Closed this issue · 6 comments
montchr commented
This particular chain of functions in yaml2attrs leads to IFD, as it forces the derivation to be evaluated before anything else can continue because we need to builtins.readFile from the resulting file.
Lines 108 to 113 in 6b404cd
Because stdenv is system-specific, that chain of requirements leads to requiring cross-compilation in order to simply run nix flake check.
It does not appear to be possible to run nix flake check on a flake containing both aarch64-darwin and x86_64-linux host configurations.
$ nf check --show-trace
trace: namaka={"dir":"tests","results":{"ops__keys":true,"ops__metadata":true}}
warning: unknown flake output '__std'
Failed to find a machine for remote build!
derivation: lm95hkshyfvyisjcxs17z4kx5mrcc3md-fromYAML.drv
required (system, features): (x86_64-linux, [])
1 available machines:
(systems, maxjobs, supportedFeatures, mandatoryFeatures)
([aarch64-linux], 4, [kvm], [])
error:
… while checking flake output 'nixosConfigurations'
at «none»:0: (source not available)
… while checking the NixOS configuration 'nixosConfigurations.freundix'
at «none»:0: (source not available)
… while calling the 'seq' builtin
at /nix/store/38rzi6kzh5ilbpdgpvd8aj34vksgkcfa-source/lib/modules.nix:326:18:
325| options = checked options;
326| config = checked (removeAttrs config [ "_module" ]);
| ^
327| _module = checked (config._module);
… while evaluating a branch condition
at /nix/store/38rzi6kzh5ilbpdgpvd8aj34vksgkcfa-source/lib/modules.nix:267:9:
266| checkUnmatched =
267| if config._module.check && config._module.freeformType == null && merged.unmatchedDefns != [] then
| ^
268| let
… in the right operand of the AND (&&) operator
at /nix/store/38rzi6kzh5ilbpdgpvd8aj34vksgkcfa-source/lib/modules.nix:267:72:
266| checkUnmatched =
267| if config._module.check && config._module.freeformType == null && merged.unmatchedDefns != [] then
| ^
268| let
… while evaluating the attribute 'unmatchedDefns'
at /nix/store/38rzi6kzh5ilbpdgpvd8aj34vksgkcfa-source/lib/modules.nix:671:7:
670| # Transforms unmatchedDefnsByName into a list of definitions
671| unmatchedDefns =
| ^
672| if configs == []
… while calling the 'concatLists' builtin
at /nix/store/38rzi6kzh5ilbpdgpvd8aj34vksgkcfa-source/lib/modules.nix:678:11:
677| else
678| concatLists (mapAttrsToList (name: defs:
| ^
679| map (def: def // {
… while calling anonymous lambda
at /nix/store/38rzi6kzh5ilbpdgpvd8aj34vksgkcfa-source/lib/attrsets.nix:539:10:
538| attrs:
539| map (name: f name attrs.${name}) (attrNames attrs);
| ^
540|
… from call site
at /nix/store/38rzi6kzh5ilbpdgpvd8aj34vksgkcfa-source/lib/attrsets.nix:539:16:
538| attrs:
539| map (name: f name attrs.${name}) (attrNames attrs);
| ^
540|
… while calling anonymous lambda
at /nix/store/38rzi6kzh5ilbpdgpvd8aj34vksgkcfa-source/lib/modules.nix:678:46:
677| else
678| concatLists (mapAttrsToList (name: defs:
| ^
679| map (def: def // {
… while calling anonymous lambda
at /nix/store/38rzi6kzh5ilbpdgpvd8aj34vksgkcfa-source/lib/modules.nix:664:22:
663| # Propagate all unmatched definitions from nested option sets
664| mapAttrs (n: v: v.unmatchedDefns) resultsByName
| ^
665| # Plus the definitions for the current prefix that don't have a matching option
… while calling anonymous lambda
at /nix/store/38rzi6kzh5ilbpdgpvd8aj34vksgkcfa-source/lib/attrsets.nix:539:10:
538| attrs:
539| map (name: f name attrs.${name}) (attrNames attrs);
| ^
540|
… from call site
at /nix/store/38rzi6kzh5ilbpdgpvd8aj34vksgkcfa-source/lib/attrsets.nix:539:16:
538| attrs:
539| map (name: f name attrs.${name}) (attrNames attrs);
| ^
540|
… while calling anonymous lambda
at /nix/store/38rzi6kzh5ilbpdgpvd8aj34vksgkcfa-source/lib/modules.nix:678:46:
677| else
678| concatLists (mapAttrsToList (name: defs:
| ^
679| map (def: def // {
… while calling anonymous lambda
at /nix/store/38rzi6kzh5ilbpdgpvd8aj34vksgkcfa-source/lib/modules.nix:664:22:
663| # Propagate all unmatched definitions from nested option sets
664| mapAttrs (n: v: v.unmatchedDefns) resultsByName
| ^
665| # Plus the definitions for the current prefix that don't have a matching option
… while calling anonymous lambda
at /nix/store/38rzi6kzh5ilbpdgpvd8aj34vksgkcfa-source/lib/modules.nix:594:46:
593| # an attrset 'name' => list of submodules that define ‘name’.
594| defnsByName = byName "config" (module: value:
| ^
595| map (config: { inherit (module) file; inherit config; }) (pushDownProperties value)
… from call site
at /nix/store/38rzi6kzh5ilbpdgpvd8aj34vksgkcfa-source/lib/modules.nix:595:69:
594| defnsByName = byName "config" (module: value:
595| map (config: { inherit (module) file; inherit config; }) (pushDownProperties value)
| ^
596| ) configs;
… while calling 'pushDownProperties'
at /nix/store/38rzi6kzh5ilbpdgpvd8aj34vksgkcfa-source/lib/modules.nix:826:24:
825| */
826| pushDownProperties = cfg:
| ^
827| if cfg._type or "" == "merge" then
… from call site
at /nix/store/fxgkj2xxcz9pn5khyq9f0ddyfpiw854y-source/stylix/palette.nix:128:26:
127| # https://github.com/SenchoPens/base16.nix#mktheme
128| lib.stylix.colors = (base16.mkSchemeAttrs cfg.base16Scheme).override override;
| ^
129| lib.stylix.scheme = base16.mkSchemeAttrs cfg.base16Scheme;
… while calling 'mkSchemeAttrs'
at /nix/store/c4ri83y4ka8q5ad1i9b278vp2qch2xx1-source/default.nix:190:5:
189| # (see https://github.com/base16-project/home/blob/main/builder.md#schemes-repository).
190| scheme:
| ^
191| let
… from call site
at /nix/store/c4ri83y4ka8q5ad1i9b278vp2qch2xx1-source/default.nix:237:25:
236|
237| populatedColors = colors inputAttrs;
| ^
238|
… while calling 'colors'
at /nix/store/c4ri83y4ka8q5ad1i9b278vp2qch2xx1-source/default.nix:44:12:
43| */
44| colors = scheme:
| ^
45| let
… from call site
at /nix/store/c4ri83y4ka8q5ad1i9b278vp2qch2xx1-source/default.nix:76:18:
75|
76| base-hex = lib.mapAttrs' (k: v: lib.nameValuePair "${k}-hex" v) base;
| ^
77| base-short = lib.mapAttrs' (k: v: lib.nameValuePair "${k}" v) base;
… while calling 'mapAttrs''
at /nix/store/38rzi6kzh5ilbpdgpvd8aj34vksgkcfa-source/lib/attrsets.nix:518:5:
517| # Attribute set to map over.
518| set:
| ^
519| listToAttrs (map (attr: f attr set.${attr}) (attrNames set));
… from call site
at /nix/store/c4ri83y4ka8q5ad1i9b278vp2qch2xx1-source/default.nix:50:9:
49| base = lib.mapAttrs (_: value: lib.toLower (lib.removePrefix "#" value)) (
50| lib.filterAttrs (name: _: lib.hasPrefix "base" name && builtins.stringLength name == 6) scheme
| ^
51| );
… while calling 'filterAttrs'
at /nix/store/38rzi6kzh5ilbpdgpvd8aj34vksgkcfa-source/lib/attrsets.nix:305:5:
304| # The attribute set to filter
305| set:
| ^
306| listToAttrs (concatMap (name: let v = set.${name}; in if pred name v then [(nameValuePair name v)] else []) (attrNames set));
… from call site
at /nix/store/c4ri83y4ka8q5ad1i9b278vp2qch2xx1-source/default.nix:202:12:
201| //
202| (yaml2attrs scheme)
| ^
203| ;
… while calling 'yaml2attrs'
at /nix/store/c4ri83y4ka8q5ad1i9b278vp2qch2xx1-source/default.nix:108:16:
107|
108| yaml2attrs = yaml:
| ^
109| builtins.fromJSON (builtins.readFile (pkgs.stdenv.mkDerivation {
error: a 'x86_64-linux' with features {} is required to build '/nix/store/lm95hkshyfvyisjcxs17z4kx5mrcc3md-fromYAML.drv', but I am a 'aarch64-darwin' with features {benchmark, big-parallel, nixos-test}
7.63s user 1.97s system 51% cpu 18.808s totalSenchoPens commented
Hi Chris,
Thank you for submitting this interesting and important issue!
I am grateful for your clear explanation of the root cause.
I reproduced the bug and I see multiple possible solutions. I will try to implement a fix today.
SenchoPens commented
Hi Chris @montchr, can you please check whether the last release works for you? Once you confirm this, I'll make a PR to the Stylix repository to update the base16.nix input.
montchr commented
@SenchoPens Thanks for the update. Unfortunately the pure-Nix approach doesn't work for me, at least on aarch64-darwin:
error:
… while checking flake output 'nixosConfigurations'
at «none»:0: (source not available)
… while checking the NixOS configuration 'nixosConfigurations.freundix'
at «none»:0: (source not available)
… while calling the 'seq' builtin
at /nix/store/38rzi6kzh5ilbpdgpvd8aj34vksgkcfa-source/lib/modules.nix:326:18:
325| options = checked options;
326| config = checked (removeAttrs config [ "_module" ]);
| ^
327| _module = checked (config._module);
… while evaluating a branch condition
at /nix/store/38rzi6kzh5ilbpdgpvd8aj34vksgkcfa-source/lib/modules.nix:267:9:
266| checkUnmatched =
267| if config._module.check && config._module.freeformType == null && merged.unmatchedDefns != [] then
| ^
268| let
… in the right operand of the AND (&&) operator
at /nix/store/38rzi6kzh5ilbpdgpvd8aj34vksgkcfa-source/lib/modules.nix:267:72:
266| checkUnmatched =
267| if config._module.check && config._module.freeformType == null && merged.unmatchedDefns != [] then
| ^
268| let
… while evaluating the attribute 'unmatchedDefns'
at /nix/store/38rzi6kzh5ilbpdgpvd8aj34vksgkcfa-source/lib/modules.nix:671:7:
670| # Transforms unmatchedDefnsByName into a list of definitions
671| unmatchedDefns =
| ^
672| if configs == []
… while calling the 'concatLists' builtin
at /nix/store/38rzi6kzh5ilbpdgpvd8aj34vksgkcfa-source/lib/modules.nix:678:11:
677| else
678| concatLists (mapAttrsToList (name: defs:
| ^
679| map (def: def // {
… while calling anonymous lambda
at /nix/store/38rzi6kzh5ilbpdgpvd8aj34vksgkcfa-source/lib/attrsets.nix:539:10:
538| attrs:
539| map (name: f name attrs.${name}) (attrNames attrs);
| ^
540|
… from call site
at /nix/store/38rzi6kzh5ilbpdgpvd8aj34vksgkcfa-source/lib/attrsets.nix:539:16:
538| attrs:
539| map (name: f name attrs.${name}) (attrNames attrs);
| ^
540|
… while calling anonymous lambda
at /nix/store/38rzi6kzh5ilbpdgpvd8aj34vksgkcfa-source/lib/modules.nix:678:46:
677| else
678| concatLists (mapAttrsToList (name: defs:
| ^
679| map (def: def // {
… while calling anonymous lambda
at /nix/store/38rzi6kzh5ilbpdgpvd8aj34vksgkcfa-source/lib/modules.nix:664:22:
663| # Propagate all unmatched definitions from nested option sets
664| mapAttrs (n: v: v.unmatchedDefns) resultsByName
| ^
665| # Plus the definitions for the current prefix that don't have a matching option
… while calling anonymous lambda
at /nix/store/38rzi6kzh5ilbpdgpvd8aj34vksgkcfa-source/lib/attrsets.nix:539:10:
538| attrs:
539| map (name: f name attrs.${name}) (attrNames attrs);
| ^
540|
… from call site
at /nix/store/38rzi6kzh5ilbpdgpvd8aj34vksgkcfa-source/lib/attrsets.nix:539:16:
538| attrs:
539| map (name: f name attrs.${name}) (attrNames attrs);
| ^
540|
… while calling anonymous lambda
at /nix/store/38rzi6kzh5ilbpdgpvd8aj34vksgkcfa-source/lib/modules.nix:678:46:
677| else
678| concatLists (mapAttrsToList (name: defs:
| ^
679| map (def: def // {
… while calling anonymous lambda
at /nix/store/38rzi6kzh5ilbpdgpvd8aj34vksgkcfa-source/lib/modules.nix:664:22:
663| # Propagate all unmatched definitions from nested option sets
664| mapAttrs (n: v: v.unmatchedDefns) resultsByName
| ^
665| # Plus the definitions for the current prefix that don't have a matching option
… while calling anonymous lambda
at /nix/store/38rzi6kzh5ilbpdgpvd8aj34vksgkcfa-source/lib/modules.nix:594:46:
593| # an attrset 'name' => list of submodules that define ‘name’.
594| defnsByName = byName "config" (module: value:
| ^
595| map (config: { inherit (module) file; inherit config; }) (pushDownProperties value)
… from call site
at /nix/store/38rzi6kzh5ilbpdgpvd8aj34vksgkcfa-source/lib/modules.nix:595:69:
594| defnsByName = byName "config" (module: value:
595| map (config: { inherit (module) file; inherit config; }) (pushDownProperties value)
| ^
596| ) configs;
… while calling 'pushDownProperties'
at /nix/store/38rzi6kzh5ilbpdgpvd8aj34vksgkcfa-source/lib/modules.nix:826:24:
825| */
826| pushDownProperties = cfg:
| ^
827| if cfg._type or "" == "merge" then
… from call site
at /nix/store/fxgkj2xxcz9pn5khyq9f0ddyfpiw854y-source/stylix/palette.nix:128:26:
127| # https://github.com/SenchoPens/base16.nix#mktheme
128| lib.stylix.colors = (base16.mkSchemeAttrs cfg.base16Scheme).override override;
| ^
129| lib.stylix.scheme = base16.mkSchemeAttrs cfg.base16Scheme;
… while calling 'mkSchemeAttrs'
at /nix/store/4wg7s6z4hmv6x7a8nzr6nznd1mz0x5cw-source/default.nix:235:5:
234| # (see https://github.com/base16-project/home/blob/main/builder.md#schemes-repository).
235| scheme:
| ^
236| let
… from call site
at /nix/store/4wg7s6z4hmv6x7a8nzr6nznd1mz0x5cw-source/default.nix:279:25:
278|
279| populatedColors = colors inputAttrs;
| ^
280|
… while calling 'colors'
at /nix/store/4wg7s6z4hmv6x7a8nzr6nznd1mz0x5cw-source/default.nix:45:12:
44| */
45| colors = scheme:
| ^
46| let
… from call site
at /nix/store/4wg7s6z4hmv6x7a8nzr6nznd1mz0x5cw-source/default.nix:77:18:
76|
77| base-hex = lib.mapAttrs' (k: v: lib.nameValuePair "${k}-hex" v) base;
| ^
78| base-short = lib.mapAttrs' (k: v: lib.nameValuePair "${k}" v) base;
… while calling 'mapAttrs''
at /nix/store/38rzi6kzh5ilbpdgpvd8aj34vksgkcfa-source/lib/attrsets.nix:518:5:
517| # Attribute set to map over.
518| set:
| ^
519| listToAttrs (map (attr: f attr set.${attr}) (attrNames set));
… from call site
at /nix/store/4wg7s6z4hmv6x7a8nzr6nznd1mz0x5cw-source/default.nix:51:9:
50| base = lib.mapAttrs (_: value: lib.toLower (lib.removePrefix "#" value)) (
51| lib.filterAttrs (name: _: lib.hasPrefix "base" name && builtins.stringLength name == 6) scheme
| ^
52| );
… while calling 'filterAttrs'
at /nix/store/38rzi6kzh5ilbpdgpvd8aj34vksgkcfa-source/lib/attrsets.nix:305:5:
304| # The attribute set to filter
305| set:
| ^
306| listToAttrs (concatMap (name: let v = set.${name}; in if pred name v then [(nameValuePair name v)] else []) (attrNames set));
… from call site
at /nix/store/4wg7s6z4hmv6x7a8nzr6nznd1mz0x5cw-source/default.nix:245:20:
244| builtins.unsafeDiscardStringContext "${scheme}"
245| )); } // yaml2attrs scheme;
| ^
246|
… while calling 'yaml2attrs'
at /nix/store/4wg7s6z4hmv6x7a8nzr6nznd1mz0x5cw-source/default.nix:115:16:
114| yaml2attrs-ifd = yaml: builtins.fromJSON (builtins.readFile (yaml2json yaml));
115| yaml2attrs = yaml: import "${fromYaml}/fromYaml.nix" { inherit lib; } (builtins.readFile yaml);
| ^
116| # Checks that the above yaml2attrs function has correctly parsed an yaml file.
… from call site
at /nix/store/4wg7s6z4hmv6x7a8nzr6nznd1mz0x5cw-source/default.nix:115:22:
114| yaml2attrs-ifd = yaml: builtins.fromJSON (builtins.readFile (yaml2json yaml));
115| yaml2attrs = yaml: import "${fromYaml}/fromYaml.nix" { inherit lib; } (builtins.readFile yaml);
| ^
116| # Checks that the above yaml2attrs function has correctly parsed an yaml file.
… while calling 'parse'
at /nix/store/lzvgfxcp5kjwgs66l8dfqvbqcy47gla4-source/fromYaml.nix:29:11:
28|
29| parse = text: let
| ^
30| lines = l.splitString "\n" text;
… from call site
at /nix/store/lzvgfxcp5kjwgs66l8dfqvbqcy47gla4-source/fromYaml.nix:177:5:
176| in
177| make filtered (-1);
| ^
178| in
… while calling 'make'
at /nix/store/lzvgfxcp5kjwgs66l8dfqvbqcy47gla4-source/fromYaml.nix:103:19:
102| */
103| make = lines: i: let
| ^
104| mNext = l.elemAt matched (i + 1);
… from call site
at /nix/store/lzvgfxcp5kjwgs66l8dfqvbqcy47gla4-source/fromYaml.nix:138:9:
137| childrenMerged =
138| l.foldl
| ^
139| (all: cObj: (
… while calling 'foldl'
at /nix/store/38rzi6kzh5ilbpdgpvd8aj34vksgkcfa-source/lib/lists.nix:81:20:
80| */
81| foldl = op: nul: list:
| ^
82| let
… from call site
at /nix/store/38rzi6kzh5ilbpdgpvd8aj34vksgkcfa-source/lib/lists.nix:87:8:
86| else op (foldl' (n - 1)) (elemAt list n);
87| in foldl' (length list - 1);
| ^
88|
… while calling 'foldl''
at /nix/store/38rzi6kzh5ilbpdgpvd8aj34vksgkcfa-source/lib/lists.nix:83:16:
82| let
83| foldl' = n:
| ^
84| if n == -1
… from call site
at /nix/store/38rzi6kzh5ilbpdgpvd8aj34vksgkcfa-source/lib/lists.nix:86:14:
85| then nul
86| else op (foldl' (n - 1)) (elemAt list n);
| ^
87| in foldl' (length list - 1);
… while calling anonymous lambda
at /nix/store/lzvgfxcp5kjwgs66l8dfqvbqcy47gla4-source/fromYaml.nix:139:15:
138| l.foldl
139| (all: cObj: (
| ^
140| if l.isAttrs cObj
… while calling anonymous lambda
at /nix/store/lzvgfxcp5kjwgs66l8dfqvbqcy47gla4-source/fromYaml.nix:134:10:
133| l.map
134| (sIdx: make lines sIdx)
| ^
135| childIdxs;
… from call site
at /nix/store/lzvgfxcp5kjwgs66l8dfqvbqcy47gla4-source/fromYaml.nix:134:16:
133| l.map
134| (sIdx: make lines sIdx)
| ^
135| childIdxs;
… while calling 'make'
at /nix/store/lzvgfxcp5kjwgs66l8dfqvbqcy47gla4-source/fromYaml.nix:103:19:
102| */
103| make = lines: i: let
| ^
104| mNext = l.elemAt matched (i + 1);
… from call site
at /nix/store/lzvgfxcp5kjwgs66l8dfqvbqcy47gla4-source/fromYaml.nix:78:17:
77| key = l.elemAt m1 1;
78| value = parseValue (l.elemAt m1 2);
| ^
79| }
… while calling 'parseValue'
at /nix/store/lzvgfxcp5kjwgs66l8dfqvbqcy47gla4-source/fromYaml.nix:16:16:
15|
16| parseValue = value: let
| ^
17| is = re: l.match re value != null;
… from call site
at /nix/store/lzvgfxcp5kjwgs66l8dfqvbqcy47gla4-source/fromYaml.nix:22:8:
21| in
22| if is singlueQuotedString then
| ^
23| contentOf singlueQuotedString
… while calling 'is'
at /nix/store/lzvgfxcp5kjwgs66l8dfqvbqcy47gla4-source/fromYaml.nix:17:10:
16| parseValue = value: let
17| is = re: l.match re value != null;
| ^
18| contentOf = re: l.elemAt (l.match re value) 0;
error: invalid regular expression ''(.*?)'[[:space:]]*'
at «none»:0: (source not available)I actually tried something like that yesterday on aarch64-darwin and got the same (or nearly-identical) error.
I've also used https://github.com/Misterio77/nix-colors, which uses a pure-Nix approach to YAML parsing, and that has no issues for me on this machine. However, I reported a similar regex issue there almost a year ago and it has since been fixed:
But it looks like the error I'm seeing on 2c292df is different:
error: invalid regular expression ''(.*?)'[[:space:]]*'
SenchoPens commented
Hi Chris, thanks for trying out the new version!
Seems like the update needs more work :)
Unfortunately I'm a little busy right now as it's the end of the semester, but I'll try to think of some robust fix this week. You can still opt-out of pure Nix parsing at the cost of doing an IFD, see the Troubleshooting section
SenchoPens commented
Hi @montchr, I might have fixed this issue in the latest release, can you please try it out?
SenchoPens commented
Closing due to innactivity. Please reopen if the issue is still there.