Cannot find `abc` binary when running `heir-opt` as a standalone binary

[kevaundray]

Issue

When running heir-opt as a standalone binary, it cannot find the abc binary.

To reproduce

(This assumes we are in the heir directory)

1. Build heir-opt using bazel

• I use bazel build @heir//tools:heir-opt
• There should be a folder under heir/bazel-bin/tools/heir-opt. The abc binary should also be located at heir/bazel-bin/external/edu_berkeley_abc/abc

2. Run heir-opt from the bazel-bin directory with a --yosys-optimizer

• I ran ./bazel-bin/tools/heir-opt --yosys-optimizer --canonicalize --cse heir/tests/yosys_optimizer/add_one.mlir
• There should be no visible output, whereas we expect for it to show the modified mlir output.

3. Run heir-opt again, but this time use strace

• I ran strace ./bazel-bin/tools/heir-opt --yosys-optimizer --canonicalize --cse heir/tests/yosys_optimizer/add_one.mlir
• The output is quite large so I've pasted it here
• The line read(3, "sh: 1: external/edu_berkeley_abc"..., 4096) = 48 seems to indicate that it cannot find the abc tool

Solution

What fixed it for me was telling bazel to link to the output location of wherever the abc binary was once it was built.

The modification was made in heir/tools/BUILD.

I changed "HEIR_ABC_BINARY=\\\"$(rootpath @edu_berkeley_abc//:abc)\\\"", to "HEIR_ABC_BINARY=\\\"$(location @edu_berkeley_abc//:abc)\\\"",

Observations

• Everything works for me when running from bazel, ie bazel build and bazel run work fine.
• After the change, bazel build and bazel run still work fine for me.
• The location seems to be an absolute path, so as long as the abc binary is not moved, it works. Otherwise, I had to override the HEIR_ABC_BINARY environment variable to point to the new location of the abc binary

[asraa]

Thanks for such a detailed issue!

What fixed it for me was telling bazel to link to the output location of wherever the abc binary was once it was built.

This also worked for me when i ran as a standalone binary from ./bazel-bin.

After the change, bazel build and bazel run still work fine for me.

For me, location stopped working. I think this is a me issue though, since I remember at some point location worked and it was always that before I recently changed it to rootpath. @code-perspective do you have some time to try too?

[asraa]

Ah, just one thing too in case this helps, but if you run with bazel build -c dbg, then you can a --debug flag on heir-opt and you'll be able to see Yosys' output log and see the error messages:

Running ABC command: "bazel-out/k8-dbg/bin/external/edu_berkeley_abc/abc" -s -f <abc-temp-dir>/abc.script 2>&1
ABC: sh: line 1: bazel-out/k8-dbg/bin/external/edu_berkeley_abc/abc: No such file or directory
ERROR: ABC: execution of command ""bazel-out/k8-dbg/bin/external/edu_berkeley_abc/abc" -s -f /tmp/yosys-abc-ipcBUL/abc.script 2>&1" failed: return code 127.

[asraa]

Could you do me a quick favor if bazel run works for you with location? Can you grep for the "Running ABC command" line of the output when you run

bazel run -c dbg  @heir//tools:heir-opt -- --yosys-optimizer $(pwd)/tests/yosys_optimizer/global_absorb.mlir --debug

[kevaundray]

Could you do me a quick favor if bazel run works for you with location? Can you grep for the "Running ABC command" line of the output when you run

bazel run -c dbg  @heir//tools:heir-opt -- --yosys-optimizer $(pwd)/tests/yosys_optimizer/global_absorb.mlir --debug

Hmm I get an error after running, this. The full output is here

Though I indeed get:

13.1.1. Executing ABC.
Running ABC command: "bazel-out/k8-dbg/bin/external/edu_berkeley_abc/abc" -s -f <abc-temp-dir>/abc.script 2>&1
ABC: sh: 1: bazel-out/k8-dbg/bin/external/edu_berkeley_abc/abc: not found
ERROR: ABC: execution of command ""bazel-out/k8-dbg/bin/external/edu_berkeley_abc/abc" -s -f /tmp/yosys-abc-BIalVV/abc.script 2>&1" failed: return code 127.

[asraa]

Thank you!

So to me rootpath gives the path external/.. path, which makes it work for bazel run since that runs inside the runfiles directory (bazel-bin/tools/heir-opt.runfiles/heir/), but not when running as a standalone binary.

While location gives bazel-out/k8-dbg/bin/external/... path, making it work for the standalone binary, but not bazel run.

My worry is that invoking it as a standalone binary outside of the project tree will fail too with $(location ...) too. So we have three situations (1) bazel run (2) running in project root as a standalone binary and (3) running in a standalone binary anywhere.

For (1)

• Should work out of the box, but we need the external/* path

For (2)

• Should work out of the box, but we need the bazel-out/k8-dbg/bin/external/* path

I can resolve these two now that I know that location is giving the full path relative to the project dir and rootpath is giving the path relative to the runfiles dir.

For (3)

• User needs to set HEIR_ABC_PATH
• User also needs to set the location of the liberty files, through HEIR_YOSYS_SCRIPTS_DIR

So running as a standalone binary should probably always have HEIR_ABC_PATH set IMO, and we can't guarantee it to work out of the box, unless you happen to be within the project tree.

[j2kun]

One thing we can also look into is having a sh_binary wrapper around heir-opt for the standalone case, so that it can set up all the paths correctly.

[kevaundray]

Thank you!

So to me rootpath gives the path external/.. path, which makes it work for bazel run since that runs inside the runfiles directory (bazel-bin/tools/heir-opt.runfiles/heir/), but not when running as a standalone binary.

While location gives bazel-out/k8-dbg/bin/external/... path, making it work for the standalone binary, but not bazel run.

My worry is that invoking it as a standalone binary outside of the project tree will fail too with $(location ...) too. So we have three situations (1) bazel run (2) running in project root as a standalone binary and (3) running in a standalone binary anywhere.

For (1)

• Should work out of the box, but we need the external/* path

For (2)

• Should work out of the box, but we need the bazel-out/k8-dbg/bin/external/* path

I can resolve these two now that I know that location is giving the full path relative to the project dir and rootpath is giving the path relative to the runfiles dir.

For (3)

• User needs to set HEIR_ABC_PATH
• User also needs to set the location of the liberty files, through HEIR_YOSYS_SCRIPTS_DIR

So running as a standalone binary should probably always have HEIR_ABC_PATH set IMO, and we can't guarantee it to work out of the box, unless you happen to be within the project tree.

Yep, I think its reasonable to say that the path for abc and yosys scripts need to be set for standalone binaries. Also +1 on shell wrapper in the long term or something that ensures the user does not need to think about the path for abc and yosys scripts

[asraa]

One thing we can also look into is having a sh_binary wrapper around heir-opt for the standalone case, so that it can set up all the paths correctly.

THANK YOU! I know there must have been some nice way for (3), and this is it.

[ai-mannamalai]

Should this be tagged as a build issue ?