move `@cImport` to the build system
andrewrk opened this issue · 29 comments
This issue is to remove the @cImport language builtin, instead relying on the feature being provided by the build system (std.Build.Step.TranslateC).
This removes one dependency on libclang inside the Zig compiler executable, which is required for #16270. In the near-term future, the translate-c implementation will still be inside the Zig compiler executable, along with the ability to compile C/C++ source files. However, we can prepare users for this transition by solving the use case via the build system rather than a language feature.
As a consolation prize, the TranslateC build step can be enhanced with advanced settings, such as namespace stripping and validation of existing bindings. Since changes to this don't require changing the language, it's OK if the scope & complexity increase to some extent.
Blockers:
- #20648
- move https://ziglang.org/learn/build-system/ to become a guide along with the language reference
- #20649
Related:
A small extra note: this eliminates one quite common use case of usingnamespace in the wild, which is to have a c.zig containing a pub usingnamespace @cImport(...) declaration. Instead, you now use a TranslateC step as described above, and directly expose the generated file to Zig code as a module.
@andrewrk would this mean that in order to interoperate with c, you need to use the zig build system?
I have a small personal project written in c built using make files. Im currently extending it by writing new zig code that cIncludes header files and invoking the zig compiler from make (i.e zig build-obj).
Would this use case not be supported anymore?
You would just have to invoke zig translate-c in your makefile; that's what std.Build.Step.TranslateC does. The compiler also does the moral equivalent of that when you use @cImport().
@149-code, @alexrp -- note that once #16270 is implemented, zig translate-c will no longer be a subcommand, with this functionality instead existing entirely within the build system. In that case, you probably will need to use zig build.
In theory, you can have your build.zig purely do the C translation, so zig build effectively becomes your drop-in for zig translate-c. However, this will almost certainly be more awkward than just migrating your C project to use the Zig build system.
with this functionality instead existing entirely within the build system
Hm, that sounds a bit concerning. Not looking to relitigate #16270 all over again, but I'm on board with it because the promise was that existing use cases would still be supported - it just might involve downloading an extra package for all the C/C++ functionality (which is totally fair). But if translate-c can't even be accessed outside the Zig build system, that leaves my Zig MSBuild SDK in a very awkward spot. I'm specifically integrating Zig into another build system entirely (MSBuild), so I would have to generate and invoke a build.zig which adds a whole other layer of complexity, performance penalty, and moving parts for (AFAICT) no benefit other than accessing functionality that would have previously been easy to integrate.
Please don't get me wrong; I really like the Zig build system and I use it extensively in my "pure Zig" projects. But if we start making existing features of the zig binary wholly exclusive to the Zig build system, I think that's going to significantly harm Zig adoption in brownfield or polyglot settings.
I'm specifically integrating Zig into another build system entirely (MSBuild), so I would have to generate and invoke a
build.zigwhich adds a whole other layer of complexity, performance penalty, and moving parts for (AFAICT) no benefit other than accessing functionality that would have previously been easy to integrate.
I don't think any code generation would be required. You could write a single, generic build.zig that exposes a CLI pretty much identical to the current zig translate-c. The only difference is that the binary exposing the CLI would not be the zig compiler.
One could probably even write a zig package that provides a translate-c binary in every way identical to the current zig translate-c subcommand except that it is not part of the zig compiler binary.
What ifreund says would work, but thinking about how the translate-c package is likely to work, it might not even be necessary. I would expect the translate-c package to expose an artifact, and users construct Run steps to run that artifact (of course, we would add any necessary helper functions to make this not-awkward). So, you could zig build the translate-c package directly to get a translate-c binary.
That sounds reasonable. My concern here was just the prospect of having to integrate all the machinery of the Zig build system as part of the build logic in my MSBuild SDK (which is currently just a zig build-exe/build-lib/test/cc/c++ invocation at the end of the day). Stacking build systems on top of each other is guaranteed pain, in my experience.
So as long as there's a way to invoke translate-c as a plain old CLI command - whether part of zig or a binary of its own - I'm a happy camper.
The only other point I would add here is that I think it would be good to distribute such a translate-c binary as part of a "C/C++ support" binary package on ziglang.org to reduce friction for cases like @149-code's (it would also make packaging a tiny bit easier for my SDK, but that's not essential by any means).
Well, I think you have provided a quite interesting and valuable real world use case, so, let's work together to come up with a migration proposal that is appealing before proceeding with removal of @cImport functionality.
To be clear @andrewrk, the use case described above doesn't depend on @cImport, only on the zig translate-c subcommand. This proposal kind of has two parts: removing @cImport seems easy and unproblematic (at least, once we improve std.Build.Step.TranslateC), while migrating zig translate-c out of the compiler is what raised concerns above.
Ah thanks for that clarification. Well, point stands for that follow-up issue but I will remove it from the blocker list above.
So as long as there's a way to invoke translate-c as a plain old CLI command - whether part of zig or a binary of its own - I'm a happy camper.
Adding my 2cts to heavily support this. We're using zig via bazel, so we directly invoke zig build-{exe,static} and would love to keep translate-c as a CLI.
I really like to use the zig run command to run some of my small projects/scripts (some are using @cImport) without dropping an exe at the current directory. So, after this change, is there way I can still simply call zig run or do I have to use a build.zig file?
Adding my 2cts to heavily support this. We're using
zigvia bazel, so we directly invokezig build-{exe,static}and would love to keeptranslate-cas a CLI.
That's a given - the build system will invoke the translate-c tool as a subprocess using CLI args.
So, after this change, is there way I can still simply call
zig runor do I have to use a build.zig file?
No, this use case will regress.
This is such a shame. Zig's main attraction is the easy & built-in interoperability with C.
Regressing features on that front is not a good sign.
@LouisLibre Your comment does not add anything of value to this discussion. A proposal like this weighs the benefits and drawbacks against each other to decide if the zig project as a whole would be improved by the proposal or not. Merely reiterating a single drawback with no nuance or discussion as you have done is useless noise.
For “hello world from C” and bug-reproduction demos, it is useful to have C-interop without having to set up the build system. (The ongoing HN thread on this topic has several anecdotes from people convinced to try out Zig because of such simple demos.) For someone new to the language, setting up the build system at all represents significant complexity.
However this use-case could be satisfied by adding an argument to zig run, zig build-exe, and friends that allowed the user to specify C files to be auto-included as modules.
This would continue to kick the can down the road with regards to #16270, but it may make sense in the short term (allowing both zig run to include C modules and removing cImport for incremental, if that was a concern).
Keeping incorporating c header imports as a feature accessible to the CLI is important to the semantic analysis pass that happens as a part of Zigler. https://github.com/E-xyza/zigler (I'm also using this in prod and this week it's saving the bacon of the startup I'm at)
@ityonemo can you be more specific? Seems like you should be fine with the zig translate-c subcommand, no? It's more powerful than @cImport.
I'm fine with deprecating @cImport, but just as the "zig module import" parameters (paths, deps) can be set via command line, if a module is imported as a c ABI, id want tobe able to do so via cli without resorting to build.zig. mentioning that zig run would regress is a bit worrying. For example, in 0.12 I can run zig run --dep foo -Mxyz.... I would have imagined something like zig run --h foo.h --define true=false -Mc_lib... etc.
I'm not exactly sure how the translate-c subcommands works (it sounds like it will? Maybe?) and I could probably figure it out, but I had lots of difficulty integrating a semantic pass with build.zig that lived side-by-side with the libs I generate, since I am doing sneaky things like substituting shims when running semantic analysis pass.
@ityonemo the @cImport feature effectively constructs a translate-c command which outputs to a file, and then that file is imported with @import. I can't see anything but a more powerful API offered to zigler by using the CLI rather than language feature. You won't be able to do it with 1 command though, you'll need to run translate-c separately and save the output somewhere. There's no plan to make run, build-exe, build-lib, or build-obj capable of executing a translate-c step automatically.
Anyway, here's a related issue I typed up just now. I think this is the actual issue that people are wanting to talk about today:
Good to know. More than one command is not ideal, but it's fine! I could probably also figure out how to do it with build.zig, I just had enough trouble with it that I gave up last time :)
From the perspective of an outsider, the fact that @cImport is probably going away, while still having blog posts written about how great it is, it would be nice to preemptively learn the "new way" to write Zig that interops with C. If the documentation could just reflect the future (assuming of course that the future is somewhat known) it would be nice, since otherwise learning Zig now feels like it could be obsolete next week. (Of course this is always to be true of an experimental language but it just feels like a significant block is all.)
To be clear, there is an upgrade path, so if you learn how to use @cImport and then you have to move it to the build system, it doesn't really mean you wasted your time. It just means you have to do something like this diff.
once this is done would it make sense to remove the top level -D CLI flag and instead force the use of it within -cflags if something like that is needed for C files? its somewhat of a "footgun" that it exists at the same time as the build systems -D which has an entirely separate meaning. there have been multiples issues surrounding the problem (e.g. #19286 was the first i found) and ive observed it countless times in zig communities
I'm not sure how to refine this into an issue, but it's imho a blocker for removing @cImport:
When using a std.Build.Step.TranslateC, we have to be able to cover the same use cases and flexibility as we have right now.
But having that property would also not be sufficient:
When building a system like Ashet OS, we need the ability to define the used libc by the consumer of a header file, not by a separate producer step.
Consider the following use case:
You have a package which exposes a module. That module internally does some @cImports, but the module itself does not link libc.
Now the consumer of that package gets the module, and adds it to three executables:
x86-hosted-linux.elf, which links musl libc (so just passing-lcon the command line)x86-pc-bios.elf, which links foundation-libc (but does not pass-lcon the command line as it would not find thelibfoundation.afile)arm-qemu-virt.elf, which linksnewlibvia--libc newlib.txt -lc
Right now, it just works, as the translate-c operation happens with the context of the std.Build.Step.Compile that is invoked. With the proposed solution, this use case will break and will require a much more complex setup, as now all logic inside the build script of the kernel for deciding the libc logic will also have to move into a unrelated package which is currently general-purpose.
What would be the way forward here?
The use case here is how zfat integrates into other projects, not linking the Zig-provided libc by default:
Usage in Ashet OS:
I don't understand why replacing @cImport would remove the dependency on clang in the Zig executable, don't you need to depend on clang anyway to be able to run build.zig? Or will you move Step.TranslateC to a different repository that you have to install as a dependency in build.zig.zon?