Tracking issue for "macro naming and modularisation" (RFC #1561)

Question

Tracking issue for "macro naming and modularisation" (RFC #1561)

nikomatsakis opened this issue 8 years ago · 164 comments

Answer 1 · 2016-08-23T17:34:00.000Z

One question I don't think was ever answered on the RFC thread is if providing a way for current macros to opt in is feasible.

Just throwing this out there as to make sure it doesn't get forgotten. My vote is obviously for yes we can, but I'm also obviously not on a team, or even an active contributor beyond these discussions.

Answer 2 · 2016-08-23T23:05:18.000Z

@camlorn this was actually address in my final edit to the RFC before merging. To summarise, it might be possible but we need implementation experience to be sure and to work out exactly how it might work. So, basically punting on the issue for now. It is certainly an area where I think we need to tread extremely carefully.

Answer 3 · 2017-01-28T02:53:16.000Z

c.f. #37732 (comment)

Answer 4 · 2017-02-07T01:18:41.000Z

Tasks

Stackless macro expansion (PR #36214).
Fix $crate with inter-crate re-exports (PR #37463).
Future-proof #[no_link] crates (PR #37247).
Parse paths in bang macro invocations (e.g. path::to::mac!();) (PR #36662).
Back-port shadowing restrictions from rust-lang/rfcs#1560 where appropriate (PR #36767).
Forbid user-defined macros named "macro_rules" (PR #36730).
Miscellaneous groundwork in syntax, resolve, and metadata (PRs #36438, #36525, #36573, #36601, #36616, #37036, #37213, #37292, and #37542).
Land #![feature(use_extern_macro)] (PRs #37732 and #38082).
Allow use to shadow macros from the prelude, currently is an ambiguity error (PR #40501).
Improve interaction with macros 1.0 exports in the same crate, i.e. emit more duplicate errors (PR #40509).
Integrate with rustdoc, issue #39436 (PR #40814).

Answer 5 · 2017-04-26T08:36:08.000Z

@jseyfried What's the status of this? Seems like the two items without checks have had their PRs merged. AFAIK the RFC was for this to be in prep for the next iteration of declarative macros, but some of the PRs seem like we could enable it for macro_rules! macros. Is that so? What's the migration story (potential breakages?) etc?

Answer 6 · 2017-05-03T17:56:00.000Z

@jseyfried @nikomatsakis I'm also interested in the status of this. Can it be activated for 1.19?

Answer 7 · 2017-05-04T21:02:13.000Z

I don't have a good feeling for current status. @jseyfried?

Answer 8 · 2017-05-05T01:59:17.000Z

@withoutboats

some of the PRs seem like we could enable it for macro_rules! macros. Is that so?

We can enable it for #[macro_export] macro_rules! in extern crates (and derive procedural macros in extern crates) but not crate-local macro_rules! (since crate-local macro_rules!'s scopes are too different from items' scopes).

This works today with #![feature(use_extern_macros)], which is implied by #![feature(proc_macro)] and #![feature(decl_macro)] (the latter has not yet landed).

@brson @nikomatsakis
IIUC, we reached consensus on the current behavior of #![feature(use_extern_macros)] in the February design sprint.

That being said, supporting use for macros from extern crates but not for crate-local macro_rules! might be confusing for end users, especially since there's no way to define a crate-local macro that can be used until declarative macros 2.0 is stable.

I don't have a strong opinion on whether we should stabilize this now, wait for more experience with declarative macros 2.0 and then stabilize, or only stabilize when we stabilize declarative macros 2.0.

Answer 9 · 2017-05-05T02:05:32.000Z

We can enable it for #[macro_export] macro_rules! in extern crates (and derive procedural macros in extern crates) but not crate-local macro_rules! (since crate-local macro_rules!'s scopes are too different from items' scopes).

Is it impossible to make crate local macro_rules! macros work with this? Probably we'd need some kind of syntactic change to avoid breakages? (Maybe just a visibility modifier?)

I think declarative macros 2.0 is pretty far off, because there are going to be many design decisions we'll be able to reconsider with the new syntax, and that's going to take time to work through. It'd be nice to support this feature sooner than that.

Answer 10 · 2017-05-05T21:05:47.000Z

Is it impossible to make crate local macro_rules! macros work with this? Probably we'd need some kind of syntactic change to avoid breakages? (Maybe just a visibility modifier?)

It is possible with a visibility modifier (e.g. pub macro_rules! m { () => {} } could have item scope), but then

the scope would be inconsistent with macro_rules! m { () => {} }
if we don't support pub custom_macro! { ... } in general, pub macro_rules! would be a strange special-case
@nrc would be unhappy (IIUC) since people might think macro_rules! is "good enough", reducing macros 2.0 adoption.

Answer 11 · 2017-05-05T22:44:17.000Z

I'm personally not concerned about making macro_rules "good enough" - I think there will be enough improvements over the current system in macro macros, and I'm not against just straight up issuing deprecation warnings if you write a macro_rules macro someday. And I'm also alright with the special case.

However, I want the story about when macros use normal namespacing and when they don't to be easy to understand. I think a solution could be to introduce a new attribute which turns this scoping on, and require it (for both local and macro_export'd macros).

Later on we could introduce an attribute to turn old-school macro scoping on, and if we do the epoch shift, we could change which is the default.

Answer 12 · 2017-05-07T05:55:02.000Z

@withoutboats Yeah, that makes sense, but I'm not sure it's worth the complexity of another dimension due to making namespacing orthogonal to macros 1.0 vs macros 2.0.

Answer 13 · 2017-05-12T01:11:45.000Z

In general I'm opposed to blurring the lines between current macros and new macros. The 'good enough' argument is part of this - I fear that at some point macros 1.0 gets good enough that there is push back against further changes (I don't believe we can simply deprecate if the community doesn't want it). To some extent I think the macros 2.0 changes are a mixture of sugar and medicine - if we add all the nice changes to macros 1.0 (naming, syntax) then it makes it harder to sell the necessary changes which are left (e.g., hygiene).

A bigger worry for me is confusion for users - there is already confusion between old decl macros, new decl macros, procedural macros, etc. I worry that adding more layers to this - old macros with feature X, old macros with feature Y will make this even more confusing. On a technical level, supporting multiple versions of macros with different features opens the gates to many more bugs in the combinations of features which might be less tested.

Answer 14 · 2017-05-22T23:30:11.000Z

I don't understand all the parts at play here, but I want to be able to reexpert macro_rules macros from other crates in stdx.

Answer 15 · 2017-05-22T23:50:47.000Z

@brson The conclusion of our lang team discussion was that we probably aren't going to migrate to this until macros 2.0.

Answer 16 · 2017-11-08T19:55:21.000Z

So right now using the macros of a different crate works by doing:

#[macro_use] extern crate foo;

macro_from_foo!(...);

However, with RFC 2126, extern crate is being brought onto the path of deprecation, meaning that there will be lints for it and maybe even hard errors in a future epoch.

Now the epochs RFC has made one thing very clear: inter-epoch interop is always guaranteed, basically allowing a newer crate crate to use the older crate's functionality. For macros I've heard assurances that there will be "epoch hygiene" or something.

But how will this apply to #[macro_use]? I see the following options:

Never lint/error for extern crate foo if there is a #[macro_use] next to it. I don't think this is a good idea however, as it would be the legacy syntax that one wants to get rid of, and it would look weird to have a bunch of #[macro_use] extern crate foo; statements inside your lib.rs
Implement the macro naming and modularisation system for macros 1.0 as well.
Implement some other system, like #[macro_use] use cratename::module; or macro_use! {cratename::macro_name}.

Regarding points 2 and 3, @withoutboats has expressed that there is a desire to make idiomatic code of the future epoch legal in the current epoch, so we need to do those backards compatibly inside the current epoch.

cc @nrc @aturon

Answer 17 · 2017-11-08T21:16:51.000Z

@est31

Implement the macro naming and modularisation system for macros 1.0 as well.

This is already implemented across crate boundaries. #[macro_export] macros from an upstream crate appear in the upstream crate's root. For example,

// crate foo
#[macro_export]
macro_rules! macro_from_foo { () => {} }

// crate bar
#![feature(use_extern_macros)]
// ^ implied by `#![feature(proc_macro)]`, `#![feature(decl_macro)]`

extern crate foo; // no #[macro_use]
use foo::macro_from_foo; // `pub use` also works, subsumes `#[macro_reexport]`
macro_from_foo!();

In other words, generally speaking you can replace

#[macro_use]
extern crate foo;

with

#![feature(use_extern_macros)]
extern crate foo;
use foo::{macro_rules_macro_1, macro_rules_macro_2, ...};

Answer 18 · 2017-11-08T22:44:02.000Z

We talked about this somewhere before @jseyfried, but I'll remind us again that macro reexporting is stable when doing it with a glob use. Crate frunk already relies on it.

Answer 19 · 2018-01-16T03:29:57.000Z

I don't know if it is the right place to mention this, but here's what I would like to achieve:
I'd like to have the possibility to put macro definitions in an impl to be able to do Struct::macro!().
My use case (in my crate tql) for that is to generate macro in a custom derive (i.e. SqlTable) and use these generated macros from another macro (sql!()).
Without this feature, that would require the users to use macros that are not in code they wrote.
Is it planned to add the ability to declare macros in an impl block or is there any alternative for this use case?
Thank you.

Answer 20 · 2018-01-18T08:12:29.000Z

@antoyo

I'd like to have the possibility to put macro definitions in an impl to be able to do Struct::macro!().

Implementing this would require dramatically restructuring the compiler, and isn't planned in the foreseeable future. Also, I believe it is a unresolved question if we can do sound type checking while macros are still being expanded (e.g. issues with coherence checks).

My use case (in my crate tql) for that is to generate macro in a custom derive (i.e. SqlTable) and use these generated macros from another macro (sql!()).

This is why we have hygiene :)

Without this feature, that would require the users to use macros that are not in code they wrote.

The point of hygiene is that names at the macro definition resolve at the macro definition, independently of where the macro is invoked. Similarly, names passed in to the macro as arguments resolve at the call site, e.g. they can't be accidentally shadowed by a name at the macro definition. For example,

#![feature(decl_macro)]

mod foo {
    pub fn f() {} // (1)

    pub macro m($arg:expr) {
        f(); // This resolves to (1)
        mod bar {
            fn f() { $arg }
        }
    }
}

fn main() {
    fn f() {} // (2) (note -- no conflict error with (1))
    foo::m!(f()); // The `f` argument resolves to (2) even though $arg is in a weird place
}

Roughly speaking, hygiene causes a macro m($arg:expr) { ... } to resolve like a corresponding fn m(arg: T) { ... } would.

This also applies to procedural macros. For example,

#[proc_macro_derive(Foo)]
fn foo(_input: TokenStream) -> TokenStream {
    quote! {
        // Due to hygiene, these names never cause conflict errors.
        extern crate sql_macros;
        use sql_macros::sql;

        ... sql!(Struct) ...
    }
}

Annoyingly, we can't declare the extern crate sql_macros; use sql_macros::sql; in the proc-macro crate, since items in the proc-macro crate are compiled for the host platform. The target platform can only see the resulting procedural macros, not the underlying functions (vice versa for the host).

Once the Cargo.toml for procedural macros crates supports declaring target dependencies in addition to today's host dependencies, the target dependencies will automatically be in scope inside quote!.

Answer 21 · 2018-04-02T21:31:36.000Z

This was discussed in the recent work week with an eye towards how we might be able to stabilize slices of macros 2.0 in the Rust 2018 edition release. When we discussed this in a group it was concluded that as far as we knew this feature was working as intended.

While there are known "oddities" with respect to how macro_rules! works locally within a crate it was decided that this is an important enough feature that it should be ok to stabilize with such behavior.

@rust-lang/lang, could I convince one of y'all to enter into FCP on this issue? In terms of timeline I'd like to ensure that a chunk of macros 2.0 (not all of it) stabilizes all at once, so if this passes through FCP and the other pieces fall through then I think we won't stabilize this issue, but if other pieces come through as well I think we'll stabilize this.

Answer 22 · 2018-04-03T14:34:22.000Z

@rfcbot fcp merge

Let me clarify first what feature we're stabilizing, as I understand it:

All proc macros can be imported using normal path syntax, including attributes and derives. This means, for example, you could do this:

extern crate serde_derive as serde;

#[derive(serde::Serialize)]
struct Foo { }

extern crate serde_derive;

use serde_derive::Serialize;

#[derive(Serialize)]
struct Foo { }

macro_rules! macros from other crates with the #[macro_export] attribute can be imported using normal imports. I do not know if they are at the location they are declared at or in the root of the crate (someone clarify).

extern crate foo;

use foo::bar;

bar! { ... }

macro_rules! macros declared within the same crate still use the legacy visibility rules, and cannot be imported using use statements and path syntax (that is - no change here).

As a result of this, the #[macro_use] system can become a legacy syntax, replaced by using normal path imports for macros from other crates. This makes macros from other crates act more like any other item, and it is useful for making extern crate a legacy syntax as well.

Answer 23 · 2018-04-03T14:34:23.000Z

Team member @withoutboats has proposed to merge this. The next step is review by the rest of the tagged teams:

No concerns currently listed.

Once a majority of reviewers approve (and none object), this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up!

See this document for info about what commands tagged team members can give me.

Answer 24 · 2018-04-03T14:36:31.000Z

@withoutboats indeed! That is my understanding as to what we're stabilizing as well :)

Answer 25 · 2018-04-03T16:26:52.000Z

@withoutboats

macro_rules! macros from other crates with the #[macro_export] attribute can be imported using normal imports. I do not know if they are at the location they are declared at or in the root of the crate (someone clarify).

They show up at top level of the crate:

// crate foo
extern crate bar;
use bar::baz; // works
use bar::bazmod::baz; // errors
fn main() { baz!() }

// crate bar
pub mod bazmod {
    #[macro_export]
    macro_rules! baz { () => { println!("hello world!"); } }
}

Answer 26 · 2018-04-03T16:56:22.000Z

Sounds good, that's what I expected since its where they show up in docs.

Answer 27 · 2018-04-04T06:36:32.000Z

I tried this feature now, generally it's very nice with simple macros, but it seems to be unusable with macros that expand to another macros, because one needs to manually import also those, even thought they can be considered as implementation details. So macros don't at the moment, "lexically close" references to another macros, right? I.e. they expand unhygienically. Is this an expected thing?

Do macros that use the new use syntax "close over" the macros they expand into?

Answer 28 · 2018-04-04T07:34:49.000Z

Yes, macro_rules! is known to be unhygienic. Fixing this is one of the goals of declarative macros 2.0 #39412

Answer 29 · 2018-04-04T08:16:11.000Z

As extern crate will be removed from the language in the 2018 epoch (I think it will??) we need a replacement for #[macro_use]. Stabilizing this is one way to do this. 👍

Answer 30 · 2018-04-04T08:26:50.000Z

So is the following true: macro_rules! expand always without hygiene, so it would be possible to have foo! { "yahoo" } that expands into bar! { "yahoo" }, where bar is not imported, or depended by the f crate at all, so that what the bar! expands into is decided at the call site, depending on what the caller has imported?

I was thinking if it would have been possible to backward-compatibly enable a limited version of "macro import hygiene" in the cases where the new use syntax is used. However, if the expansion is done completely in the context of the caller, it seems hard, because the macro authors may have written their macros with the expectation that their macro expands to some another macro that is not even in scope at the definition site, so it can't be "closed over".

Answer 31 · 2018-04-04T08:34:06.000Z

However, in the vast majority of the cases, macros that are implementation details are defined in the same crate as the user-facing macros, so they are in scope at definition site AND at call site. The use syntax causes them no longer be in scope at call site, which is the whole point of the feature, but it would improve ergonomics if there were a hygienic "closed over reference" fallback for the old-style macros imported in the new way in case the macros they expand into are not in scope at call site. That would enable people to actually use the new import syntax.

Answer 32 · 2018-04-04T13:57:49.000Z

@golddranks unhygienic expansion is a feature of macros 1.0 at this point, and so this feature, for macros 1.0, would basically be inheriting that

Answer 33 · 2018-04-04T14:14:46.000Z

Thanks; I see. So, likewise, the expansion failing in absence of the macros in the callsite scope can considered a feature, as odd as it sounds?

Answer 34 · 2018-04-04T14:20:00.000Z

Indeed yes, it's not a feature we'd choose to have but it was a tradeoff for 1.0 stabilization

Answer 35 · 2018-04-07T14:25:33.000Z

🔔 This is now entering its final comment period, as per the review above. 🔔

Answer 36 · 2018-04-08T19:29:15.000Z

How do macro! and macro_rules! interact? I fear that since there's already macro_rules! everywhere, this won't get anywhere unless either:

We come out with the shiny new Rust 2.0! (wait...) This is the Python approach where we'd have to maintain two separate but related languages.
We keep both and push for the new one. This is the JS approach where the headache is still there, we have to figure out how they should interact and, worse, there may be some who just don't migrate for whatever reason.
All I ask is which one are we going with? Because simply punting it and saying "Eh, I'll get back to on that one." won't work when it's actually implemented.

(P.S. Where is the rfcbot icon from? That looks really cool!)

Answer 37 · 2018-04-08T20:34:11.000Z

We’re not doing Rust 2.0, macro_rules! is here to stay. It’s ok if a lot of existing code doesn’t migrate. The point is providing a better alternative for new code (or code the has someone interested in refactoring/rewriting it).

Answer 38 · 2018-04-09T00:40:33.000Z

@golddranks' issue sounds to me like a good case for not being able to use macro_rules! macros, as it puts them in an entirely new context that the author could not have anticipated. (but the new proc_macros are okay, since they have to be referred to by path, and the author can clearly see from day one that their generated invocations of other macros defined in the same crate do not work without adjustment)

Edit: Turns out #[macro_use(foo)] is a thing, and therefore this problem is already faced by macro_rules macros today.

Answer 39 · 2018-04-15T00:23:32.000Z

So, how macro_export works.
If we have two crates - library and main, then

library crate is traversed in unspecified order and all legacy macro items (macro_rules) marked with macro_export and macros reexported with macro_reexport are collected into a vector (so the vector can contain duplicated names). The order is generally determined by depth-first search, but I suspect things can become more complex with macro expansion.
The vector is deduplicated and from macros with the same name only the one that happens to be the last in the vector is kept.
Remaining names are planted into the macro namespace of library's root module from main's point of view.

What I think we should do before stabilizing use of legacy macros from other crates:

Remove unstable macro_reexport, it's subsumed by use since Nov 2016 (#37732). PR is submitted: #49982.
Prohibit macro_exporting two macros with the same name from a crate (code doing this must be a mistake because one of the exports is lost in process). Duplicates macro/use vs #[macro_export] macro_rules are already prohibited.

Answer 40 · 2018-04-15T00:35:24.000Z

Stabilization of use_extern_macros will also collaterally stabilize paths with >1 segment in attributes (#[a::b::c]).
Resolution for such paths is total mess and the rules may change with #44690, so I'd prefer to keep them unstable for now (the workaround is to use use a::b::c; #[c]).

Answer 41 · 2018-04-16T15:41:14.000Z

Thanks for taking a look @petrochenkov! All of removing macro_reexport, probhibiting duplicate macro_export, and only allowing one-segment paths in attributes sounds great to me.

Answer 42 · 2018-04-16T15:44:15.000Z

Sounds good to me too. Hopefully we can get multi-segment paths in attributes working soon, but I don't think we should either block this on that or stabilize it before its ready.

Answer 43 · 2018-04-17T14:27:58.000Z

The final comment period is now complete.

Answer 44 · 2018-04-18T15:34:22.000Z

Ok great! I'm going to hold off on any actual stabilization here until the rest of Macros 1.2 is finished in FCP for stable.

Answer 45 · 2018-05-11T03:26:43.000Z

Can #[doc(hidden)] be made to work for macros reexported as in #37732 ?
or is this a limitation of current macros and rustdoc?

Answer 46 · 2018-05-11T10:28:12.000Z

@spearman
That's a bug, I've seen it while removing macro_reexport but haven't reported until now - #50647.

Answer 47 · 2018-05-14T01:27:14.000Z

Import regression with use_extern_macros: #50725.
Needs to be fixed before it's enabled by default.

Answer 48 · 2018-05-18T01:37:44.000Z

Will #[macro_use] eventually be phased out? Trying to figure out a sane way to re-export macros that are required for using a public macro. I suppose the idea is to eventually use paths like $crate::path::to::reexported_macro! so that a user never has to explicitly import any reexported macros, but if those macros don't themselves use $crate paths in macro invocations, then futher macros need to be imported at the root level, which is where #[macro_use] comes in handy. The alternative would be to put the macro definition together with any reexports in to a single module so it can be glob imported. Again if I understand correctly, if all macros correctly use $crate paths to refer to reexports then this won't be necessary and importing a macro by itself is sufficient to use it.

Answer 49 · 2018-05-18T06:41:33.000Z

Perhaps it is a good idea to make such non-$crate imports within macro definitions to be illegal?

In other words: when using use in a macro definition, it would always start with the crate name as a path root.

That way the problem is avoided in the first place. And it's not a theoretical problem either: I've run into issues like this where it's only after trying to use the crate that I discovered that the macro was exported but unusable without extra imports.

Answer 50 · 2018-05-18T19:44:18.000Z

@spearman #[macro_use] on cross crate macro usages is being phased out in favor of use crate_name::macro with this issue, although within a crate you'll still use #[macro_use]

@jjpe perhaps yeah! Although something like that will likely require an RFC

Answer 51 · 2018-05-19T01:18:59.000Z

PSA: if you, like me, are wondering why use will only work for extern macros and not crate-local macros, see #35896 (comment).

Answer 52 · 2018-05-19T05:36:40.000Z

Seems like I should have raised this last month but sorry I just found this thread: I would be strongly opposed to stabilizing use imports of extern macro_rules macros until we have a solution for private helper macros.

extern crate log;
use log::warn;

fn main() {
    // error: cannot find macro `log!` in this scope
    warn!("Warning!");
}

extern crate serde_json;
use serde_json::json;

fn main() {
    // error: cannot find macro `json_internal!` in this scope
    let j = json!({ "k": "v" });
}

Our whole system mostly works for now because of the way #[macro_use] brings in all the macros. Moving toward importing macros individually like items will make everything seem more broken and fragile and will be immensely frustrating for macro users and macro authors compared to the current way.

An additional constraint is that ideally we would solve this in a way that a crate could support macro_use and use at the same time. For example something like telling log::warn! to invoke $crate::log! would not work because that leaves them no way to support old compilers as well as the new use log::warn.

I just found the thread so here is an approach with barely more than zero thought behind it, but just to illustrate one possibility that fits my criteria:

#[macro_export]
#[bikeshed_also_export(__log)]
macro_rules! log {...} // forward to __log

// `use log::warn` brings both `warn!` and `__log!` in scope.
#[macro_export]
#[bikeshed_also_export(__log)]
macro_rules! warn {...} // call __log

#[doc(hidden)]
#[macro_export]
macro_rules! __log {...}

Here as long as bikeshed_also_export is somehow cfg'd away or ignored by old compilers then the same code works equally well both for #[macro_use] extern crate log on old compilers and use log::warn on new compilers.

Answer 53 · 2018-05-19T05:42:12.000Z

Tagging @rust-lang/libs because this stabilization would seem to put macro libraries in a tough situation.

Answer 54 · 2018-05-19T05:59:32.000Z

Here as long as bikeshed_also_export is somehow cfg'd away or ignored by old compilers then the same code works equally well both for #[macro_use] extern crate log on old compilers and use log::warn on new compilers.

Its not though, is it? Unknown attributes are a hard error. You could just as easily cfg the whole macro definition based on whether you intend to be imported with #[macro_use] or not as you could cfg the attribute you're suggesting, and $crate::log! is much cleaner.

Answer 55 · 2018-05-19T06:10:13.000Z

You could just as easily cfg the whole macro definition

I would not say that keeping two parallel macro definitions (one that calls private_helper! and one that calls $crate::private_helper!) is just as easy as a cfg_attr on one line.

Unknown attributes are a hard error.

I am not concerned about this because it seems easy to find a way around. Just for example the following compiles back to rustc 1.0.0:

#[macro_export(also_export(private_helper))]
macro_rules! ...

Answer 56 · 2018-05-19T07:26:33.000Z

@dtolnay I share your concern about the new use imports breaking stuff. What do you think about this proposal: https://internals.rust-lang.org/t/help-stabilize-a-subset-of-macros-2-0/7252/18 ?

Answer 57 · 2018-05-19T11:10:19.000Z

@dtolnay

I would be strongly opposed to stabilizing use imports of extern macro_rules macros until we have a solution for private helper macros.

I don'see how permitting use for macros is blocked by the private helpers problem.
We are not forcing use and not deprecating #[macro_use] or anything, it still can be used when needed.
(Yes, some adventurous people work on breaking lints like unnecessary_extern_crates, but thankfully they are allow by default.)

At the same time use_extern_macros is a prerequisite for stabilizing proc_macros/proc_macro_attributes that can't be imported with #[macro_use].

Answer 58 · 2018-05-19T11:20:01.000Z

#[macro_use] (at least on extern crate) works like a glob in use, but glob targeting only one namespace - macro namespace.

Perhaps more general feature - allowing use items to import only in selected namespace will help here as well.

// The syntax is exposition only
use a::b in value; // imports `fn` b, but not `type` b
use a::b::* in macro; // imports all macros from `a::b`

Answer 59 · 2018-05-19T11:48:01.000Z

The private helper problem is rather a blocker for recommending use over #[macro_use] in documentation.

Answer 60 · 2018-05-19T15:33:49.000Z

@petrochenkov

I don'see how permitting use for macros is blocked by the private helpers problem.

Without a provision for helper macros, I would actively discourage users from using use for macro_rules macros: PSA do not use this feature, do not get in the habit of using this feature, you will be confused, you will be sad, your code will break, authors of the macros you use will be sad, stay away, etc. The fact that I would discourage people from using this feature so strongly is the blocker.

On top of that I would make no effort to support use import of macros that I write because it would be an unreasonable maintenance burden. This is just a continuation of how I already make no attempt to support macro_use(...).

I hear you that we are not forcing use and not deprecating macro_use but if we are going to be saying do it the old way, please please never use the new way then we should not stabilize the new way until the recommendation is different.

Answer 61 · 2018-05-19T15:33:58.000Z

@petrochenkov

At the same time use_extern_macros is a prerequisite for stabilizing proc_macros/proc_macro_attributes that can't be imported with #[macro_use].

My objection:

I would be strongly opposed to stabilizing use imports of extern macro_rules macros until we have a solution for private helper macros.

I am concerned only about use of macro_rules macros. Proc macros can generally be factored in a way that does not require further expansion of helper macros.

Answer 62 · 2018-05-19T15:36:32.000Z

The private helper problem is rather a blocker for recommending use over #[macro_use] in documentation.

Is this saying we should stabilize use but hope nobody finds out about it...?

Answer 63 · 2018-05-19T15:55:48.000Z

@golddranks if that can be implemented, and if we can isolate the behavior to macro_rules macros, then that would be terrific and solves the problem. If I understand correctly, your idea is that just before a "cannot find macro `m!` in this scope" error, and only if the token m originated within the definition of a macro_rules macro, then it should make a last ditch effort to resolve m! within the crate that originated the m token. 👍

Answer 64 · 2018-05-20T00:50:58.000Z

@dtolnay

I am concerned only about use of macro_rules macros.

I think it's possible to stabilize/enable the general macro importing mechanism while keeping imports of macro_rules macros sort of "not working" without a feature gate.
It's not entirely trivial though, e.g. you can't just gate an import if it points to macro_rules, then e.g. use std::panic; would stop working.

#[bikeshed_also_export(__log)]
What do you think about this proposal: https://internals.rust-lang.org/t/help-stabilize-a-subset-of-macros-2-0/7252/18 ?

To clarify my priorities, I'd like to avoid:

Making import resolution more complex. It's already too complex.
Implicitly adding use my_crate::__log; on use my_crate::log; makes import resolution more complex.
I can't predict how exactly it will affect the fixed-point import resolution algorithm, but it will certainly become stuck more often because every import turns into a glob in some sense because we no longer have guarantee that use a::b; can import only items named b, if b is a macro with "linked" helpers.
Making hygiene more complex. It's already too complex.
@golddranks proposes some new bespoke hygiene scheme in which call-site hygiene for macro_rules falls back to def-site hygiene.
(Note: Currently macro_use on crates behaves sorta kinda like an additional prelude, affects only "relative" names (scope-based resolution), i.e. X, but not a::X or ::X, so it doesn't participate neither in import resolution nor in hygiene.)

So, my recommendation would be to:

For new compiler versions: do not introduce new features into import resolution or hygiene systems, fully stabilize use_extern_macros, solve the private helper problem with $crate::__log, recommend use over macro_use.
For old compiler versions if/while they need to be supported: on the user side #[macro_use] is the only way, on the library author side cfgs are somehow employed to generate both __log and $crate::__log macro paths depending on the compiler (sorry :( ). Or use is discouraged for a few compiler versions, then the switch from __log to $crate::__log happens.

Answer 65 · 2018-05-20T01:11:31.000Z

In my opinion your recommendation does not meet the bar for the cross-edition interoperability story that people envisioned. To make a library that works equally well on 2015 and 2018 (as Serde and many other libraries would want to do for some reasonable transition period) we would be telling people:

You add a build-dependency on version_check. The dependency adds 0.7 seconds to compile time.
You add a build script, build = "build.rs". The build script adds 0.6 seconds to compile time.
In the build script you use version_check to determine whether the compiler version is sufficiently new to use $crate::private_helper!. The third-party version_check crate implements this by using the std::process API to shell out to rustc --version, then parsing the numbers out of a version string that looks like rustc 1.27.0-beta.5 (84b5a46f8 2018-05-15).
If sufficiently new, print to stdout the magic string "cargo:rustc-cfg=crate-macros".
Duplicate the definition of your public macros and any private helpers transitively called by them. Change one copy to use $crate::private_helper! and tag with #[cfg(crate_macros)].
Attempt to keep the duplicated code in sync as your crate evolves.

From the library authors' perspective this seems like a tough sell. For comparison the other approaches in this thread, as tricky as they would be to implement, look like this. Some sort of bikeshed_also_export way:

You make a one-line change
```
- #[macro_export]
+ #[macro_export(...)]
```
and your macro works flawlessly through #[macro_use] on all Rust compilers back to 1.0.0 and flawlessly through use on all sufficiently new compilers.

And @golddranks' hygiene way:

(There is no step 1. Your existing macros already work flawlessly through #[macro_use] on all Rust compilers back to 1.0.0 and flawlessly through use on all sufficiently new compilers.)

Answer 66 · 2018-05-20T01:28:26.000Z

To make a library that works equally well on 2015 and 2018 (as Serde and many other libraries would want to do for some reasonable transition period) we would be telling people:

AFAIK this is a stabilization in both editions; we're talking about people who don't upgrade to the compiler version 1.29.0, not people who don't upgrade to the 2018 language edition. While libraries often support people on old compiler editions, it is a very different thing to require people to upgrade their compiler than to require that they update their code.

Answer 67 · 2018-05-20T01:39:03.000Z

The way I mean this is: for some time Serde will want to work equally well on rustc <1.29 (which only supports 2015-style macro_use) and rustc >=1.29 (on which people expect to be able to use 2018-style nice new features like use). I agree that upgrading your compiler and changing your code are different things but telling a library to follow unappealing steps 1/2/3/4/5/6 or else release a breaking change for no other reason than we couldn't figure out macro imports -- seems not great.

Answer 68 · 2018-05-20T01:52:30.000Z

@dtolnay its important to be clear here: with the $crate:: solution, you can continue to use #[macro_use] as long as your compiler is a recent enough stable (1.29 or whatever). What you call the "2015-style" system will still work, because macros can be compatible with both "styles." When you say "a breaking change" I am surprised - I did not know that serde adopted the position that increasing the minimum Rust version it required was a breaking change. Is that the case?

Answer 69 · 2018-05-20T02:00:24.000Z

The minimum required compiler version from Serde 1.0.0 through today has always been rustc 1.13. So far we have had no trouble catering to users of new compilers using only 1.13's feature set. This would be the first time in 15 rustc releases that it becomes complicated to support users of new compilers -- which I guess is driving my concern here.

I am nominating for the libs team to discuss how we imagine the library situation playing out over the rest of the year. Around 1.29 or whatever are we expecting most libraries to drop support for compilers older than 1.29, whether through a massive round of breaking changes or by patch versions that aggressively push people to upgrade compilers?

@rust-lang/core may be interested as well: this affects peoples' perception of the stability of the language which may already be a sensitive topic around publicity of the edition.

Answer 70 · 2018-05-20T12:35:39.000Z

@dtolnay
If something like rust-lang/api-guidelines#123 (comment) works, it would be great.
But I wouldn't personally mind a documentation-only solution "my macro library doesn't support importing with non-glob use and macro_use(named)" + possibly internal future-proofing with dummy internal helpers, until minimal compiler version is bumped to 1.28-1.29 naturally in the next year or two.

Now, let's assume that the combination "supporting use + simultaneously supporting older compiler versions + using same code for both" is absolutely critical and we need a language solution for it, then:

It's important to understand that the solution is a temporary hack needed only during some transition period, it will go into the deprecation land later. At the same time this hack needs to be urgently implemented as insta-stable (a bit risky).
We are limited by syntax macro_export(a, b, c) because it's accidentally accepted by older compiler versions or no new syntax at all, so we can't solve the problem by some new more generally useful mechanism, like hygiene opt-in for macro_rules (the opposite of #47992).
I think, this means we need some absolutely minimal and isolated solution getting the job done, but not interacting with other features (import resolution in particular) and complicating the whole system.

I think the most simple and local solution would be:

We have a single-segment fn-like macro call my_helper!(...) for which name resolution fails as determined.
Then we know hygienic context of the my_helper identifier, from which we can figure out definition of the macro in which it was originally written, in particular crate and kind of that macro.
Then, if the macro kind is macro_rules, we either 1) try to resolve my_helper as $crate::my_helper with that crate, i.e. automatically doing the library author's job from my previous recommendation (simpler) or 2) try to resolve my_helper at def-site of that macro (may be more complex for macro_rules (as opposed to macro), I'm not sure the necessary infrastructure is in place).
The resulting resolution is optionally filtered by the name list from macro_export(a, b, c). Increases complexity, not strictly necessary.
We'll be able to emit a warning for this resolution scenario later if necessary.

ping @jseyfried who implemented a couple of similar hacks to support legacy behavior of macro_rules (in case he's still reading messages from github).

Answer 71 · 2018-05-21T03:48:52.000Z

Thanks for bringing these points up @dtolnay, always good to know about them regardless of where we are on the stabilization timeline!

To make sure I understand the issue, the point you're bringing up @dtolnay is basically that this code doesn't work today?

#![feature(use_extern_macros)]

extern crate log;

use log::info;

fn main() {
    info!("test");
}

If that's the case, that does indeed seem worrisome! I'm not sure, though, that it necessarily implies we should delay this or add more features to macros 1.0. For example we know that all code will continue compiling as-is (as it uses #[macro_use]). The only question is how we actually signal this transition.

So far we've been saying that you should replace #[macro_use] use use krate::macro_name;, but what if we instead suggested the true replacement for #[macro_use], namely use krate::*;? That's actually (modulo namespaces) what literally #[macro_use] is doing today (whether it looks like that or not). While I agree that a glob import is indeed unsightly it's also why we're developing a new macro system!

This to me seems like it leaves us with two downsides:

We're recommending a "subpar" experience still. It's better than #[macro_use] but use log::*; isn't great either.
The "correct" experience effectively requires an opt-in from all macro authors. Every single crate exporting a macro will have to go back and rewrite the macro to do one of two things:
1. If backwards-compatibility is important, everything goes into the same macro invocation. (this I don't think works with log::info!() though?)
2. Otherwise invocations of log!() is replaced with $crate::log!()

It seems to me that the #[macro_export] solution you're thinking about still has the problem of "all current macro authors must go back and maybe edit their code", right? The main difference, I believe, is that the backwards-compatible solution is much nicer in that you can just list dependent macros.

In other words the delta over where we are today (if we stabilize), is that macro authors who both go back and take a look at their macro-exporting crates while also considering backwards-compatibility don't have to duplicate their definitions. I think though the cost of stabilizing this is still the same?

To me that seems like a wortwhile tradeoff to make. The primary use case for this features is macros 1.2 which is targeted to be stable at the same time as this feature. It's sort of secondary that we expect macro_rules! to transition to this as well, but I think it's totally fine to basically just delay our messaging here. I think we can "fix" this in a backwards-compatible way in the sense of making authoring macros a bit nicer, but I don't think we should halt or delay the stabilization of 1.2 because macro_rules! isn't so great (as it can't possibly be worse than today which we're already "happy with")

Answer 72 · 2018-05-21T04:44:43.000Z

@dtolnay and I had a chat about this on IRC, and I'll try to summarize here.

Let's say we stabilize this feature in 1.29. There exist popular crates which will maintain compatibility with pre-1.29 compilers, for example log and bitflags. We cannot as-is reasonably transition users of these two crates to using the module system instead of #[macro_use]. One of two possible options seems like a way to avoid this pain:

As a first option, we would not stabilize using the module system to import macro_rules macros. This means that you'd use this feature to import procedural macros, custom attributes, etc. You would not, however, use it to import macro_rules macros. This means that #[macro_use] extern crate foo; is still a thing, an obvious downside.
Instead we stabilize some sort of feature to allow the log and bitflags crates to get implemented in a "reasonable fashion". This would be along the lines of solutions like proposed by @dtolnay and @petrochenkov.

Alternatively we could stabilize this feature and simply not announce it. Instead we could wait until a sufficient mass of "popular crates" transition to requiring 1.29 or future compilers, in which case everything is right as rain and we can sound the trumpets at that point.

Answer 73 · 2018-05-21T09:05:41.000Z

@alexcrichton So the proposal for "last resort defsite hygiene" isn't considered feasible here? @petrochenkov already that he considers it making hygiene too complex (#35896 (comment)) but I'd like to hear your opinion too.

Answer 74 · 2018-05-21T10:03:25.000Z

@golddranks

So the proposal for "last resort defsite hygiene" isn't considered feasible here?

That's item 2. from @alexcrichton's list.
"Last resort to $crate::my_helper" (#35896 (comment)) should be simpler than "last resort to defsite hygiene" due to subtle differences like this:

// Def-site resolution for `public_macro` would refer to this private non-exported macro.
// This kind of legacy interaction is not currently supported even for `macro` items.
macro_rules! my_helper { ... }

#[macro_export]
macro_rules! public_macro {  ... my_helper!()  ...  }

// Macros with same name can shadow each other in a module.
// `$crate::my_helper` refers to this exported macro.
#[macro_export]
macro_rules! my_helper { ... }

That said, I'd still prefer not doing this.

Answer 75 · 2018-05-21T10:22:30.000Z

Ah, I see. Pardon my confusion. Indeed, that is simpler.

Answer 76 · 2018-05-23T21:47:00.000Z

Should this currently be working for macros exported by std, testing the below code gives error[E0432]: unresolved import `std::assert`:

use std::assert as std_assert;
std_assert!(2 == 3);

Answer 77 · 2018-05-23T22:34:26.000Z

@Nemo157
#48813 made assert built into the language so it's no longer defined by libstd.

Answer 78 · 2018-05-24T07:58:56.000Z

@petrochenkov interesting, follow up question then.

Should built-in macros somehow act as if they were exported from libstd/libcore for this feature?

It seems that stabilizing this feature would make it impossible to transition any other macros from being real macros into compiler builtins, otherwise any uses like above would stop compiling.

One way might be to keep the macros as real macros that expand into the compiler builtins, something like macro_rules! assert { ($($t:tt)*) => { __builtin_assert!($($t)*) } }. That way the macros themselves would still be able to participate in the normal naming/modularisation scheme.

Answer 79 · 2018-05-24T09:04:31.000Z

macro_rules! assert { ($($t:tt)*) => { __builtin_assert!($($t)*) } }

Yes, that's probably something we should do when moving a macro from the library to the language.

Answer 80 · 2018-05-24T20:50:36.000Z

Is it correct (forgive me if I am repeating) that if we supported $crate::bar!() as a way to invoke macros, then you could write macros that use private helpers without requiring users to manually import them? (Users would of course have to be using a new enough compiler to support that.)

If so, I definitely feel that just supporting that syntax is a viable solution. If a crate X wants to retain compatibility with older compilers, it means that its consumers just have to use #[macro_use] extern crate the_crate. Suboptimal but not a total disaster.

(I think we should make some effort at promoting community wide "compatibility ranges" when it comes to rustc versions, as well, but that is perhaps a discussion best had elsewhere.)

Answer 81 · 2018-05-24T20:52:26.000Z

That said, @nrc and I were talking on discord and I had an idea that I kind of like which I wanted to write down. It's a variation on a proposal that @nrc raised.

TL;DR: We try to make the ability to impor a macro via use a new feature which some macro-defining crates will have opted into and others will not have yet done. We do not attempt to "retrofit" existing crates into this model without some opt-in.

The first part is to make it "opt in" to have macros usable via use — existing macros that are not changed would still require #[macro_use] extern crate the_crate; to be used. This means then that crates can choose when to enable this feature based on whether they use helper macros and whether they are ok with requiring that their consumers have a newer compiler (e.g., one that can support the new edition).

Now, how does that opt in look? I'm actually going to spin two variants of this proposal, one more aggressive and one less aggressive, because I don't quite know how hard each one would be to do.

The more aggressive variant

We allow macro_rules! macros to be declared pub. This would be the only macro that is allowed to be declared as pub. Ideally, you would be able to declare it as pub anywhere. This would also mean the macro does not use the default macro_rules! mechanism but instead opts in to the more lexical mechanism -- also within the crate. In other words, it works just like pub macro was meant to work.

(Maybe, in the new edition, we can also make macro_rules! foo without pub work using the lexically scoped rules? We might not be able to write a migration lint for that, though, but the only real problem is in the case of shadowed macros, I guess, and that seems like a corner case that we could detect and require manual intervention? Maybe?)

Now, for compatibility, we can still permit #[macro_export] on public macros. This means that a crate can do something like this:

pub mod macros {
    #[macro_export]
    pub macro_rules! my_macro  { ... }
}

and now folks with a new compiler can do use the_crate::macros::foo or use the_crate::macros::*, but folks with an older compiler can still do #[macro_use] extern crate the_crate;.

This does not solve hygiene. If you have private "hidden" macros like _foo!, they are "observable" to your users, who must either import them manually or use a glob import. I think though that having the option to do use the_crate::macros::* is less .. unsightly than requiring use the_crate::*.

As a bonus, macro-rules within a crate work like all other items, just like we always wanted. Huzzah. (Naturally #[macro_use] on modules would have to be .. deprecated? The interactions here may just be too complex.)

The less aggressive variant

Instead of pub macro_rules!, we could do #[macro_export(pub_use)] to signal that you want this macro to be imported via use and not via #[macro_use]. This would be a breaking change. Perhaps there is some other way to say you want both: #[macro_export(macro_use, pub_use)]. Intra-crate, nothing changes (as today).

Observations:

These proposals basically add a new feature: macro imports using use. We consider this the "recommended" way to use macros, but the old way continues. Naturally this means one can support older compilers just by not adopting the new feature (or by adopting both).

There are some problems that are not "fully solved" in some sense -- in particular, if you want to have private helper macros and retain compatibility with older compiler versions, you still have to force your consumers to use #[macro_use] extern crate the_crate;. But that feels ok: it often happens that crates stick to old idioms for some time, and this is just an instance of that (as stated in the TL;DR above).

Answer 82 · 2018-05-24T21:46:42.000Z

@nikomatsakis

Is it correct (forgive me if I am repeating) that if we supported $crate::bar!() as a way to invoke macros, then you could write macros that use private helpers without requiring users to manually import them?

$crate::bar!() is already supported with #![feature(use_extern_macros)], you just can't use macros expanding into it in the same crate, but that shouldn't be a big problem for libraries that produce such macros rather than consume them.

EDIT: ... and I'm pretty sure that we can provide support for $crate::my_helper!(...) in the same crate with exactly same semantics as $crate::my_helper!(...) from other crates without any other changes, if its lack causes too much pain to library authors.

Answer 83 · 2018-05-24T21:51:45.000Z

I'm not quite sure about the motivation for proposal in #35896 (comment).
It adds some features, edition-breaks macro_rules!, but doesn't actually change the situation with @dtolnay's concern - #[macro_use] still has to be used for some time and use still can't be used everywhere for some time - this is already true without any changes.

Answer 84 · 2018-05-24T22:33:00.000Z

Ok I had a bit more discussion with @nikomatsakis on discord about his previous proposal and some things I wanted to write down...

The crux of the problem here is pre-1.29 compatibility. AFAIK all features of use_extern_macros work great, and the problem only arises when a crate wants to support using its macros via use krate::foo and also work with #[macro_use] extern crate krate; for pre-1.29 compilers.

Now this aspect of crates is actually more far reaching than just pre-1.29 compatibility. For example the whole feature here is using macros from other crates, ones that you're possibly not writing yourself. In that situation the upstream crate may have a different development policy than you, supporting different versions of the compiler. This can run the risk of having a network effect where popular crates like bitflags and log may be laggard in enabling usage of their macros via the module system and specific paths (use log::info; vs use log::*;).

The first question then is is it a goal of this issue that we want to support this pre-1.29 compatibility use case. It's not an easy one to achieve, but the cons of not supporting this are:

Crates which do want to support pre-1.29 compatibility really do have to bend over backwards. One option is to rewrite macros to only use themselves recursively. This can be a maintenance burden as well as untenable in the long run. Alternatively they need to use $crate::foo!() but use compiler version detection at build time to select one of two definitions of the macro. This duplication can also be a maintenance burden. All in all, this con is that it's likely pretty few crates which want pre-1.29 compatibility are likely to be usable with macros and the module system.
If a number of popular crates (e.g. log, bitflags, lazy_static, etc) do not support the module system then this feature may feel "incomplete". For example you'll have to remember which crates to use the module system with and which crates to use #[macro_use] with, and we'll risk being in a weird transition period for a few cycles (maybe longer?)

And I think I'm forgetting the last one! In any case though it's also worth pointing out that it's not clear what the impact is here. For example we don't know which popular macros don't work with the module system basically as-is or are easily modifiable. Additionally it's always possible to require use log::*; which will work very close to what #[macro_use] does today.

So ok, let's say we do want to consider pre-1.29 compatibility an option. So far it sounds like there's two plausible (if not-so-fun-to-implement) solutions:

@dtolnay's idea to exploit the fact that older compilers ignore extra tokens in #[macro_export ( ... )] to add annotations necessary to get use of the macro working
@petrochenkov's idea to exploit name resolution errors and fall back to attempting to resolve macro invocations at the macro's definition (pseudo-hygiene)

If we decide to go one of these two routes then we'll want to avoid stabilizing modules and the macro_rules! system for 1.29, but we'll probably want to continue to stabilize procedural macros and attributes.

And finally, one last thing worth mentioning. No matter what we do it's likely that when we stabilize macros and the module system it won't be usable with the large majority of macros already in existence in the ecosystem. In other words most macro definitions will need source level changes (in one way or another) to work with the new system (like using $crate::__helper!()). In that sense we may want to consider some possible tweaks (like @petrochenkov's idea) to reduce the impact here and make more macros usable-by-default

Answer 85 · 2018-05-24T23:05:07.000Z

Just to be clear: is the idea to fully deprecate macro_use attributes with crate-local macros too, eventually? Having two different syntaxes depending on where the macro is defined seems like a really bad state to be in for any appreciable time. Certainly not something I would think should be stabilised.

Answer 86 · 2018-05-25T01:03:21.000Z

@alexreg eventually macro-macros (macros 2.0) is the way of the future which fully integrates with the module system, there's currently no interim plans to fully deprecate #[macro_use] with macro_rules! for within a crate

Answer 87 · 2018-05-25T01:12:13.000Z

@alexcrichton Fair enough; thanks for clarifying.

Answer 88 · 2018-05-28T00:58:08.000Z

I made a rough implementation for the "macro_rules! helper fallback" as described in - #35896 (comment).

As it turns out, it's not "just a fallback" it's "fallback in the middle of fixed-point resolution" again.

To truly rely on the fallback we need to expand everything before doing it, but expansion cannot progress without performing the fallback.
The workaround is similar to other situations with fallback in resolution - optimistically perform the fallback even if the resolution is undetermined (i.e. there are macros that can potentially expand to #[macro_use]), then detect "time travel" post-factum and report an error if new #[macro_use] imports appear that would be preferred to the fallback.

I think the helper_fallbacks.contains(name) check on global_macros.insert(name) should catch all the cases of "time travel", but I'm not 100% sure.
EDIT: Unfortunately it doesn't, see the second commit.

It's certainly a trade-off, but now my feeling that supporting this would harm the language long-term in favor of short-term version-migration benefits is stronger than before.

Answer 89 · 2018-05-30T15:02:53.000Z

Status update: #51145 addresses the last known regression from enabling use_extern_macros.

I propose stabilizing use_extern_macros without providing a language solution (like #35896 (comment)) for the macro helper problem discussed above.
I think we should do it now, so $crate::my_macro becomes available on stable as soon as possible.
We could also do the backport of the stabilization PR (#50911) and bugfix PRs (#50355, #50760, #50908, #51145) to beta, then it will be available on stable starting with 1.27.

Answer 90 · 2018-05-30T16:44:11.000Z

cc @rust-lang/lang on this last comment; nominating for meeting as well.

Answer 91 · 2018-06-05T01:45:49.000Z

Here is a hopefully clearer and less shouty writeup of my perspective in anticipation of the lang team discussion.

At issue

Enabling use of individual macro_rules macros from another crate.

use log::warn;

Background: new features

We are all used to new language features and have seen some fantastic ones recently. From the point of view of a particular library, new features generally break down into one or multiple of:

Features that make it possible to solve some problem that was impossible to solve before in Rust. For example union allows a sys crate to expose a signature ABI-compatible with some C function that passes unions. Before Rust 1.19 it just couldn't have provided a binding for such functions. This is great and expands the world of problems that Rust is suitable for.
Features that prompt users to redesign the API of a library that was previously designed and working. For example Rust 1.20 added associated constants, prompting a redesign and major version bump of the bitflags crate. Bitflags existed and worked decently well before 1.20, but the new feature provided a materially better way to solve the problem bitflags intended to solve. The authors took into account the benefit of the improved API as a tradeoff against the cost of rolling out a change to the API. This is healthy and it is great when language features are adopted because of the API design improvements they make possible.
Ergonomics improvements that make Rust code easier to write, read, and maintain without much affecting API design. An example of this is default binding modes in match. These are great and a quality of life improvement for beginners as well as experienced users.

Background: foundational crates

A defining characteristic of a foundational crate like Serde is that only type-2 features are relevant to us.

The library addresses a particular problem domain, so unless we are expanding the problem domain, type-1 features geared toward things that couldn't be done before are not relevant. The library does a thing, so the thing it does is not one of the things that cannot be done prior to the new feature.
We pay close attention to type-2 features. For example if some future version of generic associated types makes it possible to do what Serde does using a radically nicer API, we would redesign the API and release the improvement as a breaking change.
We don't pick up type-3 features until such time as we bump the required compiler version for type-2 reasons. This has to do with how much weight is on one side of the tradeoff that exists between benefit of ergonomic improvements to development within the Serde codebase, versus cost of pushing a compiler upgrade to users. Regardless of how big an ergonomic improvement may be, there exists some threshold of number of downstream users beyond which their upgrade friction outweighs our internal ergonomic benefit.

Why the change is concerning

The thing that is unprecedented about use imports of macro_rules is that, while it does not break existing library APIs (a non-starter), it does break users' expectations of existing library APIs.

That is, compiler developers would categorize the change as type-2 in the sense that libraries today expose a working API that looks like #[macro_use] extern crate log, and the new feature gives them a way to expose a nicer API that behaves more like imports elsewhere in the language, use log::warn, if they choose. It is easy to see this as no different from a canonical type-2 new feature like associated constants. The authors of bitflags would have the duty of deciding whether the API improvements afforded by the new feature outweigh the cost of rolling out a change to the API.

But in this aspect the point of view of compiler developers diverges from that of library developers and library consumers. Unlike compiler developers, everybody else does not perceive the API of a crate as "it exposes such-and-such macro importable through #[macro_use]". Rather, they perceive the API as "it exposes such-and-such macro" and independently, "here is how macros in Rust are imported."

The distinction is important because it deprives library authors of the choice of following their ordinary type-2 decision process. As an author of a crate that exports macros I cannot weigh the two choices and decide whether to stick with #[macro_use] for now or jump to use. Outside of my control, and regardless of anything I may write in documentation, users will expect to be able to use my macros because that is how they understand macros are imported in Rust.

When they write a use and receive the following error:

cannot find macro `__some_internal_helper!` in this scope

then depending on their personal experience the user will either blame Rust ("macro imports sometimes work and sometimes don't work, what am I doing wrong?") or blame the library ("I tried to use your library and your code does not compile"). Either way the ecosystem feels flaky and perpetually broken in a way that it doesn't today.

2018 transition

The whole thing is only a problem during a brief 2018 transition period right?

This is true. The transition period only lasts until the macro-exporting crates people use have bumped their minimum supported compiler version and moved to invoking helpers through $crate::helper! syntax.

It is hard to say how long that would be because library authors have differing opinions about how to do this correctly and each library will have hard decisions to make: Do we force the ecosystem through a serde 2.0 upgrade? Do we prolong the transition period during which our API feels flaky and broken when people try to use it in the ways they expect? Do we aggressively force a compiler upgrade on users by breaking their builds? Regardless of which way you would decide, notice how all three of these result in an ecosystem that feels unstable.

It is possible that our attention to stability has over-indexed on one aspect of stability: the "lifespan" of code meaning how long before a compiler change breaks the code and it no longer compiles (which we promise is never, with some well-reasoned exceptions). This thread brings up "healthspan" as a different aspect of stability, how long before a compiler change breaks users' expectations around code by dropping it into a "transition period" that requires code changes to escape out of.

Large codebases

My experience in some large codebases leads me to value lifespan and healthspan as equally important. If it is expected that compiler upgrades are going to require ongoing maintenance investment in the form of periodic source-level changes to escape out of "transition periods", the value of never being actually broken by a compiler upgrade is greatly diminished.

In a large codebase we require the ability to write a library, finish it, and trust that it will age well until a type-2 redesign of its API.

Also the larger the codebase, the longer it takes to adopt compiler versions. Suppose that through perfectly legitimate inference breakage or soundness fixes we break X% of source lines of code every release (where X is a number much less than 1%). Those take increasingly long to work through. Also large codebases are increasingly likely to hit blocking perf regressions, again taking time to resolve. All of this means that for a large codebase there is value in foundational libraries supporting a generous range of old compilers.

Path forward

The notion of healthspan is why I was particularly excited about the approach in @petrochenkov's prototype #35896 (comment) which entirely avoids breaking developer's expectations of existing APIs. That is, we would change their expectations, but we would not break them because their new expectations of being able to use log::warn would work seemlessly. I would love to see something like this adopted. I am grateful that you took the time to develop the implementation and I think it was an important thing to try. Thanks also to @golddranks for the idea and internals forum discussion in #35896 (comment).

That said, if the compiler team believes that the solution there is not tenable then obviously we can't ship it.

Long-term I expect compiler versions understood by Cargo will tip the scales heavily in favor of rapid adoption of new language features including by foundational libraries. I know Josh has been working on this and I am very excited about progress there.

Answer 92 · 2018-06-07T20:39:11.000Z

OK, we had a long discussion. I think there was general consensus that current state of affairs is indeed a cause for concern and that we would rather not stabilize the features "as implemented". We did not reach any proposal that had a clear consensus. So I'm going to present a few possible routes. You will find a DropBox paper with our notes at this link.

Two goals

One of the things that we realized is that there are two features at play here:

Extern crate elision: Removing the need to write extern crate.
- In particular, with the new module system proposal, there isn't really any other reason to write extern crate besides bringing in macros.
Selective import: Basically, the ability to import individual macros, and generally to treat macros as "ordinary items". This also means that you get to treat procedural and macro-rules macros identically.
- Note that there remain distinctions that are important, for example macro-rules crates cannot be imported within a crate via use today. (One question that was raised was: how hard would this be to do anyway?)

For some people, one of these goals may be more important than the other, which influences the shape of a satisfactory solution.

Concerns

There are various concerns to be balanced:

Splitting the ecosystem. Clearly, if serde drops support for older compilers, that will cause a certain amount of churn (as @dtolnay elegantly summarized here).
User confusion. If we tell users that they can import macros, but then doing use log::debug gives an error, that's suboptimal. Furthermore, we should ideally be able to guide users relatively clearly on how to import any given macro.
Technical debt. We are particularly concerned with the idea of adding new modes of fallback into name resolution. That never seems to end well, and we don't really have a strong "model" for how name resolution works to convince ourselves that it will be ok. Plus most "fallback-like" proposals tend to sound like hygiene -- which we know is hard and has a bunch of subtle concerns to be addressed.

Some possible routes

extern crate 4eva. One point of view is that this whole conundrum is evidence that trying to "blur the line" between "macro-rules" and "macros 2.0" was a mistake. We should back off from supporting this feature at all and instead just continue to have people use #[macro_use] extern crate foo; as they ever did. Procedural macros would still use use. The main point here is that any attempt to bridge this gap will result in technical debt and language complexity that we can't get rid of. Obviously taking this route solves neither of the two goals.

Macro glob. We considered @petrochenkov's proposal of having some kind of macro glob form (some syntax proposals below). This could well address the "extern crate elision" goal but does not address selective import. This can be couple with pub macro_rules! as well to address select import. Some syntax proposals:

#[macro_use] use foo; -- presumably this would imply the old "no scope" behavior though?
use foo::macro::*;
use foo::macro *;
use foo::*!;

One thing to consider here is other namespaces. For example, we might (in the future) want to support use foo::type * or use foo::impl *.

Public macro-rules. The idea here was to have macro-rules macros opt-in to use import as a way to signal that this is how users should use them. This aims to address the selective import goal primarily while trying to avoid user confusion -- that is, it effectively defines a new set of macros, kind of "macros 1.5", which are macro-rules macros that can be brought in through use. Said macros should use $crate to invoke helpers and so forth. This allows us to give relatively clear errors: for example, trying to import an "old style" macro can result in a message like "older-style macro-rules macros can only be imported with an extern crate" (or perhaps a macro glob, if we offered that).

recurse-within-crate. We did have one fresh idea for how we might try to "have our cake and eat it too". We were thinking that we could potentially add an annotation like so (obviously the precise name is TBD):

#[macro_export(recurse_within_crate)]
macro_rules! foo { .. }

The effect of this annotation would be that any ! that appears within the definition of foo (to be determined by span information) is always resolved against the source crate (macros that appear in the arguments of foo would expand as normal, of course). This is different from @petrochenkov's proposal in a subtle, but important way: it is not a fallback mechanism. Rather, we would resolve only against the source crate, as this is usually what crates need anyway (the only exception would be macros that are using higher-order macros or something).

This allows crates like log to upgrade while still working with older versions of rustc, but new users can import crates without pain. It doesn't seem like that much technical debt to bear (my rule of thumb, at least, for name resolution is that fallback is bad, but hard choices like "always resolve from here" are fine).

Conclusion

I don't really have a conclusion, but it seemed like the tendency in the meeting was to want to push towards one extreme or the other:

Either we can make it so that you can use macros across crates and existing crates can be upgraded to work with that (e.g., via the "recurse within crate" proposal);
or, we try to back off and draw clearer lines -- but exactly what to back off to is not clear.

Answer 93 · 2018-06-07T20:39:58.000Z

@petrochenkov

I'd very much appreciate your feedback on this "recurse-within-crate" idea. I'll "quote" it from my previous comment here for ease of reading. =)

recurse-within-crate. We did have one fresh idea for how we might try to "have our cake and eat it too". We were thinking that we could potentially add an annotation like so (obviously the precise name is TBD):

#[macro_export(recurse_within_crate)]
macro_rules! foo { .. }

The effect of this annotation would be that any ! that appears within the definition of foo (to be determined by span information) is always resolved against the source crate (macros that appear in the arguments of foo would expand as normal, of course). This is different from @petrochenkov's proposal in a subtle, but important way: it is not a fallback mechanism. Rather, we would resolve only against the source crate, as this is usually what crates need anyway (the only exception would be macros that are using higher-order macros or something).

This allows crates like log to upgrade while still working with older versions of rustc, but new users can import crates without pain. It doesn't seem like that much technical debt to bear (my rule of thumb, at least, for name resolution is that fallback is bad, but hard choices like "always resolve from here" are fine).

PS, if this idea was raised before, then I missed it, and I apologize.

Answer 94 · 2018-06-07T23:19:39.000Z

I was pondering @dtolnay's breakdown of features and I wanted to relate it to the various things contained in this summary comment. @dtolnay classified features as "type 2 or 3" (I'm ignoring type 1):

[Type 2:] Features that prompt users to redesign the API of a library that was previously designed and working.

[Type 3:] Ergonomics improvements that make Rust code easier to write, read, and maintain without much affecting API design.

I think this is an insightful way of breaking things down, and I think it is useful to look at the proposals in those terms.

(As @dtolnay said, the current implementation of this feature doesn't really fit this breakdown. It's not a feature that crates choose to use or not to use. All exported macros are opted into it, for the convenience of their consumers, but many crates are not made to be used that way.)

The "pub macro-rules" proposal, which aims to make the new import style "opt-in", I think is an attempt to repackage this feature as a "type 2" feature: something that may be worth overhauling your crate in order to support, because users expect it (but not something you must support).

The "recurse_within_crate" proposal aims to exempt the feature from this breakdown. That is, this remains a feature that crates must use, but we provide a way for the vast majority of them to do so seamlessly (presuming they update their source if it is necessary).

It's not clear to me whether we should be concerned about crates that are never updated, which will have their macros "exposed" but which may not work — it depends how many such crates there are and whether they use helper macros. I suspect this will be a minor problem in practice. If we were truly paranoid, though, we could say that plain #[macro_extern] means that the macro can only be used the old way, but that one can opt-in to import, e.g., by choosing between #[macro_export(recurse_at_invocation)] (today's default behavior) or #[macro_export(recurse_at_definition)] (the new behavior, renamed).

(Actually, I think I like this way of declaring that a macro is "use"-able better than writing pub, since it does not suggest that it will work intracrate.)

Answer 95 · 2018-06-07T23:42:08.000Z

@nmatsakis:

The effect of this annotation would be that any ! that appears within the definition of foo (to be determined by span information) is always resolved against the source crate (macros that appear in the arguments of foo would expand as normal, of course).

How would that play with 'dynamic' macro calls, like nom uses heavily?

/// Wraps a parser in a closure
#[macro_export]
macro_rules! closure (
    ($ty:ty, $submac:ident!( $($args:tt)* )) => (
        |i: $ty| { $submac!(i, $($args)*) }
    );
    ($submac:ident!( $($args:tt)* )) => (
        |i| { $submac!(i, $($args)*) }
    );
);

It's parsed/broken up in argument position, not expanded - then expansion happens in the body, from an ident given by the caller, but with different arguments.

Answer 96 · 2018-06-08T00:37:12.000Z

@eternaleye

How would that play with 'dynamic' macro calls, like nom uses heavily?

Indeed, we discussed nom as an example in the meeting. My assumption would be that nom would not opt into that "recursive-call" feature. The question is whether nom also uses "hidden" helper macros that users aren't supposed to know about. Or -- at least -- I thought that was the important question. Thinking about it now, I think maybe it doesn't matter so much if the helpers are hidden or not. If we're going to support select import, you'd still like to be able to only import the things you directly reference.

So at the end of the day, the question is just whether the crate has both macros that invoke one another and the need invoke macros from the user's crate (likely via indirection). If so, the mechanism is going to be approximating hygiene to some degree, and that seems (to me) to be approaching a line of "too much complexity".

That argues I think against permitting selective import of macros, at least without some explicit opt-in.

Answer 97 · 2018-06-08T00:59:24.000Z

That argues I think against permitting selective import of macros, at least without some explicit opt-in.

That is, to clarify: if there were some opt-in mechanism, then nom could either elect to do nothing, or rewrite to use $crate:: and opt-in to permitting selective import.

Answer 98 · 2018-06-08T05:16:22.000Z

If the "extern crate 4eva" approach is taken, that means that only macro_rules! macros are forever bound to being imported by #[macro_use], correct? When we have Macros 2.0 declarative macros (i.e. the macro keyword), that would share normal use-based imports with Macros 2.0 procedural macros, correct? If I'm understanding all that right, then I'm in favor of sticking with extern crate/macro_use. It'd be nice to have a clean split between the old system and the new system. Shipping use-based imports only with Macros 2.0 (in its forms) would be a nice carrot on a stick to encourage people to port their macros to the new system. And of course it avoids the confusion of having multiple ways to import macros from the old system.

Answer 99 · 2018-06-08T20:28:04.000Z

@jimmycuadra

If the "extern crate 4eva" approach is taken, that means that only macro_rules! macros are forever bound to being imported by #[macro_use], correct?

Correct.

When we have Macros 2.0 declarative macros (i.e. the macro keyword), that would share normal use-based imports with Macros 2.0 procedural macros, correct?

Correct.

If I'm understanding all that right, then I'm in favor of sticking with extern crate/macro_use. It'd be nice to have a clean split between the old system and the new system.

I'm starting to lean that way myself, after having thought it over since yesterday. It's a tough call, though, but I feel like saying #[macro_use] extern crate foo; is the (somewhat verbose) syntax for using macros from an external crate is reasonable (and analogous to #[macro_use] mod bar being the syntax for getting macros from a module). The extern crate form also has this side-effect that foo becomes a member of that module, which isn't really needed anymore, but that's ok.

Answer 100 · 2018-06-08T23:38:39.000Z

I see removing extern crate from idiomatic Rust as a key goal of the 2018 namespacing changes, and I don't see introducing a new syntax which is semantically equivalent as "blurring the lines" at all (i.e. you still can't import 1.0 macros individually). I don't have strong opinions about what the replacement syntax should be except that it should be connected to use statements.

While from the perspective of someone who already understands extern crate, this may seem like just two ways to do the same thing, from the perspective of someone who comes into a Rust where extern crate is not normal, the current syntax will be extremely weird. It seems much more natural to tell them that you can import 1.0 style macros using use log::macro*; or whatever that's connected syntactically to other imports.