Tracking issue for RFC 1566: Procedural macros

Current Status

This issue has been closed in favor of more fine-grained tracking issues

~Updated Description~

Next steps:

Stabilize use_extern_macros
- waiting on crater
Stabilize the proc_macro feature

Possible Stabilization Showstoppers

Original Description

This RFC proposes an evolution of Rust's procedural macro system (aka syntax
extensions, aka compiler plugins). This RFC specifies syntax for the definition
of procedural macros, a high-level view of their implementation in the compiler,
and outlines how they interact with the compilation process.

At the highest level, macros are defined by implementing functions marked with
a #[macro] attribute. Macros operate on a list of tokens provided by the
compiler and return a list of tokens that the macro use is replaced by. We
provide low-level facilities for operating on these tokens. Higher level
facilities (e.g., for parsing tokens to an AST) should exist as library crates.

Roadmap: #38356 (comment).

Tasks

Implement #[proc_macro_attribute] (PR Implement #[proc_macro_attribute] #38842).
- Fix feature(proc_macro) breaks attributes on custom derive #39347 (PR macros: fix inert attributes from proc_macro_derives with #![feature(proc_macro)] #39572).
- Fix macros: don't cfg-process items before applying attribute proc macros #39336 (PR Apply attr proc macros before cfg processing #44528).
Implement #[proc_macro] (PR Implement function-like procedural macros ( #[proc_macro]) #40129).
Identify and collect uses of proc_macro_derives in the InvocationCollector (PR [WIP] Expand #[derive(..)]s in the InvocationCollector #39391).
Support macro-expanded proc_macro_derive imports.
- For example:

#[derive(Trait, OtherTrait)] struct S; // Both these derives should resolve
macro_rules! m { () => {
    #[macro_use(Trait)] extern crate derives;
    use derives::OtherTrait; // this kind of import is gated behind `#![feature(proc_macro)]`
} }
m!();

@nrc

cc @nrc @jseyfried

I'd love for #[proc_macro_attribute] to be implemented soon. I already have a prototype and test usage that I banged out before realizing there's no compiler support yet 😒 :

Prototype: https://github.com/abonander/anterofit/blob/proc_macro/macros/src/lib.rs
Example/Test: https://github.com/abonander/anterofit/blob/proc_macro/examples/post_service_proc_macro.rs

@nrc

Tasks

(dtolnay edit: moved the checklist up to the OP)

cc @nrc @petrochenkov @durka @Ralith

@jseyfried

@jseyfried I ran into an issue where if a legacy macro and an attribute with the same name are imported into the same scope, trying to use the attribute throws an error that macros cannot be used as attributes. Can we make this work so that both can be in the same scope and can be used as intended?

@abonander

@abonander All macros (bang, attribute, and derive) share the same namespace, so we can't use two different macros with the same name in the same scope. However, we could improve that error message -- could you open an issue?

Sorry I’m late to the party. I’m happy with the direction to expose tokens rather than an AST, but I have some concerns about the specific TokenStream API proposed in the RFC:

pub enum TokenKind {
    Sequence(Delimiter, TokenStream),

    // The content of the comment can be found from the span.
    Comment(CommentKind),

    // `text` is the string contents, not including delimiters. It would be nice
    // to avoid an allocation in the common case that the string is in the
    // source code. We might be able to use `&'codemap str` or something.
    // `raw_markers` is for the count of `#`s if the string is a raw string. If
    // the string is not raw, then it will be `None`.
    String { text: Symbol, raw_markers: Option<usize>, kind: StringKind },

    // char literal, span includes the `'` delimiters.
    Char(char),

    // These tokens are treated specially since they are used for macro
    // expansion or delimiting items.
    Exclamation,  // `!`
    Dollar,       // `$`
    // Not actually sure if we need this or if semicolons can be treated like
    // other punctuation.
    Semicolon,    // `;`
    Eof,          // Do we need this?

    // Word is defined by Unicode Standard Annex 31 -
    // [Unicode Identifier and Pattern Syntax](http://unicode.org/reports/tr31/)
    Word(Symbol),
    Punctuation(char),
}

pub enum StringKind {
    Regular,
    Byte,
}

It’s not clear if this API was intended as a complete plan that was accepted when the RFC was merged, or just an example to be worked out later.

Generally, this seems far from the "normal" Rust syntax accepted by the compiler outside of macros. While some macros will want to parse some ad-hoc domain-specific language, others will want to parse "actual Rust" syntax and make sense of it.

(Minor) I don’t think Eof is necessary. An Iterator will presumably be used to, well, iterate over a TokenStream and Iterator::next already returns None to signal the end of iteration.
(Minor) I don’t think Exclamation, Dollar, or Semicolon are necessary. Matching on Punctuation('!') for example is not more difficult.
(Minor) As others have mentioned in the RFC PR, we might want to omit comments that are not doc-comments. (Any use case that wants to preserve comment likely wants to preserve whitespace too.)
As far as I can tell, what to do with multi-character operators (that probably should be a single token each) is still an open question. A possible solution is discussed in PR comments, but it looks like that didn’t make it into RFC text.
Number literals are missing. Are macros supposed to parse [Punct('1'), Punct('_'), Punct('2'), Punct('3'), Punct('4'), Punct('.'), Punct('5'), Punct('e'), Punct('6')] by themselves to evaluate a literal? They can’t even use str::parse::<f32> to do that, since the syntax it accepts is not the same as Rust literal syntax (which can have _ in the middle, for example).

I imagine that there’s a stability concern here. Can we introduce new numeric types like u128 / i128 (and possibly in the future f128, u256, …) and their literals, without breaking changes to the tokens API? One way to make this possible might be:
```
struct IntegerLiteral { negative: bool, decimal_digits: String, type_suffix: Option<String> }
impl TryInto<u32> IntegerLiteral { type Err = OutOfRange; /* … */ }
// Other impls for integer types supported in this compiler version

// Something similarly for floats
```
Or maybe something else. But I don’t think "pretend numbers don’t exist" is a good way to do it.
// Word is defined by Unicode Standard Annex 31 -

This definition needs to be more precise than that. UAX 31 specifies a couple different variations of identifier syntax, and none of them is called "word". But choosing which exact variation we want is why non-ASCII identifiers are feature-gated at the moment.

Instead, I think this should be defined as "whatever the current compiler accepts as an identifier or keyword" (which can change per Tracking issue for non-ASCII identifiers (feature "non_ascii_idents") #28979). Maybe with a pub fn is_identifier(&str) -> bool public API in libmacro.
Unicode strings and byte string literals share a single token variant, which I think is wrong as the memory representations of their values have different types (str vs [u8]). It’s also not clear if the text: Symbol component is intended to be a literal slice of the source code or the value after resolving backslash escapes. I think it should definitely be the latter. (For comparison, Char(char) has to be the latter since \u{A0} takes more than one char to represent literally.)

Change tracking issue for `proc_macro` feature to rust-lang#38356

Rollup merge of rust-lang#39454 - abonander:proc_macro_tracking_issue…

Given the amount of discussion/traffic this issue gets, does it make sense to move proc_macro_diagnostic to its own tracking issue? I'd also like to figure out what the blockers are for the stabilization of that feature, and see if we can push it through. Diesel uses it, and it's been great so far. The lack of this feature on stable is leading to the community creating funky workarounds, like syn's newest version using compile_error! in its place.

@sgrif

@sgrif I've opened up such an issue: #54140.

So I'm interesting in helping to stabilise the methods in Span and the LineColumn struct. I'll look at the open issues in the first comment, but if anyone wants to point a newbie to the compiler in a particular direction I'd appreciate it 👍

The proc_macro_gen feature gate pointed me here, but in the check list at the top I see nothing that obviously refers to a (proc_)macro that generates other macro definitions. Has this been accounted for (other than in rustc's feature gate)?

@jjpe

@jjpe it's probably best at this time that we spin off dedicated tracking issues for those feature gates, this issue was largely dedicated to the initial wave of stabilizations

@alexcrichton

@alexcrichton I'm perfectly fine with that, it's more that in the process of looking at the proc_macro_gen feature, it referred me here only to find as good as no mention of it. That's a bit odd to me, so I decided to mention it, at least.

@xieyuheng What would CodeString/Code even look like i.e. what would its semantics be?
Keep in mind that TokenStream is nothing more than the source code verbatim except as a series of token values rather than bothersome text.

The extended API for TokenStream (with TokenTree) is stable in 1.29. Importing function-like proc macros will be stable in 1.30.

It seems that since rustc 1.30.0-nightly (63d51e89a 2018-09-28) it is no longer possible to traverse code inside module in separate file. If you process mod module;, you get TokenStream containing mod module;, WYSIWYG.

I understand, that this is necessary in order to preserver spans, hygiene and consistency with processed code. Is there or will be any way to work with content of modules in separate files? Its lack may be confusing for end users when trivial refactoring of module organization causes change in macro behavior.

Ok this issue is massive and has gotten to the point that I don't think it's too useful to keep open any more and track APIs. To that end I've opened #54728 which splits this issue into a number of more fine-grained tracking issues:

Tracking issue for the quote! macro in proc_macro #54722 - the quote! macro
Tracking issue for proc_macro::Ident::new_raw #54723 - Ident::new_raw
Tracking issue for Span::def_site() #54724 - Span::def_site
Tracking Issue: Procedural Macro Diagnostics (RFC 1566) #54140 - proc_macro and diagnostics
Tracking issue for proc_macro::Span inspection APIs #54725 - inspecting a proc_macro::Span
Tracking issue for custom inner attributes #54726 - allowing custom inner attributes
Tracking issue for procedural macros and "hygiene 2.0" #54727 - a sort of catch-all for "hygiene 2.0" reports, may want to be sharded itself.

At this point I'm going to close this, but if I've forgotten to split off any other tracking issues please let me know! I can certainly open up some more follow-ups

Rollup merge of rust-lang#54728 - alexcrichton:renumber-issues, r=nik…

@alexcrichton

@alexcrichton What about of sub-attributes for attribute-like macro?
#38356 (comment)
Is there a issue for this?

@XX

@XX
Such sub-attributes don't necessarily have to be a language feature?
In case of derive registration of custom attributes is needed because derive macros cannot remove attributes from their input (the input is immutable).
Attribute macros can remove #[other_attribute]s from its input, so they will never get to name resolution and never report the "unresolved attribute" error.

(Beside that classic legacy unstable custom attributes mentioned in #38356 (comment) are now compatible with proc macros.)

@petrochenkov

@petrochenkov Yes, thanks for the clarification.

Navigation Menu

Repository navigation

Tracking issue for RFC 1566: Procedural macros #38356

Current Status

~Updated Description~

Original Description

Tasks

Tasks

204 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Tracking issue for RFC 1566: Procedural macros #38356

Description

Current Status

~~~Updated Description~~~

Original Description

Tasks

Activity

aturon commented on Dec 13, 2016

abonander commented on Jan 1, 2017

jseyfried commented on Jan 23, 2017

Tasks

abonander commented on Jan 24, 2017

jseyfried commented on Jan 24, 2017

SimonSapin commented on Feb 1, 2017

204 remaining items

sgrif commented on Sep 7, 2018

SergioBenitez commented on Sep 11, 2018

xd009642 commented on Sep 17, 2018

jjpe commented on Sep 28, 2018

alexcrichton commented on Sep 28, 2018

jjpe commented on Sep 30, 2018

SimonSapin commented on Sep 30, 2018

CodeSandwich commented on Sep 30, 2018

alexcrichton commented on Oct 1, 2018

XX commented on Nov 23, 2018

petrochenkov commented on Nov 23, 2018

XX commented on Nov 23, 2018

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

~Updated Description~