Add support for splitting linker invocation to a second execution of rustc

This issue is intended to track support for splitting a rustc invocation that ends up invoking a system linker (e.g. cdylib, proc-macro, bin, dylib, and even staticlib in the sense that everything is assembled) into two different rustc invocations. There are a number of reasons to do this, including:

This can improved pipelined compilation support. The initial pass of pipelined compilation explicitly did not pipeline linkable compilations because the linking step needs to wait for codegen of all previous steps. By literally splitting it out build systems could then synchronize with previous codegen steps and only execute the link step once everything is finished.
This makes more artifacts cacheable with caching solutions like sccache. Anything involving the system linker cannot be cached by sccache because it pulls in too many system dependencies. The output of the first half of these linkable compilations, however, is effectively an rlib which can already be cached.
This can provide build systems which desire more control over the linker step with, well, more control over the linker step. We could presumably extend the second half here with more options eventually. This is a somewhat amorphous reason to do this, the previous two are the most compelling ones so far.

This is a relatively major feature of rustc, and as such this may even require an RFC. This issue is intended to get the conversation around this feature started and see if we can drum up support and/or more use cases. To give a bit of an idea about what I'm thinking, though, a strawman for this might be:

Add two new flags to rustc, --only-link and --do-not-link.
Cargo, for example, would first compile the bin crate type by passing the --do-not-link flag, passing all the flags it normally does today.
Cargo, afterwards, would then execute rustc again, only this time passing the --only-link flag.

These two flags would indicate to rustc what's happening, notably:

--do-not-link indicates that rustc should be creating a linkable artifact, such as a one of the ones mentioned above. This means that rustc should not actually perform the link phase of compilation, but rather it's skipped entirely. In lieu of this a temporary artifact is emitted in the output directory, such as *.rlink. Maybe this artifact is a folder of files? Unsure. (maybe it's just an rlib!)
The converse of --do-not-link, --only-link, is then passed to indicate that the compiler's normal phases should all be entirely skipped except for the link phase. Note that for performance this is crucial in that this does not rely on incremental compilation, nor does this rely on queries, or anything like that. Instead the compiler forcibly skips all this work and goes straight to linking. Anything the compiler needs as input for linking should either be in command line flags (which are reparsed and guaranteed to be the same as the --do-not-link invocation) or the input would be an output of the --do-not-link invocation. For example maybe the --do-not-link invocation emits an file that indicates where to find everything to link (or something like that).

The general gist is that --do-not-link says "prepare to emit the final crate type, like bin, but only do the crate-local stuff". This step can be pipelined, doesn't require upstream objects, and can be cached. This is also the longest step for most final compilations. The gist of --only-link is that it's execution time is 99% the linker. The compiler should do the absolute minimal amount of work to figure out how to invoke the linker, it then invokes the linker, and then exits. To reiterate again, this will not rely on incremental compilation because engaging all of the incremental infrastructure takes quite some time, and additionally the "inputs" to this phase are just object files, not source code.

In any case this is just a strawman, I think it'd be best to prototype this in rustc, learn some requirements, and then perhaps open an RFC asking for feedback on the implementation. This is a big enough change it'd want to get a good deal of buy-in! That being said I would believe (without data at this time, but have a strong hunch) that the improvements to both pipelining and the ability to use sccache would be quite significant and worthwhile pursuing.

@cramertj

cc @cramertj and @jsgf, we talked about this at the RustConf and figured y'all would want to be aware of this

Would the --only-link literally just invoke ld? If so, it might be useful to just be able to extract whatever extra libraries/options its adding to the link line, so we can independently regenerate the link line.

This would be useful for linking hybrid Rust/C++/(other) programs where the final executable is non-Rust. In other words, we could have C++ depending on Rust without needing to use staticlib/cdylib.

I don't think it'd just be a thin wrapper around ld, no, but it would also prepare files to get passed to the linker. For example when producing a dylib rustc will unpack an rlib and make a temporary *.a without bytecode/metadata. Additionally if performing LTO this'd probably be the time we'd take out all the bytecode and process it. (maybe LTO throws a wrench in this whole thing). Overall though I don't think it's safe to assume that it'll just be ld.

Firstly, we'd want the final linker doing LTO in order to get it cross-language, regardless of whatever language the final target is in and what mix of languages went into the target.

Secondly, since Buck has full dependency information, including Rust dependencies on C/C++, it will arrange for all the right libraries to be on the final link line. As a result we never want to use or honor #[link] directive, and our .rlibs don't contain anything to be unpacked.

(Even if that weren't true, at least on Unix systems, the .rlib is just a .a and could be used directly, except perhaps for the extension).

I like this proposal because it allows us to factor out the Rust-specific details from the language-independent ones. For example there's no reason for rustc to implement LTO if we're already having to solve that for other languages - especially when that solution pretty infrastructure-specific (distributed thin LTO, for example). There's also no real reason for us to use staticlib/cdylib if we can arrange for all the Rust linker parameters to be on the final link line, even if the final executable is C++, and it would be a significant code duplication reduction (unless LTO see it and eliminate it, but that's still a compile-time cost).

Ultimately, Rust lives in the world of linkable object files, and a final artifact is generated by calling the linker with a certain set of inputs. Since Rust doesn't have unusual requirements that make it incompatible with C/C++ linkage (eg special linkage requirements or elaborate linker scripts) then the final linker stage could be broadly language agnostic.

+1 to wanting the ability to turn off / disable #[link].

I'm generally in favor of this. Some thoughts:

One of the computationally more heavy things that linking needs is the list of exported symbols (i.e. the linker script). Getting this list involves analyzing the HIR and reading upstream crate metadata. But it should be easy to serialize this information during the --do-not-link step and store it in the .rlink output.
Could we move all of LTO out of rustc? That would make things simpler for rustc but probably has some overhead. Also, as far as I know, llvm-ar does not support doing LTO, but for staticlibs one might want to have it (Firefox does this at least).

I don't disagree that y'all's sorts of projects don't want to use the rustc-baked-in LTO, but I don't think we can remove it because many other projects do use it (and rightfully want to). Also this is still just a sort of high-level concept, but if a lot of feature requests are piled onto this it's unfortunately unlikely to happen.

@tmandry

Hi, my name is Victor and I'm workin with @tmandry.
I am very interested in this Issue as it potentially enables many other great cool. Therefore, I plan to make a prototype and ultimately implement it. @alexcrichton , do you have any suggestions/tips to start?

@0dvictor

Great @0dvictor! The steps I'd recommend for doing this would probably look like:

Poke around rustc and the linking phase, and formulate a high-level plan of how you'd like to implement this feature.
Confirm with compiler team folks that the plan of action you've got is reasonable. The details probably wouldn't be fully fleshed out, but this is likely to be a large-ish change so the compiler folks will want to be onboard and it's best to start that early.
Iterate towards a working prototype (probably with compiler team help)
Start writing tests/etc
Evaluate with the compiler team at this point if the change needs an RFC or if it's good to land unstable in the compiler

As for the actual change itself I haven't looked too much into this, so I wouldn't know where best to start there.

Some implementation notes:

Current state of things

The main run_compiler function calls Compiler::link, which ensures there is a codegen step running. It then dispatches to the trait method CodegenBackend::join_codegen_and_link.

For the LLVM backend (which is the only one right now), this method is implemented here. join_codegen_and_link joins all the threads running codegen, which save their results to an object file per thread (often named foo.bar.<hash>-cgu.<threadno>.rcgu.o on linux). The names of these object files are saved in the CodegenResults struct (specifically, look in CompiledModule).

Finally, it calls link_binary which has the main logic for invoking the linker.

Strategy

Obviously, we need to split apart all the code that assumes codegen and linking happen at the same time. This starts with the join_codegen_and_link trait method. Thankfully, it doesn't seem like there is too much code that assumes this, but there's still a question of what to do in the new code.

For the flags, we can start with unstable options (-Z no-link and -Z only-link) and later stabilize them via the RFC process. When the no-link flag is passed, we should not invoke link_binary anymore. When the only-link flag is passed, we need a way of recovering the information that was in our CodegenResults struct so we can call link_binary.

@petrhosek

In lieu of this a temporary artifact is emitted in the output directory, such as *.rlink. Maybe this artifact is a folder of files? Unsure. (maybe it's just an rlib!)

I don't much like using a directory as output, since some build systems might not support this.

Probably the best thing to do is to make an ar file (which I should note is what an rlib is today). I don't much care what the extension is (I like .rlink, since it probably gets handled differently than rlibs or staticlibs do today).

That said, the choice of extension should be up to whoever is invoking rustc, and we should use the -o flag to decide where our output goes.

I think there might be details we need to pay attention to regarding the linker's default behavior when linking object files vs archive files (like symbol visibility), but not sure what those details are. cc @petrhosek

@jsgf

@jsgf The project I'm working on seems to share similar properties to yours:

Using a non-Cargo build system based on static dependency resolution (and has 20000+ targets)
Final linking performed by an existing C++ toolchain
A few Rust .rlibs scattered throughout a very deep dependency tree, which may eventually roll up into one or multiple binaries

We can't:

Switch from our existing linker to rustc for final linking. C++ is the boss in our codebase; we're not ready to make the commitment to put Rust in charge of our final linking.
Create a Rust static library for each of our .rlibs. This works if we're using Rust in only one place. For any binary containing several Rust subsystems, there would be binary bloat and often violations of the one-definition-rule
Create a Rust static library for each of our output binaries. The build directives for the Rust .rlibs don't know what final binaries they'll end up in; and the build directives for the final binaries don't know what .rlibs they've pulled in from deep in their dependency tree. Our build system forbids that sort of global knowledge, or it would be too slow across so many targets.
Create a single Rust static library containing all our Rust .rlibs. That monster static library would depend on many C++ symbols, so each binary would become huge (and in fact not actually link properly)

Therefore we want to link rlibs directly into our final linker invocation. This is, in fact, what we're doing, but we have to add some magic:

The final C++ linker needs to pull in all the Rust stdlib .rlibs, which would be easy apart from the fact they contain the symbol metadata hash in their names.
We need to remap __rust_alloc to __rdl_alloc etc.
In future the final rustc-driven linker invocation might add extra magic.

Therefore one of the solutions you and @tmandry suggest would be awesome: either stabilizing .rlink (which sounds unlikely) or having some official way to build a linker command line which is not under the control of rustc (-Zbinary-dep-depinfo helps a bit).

I think, though, that this request is a little bit orthogonal to this issue. I wonder if we should submit a new issue? I think it's quite a big request.

That would be useful and it's also orthogonal. A new issue sounds like a good idea.

…

On Fri, May 1, 2020 at 4:17 PM adetaylor ***@***.***> wrote: @jsgf <https://github.com/jsgf> The project I'm working on seems to share similar properties to yours: - Using a non-Cargo build system based on static dependency resolution (and has 20000+ targets) - Final linking performed by an existing C++ toolchain - A few Rust .rlibs scattered throughout a very deep dependency tree, which may eventually roll up into one or multiple binaries We can't: - Switch from our existing linker to rustc for final linking. C++ is the boss in our codebase; we're not ready to make the commitment to put Rust in charge of our final linking. - Create a Rust static library for each of our .rlibs. This works if we're using Rust in only one place. For any binary containing several Rust subsystems, there would be binary bloat and often violations of the one-definition-rule - Create a Rust static library for each of our output binaries. The build directives for the Rust .rlibs don't know what final binaries they'll end up in; and the build directives for the final binaries don't know what .rlibs they've pulled in from deep in their dependency tree. Our build system forbids that sort of global knowledge, or it would be too slow across so many targets. - Create a single Rust static library containing all our Rust .rlibs. That monster static library would depend on many C++ symbols, so each binary would become huge (and in fact not actually link properly) Therefore we want to link rlibs directly into our final linker invocation. This is, in fact, what we're doing, but we have to add some magic: - The final C++ linker needs to pull in all the Rust stdlib .rlibs, which would be easy apart from the fact they contain the symbol metadata hash in their names. - We need to remap __rust_alloc to __rdl_alloc etc. - In future the final rustc-driven linker invocation might add extra magic. Therefore one of the solutions you and @tmandry <https://github.com/tmandry> suggest would be awesome: either stabilizing .rlink (which sounds unlikely) or having some official way to build a linker command line which is not under the control of rustc ( -Zbinary-dep-depinfo helps a bit). I think, though, that this request is a little bit orthogonal to this issue. I wonder if we should submit a new issue? I think it's quite a big request. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#64191 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AARMYYFU42EPAMD5YNWZ5XLRPNJ77ANCNFSM4IUB7HWA> .

@dtolnay

This issue came up in a discussion about @dtolnay's cxx which wants to be able to describe callback relationships between Rust and C++ code - eg Rust calls C++, which then calls back into Rust. If you're using linker symbols to resolve these calls (vs runtime indirect function calls via pointers), then that effectively means you have a rust.o and cxx.o with mutual references.

AFAIK the only way to correctly link this is with something like --start-group rust.o cxx.o --end-group on the linker command line. If rustc is driving the linking process, then it would need to understand this concept, which I think it out of scope for it. Alternatively, if the build system understands these kinds of relationships, then it already knows how to invoke the linker this way. (Moot for Cargo, since it doesn't have a way to model C++ dependencies on Rust, and certainly not cyclic ones.)

I'm interested in the outcome of this. Is it currently in a state where cargo could potentially implement a useful subset of it? Or is there further work needed on the final link step before cargo can do anything?

@joshtriplett

@joshtriplett Kernel?

I apologize for the long delay in continuing this work. PR #75094 is created to allow generating a single object file for a crate.

Performance was measured by compiling rustc itself:

	`-Ccodegen-units=1`	[DEFAULT]	`-Zcombine-cgu`
raw comp time (second)	987	294	303
relative perf to `-Ccodegen-units=1` (%)	100	29.79	30.70
relative perf to default (%)	335.71	100	103.06
relative perf to `-Zcombine-cgu` (%)	325.74	97.03	100

Though -Zcombine-cgu is around 3% slower than the default configuration, it is still a significant improvement comparing to the single threaded case, which would significantly improve the performance of --emit [llvm-bc|llvm-ir|asm|obj].

Rollup merge of rust-lang#75094 - 0dvictor:cgu, r=oli-obk

I tried using the -Zno-link/-Zlink-only mechanism in earnest, and unfortunately I think its deficient in a number of ways. I have some thoughts on a simpler to use mechanism. More detail: https://internals.rust-lang.org/t/alternative-approach-to-zno-link-zlink-only-split-linking/14842

Auto merge of rust-lang#93681 - Mark-Simulacrum:rlink-binary, r=david…

@bjorn3

From reading the thread at https://internals.rust-lang.org/t/alternative-approach-to-zno-link-zlink-only-split-linking/14842, I have one brief comment and several less brief ones:

@bjorn3:

Also various of the inputs of the link process don't exist in the crate metadata at all. Only the rlink files contain them.
...
What I am trying to say is that using rlib is at least as complex as rlink and may even be more complex due to no way to communicate certain things.
...
#93681 (comment)
[rlink] isn't meant to be read outside of rustc.

Synthesizing these comments of yours from 2021 and 2022 together, I now feel¹ even more like "an explicitly public format, constructed from compiler-internal data, yet designed to be abstracted from compiler internals" in the manner of "SemanticDB² for rust" is such a tantalizing idea!

@jsgf:

Personally, I'm particularly interested in the possibilities of distributed builds, and aggressively caching and reusing build outputs.

Having actually shipped a highly parallel distributed build process to production for Scala builds at Twitter¹ via working very closely with Eugene Burmako on co-engineering rsc³ and pants together for this purpose, caching and parallelization tend to reach their global maximum when you pay world-class researchers to build powerful systems in their field of expertise. Many people get this wrong and will only compensate engineers for constructing systems of enclosure and deskilling and then call the promotion ladder that valorizes stealing credit from people who devote their lives to constantly reinventing the state of the art a "free market".

In particular, the epiphany that we shipped to Twitter engineers occurred not just from Eugene Burmako and Stu Hood's initial epiphany to enable pipelining with parallel scalac invocations as per Eugene's Scale by the Bay 2018 talk⁴, but Win Wang and my extended codevelopment session which identified the relationship between the process lifetime distinction of the multithreaded local HotSpot JIT rsc invocation vs highly parallel ephemeral remote execution of graal native-image AOT-compiled scalac jobs, and how attempting to do pipelining with 100% remote execution was completely losing the benefit of pipelining in the first place by taking the network i/o hit both ways across the topologically sorted partial ordering of the outline compilation phase.

Eugene Burmako was not an expert on I/O latencies and resource saturation. I was not an expert on parallel typechecking. Win and I did not finally crack the case until we understood how computer performance is a result of bidirectional communication protocols suited to the task at hand.

In the long term I'd like to find some way where rustc can provide enough information to completely construct an external link line, so that we don't have to rely on it to invoke the linker - ie so that a top-level C++ target can link in Rust code in a native way.

Constructing a link line and providing it to the build system to invoke reminds me of how the user provides their own &mut [u8] buffer to std::io::Read and submits it to the OS. Constructing a .rlink and postprocessing it into a standardized format (the aforementioned SemanticDB for rust) reminds me of adapters like io::Chain.

Secondly since the .rlink file is a dump of internal compiler state, it must be treated as opaque. But as mentioned above, it contains a full set of paths to .rlib files that the executable depends on (directly and indirectly). These paths are absolute paths, which means that they're likely only meaningful on a single machine, implying that the -Zno-link and -Zlink-only phases can't be distributed.

The .rlink file being opaque already means it can't be distributed.

This makes split linking look a lot more normal to the surrounding build system

It is the build system's job to look normal to the compiler.

-Zlink-only should treat all its inputs as inputs and preserve them. Failing to do either of these could corrupt a cache

chmod -w

@adetaylor #64191 (comment)

either stabilizing .rlink (which sounds unlikely) or having some official way to build a linker command line which is not under the control of rustc

Going from .rlink (a rustc-specific interface) directly to a linker command line (which is tied to at least your choice of linker) seems to forget the law of excluded middle! Building a linker command line sounds like it should be definitionally the job of the build system, should it not?

"under the control" lmao. The build system is executing the compiler as a subprocess. The compiler is not responsible for the build system's architectural constraints or internal politics. Build systems are not a tool to neg the compiler after a contributor does the one thing the build system can't do without forking it. This is a document I created in 2018: https://docs.google.com/document/d/1iWINqg30nXaCYmBxma5h5RgWzx7sgFqWGJcQJ-RYpRE/edit?usp=drive_link

Compilation tends to be CPU-bound in most programming languages which have a nontrivial compilation step. Many other tools used during development, such as formatters and linters, perform much less computation per input file. Additionally, Scala and Java source files require their dependencies to be compiled beforehand, imposing a partial order, which formatters and linters typically don't require. This leads to a characterization of these operations as an example of an embarrassingly parallel IO-bound problem. While multithreading can be used to achieve parallelism and maintain JIT state, the implementation of threading in an individual command-line tool is unlikely to be optimal in all scenarios, and may conflict with attempts to further parallelize the tool invocation by a parent process.

On the other hand, process-level parallelism can be finely controlled by a parent process without requiring any optimization on the part of the command-line tool. Furthermore, the parallelism can be effectively tuned when the amount of work is known in advance, and more IO-bound workloads can often be more accurately predicted by the size of the input files. A build tool such as Pants which invokes these subprocesses has all of this information before the run begins, and can tune the number of files each subprocess operates on, and even attempt to partition the input files into buckets of similar file size in an attempt to maximize the number of processes running in parallel. Finally, using multiple subprocesses allows the build tool to scale the parallelism arbitrarily high, which saturates the operating system with IO requests and allows the operating system to perform one of the jobs it excels at, which is orchestrating IO requests from many different processes to most efficiently make use of the underlying filesystem and hardware.

btw I would absolutely recommend rustc devs read Eugene Burmako's incredibly fascinating research notes on parallelism in typechecking¹:

That was the good news. The bad news is that completers significantly affect compilation performance. First, they bring unpredictability to typechecking, making it hard for humans to understand what's going on and build a reliable mental model of performance. More importantly, they make parallelism very hard.

Let us elaborate on parallelism. Even though our explanations above may have made it look like in Scala programs everything depends on everything, in reality there is a significant degree of parallelism in typechecking. If we imagine that an oracle has magically performed naming for us, then we can type all methods completely in parallel.

Unfortunately, lazy completers prevent us from fully leveraging this parallelism. If we naively run typing in parallel right after having set up lazy completers, this will mean that multiple threads may attempt to run the same completer in parallel. In turn, this means that only the first thread will succeed, and the others will see the "in progress" mark on the associated definition and will spuriously report cyclic reference errors.

We have discovered this problem several years ago while trying to make runtime reflection in scala.reflect thread-safe. For more than a month, we tried different strategies of fixing the spurious cyclic reference errors, but eventually we gave up and hid completions behind a global lock. While this ensured correctness, this also destroyed parallelism. Maybe there exists a better solution, but we're unaware of it.

[WAIT FOR IT!!!!!!]
[...]

Thirdly, we improved upon the peculiar style of Kentucky Mule completers. These completers achieve greater flexibility (the ability to interrupt themselves at critical points of execution) via a laborious encoding that involves explicit yields. The author himself says "It's clear that for the design based on interruptible completers to scale to the size of a real compiler, a better way for expressing composition has to be found".

https://github.com/twitter/rsc/blob/master/docs/compiler.md ↩

The .rlink file being opaque already means it can't be distributed.

Not inherently. It just means that it has to be processed by the exact same rustc version as it was produced by, which is the case anyway. Rlink files could plausibly be made movable across machines. Just nobody has worked on it. Turns out something was missing when -Zno-link/-Zlink-only was originally implemented, making it not support what it was intended to support (extra build pipelining). I've been slowly chipping away on getting the rustc internals ready to actually make rlink files useful.

Building a linker command line sounds like it should be definitionally the job of the build system, should it not?

Build systems have no clue about all rust crate dependencies (even if all user dependencies are known, the build system doesn't know anything about what standard library crates need to be linked in), it doesn't know about the rustc ABI compatibility rules and rustc generates various extra input to the linker to workaround limitations in the C compilation model with those extra inputs changing between rustc versions, the build system doesn't know the list of symbols that need to be exported (those are derived from the input source code after macro expansion), how to include the crate metadata in rust dylibs and more. That is why rustc is responsible for invoking the linker when using any crate type other than staticlib. And staticlib bundles all rust crates together with most other necessary linker inputs (only the list of C libraries to link isn't included, but you can get those from the rustc output when building the staticlib if you pass --print native-static-libs). Even if we exposed all information rustc currently uses to build a linker invocation to the build system, it will probably break in the future when we add new features or change how existing features are internally implemented. For example we used to add all objects files bundled C static libraries directly into rlibs, but changed this to instead wrap the original C static library in a non-allocated section of an object file and extract them right before linking. Not including the object files directly was necessary to support conditional linking (#[link(name = "...", cfg(...))]) and wrapping the C static library in an object file is necessary to hide it from the linker. Otherwise many linker complain about unsupported archive member types.

Navigation Menu

Repository navigation

Add support for splitting linker invocation to a second execution of `rustc` #64191

Recommended reading

Current state of things

Strategy

42 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Add support for splitting linker invocation to a second execution of rustc #64191

Description

Activity

alexcrichton commented on Sep 5, 2019

jsgf commented on Sep 6, 2019

alexcrichton commented on Sep 6, 2019

jsgf commented on Sep 7, 2019

cramertj commented on Sep 9, 2019

michaelwoerister commented on Sep 11, 2019

alexcrichton commented on Sep 11, 2019

0dvictor commented on Nov 18, 2019

alexcrichton commented on Nov 18, 2019

tmandry commented on Nov 19, 2019

Recommended reading

Current state of things

Strategy

tmandry commented on Nov 19, 2019

42 remaining items

adetaylor commented on May 1, 2020

tmandry commented on May 1, 2020

jsgf commented on May 15, 2020

joshtriplett commented on Jul 17, 2020

jsgf commented on Jul 20, 2020

0dvictor commented on Aug 3, 2020

jsgf commented on Jun 6, 2021

cosmicexplorer commented on Dec 29, 2025

Footnotes

cosmicexplorer commented on Dec 29, 2025

Footnotes

bjorn3 commented on Dec 29, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Add support for splitting linker invocation to a second execution of `rustc` #64191