Repository navigation
Tracking issue for i686 msvc CI issues with latest Windows 2022 runner image 20250224.5.0+ #137733
Description
This is a tracking issue for CI compilation errors related to missing SSE2 types encountered when trying to use latest Windows 2022 runner image 20250224.5.0+ on the i686-msvc-{1,2} and dist-i686-msvc jobs. Key difference for the image version is the Windows 11 SDK version bump from 10.0.22621.0 -> 10.0.26100.0 that modified UCRT headers (see below).
Context
Zulip thread: https://rust-lang.zulipchat.com/#narrow/channel/242791-t-infra/topic/spurious.20.28.3F.29.20i686.20msvc.20errors.
cc <= 1.2.15 emits /arch:IA32 for clang-cl on x86 arch Windows targets, which in conjunction with GitHub Runner Image versions Windows 2022 20250224.5.0+ seems to cause a bunch of failures related to compilation errors failing to using SSE2 types without enabling SSE2 via /arch:SSE2. msvc seems to silently accept this1 and happily emit SSE2 instruments but clang-cl will error.
As Fulgen pointed out (thanks!) on the zulip thread:
The UCRT headers are, which is where they differ (10.0.22621.0/ucrt/wchar.h doesn't have the vectorization, 10.0.26100.0/ucrt/wchar.h does)
Current mitigation attempts
- On the
ccside: clang-cl: use/arch:SSE2forx86target arch cc-rs#1425 (In progress): Use Windows 2019 for 32-bit MSVC CI jobs #137732- Not going to work. A ui test failure. Old msvc toolchain has old cdb that will fail debuginfo tests.
(In progress) use large runners since they haven't been updated yet: Fix 32-bit MSVC CI #137749- To buy time, but going to break again once github updates large runners to use new image.
- [do not merge] fix 32-bit MSVC CI [alt version] #137754
- Downgrade 32-bit Windows SDK to 10.0.22621.0 in CI #137766
- The large runners were updated already, trying a different approach in Remove latest Windows SDK from 32-bit CI #137753.
Work needed to unblock using latest Windows 2022 runner image 20250224.5.0+ again
Warning
These steps are advisory and likely incomplete, we may find that we need to transitively bump cc versions in more deps.
- Fix
ccto not emit/arch:IA32, but instead either omit it or explicitly set/arch:SSE2forclang-clonx86: clang-cl: use/arch:SSE2forx86target arch cc-rs#1425- Release a new patch version of
cc: https://github.com/rust-lang/cc-rs/releases/tag/cc-v1.2.16
- Release a new patch version of
- Have
cargobump itsccversion to the new version: Bumpccto 1.2.16 to fixx86windows jobs in rust-lang/rust CI cargo#15245 - Sync
cargosubmodule that contains the bumpedcc: Update cargo #137791. - Bump
rustc_{codegen_ssa,llvm}ccto=1.2.16: Bumprustc_{codegen_ssa,llvm}ccto 1.2.16 to fixx86Windows jobs on newest Windows SDK #137788. - Revert remove-new-SDK workaround, and confirm that we can successfully run
i686-msvc-{1,2}anddist-i686-msvc, including the compiler crates and cargo: Revert "Remove Win SDK 10.0.26100.0 from CI" #137849.
Activity
[-]Tracking issue for i686 CI issues with latest Windows 2022 runner image 20250224.5.0+[/-][+]Tracking issue for i686 msvc CI issues with latest Windows 2022 runner image 20250224.5.0+[/+]Fulgen301 commentedon Feb 27, 2025
This has been fixed in the meantime, but to have it noted: Before this was fixed, C code with clang-cl was compiled without SSE2 support, whereas Rust code using that C code would be compiled with SSE2 enabled. The default function call ABI is likely fine, since
extern "C"passes it all over the stack anyway, but I'm not an opsem person, maybe this could have broken something that nobody ever noticed.Auto merge of rust-lang#137732 - Kobzol:ci-windows-2019-fix, r=<try>
Auto merge of rust-lang#137732 - Kobzol:ci-windows-2019-fix, r=<try>
Auto merge of rust-lang#137749 - Kobzol:fix-ci-2, r=Kobzol
Auto merge of rust-lang#137766 - ChrisDenton:winsdk=10.0.22621.0, r=<…
Auto merge of rust-lang#137766 - ChrisDenton:winsdk=10.0.22621.0, r=<…
Auto merge of rust-lang#137766 - ChrisDenton:winsdk=10.0.22621.0, r=<…
4 remaining items
Auto merge of rust-lang#137766 - ChrisDenton:winsdk=10.0.22621.0, r=<…
Bump `cc` to 1.2.16 to fix `x86` windows jobs in rust-lang/rust CI (#…
compiler: bump `cc` to 1.2.16 to fix `x86` Windows jobs on newest Win…
Rollup merge of rust-lang#137788 - jieyouxu:bump-compiler-cc, r=lqd,K…
Unrolled build for rust-lang#137788
Auto merge of rust-lang#137849 - jieyouxu:undo-workaround, r=<try>
Rollup merge of rust-lang#137849 - jieyouxu:undo-workaround, r=Kobzol
Rollup merge of rust-lang#137849 - jieyouxu:undo-workaround, r=Kobzol
Unrolled build for rust-lang#137849
Epixu commentedon Mar 5, 2025
Apologies for randomly intruding here.
I just encountered a CI failure for the same reasons and found only this relevant thread. In my case it was the cherry on top, because I'm at the brink of abandoning 32bit MSVC builds in their entirety at this point, because they have been bringing me nothing but grief lately.
I was just curious, have you often encountered as many code generation errors, slow build speeds and constexpr-related ICE's associated with MSVC v143 x86 builds in particular?
If so, considering you're still supporting them, am I doing something wrong and is it worth it to invest the time to research your coping mechanisms? Or would it be wiser for me, as a single developer, to just outright abandon these configurations...
jieyouxu commentedon Mar 5, 2025
Our most recent 32-bit msvc CI issue (this issue) was very specific. Known cause, one of our deps was incorrectly instructing msvc to use
/arch:IA32when we should've been using default or/arch:SSE2. I don't recall running intoin our CI recently. Fix for our CI purposes is to just fix the dep to emit the right
/arch, which I suspect is not remotely close to what you are experiencing (sorry to hear that).jieyouxu commentedon Mar 5, 2025
You will probably have to take that up with the MSVC team, since I assume your CI is exercising C++ code much more heavily?
Epixu commentedon Mar 5, 2025
Yes, I've already documented about three compiler bugs, one even got fixed recently. But I'm running out of energy to cope with those. I tend to test all possible combinations and it's always the particular configurations that interfere. Like, I literally yesterday excluded SSE/AVX/AVX2 x86 builds due to ICE's and code generation errors. And just now, when I decided it's all figured out, I get this absolutely horrid runner update introducing problems even for clang-cl, which pissed me even more.
Anyways, thanks for the feedback!