Why are my PCHs out of date?

Last week I got an interesting bug report from a colleague. They were minding their own business and playing with Unreal Engine 5 when suddenly a wild build error appeared and said that some precompiled headers are out of date:

fatal error: file 'D:\toolchain\sysroot\usr\include\x86_64-linux-gnu\bits\wordsize.h' has been
modified since the precompiled header 'UE5\Engine\...\SharedPCH.CoreUObject.ShadowErrors.h.gch'
was built: mtime changed

Hmm, well, it looks like some source files have changed and a simple re-build would fix the issue. However, building Unreal Engine from scratch takes considerable time, so you can understand why we’d like to avoid that.

The toolchain and the sysroot are distributed via the SDK™ and the colleague claimed that the error happened after they’ve installed a new version of the SDK. I was fairly sure that the above mentioned header files didn’t actually change – it’s a toolchain sysroot, it hasn’t been touched in years. Indeed, a simple stat shows that the modification time goes back to 2018:

> stat "D:\toolchain\sysroot\usr\include\x86_64-linux-gnu\bits\wordsize.h"
  File: D:\toolchain\sysroot\usr\include\x86_64-linux-gnu\bits\wordsize.h
  Size: 327             Blocks: 1          IO Block: 65536  regular file
Device: eab6ee59h/3937857113d   Inode: 8162774325601647  Links: 1
Access: (0644/-rw-r--r--)  Uid: (2249065/   werat)   Gid: (1049089/ UNKNOWN)
Access: 2021-12-02 13:58:10.281221500 +0100
Modify: 2018-01-14 10:39:44.000000000 +0100  # Note: 2018-01-14 09:39:44 UTC
Change: 2021-12-02 13:58:10.281221500 +0100
 Birth: 2018-01-14 10:39:44.000000000 +0100

But surprisingly right after I installed the new SDK, I was getting the same build error too. Oh boy, this is going to be interesting. Down to the rabbit hole!

quick adventure meme

The error we’re seeing when building Unreal is coming from clang – the compiler complains that some of the input files of the precompiled header have been modified (i.e. their mtime is different from what is stored in the PCH). Let’s peek inside the the PCH and see whether they are actually different:

# Dump the contents of the PCH (https://clang.llvm.org/docs/PCHInternals.html#ast-file-contents)
llvm-bcanalyzer.exe "UE5\Engine\...\SharedPCH.CoreUObject.ShadowErrors.h.gch" --dump | less
...
<INPUT_FILE abbrevid=4 op0=472 op1=327 op2=1515919184 op3=0 op4=0 op5=0/> blob data = 'D:\toolchain\sysroot\usr\include\x86_64-linux-gnu\bits\wordsize.h'

It’s size is 327 bytes (op1), and modification time is 1515919184 (op2), which corresponds to 2018-01-14 08:39:44 UTC:

>> from datetime import datetime, timezone
>> datetime.fromtimestamp(1515919184, tz=timezone.utc)
datetime.datetime(2018, 1, 14, 8, 39, 44, tzinfo=datetime.timezone.utc)

To summarize what we know so far:

The file size matches as well as the contents, I’ve checked separately. The source files used to build the precompiled header appear to be exactly the same, no real modifications has been made to them. However the mtime is indeed different (the compiler didn’t lie).

Well, I did install a newer version of the SDK recently. The sysroot headers obviously didn’t change – we can see they’re are the same. Maybe the mtime got messed up in the newer SDK? Nope, I’ve tried installing several different versions, but mtime of that file was always the same.

It’s curious though that the timestamps are different by exactly one hour. When two timestamps are different by exactly N hours programmers usually get real suspicious…

its always you three meme

The SDK is distributed via the Windows Installer (MSI). And it turns out, Windows Installer adjusts the modification time of the files according to the local timezone of the target machine! It means that the actual file mtime will be different on different machines depending on their timezone, for example:

Windows Explorer and other tools (e.g. stat) always display the local time, so both build server and the developer machine will show 2018-01-14 10:00:00, but with a different offset, of course. However Windows Explorer doesn’t show offsets or timezone, so for the user will just see the same modification time on both machines.

Same files, same installer, but the mtime will be different depending on your local timezone:

# Let's say we're in Berlin, UTC+1
> stat "D:\toolchain\sysroot\usr\include\x86_64-linux-gnu\bits\wordsize.h"
...
Modify: 2018-01-14 10:39:44.000000000 +0100

# Aaaand now we're in Moscow, UTC+3
> stat "D:\toolchain\sysroot\usr\include\x86_64-linux-gnu\bits\wordsize.h"
...
Modify: 2018-01-14 10:39:44.000000000 +0300

Typically this wouldn’t cause an issue – if local time is always the same, the timestamp will match. But wait-a-minute… what about the Daylight Saving Time? It changes the timezone offset and the files installed after the DST will have different mtime compared to the files installed before the DST.

I don’t know whether it’s a bug or a feature, but my guess it’s unlikely to be fixed anytime soon because backwards compatibility™. And that means that twice a year our users might need to rebuild their code just because. That’s why we can’t have nice things ¯\_(ツ)_/¯

Unfortunately I haven’t found a good way to alter this behaviour. Other people ran into this problem as well, but no solutions have been posted. It’s certainly possible to add a custom step to the installer to walk over the files and “correct” their mtime, but who knows what else might break because of this. If you know a better way, please, send help!

Discuss this article on lobste.rs