When there are too many debug symbols

Death, taxes, and rampant browser engines are three things you can be sure of. This was also true at the beginning of 2020, when I realized that due to the relentless growth of Chromium, sooner or later we will create PDB files (Windows debug symbols) that exceed the 4 GiB PDB format limit.

In February 2020 I filed a Visual Studio bug requesting an increase in this limit, and after three years and three days we pulled the switch, and now Chromium can create larger PDBs. At the time, Chrome’s PDB was 95% of 4 GiB, and many of the test binaries had already crossed that threshold, so it was just in time.


The PDB format is based on pages; it allows up to two to the twentieth power (2^20, or 1,048,576) pages, and the standard page size is 4 KiB. Multiplying these two numbers gives us a maximum size of 4 GiB. The maximum number of pages cannot be increased, but the page size can be increased. In fact, the developers of the PDB format have always recognized the need for different page sizes, but in fact no tool has ever supported a page size other than 4 KiB.

To support larger PDBs, it was “just” to update a few tools to support larger page sizes. These tools were:

• Visual Studio Debugger
• link.exe – Microsoft Linker
• lld-link is the linker used by Chromium
• windbg and other debuggers (kd.exe, ntsd.exe and so on)
• dbghelp.dll (used to load symbols)
• pdbstr.exe (used to index sources)
• symstore.exe (used to upload to symbol servers)
• msdia140.dll (COM API for loading symbols)
• Windows Performance Analyzer (WPA, trace viewer ETW)
• Possibly other tools

Everything is simple!

According to the comments under the bug, an internal fix was released by Microsoft in August 2021. By December, this fix was implemented in Visual Studio 2019 updates and just released by Visual Studio 2022, and lld-link also supported it. After linkers started supporting larger pages, we added a build option use_large_pdbswhich switched PDB page size to 8 KiB. However, at first this option had to be disabled by default due to the lack of full tool support.

Ideally, updates to other tools should have been released in the fall of 2021, but … this did not happen. I reached out to the Windbg and WPA development teams: I got the impression (although I could be wrong) that they didn’t realize they needed to roll out updates until I told them to.

windbg (and many related tools) and WPA there are a lot of release channels, so it’s hard to say when fixes in these tools were first released. In Microsoft Store versions and in nuget support for large PDBs appeared sometime in 2022, however, it became reasonable to release updates for all versions of all these tools only after the release of Windows 11 22H2 SDK in the fall of 2022.

Windows 7 was another issue that got in our way. Newer versions of dbghelp.dll had problems working on Windows 7. So while we were running tests on Windows 7, we had to generate PDBs that worked with older versions of dbghelp.dll. Version 109 of Chrome is the last version to support Windows 7so as soon as we branched that version, I could start working on supporting large PDBs on the main branch.

While working on the fix, I encountered a lot of mysterious crashes – this can be understood by the fact that my first public attempt create a change to move to large PDBs is dated December 28th, more than a month before the implementation of the change. The first problem was that one test stubbornly refused to load updated PDB files. Gradually, I was able to narrow the problem down to a failed test attempt to load a new version of dbghelp.dll. I then spent a lot of time figuring out which DLL imports were causing problems, but then I realized… that the new dbghelp.dll hadn’t been deployed to the test machines. It’s hard to load a DLL that doesn’t exist. It’s really funny that we haven’t deployed dbghelp.dll on test machines for several years, but the system version was good enough that it didn’t matter and this bug hadn’t been noticed before. I unfolded the cool bootloader snapshot diagnostics (based this tool), but in the end it turned out to be a waste of time, because the reason was the lack of a DLL.

Another mysterious crash occurred only on our official builders, which made testing very difficult. This failure was caused by loading the wrong version of msdia140.dll. An almost perfect parallel to the first problem: we never deployed msdia140.dll properly. I still don’t know where it was downloaded from earlier, but by copying the right version to the output folder, we solved the problem.

Finally, 37 days and 38 patch sets after uploading the first version of my change, I implemented it. The new PDB size limit is 8 GiB, but when we need to, we can double the page size again.

Changes like this can easily lead to crashes – it’s hard to foresee all the hidden dependencies or failure points, so I was happy when I deployed the change and heard almost no negative feedback after that. A couple of people complained that various tools were unable to load Chrome Canary symbols, however in all cases updating the required tools to the latest Windows 11 SDK solved the problem. It’s worth noting that by default the Windows SDK does not install the debuggers and the Windows Performance Toolkit (which includes WPA), so if you need these tools, you must select them during installation, otherwise you will end up with older versions.

Most people don’t work with Chromium, but if you’re working with Chrome, you may need to be able to profile or debug official builds someday. Can set up your tools to point to the symbol server Chrome, and then symbols and sources will be downloaded on demand, but only if you are working with tools that are compatible with large PDBs. I know that some game studios are also facing the 4 GiB limit, so those developers also need the latest tools.

I renewed UIforETW so that it installs the latest version of WPA on startup. The latest release (currently 1.58) can be found Here.