Diagnosing a VS-only build problem

I do most of my work in the google-cloud-dotnet github repo (That's the repo for the Google Cloud Client Libraries for .NET, just to get a quick bit of marketing in there.) We try to keep our build and test dependencies up to date, so I recently updated to the latest versions of Microsoft.NET.Test.Sdk and xUnit.

Leaving aside problems identified by the xUnit analyzer which is now bundled in the xUnit package (some of which were useful, and some of which weren't), this change caused another problem: while building from the command line worked fine, building some of the many solutions from Visual Studio (15.3.3) generated the following error:

Error MSB4018 The "GenerateBindingRedirects" task failed unexpectedly.
System.IO.PathTooLongException: The specified path, file name, or both are too long. The fully qualified file name must be less than 260 characters, and the directory name must be less than 248 characters.
at System.IO.PathHelper.GetFullPathName()
at System.IO.Path.LegacyNormalizePath(")
at System.IO.Path.NormalizePath(")
at System.IO.Path.NormalizePath(")

(more stack trace here)

The output window shows a file related to the error:

C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\MSBuild\15.0\Bin\Microsoft.Common.CurrentVersion.targets(2099,5)

So, what's going on, and how do we fix it?

Step 1: Check that this change is responsible

Git makes life really easy here. For peace of mind - particularly for this kind of issue - I tend to close Visual Studio when switching branches, so:

• Find a solution that fails (e.g. Google.Cloud.Diagnostics.AspNetCore)
• Close VS
• git checkout master
• Rebuild solution in VS - looks fine
• Close VS
• git checkout branch with change
• Rebuild in VS - observe error
• git checkout master
• Rebuild solution in VS - looks fine
• Close VS
• git checkout branch with change
• Rebuild in VS - observe error

Yes, I did everything twice, just to make sure it wasn't an anomaly.

So yes, it's definitely a problem.

Step 2: Moan on Twitter

This isn't the most scientific diagnostic tool in the world, but posting on Twitter about the problem did at least reassure me that I wasn't alone.

Step 3: Narrow down the cause

Even though I observed the problem in Google.Cloud.Diagnostics.AspNetCore, the full error message in VS referred to Google.Cloud.Diagnostics.Common.IntegrationTests. That's part of the Google.Cloud.Diagnostics.Common solution - the AspNetCore projects depend on the Common projects.

Try step 1 again with Google.Cloud.Diagnostics.Common (just one pass and one failure this time) - yes, it's still a problem there. That's useful.

Step 4: Try a workaround

All my source is under c:\Users\Jon\Test\Projects. The "Test" part is a bit of an anomaly, and next time I get a new machine, I'll probably work without it, but there's no need to change now. The Projects directory is actually a junction (symlink) to the c:\Users\Jon\Documents\Visual Studio 2015\Projects directory. That's quite a long path to start with" let's see if getting rid of the symlink helps.

• Delete symlink
• Move Projects file from under Documents to directly under Test
• Try to build: same failure

Okay, it looks like we'll have to be a bit more subtle.

Step 5: Play spot the difference

Given that we have code which builds on the command line in both the working and failing situations, we can reasonably easily see any differences in generated files.

• On the command line, go into the Google.Cloud.Diagnostics.Common.IntegrationTests directory
• Delete bin and obj
• git checkout master
• Run dotnet build
• Copy the resulting bin and obj directories to a "working" directory
• git checkout branch with change
• Delete bin and obj
• Run dotnet build
• Copy the resulting bin and obj directories to a "broken" directory
• Run kdiff3 against the working and broken directories

There are lots of differences between the directories, as I'd expect, but given that this is about binding redirects, it's reaosnable to use experience and start scanning for filenames ending with .config.

Sure enough, in the "broken" directory, under obj\net452, there was a file called Google.Cloud.Diagnostics.Common.IntegrationTests.csproj.Google.Cloud.Diagnostics.Common.IntegrationTests.dll.config. That's not a copy and paste error - it really is a 115-character-long filename, even leaving out any directory parts.

In the file system, the full path is: c:\users\jon\Test\Projects/google-cloud-dotnet\apis\Google.Cloud.Diagnostics.Common\Google.Cloud.Diagnostics.Common.IntegrationTests\obj\Debug\net452\Google.Cloud.Diagnostics.Common.IntegrationTests.csproj.Google.Cloud.Diagnostics.Common.IntegrationTests.dll.config - that's 266 characters.

Step 6: Try being slightly cheeky: very temporary workaround

Okay, so moving away from the "Documents\Visual Studio 2015" directory didn't help much, but given that we're just on the limit, let's try just renaming "google-cloud-dotnet" to "short" (very temporarily).

Try opening it in Visual Studio - wahoo, it works :) The .config file is generated by Visual Studio correctly.

Step 7: Work out who to blame

So, where's the bug?

• It's a shame that Visual Studio 2017 doesn't support long filenames, even though the dotnet CLI does
• Surely we don't need such a long filename anyway
• Do I need the project name to be so long?

Looking back to the very first description, let's have a look at the msbuild file that's mentioned: C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\MSBuild\15.0\Bin\Microsoft.Common.CurrentVersion.targets

That has:

<GenerateBindingRedirects AppConfigFile="@(AppConfigWithTargetPath)" TargetName="$(TargetFileName).config" OutputAppConfigFile="$(_GenerateBindingRedirectsIntermediateAppConfig)" SuggestedRedirects="@(SuggestedBindingRedirects)">

So it looks like the problem is _GenerateBindingRedirectsIntermediateAppConfig, which is defined in a property group elsewhere as:

<_GenerateBindingRedirectsIntermediateAppConfig>$(IntermediateOutputPath)$(MSBuildProjectFile).$(TargetFileName).config</_GenerateBindingRedirectsIntermediateAppConfig>

That certainly looks like it's responsible for the choice of file.

A quick search for _GenerateBindingRedirectsIntermediateAppConfig shows that I'm not the first person to have run into this - PathTooLongException with _GenerateBindingRedirectsIntermediateAppConfig describes exactly the pain I've been going through, and even helpfully references MSBuild should handle paths longer than MAX_PATH.

I'll add my little bit of feedback to the former issue as soon as I've published this post (so I can link to it).

Step 8: Decide what to do

My options appear to be:

1. Change the project name or the directory structure
2. Ask everyone on the team to work from a really short root directory name
3. Downgrade my dependencies again
4. Live with Visual Studio not building a few solutions properly - we can still work in Visual Studio, so long as we don't need a full build in it.

I went with option 4, and published this blog post.

Step 9: Embrace community feedback

Theose were the only options I considered at the time of original writing, partly as I was somewhat scared of this suggestion, being inexperienced in msbuild. However, combining that with a Directory.Build.targets file (which I didn't know about before being tipped off by Nick Guerrera), it was very easy to implement a repo-wide workaround.

My friend Kirill Osenkov also pointed me at his MSBuild Log Viewer and pointed out that if it failed in Visual Studio, it would probably fail from msbuild on the command line. He was right, and the log viewer pinpointed what it was trying to write, which would have saved time.

Points to note

• Situations where you have a working case and a broken case are great. They can help you validate your problem (does it go away if I revert to the previous commit?) and find the cause (what's the difference in output between working and broken?)
• Temporary approaches to double-check the diagnosis (such as me renaming my source directory to "short") can be useful - don't forget to undo them though!
• If you've run into a problem, someone else probably has too
• The .NET community can be really amazing - I was really impressed by the response here
• Tools are often available - in this case, msbuildlog.com would have saved me quite a bit of time.