`dotnet pack` for file-based apps fail determinism check

[richlander]

I packed a file-based tool. It fails NugetPackageExplorer determinism check.

My D.B.p:

<Project>
  <PropertyGroup>
    <NoWarn>$(NoWarn);NU5119</NoWarn>
  </PropertyGroup>
  <PropertyGroup Condition="'$(OfficialBuild)' == 'true'">
    <Deterministic>true</Deterministic>
    <ContinuousIntegrationBuild>true</ContinuousIntegrationBuild>
  </PropertyGroup>
  <ItemGroup>
    <None Include="README.md" Pack="true" PackagePath="\"/>
  </ItemGroup>
</Project>

Command to pack:

dotnet pack -p:OfficialBuild=true dotnetsay.cs -bl

App: https://github.com/richlander/dotnetsay

I wrote a little utility to look deeper:

$ dotnet run -- ~/git/dotnetsay/artifacts/dotnetsay/dotnetsay.3.0.3.nupkg
Processing local package: dotnetsay.3.0.3.nupkg
Package extracted successfully.

Found 1 PDB file(s):

=== tools/net8.0/any/dotnetsay.pdb ===
Paths Normalized: Yes
{
  "documents": {
    "/_/*": "https://raw.githubusercontent.com/richlander/dotnetsay/3662a460f940d95d65b65ec2d84b2f8ad360a3da/*"
  }
}

Checking 1 DLL(s) for embedded PDBs:

=== tools/net8.0/any/dotnetsay.dll ===
Reproducible Flag: Yes
PDB Path Normalized: No
Overall Deterministic: ✗ No

rich@richs-MacBook-Pro SourceLinkExtractor % dotnet run -- ~/git/dotnetsay/artifacts/dotnetsay/dotnetsay.3.0.3.nupkg
Processing local package: dotnetsay.3.0.3.nupkg
Package extracted successfully.

Found 1 PDB file(s):

=== tools/net8.0/any/dotnetsay.pdb ===
Paths Normalized: Yes
{
  "documents": {
    "/_/*": "https://raw.githubusercontent.com/richlander/dotnetsay/3662a460f940d95d65b65ec2d84b2f8ad360a3da/*"
  }
}

Checking 1 DLL(s) for embedded PDBs:

=== tools/net8.0/any/dotnetsay.dll ===
Reproducible Flag: Yes
PDB Path Normalized: No
Overall Deterministic: ✗ No

Non-normalized paths found:
  CodeView PDB Path: /Users/rich/Library/Application Support/dotnet/runfile/dotnetsay-62e266fb6e3dc33ba76d6e1e30a312dc0af10f763ebfa338fe8b91ad7f658370/obj/release/dotnetsay.pdb

[richlander]

@DamianEdwards @baronfel

[richlander]

Just ran sourcelink on the .pdb in the artifacts directory.

$ sourcelink print-documents artifacts/dotnetsay/dotnetsay.pdb
f09f41a2a2a96e0da4b076ac849602a6d49a1c84f255ffd3af10c40b1d264298 sha256 csharp /_/dotnetsay.cs
8f86a2cbf00ff3eee0bda2bc0eb5e1fc19c8544d5dc650c53d0132325956b592 sha256 csharp /Users/rich/Library/Application Support/dotnet/runfile/dotnetsay-62e266fb6e3dc33ba76d6e1e30a312dc0af10f763ebfa338fe8b91ad7f658370/obj/release/dotnetsay.GlobalUsings.g.cs
ae41ca1e67556e4b0622d82a2122a04f17722c0c19d3b2f1fbcab8a2aaf16d2e sha256 csharp /Users/rich/Library/Application Support/dotnet/runfile/dotnetsay-62e266fb6e3dc33ba76d6e1e30a312dc0af10f763ebfa338fe8b91ad7f658370/obj/release/.NETCoreApp,Version=v8.0.AssemblyAttributes.cs
a9117dbed36b338b966c4b02eb2b102246617a846c9ce2dd2937128233a12efa sha256 csharp /Users/rich/Library/Application Support/dotnet/runfile/dotnetsay-62e266fb6e3dc33ba76d6e1e30a312dc0af10f763ebfa338fe8b91ad7f658370/obj/release/dotnetsay.AssemblyInfo.cs

Is this because I didn't enable EmbedUntrackedSources?

[baronfel]

That's a good hypothesis - I assumed that had been done here. You've gotta embed generated files to get green.

[richlander]

What about this statement?

  <PropertyGroup>
    <!-- Optional (set by default in .NET SDK 8+): Embed source files that are not tracked by the source control manager in the PDB -->
    <EmbedUntrackedSources>true</EmbedUntrackedSources>
  </PropertyGroup>

https://github.com/dotnet/sourcelink/blob/main/README.md#using-source-link-in-net-projects

[richlander]

I set that. It didn't work. I suspect it EmbedUntrackedSources is already set, so re-setting it doesn't work.

[DamianEdwards]

@baronfel is this related to the other issue you filed about not tracking inputs that aren't in the project cone?

[jjonescz]

Normally the source directory has path mapping setup:

  <Target Name="_SetPathMapFromSourceRoots"
          DependsOnTargets="InitializeSourceRootMappedPaths"
          BeforeTargets="CoreCompile"
          Condition="'$(DeterministicSourcePaths)' == 'true'">

    <ItemGroup>
      <_TopLevelSourceRoot Include="@(SourceRoot)" Condition="'%(SourceRoot.NestedRoot)' == ''">
        <EscapedKey>$([MSBuild]::ValueOrDefault('%(Identity)', '').Replace(',', ',,').Replace('=', '=='))</EscapedKey>
        <EscapedValue>$([MSBuild]::ValueOrDefault('%(MappedPath)', '').Replace(',', ',,').Replace('=', '=='))</EscapedValue>
      </_TopLevelSourceRoot>
    </ItemGroup>

    <PropertyGroup Condition="'@(_TopLevelSourceRoot)' != ''">
      <!--
        Prepend the SourceRoot.MappedPath values to PathMap, if it already has a value.
        For each emitted source path the compiler applies the first mapping that matches the path.
        PathMap values set previously will thus only be applied if the mapping provided by 
        SourceRoot.MappedPath doesn't match. Since SourceRoot.MappedPath is also used by SourceLink 
        preferring it over manually set PathMap ensures that PathMap is consistent with SourceLink.
      -->
      <PathMap>@(_TopLevelSourceRoot->'%(EscapedKey)=%(EscapedValue)', ','),$(PathMap)</PathMap>
    </PropertyGroup>
  </Target>

The artifacts dir which contains sources isn't included. It seems this can be fixed simply by adding an item like this:

<SourceRoot Include="$([MSBuild]::EnsureTrailingSlash($(ArtifactsPath)))" />

[jjonescz]

I imagine this would fail for normal project-based apps too if you set ArtifactsPath outside the Git repo root, so perhaps this should be fixed generally in the SDK targets, not just for file-based apps?

[baronfel]

ArtifactsPath outside the Git repo root

Nice insight! I'll vibe-code out a test app and verify that a project-based app using artifacts exhibits the same failure mode. If they do, then it seems entirely reasonable to add to the ArtifactsPath-related logic the chunk that you recommend to fix the issue for the entire category of programs.

[baronfel]

Confirmed that this applies to any project that uses ArtifactsPath with a layout outside of the repo root. @jjonescz's fix almost worked exactly as written - the problem I had was that the source root must be canonicalized when added. So doing something like

<ArtifactsPath>$(MSBuildThisFileDirectory)..\artifacts\</ArtifactsPath>

didn't work because the literal ..\ would end up in some of the paths, and parts of the PathMap infrastructure didn't understand how to normalize those.

So I see two things that need to happen here:

• In DefaultArtifactsPaths.props we need to add the computed ArtifactsPath to SourceRoot
  • for now we should canonicalize this path
• it looks like Roslyn owns the targets that do source root mapping? this feels like an incorrect layering to me, but w/e - in those targets, probably InitializeSourceRootMappedPaths itself, the SourceRoot items should be canonicalized so that all downstream consumers of those Items have absolute paths to work with.

I've created issues for each of these items, and Copilot is crunching away on the SDK-side of the problem.