fix: util/path: CheckSystemDriveAndRemoveDriveLetter to preserve /

[profnandaa]

The call to CheckSystemDriveAndRemoveDriveLetter() does not preserve the trailing / or \\. This happens because filepath.Clean() strips away any trailing slashes. For example /sample/ will be \\sample on Windows. This function was mainly written for Windows scenarios, which have System Drive Letters like C:/, etc.

This was causing cases like COPY testfile /testdir/ to be intepreted as COPY testfile /testdir, and if testdir is not explictly created before the call, it ends up being treated as a destination file other than a directory.

Fix this by checking that if we have a trailing / or \\, we preserve it after the call to filepath.Clean().

Fixes #5249

PS.

Also fixed for cross-building Windows from Linux, that would fail silently:

Repro dockerfile without RUN:

FROM mcr.microsoft.com/windows/nanoserver:ltsc2022
COPY test1.txt /sample/
#RUN type \sample\test1.txt

Build log:

$ docker buildx build --platform windows/amd64 `
    --builder buildkitd-dev --no-cache --tag=windows-test . `
    --progress plain `
    --output type=local,dest=./output

#6 [2/2] COPY test1.txt /sample/
#6 DONE 0.1s

#7 exporting to client directory
#7 copying files 31B
#7 copying files 230.88MB 0.8s done
#7 DONE 0.8s

Checking results, sample is a file instead of a directory:

# before
$ ls -l output/sample
-rw-r--r-- 1 root root 6 Sep 24 08:57 output/sample

# after
$ ls -l output/sample
total 4
-rw-r--r-- 1 root root 6 Sep 24 08:57 test1.txt

NOTE: also covered cases like, where platform-specific filepath.Clean() won't strip out the \\ on Linux:

COPY test1.txt \\sample\\

[profnandaa]

Looking into the CI failures...

[profnandaa]

Found the regression caused by this, thankfully coz of the checked in integration tests!

Dockerfile:

FROM mcr.microsoft.com/windows/nanoserver:ltsc2022
COPY test1.txt /sample/
RUN type \sample\test1.txt

COPY test1.txt /
COPY test1.txt /test2.txt

RUN type test1.txt
RUN type test2.txt

Build log:

Dockerfile:5
--------------------
   3 |     RUN type \sample\test1.txt
   4 |     
   5 | >>> COPY test1.txt /
   6 |     COPY test1.txt /test2.txt
   7 |     
--------------------
error: failed to solve: removing drive letter: UNC paths are not supported
Process 5812 has exited with status 1

Fixing.

[profnandaa]

CI failure seems unrelated?

=== FAIL: client TestClientGatewayIntegration/TestClientGatewayContainerHostNetworkingValidation/worker=containerd/netmode=host (0.97s)
    build_test.go:2133: 
        	Error Trace:	/src/client/build_test.go:2133
        	            				/src/client/build_test.go:2031
        	            				/src/util/testutil/integration/run.go:97
        	            				/src/util/testutil/integration/run.go:211
        	Error:      	Received unexpected error:
        	            	expecting started message, got *moby_buildkit_v1_frontend.ExecMessage_Exit

[profnandaa]

PTAL again, addressed the comments. @tonistiigi @gabriel-samfira

[gabriel-samfira]

LGTM. Waiting for a second review before merging.

[profnandaa]

Thanks!

…

On Wed, Oct 9, 2024, 12:28 Gabriel ***@***.***> wrote: LGTM. Waiting for a second review before merging. — Reply to this email directly, view it on GitHub <#5317 (comment)>, or unsubscribe <https:/notifications/unsubscribe-auth/AAB7ZEMYKEKKZE5GDTAXVQLZ2TZKPAVCNFSM6AAAAABN6FHHEWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMBRHAYDSOBYGI> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[profnandaa]

@crazy-max @tonistiigi -- can take a look at this one?

[tonistiigi]

Still don't understand this in relation to #5317 (comment) comment.

We have a NormalizePath function that takes keepSlash parameter. Afaics that implementation is still linux-specific and does not understand windows inputOS with backslashes.

This adds a new function cleanPath() that doesn't take keepSlash but automatically adds / is input path contained it. That is not the behavior of path.Clean so minimally function should be called cleanPathKeepSlash(), but ideally we should keep the functionality to keep slash into a single function and not multiple, and NormalizePath should not be linux-specific.

[tonistiigi]

usage of filepath.Separator means current OS specific behavior, not inputOS specific behavior.

[profnandaa]

Sure, will fix that. Had this before bringing back inputOS as a param...

[profnandaa]

Hi @tonistiigi --

Still don't understand this in relation to #5317 (comment) comment.

We have a NormalizePath function that takes keepSlash parameter. Afaics that implementation is still linux-specific and does not understand windows inputOS with backslashes.

The problem is that we don't get a chance to get to NormalizePath, the trailing slash is stripped away at CheckSystemDriveAndRemoveDriveLetter. So, there's nothing to keep by that time:

buildkit/frontend/dockerfile/dockerfile2llb/convert.go

Lines 1801 to 1822 in 4b4a9aa

	func pathRelativeToWorkingDir(s llb.State, p string, platform ocispecs.Platform) (string, error) {
	dir, err := s.GetDir(context.TODO(), llb.Platform(platform))
	if err != nil {
	return "", err
	}
	p, err = system.CheckSystemDriveAndRemoveDriveLetter(p, platform.OS)
	if err != nil {
	return "", errors.Wrap(err, "removing drive letter")
	}
	if system.IsAbs(p, platform.OS) {
	return system.NormalizePath("/", p, platform.OS, true)
	}
	// add slashes for "" and "." paths
	// "" is treated as current directory and not necessariy root
	if p == "." || p == "" {
	p = "./"
	}
	return system.NormalizePath(dir, p, platform.OS, true)
	}

This adds a new function cleanPath() that doesn't take keepSlash but automatically adds / is input path contained it. That is not the behavior of path.Clean so minimally function should be called cleanPathKeepSlash(), but ideally we should keep the functionality to keep slash into a single function and not multiple, and NormalizePath should not be linux-specific.

I'm okay with renaming this for clarity. On the second part, the only problem is that we don't get to the NormalizePath function..
Another idea I'd toiled with is to have add a keepSlash param to CheckSystemDriveAndRemoveDriveLetter; but since this is already an exported function, was going to break the backward compatibility.

The more simplistic option I had is just adding back the trailing slash before we call into NormalizePath on L1821; but felt that will be too patchy...

Your thoughts?

[tonistiigi]

but since this is already an exported function

Don't worry about Go backwards compatibility. We only care about API backwards compatibility.

The problem is that we don't get a chance to get to NormalizePath, #5249 (comment). So, there's nothing to keep by that time:

NormalizePath also calls into CheckSystemDriveAndRemoveDriveLetter (2x actually) so it looks like there might be some duplication in the codepath. Better to make it clear which functions responsibility it is to keep the slash and make sure this implementation is cross-platform capable.

[profnandaa]

but since this is already an exported function

Don't worry about Go backwards compatibility. We only care about API backwards compatibility.

Thanks for this, we can then have keepSlash (preferably, be declared once and) run through both functions for consistency.

The problem is that we don't get a chance to get to NormalizePath, #5249 (comment). So, there's nothing to keep by that time:

NormalizePath also calls into CheckSystemDriveAndRemoveDriveLetter (2x actually) so it looks like there might be some duplication in the codepath. Better to make it clear which functions responsibility it is to keep the slash and make sure this implementation is cross-platform capable.

I also noticed that multiple runs through CheckSystemDriveAndRemoveDriveLetter; I can work on some optimizations as a follow up.

[profnandaa]

PTAL at the new changes I've made from the review recommendations. @gabriel-samfira @tonistiigi

[profnandaa]

refactored this since the new cleanPath called in CheckSystemDriveAndRemoveDriveLetter handles this.

[profnandaa]

Will investigate and fix the CI failures.