Add --parents opt-in flag for ADD/COPY (labs) #3001

DYefimov · 2022-08-04T14:51:49Z

fixes / addresses add parents option to dockerfile COPY command moby#35639
addresses Dockerfile COPY with file globs will copy files from subdirectories to the destination directory moby#15858
addresses Dockerfile COPY [dir1/* destdir] copies dir1/dir2/* moby#29211
addresses [Docker build] add all files of type docker/roadmap#107
addresses COPY --from does not preserve permissions on top folder moby#38710

.parents_test.sh:

#!/usr/bin/env sh
set -e

TMP_DIR=$(mktemp -dt parents_test.XXXXXXXX)

echo "Created \"$TMP_DIR\""
trap "echo \"Removing \\\"$TMP_DIR\\\"\"; rm -rf \"$TMP_DIR\"" EXIT

mkdir -p $TMP_DIR/src/dir1
mkdir -p $TMP_DIR/src/dir2/subdir2

touch $TMP_DIR/src/dir1/file1-1.py
touch $TMP_DIR/src/dir1/file1-2.go
touch $TMP_DIR/src/dir2/subdir2/file2-1.go
touch $TMP_DIR/src/dir2/subdir2/file2-2.js
touch $TMP_DIR/src/dir2/subdir2/file2-3.go

cat << DOCKERFILE > $TMP_DIR/dockerfile
# syntax=dyefimov/dockerfile:copy_parents-labs

FROM scratch

COPY ./src/*/*/*.go ./dst1/
COPY --parents ./src/*/*/*.go ./dst2/

CMD ["echo", "hello-world"]
DOCKERFILE

echo "======================================================="
SOCKET_DIR=$(realpath $(dirname $0))/.tmp
mkdir -p $SOCKET_DIR
sudo buildctl \
    --addr=unix://$SOCKET_DIR/buildkitd.socket \
    build \
        --frontend=dockerfile.v0 \
        --local context=$TMP_DIR \
        --local dockerfile=$TMP_DIR \
        --no-cache \
        --output=type=docker,name=parents_test | docker load

echo "======================================================="
echo "source:"
find $TMP_DIR -printf '%P\n' | grep -v '^[[:space:]]*$' | sort

echo "======================================================="
docker container rm -f parents_test > /dev/null 2>&1
docker container create --name=parents_test parents_test > /dev/null 2>&1
echo "COPY:"
docker container export parents_test | tar tf - | grep "dst1" | sed 's#^dst1/##' | grep -v '^[[:space:]]*$' | sort
echo "COPY --parents:"
docker container export parents_test | tar tf - | grep "dst2" | sed 's#^dst2/##' | grep -v '^[[:space:]]*$' | sort
docker container rm -f parents_test > /dev/null 2>&1

echo "======================================================="

.parents_test.sh output:

$ ./.parents_test.sh 
Created "/tmp/parents_test.KCR1qmQ3"
=======================================================
[+] Building 7.4s (8/8) FINISHED                                                                               
 => [internal] load build definition from Dockerfile                                                      0.0s
 => => transferring dockerfile: 196B                                                                      0.0s
 => [internal] load .dockerignore                                                                         0.0s
 => => transferring context: 2B                                                                           0.0s
 => resolve image config for docker.io/dyefimov/dockerfile:copy_parents-labs                              0.8s
 => CACHED docker-image://docker.io/dyefimov/dockerfile:copy_parents-labs@sha256:d610b759d921bc35515d468  0.0s
 => => resolve docker.io/dyefimov/dockerfile:copy_parents-labs@sha256:d610b759d921bc35515d468718f2609cd6  0.0s
 => [internal] load build context                                                                         0.0s
 => => transferring context: 185B                                                                         0.0s
 => [1/2] COPY ./src/*/*/*.go ./dst1/                                                                     2.6s
 => [2/2] COPY --parents ./src/*/*/*.go ./dst2/                                                           3.0s
 => exporting to oci image format                                                                         0.3s
 => => exporting layers                                                                                   0.3s
 => => exporting manifest sha256:24b4391d73d9094f6cd20457fdee6ac7fad2805988d095faabf6b9fe72ffacb6         0.0s
 => => exporting config sha256:5f6fa8bc19204a198bbb3b996f7fceb8a2674275b3182f7e804266ebc223c302           0.0s
 => => sending tarball                                                                                    0.0s
0fd58aaab3ea: Loading layer [==================================================>]     145B/145B
2fb6a3b6052f: Loading layer [==================================================>]     193B/193B
The image parents_test:latest already exists, renaming the old one with ID sha256:de30362eb4cd1c143121ff861bc844e83218abe87788889a4d707ef26209d363 to empty string
Loaded image: parents_test:latest
=======================================================
source:
dockerfile
src
src/dir1
src/dir1/file1-1.py
src/dir1/file1-2.go
src/dir2
src/dir2/subdir2
src/dir2/subdir2/file2-1.go
src/dir2/subdir2/file2-2.js
src/dir2/subdir2/file2-3.go
=======================================================
COPY:
file2-1.go
file2-3.go
COPY --parents:
src/
src/dir2/
src/dir2/subdir2/
src/dir2/subdir2/file2-1.go
src/dir2/subdir2/file2-3.go
=======================================================
Removing "/tmp/parents_test.KCR1qmQ3"

Signed-off-by: Dmitrii Efimov [email protected]

jedevc

Thanks for this! This looks like the right direction 🎉

I've left a few code comments inline, but I think the main thing to work on next will be a test suite, there's some edge cases we need to explore properly, like how this should interact with remote files, tar files, heredocs, etc, etc.

I'm particularly curious how this interacts with this from the dockerfile reference (trailing slash behavior is... hard):

If is any other kind of file, it is copied individually along with its metadata. In this case, if ends with a trailing slash /, it will be considered a directory and the contents of will be written at /base().

For example, cp produces this error:

cp: with --parents, the destination must be a directory
Try 'cp --help' for more information.

So maybe we should explicitly fail like cp does if the destination isn't a directory? Although, the behavior is already a little strange in buildkit.

Anyways, I think these issues should fall out as we build up a test suite. Nice work so far!

frontend/dockerfile/instructions/parse.go

solver/llbsolver/file/backend.go

frontend/dockerfile/dockerfile2llb/convert.go

tonistiigi · 2022-08-04T18:07:04Z

fyi @aaronlehmann

DYefimov · 2022-08-06T11:12:32Z

Excellent 🎉 Thanks.
Yep, I already has begun preparing the test suite.

solver/llbsolver/file/backend.go

DYefimov · 2022-08-08T14:14:57Z

How would you like --parents to behave in remote case you mentioned? (or URL/git cases? Not sure about current git behavior yet though - git looks very complicated inside)
Right now --parents is applied only in local/tar scenarios.
Guess we should treat URL/git as a separate decoupled "transport (or src) layer" and apply --parents indepently afterwards?

jedevc · 2022-08-08T15:19:58Z

URLs shouldn't be the determining factor here I don't think - the complexity comes with tars/gits/etc.

For a tar, I'd expect that it behave exactly like a directory, so with --parents, the entire contents being unpacked into the destination directory. But actually, I think that's how without --parents works, so I think it should be safe to discard. Same applies for .git.

thaJeztah · 2022-08-16T09:39:13Z

Looks like this needs a rebase

DYefimov · 2022-08-16T18:19:01Z

Looks like this needs a rebase

Just came back from vacation yesterday, rebase, some fixes and test suite coming shortly.

thaJeztah · 2022-08-24T14:08:37Z

I think you may have rebased with an outdated local state (I see some unrelated commits made their way in); let me know if you need help with rebasing 👍

DYefimov · 2022-08-24T14:23:31Z

I think you may have rebased with an outdated local state (I see some unrelated commits made their way in); let me know if you need help with rebasing +1

Yep, thanks.
The diff to HEAD still gives the correct change set.
Guess that's not a big deal, if I'll squash all but relevant and force push with the next rebase?

jedevc · 2022-09-07T12:41:43Z

👋 @DYefimov just wanted to follow up and check to see you weren't blocked on anything?

DYefimov · 2022-09-07T14:56:15Z

@jedevc
In fact I'm kind of do.
Failing in getting my head around some weird mounting problem.
Can't get TestParentsAddURL test compiling, and for that matter just any test using TestServer inside is failing with this strange mounting error.
I tried without container-in-container scheme with the same outcome. Other tests I've added are working just fine. Maybe I'm using the test suite the wrong way?
TEST_KEEP_CACHE=1 DOCKERFILE_RELEASES=labs TESTFLAGS="--run /TestParents*/worker=oci$/ -v" ./hack/test dockerfile

output

    dockerfile_parents_test.go:229: 
                Error Trace:    dockerfile_parents_test.go:229
                                                        run.go:85
                                                        run.go:184
                Error:          Received unexpected error:
                                lstat /tmp/bktest_buildkitd2939727001/tmp/buildkit-mount378098859/tmp/bktest_buildkitd2939727001/tmp/buildkit-mount378098859: no such file or directory
                                github.com/moby/buildkit/util/stack.Enable
                                        /src/util/stack/stack.go:77
                                github.com/moby/buildkit/util/grpcerrors.FromGRPC
                                        /src/util/grpcerrors/grpcerrors.go:196
                                github.com/moby/buildkit/util/grpcerrors.UnaryClientInterceptor
                                        /src/util/grpcerrors/intercept.go:41
                                google.golang.org/grpc.(*ClientConn).Invoke
                                        /src/vendor/google.golang.org/grpc/call.go:35
                                github.com/moby/buildkit/api/services/control.(*controlClient).Solve
                                        /src/api/services/control/control.pb.go:1535
                                github.com/moby/buildkit/client.(*Client).solve.func2
                                        /src/client/solve.go:231
                                golang.org/x/sync/errgroup.(*Group).Go.func1
                                        /src/vendor/golang.org/x/sync/errgroup/errgroup.go:57
                                runtime.goexit
                                        /usr/local/go/src/runtime/asm_amd64.s:1571
                                failed to solve
                                github.com/moby/buildkit/client.(*Client).solve.func2
                                        /src/client/solve.go:244
                                golang.org/x/sync/errgroup.(*Group).Go.func1
                                        /src/vendor/golang.org/x/sync/errgroup/errgroup.go:57
                                runtime.goexit
                                        /usr/local/go/src/runtime/asm_amd64.s:1571
                Test:           TestIntegration/TestParentsAddUrl/worker=oci/frontend=gateway
    sandbox.go:234: stdout: /usr/bin/buildkitd
    sandbox.go:234: stderr: /usr/bin/buildkitd
    sandbox.go:237: > startCmd 2022-09-07 14:38:38.14992294 +0000 UTC m=+7.105587618 [buildkitd --oci-worker=true --containerd-worker=false --oci-worker-gc=false --oci-worker-labels=org.mobyproject.buildkit.worker.sandbox=true --config=/tmp/bktest_config3250988747/buildkitd.toml --root /tmp/bktest_buildkitd2939727001 --addr unix:///tmp/bktest_buildkitd2939727001/buildkitd.sock --debug]
    sandbox.go:237: time="2022-09-07T14:38:40Z" level=info msg="auto snapshotter: using overlayfs"
    sandbox.go:237: time="2022-09-07T14:38:41Z" level=info msg="found worker \"l7i4zpx8mh85wo1txmozkpz58\", labels=map[org.mobyproject.buildkit.worker.executor:oci org.mobyproject.buildkit.worker.hostname:705f791a78df org.mobyproject.buildkit.worker.network:cni org.mobyproject.buildkit.worker.oci.process-mode:sandbox org.mobyproject.buildkit.worker.sandbox:true org.mobyproject.buildkit.worker.snapshotter:overlayfs], platforms=[linux/amd64 linux/amd64/v2 linux/amd64/v3 linux/386]"
    sandbox.go:237: time="2022-09-07T14:38:41Z" level=info msg="found 1 workers, default=\"l7i4zpx8mh85wo1txmozkpz58\""
    sandbox.go:237: time="2022-09-07T14:38:41Z" level=warning msg="currently, only the default worker can be used."
    sandbox.go:237: time="2022-09-07T14:38:41Z" level=info msg="running server on /tmp/bktest_buildkitd2939727001/buildkitd.sock"
    sandbox.go:237: time="2022-09-07T14:38:41Z" level=debug msg="session started"
    sandbox.go:237: time="2022-09-07T14:38:41Z" level=debug msg=resolving host="localhost:36511"
    sandbox.go:237: time="2022-09-07T14:38:41Z" level=debug msg="do request" host="localhost:36511" request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=buildkit/v0.10-dev request.method=HEAD url="http://localhost:36511/v2/buildkit_test/6mwgccd4ycjb9fmef2jm7icr3/manifests/latest?ns=docker.io"
    sandbox.go:237: time="2022-09-07T14:38:41Z" level=debug msg="fetch response received" host="localhost:36511" response.header.content-length=507 response.header.content-type=application/vnd.oci.image.manifest.v1+json response.header.date="Wed, 07 Sep 2022 14:38:41 GMT" response.header.docker-content-digest="sha256:c5f8c4651185c37ebb4a54be11d545b9469bb9c60d6d6b40c1bfe200416f1fa2" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:c5f8c4651185c37ebb4a54be11d545b9469bb9c60d6d6b40c1bfe200416f1fa2\"" response.status="200 OK" url="http://localhost:36511/v2/buildkit_test/6mwgccd4ycjb9fmef2jm7icr3/manifests/latest?ns=docker.io"
    sandbox.go:237: time="2022-09-07T14:38:41Z" level=debug msg=resolved desc.digest="sha256:c5f8c4651185c37ebb4a54be11d545b9469bb9c60d6d6b40c1bfe200416f1fa2" host="localhost:36511"
    sandbox.go:237: time="2022-09-07T14:38:41Z" level=debug msg=fetch digest="sha256:c5f8c4651185c37ebb4a54be11d545b9469bb9c60d6d6b40c1bfe200416f1fa2" mediatype=application/vnd.oci.image.manifest.v1+json size=507
    sandbox.go:237: time="2022-09-07T14:38:41Z" level=debug msg="do request" digest="sha256:c5f8c4651185c37ebb4a54be11d545b9469bb9c60d6d6b40c1bfe200416f1fa2" mediatype=application/vnd.oci.image.manifest.v1+json request.header.accept="application/vnd.oci.image.manifest.v1+json, */*" request.header.user-agent=buildkit/v0.10-dev request.method=GET size=507 url="http://localhost:36511/v2/buildkit_test/6mwgccd4ycjb9fmef2jm7icr3/manifests/sha256:c5f8c4651185c37ebb4a54be11d545b9469bb9c60d6d6b40c1bfe200416f1fa2?ns=docker.io"
    sandbox.go:237: time="2022-09-07T14:38:41Z" level=debug msg="fetch response received" digest="sha256:c5f8c4651185c37ebb4a54be11d545b9469bb9c60d6d6b40c1bfe200416f1fa2" mediatype=application/vnd.oci.image.manifest.v1+json response.header.content-length=507 response.header.content-type=application/vnd.oci.image.manifest.v1+json response.header.date="Wed, 07 Sep 2022 14:38:41 GMT" response.header.docker-content-digest="sha256:c5f8c4651185c37ebb4a54be11d545b9469bb9c60d6d6b40c1bfe200416f1fa2" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:c5f8c4651185c37ebb4a54be11d545b9469bb9c60d6d6b40c1bfe200416f1fa2\"" response.status="200 OK" size=507 url="http://localhost:36511/v2/buildkit_test/6mwgccd4ycjb9fmef2jm7icr3/manifests/sha256:c5f8c4651185c37ebb4a54be11d545b9469bb9c60d6d6b40c1bfe200416f1fa2?ns=docker.io"
    sandbox.go:237: time="2022-09-07T14:38:41Z" level=debug msg=fetch digest="sha256:15346172750ff1cd7d1645f8f66b3f91399d569a4ad2ff718c0a6c18da904523" mediatype=application/vnd.oci.image.config.v1+json size=1843
    sandbox.go:237: time="2022-09-07T14:38:41Z" level=debug msg="do request" digest="sha256:15346172750ff1cd7d1645f8f66b3f91399d569a4ad2ff718c0a6c18da904523" mediatype=application/vnd.oci.image.config.v1+json request.header.accept="application/vnd.oci.image.config.v1+json, */*" request.header.user-agent=buildkit/v0.10-dev request.method=GET size=1843 url="http://localhost:36511/v2/buildkit_test/6mwgccd4ycjb9fmef2jm7icr3/blobs/sha256:15346172750ff1cd7d1645f8f66b3f91399d569a4ad2ff718c0a6c18da904523?ns=docker.io"
    sandbox.go:237: time="2022-09-07T14:38:41Z" level=debug msg="fetch response received" digest="sha256:15346172750ff1cd7d1645f8f66b3f91399d569a4ad2ff718c0a6c18da904523" mediatype=application/vnd.oci.image.config.v1+json response.header.accept-ranges=bytes response.header.cache-control="max-age=31536000" response.header.content-length=1843 response.header.content-type=application/octet-stream response.header.date="Wed, 07 Sep 2022 14:38:41 GMT" response.header.docker-content-digest="sha256:15346172750ff1cd7d1645f8f66b3f91399d569a4ad2ff718c0a6c18da904523" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:15346172750ff1cd7d1645f8f66b3f91399d569a4ad2ff718c0a6c18da904523\"" response.status="200 OK" size=1843 url="http://localhost:36511/v2/buildkit_test/6mwgccd4ycjb9fmef2jm7icr3/blobs/sha256:15346172750ff1cd7d1645f8f66b3f91399d569a4ad2ff718c0a6c18da904523?ns=docker.io"
    sandbox.go:237: time="2022-09-07T14:38:41Z" level=debug msg=fetch digest="sha256:c5f8c4651185c37ebb4a54be11d545b9469bb9c60d6d6b40c1bfe200416f1fa2" mediatype=application/vnd.oci.image.manifest.v1+json size=507
    sandbox.go:237: time="2022-09-07T14:38:41Z" level=debug msg=fetch digest="sha256:15346172750ff1cd7d1645f8f66b3f91399d569a4ad2ff718c0a6c18da904523" mediatype=application/vnd.oci.image.config.v1+json size=1843
    sandbox.go:237: time="2022-09-07T14:38:42Z" level=debug msg=fetch digest="sha256:419a7037d29b1f42df90debfd2b4902306076f23f7d831da04489f4cf858e4cc" mediatype=application/vnd.oci.image.layer.v1.tar+gzip size=10572963
    sandbox.go:237: time="2022-09-07T14:38:42Z" level=debug msg="do request" digest="sha256:419a7037d29b1f42df90debfd2b4902306076f23f7d831da04489f4cf858e4cc" mediatype=application/vnd.oci.image.layer.v1.tar+gzip request.header.accept="application/vnd.oci.image.layer.v1.tar+gzip, */*" request.header.user-agent=containerd/1.6.6+unknown request.method=GET size=10572963 url="http://localhost:36511/v2/buildkit_test/6mwgccd4ycjb9fmef2jm7icr3/blobs/sha256:419a7037d29b1f42df90debfd2b4902306076f23f7d831da04489f4cf858e4cc?ns=docker.io"
    sandbox.go:237: time="2022-09-07T14:38:42Z" level=debug msg="fetch response received" digest="sha256:419a7037d29b1f42df90debfd2b4902306076f23f7d831da04489f4cf858e4cc" mediatype=application/vnd.oci.image.layer.v1.tar+gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31536000" response.header.content-length=10572963 response.header.content-type=application/octet-stream response.header.date="Wed, 07 Sep 2022 14:38:42 GMT" response.header.docker-content-digest="sha256:419a7037d29b1f42df90debfd2b4902306076f23f7d831da04489f4cf858e4cc" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:419a7037d29b1f42df90debfd2b4902306076f23f7d831da04489f4cf858e4cc\"" response.status="200 OK" size=10572963 url="http://localhost:36511/v2/buildkit_test/6mwgccd4ycjb9fmef2jm7icr3/blobs/sha256:419a7037d29b1f42df90debfd2b4902306076f23f7d831da04489f4cf858e4cc?ns=docker.io"
    sandbox.go:237: time="2022-09-07T14:38:42Z" level=debug msg="unpigz not found, falling back to go gzip" error="exec: \"unpigz\": executable file not found in $PATH"
    sandbox.go:237: time="2022-09-07T14:38:42Z" level=debug msg="diff applied" d=241.26513ms digest="sha256:419a7037d29b1f42df90debfd2b4902306076f23f7d831da04489f4cf858e4cc" media=application/vnd.oci.image.layer.v1.tar+gzip size=10572963
    sandbox.go:237: time="2022-09-07T14:38:42Z" level=debug msg="serving grpc connection"
    sandbox.go:237: time="2022-09-07T14:38:42Z" level=debug msg="> creating fv9sy8pt0khy44hi1f70mh532 [/bin/dockerfile-frontend]"
    sandbox.go:237: time="2022-09-07T14:38:43Z" level=debug msg="new ref for local: sz0fgqxdj0z7hremziol9tbte" span="[internal] load build definition from Dockerfile"
    sandbox.go:237: time="2022-09-07T14:38:43Z" level=debug msg="diffcopy took: 5.141658ms" span="[internal] load build definition from Dockerfile"
    sandbox.go:237: time="2022-09-07T14:38:43Z" level=debug msg="new ref for local: sfezqludgl413om5nz5es7i7n" span="[internal] load .dockerignore"
    sandbox.go:237: time="2022-09-07T14:38:43Z" level=debug msg="diffcopy took: 7.630458ms" span="[internal] load .dockerignore"
    sandbox.go:237: time="2022-09-07T14:38:43Z" level=debug msg="saved sz0fgqxdj0z7hremziol9tbte as dockerfile:dockerfile:" span="[internal] load build definition from Dockerfile"
    sandbox.go:237: time="2022-09-07T14:38:43Z" level=debug msg="saved sfezqludgl413om5nz5es7i7n as context:context-.dockerignore:" span="[internal] load .dockerignore"
    sandbox.go:237: time="2022-09-07T14:38:44Z" level=debug msg=resolving host="localhost:36511"
    sandbox.go:237: time="2022-09-07T14:38:44Z" level=debug msg="do request" host="localhost:36511" request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=buildkit/v0.10-dev request.method=HEAD url="http://localhost:36511/v2/library/busybox/manifests/latest?ns=docker.io"
    sandbox.go:237: time="2022-09-07T14:38:44Z" level=debug msg="fetch response received" host="localhost:36511" response.header.content-length=527 response.header.content-type=application/vnd.docker.distribution.manifest.v2+json response.header.date="Wed, 07 Sep 2022 14:38:44 GMT" response.header.docker-content-digest="sha256:98de1ad411c6d08e50f26f392f3bc6cd65f686469b7c22a85c7b5fb1b820c154" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:98de1ad411c6d08e50f26f392f3bc6cd65f686469b7c22a85c7b5fb1b820c154\"" response.status="200 OK" url="http://localhost:36511/v2/library/busybox/manifests/latest?ns=docker.io"
    sandbox.go:237: time="2022-09-07T14:38:44Z" level=debug msg=resolved desc.digest="sha256:98de1ad411c6d08e50f26f392f3bc6cd65f686469b7c22a85c7b5fb1b820c154" host="localhost:36511"
    sandbox.go:237: time="2022-09-07T14:38:44Z" level=debug msg=fetch digest="sha256:98de1ad411c6d08e50f26f392f3bc6cd65f686469b7c22a85c7b5fb1b820c154" mediatype=application/vnd.docker.distribution.manifest.v2+json size=527
    sandbox.go:237: time="2022-09-07T14:38:44Z" level=debug msg="do request" digest="sha256:98de1ad411c6d08e50f26f392f3bc6cd65f686469b7c22a85c7b5fb1b820c154" mediatype=application/vnd.docker.distribution.manifest.v2+json request.header.accept="application/vnd.docker.distribution.manifest.v2+json, */*" request.header.user-agent=buildkit/v0.10-dev request.method=GET size=527 url="http://localhost:36511/v2/library/busybox/manifests/sha256:98de1ad411c6d08e50f26f392f3bc6cd65f686469b7c22a85c7b5fb1b820c154?ns=docker.io"
    sandbox.go:237: time="2022-09-07T14:38:44Z" level=debug msg="fetch response received" digest="sha256:98de1ad411c6d08e50f26f392f3bc6cd65f686469b7c22a85c7b5fb1b820c154" mediatype=application/vnd.docker.distribution.manifest.v2+json response.header.content-length=527 response.header.content-type=application/vnd.docker.distribution.manifest.v2+json response.header.date="Wed, 07 Sep 2022 14:38:44 GMT" response.header.docker-content-digest="sha256:98de1ad411c6d08e50f26f392f3bc6cd65f686469b7c22a85c7b5fb1b820c154" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:98de1ad411c6d08e50f26f392f3bc6cd65f686469b7c22a85c7b5fb1b820c154\"" response.status="200 OK" size=527 url="http://localhost:36511/v2/library/busybox/manifests/sha256:98de1ad411c6d08e50f26f392f3bc6cd65f686469b7c22a85c7b5fb1b820c154?ns=docker.io"
    sandbox.go:237: time="2022-09-07T14:38:44Z" level=debug msg=fetch digest="sha256:7a80323521ccd4c2b4b423fa6e38e5cea156600f40cd855e464cc52a321a24dd" mediatype=application/vnd.docker.container.image.v1+json size=1457
    sandbox.go:237: time="2022-09-07T14:38:44Z" level=debug msg="do request" digest="sha256:7a80323521ccd4c2b4b423fa6e38e5cea156600f40cd855e464cc52a321a24dd" mediatype=application/vnd.docker.container.image.v1+json request.header.accept="application/vnd.docker.container.image.v1+json, */*" request.header.user-agent=buildkit/v0.10-dev request.method=GET size=1457 url="http://localhost:36511/v2/library/busybox/blobs/sha256:7a80323521ccd4c2b4b423fa6e38e5cea156600f40cd855e464cc52a321a24dd?ns=docker.io"
    sandbox.go:237: time="2022-09-07T14:38:44Z" level=debug msg="fetch response received" digest="sha256:7a80323521ccd4c2b4b423fa6e38e5cea156600f40cd855e464cc52a321a24dd" mediatype=application/vnd.docker.container.image.v1+json response.header.accept-ranges=bytes response.header.cache-control="max-age=31536000" response.header.content-length=1457 response.header.content-type=application/octet-stream response.header.date="Wed, 07 Sep 2022 14:38:44 GMT" response.header.docker-content-digest="sha256:7a80323521ccd4c2b4b423fa6e38e5cea156600f40cd855e464cc52a321a24dd" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:7a80323521ccd4c2b4b423fa6e38e5cea156600f40cd855e464cc52a321a24dd\"" response.status="200 OK" size=1457 url="http://localhost:36511/v2/library/busybox/blobs/sha256:7a80323521ccd4c2b4b423fa6e38e5cea156600f40cd855e464cc52a321a24dd?ns=docker.io"
    sandbox.go:237: time="2022-09-07T14:38:44Z" level=debug msg=fetch digest="sha256:98de1ad411c6d08e50f26f392f3bc6cd65f686469b7c22a85c7b5fb1b820c154" mediatype=application/vnd.docker.distribution.manifest.v2+json size=527
    sandbox.go:237: time="2022-09-07T14:38:44Z" level=debug msg=fetch digest="sha256:7a80323521ccd4c2b4b423fa6e38e5cea156600f40cd855e464cc52a321a24dd" mediatype=application/vnd.docker.container.image.v1+json size=1457
    sandbox.go:237: time="2022-09-07T14:38:45Z" level=debug msg=fetch digest="sha256:50783e0dfb64b73019e973e7bce2c0d5a882301b781327ca153b876ad758dbd3" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=773262
    sandbox.go:237: time="2022-09-07T14:38:45Z" level=debug msg="do request" digest="sha256:50783e0dfb64b73019e973e7bce2c0d5a882301b781327ca153b876ad758dbd3" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.6.6+unknown request.method=GET size=773262 url="http://localhost:36511/v2/library/busybox/blobs/sha256:50783e0dfb64b73019e973e7bce2c0d5a882301b781327ca153b876ad758dbd3?ns=docker.io"
    sandbox.go:237: time="2022-09-07T14:38:45Z" level=debug msg="fetch response received" digest="sha256:50783e0dfb64b73019e973e7bce2c0d5a882301b781327ca153b876ad758dbd3" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31536000" response.header.content-length=773262 response.header.content-type=application/octet-stream response.header.date="Wed, 07 Sep 2022 14:38:45 GMT" response.header.docker-content-digest="sha256:50783e0dfb64b73019e973e7bce2c0d5a882301b781327ca153b876ad758dbd3" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:50783e0dfb64b73019e973e7bce2c0d5a882301b781327ca153b876ad758dbd3\"" response.status="200 OK" size=773262 url="http://localhost:36511/v2/library/busybox/blobs/sha256:50783e0dfb64b73019e973e7bce2c0d5a882301b781327ca153b876ad758dbd3?ns=docker.io"
    sandbox.go:237: time="2022-09-07T14:38:45Z" level=debug msg="diff applied" d=47.494367ms digest="sha256:50783e0dfb64b73019e973e7bce2c0d5a882301b781327ca153b876ad758dbd3" media=application/vnd.docker.image.rootfs.diff.tar.gzip size=773262
    sandbox.go:237: time="2022-09-07T14:38:45Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Unknown desc = lstat /tmp/bktest_buildkitd2939727001/tmp/buildkit-mount378098859/tmp/bktest_buildkitd2939727001/tmp/buildkit-mount378098859: no such file or directory"
    sandbox.go:237: lstat /tmp/bktest_buildkitd2939727001/tmp/buildkit-mount378098859/tmp/bktest_buildkitd2939727001/tmp/buildkit-mount378098859: no such file or directory
    sandbox.go:237: 6419 v0.10.0-491-g81a2997b buildkitd --oci-worker=true --containerd-worker=false --oci-worker-gc=false --oci-worker-labels=org.mobyproject.buildkit.worker.sandbox=true --config=/tmp/bktest_config3250988747/buildkitd.toml --root /tmp/bktest_buildkitd2939727001 --addr unix:///tmp/bktest_buildkitd2939727001/buildkitd.sock --debug
    sandbox.go:237: github.com/moby/buildkit/solver.(*edge).execOp
    sandbox.go:237:     /src/solver/edge.go:904
    sandbox.go:237: github.com/moby/buildkit/solver/internal/pipe.NewWithFunction.func2
    sandbox.go:237:     /src/solver/internal/pipe/pipe.go:82
    sandbox.go:237: runtime.goexit
    sandbox.go:237:     /usr/local/go/src/runtime/asm_amd64.s:1571
    sandbox.go:237: 
    sandbox.go:237: 6419 v0.10.0-491-g81a2997b buildkitd --oci-worker=true --containerd-worker=false --oci-worker-gc=false --oci-worker-labels=org.mobyproject.buildkit.worker.sandbox=true --config=/tmp/bktest_config3250988747/buildkitd.toml --root /tmp/bktest_buildkitd2939727001 --addr unix:///tmp/bktest_buildkitd2939727001/buildkitd.sock --debug
    sandbox.go:237: main.unaryInterceptor.func1
    sandbox.go:237:     /src/cmd/buildkitd/main.go:570
    sandbox.go:237: github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1
    sandbox.go:237:     /src/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:25
    sandbox.go:237: github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1
    sandbox.go:237:     /src/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:34
    sandbox.go:237: github.com/moby/buildkit/api/services/control._Control_Solve_Handler
    sandbox.go:237:     /src/api/services/control/control.pb.go:1718
    sandbox.go:237: google.golang.org/grpc.(*Server).processUnaryRPC
    sandbox.go:237:     /src/vendor/google.golang.org/grpc/server.go:1283
    sandbox.go:237: google.golang.org/grpc.(*Server).handleStream
    sandbox.go:237:     /src/vendor/google.golang.org/grpc/server.go:1620
    sandbox.go:237: google.golang.org/grpc.(*Server).serveStreams.func1.2
    sandbox.go:237:     /src/vendor/google.golang.org/grpc/server.go:922
    sandbox.go:237: runtime.goexit
    sandbox.go:237:     /usr/local/go/src/runtime/asm_amd64.s:1571
    sandbox.go:237: 
    sandbox.go:237: time="2022-09-07T14:38:45Z" level=debug msg="session finished: <nil>"
--- FAIL: TestIntegration (7.09s)
    --- FAIL: TestIntegration/TestParentsAddUrl/worker=oci/frontend=gateway (7.61s)
    --- PASS: TestIntegration/TestParentsAddArchive/worker=oci/frontend=gateway (9.14s)
    --- PASS: TestIntegration/TestParentsAddBasic/worker=oci/frontend=gateway (20.64s)
    --- PASS: TestIntegration/TestParentsCopyBasic/worker=oci/frontend=gateway (21.51s)
FAIL

jedevc · 2022-09-08T08:49:11Z

Looks like the test suite is running the right way, that's how I run individual tests. I also just tried a test using TestServer, TestHTTPDockerfile, which seems to run perfectly fine.

The issue looks like a file that the test attempts to access doesn't exist at that path - could you maybe push what you have, I'm happy to take a look.

DYefimov · 2022-09-09T10:33:33Z

Thanks.
Found it! Was a tough one, and totally on me.
The only thing left to do is to make URL behave properly (apart from the mess I made with git) and I'll rebase and push, hopefully in a day or two, finally. Sorry for it taking so long 🤦

thaJeztah · 2022-09-09T11:04:22Z

If you're getting stuck on the "mess you made with git", and need help / instructions how to resolve, let us know; happy to help

DYefimov · 2022-09-09T13:07:07Z

Gave a try to new git tools and it didn't go as expected... Funny enough, I'm using git oneflow ideology for the most part - it is "based on rebasing", and now we got here :)
@thaJeztah, thanks, an advice/instructions would be much appreciated.
So I made two mistakes: accidentally pulled on rebase instead of force pushing commits, and synced master to my feature branch.
To my knowledge, I should rewrite the history now like this:

pull the master
switch to the feature branch
do an interactive rebase on top of the master (git rebase -i master)
drop all the commits slipped from the master (not squash right? as they were pushed already to the feature branch?)
force push the feature branch

Of cause, my local state has no merge op in it.
And I should not drop duplicates of my own commits.

jedevc · 2022-09-12T08:30:38Z

I tend to not worry too much about the state of the git history while I'm working in "draft mode" - at least while the code's final shape is unclear, it's quite frustrating to try and have a neat history.

Once it's all done, I either:

git rebase -i master and squash down to a single commit if it's a simple enough change, amending it later to have a nice tidy message (this is what I'd do here, since it's a fairly self-contained change)
git reset master and then manually create new detailed commits with selectively staged lines if it's a more complex change.

The steps you've outlined look good to me, when rebasing you just to only pick the commits you've written as part of this PR, everything else will already be in master.

DYefimov · 2022-09-26T11:11:16Z

Thanks, followed your instructions and squashed/amended it.

Apart from documentation, there are several issues left with ADD gitref case.

Failed ADDing individual file from gitref, like so:
ADD https:/DYefimov/buildkit.git#master:frontend/dockerfile/dockerfile.go ./dst4/
Is it an intended behavior? It seems to be failing in here:

buildkit/source/git/gitsource.go

Line 564 in dabd467

names, err := d.Readdirnames(0)
Right now ADD gitref truncates SubDir by itself somewhere inside, and passes "/" path to the CopyOp.
It looks inconsistent with other cases, and it felt wrong adding --parents to the gitsource protobuf.
It might be beneficial not to truncate SubDir in git case, and patch it first, by leting CopyOp consistently decide whether to flatten or not the directory structure.
Which way would you prefer it to work? And it feels wrong too to mix --parents + "gitref consistency fix" in one PR, as it will somewhat ruin the revert-with-one-swipe.

frontend/dockerfile/instructions/parse_noparents.go

source/http/httpsource.go

jedevc · 2022-09-29T09:10:37Z

Have left a couple comments, when you're happy with the state of this, I think it's good to come out of draft 🎉

Yup, ADDing individual files isn't supported, as you've noted. It's only for sub-directories, see here. Maybe we should also support individual files at some point? I can't find an issue for it, but it definitely falls outside of the scope of this PR.
Ugh, sorry I hadn't realized that using a sub-directory for GitSource unpacks that sub-directory to the root of the filesystem. Good call, we should definitely make that change later, I'd be happy for now with an explicit error message of the form "--parents flag is not currently support for git source".

To change that, we'd need to make a change to the gateway API, probably by allowing an additional option to the git source that allows specifying what location to unpack into, similar to how we have llb.Filename for llb.HTTP. Then if --parents is specified, we could set that option appropriately, falling back to the error message if buildkit doesn't have the appropriate caps set.

jedevc · 2022-10-10T10:01:23Z

@DYefimov all good? Just wanted to checkup to see if there's anything else or other clarification you need?

DYefimov · 2022-10-13T12:05:18Z

Great! Integrated suggested changes, fixed some linting, and rebased just in case. Coming out of draft.

What about the documentation? Guess it should be placed right after this one: https:/moby/buildkit/blob/master/frontend/dockerfile/docs/reference.md#copy---link
Do you want me to write a skeleton for it?

Regarding this:

To change that, we'd need to make a change to the gateway API, probably by allowing an additional option to the git source that allows specifying what location to unpack into, similar to how we have llb.Filename for llb.HTTP. Then if --parents is specified, we could set that option appropriately, falling back to the error message if buildkit doesn't have the appropriate caps set.

Didn't exactly follow you on this one. Is it really that complicated of a change? Please, check the placeholder I left here:
frontend/dockerfile/dockerfile2llb/convert.go
From what I understood, that should be a minor non-breaking change - just teach GitSource not to truncate parent directories, and CopyOp will do the rest automatically.

ranma42 · 2023-10-10T09:07:00Z

I noticed that there are some merge conflicts; I'd love for this to get merged soonish, so if you need someone to resolve them, I am willing to try and help :)

thaJeztah · 2023-10-11T07:14:53Z

Looks like there's a compile issue in a test;

 > [lint-labs golangci-lint 1/1] RUN --mount=target=/go/src/github.com/moby/buildkit --mount=target=/root/.cache,type=cache,sharing=locked   GOARCH=amd64 golangci-lint run --build-tags "dfrunsecurity dfparents" &&   GOARCH=arm64 golangci-lint run --build-tags "dfrunsecurity dfparents" &&   touch /golangci-lint.done:
311.3 frontend/dockerfile/dockerfile.go:1: : # github.com/moby/buildkit/frontend/dockerfile [github.com/moby/buildkit/frontend/dockerfile.test]
311.3 frontend/dockerfile/dockerfile_parents_test.go:37:14: assignment mismatch: 2 variables but integration.Tmpdir returns 1 value (typecheck)
311.3 package dockerfile

Co-Authored-By: Dmitrii Efimov <[email protected]> Co-Authored-By: Justin Chadwell <[email protected]> Signed-off-by: Tonis Tiigi <[email protected]>

jedevc · 2023-11-01T12:02:19Z

Not quite sure why I didn't merge when I approved this a couple weeks ago - but let's get this in 🎉

levrik · 2024-01-18T13:36:36Z

@jedevc Can this be used on a released Buildkit version already (when setting the right # syntax= directive) or is also an unreleased Buildkit version be required?

thaJeztah · 2024-01-18T14:00:22Z

@levrik this should work with current versions; see docker/roadmap#107 (comment) for an example

levrik · 2024-01-18T14:24:27Z

@thaJeztah Awesome. Thanks!

thaJeztah · 2024-01-18T14:46:07Z

@levrik you're welcome! And if you find any issues with the feature, please open tickets 👍. I don't know if there's a timeline already for including it in the "stable" syntax, but getting feedback helps those decisions (knowing that everything works as intended)

levrik · 2024-01-19T09:28:58Z

@thaJeztah I just tried the feature and I have to say to everyone who has been involved: Thank you so much!
We have a JS monorepo and so far to split dependency installation and building into 2 stages, we had to either manually specify all package.json and lock files manually or use scripts to isolate these files in a separate folder before copying it into the build context.

Now it's just a COPY --parents pnpm-workspace.yaml **/package.json **/pnpm-lock.yaml ./ away and I get a super nice tree that I can just pnpm install.

See: moby/buildkit#3001

nwalters512 · 2024-01-30T00:25:37Z

This feature is working great for us! It will dramatically improve our ability to rely on layer caching in our Node monorepo. I hope this ends up in the stable syntax soon!

thaJeztah · 2024-01-30T10:28:46Z

Happy to hear the feature is working well 🥳

jmurzy · 2024-03-02T18:02:19Z

Awesome work! Thanks @DYefimov!

Are there any plans to add support for combining the --parents flag with --from? The following is a noop:

COPY --from=build --parents /packages/**/dist .

tonistiigi · 2024-03-02T19:04:37Z

@jmurzy wdym it is noop?

#syntax=docker/dockerfile-upstream:1.7.0-rc1-labs
from alpine AS build

from alpine
workdir /dest
copy --parents --from=build /usr/**/keys .
run ls -lR /dest/ && stop

....
0.139
0.139 /dest/usr/share/apk/keys/aarch64:
0.139 total 8
0.139 lrwxrwxrwx    1 root     root            54 Feb 10  2023 [email protected] -> ../[email protected]
0.139 lrwxrwxrwx    1 root     root            54 Feb 10  2023 [email protected] -> ../[email protected]
0.139
0.139 /dest/usr/share/apk/keys/armhf:
0.139 total 8
0.139 lrwxrwxrwx    1 root     root            54 Feb 10  2023 [email protected] -> ../[email protected]
0.139 lrwxrwxrwx    1 root     root            54 Feb 10  2023 [email protected] -> ../[email protected]
0.139

Looks correct to me.

jmurzy · 2024-03-03T01:45:25Z

Thanks, @tonistiigi. It was an oversight on my part, but it seems to be working.

The behavior was confusing because it copied files relative to the destination directory, even when the dst was specified as an absolute path and had a common path prefix with the src directory. I'm guessing this is the intended behavior.

For example, the following command copied the files into /usr/src/app/packages/usr/src/app/packages/**/dist.

COPY --parents --from=build /usr/src/app/packages/**/dist /usr/src/app/packages

I managed to fix it with the following command:

COPY --parents --from=build /usr/src/app/packages/**/dist /

Cheers 🍻

ashb · 2024-05-03T09:58:35Z

Found this feature, tested it, it does exactly what I wanted 👍

COPY --link --parents **/go.mod **/go.sum .

Can't wait for this to make it to stable.

ashb · 2024-07-17T10:49:26Z

I see that Dockerfile 1.8 and 1.9 have been released, but this isn't yet out of labs. What is the process for that? Anything I can do to help get this feature "graduated"?

jedevc previously requested changes Aug 4, 2022

View reviewed changes

frontend/dockerfile/instructions/parse.go Outdated Show resolved Hide resolved

solver/llbsolver/file/backend.go Outdated Show resolved Hide resolved

frontend/dockerfile/dockerfile2llb/convert.go Outdated Show resolved Hide resolved

jedevc requested a review from thaJeztah August 4, 2022 17:37

jedevc reviewed Aug 8, 2022

View reviewed changes

solver/llbsolver/file/backend.go Outdated Show resolved Hide resolved

thaJeztah mentioned this pull request Aug 16, 2022

Dockerfile COPY [dir1/* destdir] copies dir1/dir2/* moby/moby#29211

Open

DYefimov force-pushed the feature/copy_parents branch 2 times, most recently from ef278e3 to f92391b Compare September 26, 2022 10:53

DYefimov requested review from jedevc and removed request for thaJeztah September 28, 2022 12:30

jedevc reviewed Sep 29, 2022

View reviewed changes

frontend/dockerfile/instructions/parse_noparents.go Outdated Show resolved Hide resolved

source/http/httpsource.go Outdated Show resolved Hide resolved

DYefimov force-pushed the feature/copy_parents branch from f92391b to b103572 Compare October 13, 2022 11:43

DYefimov marked this pull request as ready for review October 13, 2022 12:05

tonistiigi force-pushed the feature/copy_parents branch from 2beb32a to ad1cf47 Compare October 11, 2023 04:37

tonistiigi approved these changes Oct 11, 2023

View reviewed changes

tonistiigi force-pushed the feature/copy_parents branch from ad1cf47 to 6970249 Compare October 11, 2023 07:41

dockerfile: add --parents flag for COPY

dce0519

Co-Authored-By: Dmitrii Efimov <[email protected]> Co-Authored-By: Justin Chadwell <[email protected]> Signed-off-by: Tonis Tiigi <[email protected]>

jedevc force-pushed the feature/copy_parents branch from 6970249 to dce0519 Compare October 11, 2023 08:53

jedevc approved these changes Oct 11, 2023

View reviewed changes

jedevc merged commit 9dd188b into moby:master Nov 1, 2023
53 of 56 checks passed

arthur-leclerc mentioned this pull request Nov 30, 2023

Dockerfile COPY with file globs will copy files from subdirectories to the destination directory moby/moby#15858

Closed

thaJeztah mentioned this pull request Jan 3, 2024

[Docker build] add all files of type docker/roadmap#107

Open

Titozzz added a commit to Titozzz/dockerfile-utils that referenced this pull request Jan 27, 2024

feat: add COPY --parents

31fd081

See: moby/buildkit#3001

Titozzz mentioned this pull request Jan 27, 2024

feat: add COPY --parents rcjsuen/dockerfile-utils#122

Closed

nwalters512 mentioned this pull request Jan 29, 2024

Use --parents flag for Dockerfile COPY PrairieLearn/PrairieLearn#9307

Merged

tonistiigi mentioned this pull request Mar 3, 2024

dockerfile: allow pivot point for --parents flag #4728

Merged

blowfishpro mentioned this pull request May 3, 2024

copy --parents silently ignores nonexistent paths #4900

Open

uckelman mentioned this pull request May 31, 2024

Add --parents option for COPY in Dockerfiles containers/buildah#5557

Open

Add --parents opt-in flag for ADD/COPY (labs) #3001

Add --parents opt-in flag for ADD/COPY (labs) #3001

Conversation

DYefimov commented Aug 4, 2022 • edited by thaJeztah Loading

jedevc left a comment • edited Loading

Choose a reason for hiding this comment

tonistiigi commented Aug 4, 2022

DYefimov commented Aug 6, 2022

DYefimov commented Aug 8, 2022

jedevc commented Aug 8, 2022

thaJeztah commented Aug 16, 2022

DYefimov commented Aug 16, 2022 • edited Loading

thaJeztah commented Aug 24, 2022

DYefimov commented Aug 24, 2022

jedevc commented Sep 7, 2022

DYefimov commented Sep 7, 2022

jedevc commented Sep 8, 2022

DYefimov commented Sep 9, 2022

thaJeztah commented Sep 9, 2022

DYefimov commented Sep 9, 2022 • edited Loading

jedevc commented Sep 12, 2022

DYefimov commented Sep 26, 2022 • edited Loading

jedevc commented Sep 29, 2022

jedevc commented Oct 10, 2022

DYefimov commented Oct 13, 2022

ranma42 commented Oct 10, 2023

thaJeztah commented Oct 11, 2023

jedevc commented Nov 1, 2023

levrik commented Jan 18, 2024 • edited Loading

thaJeztah commented Jan 18, 2024

levrik commented Jan 18, 2024

thaJeztah commented Jan 18, 2024

levrik commented Jan 19, 2024 • edited Loading

nwalters512 commented Jan 30, 2024

thaJeztah commented Jan 30, 2024

jmurzy commented Mar 2, 2024

tonistiigi commented Mar 2, 2024

jmurzy commented Mar 3, 2024

ashb commented May 3, 2024

ashb commented Jul 17, 2024

DYefimov commented Aug 4, 2022 •

edited by thaJeztah

Loading

jedevc left a comment •

edited

Loading

DYefimov commented Aug 16, 2022 •

edited

Loading

DYefimov commented Sep 9, 2022 •

edited

Loading

DYefimov commented Sep 26, 2022 •

edited

Loading

levrik commented Jan 18, 2024 •

edited

Loading

levrik commented Jan 19, 2024 •

edited

Loading