Perl deletes files when using in-place edit

[bbarker]

Please use the following bug reporting template to help produce actionable and reproducible issues:

• A brief description

The following kind of one-liner can be used to do a recursive find and replace:

find ./ -name '*.scala' -print0 | xargs -0 perl -pi -e "s/com\.kolor/edu\.cornell\.cac/g"

However, this currently deletes all found files in BashOnWindows.

• Expected results
  From a Linux system:

brandon@euca-128-84-11-38:~/reactive-docker$ find ./ -name '*.scala' -print0 | xargs -0 perl -pi -e "s/com\.kolor/edu\.cornell\.cac/g"
brandon@euca-128-84-11-38:~/reactive-docker$ find ./ -name '*.scala' -print0
./src/test/scala/test/DockerApiSpec.scala./src/test/scala/test/DockerBuildSpec.scala./src/test/scala/test/DockerUsecaseSpec.scala./src/test/scala/test/TempFile.scala./src/test/scala/test/package.scala./src/test/scala/test/DockerQuickSpec.scala./src/test/scala/test/DockerAttachSpec.scala./src/test/scala/test/DockerEntitySpec.scala./src/main/scala/com/kolor/docker/dsl/Dockerfile.scala./src/main/scala/com/kolor/docker/dsl/package.scala./src/main/scala/com/kolor/docker/api/Endpoints.scala./src/main/scala/com/kolor/docker/api/Exception.scala./src/main/scala/com/kolor/docker/api/DockerApi.scala./src/main/scala/com/kolor/docker/api/json/Formats.scala./src/main/scala/com/kolor/docker/api/json/FormatsV112.scala./src/main/scala/com/kolor/docker/api/package.scala./src/main/scala/com/kolor/docker/api/Iteratee.scala./src/main/scala/com/kolor/docker/api/entities/DockerVolume.scala./src/main/scala/com/kolor/docker/api/entities/DockerAuth.scala./src/main/scala/com/kolor/docker/api/entities/ImageId.scala./src/main/scala/com/kolor/docker/api/entities/ConfigReader.scala./src/main/scala/com/kolor/docker/api/entities/package.scala./src/main/scala/com/kolor/docker/api/entities/DockerEntity.scala./src/main/scala/com/kolor/docker/api/entities/RepositoryTag.scala./src/main/scala/com/kolor/docker/api/entities/ContainerConfiguration.scala./src/main/scala/com/kolor/docker/api/entities/ContainerId.scala./src/main/scala/com/kolor/docker/api/Docker.scala./src/main/scala/play/custom/iteratees/Json.scala./src/main/scala/play/custom/iteratees/Combinators.scalabrandon@euca-128-84-11-38:~/reactive-docker$

• Actual results (with terminal output if applicable)

brandon@BEBSURFACEPRO3:~/workspace/reactive-docker$ git stash
Saved working directory and index state WIP on master: cff19e0 update libraries
HEAD is now at cff19e0 update libraries
brandon@BEBSURFACEPRO3:~/workspace/reactive-docker$ find ./ -name '*.scala' -print0
./src/main/scala/com/kolor/docker/api/Docker.scala ./src/main/scala/com/kolor/docker/api/DockerApi.scala ./src/main/scala/com/kolor/docker/api/Endpoints.scala ./src/main/scala/com/kolor/docker/api/entities/ConfigReader.scala ./src/main/scala/com/kolor/docker/api/entities/ContainerConfiguration.scala ./src/main/scala/com/kolor/docker/api/entities/ContainerId.scala ./src/main/scala/com/kolor/docker/api/entities/DockerAuth.scala ./src/main/scala/com/kolor/docker/api/entities/DockerEntity.scala ./src/main/scala/com/kolor/docker/api/entities/DockerVolume.scala ./src/main/scala/com/kolor/docker/api/entities/ImageId.scala ./src/main/scala/com/kolor/docker/api/entities/package.scala ./src/main/scala/com/kolor/docker/api/entities/RepositoryTag.scala ./src/main/scala/com/kolor/docker/api/Exception.scala ./src/main/scala/com/kolor/docker/api/Iteratee.scala ./src/main/scala/com/kolor/docker/api/json/Formats.scala ./src/main/scala/com/kolor/docker/api/json/FormatsV112.scala ./src/main/scala/com/kolor/docker/api/package.scala ./src/main/scala/com/kolor/docker/dsl/Dockerfile.scala ./src/main/scala/com/kolor/docker/dsl/package.scala ./src/main/scala/play/custom/iteratees/Combinators.scala ./src/main/scala/play/custom/iteratees/Json.scala ./src/test/scala/test/DockerApiSpec.scala ./src/test/scala/test/DockerAttachSpec.scala ./src/test/scala/test/DockerBuildSpec.scala ./src/test/scala/test/DockerEntitySpec.scala ./src/test/scala/test/DockerQuickSpec.scala ./src/test/scala/test/DockerUsecaseSpec.scala ./src/test/scala/test/package.scala ./src/test/scala/test/TempFile.scala brandon@BEBSURFACEPRO3:~/workspace/reactive-docker$ find ./ -name '*.scala' -print0 | xargs -0 perl -pi -e "s/com\.kolor/edu\.cornell\.cac/g"
Can't do inplace edit on ./src/main/scala/com/kolor/docker/api/Docker.scala: No such file or directory.
Can't do inplace edit on ./src/main/scala/com/kolor/docker/api/DockerApi.scala: No such file or directory.
Can't do inplace edit on ./src/main/scala/com/kolor/docker/api/Endpoints.scala: No such file or directory.
Can't do inplace edit on ./src/main/scala/com/kolor/docker/api/entities/ConfigReader.scala: No such file or directory.
Can't do inplace edit on ./src/main/scala/com/kolor/docker/api/entities/ContainerConfiguration.scala: No such file or directory.
Can't do inplace edit on ./src/main/scala/com/kolor/docker/api/entities/ContainerId.scala: No such file or directory.
Can't do inplace edit on ./src/main/scala/com/kolor/docker/api/entities/DockerAuth.scala: No such file or directory.
Can't do inplace edit on ./src/main/scala/com/kolor/docker/api/entities/DockerEntity.scala: No such file or directory.
Can't do inplace edit on ./src/main/scala/com/kolor/docker/api/entities/DockerVolume.scala: No such file or directory.
Can't do inplace edit on ./src/main/scala/com/kolor/docker/api/entities/ImageId.scala: No such file or directory.
Can't do inplace edit on ./src/main/scala/com/kolor/docker/api/entities/package.scala: No such file or directory.
Can't do inplace edit on ./src/main/scala/com/kolor/docker/api/entities/RepositoryTag.scala: No such file or directory.
Can't do inplace edit on ./src/main/scala/com/kolor/docker/api/Exception.scala: No such file or directory.
Can't do inplace edit on ./src/main/scala/com/kolor/docker/api/Iteratee.scala: No such file or directory.
Can't do inplace edit on ./src/main/scala/com/kolor/docker/api/json/Formats.scala: No such file or directory.
Can't do inplace edit on ./src/main/scala/com/kolor/docker/api/json/FormatsV112.scala: No such file or directory.
Can't do inplace edit on ./src/main/scala/com/kolor/docker/api/package.scala: No such file or directory.
Can't do inplace edit on ./src/main/scala/com/kolor/docker/dsl/Dockerfile.scala: No such file or directory.
Can't do inplace edit on ./src/main/scala/com/kolor/docker/dsl/package.scala: No such file or directory.
Can't do inplace edit on ./src/main/scala/play/custom/iteratees/Combinators.scala: No such file or directory.
Can't do inplace edit on ./src/main/scala/play/custom/iteratees/Json.scala: No such file or directory.
Can't do inplace edit on ./src/test/scala/test/DockerApiSpec.scala: No such file or directory.
Can't do inplace edit on ./src/test/scala/test/DockerAttachSpec.scala: No such file or directory.
Can't do inplace edit on ./src/test/scala/test/DockerBuildSpec.scala: No such file or directory.
Can't do inplace edit on ./src/test/scala/test/DockerEntitySpec.scala: No such file or directory.
Can't do inplace edit on ./src/test/scala/test/DockerQuickSpec.scala: No such file or directory.
Can't do inplace edit on ./src/test/scala/test/DockerUsecaseSpec.scala: No such file or directory.
Can't do inplace edit on ./src/test/scala/test/package.scala: No such file or directory.
Can't do inplace edit on ./src/test/scala/test/TempFile.scala: No such file or directory.
brandon@BEBSURFACEPRO3:~/workspace/reactive-docker$ find ./ -name '*.scala' -print0

Note the last line confirms deletion.

• Your Windows build number
  14393.51
• Steps / commands required to reproduce the error

See above commands.

• Strace of the failing command
• Required packages and commands to install

perl

See our contributing instructions for assistance.

[bbarker]

Alternative command that also deletes (showing it isn't the fault of xargs):

perl -pi -e 's/com\.kolor/edu\.cornell\.cac/g' `find ./ -name *.scala`

[bbarker]

The mystery continues ... I cloned the original repo and cannot reproduced the problem today, and could not reproduce it with your minimal example either, very strange.

[onomatopellan]

It looks like a duplicate of #904

What it probably happened is that the git directory was manipulated by Windows Git or manually through the lxss folder. Changing the contents of that folder from Windows side can corrupt files and even make them disappear. Use always linux tools to manipulate files from folders like / and /home.

[bbarker]

I can confirm I had been using Windows Git on the repo ... but, it is only under lxss via symbolic link:

0 lrwxrwxrwx 1 brandon brandon 35 Aug 10 18:07 workspace -> /mnt/c/Users/brand/Documents/GitHub
brandon@BEBSURFACEPRO3:~$ cd -
/home/brandon/workspace/reactive-docker
brandon@BEBSURFACEPRO3:~/workspace/reactive-docker$ pwd
/home/brandon/workspace/reactive-docker

[mousai]

I can reproduce this consistently - it seems like it only fails for files in DrvFs:

dbenito@cube:~$ echo "blah" > test.txt
dbenito@cube:~$ cat test.txt
blah
dbenito@cube:~$ perl -pi -e 's/blah/bleh/g' test.txt
dbenito@cube:~$ cat test.txt
bleh
dbenito@cube:~$ cd /mnt/c/tmp/
dbenito@cube:/mnt/c/tmp$ echo "blah" > test.txt
dbenito@cube:/mnt/c/tmp$ cat test.txt
blah
dbenito@cube:/mnt/c/tmp$ perl -pi -e 's/blah/bleh/g' test.txt
Can't do inplace edit on ./poo.txt: No such file or directory.
dbenito@cube:/mnt/c/tmp$ cat test.txt
cat: test.txt: No such file or directory

[laktak]

This may be a similar issue that's happening in #1615

[SvenGroot]

This is a known limitation in our unlink support for DrvFs. In LxFs, we emulate unlink by renaming the file to a temporary directory, but in DrvFs we cannot do that so we use normal Windows delete semantics.

Perl attempts to open the file, unlink it, recreate it, then copy data from the old file to the new. On DrvFs, after step 2 the file is not unlinked, but merely marked for deletion because there is still an open handle to it (from Perl itself). Recreating the file then fails because opening a file marked for deletion is not possible. When Perl closes the first file descriptor, the file is actually deleted.

We realize this is a very unfortunate limitation, and we're working on ways to improve our file system support to fix this issue in a future insider build.

[therealkenc]

Maybe the same as #1357 (message) then, too? If so might as well close and dup these all down to one.

[benhillis]

@therealkenc - good call, I'll dupe that one out as well.

[therealkenc]

In LxFs, we emulate unlink by renaming the file to a temporary directory

@SvenGroot while you are in there looking at "filesystem stuff", please see if you can squash #1529. It looks like the Meteor people are hitting it as well. The core issue here isn't about "unlink"; it's about NT semantics on open handles. You already know that of course. Just didn't want #1529 lost in the noise.

[therealkenc]

Fixed issue where deleting DrvFs files with open handles will cause the file to exhibit undefined behavior [GH 544,966,1357,1535,1615]

Very nice. So... how'd ya fix it? Or, put another way, what's the catch?

[JasonLinMS]

@therealkenc We changed NTFS-based DrvFs volumes to use "POSIX unlink" instead of the "regular Windows delete". (FileDispositionInformationEx instead of FileDispositionInformation)

[therealkenc]

Oh that's totally cheating. That wasn't even there before. 😊

#define FILE_DISPOSITION_DO_NOT_DELETE 0x00000000
#define FILE_DISPOSITION_DELETE 0x00000001
#define FILE_DISPOSITION_POSIX_SEMANTICS 0x00000002
#define FILE_DISPOSITION_FORCE_IMAGE_SECTION_CHECK 0x00000004
#define FILE_DISPOSITION_ON_CLOSE 0x00000008

[kayakyakr]

Finally got to test this on 16193 and believe it to be working. I've been able to run all dev work via my c drive. Thanks!

[KaelWD]

I'm still seeing this in 10.0.16299.15, having webpack watch a directory and then deleting a file from windows explorer causes it to stick around until webpack is killed.

Edit: Updated to 16299.19 and it's still happening, see this comment for more: #1956 (comment)

[Paul-Pushkarov]

Paul-Pushkarov unassigned JasonLinMS

Hm, what just happened? I just left a comment...

[therealkenc]

This one was left in an undead state and has been long addressed circa spring 2017. Pronoun "this" here being the repro given that doesn't presuppose some existing directory structure (as with the OP), which was (paraphrasing):

$ pushd /mnt/c>/dev/null; wintemp=$(cmd.exe /c echo %TEMP% | tr -d '\r'); popd>/dev/null
$ wsltemp=$(wslpath $wintemp)
$ cd $wsltemp
$ echo "blah" > test.txt
$ perl -pi -e 's/blah/bleh/g' test.txt
$ cat test.txt
bleh

To the extent there exists other gaps in the 9p filesystem, they are not this.

[msoutopico]

I can reproduce it in WSL (ubuntu) OS Version 10.0.18363 N/A Build 18363
uname -a
4.4.0-18362-Microsoft #836-Microsoft Mon May 05 16:04:00 PST 2020 x86_64 x86_64 x86_64 GNU/Linux