Better handling of non-ascii filepaths #434
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #394, closes #402, closes #403 (previous explorations)
There's a bit more going on here than we originally thought and the solution isn't quite as simple as "UTF-8 everywhere" (but it's close).
First, the file from the original issue (tidyverse/readr#1345) has a path containing non-ascii characters, but what's easier to miss is that the file also lacks a trailing newline. The failure cascade looks like this:
has_trailing_newline()
implicitly expects that the path is in the native encoding and, if it's not, can potentially fail to find an existing file. When that happens, it unconditionally reportsTRUE
(yes, there is a trailing new line), which is wrong in this case. A file that lacks a trailing newline should be routed through the connection logic, but, in this case is read as a "regular file", which fails and results in an empty tibble.So yes the problem is the file path encoding, but it first raises its head via
has_trailing_newline()
, not the main file-reading step.Second, by exploring various solutions, I discovered that blindly applying
enc2utf8()
is too simple, since vroom tests itself on linux in a non-UTF-8 locale. Hence we doenc2utf8()
on Windows andenc2native()
otherwise. This effort did reveal a pre-existing file path problem on this build that I ultimately decided to not fix, in the "file ends with a newline" case. I think this OS-dependent approach may be currently irrelevant in vroom, because of the mio problem I show below, but might be worth it in other settings.For normal files, on linux, in a non-UTF-8 locale, with a multi-byte file path, we get an error from mio itself, i.e. the memory mapping fails:
https:/tidyverse/vroom/runs/6315284023?check_suite_focus=true#step:7:182
I am content to
skip()
here and re-consider if an actual user ever needs to do this. There is also evidence that vroom's writing functions are not entirely prepared for this scenario, but the same posture applies.