Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Better handling of non-ascii filepaths #434

Merged
merged 11 commits into from
May 6, 2022
Merged

Better handling of non-ascii filepaths #434

merged 11 commits into from
May 6, 2022

Conversation

jennybc
Copy link
Member

@jennybc jennybc commented May 5, 2022

Closes #394, closes #402, closes #403 (previous explorations)

There's a bit more going on here than we originally thought and the solution isn't quite as simple as "UTF-8 everywhere" (but it's close).

First, the file from the original issue (tidyverse/readr#1345) has a path containing non-ascii characters, but what's easier to miss is that the file also lacks a trailing newline. The failure cascade looks like this:

has_trailing_newline() implicitly expects that the path is in the native encoding and, if it's not, can potentially fail to find an existing file. When that happens, it unconditionally reports TRUE (yes, there is a trailing new line), which is wrong in this case. A file that lacks a trailing newline should be routed through the connection logic, but, in this case is read as a "regular file", which fails and results in an empty tibble.

So yes the problem is the file path encoding, but it first raises its head via has_trailing_newline(), not the main file-reading step.

Second, by exploring various solutions, I discovered that blindly applying enc2utf8() is too simple, since vroom tests itself on linux in a non-UTF-8 locale. Hence we do enc2utf8() on Windows and enc2native() otherwise. This effort did reveal a pre-existing file path problem on this build that I ultimately decided to not fix, in the "file ends with a newline" case. I think this OS-dependent approach may be currently irrelevant in vroom, because of the mio problem I show below, but might be worth it in other settings.

For normal files, on linux, in a non-UTF-8 locale, with a multi-byte file path, we get an error from mio itself, i.e. the memory mapping fails:

https:/tidyverse/vroom/runs/6315284023?check_suite_focus=true#step:7:182

> test_check("vroom")
mapping error: No such file or directory
[ FAIL 2 | WARN 0 | SKIP 2 | PASS 1065 ]

I am content to skip() here and re-consider if an actual user ever needs to do this. There is also evidence that vroom's writing functions are not entirely prepared for this scenario, but the same posture applies.

@jennybc jennybc force-pushed the iss394 branch 4 times, most recently from fbc524b to b3d8a0b Compare May 5, 2022 20:27
The encoding improvements in R 4.2 on Windows mean that I really need to include 4.1 to make sure I can see the problem and that I have fixed it.

Also make the GHA config more obvious in other ways.
Same motivations as r-lib/tzdb#13 (comment)

It's too simplistic to always apply `enc2utf8()`, given that we have an existing ambition to work in a non-UTF-8 locale on unix, as evidenced by the GHA config.
Copy link
Member

@DavisVaughan DavisVaughan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Such a hard fought battle

@jennybc jennybc marked this pull request as ready for review May 6, 2022 06:02
@jennybc jennybc merged commit f124fcc into main May 6, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

File path encoding problem on Windows
2 participants