-
Notifications
You must be signed in to change notification settings - Fork 67
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Inconsistent errors when reading NDJSON with bad last line #4537
Comments
The fixes in #5055 have significantly improved the errors shown here. Repeating the original repro steps with Zed commit 38763f8, we now see:
@mattnibs explains in #5055 (comment) why we saw the improvement here for
Indeed this is the case.
The improvements are similar for
Since auto-detect is likely to be where most users start from, I'll hold this issue open in hopes we can one day do something about the part of this that @mattnibs attributes to the zsonio reader. |
Repro is with Zed commit d599839.
The attached NDJSON test data files lines-9.ndjson.gz and lines-10.ndjson.gz both consist of several lines of valid NDJSON and a closing incomplete line:
They otherwise only differ in that
lines-10
contains one more valid NDJSON record thanlines-9
before that bad last line.Reading both with
zq
, the reported errors differ.The difference becomes a little more significant when loading to a pool, since no error at all is reported for
line-10
.The text was updated successfully, but these errors were encountered: