Remove byte-order mark from JSON stream and first CSV chunk #53
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
When JSON input is a readable stream, a byte-order mark may be present at the beginning of the stream. When reading the first chunk of the stream, check for the presence of the byte-order mark. Remove it if it is present.
When parsing a CSV from a source that includes a byte-order mark (BOM), the BOM is present at the time the parser attempts to determine if the first column name is quoted. When the BOM is present, the parser does not recognize that the column is quoted, resulting in a failure to match an expected column name. Remove the BOM (if present) from the first chunk so that a quoted column name will be recognized and parsed as a quoted value.
mint-thompson
changed the title
shaselton-usds
approved these changes
Aug 16, 2024
eoverly
approved these changes
Aug 19, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem
When validating JSON with a readable stream, the presence of a byte-order mark causes a parser error.
When validating CSV, an initial BOM will be part of the first column name. If the first column name is also quoted, this means that the value will not be recognized as quoted. This causes the first column to not match any of the expected column names.
Solution
When reading the first chunk of a JSON stream, check for the presence of a byte-order mark. If it is present, remove it.
When reading the first chunk of CSV, check for the presence of a byte-order mark. If it is present, remove it.
Test Plan
A test case that uses a JSON file encoded as utf-8 with BOM is added to test this change.
A test case that uses CSV file encoded as utf-8 with BOM is added to test this change. The first column name in the file is quoted.
EDIT: Updated 2024/08/16 with information about the CSV changes (984cb6d)