Repository navigation
Read an NMEA log padded with NUL bytes - #923
Merged
caglarpir merged 1 commit intoOct 26, 2026
Merged
Conversation
A logger leaves NUL bytes in its log after a power cut, such as on an SD card. mapillary#915 took one NUL byte in the first 64 KiB of a file for the sign of a binary file, and failed the whole log as "not a valid NMEA file: it is not text", where earlier versions read it. With 4 KiB of NUL bytes appended to the NMEA log of an image sequence, all 113 images failed, where the log without them geotags 106, and the other 7 are duplicates. A longer log with the same tail was still read, only because its first 64 KiB hold no NUL byte. Take a file for one that is no NMEA log at all only if its first 64 KiB hold no RMC or GGA sentence, and NUL bytes or mostly bytes that do not decode as UTF-8. A video given as the NMEA file still fails at once: on 1.2 and 4.0 GB videos in 0.7 to 1.0 s and 61 MB, as before; no video or image among 18,023 sample files holds an RMC or GGA sentence in its first 64 KiB. NUL bytes now end a line, as no sentence holds one: a sentence that a logger wrote on after them is read, and one with NUL bytes inside it is skipped under the warning of malformed sentences. A UTF-16 log, which some Windows tools write, is still not read, but the error now says it is UTF-16 text, and to save it as UTF-8 or ASCII. The NUL-padded log of the image sequence now gives the same 106 geotagged images and 7 duplicates as the log without the NUL bytes, with identical descriptions. So does a video log with the same tail. The NMEA logs without NUL bytes, process of images, of GoPro MAX and BlackVue videos, video_process by distance and with an NMEA log, give the same results as before.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A logger leaves NUL bytes in its log after a power cut, such as on an
SD card. #915 took one NUL byte in the first 64 KiB of a file for the
sign of a binary file, and failed the whole log as "not a valid NMEA
file: it is not text", where earlier versions read it. With 4 KiB of NUL
bytes appended to the NMEA log of an image sequence, all 113 images
failed, where the log without them geotags 106, and the other 7 are
duplicates. A longer log with the same tail was still read, only
because its first 64 KiB hold no NUL byte.
Take a file for one that is no NMEA log at all only if its first 64 KiB
hold no RMC or GGA sentence, and NUL bytes or mostly bytes that do not
decode as UTF-8. A video given as the NMEA file still fails at once: on
1.2 and 4.0 GB videos in 0.7 to 1.0 s and 61 MB, as before; no video or
image among 18,023 sample files holds an RMC or GGA sentence in its
first 64 KiB. NUL bytes now end a line, as no sentence holds one: a
sentence that a logger wrote on after them is read, and one with NUL
bytes inside it is skipped under the warning of malformed sentences.
A UTF-16 log, which some Windows tools write, is still not read, but
the error now says it is UTF-16 text, and to save it as UTF-8 or ASCII.
The NUL-padded log of the image sequence now gives the same 106
geotagged images and 7 duplicates as the log without the NUL bytes,
with identical descriptions. So does a video log with the same tail.
The NMEA logs without NUL bytes, process of images, of GoPro MAX and
BlackVue videos, video_process by distance and with an NMEA log, give
the same results as before.
Unit and integration tests, doctests,
mypy --python-version 3.9,ruff check,ruff format --checkandusort diffare clean.