Repository navigation
Do not require the optional fields of NMEA sentences without checksums - #924
Merged
caglarpir merged 2 commits intoOct 26, 2026
Merged
Conversation
A logger leaves NUL bytes in its log after a power cut, such as on an SD card. mapillary#915 took one NUL byte in the first 64 KiB of a file for the sign of a binary file, and failed the whole log as "not a valid NMEA file: it is not text", where earlier versions read it. With 4 KiB of NUL bytes appended to the NMEA log of an image sequence, all 113 images failed, where the log without them geotags 106, and the other 7 are duplicates. A longer log with the same tail was still read, only because its first 64 KiB hold no NUL byte. Take a file for one that is no NMEA log at all only if its first 64 KiB hold no RMC or GGA sentence, and NUL bytes or mostly bytes that do not decode as UTF-8. A video given as the NMEA file still fails at once: on 1.2 and 4.0 GB videos in 0.7 to 1.0 s and 61 MB, as before; no video or image among 18,023 sample files holds an RMC or GGA sentence in its first 64 KiB. NUL bytes now end a line, as no sentence holds one: a sentence that a logger wrote on after them is read, and one with NUL bytes inside it is skipped under the warning of malformed sentences. A UTF-16 log, which some Windows tools write, is still not read, but the error now says it is UTF-16 text, and to save it as UTF-8 or ASCII. The NUL-padded log of the image sequence now gives the same 106 geotagged images and 7 duplicates as the log without the NUL bytes, with identical descriptions. So does a video log with the same tail. The NMEA logs without NUL bytes, process of images, of GoPro MAX and BlackVue videos, video_process by distance and with an NMEA log, give the same results as before.
mapillary#913 took a sentence without its checksum for cut short when it had fewer fields than a complete sentence: 11 for RMC and 14 for GGA. The fields after the date of an RMC sentence (the magnetic variation, the mode and the navigational status) and after the altitude of a GGA sentence (the height of the geoid and the DGPS fields) are optional, and receivers and converters leave them off. A log without checksums whose sentences end there failed whole, where earlier versions read it: a 650-fix log written with 9 RMC and 12 GGA fields, 10 and 12, 9 and 14, or 11 and 10, gave "1300 of its 1300 RMC and GGA sentences are malformed" or "650 of its 1300", and no track. Require only the fields that a fix is read from: those of an RMC sentence through the date, 9, and those of a GGA sentence through the altitude and its unit, 10. A sentence cut inside the date or the altitude still has fewer, or a date that is not 6 digits, and is still skipped under the warning of malformed sentences, with the reason "cut short: 9 of the 10 fields of a GGA sentence that a fix is read from". Each of those four logs now gives the track of the original, 650 points over 649 s, with no warning. Written GGA first across midnight and ending with an RMC sentence cut inside its date, the log still gives the 650 points, 0 to 649000 ms, with one warning. With a checksum on one line only, with checksums on its GGA sentences only, and as two halves with and without checksums, it reads in full as before. A log whose sentences are all cut short still fails with the reason. The NMEA logs of a Sony action camera and of an image sequence, process of the images and the video, and video_process by distance and by interval with the NMEA log write the same descriptions as before.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #923; review the last commit.
#913 took a sentence without its checksum for cut short when it had
fewer fields than a complete sentence: 11 for RMC and 14 for GGA. The
fields after the date of an RMC sentence (the magnetic variation, the
mode and the navigational status) and after the altitude of a GGA
sentence (the height of the geoid and the DGPS fields) are optional,
and receivers and converters leave them off. A log without checksums
whose sentences end there failed whole, where earlier versions read it:
a 650-fix log written with 9 RMC and 12 GGA fields, 10 and 12, 9 and
14, or 11 and 10, gave "1300 of its 1300 RMC and GGA sentences are
malformed" or "650 of its 1300", and no track.
Require only the fields that a fix is read from: those of an RMC
sentence through the date, 9, and those of a GGA sentence through the
altitude and its unit, 10. A sentence cut inside the date or the
altitude still has fewer, or a date that is not 6 digits, and is still
skipped under the warning of malformed sentences, with the reason
"cut short: 9 of the 10 fields of a GGA sentence that a fix is read
from".
Each of those four logs now gives the track of the original, 650
points over 649 s, with no warning. Written GGA first across midnight
and ending with an RMC sentence cut inside its date, the log still
gives the 650 points, 0 to 649000 ms, with one warning. With a checksum
on one line only, with checksums on its GGA sentences only, and as two
halves with and without checksums, it reads in full as before. A log
whose sentences are all cut short still fails with the reason. The NMEA
logs of a Sony action camera and of an image sequence, process of the
images and the video, and video_process by distance and by interval
with the NMEA log write the same descriptions as before.
Unit and integration tests, doctests,
mypy --python-version 3.9,ruff check,ruff format --checkandusort diffare clean.