Repository navigation
Judge an NMEA sentence cut short by its own fields, not by checksums - #913
Merged
caglarpir merged 2 commits intoOct 24, 2026
Merged
Conversation
Since mapillary#898, interval sampling with --geotag_source gpx reads the GPX again for each video, to tell which samples fall outside its track. Sampling 11 BlackVue videos every 30 s against a 163 MB GPX of 498 tracks, video_process took 293 s instead of 161 s before mapillary#898, and printed "Found 498 tracks in the GPX file" 12 times instead of once. The other readers parsed such a file once per video too: distance sampling, in a pool of processes started for each video, and the geotagging stage of videos, in each process of its pool. An NMEA file is read once per process since mapillary#909. Parse a GPX file once per process, and share its tracks among the readers: interval and distance sampling, GPX sync of --geotag_source and --video_geotag_source, and the geotagging stage of images. The tracks are kept by the path and content of the file, so a file that changes is parsed again, and only those of the last four files parsed are kept, in the main process only. The geotagging stage of videos reads a GPX that geotags several videos once, before its pool, as it reads an NMEA file; the GPX of each video is still parsed in the pool. Distance sampling reads its video in its own process, as interval sampling does. GPX sync moves copies of the points. The warnings about the tracks of a file, and about its points before 1980, are given once per file. That video_process run now takes 82 s and prints the warning once. Its 22 samples and their descriptions are the same, and its peak memory is 2.68 GB (2.84 GB on main). On one of the videos it takes 15.9 s instead of 25.3 s, and process 10.1 s with 2.69 GB, as on main. Sampling the 11 videos by distance with the GPX as --video_geotag_source takes 25 s instead of 107 s. process on them, with the GPX as --geotag_source or --video_geotag_source, takes 22 s instead of 26 to 29 s, and its processes peak at 6.4 GB together instead of 14 to 16 GB, as each of them parsed the file. Compared with main on process, video_process and sample_video, with GPX and NMEA sources and with the default sources, on BlackVue, GoPro MAX, Sony and 70mai videos and on images, the descriptions and samples are the same, and only the repeated warnings are gone. mapillary#898 said that with the legacy sources the frames before a late GPS start were kept and geotagged as before: they are kept, and fail as before.
mapillary#901 took every sentence without a checksum for cut short as soon as any other sentence of the file had one. Checksums are optional in NMEA 0183, and logs that mix sentences with and without them read in full before: a log with a checksum on one line only, or on its GGA sentences only, now failed whole as "not a valid NMEA file", and of two logs concatenated, one with checksums and one without, the half without was skipped. A sentence without its checksum is now cut short only when its own fields show it: when it has fewer fields than a complete sentence (11 for RMC, as NMEA 2.0 to 2.2 write it, 12 with the mode of NMEA 2.3 and 13 with the navigational status of NMEA 4.1; 14 for GGA), when the date of an RMC sentence is not 6 digits, or when its time does not parse. A sentence with its checksum ends where the checksum shows, however few fields it has, and one whose checksum does not match is still malformed. The checksums of other sentences no longer count. When no fix of a file can be read, the error now says why, with the first malformed sentence, its line and the reason: "track.nmea is not a valid NMEA file: 2 of its 2 RMC and GGA sentences are malformed, the first on line 1 (checksum does not match: 00 != 47): $GPRMC,...". It said only "not a valid NMEA file". (mapillary#901's message said that pynmea2 reads a time cut to "00052" as 00:00:02; it reads it as 00:05:02.) Checked on a 650-fix log, with the video it belongs to: with a checksum on one line only, with checksums on its GGA sentences only, and as two halves with and without checksums, it gives the track of the original, 650 points over 649 s, with no warning, where it failed twice and gave 325 points. Written GGA first across midnight and ending with an RMC sentence cut inside its date, it still gives the 650 points, and one warning. A log whose sentences are all cut short, which gave a stationary video, fails with the reason. The NMEA logs of a Sony action camera and of an image sequence read the same points as before, and process of the images and the video, video_process by distance and by interval with the NMEA log, and plain process and video_process of other cameras write the same descriptions.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #912; review the last commit.
#901 took every sentence without a checksum for cut short as soon as
any other sentence of the file had one. Checksums are optional in NMEA
0183, and logs that mix sentences with and without them read in full
before: a log with a checksum on one line only, or on its GGA sentences
only, now failed whole as "not a valid NMEA file", and of two logs
concatenated, one with checksums and one without, the half without was
skipped.
A sentence without its checksum is now cut short only when its own
fields show it: when it has fewer fields than a complete sentence (11
for RMC, as NMEA 2.0 to 2.2 write it, 12 with the mode of NMEA 2.3 and
13 with the navigational status of NMEA 4.1; 14 for GGA), when the date
of an RMC sentence is not 6 digits, or when its time does not parse. A
sentence with its checksum ends where the checksum shows, however few
fields it has, and one whose checksum does not match is still
malformed. The checksums of other sentences no longer count.
When no fix of a file can be read, the error now says why, with the
first malformed sentence, its line and the reason: "track.nmea is not a
valid NMEA file: 2 of its 2 RMC and GGA sentences are malformed, the
first on line 1 (checksum does not match: 00 != 47): $GPRMC,...". It
said only "not a valid NMEA file". (#901's message said that pynmea2
reads a time cut to "00052" as 00:00:02; it reads it as 00:05:02.)
Checked on a 650-fix log, with the video it belongs to: with a checksum
on one line only, with checksums on its GGA sentences only, and as two
halves with and without checksums, it gives the track of the original,
650 points over 649 s, with no warning, where it failed twice and gave
325 points. Written GGA first across midnight and ending with an RMC
sentence cut inside its date, it still gives the 650 points, and one
warning. A log whose sentences are all cut short, which gave a
stationary video, fails with the reason. The NMEA logs of a Sony action
camera and of an image sequence read the same points as before, and
process of the images and the video, video_process by distance and by
interval with the NMEA log, and plain process and video_process of other
cameras write the same descriptions.
Unit and integration tests, doctests,
mypy --python-version 3.9,ruff check,ruff format --checkandusort diffare clean.