Skip to content

Read an NMEA log padded with NUL bytes - #923

Merged
caglarpir merged 1 commit into
mapillary:mainfrom
caglarpir:nmea-binary-only-without-sentences
Oct 26, 2026
Merged

caglarpir merged 1 commit into
mapillary:mainfrom
caglarpir:nmea-binary-only-without-sentences

Conversation

@caglarpir

Copy link
Copy Markdown
Contributor

A logger leaves NUL bytes in its log after a power cut, such as on an
SD card. #915 took one NUL byte in the first 64 KiB of a file for the
sign of a binary file, and failed the whole log as "not a valid NMEA
file: it is not text", where earlier versions read it. With 4 KiB of NUL
bytes appended to the NMEA log of an image sequence, all 113 images
failed, where the log without them geotags 106, and the other 7 are
duplicates. A longer log with the same tail was still read, only
because its first 64 KiB hold no NUL byte.

Take a file for one that is no NMEA log at all only if its first 64 KiB
hold no RMC or GGA sentence, and NUL bytes or mostly bytes that do not
decode as UTF-8. A video given as the NMEA file still fails at once: on
1.2 and 4.0 GB videos in 0.7 to 1.0 s and 61 MB, as before; no video or
image among 18,023 sample files holds an RMC or GGA sentence in its
first 64 KiB. NUL bytes now end a line, as no sentence holds one: a
sentence that a logger wrote on after them is read, and one with NUL
bytes inside it is skipped under the warning of malformed sentences.
A UTF-16 log, which some Windows tools write, is still not read, but
the error now says it is UTF-16 text, and to save it as UTF-8 or ASCII.

The NUL-padded log of the image sequence now gives the same 106
geotagged images and 7 duplicates as the log without the NUL bytes,
with identical descriptions. So does a video log with the same tail.
The NMEA logs without NUL bytes, process of images, of GoPro MAX and
BlackVue videos, video_process by distance and with an NMEA log, give
the same results as before.

Unit and integration tests, doctests, mypy --python-version 3.9, ruff check, ruff format --check and usort diff are clean.

A logger leaves NUL bytes in its log after a power cut, such as on an
SD card. mapillary#915 took one NUL byte in the first 64 KiB of a file for the
sign of a binary file, and failed the whole log as "not a valid NMEA
file: it is not text", where earlier versions read it. With 4 KiB of NUL
bytes appended to the NMEA log of an image sequence, all 113 images
failed, where the log without them geotags 106, and the other 7 are
duplicates. A longer log with the same tail was still read, only
because its first 64 KiB hold no NUL byte.

Take a file for one that is no NMEA log at all only if its first 64 KiB
hold no RMC or GGA sentence, and NUL bytes or mostly bytes that do not
decode as UTF-8. A video given as the NMEA file still fails at once: on
1.2 and 4.0 GB videos in 0.7 to 1.0 s and 61 MB, as before; no video or
image among 18,023 sample files holds an RMC or GGA sentence in its
first 64 KiB. NUL bytes now end a line, as no sentence holds one: a
sentence that a logger wrote on after them is read, and one with NUL
bytes inside it is skipped under the warning of malformed sentences.
A UTF-16 log, which some Windows tools write, is still not read, but
the error now says it is UTF-16 text, and to save it as UTF-8 or ASCII.

The NUL-padded log of the image sequence now gives the same 106
geotagged images and 7 duplicates as the log without the NUL bytes,
with identical descriptions. So does a video log with the same tail.
The NMEA logs without NUL bytes, process of images, of GoPro MAX and
BlackVue videos, video_process by distance and with an NMEA log, give
the same results as before.
@meta-cla meta-cla Bot added the cla signed label Oct 26, 2026
@caglarpir
caglarpir merged commit 51ece18 into mapillary:main Oct 26, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant