Skip named sidecar files with a warning, rather than fail them by caglarpir · Pull Request #920 · mapillary/mapillary_tools

caglarpir · 2026-10-24T21:56:04Z

Stacked on #919; review the last commit.

Since #897, a file named for process that is not a supported image or
video fails with MapillaryUnsupportedFileError. That includes the files
that cameras and other tools write beside images and videos, which a
user names along with them by selecting all the files of a camera
folder: process_and_upload DCIM/100GOPRO/* named the GoPro thumbnails
(.THM) and low-resolution proxies (.LRV) beside the videos, and GPS and
metadata files beside them, so the run exited 6 and uploaded nothing,
zip files included, unless --skip_process_errors was given. An
AppleDouble file, which macOS writes as ._NAME beside each file it
copies to a drive that cannot hold its metadata, has the extension of
that file: named, ._GOPR0001.JPG was read as an image and failed as one
without GPS, and ._GH010001.MP4 as a video that cannot be parsed. The
AppleDouble file of a zip file, named for upload or process_and_upload,
was read as a zip file and stopped the run with a BadZipFile traceback
before any zip file was uploaded.

Skip a named file with a known sidecar or companion extension (.thm,
.lrv, .gpx, .nmea, .xml, .json, .srt, .txt, .csv, .log, .md5, in any
case), an AppleDouble file and .DS_Store, with the warning that #871
printed: "Skipping ..., which is not a supported image or video". The
process report still lists them as unsupported_format, AppleDouble files
too, which it counted among the processed files for the extension of an
image or video. Upload skips the AppleDouble file of a zip file with the
same warning, and uploads the zip files named with it. Any other named
unsupported file, such as an Insta360 original (.insv), a PNG or a HEIC
image, still fails with #897's error and hint. A folder scan, which
leaves hidden files and these files alone, does not change.

Checked process on a camera folder named file by file, with an image, a
video, a zip file, a thumbnail, a proxy, a GPX, an XML, a text file, the
AppleDouble files of the image and the video, .DS_Store, an Insta360
original and a PNG: the 8 sidecar files are skipped with 8 warnings and
listed in the report as before, and the original and the PNG fail as
before. Named without the last two, the folder now exits 0 (was 6).
process_and_upload of named zip files with sidecar files beside them
uploads the zip files and exits 0 without --skip_process_errors, also
with the AppleDouble file of one of them, which made process_and_upload
and upload exit 1 with nothing uploaded. Folders of images, videos of
several cameras, a mixed folder, a GPX run, and video_process and
sample_video on named videos and folders give the same descriptions and
exit codes as before.

Unit and integration tests, doctests, mypy --python-version 3.9, ruff check, ruff format --check and usort diff are clean.

Since mapillary#898, interval sampling with --geotag_source gpx reads the GPX again for each video, to tell which samples fall outside its track. Sampling 11 BlackVue videos every 30 s against a 163 MB GPX of 498 tracks, video_process took 293 s instead of 161 s before mapillary#898, and printed "Found 498 tracks in the GPX file" 12 times instead of once. The other readers parsed such a file once per video too: distance sampling, in a pool of processes started for each video, and the geotagging stage of videos, in each process of its pool. An NMEA file is read once per process since mapillary#909. Parse a GPX file once per process, and share its tracks among the readers: interval and distance sampling, GPX sync of --geotag_source and --video_geotag_source, and the geotagging stage of images. The tracks are kept by the path and content of the file, so a file that changes is parsed again, and only those of the last four files parsed are kept, in the main process only. The geotagging stage of videos reads a GPX that geotags several videos once, before its pool, as it reads an NMEA file; the GPX of each video is still parsed in the pool. Distance sampling reads its video in its own process, as interval sampling does. GPX sync moves copies of the points. The warnings about the tracks of a file, and about its points before 1980, are given once per file. That video_process run now takes 82 s and prints the warning once. Its 22 samples and their descriptions are the same, and its peak memory is 2.68 GB (2.84 GB on main). On one of the videos it takes 15.9 s instead of 25.3 s, and process 10.1 s with 2.69 GB, as on main. Sampling the 11 videos by distance with the GPX as --video_geotag_source takes 25 s instead of 107 s. process on them, with the GPX as --geotag_source or --video_geotag_source, takes 22 s instead of 26 to 29 s, and its processes peak at 6.4 GB together instead of 14 to 16 GB, as each of them parsed the file. Compared with main on process, video_process and sample_video, with GPX and NMEA sources and with the default sources, on BlackVue, GoPro MAX, Sony and 70mai videos and on images, the descriptions and samples are the same, and only the repeated warnings are gone. mapillary#898 said that with the legacy sources the frames before a late GPS start were kept and geotagged as before: they are kept, and fail as before.

mapillary#901 took every sentence without a checksum for cut short as soon as any other sentence of the file had one. Checksums are optional in NMEA 0183, and logs that mix sentences with and without them read in full before: a log with a checksum on one line only, or on its GGA sentences only, now failed whole as "not a valid NMEA file", and of two logs concatenated, one with checksums and one without, the half without was skipped. A sentence without its checksum is now cut short only when its own fields show it: when it has fewer fields than a complete sentence (11 for RMC, as NMEA 2.0 to 2.2 write it, 12 with the mode of NMEA 2.3 and 13 with the navigational status of NMEA 4.1; 14 for GGA), when the date of an RMC sentence is not 6 digits, or when its time does not parse. A sentence with its checksum ends where the checksum shows, however few fields it has, and one whose checksum does not match is still malformed. The checksums of other sentences no longer count. When no fix of a file can be read, the error now says why, with the first malformed sentence, its line and the reason: "track.nmea is not a valid NMEA file: 2 of its 2 RMC and GGA sentences are malformed, the first on line 1 (checksum does not match: 00 != 47): $GPRMC,...". It said only "not a valid NMEA file". (mapillary#901's message said that pynmea2 reads a time cut to "00052" as 00:00:02; it reads it as 00:05:02.) Checked on a 650-fix log, with the video it belongs to: with a checksum on one line only, with checksums on its GGA sentences only, and as two halves with and without checksums, it gives the track of the original, 650 points over 649 s, with no warning, where it failed twice and gave 325 points. Written GGA first across midnight and ending with an RMC sentence cut inside its date, it still gives the 650 points, and one warning. A log whose sentences are all cut short, which gave a stationary video, fails with the reason. The NMEA logs of a Sony action camera and of an image sequence read the same points as before, and process of the images and the video, video_process by distance and by interval with the NMEA log, and plain process and video_process of other cameras write the same descriptions.

With --video_geotag_source gpx followed by another source, such as --video_geotag_source native, interval and distance samples of a track placed from the GPX lost the handling of mapillary#903. A Sony video without GPS of its own, sampled against a GPX with a 73 s gap at MAPILLARY_TOOLS_MAX_GPX_GAP_SECONDS=60, got positions across the gap for 7 of its 65 interval samples (every 10 s) and 113 of its 1036 distance samples, where gpx alone leaves them out and process fails the video. With --interpolation_use_gpx_start_time, the track was not moved to start at the first interval sample, and the first sample of the Sony video was left out. The interval samples were stamped by the start time or the GPS clock of the video rather than by the GPX where they are: 1 s apart on the Sony video, and 2.9 s on a GoPro MAX timelapse with --interpolation_use_gpx_start_time. mapillary#903 took a track as placed from a file only when every source was a GPX or NMEA file, but the sources are tried in turn, and the one that read the track placed it. factory.process() now tells which source geotagged each file, and sampling decides by it. A track that a GPX or NMEA file placed has its gaps left out, its start moved and its samples stamped by it, whatever the other sources are. A track that the video's own GPS gave, before a GPX or after one that misses the video, is sampled as before. With gpx followed by native or video, and with native followed by gpx on a video without GPS of its own, the samples, descriptions and warnings are now those of gpx alone, on the Sony video and the GoPro MAX timelapse, by interval and by distance, with and without the start time. Unchanged: gpx alone; a BlackVue video that a GPX misses, geotagged by its own GPS after or before the GPX; native before a GPX on the GoPro MAX timelapse; process; and the default sources. With the legacy gopro source or exiftool_xml geotagging the samples, every sample is kept and the descriptions do not change, as before; with exiftool_xml, which reads videos as well, the interval samples are now stamped by the GPX, as they were with gopro.

Since mapillary#902 decodes an NMEA file with errors="replace", so that a byte damaged in a log costs one sentence, a file that is not text at all is decoded whole too, and mapillary#909 reads it whole to hash it. A 1.3 GB video given as the NMEA file of `process --geotag_source nmea` took 30 to 50 s and 4.5 to 8 GB of memory before it failed as "not a valid NMEA file", where it failed in a fraction of a second before mapillary#902; a video of tens of gigabytes could run the machine out of memory. Look at the first 64 KiB of the file first. If they have a NUL byte, or if most of them do not decode as UTF-8, the file is not text: it fails at once, as "not a valid NMEA file: it is not text", without being read further. A text file is read as before, a byte damaged in one of its sentences included. The 1.3 GB video now fails in 4 s (a run without the NMEA file takes 3 s on the same machine) with 64 MB of memory, and a 4.3 GB video in under 2 s. The NMEA logs of a Sony action camera and of an image sequence read the same points as before, a log with a byte that is not text in one sentence geotags the same 106 of its 113 images, and process and video_process with NMEA logs, and of other cameras without them, write the same descriptions.

A time in a gap of more than MAPILLARY_TOOLS_MAX_GPX_GAP_SECONDS (an hour by default) between two fixes of a GPX or NMEA track is outside the track. process (mapillary#899) and interval sampling compared the gaps of a track placed on a video in video seconds, against the limit divided by the rate of the video's clock. The division leaves the two a rounding error apart, so at a rate other than 1 a gap of exactly the limit could count as more than it. On a LabPano video, whose clock runs 1.143 times as fast as its video, a fix exactly an hour before a track that starts 17 seconds into the video was left out, and the track uploaded with the video started at 17.2 s rather than at 0, while interval sampling left out 2 of its 58 samples. With the limit at 60 s, a gap of exactly 60 s inside the video failed it in process, and interval sampling left out 5 of the 58 samples. Judge every gap in GPS seconds: between the GPS times that the points of a GPX or NMEA track keep once placed on the video, or, for points without one, between their times at the rate of the clock; rounded to the millisecond, which no GPS time is recorded more finely than. A gap of more than the limit is one, and a gap of exactly the limit is not, as the geotagging stage judges the gaps of a track for images and as distance sampling judged them already. Checked process, distance sampling and interval sampling of a LabPano, a BlackVue and a GoPro MAX timelapse video, each with GPX tracks made from its own track with a gap of exactly the limit, or half a second more, before the video and inside it, at limits of 3600 and 60 s (54 runs): only the LabPano runs with a gap of exactly the limit change, as above. Gaps half a second over the limit are left out as before, and the BlackVue and timelapse runs do not change. Nor do the 24 runs with real GPX and NMEA tracks of a GoPro MAX, a Sony camera and 70mai dashcams, whose logs have gaps of days.

video_process gives the sources without a source path of their own the video import path, as --geotag_source_path would (mapillary#906). A source with a path of its own keeps it, and video_process warned that "The option --geotag_source_path is ignored for source ...", although no --geotag_source_path was given: on every `video_process DIR OUT --video_geotag_source exiftool_xml`, as exiftool_xml gets the pattern %g.xml with the video import path, and for a source given a path of its own, such as {"source": "gpx", "source_path": ...}. Warn only of a --geotag_source_path that the user gave. A source with a path of its own still ignores it, with the warning, in process and in video_process. Checked process and video_process on a BlackVue video with its ExifTool XML and with a GPX: with exiftool_xml alone, and with a gpx source with a path of its own, video_process no longer warns. With a source with a path of its own and --geotag_source_path, process and video_process warn as before (once, and three times). With exiftool_xml and --geotag_source_path, with exiftool_xml in process, and with the default sources, nothing changes. The descriptions of all 8 runs are those of before.

…en in mapillary#911 rejected capture times before 1970, so that the next date tag is tried, but left three cases: - An image whose only date was rejected lost the reason: "Unable to extract timestamp from the image", where the error used to end with ": invalid capture time 0001-01-01 00:00:00". - A time with a time zone offset was judged by its own date, not in UTC: 1970-01-01 00:30 at +02:00, which is 1969-12-31 22:30 in UTC, was written as a 1969 capture time, and 1969-12-31 23:30 at -01:00, which is 1970-01-01 00:30 in UTC, was rejected. - A time in year 10000 in UTC, such as 9999-12-31 23:00 at -05:00, has no date that a description can be written with. Geotagged by a GPX, the error that names the time of an image outside the track could not write it either, and stopped the run with exit 15 and no description. A capture time is now valid from 1970-01-01 00:00 in UTC for a time with an offset, and as it is for one without, up to the end of year 9999 in UTC. A date tag that gives another time is still skipped for the next one; when none gives a valid time, the error names the first rejected one again. The time of a naive date, which is read in the local time zone, is checked once converted too, so that no capture time before 1970 or after year 9999 in UTC is written. The GPX error that an image is outside the track names a time that cannot be written as a date by its Unix time, so that a time offset that moves an image past year 9999 fails that image, not the run. Checked process with the default sources and with exiftool_runtime on 17,090 sample images from many cameras: the entries are the same, but for two images dated 0001-01-01, whose errors name that date again. With a GPX, the image in year 10000 in UTC gets an error entry and the run exits 0, and so does an image moved past year 9999 by --interpolation_offset_time. video_process and process of a GoPro and a BlackVue video do not change.

Some editors on Windows start a UTF-8 file with a byte order mark. An NMEA file saved by one lost its first sentence to it, which did not parse, with a warning that quoted it as it looked: "Skipping 1 of the 1300 RMC and GGA sentences ..., which are malformed, the first on line 1 (could not parse data): $GPRMC,...". When that sentence was a GGA, its fix was lost. Decode the file as "utf-8-sig", which leaves the mark out, with the bytes that are not text still decoded as U+FFFD, and leave out a mark that starts any line too, as where logs saved with one are appended to each other. A file in UTF-16, which Windows PowerShell can write, is still not read: its NUL bytes show it is not text. A 650-fix log and the log of a 113-image sequence, each saved with a byte order mark, read the points of the logs without it, with no warning. The NMEA logs of a Sony action camera and of an image sequence read the same points as before, and process and video_process with NMEA logs, and of other cameras without them, write the same descriptions.

Since mapillary#897, a file named for process that is not a supported image or video fails with MapillaryUnsupportedFileError. That includes the files that cameras and other tools write beside images and videos, which a user names along with them by selecting all the files of a camera folder: `process_and_upload DCIM/100GOPRO/*` named the GoPro thumbnails (.THM) and low-resolution proxies (.LRV) beside the videos, and GPS and metadata files beside them, so the run exited 6 and uploaded nothing, zip files included, unless --skip_process_errors was given. An AppleDouble file, which macOS writes as ._NAME beside each file it copies to a drive that cannot hold its metadata, has the extension of that file: named, ._GOPR0001.JPG was read as an image and failed as one without GPS, and ._GH010001.MP4 as a video that cannot be parsed. The AppleDouble file of a zip file, named for upload or process_and_upload, was read as a zip file and stopped the run with a BadZipFile traceback before any zip file was uploaded. Skip a named file with a known sidecar or companion extension (.thm, .lrv, .gpx, .nmea, .xml, .json, .srt, .txt, .csv, .log, .md5, in any case), an AppleDouble file and .DS_Store, with the warning that mapillary#871 printed: "Skipping ..., which is not a supported image or video". The process report still lists them as unsupported_format, AppleDouble files too, which it counted among the processed files for the extension of an image or video. Upload skips the AppleDouble file of a zip file with the same warning, and uploads the zip files named with it. Any other named unsupported file, such as an Insta360 original (.insv), a PNG or a HEIC image, still fails with mapillary#897's error and hint. A folder scan, which leaves hidden files and these files alone, does not change. Checked process on a camera folder named file by file, with an image, a video, a zip file, a thumbnail, a proxy, a GPX, an XML, a text file, the AppleDouble files of the image and the video, .DS_Store, an Insta360 original and a PNG: the 8 sidecar files are skipped with 8 warnings and listed in the report as before, and the original and the PNG fail as before. Named without the last two, the folder now exits 0 (was 6). process_and_upload of named zip files with sidecar files beside them uploads the zip files and exits 0 without --skip_process_errors, also with the AppleDouble file of one of them, which made process_and_upload and upload exit 1 with nothing uploaded. Folders of images, videos of several cameras, a mixed folder, a GPX run, and video_process and sample_video on named videos and folders give the same descriptions and exit codes as before.

caglarpir added 9 commits October 24, 2026 22:53

meta-cla Bot added the cla signed label Oct 24, 2026

caglarpir merged commit 9dcc72e into mapillary:main Oct 24, 2026
16 checks passed

Navigation Menu

Repository navigation

Skip named sidecar files with a warning, rather than fail them - #920

Skip named sidecar files with a warning, rather than fail them#920
caglarpir merged 9 commits into
mapillary:mainfrom
caglarpir:skip-named-sidecar-files

caglarpir commented Oct 24, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

caglarpir commented Oct 24, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant