Add MapLibre Tile (MLT) output by dktue · Pull Request #938 · systemed/tilemaker

dktue · 2026-10-12T14:27:45Z

Adds optional MapLibre Tile (MLT) output, as discussed in #856. It's off by default at build time and in the config; MVT output is unchanged.

Results (Switzerland, bundled OpenMapTiles profile, .pmtiles; details below):

12% smaller tilesets: 274.6 MB as MLT vs 313.5 MB as MVT, after gzip.
Memory: +0.9% peak RSS (7.65 vs 7.58 GB). MLT output adds almost no memory.
Time: +8% wall clock (85.2 vs 79.1 s). The encoder is configured to run about 7× faster than with its default settings, for a compressed size within 2.5% of what the defaults produce.
No change for MVT users: with this PR, MVT output and speed are the same as master within measurement noise.

Usage

Build with -DTILEMAKER_WITH_MLT=ON, then set "tile_format": "mlt" in settings. Lua profiles and layer configs don't change. In MapLibre styles, the source needs "encoding": "mlt".

.pmtiles: header tile type 0x06 (MapLibre Tile)
.mbtiles: metadata format is mlt
directory output: z/x/y.mlt
--merge is rejected with MLT (merging needs to decode existing tiles; could be added later with the MLT decoder)

How it works

Encoder: the C++ encoder from maplibre-tile-spec (merged there in June), fetched with CMake FetchContent at a pinned commit. Only the encoder target and its two needed submodules (fsst, earcut) are used. FETCHCONTENT_SOURCE_DIR_MLT can point at a local checkout for offline builds.
One code path for both formats: tile_worker.cpp's write functions are templated over a small backend. The MVT backend is plain vtzero, so the MVT code path compiles to the same thing as before. The MLT backend (mlt_writer.h) has builders with the same method names that collect a tile's features, then encodes once per tile with a thread-local encoder.
Feature order is kept (encoder sortFeatures off), so ZOrder still controls draw order.
Column types: MLT stores each attribute as a typed column per layer, and the encoder turns int + float mixes into strings. tilemaker promotes the integers of mixed int/float columns to double instead, so AttributeInteger and AttributeNumeric on the same key in one layer stay numeric.
Encoder settings: for compressed output (gzip/deflate .mbtiles/directories, and .pmtiles, which are always gzipped), FSST string compression and Morton/Hilbert vertex dictionaries are off and geometry uses plain varints. These choices make up most of the encoder's time, but after gzip they save next to nothing (see below). Uncompressed output keeps the encoder's defaults.
Build: the CMake build moves to C++20, which the encoder's headers need. Two small compatibility fixes come with it: BOOST_ASIO_DISABLE_CO_AWAIT (Boost 1.74's Asio coroutine header doesn't compile as C++20; tilemaker doesn't use coroutines), and std::uncaught_exceptions() instead of the removed std::uncaught_exception() in the bundled sqlite_modern_cpp (MSVC). The Makefile build is unchanged and has no MLT support. MLT itself needs CMake 3.25+ (required by maplibre-tile-spec).
Docker and CI: the Dockerfile builds with MLT on (adds git and ca-certificates for the fetch). The Linux CMake CI job builds with MLT and writes an MLT .pmtiles of Liechtenstein.

Testing

MLT matches MVT feature for feature. I used a test-only build that writes every tile as both MVT and MLT from the same data in the same run, then decoded both with vtzero and with maplibre-tile-spec's own decoder and compared them in order (ids, geometry type, every coordinate, every property). Clip cache lookups were disabled for this, because cache reuse is timing dependent and otherwise makes the two passes differ. Inputs: Monaco and Luxembourg, with the bundled OpenMapTiles profile.
- Monaco: identical, 10,912 features.
- Luxembourg (about 1 million features): the only geometry differences are in polygon grouping. MVT has no explicit polygon structure, so readers infer it from ring winding, and after simplification a tiny ring can have the "wrong" winding and get read as a hole. MLT stores the structure tilemaker actually wrote.
- The one other difference wasn't from MLT. A few cycleways came out as class=track instead of path. That turned out to be an existing intermittent issue on master (in 4 of 5 runs of unmodified master on Luxembourg); I'll report it separately.
Mixed int/float columns: a synthetic profile writing AttributeInteger and AttributeNumeric to the same key gives a DOUBLE column with the expected values.
Unit tests: make test passes, including a new config validator test for tile_format.
Not tested locally: MSVC and macOS with MLT enabled. CI covers the C++20 switch on all platforms but only builds MLT on Linux.

Benchmarks

Switzerland (548 MB .osm.pbf), bundled OpenMapTiles profile, .pmtiles output, 22 threads, average of 3 interleaved runs:

format	wall	CPU	peak RSS	size
MVT (master)	79.1 s	1429 s	7.58 GB	313.5 MB
MVT (this PR)	78.9 s	1444 s	7.59 GB	313.4 MB
MLT	85.2 s	1558 s	7.65 GB	274.6 MB (−12%)

MVT is unchanged within noise. MLT is about 8% slower with under 1% more memory, for tiles 12% smaller.

Encoder settings, single-threaded, Luxembourg tiles, sizes after zlib:

encoder settings	ms/tile	size after zlib
encoder defaults	4.4	27.15 MB
defaults + FSST	10.0	27.18 MB
this PR (no FSST, no Morton, plain geometry)	0.6	27.84 MB

Open questions

Should MLT be on by default? I kept it opt-in because it adds a network fetch at configure time and needs CMake 3.25.
EncoderConfig::geometryEncodingOption is commented upstream as "primarily for testing". The commit is pinned, but it may be worth asking maplibre-tile-spec to treat it as supported.
I'm happy to split out the C++20 change or the templating refactor if that's easier to review.

🤖 Generated with Claude Code

Add an optional MLT writer, built with -DTILEMAKER_WITH_MLT=ON and selected with "tile_format": "mlt". The tile writing code is templated over an MVT (vtzero) and an MLT backend; MLT tiles are encoded with the maplibre-tile-spec C++ encoder, fetched at a pinned commit. - PMTiles tile type 0x06, MBTiles format=mlt, .mlt directory output - feature order (ZOrder) preserved; mixed int/float columns kept numeric - fast encoder settings for compressed output - CMake build moves to C++20, with fixes for Boost 1.74 Asio and the bundled sqlite_modern_cpp; Dockerfile and Linux CI build with MLT Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

systemed · 2026-10-13T07:58:13Z

Thank you - this is brilliant. Fully supportive of the opt-in approach because of the increased system requirements but we can maybe move to opt-out some time in the future. I'll take a proper look in the next couple of days.

systemed · 2026-10-21T10:39:35Z

Very happy with this - have tested it and it builds fine on Mac with the relevant CMake/C++20, without breaking compatibility with earlier toolchains if MLT isn't compiled in. Thank you!

louwers · 2026-10-21T11:25:22Z

Great to see this land in tilemaker.

MapLibre GL JS requires the same winding order as MVT. I prepared a PR that updates the C++ encoder with this: maplibre/maplibre-tile-spec#1981

Hopefully we can merge it later in the week.

…does (#1981) #1694 took the C++ encoder tests out of CI because they had fallen behind. This puts `test_encode.cpp` and `test_synthetic.cpp` back, brings them up to date, and ports the polygon winding from the PR below so the C++ encoder writes what Java writes. ### What was wrong - **Undefined behavior:** `encodeVarint` took `&buf[n]` for the end of its 10-byte buffer, and `BufferStream::getReadPosition` took `&data[offset]` when reading zero bytes at the end of the data. With assertions on (`_GLIBCXX_ASSERTIONS`, hardened libc++) that aborts. It was about 480 aborts locally. - **Stale generator:** the C++ synthetic generator still used the default extent 80 (#1751 made it 64), and lacked the fixtures added since (`extent_32`, `extent_32768`, `id_max`, `id64_max`, `ids64_minmax`, `ids64_minmax_delta` and the four `prop_*_delta`). It still generated the tessellated `poly_collinear` cases, which #1614 removed, and `AllFixturesHaveGenerators` fails on those. - **Tests that invented IDs:** the re-encode helpers copied a feature's ID only if it had one, leaving `Encoder::Feature::id` at its default of 0. Tiles without IDs came back with them. - **`ByteCompare.MultiPolygonBoolean`:** the `simple/` fixtures are written by Rust now (#1870), which counts a stream's metadata when it chooses an encoding. The C++ encoder, like Java, compares only the encoded values, so it picks RLE for `[4, 4, 4]` where delta is a byte smaller. - **Winding:** the PR below regenerates the 167 polygon fixtures in `test/synthetic/0x01`, with MVT winding. The C++ encoder stored rings as given. - **googletest:** the vendored snapshot does not compile with recent clang and `-Werror` (`PrintTo(char8_t)`). ### Changes - `fix(cpp)`: use pointer arithmetic instead of subscripting one past the end. - `build(cpp)`: update googletest to v1.18.0. - `feat(cpp)`: `PolygonWinding` winds polygon rings as MVT does, as Java and Rust do. An exterior ring with a negative area, or a hole with a positive area, is reversed, keeping its first vertex (`[v0, vn-1, ..., v1]`). Rings without area are left alone. The area sum is exact (128 bits, no `__int128`, so it builds with MSVC). It is applied when the vertices are stored and when they are tessellated, so the triangles index the stored vertices. - `test(cpp)`: re-enable the encoder tests and update them as described above. The winding tests are in their own file, `test_winding.cpp`. The tests also build in more configurations (see below). ### Not byte-exact, but equivalent `id_max` and `ids64_minmax_delta` decode to the same tile as the fixture and are in the exception list of `test_synthetic.cpp` with the reason: - C++ writes a 64-bit ID column for IDs above `INT32_MAX`. - C++ does not delta-encode 64-bit values above `INT64_MAX`, where the signed delta would overflow. `ByteCompare` allows 2 bytes over the fixture, the metadata of one RLE stream. I did not change the C++ choice of integer encoding, because it matches Java, which the 315 byte-exact synthetic fixtures rely on. ### Tests capturing the change - `Winding.*` in `test_winding.cpp`: the rule itself, keeping the first vertex, multipolygons, with and without tessellation, rings without area, malformed ring sizes, and exact signs for coordinates whose products overflow 64 bits. With the rewinding disabled, 3 of them and 167 of the synthetic fixtures fail. - `SyntheticFixtureTest` compares 315 fixtures byte for byte with what Java wrote, and the other 51 by their decoded JSON. ### Other configurations - `MLT_WITH_JSON=OFF` with the tests: `test_synthetic.cpp` said so with `#warning`, which `-Werror` makes an error. It is a `#pragma message` now (an error in neither gcc nor clang with `-Werror`). - 64-bit literals in the zigzag test are `LL`: `long` is 32 bits on Windows, which made the list mix `long` and `long long`. - `MLT_AS_OBJECT_LIBRARY=ON` with the tests: `mlt-cpp` and `mlt-cpp-encoder` both contain `metadata/stream.cpp` and `metadata/tileset.cpp`, and object libraries put all their objects in a link, so a binary using both has them twice (multiple definition). The encoder tests are not built then, as before this PR. The libraries themselves are unchanged, so what `$<TARGET_OBJECTS:...>` contains for a consumer does not change. - The configuration [tilemaker](systemed/tilemaker#938) uses, `MLT_WITH_JSON=OFF MLT_WITH_TESTS=OFF MLT_WITH_TOOLS=OFF` and `mlt-cpp-encoder` only, builds with gcc and clang. ### Verified locally - `ctest`: 985 of 985 with gcc 15 and with clang 21 (Debug, `-Werror`), including the assertions in libstdc++. 985 of 985 with `MLT_WITH_JSON=OFF` (gcc), and the 118 decoder tests with `MLT_AS_OBJECT_LIBRARY=ON`. - `just cpp::test` decoder checks (`vitest`): 1169 passed. - Not run locally: Bazel, and the `gcovr` coverage report (no `gcov` in my environment). 🤖 Generated with [Claude Code](https://claude.com/claude-code)

louwers · 2026-10-21T18:42:05Z

Please pick up a new commit. Thanks!

dktue mentioned this pull request Oct 14, 2026

perf: 3.4x faster planet build (2h10 -> 39 min), less RAM, deterministic output #939

Draft

systemed merged commit c178dba into systemed:master Oct 21, 2026
11 checks passed

systemed mentioned this pull request Oct 21, 2026

Support for generating MapLibre Tiles (MLT)? #856

Closed

louwers mentioned this pull request Oct 21, 2026

test(cpp): run the encoder tests again and wind polygon rings as MVT does maplibre/maplibre-tile-spec#1981

Merged

Navigation Menu

Repository navigation

Add MapLibre Tile (MLT) output - #938

Add MapLibre Tile (MLT) output#938
systemed merged 1 commit into
systemed:masterfrom
dktue:mlt-output

dktue commented Oct 12, 2026

Uh oh!

systemed commented Oct 13, 2026

Uh oh!

Uh oh!

systemed commented Oct 21, 2026

Uh oh!

louwers commented Oct 21, 2026

Uh oh!

louwers commented Oct 21, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

dktue commented Oct 12, 2026

Usage

How it works

Testing

Benchmarks

Open questions

Uh oh!

systemed commented Oct 13, 2026

Uh oh!

Uh oh!

systemed commented Oct 21, 2026

Uh oh!

louwers commented Oct 21, 2026

Uh oh!

louwers commented Oct 21, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants