Skip to content

Look up the visitor in the block and allow list by index (65 ms -> 1 ms per request) - #240

Open
codeling wants to merge 1 commit into
mainfrom
cidr-lookup-cost
Open

codeling wants to merge 1 commit into
mainfrom
cidr-lookup-cost

Conversation

@codeling

Copy link
Copy Markdown
Owner

Problem

onAfterRoute() runs for every request of every visitor and looked the address up in the block list and in the allow list. Each lookup read every network (CIDR) row of the table and tested it in PHP, and the exact-address part compared LOWER(ipaddress), which no index can serve (and there was no index). Every IPv6 client the plugin blocks adds a network row, so a botnet could make every page of the site slower.

Measured here (PostgreSQL 16, 150,000 blocked addresses + 20,000 blocked IPv6 /64 networks + 50 allow list ranges, all active): 65-70 ms of database work per request, for IPv4 and IPv6 visitors alike.

Change

  • Addresses and networks of both lists are stored in one canonical form (IpHelper::canonicalEntry()): compressed lower case, networks as their masked network address and prefix length (2001:db8:1::/48, 203.0.113.0/24).
  • With that, the entries that can contain an address are known: the address and its network for each prefix length (IpHelper::lookupCandidates(): 34 strings for IPv4, 130 for IPv6). One ipaddress IN (...) query on a new index finds them; the matching in PHP is gone.
  • Result: 1.1-2 ms per request with the same data.
  • Writers store canonical entries: blockIP(), the tracked address of a client (IpHelper::trackedAddress()), and - in the matching com_bfstop PR - the edit forms' tables.
  • Existing entries are converted once by the plugin, before its first lookup after the update (marker setting addressesCanonical). The installer script can't do that reliably: during an update the previous version's DatabaseHelper may already be loaded in that request. Scanning 170,000 entries takes 0.3 s; the conversion is idempotent (it also runs again after a settings save, which drops bookkeeping values).
  • Indexes on bannedip.ipaddress and allowlist.ipaddress in the install scripts (MySQL, PostgreSQL) and sql/updates/2.0.0.sql (2.0.0 is not released yet).

Behaviour changes to be aware of

  • The stored spelling of an entry changes (203.0.113.77/24 is kept as 203.0.113.0/24, 2001:DB8::1 as 2001:db8::1). What is blocked or allowed stays the same.
  • Entries added to the tables by other means (SQL, other tools) must be in canonical form; entries that are no address or network are ignored, as before.
  • The tracked key of a client that is not aggregated (e.g. NAT64) is now canonical too (64:ff9b::c000:207 instead of 64:ff9b::192.0.2.7) - consistent everywhere, since all code uses trackedAddress().

Tests

  • Unit: canonicalEntry() (19 cases incl. malformed prefixes), lookupCandidates(), and that every candidate is canonical and contains the address.
  • Integration: migration (incl. batches beyond one batch of 5,000, garbage left alone, idempotent), blocks stored canonical, candidates don't pull in other networks, and the existing active-block/allow-list tests now go through the migration.
  • Full suite with the com_bfstop branch of the same name: 352 tests pass on Joomla 5.4.8 / PostgreSQL 16 (1 skipped: read-only file, because of running as root). Not run locally: MySQL/MariaDB, Joomla 6, PHP 8.1 (CI covers them).

Component side: codeling/com_bfstop (branch cidr-lookup-cost).


Generated by Claude Code

Every request of every visitor read all networks of both lists and tested each
in PHP; the exact-address part compared LOWER(ipaddress), which no index can
serve. With 150000 blocked addresses and 20000 blocked IPv6 networks that was
65-70 ms of database work per request (1-2 ms now).

Addresses and networks are stored in one canonical form
(IpHelper::canonicalEntry()), so the entries which can contain an address are
known (IpHelper::lookupCandidates(): the address and its network for each
prefix length) and one indexed IN query finds them. The plugin converts the
existing entries once, before its first lookup after the update (the
installer script can't, the previous version's class may be loaded already);
blocks and tracked addresses are written canonical. Indexes on both tables.

The component's forms store canonical entries as well (matching branch of
com_bfstop).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LhhhdjhErC9ZiaS4HyRjsh
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants