Repository navigation
feat: load policy with a single LRANGE instead of one LINDEX per rule - #4
Merged
Merged
Conversation
load_policy issued LLEN and then one LINDEX per rule, so loading N rules took N+1 client/Redis round trips. With a few thousand rules over a ~1 ms network that is several seconds per load, almost all of it spent waiting on round trips. A single LRANGE fetches the whole list in one round trip and returns an atomic snapshot, so a concurrent LREM can no longer shrink the list mid-load and make LINDEX return None. This matches the Go redis-adapter, which already loads with LRANGE.
Coverage Report for CI Build 37948353873Coverage increased (+0.07%) to 98.305%Details
Uncovered ChangesNo uncovered changes found. Coverage RegressionsNo coverage regressions found. Coverage Stats
💛 - Coveralls |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adapter.load_policyusedLLENfollowed by oneLINDEXper rule:This PR fetches the whole list with a single
LRANGE key 0 -1instead. Parsing is unchanged (json.loads→CasbinRule→persist.load_policy_line), and nothing on the write path changes.Why
Latency. Each
LINDEXis a separate client↔Redis round trip, so loading N rules costs N+1 round trips. The Redis server-side work is negligible; the time goes into waiting on the network. Measured against a managed Redis (~0.75 ms RTT) with ~2,950 rules:LLEN+ N ×LINDEX(current)LINDEXcalls pipelined (1 round trip)LRANGE key 0 -1(this PR)load_policyruns when an enforcer is created and again every time policy is reloaded (watcher callback, explicit reloads), so applications pay this cost repeatedly.Consistency.
LLEN+LINDEXis not atomic. If another process removes a rule (LREM) while the loop is running, the list shrinks, the lastLINDEXcalls returnNone, andjson.loads(None)raises aTypeError; inserts or removals in the middle can also skip or duplicate rules.LRANGEis a single command, so it reads a consistent snapshot of the list.This also brings the Python adapter in line with the Go redis-adapter, whose
LoadPolicyalready reads the list with a singleLRANGE.Tests
Added
test_load_policy_single_round_trip: it reloads the policy withlindexpatched to fail and checks thatlrangeis called exactly once with(key, 0, -1)and that the loaded policy enforces correctly. It fails on currentmaster(LINDEX used) and passes with this change. The full suite passes locally (10 tests), andblack --checkis clean.