Conversation
…the answer `matches` evaluated all six fields over every row, every time. Three of those passes are avoidable from the selection alone, before a row is read: * a field NO value of which renders the term contributes nothing, and when that holds for every field the answer is empty without touching the column; * a field EVERY value of which renders the term means every present row matches whatever the other fields hold, so the answer IS the present mask; * once the accumulated mask is all true, no remaining field can add to it. Within a field, selecting rows by comparing against each chosen value replaces gathering through a lookup table while few values are chosen. `_COMPARE_UPTO` is the measured crossover on a 30M-row column, not a round number, and both sides of it are reachable. The answer is unchanged -- this only stops computing parts of it that cannot matter. The differential suite (the render as oracle, six shapes x five zones x forty terms) covers that, and a new class pins the ROUTE, because a version that quietly stopped taking these shortcuts would still be exact and every differential test would still pass. Each shortcut is mutation-verified: removing it, or moving the threshold to either extreme, fails a test that names it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017ropeBMLJUuy6ViYwy15ud
This was referenced Sep 23, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
DatetimeSearchIndex.matchesevaluated all six rendered fields over every row on every call.Three of those passes are decidable from the selection alone, before a row is read:
every field, the answer is empty without touching the column;
other fields hold, so the answer is the present mask;
Within a field, comparing against each chosen value costs one pass per value while a lookup-table
gather costs one pass regardless.
_COMPARE_UPTO = 11is where those meet, measured.The answer is unchanged. This only stops computing parts of it that cannot matter.
Why it matters:
2024was the easy caseMeasured at 30M before this change — the term a live-typing session opens with is the expensive one:
22020240,1,5all land at ~490 ms. Every session's first keystroke is a single character.Measured (dgx, under the perf lock, 3 interleaved arms each, exactness asserted per term)
search_any_maskend to end at 30M rows:22009520242→20→202→2024→20241Every term now bottoms out at ~178 ms, and that floor is the cache-key digest (blake2b over
240 MB, 173.4 ms measured, 72.7% of a keystroke) — the match has left the critical path. That
digest is a separate, API-level question and is untouched here.
No regression on small frames (faster or equal at 100 → 1M rows); the only marginal cell,
5at 100 rows, reverses at 1,000 rows.Tests pin the ROUTE, because the answer cannot distinguish these
A version that quietly stopped taking the shortcuts would still be exact, and the existing
differential suite (the render as oracle, 6 shapes × 5 zones × ~40 terms) would still pass. The
new class counts the whole-column passes
matchesperforms.Every shortcut is mutation-verified:
test_a_field_every_value_of_which_matches_reads_no_rowstest_a_term_no_value_can_contain_reads_no_rows_COMPARE_UPTO = 0(always gather)test_one_selected_value_is_a_comparison_not_a_gather_COMPARE_UPTO = 100(always compare)test_many_selected_values_switch_to_a_gatherMy first version of those tests counted only
equal/takeand did not catch the empty fastpath — with no live field the loop never runs, so the counts stayed 0 either way while the code
still allocated a scratch buffer and did a final
logical_and. Widened to countlogical_and.Gates, run locally before the first push
ruff/lint · mypy (352 files) · pyright guard (230 grandfathered, no growth) · comment-density ·
changed-line coverage 28/28 = 100% ·
graphistry/tests/compute/gfql/5,838 passed.The comment-density guard caught a
'slower'in a test docstring and a 3-line#:run; bothrewritten rather than baselined.
🤖 Generated with Claude Code
https://claude.ai/code/session_017ropeBMLJUuy6ViYwy15ud
Stack
This is PR 1 of 3 (base) in a stack that lands together, in order: #2110 kernel → #2111 sha256 cache key → #2112 thrash detector + budget. Each is rebased onto the one before it, so the diff shown is only this PR's change. Merge #2110 first; GitHub retargets the next base automatically. Combined measurements are in #2112.