Conversation
lmeyerov
force-pushed
the
fix/docs-revendor-bench-numbers
branch
from
September 25, 2026 08:17
fa79338 to
7ef80c1
Compare
…PU PageRank ratio
The vendored `docs/source/_data/gfql_benchmarks.json` carried GraphBench q1-q9 runs measured
2026-07-26. The publication policy allows 60 days; today is day 61, so every CI lane that runs
`docs/test_bench_numbers.py` -- test-minimal-python, test-docs, the Read the Docs build -- fails
on master for any change. pyg-bench re-measured those lanes on 2026-09-17
(`graphbench-q1q9-{20k,100k}-master-f283a305e-20260917`); this vendors that artifact. The
contract file is unchanged (identical to pyg-bench `manifests/docs-numbers.contract.json`).
Two documented numbers change status with it, and the docs follow the artifact rather than
the other way round:
* `pagerank.gplus.gfql_gpu_vs_gfql_cpu` is no longer published, and `pagerank.gplus.gfql_gpu`
is now diagnostic-only, with pyg-bench's disclosure: the GPU arm's selected-node set differs
from the CPU arm's on gplus (Jaccard 0.91065 against a 0.95 gate), so it is a direct pipeline
time, not a comparison. `benchmark_filter_pagerank.rst` and its chart stop claiming a GPlus
GPU-vs-CPU speedup and present the GPU time as `:bench-diag:`. Twitter's cells are unchanged
upstream and untouched here.
* `graphbench.{20k,100k}.q8.{pandas,polars}` were promoted from diagnostic to published
results upstream; `performance.rst` upgrades their role from `:bench-diag:` to `:bench:`.
This withdraws a published comparison, so it is for review rather than self-merge. Note for the
publisher: `snb-is3-single-query-20260727` is at 60 days in both this artifact and upstream and
crosses the same policy tomorrow.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017ropeBMLJUuy6ViYwy15ud
lmeyerov
force-pushed
the
fix/docs-revendor-bench-numbers
branch
from
September 25, 2026 08:25
7ef80c1 to
69c6a0d
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why master is red today
docs/source/_data/gfql_benchmarks.json— the vendored copy of pyg-bench'spublished/docs-numbers.json— carried GraphBench q1–q9 runs measured 2026-07-26. The publication policy allows 60 days; 2026-09-25 is day 61, sodocs/test_bench_numbers.py::test_no_published_number_is_stalefails in every lane that runs it —test-minimal-python,test-docs, the Read the Docs build — on master, for any change. Found because #2113 touched a path routed into those lanes; #2110–#2112 ran on days 59–60 and passed.What this does
pyg-bench already re-measured those lanes on 2026-09-17 (
graphbench-q1q9-{20k,100k}-master-f283a305e-20260917, pyg-bench7057a170). This vendors that artifact. The contract file is byte-identical to pyg-bench'smanifests/docs-numbers.contract.json. Chart SVGs regenerated with the documentedgfql_bench_charts.py --write.docs/test_bench_numbers.py: 26/26.Two documented numbers change status — the docs follow the artifact
Withdrawn (for your review — this removes a published comparison):
pagerank.gplus.gfql_gpu_vs_gfql_cpuis no longer published upstream, andpagerank.gplus.gfql_gpuis diagnostic-only, with pyg-bench's disclosure:benchmark_filter_pagerank.rstand its chart stop claiming a GPlus GPU-vs-CPU speedup and show the GPU time as:bench-diag:with that reason. Twitter's cells are unchanged upstream and untouched.Promoted (mechanical):
graphbench.{20k,100k}.q8.{pandas,polars}moved from diagnostic to published upstream;performance.rstupgrades them from:bench-diag:to:bench:.For the publisher
snb-is3-single-query-20260727is at 60 days in both this artifact and upstream and crosses the same policy tomorrow. Needs a re-measure, a drop, or a policy decision in pyg-bench.Not self-merged: it withdraws a published claim.
🤖 Generated with Claude Code
https://claude.ai/code/session_017ropeBMLJUuy6ViYwy15ud