chore: add non-cached benchmark to find the price of costs control#2929
Merged
Conversation
We already have two benchmark that use router with caching enabled.
That does not make a good indicator of the delta when we enable
the cost control feature.
This PR adds two benchmarks where caching is disabled and we could see
the real price of Cost Control:
$ go test -bench=BenchmarkSequentialBigNotCached -benchmem -count=20 -run=^$ . > new.txt
$ benchstat -col .name new.txt
goos: darwin
goarch: arm64
pkg: github.com/wundergraph/cosmo/router-tests/protocol
cpu: Apple M4 Max
│ SequentialBigNotCached │ SequentialBigNotCachedCostControl │
│ sec/op │ sec/op vs base │
*-14 1.092m ± 1% 1.125m ± 0% +3.05% (p=0.003 n=20)
│ SequentialBigNotCached │ SequentialBigNotCachedCostControl │
│ B/s │ B/s vs base │
*-14 2.952Mi ± 1% 2.871Mi ± 0% -2.75% (p=0.003 n=20)
│ SequentialBigNotCached │ SequentialBigNotCachedCostControl │
│ B/op │ B/op vs base │
*-14 998.0Ki ± 0% 986.5Ki ± 0% -1.15% (p=0.000 n=20)
│ SequentialBigNotCached │ SequentialBigNotCachedCostControl │
│ allocs/op │ allocs/op vs base │
*-14 11.15k ± 0% 11.22k ± 0% +0.66% (p=0.000 n=20)
It shows that costs adds ~80 allocations, 3% of CPU time
when caching is disabled.
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Claude Code Review
This repository is configured for manual code reviews. Comment @claude review to trigger a review and subscribe this PR to future pushes, or @claude review once for a one-time review.
Tip: disable this comment in your organization's Code Review settings.
Router image scan passed✅ No security vulnerabilities found in image: |
Contributor
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughPause benchmark timing around assertions and run cache-metric checks off-timer; reconfigure sequential cost-control benchmarks to expose headers and validate estimated/actual cost per iteration. Add a k6 big-response benchmark script, add a local ChangesBenchmark Measurement and Cost-Control Validation
Load Test Script, Schema, and Docs
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. Comment |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #2929 +/- ##
==========================================
- Coverage 66.38% 66.17% -0.22%
==========================================
Files 258 258
Lines 27539 27539
==========================================
- Hits 18282 18223 -59
- Misses 7805 7853 +48
- Partials 1452 1463 +11 🚀 New features to boost your workflow:
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@router-tests/protocol/integration_test.go`:
- Around line 1603-1614: The benchmark loop in Benchmark... (the for b.Loop()
block that issues xEnv.MakeGraphQLRequestOK with bigEmployeesQuery) is measuring
assertion overhead because the body-size check and header assertions run while
the timer is active; stop the benchmark timer before validating res.Body and the
header checks (b.StopTimer()), perform the len(res.Body) validation and the
require.Equal checks for core.CostEstimatedHeader and core.CostActualHeader,
then restart the timer with b.StartTimer() so only request/processing time is
measured; locate the assertions inside the for b.Loop() in the benchmark and
wrap them with b.StopTimer()/b.StartTimer() around the validations.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: c7ac9107-18df-4b43-8300-32827dd74962
📒 Files selected for processing (2)
router-tests/protocol/integration_test.gorouter-tests/testenv/testenv.go
devsergiy
approved these changes
Jun 9, 2026
…_cached_benchmarks_for_costs
Contributor
There was a problem hiding this comment.
Actionable comments posted: 4
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@router/bench-big-response.js`:
- Line 77: Replace the substring check in the 'no graphql errors' response
assertion with proper JSON parsing: in the arrow function keyed 'no graphql
errors' parse r.body (try/catch) into an object and then assert r.status === 200
and (parsed.errors === undefined || (Array.isArray(parsed.errors) &&
parsed.errors.length === 0)); if JSON.parse throws or parsed.errors indicates
problems, return false (or DISCARD || false) so we don't misclassify responses.
Ensure you still short-circuit with the existing DISCARD condition and reference
the same function/arrow expression handling r.
- Around line 71-79: Wrap the HTTP request and subsequent check calls inside a
try/catch block in the default exported function: perform const res =
http.post(URL, body, { headers }) and the existing check(...) inside try, and in
catch catch any runtime error, log the error (console.error or a logger) and
report a controlled failed check (e.g., call check with a single failing
assertion so the iteration is recorded as failed instead of throwing). Ensure
references to res, http.post, check, DISCARD, URL, body and headers are guarded
so the catch path does not rethrow.
In `@router/docs/Profiling.md`:
- Line 110: Replace the non-ASCII “smart” single quotes around the pprof URL in
the documentation line containing go tool pprof
‘http://localhost:6060/debug/pprof/profile?seconds=5’ with plain ASCII quotes
(either ' or ") so the shell can execute the command; update the documentation
text to use go tool pprof 'http://localhost:6060/debug/pprof/profile?seconds=5'
(or with double quotes) wherever that exact pprof invocation appears.
- Around line 33-51: Add a language identifier to the unlabeled fenced code
block in router/docs/Profiling.md (the block that starts with "✓ is status 200")
to satisfy markdownlint MD040; update the opening fence from ``` to ```text so
the block becomes ```text ... ``` and leave the content unchanged.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 88adac9b-ce82-43cd-b15a-abc75c0910b8
📒 Files selected for processing (3)
router/__schemas/graph.yamlrouter/bench-big-response.jsrouter/docs/Profiling.md
✅ Files skipped from review due to trivial changes (1)
- router/__schemas/graph.yaml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
We already have two benchmark that use router with caching enabled. That does not make a good indicator of the delta when we enable the cost control feature.
This PR adds two benchmarks where caching is disabled and we could see the real price of Cost Control:
It shows that Cost Control adds ~80 allocations, 3% of CPU time when caching is disabled.
Summary by CodeRabbit