feat(export): full data export — ZIP download + S3 presigned upload#1475
Merged
Conversation
Adds the ZIP-streaming library the upcoming "Full Data Export" feature uses to produce the bundle. Locked at 3.2.2.
Lifts the data-export subsystem out of nothing into the five classes
the plan calls for: BundleWriter (orchestrator), EntityExporter
(per-entity streaming SELECT + JSONL emission), ManifestBuilder
(pre-flight pass + bundle cap math), S3PresignedUploader (HTTPS
PUT with single-flight test override), ExportError (typed exception
with code constants), plus a readonly Manifest DTO.
Producer contracts pinned per the plan:
- null-for-absent (never empty string, never omitted)
- decimal-string Steam64 + preserved Steam2 via validate-then-convert
through SteamID::isValidID + toSteam64/toSteam2
- unix-seconds integers for every timestamp
- forbidden columns (admins.{password,validate,attempts,lockout_until},
servers.rcon, settings.{smtp.pass,telemetry.instance_id}) dropped
at the SQL projection level so a future code-path that bypasses
the row builder still cannot leak them
- JSON_INVALID_UTF8_SUBSTITUTE on every encode site to defend against
pre-#1108 Latin-1-on-utf8 truncation in player names
- 4 GiB cap minus 64 MiB safety margin enforced both pre-flight (via
ManifestBuilder) and at-write (via BundleWriter's running-byte gate)
Entry point (web/export.php) lands in the next commit. PHPStan: clean.
Top-level entry point that lifts the data-export bundle out of the
admin page-handler flow. Lives at panel-root for the same reason
`exportbans.php` / `getdemo.php` do: the wire format is binary
(`Content-Type: application/zip`), so the JSON dispatcher's
`Content-Type: application/json` envelope doesn't fit and there's
no clean extension point for a streaming binary handler.
Hardening (each justified in the file's docblock):
- POST-only (GET / HEAD → 405; the form posts, direct GET is a probe)
- CSRF gate via `CSRF::rejectIfInvalid()` (state-changing PII surface;
a stale tab MUST NOT trigger an export)
- Owner-only via `HasAccess(WebPermission::Owner)` (navbar / palette
is UX gating; this is the load-bearing security check)
- `@set_time_limit(0)` + `@ini_set('memory_limit', '256M')` +
`ignore_user_abort(true)` (shared-host safe — the `@` swallows
`disable_functions = set_time_limit,ini_set` warnings)
- Output-buffer drain + `X-Accel-Buffering: no` + Apache's `no-gzip`
so streamed bytes flow directly without re-buffering
Two delivery modes:
- ZIP: streams `php://output` straight to the wire; `flush()` after
each entry keeps the browser's download progress bar moving;
`application/zip` + `Content-Disposition: attachment` + aggressive
`Cache-Control: no-store` so the dynamic response can't leak
across operators
- S3: stages under `SB_CACHE/exports/<bundle-id>.zip` (S3 presigned
PUT requires `Content-Length` upfront — can't be set on a
chunked-streaming response), then `S3PresignedUploader::upload`
PUTs to the operator URL; a `register_shutdown_function` wipes
the staging file before the build even runs so a mid-build fatal
still cleans up
Error handling: only `ExportError` is caught — anything else
(real DB outage, memory exhaustion, regression in the writer)
propagates to the dispatcher's generic 500 so the stack trace
lands in the audit log via the project's error handler. Catching
`Throwable` blanket would mask real bugs behind a generic toast.
Every reachable terminal branch (success + every error class) emits
an audit-log entry that includes the operator's `aid`, the
`bundle_id`, and (on failure) the `ExportError::code()` value so
operators can correlate the painted toast with the audit row.
The `Variable \$userbank might not be defined` PHPStan note lands
in `phpstan-baseline.neon` matching the existing `exportbans.php`
shape — same root cause (PHPStan can't follow $userbank from
init.php's top-level scope across a require_once).
Admin page handler + form template land in the next commit.
…ort.tpl form Adds the owner-only admin landing page for the Full data export feature. Mirrors the back-link-only AdminTabs shape used by admin.email.php / admin.rcon.php (single section, no sub-routes). The page handler re-runs the ManifestBuilder pre-flight (counts only — no JSONL emission) so the form can surface the row counts + the demo-on-disk byte budget + the 4 GiB cap status before the operator commits to a stream. Toast feedback consumed from ?result=success&bid=… or ?result=error&code=… set by web/export.php's S3 mode redirect. Both submit buttons ride the window.SBPP.setBusy triple (data-loading + aria-busy + disabled) via the local fallback wrapper per AGENTS.md "Loading state on action buttons" — neither branch flips busy back to false because both tear the page down (ZIP streams the response body, S3 mode 302-redirects after the upload completes).
…alette Adds the discovery surfaces for the Full data export admin page. All three are gated on WebPermission::Owner only — the feature is intentionally NOT delegated via a per-flag permission today because every PII category in scope (admin emails, IP addresses, Steam IDs, unban reasons) is owner-territory by default; a future split into per-category exports can revisit this without breaking the v1 manifest contract. No web.json edit; no WebPermission::ExportData case; no api-contract regen (the entry point is web/export.php — a top-level streaming script, not a JSON action — so the Actions/Perms surfaces are unchanged).
Five PHPUnit files + one Playwright spec covering the data export
wire-format contract end-to-end:
- ExportBundleWriterTest (integration) — opens the streamed ZIP via
ZipArchive, asserts manifest.json is the first entry, every entity's
manifest row_count matches its JSONL line count, demos are STORE
compression (already ZIP-style compressed binary), every Steam ID
is a quoted decimal string never a JSON number, every timestamp is
an integer, no forbidden column value leaks. 393 assertions.
- EntityExporterTest (unit) — forbidden columns absent per entity,
SteamID always quoted decimal, mute_kind enum, BanType×BanRemoval
state derivation matrix, null-for-absent contract, log level
derivation, timestamps as int.
- ManifestBuilderTest (unit) — 4 GiB cap math, pii_policy block,
format_version pin, UUIDv4 shape, created_at as unix-seconds int,
exceeds_cap flag exposure.
- AdminExportPermissionTest + AdminExportRuntimePermissionTest —
three permission gates per AGENTS.md "Public auth surfaces" sister
pattern: anonymous → 403, non-owner with every other web flag →
403, owner → allowed. Form GET access mirrors the gate.
- S3PresignedUploaderTest — scheme guard (http:// rejected), URL
parser guard (garbage rejected), happy path with mocked transport
via _setHttpTransportForTests, 200/201/204 all accepted, non-2xx
surfaces as s3_put_failed with response body truncated to 2 KiB,
byte-stable error code constants.
- data-export.spec.ts (Playwright, 6 tests across chromium + mobile
chromium) — log in as seeded admin, navigate to ?p=admin&c=export,
click Export as ZIP, page.waitForEvent('download'), parse with
jszip, assert manifest-first contract + row_counts shape + 405 on
direct GET.
The paired ManifestBuilder.php change filters the settings row count
through the same EntityExporter::FORBIDDEN_SETTING_KEYS allowlist the
exporter applies — without it the manifest's row_counts.settings was
N + |forbidden_keys| while the JSONL line count was N, which
ExportBundleWriterTest's "row_counts agrees with JSONL line counts"
assertion caught immediately.
…rt.mdx Three documentation surfaces for the Full data export feature: - AGENTS.md — new "Full data export" Conventions block (synchronous in-request execution model, ZIP-vs-S3 delivery asymmetry, the ignore_user_abort + @set_time_limit(0) + @ini_set memory_limit shared-host triplet, manifest-first contract, null-for-absent + Steam64-as-decimal-string + unix-seconds wire contracts, the hard-coded forbidden columns list with the "never relax" rationale, owner-only access with the per-flag delegation deferral, no schema change + no JSON handler rationale, persistent error toasts, HTTPS-only S3 scheme guard) + new "Where to find what" rows pointing at the entry point / page handler / View / template / presigned-URL workflow + a new sync-docs table row covering the Export/** + admin.export.* + page_admin_export.tpl surfaces. - ARCHITECTURE.md — new "Data export" subsystem section under the panel's request-lifecycle area describing the Sbpp\Export\* classes, the entry-point dispatch, the manifest-first contract, and the 4 GiB cap. The web/ Directory layout listing gets web/export.php (in the panel-root scripts list) + the Export/ namespace entry. - docs/src/content/docs/configuring/data-export.mdx — operator- facing. What's in the bundle (entity list + spec contract: ZIP 2.0 / 4 GiB cap / manifest-first / JSONL / Steam64-as-string / unix-seconds), two delivery modes (ZIP download vs S3 upload), worked presigned-URL generation examples for AWS S3 / Cloudflare R2 / MinIO, the 4 GiB cap, audit logging, and a security-model summary (owner-only gate, single-use presigned URLs, ≤1h expiry, never paste the URL into chat / logs) + a troubleshooting card per ExportError code (cap_exceeded / s3_put_failed / presign_invalid_scheme / presign_invalid_url / disk_write_failed / disk_full). New sidebar entry in docs/astro.config.mjs under the "Configuring" group, next to the announcements page.
Reviewer flagged six issues against the worker-produced export feature. All six addressed in-place; runtime contract / streaming semantics preserved. - **B1 (BLOCKER, security)**: drop `admins.srv_password` from the export. The panel stores this field cleartext (varchar(128) with stringwise compare in `account.php`), so exporting it lets a bundle recipient authenticate as the admin against every game server they manage. Add the column to `FORBIDDEN_ADMIN_COLUMNS` so the manifest's `pii_policy.password_hashes="never"` attestation stays truthful. Regression test seeds a distinctive marker AND asserts the constant carries every credential-class field. - **H1 (HIGH, OOM)**: switch `BundleWriter::writeEntity` from in-memory concatenation inside `addFileFromCallback` to `php://temp/maxmemory:8M` spill via `addFileFromStream`. Pre-fix the writer buffered the entire entity into a single PHP string, defeating the documented streaming contract and risking OOM on installs with multi-hundred-MB audit logs. Spill threshold matches the 256 MiB runtime hardening ceiling minus PDO/Smarty/Composer headroom. Regression test seeds >15 MiB of comments and asserts peak memory delta stays under 64 MiB. - **M1 (MEDIUM, UX)**: BundleWriter now accepts an optional `$outputHandle` parameter; in s3 mode (build-to-disk path) the cap counter snaps to `fstat($handle)['size']` after each entry for exact compressed-byte tracking. Pre-fix the counter advanced by uncompressed bytes against a compressed limit, prematurely tripping `CAP_EXCEEDED` on JSONL bundles (DEFLATE typically compresses 3-8x). zip mode (php://output, unseekable) keeps the conservative uncompressed estimate as the documented fallback — over-counting is the safe direction (a premature error message beats silently exceeding ZIP 2.0's 4 GiB ceiling). Regression test asserts `bytesWritten()` matches on-disk size byte-for-byte. - **L1**: emit `LogType::Warning` audit-log entries on the `mode_invalid` and presign-URL-malformed rejection branches so the painted toast correlates with an audit trail. URL length is logged (not the raw URL — operator-typed presign params can carry sensitive signature data even on the failure branch). - **L2**: include `bundle_id` in the disk-write-failed and fopen-failed log entries so the audit row can be correlated with the manifest-build event upstream. - **L3**: empty-state CTA in `page_admin_export.tpl` pointed at the non-existent `?p=admin&c=bans§ion=add-ban` Demo column; retarget to the public banlist where per-row demo affordances actually live.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add a one-shot full data export feature to the panel. Owner-only
(no new permission flag — strictly
ADMIN_OWNER). Two deliverymodes:
Content-Type: application/zipbodystraight from
php://outputthroughmaennchen/zipstream-php.No staging file; the browser's progress bar moves in real time
via per-entry
flush()calls.tempfile under
SB_CACHE/exports/, then PUTs it via cURL to theoperator-supplied presigned URL. The tempfile is registered for
register_shutdown_functioncleanup so a mid-build fatal stillwipes the staging file.
Both modes honour the bundle spec (manifest-first, per-
entity JSONL, STORE-mode demo entries, SteamID64-as-decimal-string,
unix-seconds integer timestamps, ZIP 2.0 with 4 GiB hard cap minus
64 MiB safety margin,
pii_policy.password_hashes="never"attestation).
Runtime model is synchronous in-request with PHP hardening
(
@set_time_limit(0),@ini_set('memory_limit', '256M'),ignore_user_abort(true), ob_buffer drain,X-Accel-Buffering: no,Apache
no-gzip). Works on non-FPM hosts, shared hosting, andbehind reverse proxies without requiring a background-worker
subsystem the panel doesn't have today.
Security highlights
ADMIN_OWNER-only. Navbar + command palette + page-builderroute + page-handler defence-in-depth + top-level entry-point
gate all check the same permission.
FORBIDDEN_ADMIN_COLUMNS(password,validate,attempts,lockout_until,srv_password) NEVER projected from the SELECT.The list is the load-bearing contract that keeps the manifest's
pii_policy.password_hashes="never"attestation truthful.srv_passwordspecifically was added in the adversarial-reviewfix sweep (B1): the panel stores SourceMod admin server-login
passwords cleartext, exporting them would let a bundle recipient
authenticate against every game server the admin manages.
FORBIDDEN_SERVER_COLUMNS(rcon) NEVER projected — RCON isthe keys-to-the-game-server credential.
FORBIDDEN_SETTING_KEYSfiltered at the SQL WHERE level so afuture post-fetch helper can't re-add them.
every error code), tagged with
bundle_idso an operator cancorrelate a painted toast with the audit row.
Adversarial review findings (all addressed)
A read-only adversarial subagent reviewed the worker's first pass
and surfaced six issues; the final commit on this branch (
ab167a41)addresses all of them:
admins.srv_password.BundleWriter::writeEntitynow spills tophp://temp/maxmemory:8Minstead of concatenating the fullentity into a PHP string.
fstat($outputHandle)['size']in s3 mode for exact compressed-byte tracking; zip-mode keeps
the conservative uncompressed-byte estimate (over-counting is the
safe direction).
Log::addonmode_invalid+ presign-URL-malformedrejection branches.
bundle_idin disk-write-failed + fopen-failed log entries.?p=admin&c=bans§ion=add-banDemo column to the public banlist.Tests
tests/unit/EntityExporterTest.php— per-entity column rules,forbidden-column non-leak, SteamID-as-string contract, mute-kind
derivation, bans state matrix, unix-seconds timestamps. The
forbidden-admin-column test now seeds a distinctive
srv_passwordmarker AND asserts the constant list itself includes every
credential field.
tests/integration/ExportBundleWriterTest.php— full pipeline(ManifestBuilder → BundleWriter → on-disk ZIP), structural
contract assertions (manifest-first, row-count agreement,
STORE-mode demos, no bare-number SteamIDs, no leaked credentials).
Two new regression tests:
testWriterBytesWrittenMatchesOnDiskSizeInS3Modepins M1'sexact-byte tracking.
testWriterDoesNotBufferLargeEntityIntoMemorypins H1's spillbehaviour by seeding >15 MiB of comments and asserting peak
memory delta stays under 64 MiB.
tests/unit/ManifestBuilderTest.php— manifest shape, cap math.tests/integration/AdminExportPermissionTest.php+AdminExportRuntimePermissionTest.php— HTTP entry gates(CSRF, owner-only, method gate).
tests/integration/S3PresignedUploaderTest.php— cURL stub paths(success, network failure, non-2xx response, oversized body).
tests/e2e/specs/flows/data-export.spec.ts— Playwright end-to-endzip-mode download + jszip-based bundle inspection.
Documentation
AGENTS.md— "Full data export" conventions block + "Where tofind what" rows for the entry point, writer, exporter, manifest.
ARCHITECTURE.md— "Data export" subsystem section.docs/src/content/docs/configuring/data-export.mdx— operator-facing guide covering both modes, the bundle wire format, the
pii_policy attestation, and S3 presign-URL setup.
Test plan
PluginVersionResolveTestcases (missingsbpp_version.incin the dev container; unrelated to this branch, confirmed
reproducing on stashed-clean main)