GitHub - Kodiqa-Solutions/VaultS3: Lightweight, S3-compatible object storage server with built-in web dashboard. Single binary, low memory, encryption at rest.

Lightweight S3-compatible object storage. Single binary, <80MB RAM, built-in dashboard.

Quick Start · Features · S3 API · Dashboard · CLI · Docker

Why VaultS3?

MinIO stripped the admin console from its Community Edition in mid-2025 and archived the open-source repository in February 2026, it is no longer actively maintained, and full management now requires the paid AIStor tier. SeaweedFS is capable but spreads across multiple components (master / volume / filer). It does have a web admin UI (weed admin) and a working FUSE mount. Garage still has no object versioning, WORM/object-lock, or event notifications.

VaultS3 keeps everything in one self-contained binary under 80 MB RAM:

	VaultS3	MinIO	SeaweedFS	Garage
RAM (small deploy)	<80 MB	512 MB+	50-200 MB	50-150 MB
Single binary	Yes	Yes	No	Yes
Web dashboard	Built-in	Paid¹	Yes	No
Open-source maintained	Yes	Archived¹	Yes	Yes
Raft clustering	Yes	Yes	Yes	Yes
Erasure coding	Yes	Yes	Yes	No
Active-active replication	Yes	Yes	No	No
FUSE mount	Built-in	No	Yes	No
Full-text search	Yes	No	No	No
Version diff/tags	Yes	No	No	No
Lambda triggers	Yes	No	No	No
Virus scanning	Yes	No	No	No
Backup scheduler	Yes	No	No	No

¹ MinIO removed the admin console from its Community Edition in 2025 and archived the open-source repository in February 2026. Full management now requires the paid AIStor product. This comparison reflects publicly available information as of June 2026, please open an issue if a cell is out of date.

make build && ./vaults3
# Server at http://localhost:9000
# Dashboard at http://localhost:9000/dashboard/

Production Readiness

VaultS3 is honest about what's battle-tested versus still maturing. Pick the lane that matches your risk tolerance:

Path	Maturity	Notes
Single-node (S3 API, versioning, IAM, dashboard)	✅ Stable	The default deployment. Broad test coverage. Runs in production today.
Erasure coding (single-node, multi-disk)	✅ Stable	Reed-Solomon encode/reconstruct and the background healer have fault-injection tests (lose disks → reconstruct → heal).
Tiering & backup	✅ Stable	Hot/cold migration, transparent promotion, and full/incremental backup are tested. Restore is a manual file copy.
Multi-node Raft clustering	🟡 Beta	Metadata writes replicate via Raft consensus (writes accepted on any node via leader-forwarding), object data is placed/served by a live-membership hash ring, inter-node calls are authenticated, and on Kubernetes the cluster auto-forms (leader bootstrap + auto-join + self-heal). Validated end-to-end on a real 3-node cluster: leader election & failover, node recovery with catch-up, cross-node reads, and concurrent load, including a 10,000-write list-then-write-then-read workload behind a gateway with a node restarted mid-run. Still operationally newer, not yet stress/scale/multi-region hardened, so validate against your workload before trusting it as the only copy of critical data.
Active-active replication	🟡 Beta	Vector-clock conflict resolution is unit-tested. The cross-site sync worker is less exercised in the wild.

Recommendation: run single-node (optionally with erasure coding across local disks) for production data you care about, and treat clustering/active-active as advanced opt-in features you validate first. Always keep an independent backup. See the Scaling & Operations Guide for redundancy layering and recovery runbooks, and the Benchmarks guide for a reproducible way to measure throughput and RAM on your own hardware.

Features

S3-compatible API: Works with any S3 client (AWS CLI, mc, boto3, minio-js, s3fs), including directory-marker objects (folder/ keys) so tools that represent folders as zero-byte objects work correctly
Single binary: One file, no runtime dependencies, no Docker required
Low memory: Targets <80MB RAM (vs MinIO's 300-500MB)
BoltDB metadata: Embedded key-value store, no external database needed
S3 Signature V4: Standard AWS authentication
AES-256-GCM encryption at rest: SSE-S3 (static key) and SSE-KMS (HashiCorp Vault or local key provider) encryption modes
Per-bucket encryption keys: For bucket-per-tenant setups, each bucket can be encrypted with its own key that is not shared with other tenants (or opt out and stay plaintext). Envelope encryption (master KEK wraps a per-bucket data key). Opt in per bucket via PUT /{bucket}?encryption or the dashboard. Supports key rotation and crypto-shredding. Enable with encryption.per_bucket: true, see design doc
SSE-C (customer-provided keys): Operator-blind per-object encryption: the client supplies the key per request (x-amz-server-side-encryption-customer-*). The server encrypts/decrypts with it and stores only the key's MD5, never the key
Bucket policies: Public-read, private, custom S3-compatible JSON policies. Supports the standard AWS Principal forms ("*", {"AWS": "*"}, {"AWS": ["*"]}), wildcard actions, explicit Deny precedence, and per-bucket Resource matching. Granting s3:GetObject to everyone makes objects publicly readable and s3:ListBucket makes the listing public, as separate permissions; bucket sub-resources (?policy, ?acl, ...) always require authentication. Public Access Block (BlockPublicPolicy / RestrictPublicBuckets) overrides any policy and blocks anonymous access
Quota management: Per-bucket size and object count limits
Rate limiting: Token bucket rate limiter per client IP and per access key to prevent abuse
S3 Select: Execute SQL queries on CSV, JSON, and Parquet objects without downloading the full file
Multipart upload: Full lifecycle (Create, UploadPart, UploadPartCopy, Complete, Abort, ListUploads, ListParts)
Bucket tagging: S3-compatible tag sets with PUT/GET/DELETE
Bucket/Object ACL: S3-compatible ACL responses (GET/PUT)
Multiple access keys: Dynamic key management via BoltDB
Object tagging: Up to 10 tags per object
Range requests: Partial content downloads (206 responses)
Copy object: Same-bucket and cross-bucket copies
Batch delete: Multi-object delete with XML body
Virtual-hosted style URLs: bucket.domain/key in addition to path-style
Bucket default retention: Set default GOVERNANCE or COMPLIANCE retention on a bucket, auto-applied to new objects
Per-bucket Prometheus metrics: Request counts, bytes in/out, and errors with bucket labels at /metrics
Prometheus metrics: /metrics endpoint with storage, request, and runtime stats
Presigned URLs: Pre-authenticated URL generation
Web dashboard: Built-in React UI at /dashboard/ with home overview page, file browser (grid or list layout with file-type icons, sortable columns, pagination, file preview, metadata panel, version history panel with diff viewer/rollback/tagging, multi-select, bulk delete, bulk zip download, breadcrumb navigation), drag-and-drop file and folder upload (streamed straight to storage so large files work, with subfolder structure preserved), copy-to-clipboard buttons, access key management, activity log, storage stats with auto-refresh, read-only settings viewer, IAM management, audit trail viewer (sortable, paginated), search (sortable, paginated), notifications, replication status, lambda triggers, backup management, bucket config (versioning toggle with status indicator, lifecycle editor, CORS editor), keyboard shortcuts (/ search, ? help), toast notifications (success/error/info), dark/light theme, language switcher, collapsible sidebar, remember-me sign-in, responsive layout
Dashboard in your language: The Web UI ships English, German, French, and Simplified Chinese, picked automatically from the browser's language and switchable from the top bar (the choice is remembered). Adding a language is one JSON file and no code, see Translating the dashboard
Health checks: /health (liveness) and /ready (readiness) endpoints for load balancers and Kubernetes
Buckets on first start: Declare the buckets a deployment needs (VAULTS3_DEFAULT_BUCKETS=app-data,backups, storage.default_buckets, or the chart's defaultBuckets) and the missing ones are created while the server starts, so a container needs no init container or one-off S3 client call to become usable. Existing buckets are never touched, and an invalid name or a failed create stops startup instead of coming up quietly incomplete
Graceful shutdown: Drains in-flight requests on SIGTERM/SIGINT with configurable timeout
TLS support: Optional HTTPS with configurable cert/key paths
Separate dashboard port: Optionally serve the Web UI + its API on a dedicated port (server.console_port, e.g. 9001) apart from the S3 API, so each can have its own firewall rules / TLS / reverse proxy (MinIO-style)
Reverse-proxy subpath: Host the whole app under a subpath (server.base_path, e.g. /vaults3) so both the dashboard (https://example.com/vaults3/dashboard/) and the S3 API work behind it — asset URLs and SPA routes are rewritten at serve time, and S3 SigV4 signatures are verified against the original (pre-strip) path (optionally auto-detect the subpath from the proxy's X-Forwarded-Prefix with server.trust_forwarded_prefix, off by default since that header is client-supplied)
Object versioning: Per-bucket versioning with version IDs, delete markers, version-specific GET/DELETE/HEAD
Object locking (WORM): Legal hold and retention (GOVERNANCE/COMPLIANCE) to prevent deletion
Lifecycle rules: Per-bucket object expiration (auto-delete after N days) and aborting incomplete multipart uploads after N days (S3 AbortIncompleteMultipartUpload, reclaims the parts left by killed/failed clients), run by a background worker
Zstandard compression: Transparent compress-on-write, decompress-on-read with zstd (better ratio and speed than gzip). Reads stream the decoder, so GET time-to-first-byte stays flat regardless of object size (no whole-object buffering). Objects written by older gzip builds are still read transparently (codec auto-detected by magic number)
Small-file packing (experimental): Optionally pack objects up to a size threshold into large append-only volume files (each as an independent zstd frame) with byte-offset locations in BoltDB, plus background dead-space compaction (POST /api/v1/compact), avoids the per-file overhead (inodes, syscalls, disk blocks) of millions of tiny objects. Larger objects fall through to individual files. Not yet composable with encryption or erasure coding (skipped if either is enabled)
Scales to millions of objects: Listing and storage stats are served from a sorted BoltDB metadata index with maintained per-bucket counters (size/count updated incrementally on every write), never a filesystem walk. So dashboard stats are O(1) and the object browser pages in milliseconds regardless of bucket size, verified at 1M+ objects (stats 13s → 0.4ms)
Access logging: Structured JSON lines log file of all S3 operations
Static website hosting: Serve index/error documents from buckets, no auth required
IAM users, groups & policies: Fine-grained access control with S3-compatible policy evaluation, default deny, wildcard matching
CORS per bucket: S3-compatible CORS configuration with OPTIONS preflight support
STS temporary credentials: Short-lived access keys with configurable TTL, auto-cleanup of expired keys
Audit trail: Persistent audit log with filtering by user, bucket, time range. Auto-pruning via lifecycle worker
IP allowlist/blocklist: Global and per-user CIDR-based IP restrictions with IPv4/IPv6 support
S3 event notifications: Per-bucket webhook notifications on object mutations with event type and key prefix/suffix filtering, plus Kafka, NATS, Redis, AMQP/RabbitMQ, PostgreSQL, and Elasticsearch backends
Raft clustering: Multi-node cluster with Hashicorp Raft consensus for strongly consistent distributed metadata, automatic leader election, and node join/leave via HTTP API
Consistent hashing: xxhash64-based hash ring with virtual nodes for automatic data placement and request routing across cluster nodes via reverse proxy. A read whose data has not yet been copied to the node serving it is fetched from a holder that has it, so a GET never reports "not found" for an object that was just written, and a hop that fails before any response reaches the client is retried against the object's other holders instead of surfacing as a gateway error
Erasure coding: Reed-Solomon encoding (configurable data/parity shards) for disk-failure protection with background healer that auto-reconstructs degraded objects. Reads stream the data shards, so GET time-to-first-byte stays flat regardless of object size, and fall back to parity reconstruction only when a shard is actually missing. Settable per bucket alongside replica count (vaults3-cli bucket durability), so scratch data can be stored once while the buckets that matter keep their parity and copies: on a 3-node cluster with 4+2 coding and 3 replicas, the same data costs 4.52x with the defaults and 1.00x with both turned off
High availability: Automatic failure detection (health probes with suspect/down state machine), failover proxy routing to healthy replicas, and background rebalancer for membership changes. Inter-node traffic shares a pooled HTTP transport (connection reuse instead of a new socket per call), and a node that genuinely cannot serve a request answers 503 SlowDown with an S3 error document, which every mainstream SDK retries on its own
Scalable listing: Object listing is served from the sorted BoltDB metadata index (seek to the page, O(log n + page_size)), so ListObjectsV2 page latency stays flat (~0.7 ms per 1000-key page) whether a prefix holds a thousand or a hundred million objects (measured, not extrapolated), no full-bucket scan
- 📖 See the Scaling & Operations Guide for multi-disk erasure coding, multi-node cluster setup, large-prefix listing, and lost-disk / lost-server recovery runbooks
Active-active replication: Bidirectional site-to-site sync with vector clocks for causal ordering, pluggable conflict resolution (last-writer-wins, largest-object, site-preference), and change log for efficient delta sync
Async replication: One-way async replication to peer VaultS3 instances with BoltDB-backed queue, retry with exponential backoff, and loop prevention
CLI tool: Standalone vaults3-cli binary for bucket, object, user, and replication management without AWS CLI, plus vaults3-cli info for server version and storage capacity (used / free / total) and vaults3-cli cluster for day-2 cluster operations (status, join, leave, drain/undrain a member, rebalance, decommission — see docs/SCALING.md)
Capacity overview: GET /api/v1/system and the dashboard Stats page report the version and storage usage; in a cluster, GET /api/v1/cluster/info (the same dashboard panel, and vaults3-cli info) roll it up across all nodes with a per-node breakdown, an mc admin info-style view. Three sizes are reported separately because they answer different questions and are not meant to match: logical (each object's current version, counted once cluster-wide, since object metadata is the same on every node), VaultS3 on disk (what its data, metadata, erasure, cold-tier and Raft directories actually occupy, summed per node, so it includes replicas, parity shards and non-current versions), and filesystems (statfs of the whole volumes, which usually also hold the OS, container images and logs). The middle figure is the one to compare against logical for a real amplification ratio, with a per-directory split to tell object data apart from metadata and Raft logs. It comes from a cached background walk, storage.usage_scan_interval_secs (default 300, 0 disables it), and is also exported as vaults3_disk_usage_bytes{dir=...}
Presigned upload restrictions: Enforce max file size, content type whitelist, and key prefix on presigned PUT URLs
Full-text search: In-memory search index over object metadata, tags, content type, and key patterns with incremental updates
Semantic / vector search (optional): Embeds object text via any OpenAI-compatible endpoint (Ollama, llama.cpp, OpenAI…) and serves similarity search + RAG retrieval from POST /api/v1/vectors/query, all in the single binary, no external vector database. Searchable from the dashboard (Keyword / Semantic toggle)
Migrate from S3: Import buckets and objects from MinIO, SeaweedFS, Garage, Ceph, AWS S3, Cloudflare R2, Wasabi, Backblaze B2, or any S3-compatible source via a dashboard wizard (pick a source preset that pre-fills the endpoint and SigV4 region, test connection, pick buckets, live progress, cancel in-flight jobs). preserves each object's original modified date, user metadata, and content headers plus bucket policies and tags (a faithful copy, not a same-day re-upload, something mc mirror can't do since it writes via PutObject). Streams objects (no in-memory buffering, so arbitrarily large objects transfer without a request-size timeout), copies objects in parallel (bounded worker pool), and is resumable: a restarted or crashed migration skips objects already copied and continues instead of re-transferring the whole bucket. Retries transient errors, no AWS SDK required
Cost estimator: A dashboard panel that estimates what your stored data would cost on AWS S3, GCS, Cloudflare R2, Backblaze B2, and Wasabi (storage + egress) vs. self-hosting, egress-free, for free (GET /api/v1/tco)
Bucket snapshots ("git-for-buckets"): Capture a bucket's state, diff against the live bucket, and roll back in one click (commit / diff / restore), git-style history for your data, built on versioning with no external stack. Restore even resurrects deleted objects, and is itself reversible
Webhook virus scanning: POST uploaded objects to a configurable scan endpoint (ClamAV, VirusTotal, etc.) with quarantine bucket for infected files
Data tiering: Automatic hot/cold storage migration based on access patterns with transparent reads and manual migration API
Backup scheduler: Scheduled full/incremental backups to local directory targets with cron-like scheduling and backup history
Git-like versioning: Visual diff between object versions (text and binary), version tagging with labels, one-click rollback to any version
FUSE mount: Mount VaultS3 buckets as local filesystem directories with read/write support, lazy loading, and SigV4 authentication. LRU block cache (256KB blocks, configurable size), metadata cache with TTL, kernel attribute caching, and SigV4 derived key caching for fast repeated reads
OIDC/JWT SSO: Sign in to the dashboard with external identity providers (Google, Keycloak, Auth0, Authentik) via OpenID Connect, using the authorization-code flow with PKCE (validated end-to-end against a real Keycloak and a real Authentik). The PKCE verifier, nonce and client secret never leave the server, and the login state is sealed so a login started on one cluster node can finish on another. The authorization endpoint, expected issuer and requested scopes all come from the provider's discovery document, so providers that serve a global authorization endpoint separate from each application's issuer need no rewrite rules, and a scope the provider does not define is never requested. RS256 JWT verification with JWKS auto-discovery and caching. Email domain filtering, auto-create users, OIDC group to policy mapping.
Lambda compute triggers: Webhook-based function triggers on S3 events. Call external URLs with event payload and optional object body, optionally store the response as a new object. Per-bucket trigger configuration with event type and key prefix/suffix filtering. Worker pool with non-blocking dispatch.
SVG dashboard charts: Pure SVG bar chart (per-bucket sizes), donut chart (request method distribution), and sparkline (request activity) on the stats page, zero dependencies
GitHub Actions CI: Automated build, test, lint, and coverage on push/PR
pprof debug endpoint: /debug/pprof/* available when debug: true in config for CPU/memory profiling
Structured logging (slog): All server logs use Go's log/slog with key-value pairs. Configurable log level (debug, info, warn, error) via logging.level in config
Request ID middleware: Every response includes an X-Request-Id header for request tracing
Panic recovery middleware: Catches panics, logs full stack trace, returns 500 without crashing the server
Request latency histogram: vaults3_request_duration_seconds_bucket Prometheus histogram with 11 bucket boundaries (5ms to 10s)
Security headers: CSP, X-Frame-Options (DENY), X-Content-Type-Options (nosniff), HSTS (1 year), Referrer-Policy on all dashboard responses
CORS origin validation: Dashboard API restricts Access-Control-Allow-Origin to same-origin and localhost (replaces wildcard *)
Dashboard API rate limiting: Uses existing token bucket rate limiter on /api/v1/ endpoints, returns 429 when exceeded
Input validation: DNS-compatible bucket name validation (3-63 chars, lowercase, no leading/trailing hyphen) and object key validation (max 1024 chars, no null bytes)
RAM optimization: Slim search index with LRU eviction cap (50K entries default), batched last-access updates (30s flush interval), configurable Go memory limit (GOMEMLIMIT)
Streaming uploads: A PUT streams to disk while its checksums are computed in passing, and compression encodes as the object flows through, so peak memory scales with concurrency rather than with concurrency multiplied by object size. A 64 MiB object at 32 concurrent uploads no longer costs gigabytes of buffered copies (SSE-C and uploads with no declared length still buffer by necessity)
GetObjectAttributes: Returns object size, ETag, and storage class. Used internally by AWS SDK v2
Bucket encryption config: Per-bucket server-side encryption configuration (AES256, aws:kms) via PUT/GET/DELETE /{bucket}?encryption
Public access block: Per-bucket public access block with 4 boolean flags (BlockPublicAcls, IgnorePublicAcls, BlockPublicPolicy, RestrictPublicBuckets). BlockPublicPolicy and RestrictPublicBuckets are enforced: either one blocks anonymous access to the bucket regardless of its policy. The two ACL flags are accepted and stored for API compatibility but have no effect, since VaultS3 uses policies rather than ACLs (PUT ?acl is a no-op)
Bucket logging config: Per-bucket access logging configuration with target bucket and prefix
User metadata: Custom x-amz-meta-* headers on PUT/GET/HEAD
Conditional requests: If-Modified-Since, If-None-Match (304), If-Match, If-None-Match (412) on GET and PUT
Content-MD5 validation: Server-side integrity check on PUT with Content-MD5 header
Metadata-only copy: x-amz-metadata-directive: REPLACE for updating metadata without re-uploading
Conditional copy: x-amz-copy-source-if-* headers for conditional CopyObject
Response header overrides: ?response-content-type, ?response-content-disposition, etc. on GET
Inline tagging on PUT: x-amz-tagging header to set tags during object upload
Inline retention on PUT: x-amz-object-lock-mode header to set retention during upload
Canned ACL headers: x-amz-acl and x-amz-grant-* headers on PUT
Replication status header: x-amz-replication-status on GET/HEAD responses
Website redirect: x-amz-website-redirect-location header for per-object redirects
S3 Checksum API: CRC32, CRC32C, SHA1, SHA256 checksums on upload and download
Parts count header: x-amz-mp-parts-count on HEAD for multipart objects
ListObjectsV1: Marker-based pagination (GET /{bucket}?marker=) for legacy client compatibility
ListBuckets with prefix filter: Filter bucket listing by name prefix
Versioning suspend: Suspend versioning on a bucket while preserving existing versions
GetObject by part number: ?partNumber=N to retrieve individual parts of multipart objects
Multiple lifecycle rules: Multiple rules per bucket with prefix, tag, and size filters
NoncurrentVersionExpiration: Auto-expire non-current object versions after N days
AbortIncompleteMultipartUpload: Auto-cleanup stale multipart uploads after N days
MaxNoncurrentVersions: Cap retained non-current versions per object
ExpiredObjectDeleteMarker cleanup: Remove orphaned delete markers automatically
Object size filter: Lifecycle rules with ObjectSizeGreaterThan / ObjectSizeLessThan conditions
IAM policy conditions: StringEquals, StringLike, IpAddress, DateLessThan condition operators
Policy variables: ${aws:username}, ${aws:userid} substitution in policy resources
LDAP authentication: Bind-based LDAP/LDAPS authentication with group mapping
STS AssumeRole: Federated access via STS AssumeRole with session policies
External auth webhook: Delegate authentication to an external HTTP endpoint
Access key description & status: Metadata fields and active/inactive status on access keys
Governance bypass: x-amz-bypass-governance-retention header for admin override of GOVERNANCE locks
Per-bucket replication rules: Replication rules with prefix and tag filters per bucket
Replication Config API: S3-compatible PUT/GET/DELETE /{bucket}?replication endpoints
Real-time event-driven replication: Replicate objects immediately on mutation events
Existing object replication: Replicate pre-existing objects when enabling replication rules
Delete marker replication: Optionally replicate delete markers to target buckets
Site replication: IAM and bucket configuration sync across sites
KMS integration: HashiCorp Vault and local key provider for envelope encryption
Remote tiering: Tier cold objects to an S3-compatible remote backend
RestoreObject API: POST /{bucket}/{key}?restore to initiate restore from cold tier
Storage classes: STANDARD and REDUCED_REDUNDANCY storage class support
Compression exclusions: Skip compression for already-compressed file types (GZIP, JPEG, MP4, etc.)
Real-time event streaming: Server-Sent Events at /api/v1/events for live S3 event monitoring
Real-time log streaming: Server-Sent Events at /api/v1/logs for live access log tailing
Request tracing: Server-Sent Events at /api/v1/trace for per-request latency tracing
Health diagnostics: Detailed system diagnostics at /api/v1/diagnostics (disk, memory, goroutines, DB stats)
Manual heal API: POST /api/v1/heal to trigger erasure-coded object repair on demand
Speedtest: POST /api/v1/speedtest to benchmark storage throughput
Batch operations: Bulk delete and copy processor for large-scale object operations
PROXY protocol v1: Accept PROXY protocol connections for real client IP behind load balancers
Auto-TLS: Automatic Let's Encrypt certificates with self-signed fallback
Inter-node network separation: Bind cluster traffic to a dedicated network interface
Bucket bandwidth throttling: Per-bucket upload/download rate limits
S3 Select on compressed files: Query GZIP and BZIP2 compressed CSV/JSON objects with S3 Select
S3 POST policy: HTML form-based upload with policy document validation
S3 Inventory reports: Periodic CSV inventory of bucket contents
Snowball/TAR bulk upload: Upload TAR archives that are automatically extracted into objects
FIFO quota: Automatically delete oldest objects when bucket quota is exceeded
AMQP/RabbitMQ notifications: Publish S3 events to RabbitMQ exchanges
PostgreSQL notifications: Insert S3 events into a PostgreSQL table
Elasticsearch notifications: Index S3 events in Elasticsearch
Docker image: Multi-stage Dockerfile with built-in health check
YAML config: Simple configuration, sensible defaults

Supported S3 Operations

Operation	Endpoint	Status
List Buckets	`GET /`	Done
Create Bucket	`PUT /{bucket}`	Done
Delete Bucket	`DELETE /{bucket}`	Done
Head Bucket	`HEAD /{bucket}`	Done
Put Object	`PUT /{bucket}/{key}`	Done
Get Object	`GET /{bucket}/{key}`	Done
Delete Object	`DELETE /{bucket}/{key}`	Done
Head Object	`HEAD /{bucket}/{key}`	Done
List Objects V2	`GET /{bucket}?prefix=&max-keys=`	Done
Copy Object	`PUT /{bucket}/{key}` + `x-amz-copy-source`	Done
Batch Delete	`POST /{bucket}?delete`	Done
Multipart Upload	`POST/PUT/DELETE /{bucket}/{key}?uploads&uploadId`	Done
UploadPartCopy	`PUT /{bucket}/{key}?partNumber&uploadId` + `x-amz-copy-source`	Done
S3 Select	`POST /{bucket}/{key}?select&select-type=2`	Done
Object Tagging	`PUT/GET/DELETE /{bucket}/{key}?tagging`	Done
Bucket Policy	`PUT/GET/DELETE /{bucket}?policy`	Done
Bucket Quota	`PUT/GET /{bucket}?quota`	Done
Bucket Durability (erasure + replicas)	`PUT/GET /{bucket}?durability`	Done
Bucket Versioning	`PUT/GET /{bucket}?versioning`	Done
List Object Versions	`GET /{bucket}?versions`	Done
Object Locking (Legal Hold)	`PUT/GET /{bucket}/{key}?legal-hold`	Done
Object Locking (Retention)	`PUT/GET /{bucket}/{key}?retention`	Done
Bucket Default Retention	`PUT/GET /{bucket}?object-lock`	Done
Lifecycle Rules	`PUT/GET/DELETE /{bucket}?lifecycle`	Done
Website Hosting	`PUT/GET/DELETE /{bucket}?website`	Done
Bucket CORS	`PUT/GET/DELETE /{bucket}?cors`	Done
Presigned URLs	,	Done
Get Bucket Location	`GET /{bucket}?location`	Done
Bucket Tagging	`PUT/GET/DELETE /{bucket}?tagging`	Done
Bucket ACL	`GET/PUT /{bucket}?acl`	Done
Object ACL	`GET/PUT /{bucket}/{key}?acl`	Done
Get Object Attributes	`GET /{bucket}/{key}?attributes`	Done
Bucket Encryption	`PUT/GET/DELETE /{bucket}?encryption`	Done
Public Access Block	`PUT/GET/DELETE /{bucket}?publicAccessBlock`	Done
Bucket Logging	`PUT/GET /{bucket}?logging`	Done
List Multipart Uploads	`GET /{bucket}?uploads`	Done
List Parts	`GET /{bucket}/{key}?uploadId=X`	Done
Metrics	`GET /metrics`	Done
IAM (Users/Groups/Policies)	Dashboard API `/api/v1/iam/*`	Done
STS Temporary Credentials	`POST /api/v1/sts/session-token`	Done
Audit Trail	`GET /api/v1/audit`	Done
IP Restrictions	`PUT /api/v1/iam/users/{name}/ip-restrictions`	Done
Bucket Notifications	`PUT/GET/DELETE /{bucket}?notification`	Done
Notification Configs	`GET /api/v1/notifications`	Done
Replication Status	`GET /api/v1/replication/status`	Done
Replication Queue	`GET /api/v1/replication/queue`	Done
Presigned URL Generation	`POST /api/v1/presign`	Done
Full-Text Search	`GET /api/v1/search?q=...`	Done
Scanner Status	`GET /api/v1/scanner/status`	Done
Quarantine List	`GET /api/v1/scanner/quarantine`	Done
Tiering Status	`GET /api/v1/tiering/status`	Done
Tiering Migrate	`POST /api/v1/tiering/migrate`	Done
Backup List	`GET /api/v1/backups`	Done
Backup Trigger	`POST /api/v1/backups/trigger`	Done
Backup Status	`GET /api/v1/backups/status`	Done
Version Diff	`GET /api/v1/versions/diff`	Done
Version Tags	`GET/POST/DELETE /api/v1/versions/tags`	Done
Version Rollback	`POST /api/v1/versions/rollback`	Done
Rate Limit Status	`GET /api/v1/ratelimit/status`	Done
OIDC Config	`GET /api/v1/auth/oidc/config`	Done
OIDC Login (code flow)	`POST /api/v1/auth/oidc/start`, `POST /api/v1/auth/oidc/callback`	Done
OIDC Login (implicit)	`POST /api/v1/auth/oidc`	Done
Lambda Triggers	`PUT/GET/DELETE /{bucket}?lambda`	Done
Lambda Trigger List	`GET /api/v1/lambda/triggers`	Done
Lambda Trigger CRUD	`GET/PUT/DELETE /api/v1/lambda/triggers/{bucket}`	Done
Lambda Status	`GET /api/v1/lambda/status`	Done
Bucket Versioning (Dashboard)	`GET/PUT /api/v1/buckets/{name}/versioning`	Done
Bucket Lifecycle (Dashboard)	`GET/PUT/DELETE /api/v1/buckets/{name}/lifecycle`	Done
Bucket CORS (Dashboard)	`GET/PUT/DELETE /api/v1/buckets/{name}/cors`	Done
Bulk Delete (Dashboard)	`POST /api/v1/buckets/{name}/bulk-delete`	Done
Bulk Download Zip	`GET /api/v1/buckets/{name}/download-zip?keys=...`	Done
Version List (Dashboard)	`GET /api/v1/versions?bucket=X&key=Y`	Done
Settings	`GET /api/v1/settings`	Done
System / Capacity	`GET /api/v1/system`	Done
Cluster Capacity	`GET /api/v1/cluster/info`	Done
Cluster Status	`GET /cluster/status`, `GET /api/v1/cluster/status`	Done
Cluster Join	`POST /cluster/join`, `POST /api/v1/cluster/join`	Done
Cluster Leave	`POST /cluster/leave`, `POST /api/v1/cluster/leave`	Done
Cluster Drain / Undrain	`POST /api/v1/cluster/{drain,undrain}`	Done
Cluster Rebalance	`POST /api/v1/cluster/rebalance`	Done
Replication Sync	`POST /_replication/sync`	Done
List Objects V1	`GET /{bucket}?marker=`	Done
Replication Config	`PUT/GET/DELETE /{bucket}?replication`	Done
Restore Object	`POST /{bucket}/{key}?restore`	Done
POST Upload (Form)	`POST /{bucket}` (multipart/form-data)	Done
Get Object (Part)	`GET /{bucket}/{key}?partNumber=N`	Done
Event Stream	`GET /api/v1/events` (SSE)	Done
Log Stream	`GET /api/v1/logs` (SSE)	Done
Request Trace	`GET /api/v1/trace` (SSE)	Done
Health Diagnostics	`GET /api/v1/diagnostics`	Done
Manual Heal	`POST /api/v1/heal`	Done
Speedtest	`POST /api/v1/speedtest`	Done
Batch Operations	`POST /api/v1/batch`	Done
STS AssumeRole	`POST /api/v1/sts/assume-role`	Done
Inventory Reports	`GET /api/v1/inventory`	Done

Quick Start

Build

make build

Run

./vaults3

Server starts on http://localhost:9000 by default.

Configure

Edit configs/vaults3.yaml:

server:
  address: "0.0.0.0"
  port: 9000
  domain: ""  # set to enable virtual-hosted URLs (e.g. "s3.example.com")
  shutdown_timeout_secs: 30
  tls:
    enabled: false
    cert_file: ""
    key_file: ""

storage:
  data_dir: "./data"
  metadata_dir: "./metadata"

auth:
  admin_access_key: "vaults3-admin"
  admin_secret_key: "vaults3-secret-change-me"

encryption:
  enabled: false
  key: ""  # 64-character hex string (32 bytes) for SSE-S3
  kms:     # SSE-KMS (optional, overrides static key when enabled)
    enabled: false
    provider: "vault"          # "vault" or "local"
    vault_addr: ""
    vault_token: ""
    key_name: "vaults3-dek"
    local_key: ""

compression:
  enabled: false

logging:
  enabled: false
  file_path: "./access.log"
  level: "info"  # debug, info, warn, error

lifecycle:
  scan_interval_secs: 3600

security:
  ip_allowlist: []     # global CIDR allow list, empty = allow all
  ip_blocklist: []     # global CIDR deny list
  audit_retention_days: 90
  sts_max_duration_secs: 43200  # max STS token duration (12 hours)

# Distributed clustering (optional)
cluster:
  enabled: false
  node_id: "node-1"
  bind_addr: "0.0.0.0"
  raft_port: 9001
  api_port: 9000
  bootstrap: true          # true for the first node
  peers: []                # ["node-2@host2:9001", "node-3@host3:9001"]
  peer_apis:               # nodeID → "host:apiPort"
    node-2: "host2:9000"
    node-3: "host3:9000"
  placement:
    replica_count: 3
    read_quorum: 2
    write_quorum: 2
    virtual_nodes: 128
  detector:
    probe_interval_secs: 5
    suspect_after: 3
    down_after: 6
  rebalance:
    max_bandwidth_mbps: 50
    batch_size: 100

# Erasure coding (optional, works with or without clustering)
erasure:
  enabled: false
  data_shards: 4
  parity_shards: 2
  block_size: 4194304      # 4MB, objects smaller than this bypass EC
  data_dirs: []            # multiple disk paths for shard distribution
  heal_interval: 300       # seconds between heal scans

# Replication
replication:
  enabled: false
  mode: "push"             # "push" (one-way) or "active-active" (bidirectional)
  site_id: "site-1"        # unique site ID for active-active mode
  conflict_strategy: "last-writer-wins"  # "last-writer-wins", "largest-object", "site-preference"
  preferred_site: ""       # for site-preference strategy
  peers: []
  scan_interval_secs: 10
  max_retries: 5
  batch_size: 100

Encryption at Rest

VaultS3 supports two encryption modes:

SSE-S3 (Static Key), Simple setup with a hex-encoded 32-byte key:

encryption:
  enabled: true
  key: ""  # 64-char hex string: openssl rand -hex 32

SSE-KMS (Key Management Service), Per-object encryption with KMS-managed data encryption keys:

encryption:
  enabled: true
  kms:
    enabled: true
    provider: "vault"          # "vault" (HashiCorp Vault) or "local" (fallback)
    vault_addr: "http://vault:8200"
    vault_token: "hvs.xxx"
    key_name: "vaults3-dek"    # Transit engine key name
    local_key: ""              # hex-encoded fallback key (when provider: "local")

SSE-KMS fetches data encryption keys from HashiCorp Vault's Transit engine, caches them in memory, and supports key rotation. All objects are encrypted with AES-256-GCM using a random nonce per object.

Virtual-Hosted Style URLs

Set server.domain to enable virtual-hosted style access:

server:
  domain: "s3.example.com"

This enables bucket-name.s3.example.com/key in addition to the default s3.example.com/bucket-name/key path-style.

Prometheus Metrics

Access metrics at GET /metrics:

curl http://localhost:9000/metrics

Exposes: request counts by method, bytes in/out, per-bucket storage size and object counts, per-bucket request/bytes/error counters, quota usage, Go runtime stats (goroutines, memory, GC).

Storage is reported both ways, which is what makes growth diagnosable:

Series	Measures
`vaults3_storage_size_bytes_total`	Logical: each object's current version, counted once.
`vaults3_disk_usage_bytes{dir="..."}`	Physical: what that directory actually occupies on disk, per data / metadata / erasure / cold-tier / Raft directory. Includes replicas, parity shards and non-current versions.
`vaults3_disk_usage_files{dir="..."}`	File count behind the figure above.
`vaults3_disk_usage_bytes_total`	Physical total for this node.
`vaults3_disk_usage_scanned_timestamp_seconds`	When the footprint was last measured, so a stale reading is detectable.

Graphing physical against logical shows amplification directly. The vaults3_disk_usage_* series come from a cached background walk and are absent when storage.usage_scan_interval_secs is 0.

Web Dashboard

The built-in dashboard is available at http://localhost:9000/dashboard/. Login with your admin credentials. Features:

Bucket browser, list, create, delete buckets
Bucket detail, view/edit policies and quotas
File browser, list, upload (drag & drop files and folders), download, delete objects with folder navigation, multi-select with bulk delete and bulk zip download, copy-to-clipboard for S3 URIs and keys
Access key management, create, list, revoke S3 API keys
IAM management, users, groups, policies CRUD with attach/detach operations
Audit trail, filter by user, bucket, time range with auto-refresh
Search, full-text search across all buckets by key, content type, tags
Notifications, view webhook notification configurations
Replication, peer status cards, pending queue table
Lambda triggers, status overview, trigger table with event filtering
Backups, status cards, history table, manual trigger button
Activity log, real-time S3 operation feed with auto-refresh
Storage stats, logical size, VaultS3's measured on-disk footprint and total filesystem usage side by side (with a per-directory and per-node breakdown), per-bucket breakdown, runtime metrics, auto-refresh toggle (30s)
Migrate, import buckets from any S3-compatible source with live progress and a Cancel button for in-flight jobs
Version indicator, the running version is shown at the bottom of the sidebar, with an "update available" hint linking to releases
Dark/light theme, toggle with system preference detection
Language, English, German, French, Simplified Chinese, detected from the browser and switchable in the top bar
Responsive layout, mobile-friendly with collapsible sidebar
JWT-based authentication (24h tokens)

The dashboard is embedded into the binary, no separate web server needed.

Language

The dashboard picks a language from the browser on first load and falls back to English. Change it with the selector in the top bar; the choice is stored per browser, so different people using the same server can each read it in their own language. There is no server-side setting.

Shipping today: English, Deutsch, Francais, and simplified Chinese. The non-English files were drafted without a native-speaker review, so corrections are welcome.

Adding a language takes one JSON file and no code: copy web/src/i18n/locales/en.json, translate the values, and add one entry to LOCALES in web/src/i18n/index.tsx. Any key you leave out falls back to English, so a partial translation is fine to send. See Translating the dashboard for the full steps and the test that checks a locale file.

Server-side output (S3 API error codes, log lines) is English only.

Screenshots

Buckets	File Browser

Bucket Detail	Search

Dark Mode	Settings

Health Checks

curl http://localhost:9000/health   # liveness: {"status":"ok","uptime":"5h23m"}
curl http://localhost:9000/ready    # readiness: checks BoltDB, returns 503 if unhealthy

TLS

Enable HTTPS by providing cert and key files:

server:
  tls:
    enabled: true
    cert_file: "/path/to/cert.pem"
    key_file: "/path/to/key.pem"

Docker

# From Docker Hub
docker pull eniz1806/vaults3
docker run -p 9000:9000 \
  -e VAULTS3_ACCESS_KEY=myadmin \
  -e VAULTS3_SECRET_KEY=mysupersecret \
  -v ./data:/data -v ./metadata:/metadata \
  eniz1806/vaults3

# Or build locally
docker build -t vaults3 .
docker run -p 9000:9000 -v ./data:/data -v ./metadata:/metadata vaults3

Images are automatically published to Docker Hub on every push to main.

Buckets on First Start

A fresh container has no buckets, which normally means an init container or a one-off S3 client call before the app can write anything. Name the buckets you need instead and VaultS3 creates the missing ones while it starts:

docker run -p 9000:9000 \
  -e VAULTS3_DEFAULT_BUCKETS=app-data,backups \
  -v ./data:/data -v ./metadata:/metadata \
  eniz1806/vaults3

or in vaults3.yaml:

storage:
  default_buckets: ["app-data", "backups"]

Buckets that already exist are left completely alone: no data, policy, versioning, or lifecycle setting is touched, so the variable is safe to keep in place across restarts and upgrades.
Removing a name from the list never deletes anything.
The setting means "these buckets must exist", so if you delete one while its name is still listed, the next restart creates it again, empty. Take the name out of the list first if you mean the deletion to stick.
An invalid bucket name, or a bucket that cannot be created, stops startup with an error naming the bucket, rather than starting up quietly incomplete.
On a cluster, creation is a replicated write like any other, so the nodes agree on one bucket no matter how many of them boot with the same setting.

Environment Variables

All settings can be overridden via environment variables (takes precedence over config file):

Variable	Description	Default
`VAULTS3_ACCESS_KEY`	Admin access key	`vaults3-admin`
`VAULTS3_SECRET_KEY`	Admin secret key	`vaults3-secret-change-me`
`VAULTS3_PORT`	Server port	`9000`
`VAULTS3_ADDRESS`	Bind address	`0.0.0.0`
`VAULTS3_DOMAIN`	Domain for virtual-hosted URLs	(empty)
`VAULTS3_BASE_PATH`	Reverse-proxy subpath (e.g. `/vaults3`)	(empty)
`VAULTS3_TRUST_FORWARDED_PREFIX`	Auto-detect subpath from `X-Forwarded-Prefix`	`false`
`VAULTS3_DATA_DIR`	Object storage directory	`./data`
`VAULTS3_METADATA_DIR`	BoltDB metadata directory	`./metadata`
`VAULTS3_DEFAULT_BUCKETS`	Comma-separated buckets to create on startup if missing	(none)
`VAULTS3_USAGE_SCAN_INTERVAL_SECS`	How often VaultS3 may re-measure its own on-disk footprint (0 disables)	`300`
`VAULTS3_ENCRYPTION_KEY`	64-char hex key (enables encryption)	(disabled)
`VAULTS3_TLS_CERT`	TLS certificate file path	(disabled)
`VAULTS3_TLS_KEY`	TLS private key file path	(disabled)
`VAULTS3_LOG_LEVEL`	Log level (`debug`, `info`, `warn`, `error`)	`info`
`VAULTS3_TRACE_FORWARD`	Log per-hop latency (DNS/connect/reuse/TTFB) for proxied cluster reads	`0`
`VAULTS3_TRACE_READS`	Log the cause (`meta_nil` vs `data_missing`) of cluster `GET`/`HEAD` 404s	`0`
`VAULTS3_OIDC_CLIENT_SECRET`	OAuth client secret for SSO (keeps it out of the config file)	(public client)

Storage requirements

VaultS3 stores each object as a regular file under data_dir and keeps metadata in a BoltDB file under metadata_dir, so both must point at a mounted filesystem, not a raw block device. Format the disk first (XFS recommended. ext4 also works) and mount it, then point data_dir at a directory on the mount. This is the same model as MinIO.

One file per object means a filesystem with plenty of inodes (XFS handles this well). For workloads with millions of tiny objects, enable the experimental small-file packing mode to pack them into large volume files and cut per-file overhead.
On Kubernetes, a CSI driver like DirectPV is a good fit, it formats disks with XFS and presents them as mounted PVCs, which is exactly what VaultS3 wants.

Kubernetes

Deploy with the bundled Helm chart or a single plain-manifest quickstart (both under deploy/):

# Helm (configurable, production-grade)
helm install vaults3 ./deploy/helm/vaults3 \
  --namespace vaults3 --create-namespace \
  --set auth.secretKey="$(openssl rand -hex 20)" \
  --set defaultBuckets="{app-data,backups}"

# Or plain manifests (single-node, no Helm)
kubectl apply -f deploy/k8s/quickstart.yaml

defaultBuckets creates those buckets on startup if they are missing, so the release needs no init container to become usable. See Buckets on First Start.

Both deploy a StatefulSet (S3 API + dashboard on port 9000), admin keys via a Secret, vaults3.yaml via a ConfigMap, persistent volumes for /data and /metadata, liveness/readiness probes (/health, /ready), and an optional Ingress + Prometheus ServiceMonitor. See deploy/README.md and the chart reference.

The Helm chart can also auto-form a Raft cluster (Beta) with --set cluster.enabled=true --set replicaCount=3: pod-0 bootstraps as leader and the others auto-join, with self-healing on pod restart. Metadata writes replicate across nodes via Raft consensus (a write to any node is forwarded to the leader), so the cluster stays consistent. Clustering is functional but newer than single-node + erasure coding, which remains the recommendation for maximum production durability.

For backup/restore workflows, the chart also supports a single-node Deployment mode (controller.kind=Deployment) and existing PVCs (persistence.data.existingClaim), so you can mount a claim restored from Velero, k8up, or a CSI snapshot, see the chart reference.

Object Versioning

Enable versioning on a bucket to keep multiple versions of objects:

import boto3

s3 = boto3.client('s3', endpoint_url='http://localhost:9000',
    aws_access_key_id='vaults3-admin',
    aws_secret_access_key='vaults3-secret-change-me')

# Enable versioning
s3.put_bucket_versioning(Bucket='my-bucket',
    VersioningConfiguration={'Status': 'Enabled'})

# Upload creates a new version each time
s3.put_object(Bucket='my-bucket', Key='file.txt', Body=b'v1')
s3.put_object(Bucket='my-bucket', Key='file.txt', Body=b'v2')

# Get specific version
s3.get_object(Bucket='my-bucket', Key='file.txt', VersionId='...')

# Delete creates a delete marker (versions preserved)
s3.delete_object(Bucket='my-bucket', Key='file.txt')

# Permanently delete a specific version
s3.delete_object(Bucket='my-bucket', Key='file.txt', VersionId='...')

Object Locking (WORM)

Protect objects from deletion with legal holds or retention policies:

# Legal hold — prevents deletion regardless
s3.put_object_legal_hold(Bucket='my-bucket', Key='file.txt', VersionId='...',
    LegalHold={'Status': 'ON'})

# Retention — prevents deletion until date
s3.put_object_retention(Bucket='my-bucket', Key='file.txt', VersionId='...',
    Retention={'Mode': 'COMPLIANCE', 'RetainUntilDate': '2030-01-01T00:00:00Z'})

Lifecycle Rules

Auto-delete objects after a specified number of days:

s3.put_bucket_lifecycle_configuration(Bucket='my-bucket',
    LifecycleConfiguration={
        'Rules': [{
            'ID': 'expire-logs',
            'Expiration': {'Days': 30},
            'Filter': {'Prefix': 'logs/'},
            'Status': 'Enabled',
        }]
    })

Abort incomplete multipart uploads (from killed or failed clients) after a number of days, reclaiming the uploaded parts. A rule may specify only this action, with no object expiration:

s3.put_bucket_lifecycle_configuration(Bucket='my-bucket',
    LifecycleConfiguration={
        'Rules': [{
            'ID': 'abort-stale-uploads',
            'AbortIncompleteMultipartUpload': {'DaysAfterInitiation': 7},
            'Filter': {'Prefix': ''},
            'Status': 'Enabled',
        }]
    })

The background worker scans periodically (configurable interval, default 1 hour) and deletes expired objects and aborts stale multipart uploads (removing both their metadata and their part files on disk). Locked objects (legal hold or retention) are skipped.

Compression

Enable zstd compression to reduce storage usage:

compression:
  enabled: true

All objects are transparently compressed (zstd) on write and decompressed on read. Objects written by older gzip builds are still read correctly. Works with encryption (data is compressed then encrypted on disk).

Both directions stream, so a large object costs a compression window rather than a copy of itself: peak memory scales with concurrency, not with concurrency multiplied by object size. An upload that does not declare its length falls back to buffering, because the decompressed size has to be recorded in the frame header for reads to stream.

Small-file packing (experimental)

For workloads with huge numbers of tiny objects, packing stores small objects as independent zstd frames inside large append-only volume files (with byte-offset locations in BoltDB) instead of one file per object, avoiding per-file overhead. Objects larger than max_object_size are stored as individual files as usual.

packing:
  enabled: true
  max_object_size: 1048576       # objects this size (bytes) or smaller are packed
  volume_max_size: 1073741824    # roll to a new volume past this size
  compact_interval_hours: 24     # background dead-space reclamation; 0 = disabled
  compact_min_dead_ratio: 0.5    # compact a volume once half of it is dead space

Deleted/overwritten objects leave dead space in volumes. It is reclaimed by background compaction (or on demand via POST /api/v1/compact). Packing is experimental and does not yet compose with encryption or erasure coding (it is skipped, with a warning, if either is enabled).

Access Logging

Enable structured JSON access logs:

logging:
  enabled: true
  file_path: "./access.log"

Each S3 operation is logged as a JSON line with timestamp, method, bucket, key, status code, bytes, and client IP.

Static Website Hosting

Serve static websites directly from buckets:

s3.put_bucket_website(Bucket='my-site',
    WebsiteConfiguration={
        'IndexDocument': {'Suffix': 'index.html'},
        'ErrorDocument': {'Key': 'error.html'}
    })

Website-enabled buckets serve index.html for directory paths and a custom error page for missing objects. No authentication required for GET/HEAD requests.

IAM (Users, Groups & Policies)

Fine-grained access control with S3-compatible IAM policies:

import requests, json

API = "http://localhost:9000/api/v1"
headers = {"Authorization": "Bearer <jwt-token>", "Content-Type": "application/json"}

# Create an IAM user
requests.post(f"{API}/iam/users", headers=headers, json={"name": "alice"})

# Attach a built-in policy (ReadOnlyAccess, ReadWriteAccess, FullAccess)
requests.post(f"{API}/iam/users/alice/policies", headers=headers,
    json={"policyName": "ReadOnlyAccess"})

# Create an access key for the user
resp = requests.post(f"{API}/keys", headers=headers, json={"userId": "alice"})
key = resp.json()  # {"accessKey": "...", "secretKey": "..."}

# Create groups and attach policies
requests.post(f"{API}/iam/groups", headers=headers, json={"name": "developers"})
requests.post(f"{API}/iam/groups/developers/policies", headers=headers,
    json={"policyName": "ReadWriteAccess"})

# Add user to group
requests.post(f"{API}/iam/users/alice/groups", headers=headers,
    json={"groupName": "developers"})

# Create custom policies
custom_policy = json.dumps({
    "Version": "2012-10-17",
    "Statement": [{
        "Effect": "Allow",
        "Action": ["s3:GetObject"],
        "Resource": ["arn:aws:s3:::my-bucket/*"]
    }]
})
requests.post(f"{API}/iam/policies", headers=headers,
    json={"name": "MyBucketReadOnly", "document": custom_policy})

Policy evaluation follows AWS IAM semantics: default deny, explicit Allow required, explicit Deny always wins. Admin keys and legacy keys (without a user) retain full access.

CORS per Bucket

Configure Cross-Origin Resource Sharing on a per-bucket basis:

s3.put_bucket_cors(Bucket='my-bucket', CORSConfiguration={
    'CORSRules': [{
        'AllowedOrigins': ['https://example.com'],
        'AllowedMethods': ['GET', 'PUT'],
        'AllowedHeaders': ['*'],
        'MaxAgeSeconds': 3600,
    }]
})

The server responds to OPTIONS preflight requests with the configured CORS headers. Unknown origins are rejected with 403.

STS Temporary Credentials

Issue short-lived access keys for temporary access:

import requests, boto3

API = "http://localhost:9000/api/v1"
headers = {"Authorization": "Bearer <jwt-token>", "Content-Type": "application/json"}

# Create temporary credentials for an IAM user (max 12 hours)
resp = requests.post(f"{API}/sts/session-token", headers=headers,
    json={"durationSecs": 3600, "userId": "alice"})
creds = resp.json()  # {"accessKey", "secretKey", "sessionToken", "expiration"}

# Use temporary credentials with any S3 client
s3 = boto3.client("s3", endpoint_url="http://localhost:9000",
    aws_access_key_id=creds["accessKey"],
    aws_secret_access_key=creds["secretKey"])

Temporary keys inherit the IAM user's policies. Expired keys are automatically cleaned up by the lifecycle worker.

Audit Trail

Query the persistent audit log of all S3 operations:

# List recent audit entries
requests.get(f"{API}/audit?limit=50", headers=headers)

# Filter by user, time range, or bucket
requests.get(f"{API}/audit?user=alice&limit=10", headers=headers)
requests.get(f"{API}/audit?from=1700000000&to=1700100000", headers=headers)
requests.get(f"{API}/audit?bucket=my-bucket", headers=headers)

Each entry records: timestamp, principal, user ID, action, resource, effect (Allow/Deny), source IP, and status code. Old entries are automatically pruned based on security.audit_retention_days.

IP Restrictions

Control access by IP address at global or per-user level:

# Global restrictions in config
security:
  ip_allowlist: ["10.0.0.0/8", "192.168.0.0/16"]  # empty = allow all
  ip_blocklist: ["10.0.0.99/32"]  # deny always wins

# Per-user IP restrictions via API
requests.put(f"{API}/iam/users/alice/ip-restrictions", headers=headers,
    json={"allowedCidrs": ["10.0.0.0/8", "::1/128"]})

# Clear restrictions (allow from anywhere)
requests.put(f"{API}/iam/users/alice/ip-restrictions", headers=headers,
    json={"allowedCidrs": []})

Evaluation order: global blocklist (deny wins) → global allowlist → per-user allowlist. Admin keys are exempt from IP restrictions. Supports both IPv4 and IPv6 CIDR notation.

S3 Event Notifications

Configure webhooks on buckets to receive notifications when objects are created or deleted:

from botocore.auth import SigV4Auth
from botocore.credentials import Credentials
from botocore.awsrequest import AWSRequest
import requests

# PUT notification configuration (S3-compatible XML)
notif_xml = b"""<?xml version="1.0" encoding="UTF-8"?>
<NotificationConfiguration>
  <TopicConfiguration>
    <Id>my-webhook</Id>
    <Topic>https://example.com/webhook</Topic>
    <Event>s3:ObjectCreated:*</Event>
    <Event>s3:ObjectRemoved:*</Event>
    <Filter>
      <S3Key>
        <FilterRule>
          <Name>prefix</Name>
          <Value>images/</Value>
        </FilterRule>
      </S3Key>
    </Filter>
  </TopicConfiguration>
</NotificationConfiguration>"""

# Sign and send (using botocore for SigV4)
url = "http://localhost:9000/my-bucket?notification"
creds = Credentials("vaults3-admin", "vaults3-secret-change-me")
req = AWSRequest(method="PUT", url=url, data=notif_xml,
    headers={"Content-Type": "application/xml"})
SigV4Auth(creds, "s3", "us-east-1").add_auth(req)
requests.put(url, headers=dict(req.headers), data=notif_xml)

Supported events: s3:ObjectCreated:Put, s3:ObjectCreated:Copy, s3:ObjectCreated:CompleteMultipartUpload, s3:ObjectRemoved:Delete. Use wildcards like s3:ObjectCreated:*. Webhook payloads follow the AWS S3 event notification JSON format.

Configure webhook delivery in configs/vaults3.yaml:

notifications:
  max_workers: 4       # concurrent webhook delivery goroutines
  queue_size: 256      # buffered event queue size
  timeout_secs: 10     # webhook HTTP timeout
  max_retries: 3       # retry attempts for failed webhooks
  kafka:
    enabled: true
    brokers: ["localhost:9092"]
    topic: "vaults3-events"
  nats:
    enabled: true
    url: "nats://localhost:4222"
    subject: "vaults3.events"
  redis:
    enabled: true
    addr: "localhost:6379"
    channel: "vaults3-events"   # pub/sub mode
    list_key: ""                # set for LPUSH queue mode
  amqp:
    enabled: true
    url: "amqp://guest:guest@localhost:5672/"
    exchange: "vaults3-events"
    routing_key: "s3.events"
  postgres:
    enabled: true
    dsn: "postgres://user:pass@localhost:5432/vaults3?sslmode=disable"
    table: "s3_events"
  elasticsearch:
    enabled: true
    urls: ["http://localhost:9200"]
    index: "vaults3-events"

Additional backends: AMQP/RabbitMQ (publish to exchanges), PostgreSQL (insert into table), Elasticsearch (index events). In addition to per-bucket webhooks, you can enable global notification backends. All S3 events are published to every enabled backend. Multiple backends can be active simultaneously. Disabled backends add zero overhead.

Async Replication

Replicate objects to a peer VaultS3 instance automatically:

replication:
  enabled: true
  peers:
    - name: "dc2"
      url: "http://peer-vaults3:9000"
      access_key: "peer-admin"
      secret_key: "peer-secret"
  scan_interval_secs: 30   # queue processing interval
  max_retries: 5           # retry before dead-letter
  batch_size: 100          # events per scan cycle

Objects created, copied, or deleted on the primary, whether through the S3 API or the web dashboard, are asynchronously pushed to all configured peers over the S3 protocol. Buckets are auto-created on peers. Failed deliveries retry with exponential backoff (5s, 15s, 45s, 135s, 405s). The X-VaultS3-Replication header prevents infinite loops. Monitor via dashboard API:

curl http://localhost:9000/api/v1/replication/status   # per-peer sync stats
curl http://localhost:9000/api/v1/replication/queue     # pending queue entries

For one-way push, replication only needs to be enabled on the source. The target does not need replication.enabled, it just needs the peer access_key/secret_key (from the source's config) to be valid credentials on it. Enable replication on both sides only for mode: active-active.

CLI Tool

VaultS3 includes a standalone CLI binary (vaults3-cli) for managing the server:

# Set credentials via environment or flags
export VAULTS3_ENDPOINT=http://localhost:9000
export VAULTS3_ACCESS_KEY=vaults3-admin
export VAULTS3_SECRET_KEY=vaults3-secret-change-me

# Server version and storage capacity (used / free / total)
vaults3-cli info

# Bucket operations
vaults3-cli bucket list
vaults3-cli bucket create my-bucket
vaults3-cli bucket info my-bucket
vaults3-cli bucket delete my-bucket

# Object operations
vaults3-cli object put my-bucket docs/readme.md ./README.md
vaults3-cli object ls my-bucket                       # folder view (objects + prefixes)
vaults3-cli object ls my-bucket --prefix=docs/        # list inside a prefix
vaults3-cli object ls my-bucket --recursive           # all nested objects (paginates past 1000)
vaults3-cli object get my-bucket docs/readme.md ./downloaded.md
vaults3-cli object cp my-bucket/file.txt my-bucket/copy.txt
vaults3-cli object rm my-bucket docs/readme.md
vaults3-cli object presign my-bucket file.txt --expires=3600
vaults3-cli object verify my-bucket                   # find objects that list but cannot be read (metadata/data desync)
vaults3-cli object verify my-bucket --repair          # remove orphaned metadata for unreadable objects

# IAM user operations
vaults3-cli user list
vaults3-cli user create alice --access-key=ak --secret-key=sk
vaults3-cli user attach-policy alice ReadWriteAccess
vaults3-cli user delete alice

# Replication monitoring
vaults3-cli replication status
vaults3-cli replication queue

# Cluster operations (see docs/SCALING.md)
vaults3-cli bucket durability scratch --erasure=off --replicas=1  # store this bucket once
vaults3-cli cluster status                     # members, leader, drain state
vaults3-cli cluster join node-3 10.0.0.4:7000  # add a member (against the leader)
vaults3-cli cluster drain node-2               # stop a node accepting writes (reads continue)
vaults3-cli cluster rebalance                  # move objects to their correct owner
vaults3-cli cluster decommission node-2        # guided drain + rebalance before replacing a node

Build both binaries with make build or just the CLI with make cli.

Presigned Upload Restrictions

Generate presigned PUT URLs with server-enforced restrictions:

import requests

API = "http://localhost:9000/api/v1"
headers = {"Authorization": "Bearer <jwt-token>", "Content-Type": "application/json"}

# Generate restricted presigned PUT URL
resp = requests.post(f"{API}/presign", headers=headers, json={
    "bucket": "uploads",
    "key": "images/photo.jpg",
    "method": "PUT",
    "expires": 3600,
    "maxSize": 10485760,               # 10MB max
    "allowTypes": "image/jpeg,image/png",  # only images
    "requirePrefix": "images/"         # must upload to images/
})
url = resp.json()["url"]

# Upload within restrictions — succeeds
requests.put(url, data=image_data, headers={"Content-Type": "image/jpeg"})

# Upload too large / wrong type / wrong prefix — 403 Forbidden

Restriction parameters (X-Vault-MaxSize, X-Vault-AllowTypes, X-Vault-RequirePrefix) are embedded in the signed URL and validated server-side.

Full-Text Search

Search objects by key, content type, and tags across all buckets:

# Search by key substring
curl "http://localhost:9000/api/v1/search?q=readme" -H "Authorization: Bearer <token>"

# Search by content type
curl "http://localhost:9000/api/v1/search?q=type:image" -H "Authorization: Bearer <token>"

# Search by tag
curl "http://localhost:9000/api/v1/search?q=tag:project=vaults3" -H "Authorization: Bearer <token>"

# Filter by bucket and limit results
curl "http://localhost:9000/api/v1/search?q=docs&bucket=my-bucket&limit=10" -H "Authorization: Bearer <token>"

The search index is built on startup from BoltDB metadata and updated incrementally on every object put, delete, copy, and tag change. Supports plain text (substring match), type: prefix for content-type filtering, and tag:key=value for tag matching.

Webhook Virus Scanning

Scan uploaded objects with an external virus scanner (ClamAV REST, VirusTotal, etc.):

scanner:
  enabled: true
  webhook_url: "http://localhost:3310/scan"
  timeout_secs: 30
  quarantine_bucket: "vaults3-quarantine"
  fail_closed: false          # false=fail-open (keep file), true=quarantine on error
  max_scan_size_bytes: 104857600  # 100MB
  workers: 2

When enabled, every uploaded object is POSTed to the webhook URL as multipart/form-data. If the scanner returns 406/403 (infected), the object is moved to the quarantine bucket and deleted from the original. Monitor via dashboard API:

curl http://localhost:9000/api/v1/scanner/status       # queue depth + recent scans
curl http://localhost:9000/api/v1/scanner/quarantine    # quarantined objects

Data Tiering

Automatically migrate infrequently accessed objects to a cold storage directory:

tiering:
  enabled: true
  cold_data_dir: "./cold_data"
  migrate_after_days: 30
  scan_interval_secs: 3600

Objects not accessed for migrate_after_days are moved to the cold data directory. On read, cold objects are transparently served and promoted back to hot storage. Manual migration is available via API:

# Check tiering status (hot/cold counts and sizes)
curl http://localhost:9000/api/v1/tiering/status -H "Authorization: Bearer <token>"

# Manually migrate an object to cold tier
curl -X POST http://localhost:9000/api/v1/tiering/migrate \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{"bucket":"my-bucket","key":"archive/old-file.zip","direction":"cold"}'

Backup Scheduler

Schedule automatic backups to local directories:

backup:
  enabled: true
  targets:
    - name: "local-backup"
      type: "local"
      path: "/backups/vaults3"
  schedule_cron: "0 2 * * *"   # daily at 2am
  retention_days: 30
  incremental: false            # true for incremental backups

Monitor and trigger backups via API:

# Check backup status
curl http://localhost:9000/api/v1/backups/status -H "Authorization: Bearer <token>"

# List backup history
curl http://localhost:9000/api/v1/backups -H "Authorization: Bearer <token>"

# Trigger immediate backup
curl -X POST http://localhost:9000/api/v1/backups/trigger -H "Authorization: Bearer <token>"

Incremental backups only copy objects modified since the last successful backup. Full backups mirror the complete object store.

Git-like Versioning

Compare, tag, and rollback object versions:

# Diff two versions (text files show line-by-line diff, binary shows metadata only)
curl "http://localhost:9000/api/v1/versions/diff?bucket=my-bucket&key=file.txt&v1=VERSION_A&v2=VERSION_B" \
  -H "Authorization: Bearer <token>"

# Tag a version with a label
curl -X POST http://localhost:9000/api/v1/versions/tags \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{"bucket":"my-bucket","key":"file.txt","versionId":"VERSION_ID","tag":"v1.0"}'

# List tags for an object
curl "http://localhost:9000/api/v1/versions/tags?bucket=my-bucket&key=file.txt" \
  -H "Authorization: Bearer <token>"

# Delete a tag
curl -X DELETE "http://localhost:9000/api/v1/versions/tags?bucket=my-bucket&key=file.txt&tag=v1.0" \
  -H "Authorization: Bearer <token>"

# Rollback to a specific version (copies old version content as latest)
curl -X POST http://localhost:9000/api/v1/versions/rollback \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{"bucket":"my-bucket","key":"file.txt","versionId":"VERSION_ID"}'

Text diffs use LCS (Longest Common Subsequence) to produce unified diffs with add/remove/equal lines. Binary files show only size and metadata differences.

FUSE Mount

Mount a VaultS3 bucket as a local filesystem directory:

# Mount a bucket (requires macFUSE on macOS or FUSE on Linux)
vaults3-cli mount my-bucket /mnt/vaults3

# Browse files
ls /mnt/vaults3
cat /mnt/vaults3/docs/readme.txt

# Write files (creates objects in VaultS3)
echo "hello" > /mnt/vaults3/new-file.txt

# Unmount
# Press Ctrl+C in the mount terminal, or:
fusermount -u /mnt/vaults3

FUSE mount uses range requests for lazy loading, only the requested bytes are fetched from the server. Write support buffers data and uploads on file close.

S3 Select (SQL on Objects)

Execute SQL queries on CSV and JSON objects without downloading the full file:

from botocore.auth import SigV4Auth
from botocore.credentials import Credentials
from botocore.awsrequest import AWSRequest
import requests

# Query a CSV file
url = "http://localhost:9000/my-bucket/data.csv?select&select-type=2"
body = b"""<?xml version="1.0"?>
<SelectObjectContentRequest>
    <Expression>SELECT name, age FROM s3object WHERE city = 'New York' AND age > '25'</Expression>
    <ExpressionType>SQL</ExpressionType>
    <InputSerialization><CSV><FileHeaderInfo>USE</FileHeaderInfo></CSV></InputSerialization>
    <OutputSerialization><JSON/></OutputSerialization>
</SelectObjectContentRequest>"""

creds = Credentials("vaults3-admin", "vaults3-secret-change-me")
req = AWSRequest(method="POST", url=url, data=body, headers={"Content-Type": "application/xml"})
SigV4Auth(creds, "s3", "us-east-1").add_auth(req)
r = requests.post(url, headers=dict(req.headers), data=body)
# Returns JSON lines: {"name":"Alice","age":"30"}\n{"name":"Charlie","age":"35"}

Supported SQL features:

SELECT * or SELECT col1, col2 (column projection)
FROM s3object (required table name)
WHERE col = 'value', !=, <, >, <=, >= (comparisons, numeric-aware)
AND / OR (logical operators)
LIKE 'pattern%' (SQL wildcards: % = any chars, _ = single char)
IS NULL / IS NOT NULL
LIMIT N
Column references: name, s3object.name, s.name, _1 (positional for CSV without headers)

Input formats: CSV (with/without headers, custom delimiters), JSON Lines, JSON Document (array), Parquet (columnar format via parquet-go). Compressed input: GZIP and BZIP2 compressed CSV/JSON files are transparently decompressed before query execution. Output formats: JSON (one object per line) or CSV.

Bucket Default Retention

Set default object retention on a versioned bucket, all new objects automatically inherit the retention policy:

from botocore.auth import SigV4Auth
from botocore.credentials import Credentials
from botocore.awsrequest import AWSRequest
import requests

# Set default retention (requires versioning enabled)
url = "http://localhost:9000/my-bucket?object-lock"
body = b"""<?xml version="1.0" encoding="UTF-8"?>
<ObjectLockConfiguration>
  <Rule>
    <DefaultRetention>
      <Mode>GOVERNANCE</Mode>
      <Days>30</Days>
    </DefaultRetention>
  </Rule>
</ObjectLockConfiguration>"""

creds = Credentials("vaults3-admin", "vaults3-secret-change-me")
req = AWSRequest(method="PUT", url=url, data=body, headers={"Content-Type": "application/xml"})
SigV4Auth(creds, "s3", "us-east-1").add_auth(req)
requests.put(url, headers=dict(req.headers), data=body)

# All new objects now get 30-day GOVERNANCE retention automatically
s3.put_object(Bucket='my-bucket', Key='file.txt', Body=b'protected')
# file.txt cannot be deleted for 30 days

Modes: GOVERNANCE (admin can bypass with special header) or COMPLIANCE (nobody can shorten/remove until expiry). Requires versioning to be enabled on the bucket.

Per-Bucket Prometheus Metrics

The /metrics endpoint includes per-bucket counters (limited to top 100 buckets):

vaults3_bucket_requests_total{bucket="my-bucket",method="PUT"} 42
vaults3_bucket_requests_total{bucket="my-bucket",method="GET"} 156
vaults3_bucket_bytes_in_total{bucket="my-bucket"} 10485760
vaults3_bucket_bytes_out_total{bucket="my-bucket"} 52428800
vaults3_bucket_errors_total{bucket="my-bucket"} 3

These complement the existing global metrics and per-bucket storage metrics, enabling monitoring and alerting per bucket.

Rate Limiting

Protect against abuse and DDoS with token bucket rate limiting:

rate_limit:
  enabled: true
  requests_per_sec: 100   # per client IP
  burst_size: 200
  per_key_rps: 50         # per access key
  per_key_burst: 100

When enabled, each client IP and access key gets an independent token bucket. Requests exceeding the limit receive 429 Too Many Requests with a Retry-After: 1 header. Stale buckets are cleaned up after 5 minutes of inactivity.

# Check rate limiter status
curl http://localhost:9000/api/v1/ratelimit/status -H "Authorization: Bearer <token>"

Test with mc (MinIO Client)

mc alias set vaults3 http://localhost:9000 vaults3-admin vaults3-secret-change-me
mc mb vaults3/my-bucket
mc cp file.txt vaults3/my-bucket/
mc ls vaults3/my-bucket/
mc cat vaults3/my-bucket/file.txt

Security

VaultS3 is designed with security in mind:

S3 Signature V4: full signature verification including presigned URLs
Presigned URL validation: signature, expiry, and restrictions enforced server-side
Constant-time credential comparison: crypto/hmac.Equal prevents timing attacks on login
Path traversal protection: .. segments rejected at S3, API, versioning API, CopyObject/UploadPartCopy source, and filesystem layers
SSRF prevention: webhook, lambda, and notification URLs blocked from targeting localhost, private IPs, and cloud metadata endpoints
Upload size limits: 5GB per PUT (S3 spec), enforced with http.MaxBytesReader
Rate limiting: per-IP token bucket using RemoteAddr (not spoofable via X-Forwarded-For)
AES-256-GCM encryption at rest: SSE-S3 (static key) and SSE-KMS (HashiCorp Vault / local key) modes
IAM with default-deny: policy evaluation engine with wildcard matching
Security headers: CSP, HSTS, X-Frame-Options, X-Content-Type-Options, Referrer-Policy
Non-root Docker: container runs as vaults3 user (UID 1000)
Default credential warning: startup log warns if admin credentials haven't been changed
Error message sanitization: OIDC and health check errors return generic messages, preventing internal detail leaking
Race condition safety: Replication handler creates per-request struct copy instead of mutating shared state
UploadID validation: Hex-only regex validation prevents path traversal via crafted multipart upload IDs
Bounded request bodies: All JSON API endpoints use readJSON() with 1MB io.LimitReader. Bucket policy body capped at 1MB
OIDC SSRF prevention: Issuer URL validated against loopback, private, and link-local addresses before JWKS discovery
IPv6-safe rate limiting: Uses net.SplitHostPort for correct IP extraction from IPv6 [::1]:port addresses
OIDC authorization layer: Dashboard admin routes (IAM, keys, STS, audit, settings, lambda, backups) restricted to admin user. OIDC users get read-only access
Encryption size cap: 1GB max object size for encrypted reads/writes prevents OOM from 3x RAM amplification
Compression size cap: 1GB max decompressed size prevents decompression-bomb DoS (gzip/zstd)
Version path traversal protection: versionId parameter validated against directory escape in version storage
BatchDelete lock enforcement: Batch delete respects WORM/legal-hold and validates keys against path traversal
SigV4 timestamp validation: Requests with X-Amz-Date skewed more than 15 minutes are rejected (prevents replay)
Presigned URL expiry cap: Maximum 7 days (604800 seconds), matching AWS behavior
Atomic file writes: PutObject writes to temp file then renames, preventing corruption from concurrent writes
Backup scheduler thread safety: Atomic bool prevents concurrent backup races
OIDC admin name reservation: OIDC users cannot claim the "admin" username
OIDC domain validation enforcement: Tokens without email are rejected when domain filtering is enabled
OIDC code flow hardening: The authorization code is redeemed server-side, so the ID token never travels through the browser. The PKCE verifier, nonce and client secret stay on the server; the CSRF state is sealed with AES-GCM and expires after 15 minutes, so it cannot be read, forged, or replayed from another deployment. The ID token's nonce is checked against the login that requested it
CORS port restriction: Localhost CORS only allowed on the server's own port
Presigned URL credential isolation: Presigned URLs use a dedicated non-admin key, preventing privilege escalation
CORS Host header protection: Origin validation uses configured server address, not attacker-controlled Host header
Admin-only route expansion: Backup, replication, scanner, and tiering endpoints restricted to admin users
Replication queue limit cap: Queue listing capped at 1000 entries to prevent memory exhaustion
Backup path traversal protection: Backup target validates resolved paths stay within base directory
LIKE pattern O(n*m) matching, Iterative DP algorithm replaces recursive backtracking, preventing ReDoS
Tiering promotion safety: Async cold-to-hot promotion re-checks tier state and orders operations safely
Lambda output key validation: Output key template expansion validated against path traversal
S3 Select record cap: JSON/CSV parsing capped at 1M records to prevent memory exhaustion
FUSE cache size caps: Signature cache, HEAD cache, and LIST cache bounded to prevent unbounded memory growth
GetObjectAttributes version support: Respects versionId parameter and handles delete markers
LDAP authentication: Bind-based LDAP/LDAPS with group-to-policy mapping and TLS support
External auth webhook: Delegate authentication to an external HTTP endpoint with request validation
KMS envelope encryption: HashiCorp Vault and local key provider for data encryption key management
Auto-TLS: Automatic Let's Encrypt certificate provisioning with self-signed fallback
PROXY protocol v1: Real client IP extraction behind PROXY protocol-aware load balancers
Governance bypass protection: x-amz-bypass-governance-retention restricted to authorized principals
IAM policy conditions: StringEquals, StringLike, IpAddress, DateLessThan condition evaluation
Bucket bandwidth throttling: Per-bucket upload/download rate limits prevent resource monopolization
POST policy validation: HTML form upload policies validated for expiration, conditions, and signature
Content-MD5 validation: Server-side integrity verification on PUT rejects corrupted uploads
S3 Checksum API: CRC32, CRC32C, SHA1, SHA256 checksums verified on upload and returned on download
Conditional request handling: If-Match/If-None-Match ETag checks prevent lost updates (412 Precondition Failed)
Dependency hygiene: Dashboard dependencies kept current against Dependabot advisories (latest: react-router 7.18.1 and postcss 8.5.24, closing a backslash open redirect in <Link>/useNavigate, an unauthenticated route-matching DoS, an SSR hydration constructor injection, an RSCErrorHandler XSS, and a postcss path traversal; earlier: react-router 7.17.0 closing 6 alerts, turbo-stream RCE, RSC/Location XSS, __manifest/single-fetch DoS, protocol-relative open redirect). The one advisory left open is a React Server Components CSRF bypass that is only patched in react-router 8.x: the dashboard is a client-rendered SPA and never uses RSC mode, so it is not affected, and 8.x would additionally require Node 22.22+ and dropping react-router-dom

See SECURITY.md for vulnerability reporting policy and deployment best practices.

Project Structure

VaultS3/
├── cmd/vaults3/main.go        — Server entry point
├── cmd/vaults3-cli/           — CLI tool (bucket, object, user, replication, cluster commands)
├── internal/
│   ├── config/                — YAML config loader
│   ├── server/                — HTTP server, routing, and auto-TLS
│   ├── s3/                    — S3 API handlers (auth, buckets, objects, multipart, checksums, preconditions, replication config, restore, POST upload, snowball)
│   ├── storage/               — Storage engine interface + filesystem + encryption + KMS + storage classes
│   ├── metadata/              — BoltDB metadata store
│   ├── metrics/               — Prometheus-compatible metrics collector
│   ├── iam/                   — IAM policy engine, identity, IP access control, conditions, LDAP, external auth, STS AssumeRole
│   ├── notify/                — Event notification dispatcher (webhook, Kafka, NATS, Redis, AMQP, PostgreSQL, Elasticsearch)
│   ├── replication/           — Async + active-active replication (SigV4 signer, queue processor, vector-clock conflict resolution)
│   ├── erasure/               — Reed-Solomon erasure coding + background healer (multi-disk shard placement)
│   ├── cluster/               — Raft metadata, consistent-hash ring, failure detector, failover proxy, rebalancer
│   ├── search/                — In-memory full-text search index
│   ├── vector/                — Optional vector store: cosine kNN index + OpenAI-compatible embedder (semantic search / RAG)
│   ├── migrate/               — Import from any S3-compatible source (MinIO/AWS/...): SigV4 source client + async migrator
│   ├── snapshot/              — Bucket snapshots ("git-for-buckets"): commit / diff / restore on version pointers
│   ├── scanner/               — Webhook virus scanning with quarantine
│   ├── ratelimit/             — Token bucket rate limiter (per IP, per key, per bucket bandwidth)
│   ├── tiering/               — Hot/cold data tiering manager + remote S3-compatible tier
│   ├── backup/                — Backup scheduler with local targets
│   ├── versioning/            — Version diff (LCS), tagging, rollback
│   ├── fuse/                  — FUSE filesystem mount (go-fuse/v2)
│   ├── middleware/             — HTTP middleware (request ID, panic recovery, latency, security headers, PROXY protocol)
│   ├── api/                   — Dashboard REST API (JWT auth, IAM, STS, audit, events, logs, trace, diagnostics, heal, speedtest)
│   ├── batch/                 — Batch operations processor (bulk delete/copy)
│   ├── inventory/             — S3 Inventory report generator (periodic CSV)
│   └── dashboard/             — Embedded React SPA
├── web/                       — React dashboard source (Vite + Tailwind)
├── configs/vaults3.yaml       — Default configuration
├── Makefile                   — Build commands
├── Dockerfile                 — Multi-stage Docker build
└── README.md

Tech Stack

Go: net/http (no frameworks)
React 19: Dashboard UI (embedded via //go:embed)
Tailwind CSS: Dashboard styling
BoltDB: Embedded key-value store for metadata
Local filesystem: Object storage backend
AES-256-GCM: Server-side encryption (SSE-S3 and SSE-KMS with HashiCorp Vault)

Requirements

Go 1.26+ (build)
Node.js 20.19+ (dashboard build only, Vite 8 / Rolldown)
No runtime dependencies

Updating

VaultS3 can check GitHub Releases once a day and show a dashboard banner when a newer version is out. Updates only ever replace the binary or image, your object data, metadata, and config are never touched.

Docker (recommended): Watchtower watches for a new image and recreates the container. Your data volumes are preserved:

services:
  vaults3:
    image: eniz1806/vaults3:latest
    volumes: [vaults3-data:/data, vaults3-meta:/metadata]
  watchtower:
    image: containrrr/watchtower
    volumes: [/var/run/docker.sock:/var/run/docker.sock]
    command: --interval 86400   # check daily

Binary / systemd: enable the built-in updater in vaults3.yaml. With apply: true it downloads the new release for your platform, verifies its SHA-256 checksum, swaps the binary, and restarts into the new version (checked daily. Never auto-crosses a major version):

auto_update:
  enabled: true     # daily check + dashboard banner
  apply: true       # also install automatically (omit for notify-only)

The current/latest version is also exposed at GET /api/v1/version.

Contributing

Contributions are welcome! See CONTRIBUTING.md for build, test, and PR guidelines, and CHANGELOG.md for release notes.

make build                      # React dashboard + server + CLI binaries
go test ./...                   # run the Go test suite
go test -race ./internal/erasure/ ./internal/cluster/ ./internal/replication/

The data-durability subsystems (erasure, cluster, replication) carry fault-injection tests, corrupt a shard and heal it, lose a node and re-route, partition two sites and resolve the conflict. New logic there should keep that bar.

Name		Name	Last commit message	Last commit date
Latest commit History 193 Commits
.github		.github
assets		assets
cmd		cmd
configs		configs
deploy		deploy
docs		docs
internal		internal
signatures/version1		signatures/version1
web		web
.dockerignore		.dockerignore
.gitignore		.gitignore
CHANGELOG.md		CHANGELOG.md
CONTRIBUTING.md		CONTRIBUTING.md
DOCKERHUB.md		DOCKERHUB.md
Dockerfile		Dockerfile
LICENSE		LICENSE
Makefile		Makefile
README.md		README.md
SECURITY.md		SECURITY.md
go.mod		go.mod
go.sum		go.sum

Uh oh!

Folders and files

Latest commit

History

Repository files navigation

Why VaultS3?

Production Readiness

Features

Supported S3 Operations

Quick Start

Build

Run

Configure

Encryption at Rest

Virtual-Hosted Style URLs

Prometheus Metrics

Web Dashboard

Language

Screenshots

Health Checks

TLS

Docker

Buckets on First Start

Environment Variables

Storage requirements

Kubernetes

Object Versioning

Object Locking (WORM)

Lifecycle Rules

Compression

Small-file packing (experimental)

Access Logging

Static Website Hosting

IAM (Users, Groups & Policies)

CORS per Bucket

STS Temporary Credentials

Audit Trail

IP Restrictions

S3 Event Notifications

Async Replication

CLI Tool

Presigned Upload Restrictions

Full-Text Search

Webhook Virus Scanning

Data Tiering

Backup Scheduler

Git-like Versioning

FUSE Mount

S3 Select (SQL on Objects)

Bucket Default Retention

Per-Bucket Prometheus Metrics

Rate Limiting

Test with mc (MinIO Client)

Security

Project Structure

Tech Stack

Requirements

Updating

Contributing

Roadmap

About

Resources

Contributing

Security policy

Stars

Watchers

Forks

Releases

Packages

Used by

Contributors

Languages