Skip to content
Open
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 24 additions & 14 deletions docs/features/identifiers.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,15 +57,15 @@ private token and generation never appear in public lease records. Fixed-ID
to own replay, and caller cancellation never releases them.

Automation may instead supply the canonical ID with `warmup --lease-id`. For
direct AWS, direct Machine0, direct local-container, and managed coordinator
leases, that ID is an immutable create identity: an identical semantic replay
returns the same lease, while intent drift returns `lease_id_conflict`. External
providers also accept requested IDs when their protocol explicitly advertises
idempotent lease identity support. The coordinator durably stores a versioned
normalized request hash. Direct AWS durably stores the intent and current
resolved EC2 attempt in the normal lease claim before `RunInstances`, then uses
a deterministic regional/zonal client token. No path uses the slug to decide
replay ownership.
direct AWS, direct Machine0, direct local-container, direct Proxmox, and managed
coordinator leases, that ID is an immutable create identity: an identical
semantic replay returns the same lease, while intent drift returns
`lease_id_conflict`. External providers also accept requested IDs when their
protocol explicitly advertises idempotent lease identity support. The
coordinator durably stores a versioned normalized request hash. Direct AWS
durably stores the intent and current resolved EC2 attempt in the normal lease
claim before `RunInstances`, then uses a deterministic regional/zonal client
token. No path uses the slug to decide replay ownership.

Direct Machine0 binds the intent to its deterministic VM name before creation;
the durable attempt binds the first visible match to its Machine0 resource ID,
Expand All @@ -81,6 +81,15 @@ missing acquired container fails closed instead of starting another container.
Its fixed claims use the downgrade-safe `local-container-fixed-v1` marker, so
older clients cannot mistake them for ordinary local-container claims.

Direct Proxmox selects a free VMID through the cluster allocator, then durably
binds that exact VMID, the normalized intent, source node, and cluster scope
before submitting the template clone with an explicit `newid`. Replay inspects
that VMID and requires matching lease labels, intent fingerprint, provider
scope, and native `vmgenid`; it never derives a VMID from the lease ID or adopts
by slug. Missing or ambiguous post-submit state retains the attempt and cannot
issue another clone. Its fixed claims use the downgrade-safe
`proxmox-fixed-v1` marker.

After the direct AWS launch attempt is durable, Crabbox never submits that
attempt again. An ambiguous replay with no visible tagged instance fails closed;
a later replay can adopt the one instance after inventory converges only when
Expand All @@ -89,11 +98,12 @@ match the persisted attempt exactly. Fixed AWS
claims use the downgrade-safe local discriminator `aws-fixed-v1`; current
clients map it to runtime AWS, while older clients skip/refuse it.

Fixed IDs are single-use operation identities. Direct AWS, Machine0, and
local-container keep a compact terminal claim tombstone after successful
destroy release or exact missing-resource cleanup. Tombstones contain only the
ID, slug, provider scope, versioned intent hash, timestamps, and terminal
state; automatic provider cleanup never prunes them.
Fixed IDs are single-use operation identities. Direct AWS, Machine0,
local-container, and Proxmox keep a terminal claim tombstone after successful
destroy release or exact missing-resource cleanup. Proxmox retains the selected
VMID and, when observed, its native generation identity so release
reconciliation remains exact. Automatic provider cleanup never prunes fixed-ID
tombstones.
There is no time-based reuse window. Explicitly deleting local Crabbox claim
state forfeits this replay protection, so automation must instead mint a new
operation ID.
Expand Down
18 changes: 17 additions & 1 deletion docs/providers/proxmox.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,9 @@ then drives the normal SSH sync/run/release path.
The provider is direct-only: it talks to the Proxmox API straight from the CLI.
The Crabbox coordinator (broker) does not provision or broker Proxmox capacity,
so brokered shared-team leases are not available here. Proxmox supports the
`ssh`, `crabbox-sync`, and `cleanup` features on `target=linux` only.
`ssh`, `crabbox-sync`, and `cleanup` features on `target=linux` only. Direct
Proxmox also supports caller-supplied fixed lease IDs with
`warmup --lease-id cbx_<12 lowercase hex>`.

## When to use

Expand Down Expand Up @@ -378,6 +380,20 @@ only to discover candidates; they never authorize deletion by themselves.
Failed acquisition cleanup removes the per-lease SSH key only after confirming
the VM is absent across the cluster.

For an ordinary acquire, step 2 remains a per-create `/cluster/nextid` lookup.
For `warmup --lease-id`, Crabbox instead persists the normalized create intent,
selected VMID, source node, and cluster scope in the fixed lease claim before
submitting the clone, then passes that exact VMID as the clone API's `newid`.
An identical replay inspects the persisted VMID and adopts only the VM whose
lease labels, intent fingerprint, cluster scope, VMID, and native `vmgenid`
match. Slugs are never replay authority. A changed intent, copied labels,
different VMID or generation, duplicate match, or unresolved post-submit
attempt returns `lease_id_conflict` without issuing another clone.

Successful fixed-ID release, including authoritative confirmation that the
selected VMID is absent, retains a terminal local tombstone. The fixed lease ID
is single-use and cannot allocate another VM after release.

### Automatic cleanup ownership

Cleanup requires exactly one local claim matching the provider, configured API
Expand Down
3 changes: 3 additions & 0 deletions internal/cli/claim.go
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,7 @@ type FixedCreateIntent struct {
const FixedAWSClaimProvider = "aws-fixed-v1"
const FixedMachine0ClaimProvider = "machine0-fixed-v1"
const FixedLocalContainerClaimProvider = "local-container-fixed-v1"
const FixedProxmoxClaimProvider = "proxmox-fixed-v1"

const maxLocalClaimInventoryFileBytes int64 = 1 * 1024 * 1024

Expand Down Expand Up @@ -1180,6 +1181,8 @@ func canonicalClaimProvider(provider string) string {
return "machine0"
case FixedLocalContainerClaimProvider:
return "local-container"
case FixedProxmoxClaimProvider:
return "proxmox"
case "exec-provider":
return "external"
}
Expand Down
6 changes: 6 additions & 0 deletions internal/cli/claim_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,12 @@ func TestFixedLocalContainerClaimProviderCanonicalizesWithoutOverwritingMarker(t
}
}

func TestFixedProxmoxClaimProviderCanonicalizes(t *testing.T) {
if got := canonicalClaimProvider(FixedProxmoxClaimProvider); got != "proxmox" {
t.Fatalf("fixed Proxmox marker canonicalized to %q", got)
}
}

func TestClaimEndpointReservationDeadlineStartsAfterClaimLockAcquired(t *testing.T) {
t.Setenv("XDG_STATE_HOME", t.TempDir())
const leaseID = "cbx_reservation_lock"
Expand Down
20 changes: 17 additions & 3 deletions internal/cli/proxmox.go
Original file line number Diff line number Diff line change
Expand Up @@ -838,6 +838,10 @@ func (c *ProxmoxClient) nextID(ctx context.Context) (int, error) {
}
}

func (c *ProxmoxClient) NextVMID(ctx context.Context) (int, error) {
return c.nextID(ctx)
}

type proxmoxVM struct {
VMID int `json:"vmid"`
Name string `json:"name"`
Expand Down Expand Up @@ -998,15 +1002,22 @@ func (c *ProxmoxClient) VMExistsInCluster(ctx context.Context, id string) (bool,
}

func (c *ProxmoxClient) CreateServer(ctx context.Context, cfg Config, publicKey, leaseID, slug string, keep bool) (Server, error) {
vmid, err := c.nextID(ctx)
if err != nil {
return Server{}, err
}
return c.CreateServerWithVMID(ctx, cfg, publicKey, leaseID, slug, keep, vmid, nil)
}

func (c *ProxmoxClient) CreateServerWithVMID(ctx context.Context, cfg Config, publicKey, leaseID, slug string, keep bool, vmid int, extraLabels map[string]string) (Server, error) {
if cfg.TargetOS != targetLinux {
return Server{}, exit(2, "proxmox provider currently supports target=linux only")
}
if cfg.Proxmox.TemplateID <= 0 {
return Server{}, exit(3, "proxmox templateId is required (set proxmox.templateId or CRABBOX_PROXMOX_TEMPLATE_ID)")
}
vmid, err := c.nextID(ctx)
if err != nil {
return Server{}, err
if vmid <= 0 {
return Server{}, exit(2, "proxmox VMID must be positive")
}
name := leaseProviderName(leaseID, slug)
full := "1"
Expand Down Expand Up @@ -1041,6 +1052,9 @@ func (c *ProxmoxClient) CreateServer(ctx context.Context, cfg Config, publicKey,

now := time.Now().UTC()
labels := directLeaseLabels(cfg, leaseID, slug, "proxmox", "", keep, now)
for key, value := range extraLabels {
labels[key] = value
}
labels["node"] = cfg.Proxmox.Node
labels["template_id"] = strconv.Itoa(cfg.Proxmox.TemplateID)
description := proxmoxDescription(labels)
Expand Down
37 changes: 37 additions & 0 deletions internal/cli/proxmox_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -1421,6 +1421,43 @@ func TestProxmoxCreateServerCleansUpCloneOnConfigFailure(t *testing.T) {
}
}

func TestProxmoxCreateServerWithVMIDSkipsNextIDAndUsesExplicitCloneTarget(t *testing.T) {
nextIDCalls := 0
var clone url.Values
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch {
case r.Method == http.MethodGet && r.URL.Path == "/api2/json/cluster/nextid":
nextIDCalls++
t.Fatal("explicit VMID create queried /cluster/nextid")
case r.Method == http.MethodPost && r.URL.Path == "/api2/json/nodes/pve1/qemu/9000/clone":
clone = readForm(t, r)
http.Error(w, "stop after clone request", http.StatusInternalServerError)
default:
t.Fatalf("%s %s", r.Method, r.URL.String())
}
}))
defer server.Close()

cfg := baseConfig()
cfg.Provider = "proxmox"
cfg.Proxmox.APIURL = server.URL
cfg.Proxmox.TokenID = "runner@pve!crabbox"
cfg.Proxmox.TokenSecret = "secret"
cfg.Proxmox.Node = "pve1"
cfg.Proxmox.TemplateID = 9000
client, err := NewProxmoxClient(cfg)
if err != nil {
t.Fatal(err)
}
_, err = client.CreateServerWithVMID(context.Background(), cfg, "ssh-ed25519 AAAA test", "cbx_123456abcdef", "blue-crab", false, 417, nil)
if err == nil {
t.Fatal("expected fixture clone failure")
}
if nextIDCalls != 0 || clone.Get("newid") != "417" {
t.Fatalf("nextIDCalls=%d clone=%v", nextIDCalls, clone)
}
}

func testProxmoxClient(t *testing.T, serverURL string) *ProxmoxClient {
t.Helper()
cfg := baseConfig()
Expand Down
Loading