Skip to content

fix(job-runs): resolve upload path and S3 presigned URL signature errors#9

Open
haizhou-zhao wants to merge 1 commit intomainfrom
haizhou/fix-job-create-upload-path
Open

fix(job-runs): resolve upload path and S3 presigned URL signature errors#9
haizhou-zhao wants to merge 1 commit intomainfrom
haizhou/fix-job-create-upload-path

Conversation

@haizhou-zhao
Copy link
Copy Markdown

Summary

These bugs were discovered while using the CLI to submit a local Python file as a Wherobots job (via wherobots job-runs create test_connection.py --name test_connection --watch). All three blocked the upload step before the job could even be created.

Bug 1: /files/dir called with s3:// URI instead of bare bucket name (jobs.go)

The resolveManagedUploadTarget function constructed rootDir as s3://bucket/ and passed it as the dir query parameter. The /files/dir API only accepts a bare bucket name (e.g. wbts-wbc-m97rcg45xi), so the call failed with HTTP 400 InvalidInputException (No storage source found for bucket: s3:).

Fix: Pass the bucket name directly. Also normalize the response path field (which lacks the s3:// prefix) before parsing it with splitS3Path.

Bug 2: /files/upload-url key missing bucket prefix, and wrong upload target directory (jobs.go)

The key sent to /files/upload-url was prefix/name/file, but the API expects the full path starting with the bucket name: bucket/prefix/name/file. Without it, the server parsed the org ID as the bucket and returned HTTP 400 InvalidInputException (No storage source found for bucket: <org_id>).

Additionally, the upload was targeting the org-level managed storage root, which has uploadProtected: true. The actual writable directory per user is bucket/org_id/data/customer-{user_id}/. Added a GET /users/me call to resolve the caller's user ID and navigate to their personal directory before falling back to the legacy root path.

Fix: Prepend bucket to the upload key. Add getMe operation and resolve the user's personal directory first.

Bug 3: Content-Type: application/octet-stream breaks S3 presigned URL signature (upload.go)

AWS S3 Signature Version 2 includes Content-Type in the canonical string used to verify presigned URLs. The presigned URL from /files/upload-url is signed without a Content-Type constraint, but UploadFileToPresignedURL was unconditionally setting Content-Type: application/octet-stream on the PUT request. This caused HTTP 403 SignatureDoesNotMatch on every upload.

Fix: Remove the Content-Type header from the presigned PUT request.

Test plan

  • Run wherobots job-runs create <local-script.py> --name <name> --watch and verify it uploads, submits, and streams logs to completion
  • Verify the uploaded script appears at s3://bucket/org_id/data/customer-{user_id}/name/script.py
  • Verify --upload-path s3://custom/path override still works (code path unchanged)
  • Verify --no-upload with an s3:// URI still works (skips upload entirely)

🤖 Generated with Claude Code

Copy link
Copy Markdown

@claude claude bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review to trigger a review.

Tip: disable this comment in your organization's Code Review settings.

@haizhou-zhao haizhou-zhao changed the title Fix three bugs preventing job-runs create from uploading local scripts fix(job-runs): resolve upload path and S3 presigned URL signature errors Mar 23, 2026
@haizhou-zhao
Copy link
Copy Markdown
Author

@salty-hambot review

Discovered while submitting a local Python file via:
  wherobots job-runs create test_connection.py --name test_connection --watch

Three bugs blocked the upload step before the job could be created:

Bug 1 (jobs.go): /files/dir called with s3:// URI instead of bare bucket name
The dir query param was constructed as s3://bucket/ but the API only accepts a
bare bucket name. This caused HTTP 400 "No storage source found for bucket: s3:"
on every auto-upload attempt. Also normalize the response path field (which lacks
the s3:// prefix) before parsing with splitS3Path.

Bug 2 (jobs.go): upload key missing bucket prefix; wrong upload target directory
The key sent to /files/upload-url was prefix/name/file, but the API expects the
full path starting with the bucket: bucket/prefix/name/file. Without it, the
server parsed the org ID as the bucket and returned HTTP 400 "No storage source
found for bucket: <org_id>". Additionally, the upload targeted the org-level
managed storage root (uploadProtected: true). Added a GET /users/me call to
resolve the caller's user ID and navigate to their personal writable directory
(bucket/org_id/data/customer-{user_id}) before falling back to the legacy root.

Bug 3 (upload.go): Content-Type header breaks S3 presigned URL signature
AWS S3 Signature V2 includes Content-Type in the canonical string. The presigned
URL from /files/upload-url is signed without a Content-Type constraint, but
UploadFileToPresignedURL was setting Content-Type: application/octet-stream on
the PUT request. This caused HTTP 403 SignatureDoesNotMatch on every upload.
Fixed by removing the Content-Type header from the presigned PUT request.

Update three test assertions that encoded the old (buggy) behavior:
- Expected s3://managed-bucket/ dir param → now bare managed-bucket
- Expected key prefix flag-prefix/ → now flag-bucket/flag-prefix/
- Expected key prefix env-prefix/ → now env-bucket/env-prefix/

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@haizhou-zhao haizhou-zhao force-pushed the haizhou/fix-job-create-upload-path branch from 84167cb to f6a6eae Compare March 23, 2026 22:50
@ClayMav
Copy link
Copy Markdown
Member

ClayMav commented Apr 1, 2026

@haizhou-zhao was this tested manually?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants