Skip to content

perf(memory): Minimizing the gcs.MinObject memory footprint#4830

Draft
raj-prince wants to merge 1 commit into
masterfrom
min_object_time
Draft

perf(memory): Minimizing the gcs.MinObject memory footprint#4830
raj-prince wants to merge 1 commit into
masterfrom
min_object_time

Conversation

@raj-prince

Copy link
Copy Markdown
Collaborator

Description

Optimized the memory footprint of gcs.MinObject by changing the representation of its Updated and Finalized timestamp fields from time.Time (24 bytes each) to int64 Unix nanoseconds (8 bytes each).

Rationale

gcs.MinObject is the core structure used to represent GCS objects in the metadata stat cache and is embedded by value in FileInode and SymlinkInode. Storing them as time.Time is memory-inefficient because time.Time carries additional overhead (like timezone pointers and monotonic clock readings) that are unnecessary for static caching.

By converting these to int64 nanoseconds, we save 16 bytes per timestamp, reducing the struct size significantly.

Changes

  1. gcs.MinObject Refactoring (object.go):
    • Changed Updated and Finalized fields to int64.
    • Added UpdatedTime() time.Time and FinalizedTime() time.Time helper methods to reconstruct time.Time on the fly for callers.
    • Added package-level TimeToNS(time.Time) int64 and NSToTime(int64) time.Time helpers to handle safe conversion, specifically protecting against panics when calling UnixNano() on Go's zero time.Time value.
  2. Production Code Updates:
    • storageutil (object_attrs.go): Updated conversion helpers (ObjectAttrsToMinObject, ConvertObjToMinObject, etc.) to convert timestamps using the new helpers.
    • fake GCS (bucket.go): Updated copyMinObject helper.
    • gcsx (garbage_collect.go): Updated stale object age calculation to use o.UpdatedTime().
    • inode (file.go, symlink.go): Updated attribute mapping and constructor code to use UpdatedTime() and gcs.TimeToNS().
  3. Test Suite Refactoring:
    • Updated all unit and integration tests across util, storage, gcsx, inode, and handle packages to use int64 literals or gcs.TimeToNS() in MinObject struct literals.
    • Fixed assert.Equal and ExpectThat assertions in tests (including the shared GCS bucket test suite in bucket_tests.go) to compare time.Time with UpdatedTime() / FinalizedTime(), preventing runtime type mismatch failures.
    • Simplified timestamp comparisons in local_modifications_test.go and symlink_test.go to use direct int64 integer comparisons (< and ==) instead of time.Time.Before() / Compare().

Impact & Memory Savings (Verified via unsafe.Sizeof)

  • gcs.MinObject size: 120B → 88B (-32 bytes / 26.7% reduction).
  • Stat Cache: Saves 32 MB of RAM per 1 million cached objects.
  • Inodes: Since FileInode embeds MinObject by value, its size automatically shrank by 32 bytes (488B → 456B) on master, saving another 32 MB of RAM per 1 million cached files.
  • Total Resident Memory Saved: ~64 MB of RAM saved permanently for a mount caching 1 million files.

All tests compile and pass successfully.

Link to the issue in case of a bug fix.

Testing details

  1. Manual - NA
  2. Unit tests - NA
  3. Integration tests - NA

Any backward incompatible change? If so, please explain.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant