Skip to content

feat(cache): add ListableCache interface for enumerating cache keys#940

Open
ecordell wants to merge 1 commit into
warpstreamlabs:mainfrom
ecordell:listable-cache
Open

feat(cache): add ListableCache interface for enumerating cache keys#940
ecordell wants to merge 1 commit into
warpstreamlabs:mainfrom
ecordell:listable-cache

Conversation

@ecordell

Copy link
Copy Markdown
Contributor

Description

Adds an optional ListableCache interface to the service package for caches that are able to enumerate the keys they hold:

type ListableCache interface {
	Cache

	// ListKeys returns a slice of all keys currently held by the cache.
	ListKeys(ctx context.Context) ([]string, error)
}

Not all caches can support key listing, so support is detected via type assertion on the cache obtained from *Resources.AccessCache. The air gap, metrics and reverse air gap wrappers propagate the capability only when the underlying cache implements it, so the assertion stays truthful through the whole chain. The metrics wrapper emits the standard cache metrics with operation="list_keys".

Implementations included: memory (walks shards, omits TTL-expired items to match Get semantics), file (recursive directory walk), aws_s3 (paginated ListObjectsV2 with the cache's existing backoff pattern) and gcp_cloud_storage (bucket object iterator). The mock cache behind MockResourcesOptAddCache also implements it so plugin authors can test listable-cache consumers.

Motivation: components consuming a cache resource currently have no way to enumerate its contents, so patterns like "replay everything stored in this cache" require maintaining a separate index entry alongside the cached items. A follow-up PR adds a cache input built on this interface.

Testing

  • Unit tests for the type-assertion passthrough at each wrapper layer, including a round trip through all of them
  • ListKeys unit tests for memory (including shards and TTL expiry) and file (including nested directories)
  • New CacheTestListKeys integration test definition, wired into the aws_s3 localstack suite and verified locally

Adds an optional ListableCache interface to the service package for
caches that are able to enumerate the keys they hold. Consumers detect
support via type assertion on caches obtained from
*Resources.AccessCache; the air gap, metrics and reverse air gap
wrappers only expose ListKeys when the underlying cache implements it.

The memory, file, aws_s3 and gcp_cloud_storage caches implement the new
interface, as does the mock cache provided by MockResourcesOptAddCache.
The cache metrics wrapper emits standard cache metrics for the
list_keys operation, and a CacheTestListKeys definition is added to the
integration test framework.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant