Skip to content

fix(filesystem): default bucket_url to '.' for bare destination string#3955

Open
Bhagwat-45 wants to merge 1 commit into
dlt-hub:develfrom
Bhagwat-45:feat/3886-filesystem-bare-destination
Open

fix(filesystem): default bucket_url to '.' for bare destination string#3955
Bhagwat-45 wants to merge 1 commit into
dlt-hub:develfrom
Bhagwat-45:feat/3886-filesystem-bare-destination

Conversation

@Bhagwat-45
Copy link
Copy Markdown

Description

Allows bare dlt.pipeline(..., destination="filesystem") without requiring bucket_url to be set in config.toml, matching the existing behavior of the duckdb destination.

Previously this would throw a ConfigFieldMissingException. The fix sets bucket_url = "." (current directory) as the default in FilesystemConfiguration. The existing normalize_bucket_url() method already handles "." correctly, converting it to an absolute file:// URL at resolve time — so this is a minimal, safe one-line change.

Files changed:

  • dlt/common/storages/configuration.py — set bucket_url: str = "." as the default
  • tests/sources/filesystem/test_config_sections.py — rewrote test to reflect new default; added test verifying config section path for cloud destinations (which still require credentials)
  • tests/load/filesystem/test_filesystem_factory.py — updated stale comment
  • tests/workspace/cli/test_config_toml_writer.py — updated assertion and comment

Related Issues

Additional Context

Tests were run locally and passed:

pytest tests/common/storages/test_local_filesystem.py \
       tests/sources/filesystem/test_config_sections.py \
       tests/load/filesystem/test_filesystem_factory.py \
       tests/workspace/cli/test_config_toml_writer.py -v

Result: 55 passed, 21 skipped, 0 failed.

@zilto zilto self-requested a review May 20, 2026 17:24
@zilto zilto self-assigned this May 20, 2026
@rudolfix
Copy link
Copy Markdown
Collaborator

rudolfix commented May 22, 2026

@Bhagwat-45 by setting a default you are making bucket_url optional. this will impact a lot of users on filesystems different than local. same configuration is shared with ducklake, lance and iceberg catalogs. tbh. it will be very confusing if instead of failing with missing configuration, files are created in locally. very hard to diagnose and also leaks data into place where it should not be. my take is that you add the following to your global config toml (so all your projects use it)

[credentials]
bucket_url="file:///."

(I didnt check it, pwd may be slightly different)
this will act as an default that any more specific configuration can overwrite

@zilto
Copy link
Copy Markdown
Collaborator

zilto commented May 22, 2026

@rudolfix the initial issue was for quality of life and supporting, similar to destination="duckdb"

dlt.pipeline(..., destination="filesystem")

Setting up config.toml or dlt.destinations.filesystem(bucket_url=...) can trip up new users and agents

Potential solutions

  • IMHO, it makes sense to have filesystem default to local. It doesn't make sense to expect your pipeline will write to S3 if you didn't set any credentials
  • The tricky part is how filesystem credentials are used by other destinations. In these cases, can the "parent config" set default value to None / missing? For example, ducklake could remove bucket_url="."

If this is too complicated, we should avoid changes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(filesystem): allow bare dlt.pipeline(..., destination="filesystem")

3 participants