How to restrict tmp dir usage

[Peiqi0714]

If user write in tmp file mode, I'm worried about the diskusage. Is there any way to restrict it?

[vipnydav]

Hi @Peiqi0714

Since GCS objects are immutable, writing typically requires overwriting the entire file. We currently support streaming writes, which upload data directly without using local disk, but this is limited to newly created or empty files undergoing sequential writes. For random writes, GCSFuse currently falls back to legacy writes (using temporary local files), as streaming support for random I/O is not yet available.

[vipnydav]

Hi @Peiqi0714,

If you want to stage writes on a specific disk or in memory to avoid using the default /tmp partition, you can use the temp-dir flag. This flag allows you to specify the directory where files are staged prior to being uploaded to GCS.

[Peiqi0714]

Hi @Peiqi0714,

If you want to stage writes on a specific disk or in memory to avoid using the default /tmp partition, you can use the temp-dir flag. This flag allows you to specify the directory where files are staged prior to being uploaded to GCS.

Yes, I know what you mean. But there is no conf to restrict the usage of tmp dir, if users open too many files to write at the same time, the disk usage of tmp dir maybe huge. Even worse, gcsfuse may can not create tmp file in this dir.

[vipnydav]

That is a valid concern. However, enforcing a limit on temp-dir usage could cause write operations to stall until space becomes available. This blocking behavior risks creating deadlock-like situations, which we aim to avoid to ensure a stable user experience.

CC : @ashmeenkaur.

[Peiqi0714]

That is a valid concern. However, enforcing a limit on temp-dir usage could cause write operations to stall until space becomes available. This blocking behavior risks creating deadlock-like situations, which we aim to avoid to ensure a stable user experience.

CC : @ashmeenkaur.

But if you don‘t enforce a limit on tmp-dir, when it don‘t have enough space, we can't create tmp file, users will receive i/o error. I think this is worse than stall user writes.

[vipnydav]

We prefer returning an EIO over implementing a temp-dir limit due to the risk of deadlocks. A hard limit could cause multiple concurrent write operations to block each other, if all open handles are waiting for space to free up, they won't be able to complete the syncs necessary to release that space. We would also face issues where a single file's size exceeds the defined limit, creating a deadlock situation with no easy fix. For these reasons, letting the operation fail with an I/O error is currently the better alternative to a total stall.

[Peiqi0714]

We prefer returning an EIO over implementing a temp-dir limit due to the risk of deadlocks. A hard limit could cause multiple concurrent write operations to block each other, if all open handles are waiting for space to free up, they won't be able to complete the syncs necessary to release that space. We would also face issues where a single file's size exceeds the defined limit, creating a deadlock situation with no easy fix. For these reasons, letting the operation fail with an I/O error is currently the better alternative to a total stall.

Thanks！

[Peiqi0714]

@vipnydav but sometimes，when enable kernel writeback caching，pwrite will stall, not return disk full error

[Peiqi0714]

@vipnydav , I want to reopen this issue to discuss it

[vipnydav]

Reopening the issue and assigning to @PranjalC100 who is current oncall.

[PranjalC100]

Hi @Peiqi0714 , thanks for reporting and following up on this.

When Kernel Writeback Cache is enabled, pwrite() calls succeed instantly by buffering to RAM, and failures happen silently in the background. The failure happens asynchronously in the background, and errors only become visible to the application later when it attempts to flush or close() the file.

Depending on the kernel, this deferred error leads to:

• Total Deadlock: The kernel stalls the app in pwrite() when RAM fills up. Because it hangs, the app can never call close(), meaning GCSFuse never uploads the file or frees the temp space.
• Silent Data Loss: The kernel drops failed pages from RAM. The app thinks the writes succeeded, but the data is lost and never reaches GCSFuse. The app only discovers the failure when closing the file, or not at all if it crashes before then.

Since writeback caching negates the immediate error and forces stalls or data loss anyway, we should reconsider an explicit mechanism to monitor temp-dir usage and fail early.

cc @trinadhkotturu , what is your opinion on this?