Skip to content

Batch the order & copy from prescribing_norm_tmp#390

Open
madwort wants to merge 1 commit into
mainfrom
madwort/batch-rx-insert-from-tmp
Open

Batch the order & copy from prescribing_norm_tmp#390
madwort wants to merge 1 commit into
mainfrom
madwort/batch-rx-insert-from-tmp

Conversation

@madwort

@madwort madwort commented Jun 4, 2026

Copy link
Copy Markdown
Contributor
  • I tested this locally with a constrained environment:
    • 500MB RAM & 50GB tmp disk available to duckdb
    • just the 2026 data
  • For the current main ingest:
    • tmp disk usage reached 2.5GB whilst creating the tmp table
    • tmp disk usage reached 6.5GB whilst inserting from the tmp table to the actual table
  • For this branch ingest:
    • tmp disk usage reached 3.3GB whilst creating the tmp table
    • tmp disk upage remained at 3.3GB whilst inserting from the tmp table to the actual table
  • It was a surprise that the tmp table appeared to require more tmp disk space to do the same thing, however the overall usage was significantly lower
  • Wall clock time on my developer laptop was similar for this PR compared to main, however on the production server I would suspect that it would be quicker
    • this is because I think the tmp disk is on a slower hard-drive type device, and attached over a network, which I think it would make it relatively a lot more expensive than the storage on my developer laptop
    • the only real way to compare this is to experiment with it on production or a production-class machine

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant