Slowdown of data staging transfer

[scwatts]

Background

• Inputs are transferred from GDS to AWS on a single r5dn.large instance
• Transfer facilitated by rclone, streaming via the instance
• Slowdown of transfer speed often occurs and is seen ~1 hour into transfer

This behaviour is probably likely related to network bursting. It hasn't been a significant issue for average sized datasets (~200-250 GB) but may cause issues for very large inputs. If so, it is potentially solvable by spinning out a better resourced network-optimised instance or individual instances for each file transfer.

[brainstorm]

Also might help to use a recent CLI that has throughput optimizations: https://aws.amazon.com/blogs/storage/improving-amazon-s3-throughput-for-the-aws-cli-and-boto3-with-the-aws-common-runtime/