Problem
See the issue raised by this comment: #131 (comment)
The /file endpoint doesn't use a streaming API, meaning that potentially large files must be put in memory (in their entirety) while the upload function processes the upload.
The current HAPI API doesn't seem to be streaming either due to parse: true, so this isn't exactly a regression, but an enhancement.
Solution
Implement a streaming consumer for the multipart upload so that large files can be processed with a smaller memory footprint.
Note that this issue has two parts for it to be useful. The first part is as described (replace current logic with streaming logic, server-side). However, Group Income currently uploads all files as a single chunk, regardless of size. We need to implement splitting files up there too for the effects of this improvement to be had. For this second part we also need to decide whether this is the app's responsibility or whether we want to add this functionality transparently to chelonia/fileUpload. (For example, we could automatically split the payload if we receive a Blob but leave the chunks untouched if we receive an array of blobs.)
Problem
See the issue raised by this comment: #131 (comment)
The
/fileendpoint doesn't use a streaming API, meaning that potentially large files must be put in memory (in their entirety) while the upload function processes the upload.The current HAPI API doesn't seem to be streaming either due to
parse: true, so this isn't exactly a regression, but an enhancement.Solution
Implement a streaming consumer for the multipart upload so that large files can be processed with a smaller memory footprint.
Note that this issue has two parts for it to be useful. The first part is as described (replace current logic with streaming logic, server-side). However, Group Income currently uploads all files as a single chunk, regardless of size. We need to implement splitting files up there too for the effects of this improvement to be had. For this second part we also need to decide whether this is the app's responsibility or whether we want to add this functionality transparently to
chelonia/fileUpload. (For example, we could automatically split the payload if we receive aBlobbut leave the chunks untouched if we receive an array of blobs.)