Skip to content

/file endpoint should be streaming #133

Description

@corrideat

Problem

See the issue raised by this comment: #131 (comment)

The /file endpoint doesn't use a streaming API, meaning that potentially large files must be put in memory (in their entirety) while the upload function processes the upload.

The current HAPI API doesn't seem to be streaming either due to parse: true, so this isn't exactly a regression, but an enhancement.

Solution

Implement a streaming consumer for the multipart upload so that large files can be processed with a smaller memory footprint.

Note that this issue has two parts for it to be useful. The first part is as described (replace current logic with streaming logic, server-side). However, Group Income currently uploads all files as a single chunk, regardless of size. We need to implement splitting files up there too for the effects of this improvement to be had. For this second part we also need to decide whether this is the app's responsibility or whether we want to add this functionality transparently to chelonia/fileUpload. (For example, we could automatically split the payload if we receive a Blob but leave the chunks untouched if we receive an array of blobs.)

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions