services/connectors downloads and fully parses a caller-supplied file before any size limit is applied. Surfaced by the final review of #163 (PDF/.docx extractors).
What's actually unbounded
PdfExtractor and DocxExtractor each pull the entire file into memory via files().get(fileId=..., alt="media") → io.BytesIO(payload), parse it in full, and only then return text to google.py, where max_content_chars (default 1,200,000) truncates.
The cap is applied after the cost has already been paid. Its own comment describes it as a transport guard that "stops a pathological file becoming a huge HTTP response" — it was never a guard on input size.
Two things that look like they'd catch this and don't:
request_timeout_s = 30 is an httplib2 socket timeout, not a request deadline. A large file that streams steadily never trips it.- Nothing bounds pypdf's parse time. It's pure Python;
extract_text() across a few hundred pages pins a FastAPI threadpool worker for minutes while holding the byte buffer and the extracted text.
This also predates the PDF work: the text/* media path has the same shape.
Why it's inconsistent with the rest of the package
SheetsExtractor sets an explicit MAX_ROWS_PER_TAB = 2000 and warns when it truncates. PDF and .docx got neither a cap nor a warning, and the connectors README's "Known limitations" lists only the spreadsheet cap and the scanned-PDF case — an operator reading it would conclude there is no other size exposure.
The fix is one place, not three
google.py already calls files().get(fileId=..., fields="name,mimeType") for metadata before dispatching to an extractor. Adding size to that fields list and rejecting oversized files with SourceUnsupported guards PDF, .docx, and the pre-existing text/* path from a single point — no per-extractor changes, no duplication of the check across the three call sites that currently repeat the media download (drive_export.py, pdf.py, docx.py).
Note size is absent for native Google types (Docs/Sheets/Slides/Forms), which are fetched through their own APIs rather than downloaded — the guard should skip rather than reject when the field is missing.
Open decisions
- What limit? Needs a number. Resumes are a few hundred KB; the exposure is a caller pointing at something pathological.
- Reject or truncate? Rejecting with a clear 422 is honest; silently truncating a 500-page PDF reproduces the scanned-PDF failure mode of looking successfully ingested.
- Worth a
MAX_FILE_BYTES in config.py alongside max_content_chars, and a README line so the limitation is documented rather than discovered.
Why it was deferred out of #163
Choosing the limit is a design call, and the text/* exposure isn't a regression from that branch. Not a today-outage — the shared folder holds ordinary resumes — but it's an unbounded blast radius on a service that accepts a caller-supplied URL.
services/connectorsdownloads and fully parses a caller-supplied file before any size limit is applied. Surfaced by the final review of #163 (PDF/.docx extractors).What's actually unbounded
PdfExtractorandDocxExtractoreach pull the entire file into memory viafiles().get(fileId=..., alt="media")→io.BytesIO(payload), parse it in full, and only then return text togoogle.py, wheremax_content_chars(default 1,200,000) truncates.The cap is applied after the cost has already been paid. Its own comment describes it as a transport guard that "stops a pathological file becoming a huge HTTP response" — it was never a guard on input size.
Two things that look like they'd catch this and don't:
request_timeout_s = 30is an httplib2 socket timeout, not a request deadline. A large file that streams steadily never trips it.extract_text()across a few hundred pages pins a FastAPI threadpool worker for minutes while holding the byte buffer and the extracted text.This also predates the PDF work: the
text/*media path has the same shape.Why it's inconsistent with the rest of the package
SheetsExtractorsets an explicitMAX_ROWS_PER_TAB = 2000and warns when it truncates. PDF and.docxgot neither a cap nor a warning, and the connectors README's "Known limitations" lists only the spreadsheet cap and the scanned-PDF case — an operator reading it would conclude there is no other size exposure.The fix is one place, not three
google.pyalready callsfiles().get(fileId=..., fields="name,mimeType")for metadata before dispatching to an extractor. Addingsizeto thatfieldslist and rejecting oversized files withSourceUnsupportedguards PDF,.docx, and the pre-existingtext/*path from a single point — no per-extractor changes, no duplication of the check across the three call sites that currently repeat the media download (drive_export.py,pdf.py,docx.py).Note
sizeis absent for native Google types (Docs/Sheets/Slides/Forms), which are fetched through their own APIs rather than downloaded — the guard should skip rather than reject when the field is missing.Open decisions
MAX_FILE_BYTESinconfig.pyalongsidemax_content_chars, and a README line so the limitation is documented rather than discovered.Why it was deferred out of #163
Choosing the limit is a design call, and the
text/*exposure isn't a regression from that branch. Not a today-outage — the shared folder holds ordinary resumes — but it's an unbounded blast radius on a service that accepts a caller-supplied URL.