Skip to content

Fix embed and similar commands using platform default encoding instead of UTF-8 for file input - #1559

Open
ikatyal2110 wants to merge 1 commit into
simonw:mainfrom
ikatyal2110:fix-embed-utf8-encoding
Open

Fix embed and similar commands using platform default encoding instead of UTF-8 for file input#1559
ikatyal2110 wants to merge 1 commit into
simonw:mainfrom
ikatyal2110:fix-embed-utf8-encoding

Conversation

@ikatyal2110

Copy link
Copy Markdown
Contributor

When llm embed -i file or llm similar -i file read text files, they called open(path, "r") without an explicit encoding, letting Python fall back to the platform default (which can be latin-1 or ascii on some systems). On those systems, UTF-8 files containing characters like æ or ø were silently misread, and the corrupted text was stored in embeddings.db. The stdin path had the same problem, reading from sys.stdin instead of decoding sys.stdin.buffer as UTF-8. Both paths now explicitly use UTF-8 for text mode, matching the embed-multi --files path which already tried UTF-8 first. Fixes #283


Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Foreign characters stored in content with embeddings malformed

2 participants