Follow up on merged PR #756
Inside main, code_puppy/tools/common.py::_sanitize_string is implemented like this:
def _sanitize_string(text: str) -> str:
"""Sanitize a string to remove invalid Unicode surrogates.
This handles encoding issues common on Windows with copy-paste operations.
"""
if not text:
return text
try:
# Try encoding — if it works, string is clean.
text.encode("utf-8")
return text
except UnicodeEncodeError:
pass
try:
# Encode allowing surrogates, then decode replacing them.
return text.encode("utf-8", errors="surrogatepass").decode(
"utf-8", errors="replace"
)
except (UnicodeEncodeError, UnicodeDecodeError):
# Last resort: filter out surrogate characters.
return "".join(
char if ord(char) < 0xD800 or ord(char) > 0xDFFF else "\ufffd"
for char in text
)
Additional duplications
I ran rg "surrogatepass", and found these:
1. code_puppy/command_line/completers.py::_sanitize_for_encoding
Appears to be dead code in production; its only reference is tests/command_line/test_remaining_coverage.py::test_sanitize_for_encoding_unicode_error, which exists solely for test coverage.
Candidates for removal are this function + its dedicated test.
2. code_puppy/agents/_runtime.py::_sanitize_prompt
Actively used with duplicated surrogate-stripping logic as code_puppy/tools/common.py::_sanitize_string.
Consolidation candidate here — call _sanitize_string via a thin wrapper. Like this:
from code_puppy.tools.common import _sanitize_string
def _sanitize_prompt(prompt: str) -> str:
"""Strip lone UTF-16 surrogates (common on Windows copy-paste)."""
return _sanitize_string(prompt)
3. code_puppy/config.py::normalize_command_history
Consolidation candidate here — call _sanitize_string.
4. code_puppy/config.py::save_command_to_history
Consolidation candidate here — call _sanitize_string.
Plan
Will open a PR to cover them all.
Follow up on merged PR #756
Inside main,
code_puppy/tools/common.py::_sanitize_stringis implemented like this:Additional duplications
I ran
rg "surrogatepass", and found these:1.
code_puppy/command_line/completers.py::_sanitize_for_encodingAppears to be dead code in production; its only reference is
tests/command_line/test_remaining_coverage.py::test_sanitize_for_encoding_unicode_error, which exists solely for test coverage.Candidates for removal are this function + its dedicated test.
2.
code_puppy/agents/_runtime.py::_sanitize_promptActively used with duplicated surrogate-stripping logic as
code_puppy/tools/common.py::_sanitize_string.Consolidation candidate here — call
_sanitize_stringvia a thin wrapper. Like this:3.
code_puppy/config.py::normalize_command_historyConsolidation candidate here — call
_sanitize_string.4.
code_puppy/config.py::save_command_to_historyConsolidation candidate here — call
_sanitize_string.Plan
Will open a PR to cover them all.