Skip to content

Additional surrogate-sanitizing file read outside code_puppy/tools. #893

Description

@StarsExpress

Follow up on merged PR #756

Inside main, code_puppy/tools/common.py::_sanitize_string is implemented like this:

def _sanitize_string(text: str) -> str:
    """Sanitize a string to remove invalid Unicode surrogates.
    This handles encoding issues common on Windows with copy-paste operations.
    """
    if not text:
        return text

    try:
        # Try encoding — if it works, string is clean.
        text.encode("utf-8")
        return text

    except UnicodeEncodeError:
        pass

    try:
        # Encode allowing surrogates, then decode replacing them.
        return text.encode("utf-8", errors="surrogatepass").decode(
            "utf-8", errors="replace"
        )

    except (UnicodeEncodeError, UnicodeDecodeError):
        # Last resort: filter out surrogate characters.
        return "".join(
            char if ord(char) < 0xD800 or ord(char) > 0xDFFF else "\ufffd"
            for char in text
        )

Additional duplications

I ran rg "surrogatepass", and found these:

Image

1. code_puppy/command_line/completers.py::_sanitize_for_encoding

Appears to be dead code in production; its only reference is tests/command_line/test_remaining_coverage.py::test_sanitize_for_encoding_unicode_error, which exists solely for test coverage.

Candidates for removal are this function + its dedicated test.

2. code_puppy/agents/_runtime.py::_sanitize_prompt

Actively used with duplicated surrogate-stripping logic as code_puppy/tools/common.py::_sanitize_string.

Consolidation candidate here — call _sanitize_string via a thin wrapper. Like this:

from code_puppy.tools.common import _sanitize_string

def _sanitize_prompt(prompt: str) -> str:
    """Strip lone UTF-16 surrogates (common on Windows copy-paste)."""
    return _sanitize_string(prompt)

3. code_puppy/config.py::normalize_command_history

Image

Consolidation candidate here — call _sanitize_string.

4. code_puppy/config.py::save_command_to_history

Image

Consolidation candidate here — call _sanitize_string.

Plan

Will open a PR to cover them all.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions