co-mono/packages/ai/src/utils
Mario Zechner 4e7a340460 Add Unicode surrogate sanitization for all providers
Fixes issue where unpaired Unicode surrogates in tool results cause JSON serialization errors in API providers, particularly Anthropic.

- Add sanitizeSurrogates() utility function to remove unpaired surrogates
- Apply sanitization in all provider convertMessages() functions:
  - User message text content (string and text blocks)
  - Assistant message text and thinking blocks
  - Tool result output
  - System prompts
- Valid emoji (properly paired surrogates) are preserved
- Add comprehensive test suite covering all 8 providers

Previously only Google and Groq handled unpaired surrogates correctly.
Now all providers (Anthropic, OpenAI Completions/Responses, Google, xAI, Groq, Cerebras, zAI) sanitize text before API submission.
2025-10-13 14:26:54 +02:00
..
event-stream.ts refactor(ai): improve error handling and stop reason types 2025-09-18 19:57:13 +02:00
json-parse.ts refactor(ai): improve error handling and stop reason types 2025-09-18 19:57:13 +02:00
sanitize-unicode.ts Add Unicode surrogate sanitization for all providers 2025-10-13 14:26:54 +02:00
typebox-helpers.ts refactor(ai): improve error handling and stop reason types 2025-09-18 19:57:13 +02:00
validation.ts More browser extension work, disable ajv validation in browser extensions, it uses eval/new Function, which is not allowed in manifest v3 extensions 2025-10-01 20:30:49 +02:00