analyze

prompts/analyzer-developer.md

@@ -0,0 +1,72 @@
+# EvalSpec Schema Reference
+
+## Assertion Types
+
+### Deterministic Assertions (for pure functions, exact outputs)
+
+| Type | Properties | Use Case |
+|------|------------|----------|
+| `equals` | `expected`, `path?` | Exact value match |
+| `contains` | `value`, `path?` | Substring or array element |
+| `throws` | `errorType?`, `messageContains?` | Exception expected |
+| `typeof` | `expected`, `path?` | Type checking |
+| `matches` | `pattern`, `path?` | Regex pattern match |
+| `truthy`/`falsy` | `path?` | Boolean coercion |
+| `custom` | `description`, `check` | Complex validation |
+
+### LLM Rubric Assertions (for subjective quality, UI, user experience)
+
+| Type | Properties | Use Case |
+|------|------------|----------|
+| `llm-rubric` | `rubric`, `criteria[]`, `passingThreshold?` | Quality evaluation by Claude |
+
+**When to use LLM rubrics:**
+- Error message quality (is it helpful? actionable?)
+- UI component output (does it render correctly? accessible?)
+- API response format (well-structured? consistent?)
+- Generated content quality (documentation, code suggestions)
+
+**Example:**
+```json
+{
+  "type": "llm-rubric",
+  "rubric": "error-message-quality",
+  "criteria": ["clarity", "actionability", "includes context"],
+  "passingThreshold": 0.7,
+  "description": "Error message should clearly explain what went wrong and how to fix it"
+}
+```
+
+## Formatting Rules
+
+- **Scenario IDs**: kebab-case, descriptive (e.g., `user-auth-invalid-token`)
+- **Module paths**: Match source file paths exactly (e.g., `src/auth/login.py`)
+- **Function names**: Match source exactly, including case
+- **Tags**: lowercase, categorize by domain (`auth`, `api`, `database`, etc.)
+
+## Priority Guidelines
+
+| Priority | When to Use |
+|----------|-------------|
+| `critical` | Core business logic, security-sensitive, payment flows |
+| `high` | Public API, user-facing, data integrity |
+| `medium` | Internal utilities, helper functions |
+| `low` | Convenience methods, formatting, logging |
+
+## Mock Specification
+
+When specifying mocks:
+```json
+{
+  "target": "module.external_api.fetch",
+  "returnValue": {"status": "ok"},
+  "sideEffect": "raises ConnectionError"
+}
+```
+
+## Input Generation
+
+Generate realistic inputs based on:
+1. Parameter types from signatures
+2. Docstring examples
+3. Domain semantics (emails, UUIDs, timestamps)