Everything You Need to Know
How blind evaluation works, what Monitors deliver, what a plan includes, and why teams switch from manual testing.
What is blind comparative ranking?
PeerLM anonymizes all model responses, shuffles the order, and asks evaluator models to rank them best-to-worst. The evaluator never knows which model produced which response — eliminating name bias and positional bias. Rankings map back to source models as normalized scores.
How is PeerLM different from testing models manually?
Manual testing means one person, one prompt, one afternoon — and a decision nobody can reproduce. PeerLM runs blind evaluations across dozens of scenarios, personas, and criteria simultaneously. You get statistically meaningful data instead of anecdotes, and audit-ready reports instead of Slack threads.
What models can I evaluate?
200+ models across OpenAI, Anthropic, Google, Meta, Mistral, and more — via OpenRouter and Groq integrations. Evaluate any combination against each other. Model capabilities sync automatically so you always have access to the latest releases.
What do I pay for?
Monitors, runs, and triggers. Model inference is included — PeerLM pays the providers, and we do not meter it or pass a token bill through to you. $99/month buys the default Pro monitor: weekly, 150 of your own prompts, 2 candidates, 5 judges rotated from a frozen pool of 8, plus catalog and price triggers, 10 runs a month. Team is $499/month with Prompt CI and a higher run allowance. Unused runs do not roll over.
What happens when I subscribe to a plan?
Your workspace is activated immediately. All evaluations and data are retained — nothing resets. Pro unlocks standing monitors, Premium candidates, custom criteria, and API & MCP access. Team adds Prompt CI (deploy webhook and GitHub Action). Start running evals right away. Reach out to sales@peerlm.com if you have any questions.
What are Monitors?
A Monitor connects PeerLM to a production use case and accumulates comparison Runs against live traffic. Each Run samples real prompts, replays candidates on the same inputs, and judges quality retained vs your control alongside cost. The Monitor rolls those Runs into a living switch / route / hold verdict. One-off authored evaluations stay under Suites and Runs; Monitors are for continuous comparison from production traffic. Standing monitors start on Pro. Free includes one sponsored comparison.
What is a living verdict?
The living verdict is the Monitor's recommendation — switch, route, or hold — refreshed by every comparison Run. It surfaces quality retained vs control, projected savings, and Switch Confidence (SCS), a 0–100 composite of quality parity, latency, cost savings, sample power, and judge calibration. Results break down by prompt cluster so you can see where a candidate excels or struggles. Export routing recommendations as JSON, YAML, LiteLLM config, Portkey config, or a code snippet. Receipts live on the Run; the verdict lives on the Monitor.
What log sources does PeerLM connect to?
Langfuse, Helicone, Cloudflare AI Gateway, and any OpenTelemetry-compatible source — plus file upload and PeerLM's SDK (TypeScript, Python, Go, Ruby) for in-app logging with configurable sampling. Connect Sources on each Monitor, then start a comparison Run from that traffic.
What triggers an Auto-Run?
On Pro, four events can start another comparison Run: the weekly heartbeat, a model release (catalog), a price change, and quality drift. Team and Enterprise add deploy triggers for Prompt CI, which bypass the daily debounce so each ship is its own run. Each monitor has a monthly run allowance (10 on Pro, 30 on Team; Enterprise is stated in the contract). Configure on the Monitor → Schedule.
Who picks the judges?
PeerLM selects the judges. A model from the vendor under test can't judge it. Default cycles seat 5 judges rotated from a frozen pool of 8, so consecutive runs compare like with like. Choosing your own judges is available on Enterprise. Pro and above can add one premium model to the pool for $49 per monitor per month — it sits in the rotation, it does not become the sole judge.
What is Prompt CI?
On Team and Enterprise, a deploy webhook and GitHub Action validate every system-prompt change against real production traffic before it ships. The control is the old prompt on the same model — not a cheaper candidate. Deploy runs bypass the daily debounce and count against the monitor's monthly run allowance.
What are system prompts?
System prompts (personas) let you test models in context — as a customer support agent, code reviewer, creative writer, etc. Each system prompt defines the role and instructions given to the model. PeerLM breaks down results by system prompt so you see which model wins for each use case.
What's the difference between JSON output and text output?
JSON mode returns structured arrays (e.g., '3 one-liner jokes') where each item is scored independently — giving granular data. Text mode evaluates free-form prose as a whole. JSON mode is recommended for comparative evaluations as it produces more reliable rankings.
Can I share results with my team?
Yes. All paid plans include shareable reports — a public link anyone can view without an account. Reports include rankings, per-persona breakdowns, and response samples. Export as CSV or JSON for further analysis.
How does response caching work?
PeerLM hashes the combination of model, persona, and topic content. Matching cached responses are reused instead of regenerated. Edit any part of the prompt and the cache auto-invalidates. Caching makes runs faster; it does not change your run allowance.
What is deterministic mode?
PeerLM sets temperature to 0 and uses fixed seeds where supported. It checks each model's capability registry first to prevent silent failures. Parameters used are logged for audit purposes — fully reproducible results.
Do you store my API keys?
Model calls use PeerLM-managed keys by default — no setup needed. On Enterprise, Bring Your Own Key covers both the judge panel and the models under test. Keys are encrypted with AES-256-GCM at rest under a dedicated key, and only the last four characters are ever readable.
What is the PeerLM MCP server?
The PeerLM MCP (Model Context Protocol) server lets you run evaluations directly from Claude Desktop, Cursor, or any MCP-compatible client. Create prompts, configure suites, trigger runs, and check results — all without leaving your IDE. Available on Pro, Team, and Enterprise. Set it up with: npx -y @peerlm/mcp
Does PeerLM have an API?
Yes. The REST API lets you manage suites, create system prompts and test prompts, trigger evaluation runs, start Monitor comparison Runs, and retrieve results programmatically. Available on Pro, Team, and Enterprise. Generate an API key from Settings > API Keys.
Can I cancel anytime?
Yes. Month-to-month, no long-term commitment. Cancel or downgrade anytime from billing settings. You keep access through the end of your billing period.
Still have questions?
Contact support