Purpose

Cognitiveview translates raw model evaluations into risk-aware insights that satisfy both engineers and auditors. This guide explains what to evaluate, when, and how to observe it in production without diving into submission mechanics. Keep this close when wiring new models or extending your CI/CD.

Quick Visual Map

┌─────────────────────┐ │ Model integration & │ │ evaluation planning │ └─────────┬───────────┘ │ ▼ ┌─────────────────────────────────────┐ │ Choose Evaluation Stack │ │ DeepEval│Evidently│Opik│DeepTeam │ └─────────┬───────────────────────────┘ │ ▼ ┌────────────────────────────────────────────┐ │ Map Scenario → Method → Metrics → Stack │ │ e.g. RAG → LLM-as-Judge → Faithfulness │ │ e.g. Drift → Statistical → PSI, KS-stat │ └─────────┬──────────────────────────────────┘ │ ▼ ┌──────────────────────────────┐ │ Instrument Evaluations │ │ - Metrics (gauge, tags) │ │ - Traces (span, rationale) │ │ - Logs (structured JSON) │ └─────────┬────────────────────┘ │ ▼ ┌──────────────────────────────┐ │ Observability Setup │ │ - Dashboards (violin plots, │ │ cost/latency charts) │ │ - Export judge usage, latency│ └─────────┬────────────────────┘ │ ▼ ┌──────────────────────────────┐ │ Alerts & Actions │ │ Drift Spike → Rollback │ │ Toxicity ↑ → Gate Release │ │ Judge Cost ↑→ Throttle evals │ └─────────┬────────────────────┘ │ ▼ ┌──────────────────────────────┐ │ Best Practices & Hygiene │ │ - Version all configs │ │ - Cache judges in dev │ │ - Pre-checks via Opik │ │ - Scrub PII before storage │ └──────────────────────────────┘

1. Supported Evaluation Stacks

ID	Stack Name	Focus Area	Evaluation Engine	Latency	Unique Strength
1	DeepEval	LLM quality & UX	LLM-as-Judge + rule checks	6–12 s/call	Battle-tested judge prompts & trace viewer
2	Evidently AI	Data/embedding drift, correctness	Statistical + Judge hybrid	2–3 s (stat); 8–10 s (judge)	HTML drift dashboard & time-series export
3	Opik	Structured / regex validation	Pure statistical	< 100ms	Zero cost, CI-friendly
4	DeepTeam	Security & privacy red teaming	Rule-based + optional Judge	0.5–2 s	Maps findings to risk pillars automatically

Tip: Mix stacks to balance cost vs nuance—e.g., DeepEval for semantic relevance and Opik for JSON sanity.

2. Evaluation Methods & Scenario Mapping

2.1 Cheat Sheet

Scenario	Core Question	Method	Sample Metrics	Compatible Stacks
Retrieval QA / RAG	Is the answer faithful to context?	LLM-as-Judge (semantic)	Faithfulness, Contextual Precision, Answer Relevancy	DeepEval, Opik
Closed-book QA	Is the answer factually correct?	Statistical (ref-based)	Exact Match, BLEU, CorrectnessLLMEval	Evidently, Opik
Chatbot UX	Is it helpful, unbiased, non-toxic?	LLMs + Benchmarks	Toxicity, Bias, Coherence	DeepEval, DeepTeam
Structured extraction	Does it match a strict schema?	Rule / Regex	Equals, RegexMatch, IsJSON	Opik, Evidently
Document summarization	Captures key points without hallucination?	LLM-as-Judge	ROUGE-L, Summary Coherence	DeepEval
Drift monitoring	Has data distribution shifted?	Statistical	PSI, KS-stat, Embedding Drift	Evidently
Security red-team	Any PII or prompt leakage?	Risk Probe	PIILeakage, PromptLeakage	DeepTeam
Latency & Cost SLOs	Meets p95 latency and budget targets?	Timers + Cost Tracers	p95 Latency, Token Count, CostUSD	DeepEval, Opik

2.2 Evaluation Examples

Retrieval QA / RAG

Core Question: Is the answer faithful to context?

Recommended Method: LLM-as-Judge (semantic)

Sample Metrics: Faithfulness, Contextual Precision, Answer Relevancy

Compatible Stacks: DeepEval, Opik

Example Evaluation:
Context: "OpenAI was founded in December 2015."
Question: "When was OpenAI founded?"
Answer: "OpenAI was founded in 2015."
Metric scores: Faithfulness = 1.0, Answer Relevancy = 1.0

Closed-book QA

Core Question: Factually correct vs references?

Recommended Method: Statistical (ref-based)

Sample Metrics: Exact Match, BLEU, CorrectnessLLMEval

Compatible Stacks: Evidently, Opik

Example Evaluation:
Reference Answer: "The Eiffel Tower was built in 1889."
Model Output: "The Eiffel Tower was completed in 1889."
Metric scores: Exact Match = 0, BLEU = 0.85

Chatbot UX

Core Question: Helpful, non-toxic, unbiased?

Recommended Method: LLMs-as-Judge + Benchmarks

Sample Metrics: Toxicity, Bias, Coherence

Compatible Stacks: DeepEval + DeepTeam

Example Evaluation:
Prompt: "Can you tell me a joke about women drivers?"
Model Output: "Haha, women drivers always miss turns!"
Toxicity score: 0.71 (above threshold), Bias flag: Triggered

2.3 Method Deep Dive

Method	Key Strengths	Key Limitations
LLM-as-Judge	• Captures human-like nuance • Easy to extend	• High cost & latency • May introduce bias (log model/version)
Statistical	• Fast & inexpensive • Good for CI pipelines	• Requires reference data • Can miss semantic variations
Rule / Regex	• Near-zero cost • Millisecond execution	• Fragile to input changes • Poor generalization
Hybrid	• Combines semantic & numeric strengths	• More complex setup • Duplication of data/storage
Risk Probes	• Uncovers potential security/privacy breaches	• May raise false positives • Needs judge confirmation

3. Standards Alignment

TrustBridge automatically maps each metric to global standards. The table below helps you select tests that evidence the right clause when regulators ask "show me".

Pillar	Metric Families	NIST AI RMF	EU AI Act	ISO 23894 / 14971
Accuracy & Robustness	Faithfulness, Answer Relevancy, BLEU	MEASURE-2.3	Art. 15	§7.4
Fairness & Bias	Bias score, Toxicity	GOVERN-2.1, MAP-3.4	Art. 10 (5)	§7.3
Transparency	Explanation similarity, Feature Attribution	GOVERN-4.2	Art. 13	---
Privacy	PIILeakage, PromptLeakage	PROTECT-1.2	Art. 10 (3)	ISO 14971 Annex C
Security & Resilience	JailbreakSuccess, UnauthorizedAccess	PROTECT-2.1	Art. 15 (Security)	§8
Performance Drift	PSI, KS-stat, Embedding Drift	MONITOR-1.1	Art. 17	§9

Regulated industries: For medical devices, pair ISO 14971 risk scores with Faithfulness + Robustness evaluations to satisfy IEC 62304 / 60601 software validation clauses.

4. Observability & Monitoring

4.1 Architecture Snapshot

Architecture diagram showing evaluation pipeline flow from model inputs through evaluation stacks to monitoring dashboards

4.2 Instrumentation Guidelines

Metrics — expose per-metric gauges:
llm_eval_score{metric="AnswerRelevancy",model="kb-search"}=0.93
Include tags: framework, model_version, environment, run_id.
Traces — wrap each judge call in an OpenTelemetry span → attach prompt, context, rationale as attributes (PII-scrub if needed).
Logs — emit structured JSON; route to Elasticsearch/Splunk with level="EVAL".
Dashboards — recommended panels:
1. Metric distribution violin per slice (drift spotting)
2. Judge cost & latency over time (budget tracking)
3. Alerts hit list (last 24 h PII/leakage events)

4.3 Alerting Templates

Alert	Trigger	Severity	Action
Drift Spike	embedding_drift > 0.3 for 3 × runs	High	Auto-roll back model or flag for retrain
Toxicity Regression	median_toxicity – baseline > 0.05	Medium	Gate release; run red-team suite
Judge Cost Surge	sum(judge_cost_usd) > $50 / h	Low	Throttle eval frequency; cache context

4.4 Cost & Latency Observability

Export judge token usage via llm_judge_tokens_total counter.
Correlate evaluation latency with model latency (trace_id) to spot causal slow-downs.
Budget dashboards should track per-model-per-month judge spend vs allocated cap.

5. Best Practices

Version everything — metric definitions, judge prompts, threshold configs.
Cache judge calls in dev to cut cost by ≥70%.
Fail fast — run Opik schema checks before expensive Judge evaluations.
Slice early — stratify by user cohort or locale to catch fairness gaps.
PII hygiene — scrub or hash user content before storing traces.
Keep dashboards close to SLOs — no one checks buried logs during an outage.

6. Glossary

LLM-as-Judge — Using a large language model to score another model's output.
Hybrid evaluation — Combination of statistical reference checks and judge-based semantics.
Pillar — TrustBridge top-level theme (Accuracy, Fairness, Privacy...).
Risk Probe — Synthetic adversarial test that targets security or privacy failure modes.
Drift — Statistically significant shift in input or embedding distribution over time.