ENGINEER / TRACE BEFORE GUESSING
Once AI is inside a system, bad output may come from retrieval, permissions, tool calls, latency, graders or infrastructure. Engineer helps you trace the request path and decide what to measure first.
DIAGNOSTIC MAP
Look here when: output quality drops after a model, prompt or tool change. The failure may be an unrepresentative task set or vague grader.
Measure first: representative task set, pass/fail criteria, failure taxonomy and version regression.
Look here when: answers sound plausible but miss context or retrieve the wrong documents.
Measure first: query set, relevance judgment, missing-context failures and reranking impact. Separate retrieval quality from answer quality.
Look here when: an agent can call tools, read protected data or take consequential actions.
Measure first: tool permissions, protected-data boundaries, human approval points and audit trail.
OWASP GenAI ↗ · NIST GenAI Profile ↗
Look here when: the app feels slow even though a model benchmark looks fast. Vendor throughput is not app latency.
Measure first: network, queue, TTFT, generation, tool execution and retries in an end-to-end trace.
Look here when: you own SLA, throughput, memory or cost for a production workload.
Measure first: workload shape, latency target, throughput, memory, batching and failure budget before choosing serving trade-offs.