n8n-nodes-deepeval

n8n community nodes powered by DeepEval: 33 metric nodes, DeepEval Trigger, DeepEval Aggregate, and DeepEval Consistency.

• n8n-nodes-deepeval
  • Synopsis
    • Install
  • Architecture
    • Required N8N Nodes
    • Optional Dashboard
  • Kitchen-sink examples
    • Non-conversational kitchen sink
    • Conversational kitchen sink
  • Available Nodes
    • n8n → DeepEval field mapping
    • DeepEval Trigger
      • Sources
    • DeepEval Metrics
      • Custom
        • G-Eval
        • DAG
        • Conversational G-Eval
        • Conversational DAG
      • Agentic
        • Task Completion
        • Step Efficiency
        • Argument Correctness
        • Tool Correctness
        • Plan Adherence
        • Plan Quality
      • Multi-Turn
        • Turn Relevancy
        • Role Adherence
        • Knowledge Retention
        • Conversation Completeness
        • Goal Accuracy
        • Tool Use
        • Topic Adherence
        • Turn Faithfulness
        • Turn Contextual Precision
        • Turn Contextual Recall
        • Turn Contextual Relevancy
      • Safety
        • Bias
        • Toxicity
        • Non-Advice
        • Misuse
        • PII Leakage
        • Role Violation
      • Others
        • Summarization
        • Prompt Alignment
        • Hallucination
      • Community
        • Citation Faithfulness
        • Agent Loop Detection
        • Tool Permission
    • DeepEval Aggregate
      • Sinks
      • Typical wiring
    • DeepEval Consistency

Synopsis

This package integrates DeepEval into n8n 2.x. It requires Node.js 20 or newer. All 36 nodes are listed under n8n's AI, LLM & Voice category and are searchable with the DeepEval Benchmarking alias.

The current package intentionally contains only the n8n nodes. The dashboard, dashboard hooks, Google Sheets adapters, and Microsoft Excel adapters are deferred.

Install

Install the published community package in n8n:

npm install n8n-nodes-deepeval

Repository development, testing, and contributor workflows are documented in AGENTS.md.

Architecture

DeepEval executes inline through an embedded Python runtime bundled with the package. Evaluations run in-process with n8n via a warmed Pyodide pool (default size 4, set DEEPEVAL_PYODIDE_POOL_SIZE). Each metric evaluation borrows a pool slot, runs in an isolated WASM interpreter, and returns the slot after cleanup. Plan worker memory and throughput around pool init cost and per-slot RAM. Multi-run consistency (runsPerRow

1 on DeepEval Trigger) multiplies metric evaluations by that factor — size DEEPEVAL_PYODIDE_POOL_SIZE and worker memory accordingly. See AGENTS.md for implementation details (Pyodide, vendoring, E2E).

Pyodide isolation

After every evaluation, the runtime clears judge globals and resets DeepEval trace/context state on the borrowed slot before returning it to the pool. That mandatory cleanup prevents one metric run from poisoning the next borrower when multiple metrics execute in parallel.

Clean Session is an optional per-metric node setting (default off). When enabled, the runtime discards that slot's Pyodide VM after the evaluation and warms a fresh interpreter before the slot re-enters the pool. Use it when you want maximum isolation at the cost of slower runs on that node; leave it off for normal throughput. Clean Session is a runtime option — it is not passed to DeepEval's Python metric constructor.

Required N8N Nodes

Offered as N8N community nodes, these are the core components that allow you to integrate DeepEval into your N8N workflows. They are designed to be flexible and easy to use, enabling you to evaluate various data types and models directly within your N8N environment.

Optional Dashboard

The dashboard is not included in this implementation. No frontend injection, hooks, parallel database, or transparent recording behavior is installed.

Kitchen-sink examples

Two importable stress-test workflows exercise the full evaluation pipeline end to end: DeepEval Trigger loads one dataset row, Enrich Evaluation Data adds fixture fields the source table does not carry, AI Agent (with Calculator and shared judge model) produces agent output and traces, metrics score through DeepEval Aggregate, and Persist Results writes the run to a Data Table.

Use them to validate wiring, Pyodide throughput, judge-model fan-out, and Aggregate fan-in under realistic load.

Non-conversational kitchen sink

packages/nodes/examples/kitchenSinkNonConversational.workflow.json runs all 20 single-turn and agentic metrics (requiresMemory: false):

G-Eval, DAG, Task Completion, Step Efficiency, Argument Correctness, Tool Correctness, Plan Adherence, Plan Quality, Bias, Toxicity, Non-Advice, Misuse, PII Leakage, Role Violation, Summarization, Prompt Alignment, Hallucination, Citation Faithfulness, Agent Loop Detection, and Tool Permission.

Shared infrastructure: one OpenAI Chat Model sub-node wired to AI Agent and every LLM-judge metric; Calculator on AI Agent for trace-dependent metrics; Trigger limited to one row so the stress is metric fan-out, not dataset size. Metrics fan out in parallel from Prepare Metric Input into Collect Metric Results (Merge append).

Conversational kitchen sink

packages/nodes/examples/kitchenSinkConversational.workflow.json runs all 13 conversational and turn-based metrics (requiresMemory: true):

Conversational G-Eval, Conversational DAG, Turn Relevancy, Role Adherence, Knowledge Retention, Conversation Completeness, Goal Accuracy, Tool Use, Topic Adherence, Turn Faithfulness, Turn Contextual Precision, Turn Contextual Recall, and Turn Contextual Relevancy.

Same Trigger → enrich → Agent → Aggregate → Persist pipeline as the non-conversational sink, plus Simple Memory shared by AI Agent and every conversational metric (session key ={{ $json.evalContext.runId }} so each Trigger run gets an isolated buffer). Calculator remains on AI Agent for Goal Accuracy and Tool Use. Metrics fan out in parallel from Prepare Metric Input into Collect Metric Results (Merge append); the Pyodide pool executes up to DEEPEVAL_PYODIDE_POOL_SIZE evaluations concurrently.

Available Nodes

Metric nodes cover the DeepEval Eval Metrics catalog, grouped as Custom, Agentic, Multi-Turn, Safety, Others, and Community (MCP, Images, and RAG are not included).

Wiring follows normal n8n data flow:

• Main connection — evaluation data arrives as item fields from upstream nodes. Connect AI Agent (or any node that produced the run) on main for agentic metrics; enable Return Intermediate Steps so output and intermediateSteps are available. Single-turn metrics use fields such as input, actualOutput, context, and retrievalContext. See n8n → DeepEval field mapping for how these map into DeepEval test cases and traces.
• Language Model sub-node — LLM-judge metrics accept an aiLanguageModel connection (the same OpenAI, Anthropic, and related sub-nodes used by AI Agent). AI Agent and Chat Trigger are not sub-nodes; only the judge model uses that port.
• Memory sub-node — conversational and turn-based metrics require aiMemory (Simple Memory, Postgres Chat Memory, and related memory sub-nodes). Connect the same Memory used by AI Agent; the metric reads chat history and builds DeepEval turns internally.

Remaining DeepEval constructor options (threshold, criteria, allowlists, and so on) appear as Config on the node UI. Clean Session (see Pyodide isolation) is also on every metric node. Every metric node emits the same output shape: score, reason, success. Most metrics pass when score >= threshold; lower-is-better safety metrics (Bias, Toxicity, Hallucination, Misuse) pass when score <= threshold — each metric section documents its direction.

n8n → DeepEval field mapping

Metric nodes accept n8n item fields on the main connection and/or special sub-node connections, then map them internally before calling DeepEval. Canvas wiring: main data in for agentic and single-turn metrics; Memory sub-node (required) for conversational metrics; Language Model sub-node for LLM judges.

| Source | DeepEval field | Notes | | --- | --- | --- | | output (main) | actual_output | AI Agent final response | | input (main) | input | User prompt or golden input | | actualOutput (main) | actual_output | Explicit field name when set upstream | | intermediateSteps (main) | tools_called, synthetic trace | AI Agent with Return Intermediate Steps enabled | | expectedTools (main) | expected_tools | Trigger column mapping or upstream field | | aiMemory (sub-node) | turns (ConversationalTestCase) | Required for conversational / turn-based metrics; built by memory adapter | | context, retrievalContext (main) | context, retrieval_context | Grounding / RAG metrics |

Conversational metrics — Conversational G-Eval, Conversational DAG, and all Multi-Turn metrics read conversation history only from the aiMemory sub-node. Connect the same Memory instance your AI Agent uses. No turns field on main.

Trace-dependent metrics — Task Completion, Step Efficiency, Plan Adherence, Plan Quality, and Agent Loop Detection analyze an agent execution trace in DeepEval (normally from @observe tracing). In n8n, the metric builds a synthetic trace from intermediateSteps when native tracing is unavailable. If intermediate steps are missing when required, the metric errors with a clear message.

Goal Accuracy and Tool Use — turns come from Memory. Connect AI Agent on main (with Return Intermediate Steps) to enrich relevant turns with tools_called from intermediateSteps.

Deterministic agent metrics — Argument Correctness, Tool Correctness, and Tool Permission map intermediateSteps → tools_called directly (plus input and expected_tools where applicable).

DeepEval Trigger

Starts an evaluation run from rows supplied by n8n's official Data Table node. It emits one mapped item and evaluation context per row. It does not score and does not own Language Model or metric configuration.

n8n 2.x restricts the internal Data Table proxy to built-in node types. The example therefore uses the supported composition Data Table (Get rows) → DeepEval Trigger instead of bypassing that access control.

Config

• runName — human label for the evaluation run
• dataTableId — source table identity recorded in evalContext
• columnMapping — map source columns → DeepEval fields (input, expectedOutput, context, retrievalContext, expectedTools, and so on)
• limitRows — whether to cap how many rows are processed
• maxRows — maximum rows when limitRows is enabled
• filters — optional column=value filters on the dataset
• runsPerRow — emit this many runs per source row for consistency scoring (default 1)

Each emitted item includes evalContext.workflowHash (SHA-256 of the canonical workflow graph), evalContext.caseId (${workflowHash}:${row.id}), evalContext.runIndex, and evalContext.runId (${caseId}:${runIndex}). Workflow edits change workflowHash; canvas layout changes do not.

Sources

This pass supports n8n Data Tables only. Google Sheets and Excel sources are deferred.

DeepEval Metrics

Custom

Custom metrics let you define evaluation criteria with natural language (G-Eval) or deterministic decision trees (DAG).

G-Eval

Uses LLM-as-a-judge with chain-of-thought to score an output against any criteria you define (correctness, tone, safety, and so on).

Config

• name (required) — metric display name
• criteria — natural-language evaluation rubric (required unless evaluationSteps is set; mutually exclusive with evaluationSteps)
• evaluationSteps — fixed chain-of-thought steps; skips auto-generation from criteria (mutually exclusive with criteria)
• evaluationParams (required at run) — which test-case fields the judge may use (INPUT, ACTUAL_OUTPUT, EXPECTED_OUTPUT, CONTEXT, and so on)
• rubric — score bands (scoreRange 0–10, expectedOutcome) to confine LLM scoring
• threshold — pass cutoff; success when score >= threshold (default 0.5)
• strictMode — binary 1/0 scoring; forces threshold to 1
• asyncMode — run internal LLM calls concurrently (default true)
• verboseMode — print intermediate steps to console

DAG

Runs a deep acyclic graph of LLM-powered decision nodes for deterministic, rule-based scoring when G-Eval is too subjective.

Config

• name (required) — metric display name
• dag (required) — decision graph built from Task, Binary/Non-Binary Judgement, and Verdict nodes
• threshold — pass cutoff; success when score >= threshold (default 0.5)
• includeReason — whether to generate a human-readable reason (default true)
• strictMode — binary 1/0 scoring; forces threshold to 1
• asyncMode — run internal LLM calls concurrently (default true)
• verboseMode — log each node verdict

Conversational G-Eval

G-Eval adapted for full conversations: scores the entire dialogue against custom criteria with prior context in mind.

Config

• name (required) — metric display name
• criteria — conversation-level rubric (required unless evaluationSteps is set; mutually exclusive with evaluationSteps)
• evaluationSteps — fixed chain-of-thought steps (mutually exclusive with criteria)
• evaluationParams — turn fields to evaluate (defaults include CONTENT)
• rubric — score bands to confine LLM scoring
• threshold — pass cutoff; success when score >= threshold (default 0.5)
• strictMode — binary 1/0 scoring; forces threshold to 1
• asyncMode — run internal LLM calls concurrently (default true)
• verboseMode — print intermediate steps to console

Connect the same Memory as AI Agent.

Conversational DAG

DAG adapted for multi-turn evaluation: deterministic decision trees over conversation windows.

Config

• name (required) — metric display name
• dag (required) — conversational decision graph (supports turnWindow on nodes)
• threshold — pass cutoff; success when score >= threshold (default 0.5)
• includeReason — whether to generate a human-readable reason (default true)
• strictMode — binary 1/0 scoring; forces threshold to 1
• asyncMode — run internal LLM calls concurrently (default true)
• verboseMode — log each node verdict

Connect the same Memory as AI Agent.

Agentic

Agentic metrics evaluate LLM agent execution. Connect AI Agent (or an equivalent upstream node) on the main connection; enable Return Intermediate Steps. The metric maps n8n fields to DeepEval test cases and traces (see n8n → DeepEval field mapping). Attach a Language Model sub-node when the metric needs an LLM judge.

Task Completion

Judges whether the agent accomplished the task by aligning the extracted outcome with the inferred (or configured) goal.

Config

• task — explicit goal; if omitted, inferred from the agent trace
• threshold — pass cutoff; success when score >= threshold (default 0.5)
• includeReason — whether to generate a human-readable reason (default true)
• strictMode — binary 1/0 scoring; forces threshold to 1
• asyncMode — run internal LLM calls concurrently (default true)
• verboseMode — print intermediate steps to console

Connect AI Agent on main (with Return Intermediate Steps). Maps to DeepEval input, actual_output, and a synthetic trace from intermediateSteps. DeepEval infers task from the trace when not set in Config.

Step Efficiency

Measures how efficiently the agent completed the task, penalizing unnecessary steps, retries, and detours in the trace.

Config

• threshold — pass cutoff; success when score >= threshold (default 0.5)
• includeReason — whether to generate a human-readable reason (default true)
• strictMode — binary 1/0 scoring; forces threshold to 1
• asyncMode — run internal LLM calls concurrently (default true)
• verboseMode — print intermediate steps to console

Trace-only in DeepEval. Requires synthetic trace from intermediateSteps; fails clearly if intermediate steps are absent.

Argument Correctness

Checks whether each tool call received correct arguments for the user request (referenceless LLM judge).

Config

• threshold — pass cutoff; success when score >= threshold (default 0.5)
• includeReason — whether to generate a human-readable reason (default true)
• strictMode — binary 1/0 scoring; forces threshold to 1
• asyncMode — run internal LLM calls concurrently (default true)
• verboseMode — print intermediate steps to console

Maps input → DeepEval input, output → actual_output, intermediateSteps → tools_called.

Tool Correctness

Compares tools the agent called against expected tools (selection, order, and optionally inputs/outputs).

Config

• availableTools — tools the agent could use; enables LLM tool-selection optimality sub-score
• evaluationParams — strictness for matching (INPUT_PARAMETERS, OUTPUT; names always matched)
• shouldExactMatch — toolsCalled must exactly match expectedTools (name + optional input/output)
• shouldConsiderOrdering — LCS-based order-aware matching (ignored if shouldExactMatch is true)
• threshold — pass cutoff; success when score >= threshold (default 0.5)
• includeReason — whether to generate a human-readable reason (default true)
• strictMode — binary 1/0 scoring; forces threshold to 1
• asyncMode — run internal LLM calls concurrently (default true)
• verboseMode — print intermediate steps to console

Maps input, intermediateSteps → tools_called, and expectedTools → expected_tools. expectedTools may come from the Trigger column mapping or another upstream field.

Plan Adherence

Scores how closely the agent's execution followed the plan inferred from its reasoning in the trace.

Config

• threshold — pass cutoff; success when score >= threshold (default 0.5)
• includeReason — whether to generate a human-readable reason (default true)
• strictMode — binary 1/0 scoring; forces threshold to 1
• asyncMode — run internal LLM calls concurrently (default true)
• verboseMode — print intermediate steps to console

Trace-only. Synthetic trace from intermediateSteps. When DeepEval finds no plan in the trace, score defaults to 1.

Plan Quality

Scores the quality of the plan itself (task vs. plan alignment), independent of whether execution stuck to it.

Config

• threshold — pass cutoff; success when score >= threshold (default 0.5)
• includeReason — whether to generate a human-readable reason (default true)
• strictMode — binary 1/0 scoring; forces threshold to 1
• asyncMode — run internal LLM calls concurrently (default true)
• verboseMode — print intermediate steps to console

Trace-only. Synthetic trace from intermediateSteps. When DeepEval finds no plan in the trace, score defaults to 1.

Multi-Turn

Multi-turn metrics evaluate chatbots over a full conversation. Each metric requires an aiMemory sub-node — connect the same Memory used by AI Agent. Goal Accuracy and Tool Use also connect AI Agent on main (with Return Intermediate Steps) to enrich turns with tool-call data from intermediateSteps.

Turn Relevancy

Checks that each assistant reply stays relevant given prior turns in a sliding window.

Config

• windowSize — sliding-window size in unit interactions (default 10)
• threshold — pass cutoff; success when score >= threshold (default 0.5)
• includeReason — whether to generate a human-readable reason (default true)
• strictMode — binary 1/0 scoring; forces threshold to 1
• asyncMode — run internal LLM calls concurrently (default true)
• verboseMode — print intermediate steps to console

Connect the same Memory as AI Agent.

Role Adherence

Measures whether the assistant stayed in character across every turn against a defined persona.

Config

• threshold — pass cutoff; success when score >= threshold (default 0.5)
• includeReason — whether to generate a human-readable reason (default true)
• strictMode — binary 1/0 scoring; forces threshold to 1
• asyncMode — run internal LLM calls concurrently (default true)
• verboseMode — print intermediate steps to console

Connect the same Memory as AI Agent.

chatbotRole is set in Config or supplied as an item field.

Knowledge Retention

Detects when the bot forgets facts the user already provided earlier in the conversation.

Config

• threshold — pass cutoff; success when score >= threshold (default 0.5)
• includeReason — whether to generate a human-readable reason (default true)
• strictMode — binary 1/0 scoring; forces threshold to 1
• asyncMode — run internal LLM calls concurrently (default true)
• verboseMode — print intermediate steps to console

Connect the same Memory as AI Agent.

Conversation Completeness

Checks whether all user intentions raised in the dialogue were satisfied by the assistant.

Config

• windowSize — intent-window size (default 3)
• threshold — pass cutoff; success when score >= threshold (default 0.5)
• includeReason — whether to generate a human-readable reason (default true)
• strictMode — binary 1/0 scoring; forces threshold to 1
• asyncMode — run internal LLM calls concurrently (default true)
• verboseMode — print intermediate steps to console

Connect the same Memory as AI Agent.

Goal Accuracy

Evaluates whether the agent reached the user's goal and how well its plan and steps supported that outcome.

Config

• threshold — pass cutoff; success when score >= threshold (default 0.5)
• includeReason — whether to generate a human-readable reason (default true)
• strictMode — binary 1/0 scoring; forces threshold to 1
• asyncMode — run internal LLM calls concurrently (default true)
• verboseMode — print intermediate steps to console

Connect the same Memory as AI Agent.

Also connect AI Agent on main (with Return Intermediate Steps) to enrich turns with tools_called from intermediateSteps.

Tool Use

Scores tool selection and argument correctness per interaction against available tools.

Config

• availableTools (required) — tool catalog for selection and argument judging per interaction
• threshold — pass cutoff; success when score >= threshold (default 0.5)
• includeReason — whether to generate a human-readable reason (default true)
• strictMode — binary 1/0 scoring; forces threshold to 1
• asyncMode — run internal LLM calls concurrently (default true)
• verboseMode — print intermediate steps to console

Connect the same Memory as AI Agent.

Also connect AI Agent on main (with Return Intermediate Steps) to enrich turns with tools_called from intermediateSteps. availableTools is required in Config.

Topic Adherence

Penalizes answers to off-topic questions and rewards correct refusals when a question is outside allowed topics.

Config

• relevantTopics (required) — allowed topic list for TP/TN/FP/FN classification
• threshold — pass cutoff; success when score >= threshold (default 0.5)
• includeReason — whether to generate a human-readable reason (default true)
• strictMode — binary 1/0 scoring; forces threshold to 1
• asyncMode — run internal LLM calls concurrently (default true)
• verboseMode — print intermediate steps to console

Connect the same Memory as AI Agent.

Turn Faithfulness

Verifies assistant claims are grounded in retrievalContext attached to turns (RAG chatbots).

Config

• windowSize — sliding window over turns (default 10)
• truthsExtractionLimit — cap truths extracted from retrievalContext per window
• penalizeAmbiguousClaims — penalize idk verdicts on claims (default false)
• threshold — pass cutoff; success when score >= threshold (default 0.5)
• includeReason — whether to generate a human-readable reason (default true)
• strictMode — binary 1/0 scoring; forces threshold to 1
• asyncMode — run internal LLM calls concurrently (default true)
• verboseMode — print intermediate steps to console

Connect the same Memory as AI Agent.

Per-turn retrievalContext may be set in Config or supplied on turns built from Memory.

Turn Contextual Precision

Measures whether relevant retrieval nodes are ranked above irrelevant ones per turn against an expected outcome.

Config

• windowSize — sliding window over turns (default 10)
• threshold — pass cutoff; success when score >= threshold (default 0.5)
• includeReason — whether to generate a human-readable reason (default true)
• strictMode — binary 1/0 scoring; forces threshold to 1
• asyncMode — run internal LLM calls concurrently (default true)
• verboseMode — print intermediate steps to console

Connect the same Memory as AI Agent.

expectedOutcome is Config or an item field.

Turn Contextual Recall

Checks whether retrieved context per turn contains enough information to support the expected outcome.

Config

• windowSize — sliding window over turns (default 10)
• threshold — pass cutoff; success when score >= threshold (default 0.5)
• includeReason — whether to generate a human-readable reason (default true)
• strictMode — binary 1/0 scoring; forces threshold to 1
• asyncMode — run internal LLM calls concurrently (default true)
• verboseMode — print intermediate steps to console

Connect the same Memory as AI Agent.

expectedOutcome is Config or an item field.

Turn Contextual Relevancy

Measures signal-to-noise in each turn's retrievalContext relative to the user's input.

Config

• windowSize — sliding window over turns (default 10)
• threshold — pass cutoff; success when score >= threshold (default 0.5)
• includeReason — whether to generate a human-readable reason (default true)
• strictMode — binary 1/0 scoring; forces threshold to 1
• asyncMode — run internal LLM calls concurrently (default true)
• verboseMode — print intermediate steps to console

Connect the same Memory as AI Agent.

Safety

Safety metrics flag harmful or policy-violating outputs. Supply input and actualOutput on the main connection from any upstream node (typically an LLM or AI Agent).

Bias

Detects gender, racial, political, or geographical bias in opinions expressed in the output. Lower scores are safer; success when score <= threshold.

Config

• threshold — pass cutoff; success when score <= threshold (default 0.5)
• includeReason — whether to generate a human-readable reason (default true)
• strictMode — binary 1/0 scoring; forces threshold to 0
• asyncMode — run internal LLM calls concurrently (default true)
• verboseMode — print intermediate steps to console

Toxicity

Flags toxic opinions (attacks, mockery, hate, threats). Lower scores are safer; success when score <= threshold.

Config

• threshold — pass cutoff; success when score <= threshold (default 0.5)
• includeReason — whether to generate a human-readable reason (default true)
• strictMode — binary 1/0 scoring; forces threshold to 0
• asyncMode — run internal LLM calls concurrently (default true)
• verboseMode — print intermediate steps to console

Non-Advice

Detects inappropriate professional advice (financial, medical, legal, and so on) that should be deferred to licensed experts.

Config

• adviceTypes (required) — prohibited advice categories (e.g. financial, medical, legal)
• threshold — pass cutoff; success when score >= threshold (default 0.5)
• includeReason — whether to generate a human-readable reason (default true)
• strictMode — binary 1/0 scoring; forces threshold to 1
• asyncMode — run internal LLM calls concurrently (default true)
• verboseMode — print intermediate steps to console

Misuse

Flags when a domain-specific bot answers off-topic or general-knowledge requests outside its scope.

Config

• domain (required) — bot's allowed domain (e.g. financial)
• threshold — pass cutoff; success when score <= threshold (default 0.5)
• includeReason — whether to generate a human-readable reason (default true)
• strictMode — binary 1/0 scoring; forces threshold to 0
• asyncMode — run internal LLM calls concurrently (default true)
• verboseMode — print intermediate steps to console

PII Leakage

Detects personally identifiable information exposed in the output (names, financial, medical, government IDs, and so on).

Config

• threshold — pass cutoff; success when score >= threshold (default 0.5)
• includeReason — whether to generate a human-readable reason (default true)
• strictMode — binary 1/0 scoring; forces threshold to 1
• asyncMode — run internal LLM calls concurrently (default true)
• verboseMode — print intermediate steps to console

Role Violation

Binary check for a single-turn output breaking the assigned role or persona (breaking character, policy violations, and so on). Score is higher-is-safer: 1.0 when no violation, 0.0 when any violation is detected.

Config

• role (required) — expected persona (e.g. helpful assistant)
• threshold — pass cutoff; success when score >= threshold (default 0.5)
• includeReason — whether to generate a human-readable reason (default true)
• strictMode — binary 1/0 scoring; forces threshold to 1 (pass only on a perfect 1.0 score)
• asyncMode — run internal LLM calls concurrently (default true)
• verboseMode — print intermediate steps to console

Upstream DeepEval docs describe strictMode inconsistently (Bias-style “0 for perfection” vs higher-is-safer FAQ). This node follows higher-is-safer semantics and pins strictMode to threshold 1 accordingly.

Others

General-purpose metrics for summarization, prompt compliance, and factual grounding.

Summarization

Scores whether a summary is factually aligned with the source and covers required details (alignment + coverage).

Config

• n — number of auto-generated assessment questions when assessmentQuestions is not set (default 5)
• assessmentQuestions — custom coverage questions; skips auto-generation
• truthsExtractionLimit — cap source truths for alignment check
• threshold — pass cutoff; success when score >= threshold (default 0.5)
• includeReason — whether to generate a human-readable reason (default true)
• strictMode — binary 1/0 scoring; forces threshold to 1
• asyncMode — run internal LLM calls concurrently (default true)
• verboseMode — print intermediate steps to console

Prompt Alignment

Checks whether the output follows each instruction listed in your prompt template.

Config

• promptInstructions (required) — instructions from the prompt template to check compliance against
• threshold — pass cutoff; success when score >= threshold (default 0.5)
• includeReason — whether to generate a human-readable reason (default true)
• strictMode — binary 1/0 scoring; forces threshold to 1
• asyncMode — run internal LLM calls concurrently (default true)
• verboseMode — print intermediate steps to console

Hallucination

Measures contradictions between the output and ground-truth context. Lower scores are better; success when score <= threshold.

Config

• threshold — pass cutoff; success when score <= threshold (default 0.5)
• includeReason — whether to generate a human-readable reason (default true)
• strictMode — binary 1/0 scoring; forces threshold to 0
• asyncMode — run internal LLM calls concurrently (default true)
• verboseMode — print intermediate steps to console

Community

Community metrics are contributed DeepEval extensions. Citation Faithfulness imports from deepeval.metrics.community. Agent Loop Detection and Tool Permission are grouped here for product navigation; with the pinned runtime (deep-eval-web / deepeval v4.0.7), import paths may be deepeval.metrics.community or deepeval.metrics depending on the wheel build — the node resolves the correct class at runtime.

Citation Faithfulness

Stricter than Faithfulness: every [N] citation in the output must point to the passage that actually supports that claim.

Config

• threshold — pass cutoff; success when score >= threshold (default 1.0)
• includeReason — whether to generate a human-readable reason (default true)
• strictMode — binary 1/0 scoring; forces threshold to 1
• asyncMode — run internal LLM calls concurrently (default true)
• verboseMode — print intermediate steps to console

Agent Loop Detection

Deterministic detection of infinite loops in an agent trace (tool repetition, reasoning stagnation, call-graph cycles). No LLM required.

Config

• repetitionThreshold — identical tool-call count before repetition penalty (default 3)
• similarityThreshold — reasoning stagnation similarity cutoff (default 0.85)
• checkToolRepetition — enable tool-repetition sub-signal (default true)
• checkReasoningStagnation — enable reasoning stagnation check (default true)
• checkCallGraphCycles — enable call-graph cycle detection (default true)
• threshold — pass cutoff; success when score >= threshold (default 0.5)
• includeReason — whether to generate a human-readable reason (default true)
• strictMode — binary 1/0 scoring; forces threshold to 1
• verboseMode — print intermediate steps to console

Trace-only and deterministic. Maps intermediateSteps to a synthetic DeepEval trace. No Language Model sub-node. Fails clearly if intermediate steps are missing.

Tool Permission

Enforces least privilege: flags any tool call outside an allowlist or on a denylist. Deterministic; no LLM required.

Config

• allowedTools — allowlist (least privilege); at least one of allowedTools or deniedTools is required
• deniedTools — denylist (deny wins over allow)
• threshold — pass cutoff; success when score >= threshold (default 1.0)
• includeReason — whether to generate a human-readable reason (default true)
• strictMode — binary 1/0 scoring; forces threshold to 1
• verboseMode — print intermediate steps to console

Maps intermediateSteps → DeepEval tools_called. No Language Model sub-node. No asyncMode (deterministic, synchronous in DeepEval).

DeepEval Aggregate

Fan-in transformer for an evaluation branch. It collects normalized metric results, computes the overall score and success state, and prepares a row for n8n's official Data Table node. The importable example persists that row with DeepEval Aggregate → Data Table (Insert).

Config

• dataTableId — sink table identity included in the output
• writeMode — intended downstream persistence mode
• output column names for run ID, score, success, and serialized metrics
• metrics — which incoming metric nodes to include (allConnected or explicit list)
• passRule — overall success rule (allPass, anyFail, and so on)

Sinks

This pass supports n8n Data Tables only, through the built-in Data Table node connected after Aggregate. Google Sheets and Excel sinks are deferred.

Typical wiring

Batch eval (dataset-driven): import kitchenSinkNonConversational.workflow.json for a full single-turn and agentic benchmark, or kitchenSinkConversational.workflow.json for all Memory-backed metrics. Both wire Data Table (Get rows) → DeepEval Trigger → enrich → AI Agent → parallel metrics → DeepEval Aggregate → Data Table (insert).

Live chat eval (conversational metrics — Memory required):

Connect the same Memory sub-node to AI Agent and to each conversational / turn-based metric. Enable Return Intermediate Steps on AI Agent when using Goal Accuracy, Tool Use, or other metrics that enrich turns from intermediateSteps.

DeepEval Consistency

Cross-run aggregation for multi-run benchmarks. Consumes DeepEval Aggregate outputs, groups them by evalContext.caseId, and emits per-case score variance, coefficient of variation, and optional label entropy or agreement. The importable example wires Data Table (Get rows) → DeepEval Trigger (runsPerRow: 3) → enrich → AI Agent → G-Eval → DeepEval Aggregate → DeepEval Consistency → Data Table (Insert).

Config

• groupByField — evalContext field used to group runs (default caseId)
• labelField — optional item path for categorical labels (required for entropy / agreement basis)
• consistencyBasis — cv (default), entropy, or agreement
• dataTableId — sink table identity included in the output
• writeMode — intended downstream persistence mode (append or upsert)
• output column names for case ID, mean score, consistency score, and serialized stats

Multi-run wiring: set runsPerRow on DeepEval Trigger, run metrics and Aggregate as usual (Aggregate emits one item per evalContext.runId), then fan in Aggregate outputs to DeepEval Consistency for one row per case.