A local-first CLI and library for evaluating and red-teaming LLM applications, enabling systematic assessment of prompts, models, agents, and RAG pipelines through declarative YAML configuration.
promptfoo is a TypeScript-based LLM evaluation and security testing platform driven by declarative YAML configuration (promptfooconfig.yaml), executing all assessments locally to ensure data privacy. Its core capabilities span three dimensions: evaluation, security, and engineering integration.
For evaluation, promptfoo supports systematic prompt variant assessment through variable interpolation and assertion mechanisms, enabling side-by-side comparison of 60+ LLM providers (OpenAI, Anthropic, Google, AWS Bedrock, Azure, Ollama, Hugging Face, etc.) on the same test set. It provides specialized metrics for RAG pipelines including factuality, context relevance, and faithfulness, along with multi-turn agent validation for task completion and tool call traces.
For security, promptfoo features a built-in red-teaming engine that automatically generates adversarial inputs to detect prompt injection, jailbreaking, harmful content generation, PII leakage, and privilege escalation, aligned with OWASP LLM Top 10 standards. A standalone code-scan-action GitHub Action enables PR-level code security scanning.
For engineering, promptfoo offers full CI/CD integration with GitHub Actions for automated evaluation gates; custom model integration via JavaScript, Python, Ruby, Shell, or HTTP API; MCP protocol support for advanced agent capabilities; containerized deployment through Dockerfile and Helm Chart; and a Web UI (promptfoo view) for visual result comparison. The project uses a pnpm monorepo structure with Vitest for testing, Drizzle ORM for persistence, and Biome for code quality. Announced acquisition by OpenAI on March 9, 2026, with commitment to continued open-source maintenance.
Installation:
npm install -g promptfoo
brew install promptfoo
pip install promptfoo
No-install usage: npx promptfoo@latest <command>
Quick Start:
promptfoo init --example getting-started
export OPENAI_API_KEY=sk-abc123
cd getting-started
promptfoo eval
promptfoo view
Key CLI Commands:
promptfoo init— Interactive config creationpromptfoo eval— Run evaluationspromptfoo eval setup— Web UI config wizardpromptfoo view— Open Web viewerpromptfoo redteam— Run red-team tests
Configuration Example (promptfooconfig.yaml):
prompts:
- 'Convert the following English text to {{language}}: {{input}}'
providers:
- openai:chat:gpt-5.4
- anthropic:messages:claude-opus-4-6
tests:
- vars:
language: French
input: Hello world
assert:
- type: contains
value: 'Bonjour le monde'
Unconfirmed: Acquisition closing status, Python package feature parity with npm, Promptfoo Cloud details and pricing, RAG evaluation metric calculation methods, red-team adversarial strategies, complete assertion type list.