Skip to content

promptfoo

by Promptfoo (now part of OpenAI)

Open-source CLI and library to evaluate and red-team LLM apps, agents and RAG systems in CI.

  • Self-hosted
  • Small business
  • Mid-size
  • Enterprise
  • Open source
  • Free tier

From promptfoo's own website and public documents, updated 4 Nov 2026. Not a hands-on review: I add a verdict only to products I use myself.

What is promptfoo?

Promptfoo is a CLI and library for evaluating and red-teaming LLM-based apps, including prompts, agents and RAG systems. It runs automated evaluations and compares models side by side, among them OpenAI, Anthropic, Azure, Bedrock and Ollama.

Red teaming covers vulnerability scanning and security vulnerability reports. A code scanning feature reviews pull requests for LLM-related security and compliance issues. Configs are declarative and checks can run in CI/CD pipelines. The README says evals run locally, so prompts stay on the user's machine.

The licence is MIT and the repository shows about 26.8k stars. The README states that Promptfoo is now part of OpenAI and that the project remains open source.

Best forDevelopers who want LLM tests and red-team scans inside their CI pipeline

promptfoo features

  • LLM evals

    Automated evaluations and side-by-side model comparison.

  • Red teaming

    Vulnerability scanning with security reports.

  • Code scanning

    Reviews pull requests for LLM security and compliance issues.

  • CI/CD checks

    Declarative configs run in pipelines.

  • Local runs

    Evals run locally, according to the README.

  • Many providers

    OpenAI, Anthropic, Azure, Bedrock and Ollama.

promptfoo integrations

  • OpenAI
  • Anthropic
  • Azure
  • Amazon Bedrock
  • Ollama

From the vendor's own integration list or documentation.

promptfoo pricing

Starting price

Free (open source)

  • Pricing modelFlat fee
  • Free tierYes
  • Free trialNo

Current prices on the vendor's site · Checked 4 Nov 2026. Prices change often; the vendor's page wins.

Security and compliance

Certifications
None listed on a public trust page
EU data residency
Not documented

Where it fits (NIST CSF 2.0)

  1. Govern (not covered)
  2. Identify
  3. Protect
  4. Detect (not covered)
  5. Respond (not covered)
  6. Recover (not covered)

Independent audits

No public independent audit found.

Good to know before you buy

  • Promptfoo is now part of OpenAI, which matters if vendor neutrality is a requirement.
  • The README does not describe a paid enterprise offering.

promptfoo alternatives

garak

NVIDIA

Open-source NVIDIA scanner that probes LLMs for jailbreaks, prompt injection, leakage and toxicity.

Commercial platform for continuous AI red teaming and attack-led security of AI systems and agents.

Agent security platform with AI-SPM, runtime detection and response, access control and red teaming.

PyRIT

Microsoft

Open-source Microsoft AI Red Team toolkit for scanning and attacking generative AI systems.

promptfoo vs garakpromptfoo vs Mindgardpromptfoo vs Noma Security

Frequently asked questions

What is promptfoo?

Open-source CLI and library to evaluate and red-team LLM apps, agents and RAG systems in CI.

How much does promptfoo cost?

promptfoo starts at Free (open source), according to its pricing page (checked 4 Nov 2026).

Is there a free version of promptfoo?

Yes. promptfoo has a free plan or tier; check the pricing page for its limits.

What are the alternatives to promptfoo?

Products in the same category include garak, Mindgard, Noma Security, PyRIT.

Have you tested promptfoo?

Not yet. This page lists published facts only. When I test it, my verdict will appear at the top.

Sources (2), checked 4 Nov 2026
  1. github.com/promptfoo/promptfoo
  2. promptfoo.dev

Spotted something out of date? Tell me through the contact page and I will fix it.

By using this site, you agree to the Privacy Policy.