AI Evaluation Engineer

The Role

We're hiring an AI Evaluation Engineer to own the quality bar for every LLM-powered feature we ship. You'll design, build, and scale the infrastructure that tells us — with evidence — whether a prompt change, model swap, or agent refactor made things better or worse.

This is a high-leverage role. Every customer-facing AI capability at FirstIgnite flows through your evals. You'll work directly with the Head of Engineering and partner closely with product, applied AI, and the full-stack team to establish evaluation as a first-class discipline across the company.

What You'll Do

• Build evaluation infrastructure: Design and maintain eval suites using Promptfoo, LLM-as-judge methodologies, and custom harnesses for features like our expert search system, natural language grants search, and AI SDR agents.
• Define what good means: Partner with product and domain experts to translate fuzzy customer outcomes (does this surface the right p...