AI Engineer, Post-Training & Evaluation
Coval
About the role
The role
Own the models behind Coval’s judges. Coval is the evaluation platform for conversational AI: we help enterprise teams simulate, monitor and improve voice and chat agents before those agents ever reach a customer. Because our product is the judge, this role sits at the center of the company.
You’ll own the model side end to end. You’ll design and ship the LLM-as-a-judge evaluations that score voice and chat conversations at scale, then sharpen them with LoRA fine-tunes on open-weight models and non-tuning work like tool use and context engineering. Your improvements ship directly into the product enterprises rely on to launch dependable voice agents.
What you’ll do
Design and ship the LLM judges that score conversations: rubrics, judge prompting, and calibration against human raters
Build the reliability tracking that keeps judges honest from release to release
Run post-training where training beats prompting: LoRA adapters on open-weight models, trained on data distilled from our own scored conversations
Everywhere else, do the non-tuning capability work: tool use, structured outputs, and context engineering
Keep asking which lever actually moves the metric, and focus there
What we’re looking for
4+ years as an engineer, with strong systems instincts
A track record of shipping measurable quality gains by pairing rigorous measurement with targeted training, not vibes and prompt tweaks
You’ve built eval infrastructure you actually trust
You’ve run LoRA/SFT experiments end to end, including preparing your own data rather than relying on clean data from other teams, so you know which lever really wins
High agency: on a small Series A team, nobody hands you tickets. You figure out what matters next and drive it there
Ambitious and kind. You’ll be part of the group that founds our engineering org, sets its bar and grows the team
Responsibilities
- Design and ship LLM judges to score voice and chat conversations, including rubrics, prompting, and calibration against human raters
- Build reliability tracking to monitor judge performance across releases
- Run LoRA fine-tuning experiments on open-weight models using data distilled from scored conversations
- Improve model capabilities through tool use, structured outputs, and context engineering
- Measure quality gains and prioritize the approaches that improve results
Qualifications
- 4+ years of engineering experience with strong systems instincts
- Track record of delivering measurable quality improvements through rigorous measurement and targeted training
- Experience building trusted evaluation infrastructure
- End-to-end experience running LoRA/SFT experiments, including preparing training data
- High agency and ability to identify priorities and drive work forward on a small team