Software Engineer
Harvey NashAbout the role
We are seeking a Software Engineer to help build and scale the tooling we use to measure how well AI-powered software development tools actually perform. You'll develop evaluation harnesses, automate benchmark runs, and help make sure the results we produce are reproducible and hold up to scrutiny. This is an engineering role, but a lot of the work is about getting the measurement right, not just automating it.
Key Responsibilities
Build and integrate evaluation harnesses and automation for software development use cases, including turning real engineering artifacts like merged pull requests into repeatable benchmark tasks.
Build versioned, repeatable processes to evaluate AI tools, models, and harnesses with reproducible run environments (pinned dependencies, containerized runs,.isolated worktrees) so results stay comparable over time.
Validate and calibrate evaluation approaches against human judgment, so scores are consistent and correct rather than just repeatable.
Analyze results across repeated runs, looking at variance, failure patterns, and cost per outcome, and finding ways to make the workflows more reliable and more automated.
Work with engineering and data teams to improve the tooling, and document how the evaluations work and what they find for both technical and leadership audiences.
Job Title: Software Engineer — AI Evaluation & Automation.
Location: San Diego CA (Hybrid).
Duration: 6+ Months Contract with potential extension.
Required Skills & Experience
Must have experience using existing AI coding tools with harnesses such Claude, Devin, or Cursor.
Proficiency in Python and experience with Java, JavaScript, or a similar programming language.
Hands-on knowledge/experience of Git, including branches, history, and working trees.
Experience with Docker/containerization, APIs, development environments, and CI/CD pipelines.
Experience with AI/LLM/agent evaluation concepts, including benchmark reliability, repeated runs, and evaluation accuracy.
A reasonable, good-faith estimate of the minimum and maximum hourly wage for this position rate may differ based on current location & experience level. Benefits will be available, and details are available at the following links:
Benefits Details: https://view.onedigital.com/harveynash2026contractors
401K Plan: Our employees work hard, which is why Harvey Nash is proud to contribute to their hard-earned savings with a 401(k)-retirement plan that includes a 10% company match on all deferrals. We also offer a Roth 401(k) for even more flexibility. Employees 21 years of age or older, and have completed 3 months of service, are eligible to participate.
Responsibilities
- Build and integrate evaluation harnesses and automation for software development use cases
- Build versioned, repeatable processes to evaluate AI tools, models, and harnesses
- Validate and calibrate evaluation approaches against human judgment
- Analyze results across repeated runs and find ways to improve workflows
- Work with engineering and data teams to improve tooling and document evaluations
Qualifications
- Experience using existing AI coding tools like Claude, Devin, or Cursor
- Proficiency in Python and experience with Java, JavaScript, or similar languages
- Hands-on knowledge of Git, including branches and history
- Experience with Docker/containerization, APIs, development environments, and CI/CD pipelines
- Experience with AI/LLM/agent evaluation concepts
Benefits
- 401(k) retirement plan with 10% company match
- Roth 401(k) for flexibility
Skills mentioned
About Harvey Nash
We’re Harvey Nash, and we build amazing technology and digital teams. From senior appointments through to recruitment and project solutions, we are experts in the tech sector. Established in 1988, we have helped over half the world’s leading companies recruit, retain, source and manage the highly skilled tech talent they need to succeed in an increasingly competitive, global and technology-driven world. With over 3.300 staff in more than 49 offices across Europe, Asia-Pacific and the U.S. and Canada, we have built a reputation amongst candidates and clients alike for our innovative, open and fair approach to business. We work with clients, both big and small, to deliver a range of key recruitment solutions including Executive Search & Interim, Permanent and Contractor Recruitment, Direct Sourcing, MSP & VMS Partnership, NextGen – our Hire Train Deploy solution through to broader workforce solutions. You can find out more about us, our recruitment solutions, our people and our insights at www.harveynash.com. Please be aware that Harvey Nash will never contact you using personal email accounts such as Gmail or Hotmail. Genuine communications will only come from an official Harvey Nash email domain. If you are unsure whether a message is legitimate, please contact us through our official LinkedIn page or email info@harveynash.com.
H-1B sponsorship history
Historical employer filing data was found for Harvey Nash. The employer record includes 346 historical certified applications. This is employer-level history, not a guarantee that this role currently offers sponsorship.