Staff Applied AI Engineer, Product & Agent Performance
LaddersAbout the role
For our client, we are seeking a Staff Applied AI Engineer, Product & Agent Performance to join the team of a leader in the Healthcare space.
This role will shape how AI agents behave across healthcare workflows, with a strong focus on performance, safety, and reliability. You will partner closely with cross-functional stakeholders to turn production evidence into practical model and system improvements. The position is highly technical and hands-on, blending software engineering, evaluation design, and applied AI to improve patient care delivery.
Location: Remote - US based candidates only, no visa sponsorship available
Compensation: $175,000 – $200,000 annually
Responsibilities
Design and iterate on agent behavior across healthcare workflows
Architect retrieval and context to ensure data integrity for AI models
Handle memory and state in multi-turn agent interactions
Create prompt templates integrating structured formatting and reasoning
Conduct evaluations to measure performance and identify regressions
Author evaluation rubrics that weight failures by severity and cost
Design escalation paths for agents based on confidence levels
Qualifications
8+ years of production software engineering experience
3+ years of experience in ML, LLM, or agentic systems
Demonstrated expertise in diagnosing agent failures and implementing effective solutions
Hands-on experience with RAG architecture and frameworks for evaluating automated systems
Familiarity with AWS AI/ML services, particularly Bedrock and SageMaker
Ability to make evidence-led judgments, with experience in experimental design and implementation
Benefits
Ownership over agent performance and safety metrics for healthcare workflows
Cross-functional role linking production evidence to AI enhancements
Mission-driven work aimed at improving patient care delivery
Flexible and remote-friendly work culture
Access to employee-driven development programs
Integration into a diverse and purpose-driven community
Our client is an equal opportunity employer. We encourage you to apply even if you don’t meet every qualification—your background could be exactly what this team needs.
Responsibilities
- Design and iterate on agent behavior across healthcare workflows
- Architect retrieval and context to ensure data integrity for AI models
- Handle memory and state in multi-turn agent interactions
- Create prompt templates integrating structured formatting and reasoning
- Conduct evaluations to measure performance and identify regressions
- Author evaluation rubrics that weight failures by severity and cost
- Design escalation paths for agents based on confidence levels
Qualifications
- 8+ years of production software engineering experience
- 3+ years of experience in ML, LLM, or agentic systems
- Demonstrated expertise in diagnosing agent failures and implementing effective solutions
- Hands-on experience with RAG architecture and frameworks for evaluating automated systems
- Familiarity with AWS AI/ML services, particularly Bedrock and SageMaker
- Ability to make evidence-led judgments, with experience in experimental design and implementation
Benefits
- Ownership over agent performance and safety metrics for healthcare workflows
- Cross-functional role linking production evidence to AI enhancements
- Mission-driven work aimed at improving patient care delivery
- Flexible and remote-friendly work culture
- Access to employee-driven development programs
- Integration into a diverse and purpose-driven community
Skills mentioned
About Ladders
We lead the leaders. Ladder’s community of 10 mm leading professionals come to us to be inspired, grow in their careers, and get ahead in their professional lives. We focus on the $100K+ job market, representing 25% of all jobs and about 50% of income in the US & Canada. We build data-driven tools, marketplaces, news and entertainment products for our growing audience, and the companies, recruiters, and advertisers that want to reach them. We work best with people who enjoy using their talents, commitment and hard work to achieve great successes for millions of real life users through products that impact their careers. We are deeply technology-driven -- about half the company are engineers, technical product people or product designers -- but we never build technology for technology’s sake. We’re always trying to connect our efforts to the good it can do in people’s lives.
H-1B sponsorship history
Historical employer filing data was found for Ladders. The employer record includes 8 historical certified applications. This is employer-level history, not a guarantee that this role currently offers sponsorship.