AI Software Development Engineer in Test
TalentOlaAbout the role
Role: AI SDET
Location –Sunnyvale, CA (Onsite)
Contract role
Looking for 10+ years of experience
Job Summary:
We are seeking an innovative AI Software Development Engineer in Test (AI-SDET) to join our client’s quality engineering team. An ideal candidate would bridge the gap between traditional QA and artificial intelligence by designing self-healing test automation frameworks, validating Large Language Models (LLMs), and integrating AI agents into CI/CD.
Experience Required:
6 to 8 years of hands on experience in software quality engineering, test automation and AI ML development.
Qualifications/Required Skills:
Bachelor’s degree in computer science, or a related field.
Proficiency in Python/Java/Scala.
Proficiency with modern tools - Playwright.
Experience with LLM APIs (e.g. Claude, OpenAI) and agent frameworks (e.g. LangChain0
Clouse based plotforms with AI integration ( Azure Open AI, AWS Open AI)
Strong understanding of CI/CD pipelines and containerization (Docker).
Strong experience validating complex RESTful APIs, microservices and databases (SQL/NoSQL).
Experience with Retail Domain is preferred, not a constraint though
Key responsibilities:
AI Framework Architecture: Architect and maintain enterprise-grade, scalable automation frameworks.
GenAI and Agentic automation: Build intelligent test agents, automated prompt-engineering validators, and self-healing test execution scripts.
Model and Evals validation: Design and maintain evaluation frameworks covering hallucination detection, roundedness, toxicity, bias and safety guardrails.
Synthetic Data Generation: Design AI-driven data generation pipelines to simulate complex, production-like testing scenarios.
ML Pipeline testing: Validate RAG (Retrieval-Augmented Generation) architectures, embedding quality, retrieval accuracy, and vector search relevance.
Intelligent testing: Build smart test selection and impact analysis systems that predict which test suits to run based on specific code changes.
Framework Development: Develop and enhance scalable, robust automation scripts for UI, API, and backend validation using modern languages and frameworks.
CI CD Integration: Work alongside MLOps teams to build continuous testing and automated verification pipelines.
Behavioral Skills:
Strong analytical and problem-solving skills with a passion for adoption new AI Technologies.
Excellent Communication skills and collaboration skills
Ability to propose and implement improvements in the system
Ability to work with cross-functional stakeholders
Responsibilities
- Architect and maintain enterprise-grade, scalable automation frameworks
- Build intelligent test agents and automated prompt-engineering validators
- Design and maintain evaluation frameworks for model validation
- Design AI-driven data generation pipelines for testing scenarios
- Validate RAG architectures and embedding quality
- Build smart test selection and impact analysis systems
- Develop and enhance scalable automation scripts for validation
- Work alongside MLOps teams for continuous testing integration
Qualifications
- Bachelor’s degree in computer science or related field
- Proficiency in Python/Java/Scala
- Experience with modern tools like Playwright
- Experience with LLM APIs and agent frameworks
- Strong understanding of CI/CD pipelines and containerization
- Experience validating complex RESTful APIs and databases
Skills mentioned
About TalentOla
TalentOla is a search firm focused on meeting our customers’ diversified talent needs across industry verticals and functions. We have expertise in supporting clients’ deep technical and techno commercial expertise. We have a one-of-a-kind AI led search and delivery model that uses technologies across the entire recruitment life cycle. Our processes provide our experienced recruiters with the actionable intelligence needed to understand customers' real needs and find the best talent across the globe extremely fast. TalentOla is a journey embarked on by a group of senior talent professionals and is mentored by the industry veterans with a mission to increase workforce diversity. TalentOla is committed to supporting local initiatives by partnering with organizations that share our values, both in cash and kind