Senior AI Engineer

About the role

Who We Are

DoubleVerify is the leading independent provider of marketing measurement software, data and analytics that authenticates the quality and effectiveness of digital media for the world's largest brands and media platforms. DV provides media transparency and accountability to deliver the highest level of impression quality for maximum advertising performance. Since 2008, DV has helped hundreds of Fortune 500 companies gain the most from their media spend by delivering best-in-class solutions across the digital advertising ecosystem, helping to build a better industry. Learn more at www.doubleverify.com.

Role Overview

We are looking for a talented and driven Senior AI Engineer to join the Pinnacle AI (PinAI) team at DoubleVerify. In this role, you will be a core builder of AgentOS — our agentic AI platform that powers conversational, intelligent experiences across the Pinnacle advertising analytics suite.

You will design and implement AI agents, Model Context Protocol (MCP) servers, and LLM-backed tools that allow Fortune 500 clients to interact with their advertising data through natural language. This is a hands-on engineering role for someone who is passionate about applied AI, thrives in a fast-moving environment, and wants to see their work directly impact how the world's largest advertisers make decisions.

What You'll Do

Agentic AI Development

Design, build, and maintain AI agents within the AgentOS framework using architectures based on MCP (Model Context Protocol), LangChain, LangGraph, and related tooling

Develop new MCP servers and tools that expose Pinnacle analytics capabilities — including metrics, dimensions, settings, and user data — to AI agents

Implement multi-turn conversational flows, tool-use reasoning chains, and agent orchestration logic that handle complex, real-world advertising analytics queries

Build and maintain the chat UI integration layer, connecting AgentOS to the Pinnacle front-end experience

LLM Engineering & Optimization

Work with major LLM providers (OpenAI, Anthropic, Google, and others) to select, prompt, and integrate models appropriate for specific agent tasks

Develop and own prompt engineering strategies, including system prompts, few-shot examples, and chain-of-thought techniques

Implement Retrieval-Augmented Generation (RAG) pipelines to ground AI responses in Pinnacle's proprietary data and documentation

Continuously optimize model performance through evals, latency tuning, and cost management

Evaluation & Quality

Build and maintain evaluation (evals) frameworks that measure agent accuracy, instruction-following, hallucination rates, and user satisfaction

Define quality benchmarks and run regression evals on every agent change before production deployment

Instrument agents with observability tooling (e.g., LangSmith, Weights & Biases, or custom tracing) to support debugging and continuous improvement

Integration & Platform Engineering

Integrate AgentOS with Pinnacle's existing platform services — including PIAM/Keycloak authentication, Settings APIs, and analytics data layers

Collaborate with backend teams to build and consume APIs that surface data cleanly and securely to AI agents

Participate in architecture reviews and contribute to technical design documents for new AI-powered features

Cross-Functional Collaboration

Partner closely with Product, UX, and Data Science to translate business requirements into agent capabilities and conversational flows

Work with QE teams to define testing strategies for AI features, including behavioral testing and adversarial prompting

Contribute to the team's shared AI engineering standards, best practices, and documentation

Who You Are

Required Qualifications

Experience:

5+ years of professional software engineering experience building production-grade systems

2+ years of hands-on experience developing LLM-based applications or AI agent systems

Demonstrated experience shipping AI-powered features end-to-end, from prototype to production

Technical Expertise:

Agentic AI Frameworks: Solid hands-on experience with LangChain, LangGraph, or equivalent agent orchestration frameworks

Model Context Protocol (MCP): Familiarity with MCP or similar modular tool-integration protocols for AI agents (or strong appetite to ramp quickly)

LLM Integration: Experience calling and integrating with LLM APIs (OpenAI, Anthropic Claude, Google Gemini, or similar), including streaming, tool-calling, and structured output patterns

Python: Strong Python proficiency; ability to build clean, testable, production-quality backend services (FastAPI experience a plus)

RAG & Embeddings: Understanding of retrieval-augmented generation patterns, vector stores, and semantic search

Evaluation Mindset: Experience building or using eval frameworks to measure LLM/agent quality and regression-test AI behavior

Cloud & Infrastructure: Familiarity with cloud environments (GCP, AWS, or Azure) and containerization basics (Docker/Kubernetes)

Soft Skills:

Strong problem-solving skills with a bias toward shipping and iterating

Clear communicator — able to explain AI behavior and limitations to both technical peers and non-technical stakeholders

Comfortable operating with ambiguity in a fast-paced, evolving AI landscape

Collaborative team player who contributes actively to shared standards and code quality

Education:

BS/MS in Computer Science, Machine Learning, AI, or a related technical field

Preferred Qualifications

Experience with MCP server development and tool-use patterns in production agentic systems

Familiarity with IAM, Keycloak, or OAuth 2.0-based authentication patterns

Knowledge of advertising analytics, AdTech platforms, or BI tools (Looker, Snowflake, etc.)

Experience with modern front-end frameworks (React/Angular) for conversational UI features

Contributions to open-source AI projects or published work in AI/ML

Experience with agent evaluation tools such as DeepEval, LangSmith, or Braintrust

Knowledge of responsible AI practices, including bias mitigation and hallucination reduction strategies