Staff Software Engineer, Data Infrastructure

About the role

## About the Role

Instacart is seeking a **Staff Software Engineer, Data Infrastructure** to join the **Data Governance and Foundations Team** and serve as a senior technical leader for the company's data platform. The role focuses on the architecture and delivery of Instacart's open lakehouse foundation, data governance and access patterns, and multi-engine compute strategy, balancing reliability, scalability, maturity, and cost efficiency.

You will translate data strategy into a multi-year architecture roadmap and work closely with engineering leadership and teams across Data Science, ML Platform, Ads Infrastructure, Finance Engineering, Product Engineering, and Security. The role offers significant ownership and the opportunity to shape large-scale data infrastructure using technologies including **Apache Iceberg, Apache Flink, Trino, ClickHouse, Apache Kafka, Apache Spark, Snowflake, Databricks, Airflow, dbt, Scala, Python, Postgres, and AWS**.

## Key Responsibilities

• Translate Instacart's data strategy, including monetization, federated access, and real-time data, into an actionable multi-year architecture roadmap.
• Align architecture strategy with engineering leadership while evolving the platform for scale, maturity, reliability, and cost efficiency.
• Own the open lakehouse foundation, including unified table formats, storage governance, and a multi-engine compute portfolio.
• Define interactive, batch, and streaming compute capabilities that enable portability and help prevent technology lock-in.
• Drive real-time and streaming infrastructure for critical use cases including Ads, Fraud, and ML.
• Establish deployment patterns, SLAs, and operational practices that balance performance, availability, and infrastructure spend.
• Apply LLM and AI tools to the data platform lifecycle to accelerate development, automation, observability, and cost optimization.
• Partner with teams to embed AI-powered capabilities into the data platform.
• Lead architecture reviews and communicate complex technical and business trade-offs.
• Mentor senior and staff engineers.
• Influence hiring and contribute to engineering team development.
• Build alignment across organizations and leadership levels through architecture documentation and strategy communications.
• Own major platform transitions and migrations, including build-versus-buy decisions, migration design, and risk management.

## Required Qualifications

• **10+ years of software engineering experience** building and operating data infrastructure or distributed systems at production scale.
• Hands-on expertise with modern **data lakehouse architectures** and open table formats such as **Apache Iceberg, Delta Lake, or Hudi**.
• Experience with distributed query or compute engines such as **Trino, Spark, or ClickHouse**, including performance tuning and production reliability.
• Experience with event-driven and streaming infrastructure such as **Kafka or Flink** for real-time pipelines and serving systems.
• Proven ownership of major platform transitions or migrations delivered to production.
• Experience with migration design, risk management, and build-versus-buy decisions.
• Ability to develop cost/benefit and **total cost of ownership (TCO)** models for infrastructure investments.
• Ability to drive technical alignment through clear architecture documents and strategy communications across multiple teams and leadership levels.

## Preferred Qualifications

• Experience designing platform-level data governance controls.
• Familiarity with compliance frameworks such as **SOX, CPRA, or GDPR**.
• FinOps experience optimizing data platform spending.
• Experience managing multi-million-dollar infrastructure budgets.
• Experience negotiating vendor contracts.
• Deep SQL proficiency.
• Strong Python or Scala skills for systems-level development.
• Experience with orchestration tools such as **Apache Airflow**.
• Experience with data transformation pipelines such as **dbt** in large-scale production environments.
• Bachelor's, Master's, or PhD in Computer Science, Computer Engineering, Electrical Engineering, or equivalent practical experience.

## Skills & Competencies

• Staff-level software engineering
• Data infrastructure
• Distributed systems
• Data lakehouse architecture
• Apache Iceberg and open table formats
• Distributed query and compute engines
• Apache Spark
• Trino
• ClickHouse
• Apache Kafka
• Apache Flink
• Real-time and streaming data infrastructure
• Production reliability
• Performance tuning
• Data governance
• Storage governance
• Platform architecture
• Cloud infrastructure
• AWS
• SQL
• Python
• Scala
• Airflow
• dbt
• Platform migrations
• Build-versus-buy analysis
• Risk management
• Cost optimization and FinOps
• TCO modeling
• AI/LLM-assisted engineering
• Architecture documentation
• Technical strategy
• Cross-functional leadership
• Technical mentoring
• Executive communication

## Education & Experience

**Education:**

• Bachelor's, Master's, or PhD in Computer Science, Computer Engineering, Electrical Engineering, or equivalent practical experience is preferred.

**Experience:**

• **10+ years of software engineering experience** building and operating production-scale data infrastructure or distributed systems is required.
• Experience with lakehouse architectures, open table formats, distributed compute/query engines, and streaming infrastructure is required.
• Experience owning major production platform transitions or migrations is required.
• Experience with governance, compliance frameworks, FinOps, SQL, Python/Scala, Airflow, or dbt is preferred.

## Work Arrangement & Schedule

• **Work arrangement:** Remote
• **Work model:** Instacart Flex First
• **Location:** Lansing, MI, US (Remote)
• Employees may choose where they do their best work, including from home, an office, or another preferred location, while participating in regular in-person events.
• **Employment type:** Not explicitly specified in the original job description.
• **Expected weekly hours:** Not specified.
• **Schedule flexibility:** Not specified.
• **Weekend requirements:** Not specified.

## Compensation & Benefits

The base pay range depends on the employee's permanent U.S. work location:

• **California, New York, Connecticut, New Jersey:** $265,000–$279,500 USD
• **Washington:** $254,000–$268,000 USD
• **Oregon, Delaware, Maine, Massachusetts, Maryland, New Hampshire, Rhode Island, Vermont, Washington DC, Pennsylvania, Virginia, Colorado, Texas, Illinois, Hawaii:** $243,000–$256,500 USD
• **All other states:** $221,000–$233,000 USD

Additional compensation and benefits include:

• New-hire equity grant
• Annual equity refresh grants
• Market-competitive compensation and benefits based on work location

## Compliance / Additional Information

• Instacart is committed to an inclusive workplace and provides flexibility through its Flex First work model.
• Compensation offers may vary based on candidate experience and the skills required for the role.
• The listed base pay ranges are dependent on the employee's permanent work location.