Senior Software Engineer - Model Hosting
CalanceAbout the role
Job Type: Direct Hire
Job Title: Senior Software Engineer - Model Hosting
Location: Remote
Travel: Yes, 10 days per year
Must Be authorized to work in the USA: No Sponsorship
Must Have: Hosting Models, IaaS
Compensation: $180K+ plus bonus and benefits.
The Model Hosting team sits at the heart of our infrastructure, working deeply within our bare metal and Kubernetes stack to ensure every bit of compute is used to its fullest. If you're excited about squeezing maximum performance out of hardware and building the systems that serve large language models at scale, we'd love to talk.
About the Role
You'll help design and operate the platform that serves our models in production. Our stack spans several languages, each chosen for the job it does best: Go powers our gateway layer, Rust handles fast real-time decision-making, and Python supports model dependencies that require it. Kubernetes orchestrates all model deployments across the company.
Your work will touch some of the most interesting problems in high-performance model serving, including:
RDMA technologies (InfiniBand and RoCE) across multiple nodes
Disaggregated serving and KV cache offloading
Quantization, speculative-decoding, MTP, and other inference performance optimizations
Working with modern inference engines such as SGLang, vLLM, and others
What We're Looking For
We are primarily looking for someone with 1-3 years of experience hosting models (a fairly new field), but who also has 5-10+ years of broader software engineering experience.
Experience serving LLMs in production
Strong programming skills in Python and/or Go
A track record of designing and operating highly scalable, highly available distributed services
Working knowledge of InfiniBand, RoCE, or other high-performance networking
Nice to Have
Contributions to vLLM, SGLang, TRT-LLM, or other NVIDIA ecosystem open-source projects
Deep understanding of Kubernetes
Experience developing in Rust
Familiarity with storage over RDMA
Responsibilities
- Design and operate the platform that serves models in production
- Work with high-performance model serving problems
- Utilize RDMA technologies across multiple nodes
- Implement disaggregated serving and KV cache offloading
- Optimize inference performance
Qualifications
- 1-3 years of experience hosting models
- 5-10+ years of broader software engineering experience
- Experience serving LLMs in production
- Strong programming skills in Python and/or Go
- Track record of designing and operating highly scalable distributed services
- Working knowledge of InfiniBand, RoCE, or high-performance networking
Benefits
- Bonus and benefits package
Skills mentioned
About Calance
Calance is a global IT services and technology solutions provider with over 28 years of expertise in Business Intelligence, Data Science, Generative AI, Application Development, Robotic Process Automation (RPA), DevOps Enablement, Managed Services, Cybersecurity, and IT Staffing. Headquartered in Southern California, Calance operates across the United States, Canada, and India, offering reliable, innovative, and customized solutions for diverse industries. Why Calance? Our strength lies in our ability to leverage cutting-edge technologies like Generative AI to deliver tailored solutions that optimize digital ecosystems. We help businesses automate processes, unlock data-driven insights, and streamline operations to stay competitive. Our focus on collaboration ensures we understand your unique challenges and deliver results that align with your goals. Services Offered 1. Business Intelligence & Data Science: Harness the power of data for informed decisions. 2. Generative AI Solutions: Develop AI tools to enhance user experiences and workflows. 3. RPA & DevOps Enablement: Simplify processes and ensure efficient development pipelines. 4. Managed Services & Cybersecurity: Secure, monitor, and maintain IT infrastructure. 5. Application Development: Build scalable, business-focused apps tailored to your needs. Global Presence & Impact Calance combines global expertise with local insights to address industry-specific challenges in healthcare, finance, retail, and manufacturing. Our experienced teams in three countries bring innovative strategies and solutions to clients worldwide. Your Vision, Our Mission Explore our case studies to discover how we have partnered with businesses to achieve exceptional outcomes. Calance goes beyond providing IT solutions; we are committed to empowering companies to innovate and grow sustainably. Connect with us today to transform your ideas into reality with expert talent, advanced technology, and disciplined execution.