Engineering Manager, Serving/API

<h3>Accountabilities</h3><li>Lead, coach, hire, and develop engineers working across API serving, tokenization, structured generation, reasoning, and model integration.</li><li>Establish and maintain a clear technical and organizational roadmap covering API compatibility, chat templates, tool calling, reasoning, request budgets, speculative decoding, and new-model support.</li><li>Own API fidelity, including OpenAI-compatible behavior, HTTP endpoints, request validation, streaming, usage accounting, and robust error handling.</li><li>Lead support for reasoning models, including reasoning formats, reasoning-effort controls, token limits, and per-request accounting.</li><li>Partner with compiler and execution teams on speculative decoding, including draft-model integration, request-side infrastructure, acceptance metrics, and policies for evaluating when speculation improves performance.</li><li>Define and lead the roadmap for vision-language model support and interoperability with modern serving frameworks, making informed decisions about what to adopt, integrate, or build internally.</li><li>Ensure serving-layer overhead remains off the critical path for time-to-first-token and streaming latency.</li><li>Design the API layer to remain independent of underlying hardware topology as inference platforms evolve and expand across multiple hosts.</li><li>Establish clear ownership boundaries for shared host-level load balancing, request policies, and system protections in partnership with production platform teams.</li><li>Build robust release gates, including API conformance tests, tool-calling and reasoning evaluations, and regression suites, so new models and capabilities can ship with confidence.</li><li>Collaborate with production platform and orchestration teams to turn new serving capabilities into reliable, supportable production endpoints.</li><li>Translate customer and business priorities into sequenced engineering initiatives while protecting the team from reactive or unstructured demands.</li><li>Establish measurable outcomes around API fidelity, tool-call accuracy, serving latency, throughput, reliability, and customer-visible quality.</li><li>Develop clear ownership boundaries and grow emerging technical leaders as the organization and product surface expand.</li><li>Support longer-term evolution of the function as it scales into areas such as API and protocol compatibility, structured generation, tool calling, speculative decoding, and multimodal serving.</li><h2>Requirements</h2><li>Demonstrated success managing and growing engineering teams responsible for LLM inference, model serving, ML systems, or closely related technical domains.</li><li>Strong systems engineering background with expertise in C++ and working proficiency in Python, including the ability to review performance- and correctness-critical code.</li><li>Proven ability to turn ambiguous product and customer requirements into coherent roadmaps, clear ownership models, and measurable engineering outcomes.</li><li>Experience recruiting, coaching, developing, and retaining engineers across different levels of experience while maintaining high technical standards.</li><li>Strong understanding of LLM serving concepts, including tokenization, chat templates, sampling, streaming APIs, and OpenAI-compatible APIs.</li><li>Hands-on understanding of tool calling, function-calling formats, structured output, and the practical failure modes associated with these capabilities.</li><li>Experience with serving frameworks or open-source technologies such as vLLM, SGLang, TensorRT-LLM, llguidance, XGrammar, or Hugging Face tokenizers is highly valuable.</li><li>Familiarity with multimodal serving, including vision-language models, image preprocessing, vision encoders, and multimodal token handling.</li><li>Experience supporting reasoning models and associated output formats, such as OpenAI Harmony, is advantageous.</li><li>Experience working with GPU, FPGA, ASIC, or other AI accelerators, particularly for memory-bandwidth-bound inference, is valuable.</li><li>Experience developing customer-facing API products where engineering teams are responsible for compatibility, escalations, and release readiness.</li><li>Ability to make sound technical and organizational trade-offs and communicate decisions clearly to engineers, cross-functional partners, and executive stakeholders.</li><li>Hands-on leadership style with enough technical depth to engage meaningfully in architecture and code discussions without becoming a delivery bottleneck.</li><li>Strong written and verbal communication, prioritization, collaboration, and organizational skills.</li><li>Ability to operate effectively in a fast-moving environment where models, modalities, serving technologies, and customer requirements evolve rapidly.</li><li>Candidates must currently be authorized to work in the United States. New visa sponsorship is not available, though eligible H-1B transfers may be supported.</li><h2>Benefits</h2><li>Base salary range of<strong>$225,000–$350,000</strong>, with final compensation determined by experience, qualifications, internal equity, and expected impact.</li><li>Competitive equity as part of the total compensation package.</li><li>Fully company-paid medical, dental, and vision insurance for employees and dependents.</li><li>Company-paid life and disability insurance, with voluntary supplemental coverage options.</li><li>Supplemental hospital, critical illness, and accident insurance options.</li><li>Unlimited paid time off, with an emphasis on taking meaningful time to rest and recharge.</li><li>13 paid company holidays.</li><li>Remote-first work environment.</li><li>Company-provided computer and home office setup.</li><li>401(k) retirement plan with company matching, eligible from day one.</li><li>Opportunity to shape both advanced AI inference technology and the engineering organization supporting it.</li><li>Opportunity to work on rapidly evolving accelerator, model-serving, and production AI technologies.</li>