About the role
· Define and own the technical direction of the LLM serving deployment, i ncluding engine integration, continuous batching, KV cache management and reuse, and quantized execution · Partner with inference performance engineering teams, with accountability for time-to-first-token, inter-token latency, throughput per GPU, and P95/P99 tail latency Found on 1752vc Careers, the job board for startup and VC roles.
What they're looking for
- Production experience with LLM inference engines such as vLLM, SGLang, TensorRT-LLM, TGI, or NVIDIA NIM, including modification of engine internals
- Comprehensive understanding of inference internals, including continuous batching, paged attention, KV cache reuse and prefix caching, chunked prefill, quantization, and speculative decoding
- Tensor and pipeline parallelism, NCCL collective operations, and the GPU memory hierarchy and interconnect characteristics that govern them
- Expert level proficiency in Kubernetes platform architectures, including operators, custom resources, controllers, and scheduling
- Strong programming proficiency in Go and Python, with the ability to read, debug, and profile C++/CUDA using tools such as Nsight
- Experience with debugging/profiling multi-tier application workloads such as RAG, Agents, etc
More about this role
Principal Software Engineer, Inference
This role has been designed as 'Hybrid' with a requirement that you will work on average 2 days per week from an HPE office.
Hewlett Packard Enterprise is the global edge-to-cloud company advancing the way people live and work. We help companies connect, protect, analyze, and act on their data and applications wherever they live, from edge to cloud, so they can turn insights into outcomes at the speed required to thrive in today’s complex world. Our culture thrives on finding new and better ways to accelerate what’s next. We know varied backgrounds are valued and succeed here. We have the flexibility to manage our work and personal needs. We make bold moves, together, and are a force for good. If you are looking to stretch and grow your career our culture will embrace you. Open up opportunities with HPE.
HPE's Private Cloud AI organization is seeking a Principal Software Engineer to lead the model runtime within HPE AI Essentials, the inference platform used by enterprises to operate large language models on infrastructure they own, including air-gapped and sovereign environments. The principal engineering challenge in this domain is not...
Read the full posting on Hewlett Packard Enterprise's site ↗
Browse similar: AI jobs · AI startup jobs · Startup jobs