# Member of Technical Staff - Research Engineer at Black Forest Labs

- Company: Black Forest Labs
- What the company does: Backed by a16z.
- Type: Startups
- Level: Senior
- Location: San Francisco (United States)
- Work setup: On-site
- Pay: $180K to $290K base salary per year (USD)
- Posted: 2026-06-29
- Apply by: 2026-10-08
- Apply: https://jobs.ashbyhq.com/black-forest-labs/362e79d1-1b62-4a28-ba4d-46484cbfcf3c
- Page: https://www.1752.vc/careers/jobs/black-forest-labs-member-of-technical-staff-research-engineer/

## About the role

Improve the performance, reliability, and numerical stability of production training runs for large multimodal generative models Profile full training steps across model code, attention, kernels, data loading, encoders, communication, optimizer steps, checkpointing, and memory pressure

## What they're looking for

- Experience working deeply on large-scale training systems, ideally as part of a training group working closely with researchers
- Strong PyTorch fluency, including comfort reading and modifying low-level training code rather than only using high-level APIs
- Experience with distributed training concepts such as FSDP, tensor/model/context/sequence parallelism, activation checkpointing, NCCL, and overlapping compute and communication
- Hands-on experience improving training throughput, memory footprint, or stability in real training runs
- Experience profiling GPU workloads with tools like Nsight Systems, Nsight Compute, torch profiler, trace viewers, or custom telemetry
- Practical GPU performance judgment: you may use modern coding agents and tools as much as you want, but you need the understanding to verify correctness, numerical behavior, and performance, and to own the result

Tags: Research