caferemote
Robots & Pencils

Staff ML Engineer – AWS Trainium & SageMaker

Remote United States Full-time Lead Posted 3 days ago
Apply now
Work from
United States
Contract
Full-time
Experience
Lead
Posted
17 November 2026
Apply by
16 January 2027

We're looking for an engineer who can operate and train models on Amazon SageMaker running on AWS Trainium, AWS's custom silicon built specifically for large-scale model training. You'll be walking up the stack: understanding what a training request actually looks like at the Trainium hardware and compiler level, then carrying that understanding all the way up through PyTorch training code and into a production SageMaker pipeline.PyTorch is the backbone of this work.

About the role

This isn't a role where you call an API and wait. If you know the framework deeply and you're comfortable reasoning about how your code actually behaves on custom accelerator hardware rather than treating it as a black box, this role is built around that skill set specifically.

Responsibilities

  • Train and operate models on Amazon SageMaker with AWS Trainium as the underlying compute
  • Write and optimize PyTorch training code with a real understanding of how it compiles and executes on Trainium (NeuronCore architecture, compiler behavior, memory and throughput tradeoffs)
  • Diagnose training run issues that show up specifically because of the hardware, not just the model, distinguishing a data or code problem from a compiler or device-level one
  • Translate a request for "a Trainium job" into an actual working, cost-aware training pipeline, end to end
  • Tune distributed training runs for throughput and cost on SageMaker's training infrastructure
  • Work directly with client and internal engineering teams to scope and deliver real production training workloads, not experiments that stay in a notebook

Requirements

  • Strong, hands-on PyTorch experience, ideally including distributed or multi-device training
  • Production experience with Amazon SageMaker for training and/or inference
  • Comfort working close to the hardware layer: you understand device-specific compilation and can debug issues that are actually about the accelerator, not just the model
  • AWS Trainium or Inferentia (Neuron SDK) experience is a strong plus; if you don't have it yet but have deep PyTorch and a track record of picking up new hardware targets fast, we want to talk to you
  • Solid Python fundamentals and comfort operating in a client-facing, production engineering environment

About Robots & Pencils

Robots & Pencils is an AWS Partner building production AI systems for enterprise clients who need real engineering, not proofs of concept that never ship. We work forward-deployed, embedded directly with client teams, solving the problems that are too new or too specialized for a typical vendor relationship to handle.

Discussion

No comments yet. Ask a question or share a tip about this role.

Apply now

More jobs from Robots & Pencils

View company →

UX/UI Designer

Robots & Pencils
Remote Canada Full-time

Company Overview APPLICATIONS FROM OUTSIDE CANADA WILL NOT BE CONSIDERED FOR THIS ROLE Robots & Pencils is an applied AI engineering firm building the next…

3 days ago Design

Staff ML Engineer – AWS Trainium & SageMaker

Robots & Pencils
Remote Canada Full-time

Robots & Pencils is an AWS Partner building production AI systems for enterprise clients who need real engineering, not proofs of concept that never ship. We…

3 days ago Data

Principal Applied Scientist

Robots & Pencils
Remote United States Full-time $200K – $300K / year

We're looking for a Principal Applied Scientist to lead the design and build of an evaluation framework for an enterprise planner-based agent. You will work…

7 days ago Data

Senior Delivery Manager

Robots & Pencils
Remote Canada Full-time CAD 170K – CAD 220K / year

Robots & Pencils is an applied AI engineering firm building the next frontier of business architecture. We design and ship AI co-workers that integrate into…

8 days ago Software Development

Similar remote jobs

Browse all →

NEN SMIT Senior Solution Architect

Leidos
Remote United States Full-time $107K – $195K / year

Leidos is currently seeking a Senior Solution Architect to join our SMIT Growth Team. The Senior Solution Architect will be responsible for providing technical leadership and delivering…

2h ago DevOps and Sysadmin

Cloud Architecture Manager

Smart Tech Contracting
Remote United States Full-time $140K – $170K / year

Smart Tech Contracting delivers turnkey BAS & EPMS solutions for mission-critical colocation and hyperscale data centers operating at national program scale. As a Master Systems Integrator…

2h ago DevOps and Sysadmin

Cloud System Administrator (Shift B - Day)

NexusTek
Remote United States Full-time $28 – $31 / hour

Shift Schedule: This is a 12-hour day shift (5:45 AM to 6:15 PM Mountain Time), operating on a 12/7 rotational schedule . The rotation is as follows: Week 1: Work Thursday through Saturday…

4h ago DevOps and Sysadmin

Principal, Software Development

Common Securitization Solutions
Remote United States Full-time $181K – $208K / year

OVERVIEW The Company U.S. Financial Technology (U.S. FinTech) is seeking an experienced Principal, Software Development to join our team of talented professionals. This is a full-time…

5h ago DevOps and Sysadmin

Lead Devops Engineer

Way2B1
Remote United States Full-time

Our Company and the Role This is a hands-on Lead DevOps Engineer role responsible for designing, operating, and evolving a highly available, multi-tenant platform on AWS. You will work…

5h ago DevOps and Sysadmin

Senior Platform Engineer — Kubernetes, DevOps, & Cloud Infra (US)

Genios AI
Remote United States Full-time

We are on a mission to build the AI-native platform for finance —a multi-agent system that reimagines how investment teams make decisions and empowers them to move faster, think deeper,…

6h ago DevOps and Sysadmin

Get remote jobs the moment they are posted

Choose your categories and skills. We push matching jobs to your device. No email, ever.