University of Pittsburgh deploys Meta's DINOv3 and Segment Anything on a $41.5M ARPA-H wheelchair-robotics platform

ARPA-H, the federal government's health-focused high-risk-R&D agency, is funding a robotic mobility platform for wheelchair users at the University of Pittsburgh — and the perception system underneath it runs entirely on Meta's open-source computer vision models.

What's new

The University of Pittsburgh's Human Engineering Research Laboratories (HERL), working with partner ATDev, is building RAMMP (Robotic Assistive Mobility and Manipulation Platform) under up to $41.5 million in ARPA-H funding, targeting the estimated 5.5 million wheelchair users in the United States. The perception stack combines three Meta models, all running on-device rather than via a cloud connection:

• DINOv3 — a self-supervised vision transformer Meta calls a compact 'visual brain' for edge hardware.
• Segment Anything Model (SAM) — outlines and identifies objects in images/video with minimal prompting, and doubles as an auto-labeling tool for RAMMP's training data.
• RF-DETR — a lightweight detector fine-tuned on DINOv2 embeddings, giving the platform 360-degree awareness.

Engineers optimized all three for the constraints of a battery-powered mobile robot — heat, power draw, and limited onboard compute. Current prototype capability includes recognizing automatic door buttons, cups, and curbs to assist with navigation and retrieval tasks.

Sivashankar Sivakanthan, RAMMP's chief of staff, said the on-device approach is the point: 'Running models like DINOv3 and SAM on-device is what enables real-time perception that users can trust — without relying on connectivity.' ARPA-H program manager Mansoor Khan called the vision models 'a critical technology that allows the robotics platform to see,' and ARPA-H director Alicia Jackson tied the work to its intended beneficiaries: 'Millions of Americans with limited mobility deserve technology that meets them where they are.'

Context

DINO and SAM are among Meta's most reused open-weight releases — general-purpose segmentation and representation-learning models that were never built for any single downstream task. RAMMP is one of a growing list of outside teams, spanning government, university, and industry, adopting them as off-the-shelf perception backbones rather than training custom vision systems from scratch. Pairing a DARPA-style biomedical funding agency with commodified open models is itself a marker of how far the cost of building capable perception systems has dropped since these models shipped.

Why it matters

A wheelchair-mounted robot that has to reliably spot a door button or a curb edge in unpredictable real-world lighting is a hard computer vision problem, and building a bespoke stack to solve it has historically been slow and expensive. Running proven open models on-device instead sidesteps both that cost and the latency/privacy tradeoffs of streaming a disabled user's camera feed to the cloud. It's also a concrete instance of the argument open-model advocates have made all year: open weights show up in places — federally funded assistive technology, in this case — that a closed, API-gated model would reach far more slowly, if at all.