Endoscopic Navigation

Sep 2025 – Present · Lab member, UC San Diego Advanced Robotics and Controls Lab

Exploration code · Simulator

I work on this at UC San Diego's Advanced Robotics and Controls Lab, with a medical robotics company and physicians at UC San Diego Health. The goal is a system that steers a robotic endoscope through the kidney on its own, finds kidney stones and helps clear them. The procedure is called flexible ureteroscopy.

The problem

In ureteroscopy, a surgeon passes a thin, flexible scope up the urinary tract into the kidney and steers it through the kidney's branches (the calyces) looking for stones. Even experienced surgeons sometimes miss a branch, and a missed branch can mean a missed stone and a second procedure. A robot that checks every branch in order would fix that.

What we know before the procedure

Patients usually get a CT scan first. From it we can build a 3D model of their kidney and pull out:

The rest of the approach leans on that prior.

Why SLAM didn't work

My first try was to have the robot follow the CT centerline while a visual SLAM system (DROID-SLAM, then VGGT-SLAM) tracked where the scope really was.

3D kidney reconstruction from a CT scan

It didn't hold up. The inside of the kidney is wet, smooth tissue with almost no texture, and the scope's light reflects off it. Feature matching kept failing, and so did loop closure, even with dense reconstruction.

Navigating on the graph

We don't need the scope's exact position. The kidney is a tree, and the CT already tells us its shape and how many branches each junction has. So the question becomes "which node of the graph am I at?" and not "what is my exact 6-DoF pose?"

That changes what the robot has to see. If the CT says a junction has three branches and the camera shows two, the robot knows to rotate until it finds the third. Depth is enough for that: it shows how many openings are in view and keeps the scope off the walls (touching them isn't a disaster, but it's better avoided).

Depth estimation and junction detection from the endoscope feed

The topological graph the system builds while it explores

What I've built

Exploration on real footage. A per-frame pipeline runs on video from a kidney phantom (endonav-exploration). It estimates depth from a single camera with Depth Anything v2, uses that depth to avoid walls and count branch openings, and uses DINOv2 features (the AnyLoc method) to notice when it's back somewhere it has already been. A state machine turns all of that into a graph of where it has explored and picks where to look next. There's no SLAM in it.

A simulator with many kidneys. We can't just have a robot testing on a real patient, a 3D print isn't visually believable enough, and a phantom is only one kidney. So I wrote a procedural generator (kidney-meshgen). Each random seed produces a different kidney, shaped after a published classification of kidney anatomy, and exports what a real-time Unity simulator needs: the visible mesh, a collision mesh, a distance field, the centerline graph, waypoints and stones.

Endoscope views rendered across twelve generated kidneys

The matching depth maps, the signal the navigation uses

It replaced an earlier all-Python simulator (endonav-sim) that modeled the scope's mechanics in detail (encoder steps, cable backlash, drift) and the find, break up and remove loop for stones. The move to Unity was for more flexibility in rendering, and to experiment with SofaUnity, which runs the SOFA physics engine for soft, deformable objects inside Unity.

An autonomous agent, started. endonav-agent is meant to explore every calyx depth-first, deal with the stones and check that each calyx is clear, using only what a real scope sees: the camera and noisy joint readings. The simulator's ground truth is only there to score it. It's still a scaffold.

Where it stands

I paused the project in June 2026 and came back to it in October 2026. The exploration pipeline and the simulator work. The agent doesn't yet.

Code