Robotic bin picking with 3D computer vision: how it works end to end

Bin picking sounds simple. A robot looks into a box of parts thrown in at random and picks them out one by one. People do it without thinking. For a machine, it's one of the classic hard problems in industrial automation, because every pick starts from a different, messy scene.

I was responsible for the software of PickNet, the vision and control pipeline for an industrial bin-picking cell built by the AutoMates team. The robot cell, hardware and integration were a team effort; the pipeline from camera to grasp was my part. Here's how a system like that works, stage by stage, and where the difficulty really is.

What robotic bin picking has to solve

To pick one part, the system needs to answer four questions, every single time:

Where are the parts? Which pixels and which 3D points belong to which object.
How is each part oriented? Its full 6D pose: position plus rotation in three dimensions.
Which part should I pick, and where do I grab it? A grasp point the gripper can reach without hitting the bin or other parts.
How does the robot get there? A collision-free motion from where the arm is to the grasp, and back out.

Get any one of those wrong and the robot misses, drops the part or crashes into the bin wall.

The pipeline, stage by stage

Here's how PickNet does it.

1. A 3D camera looks into the bin

A Zivid 3D camera captures the bin. Instead of a flat photo, you get a point cloud: every pixel comes with a depth value. That depth is what makes 3D bin picking possible. A 2D image can tell you there's a part there; it can't reliably tell you how it's tilted.

2. Segmentation: one mask per part

The first model separates the scene into individual parts. PickNet uses FastSAM, a foundation segmentation model, running on a dedicated GPU worker. It returns a mask for every part in the frame in about one second.

Using a foundation model here has a big advantage: you don't need to train a segmentation network from scratch for every new part. It already knows how to separate objects in general.

3. 6D pose estimation against the CAD model

A mask tells you which points belong to a part. It doesn't tell you its pose. For that, each mask is cut out of the point cloud and matched against the part's CAD model.

This is classical 3D geometry, built with Open3D:

RANSAC finds a rough alignment between the CAD model and the observed points, even with lots of noise and missing data.
ICP (iterative closest point) refines that alignment until the model sits tightly on the observed surface.
VSD (visible surface discrepancy) scores the fit, so only confident poses go forward.

That last step matters more than it sounds. A wrong pose with high confidence is the worst outcome in bin picking. It's much better to skip a part than to send the robot to a place where the part isn't.

4. Choosing a grasp

With a confident pose, the system knows exactly where the part is and how it's oriented, so it can place a grasp point on it in a way the gripper can actually use. Alongside the geometry, I also worked on grasp-quality models (CNNs and Vision Transformers), trained partly on synthetic data to cover rare cases.

5. Handing off to the robot

A ROS 2 bridge converts the grasp into the robot's base frame and publishes it to a MoveIt 2 state machine. MoveIt plans the motion, with the gripper modelled for collision checking, so the arm doesn't hit the bin or neighbouring parts on the way in.

How the software is structured

The pipeline runs as five Docker services on a shared network, with Redis connecting them. Segmentation is served as a GPU microservice over Redis, and the rest of the pipeline is split across the other services.

Splitting it this way has practical benefits:

Each piece can be developed, restarted and replaced on its own. Swap the segmentation model without touching the robot code.
The GPU work is isolated. The heavy model lives in one place and the rest stays light.
It's measurable. Every run writes a timeline and a performance report. That's how the end-to-end time, about 3.3 seconds from camera frame to grasp pose, was profiled and tuned.

Instrumenting everything from day one is the habit I'd recommend most. You can't make a pipeline faster or more reliable if you don't know where the time and the failures go.

The data work behind it

The models are only as good as the data. Behind the pipeline sits the less visible part: labelling real captures, generating synthetic data, training, running inference and deploying the models.

Synthetic data deserves a special mention. Real bins rarely give you enough examples of the awkward cases: parts standing on edge, parts half-buried, unusual lighting. Generating those scenes synthetically lets you train on situations that would take weeks to collect by hand.

The hard parts of 3D bin picking

Some problems come up in almost every bin-picking project:

Shiny and reflective parts. Metal reflects light in ways that leave holes or noise in the point cloud. Camera settings, viewing angle and lighting all matter.
Occlusion. Parts lying on top of each other are only partly visible, which makes both segmentation and pose estimation harder.
Tangled or interlocking parts. Picking one can drag another along.
The last few parts. Parts lying flat against the bin walls or corners are hard to reach without a collision.
Knowing when to give up. The system needs a clear rule for "no confident pick available": shake the bin, re-capture or ask for help, rather than guess.

None of these are solved by a better model alone. They're solved by the whole system working together: camera, vision, scoring, motion planning and sensible fallbacks.

When bin picking is worth it

Robotic bin picking makes sense when parts arrive in bulk, the task is repetitive, and the alternative is a person feeding a machine all day, or expensive fixtures and feeders for every part type. It's harder to justify when parts are extremely varied, very small, or easier to present in an orderly way earlier in the process.

The best way to find out is a feasibility test with your real parts and a 3D camera, before committing to a full cell.

The same principles carry over to other AI systems: measure everything and prefer confident skips over wrong guesses. See how I design multi-agent systems and private AI on your own hardware.

Working together

I'm Driss Briksine, a software and ML engineer based in Morocco. I build computer vision, 3D perception and AI systems end to end, from data and training to production software. See the PickNet case study and my other work.

If you're exploring bin picking or another vision problem, send me a short brief and I'll tell you honestly what's feasible.