Body & pose tracking

Track a person with nothing but a webcam and turn their movement into visuals — a glowing skeleton, dots that follow the hands and feet, motion trails that smear behind the body. tdmcp builds these from plain language, the same way it builds everything else.

No Kinect, no depth camera, no Windows-only hardware: the tracking comes from MediaPipe, Google's free pose model, which runs on macOS and Windows from an ordinary webcam.

You can try it before installing anything

Every body-tracking tool defaults to a synthetic source — a self-contained animated figure — so the network builds and previews instantly with no camera and no plugin. Use it to dial in the look, then switch the source to your webcam.

1. Try it now (no camera, no plugin)

In your AI assistant:

"Build a pose skeleton and show me a preview."

You'll get a stick figure — 33 landmarks (head, shoulders, elbows, wrists, hips, knees, ankles) joined by glowing lines — moving on its own. Then iterate:

• "Make the lines magenta and thicker."
• "Now give me body-reactive glowing dots instead."
• "Add motion trails."

2. Install the MediaPipe plugin (for a real performer)

To track a real person you need the free, GPU-accelerated MediaPipe plugin (Mac + PC). tdmcp can fetch it for you — in a terminal:

npx --yes --package=@dpantani/tdmcp tdmcp install mediapipe-touchdesigner

tdmcp install <lib> stages manifest-listed community libraries safely; here it grabs the MIT-licensed torinmb/mediapipe-touchdesigner and extracts it under ~/.tdmcp/packages.

Then, with TouchDesigner open, ask the assistant:

"set up body tracking"

The setup_body_tracking tool loads the plugin into your project, finds its pose-landmarks CHOP, and wires up create_pose_tracking plus a live skeleton — all you do is pick your webcam on the new mediapipe_pose component and enable Pose.

Prefer to do it by hand? Open MediaPipe TouchDesigner.toe (or drag a .tox from the staged package under ~/.tdmcp/packages) into your project, enable Pose, then point a tool's mediapipe_chop_path at the plugin's pose-landmarks CHOP (33 samples, channels tx/ty/tz).

macOS camera permission

The first time TouchDesigner reads your webcam, macOS pops a permission dialog. Click Allow — until you do, TouchDesigner may look frozen. See Troubleshooting.

3. Point the visuals at your body

Tell the assistant to use the plugin as the source:

"Build a pose skeleton from the MediaPipe plugin at /project1/mediapipe1/select_pose and show a preview."

"Make body-reactive trails from my webcam pose."

The assistant sets the tool's source to mediapipe and points mediapipe_chop_path at the plugin's pose CHOP. Everything else — the skeleton, the dots, the trails — works exactly as it did in synthetic mode.

4. What you can build

Three tools cover the pipeline; the AI picks them for you, but it helps to know the vocabulary:

Tool	What it makes
create_pose_tracking	The foundation. A clean pose signal (33 landmarks) plus ready-to-bind scalar channels like r_wrist_y, hand_span, height. Smoothing and Mirror built in.
create_pose_skeleton	The classic stick-figure skeleton rendered to a TOP — glowing lines connecting the landmarks.
create_body_reactive	Glowing marks that follow the body, in three styles: points (crisp dots), glow (bloomed dots), trails (motion smears).

You can chain them: build create_pose_tracking once, then point both the skeleton and the reactive visual at its output so they share one tracked body.

5. Bind your body to anything

create_pose_tracking also exposes a keypoints CHOP of plain scalar channels, so you can drive any parameter with a body part:

• "Make the blur amount follow my right hand height."
• "Map how wide my arms are open to the feedback feedback amount."
• "When I crouch, dim the lights." (the height channel shrinks)

Bind a parameter to op('…/pose_tracking/keypoints')['r_wrist_y'] (or hand_span, hips_x, height, …) and it tracks your movement live.

6. Put a hologram in your hand

Hands are not only controller data. create_hand_hologram builds a complete visual system where an open palm reveals a translucent cube floating just above the hand. The visual defaults to a synthetic two-hand source, so you can preview the look before touching the webcam:

"Build a futuristic holographic cube floating above my open palm. Start with a synthetic preview, make it transparent cyan with a violet accent, and let a pinch from my other hand grow the cube."

Under the hood, create_hand_hologram creates a nested create_hand_gesture_bus. That bus publishes stable channels like float_x, float_y, palm_size, pinch_power, light_gain and audio_level. The hologram shader reads those channels instead of raw MediaPipe landmarks, so the same bus can later drive lasers, particles, projection cues or audio controls.

When you are ready for the real hand, switch the source:

"Rebuild the hand hologram with MediaPipe hands from my webcam, keep it higher above the palm, and send the futuristic audio to my audio device."

audio_mode defaults to none; use synth for an internal signal or device_out when you explicitly want an Audio Device Out CHOP.

7. Use hands as Ableton filter controls

Hands can also become a four-channel Ableton controller when you use TDAbleton. The create_hand_ableton_mapper tool builds the TouchDesigner side: MediaPipe hands in, a skeleton overlay with star joints, and a mapper_send CHOP for TDAbleton. AbletonMCP is not required for this recipe.

Default mapping:

Mapper slot	Gesture
map1	Left-hand thumb/index distance
map2	Right-hand thumb/index distance
map3	Left wrist roll
map4	Right wrist roll

On the TDAbleton TDA_Mapper, point the input CHOP to /project1/hand_ableton_mapper/mapper_send, set Reorder to map1 map2 map3 map4, keep Bypass1..4 off, and map the four slots manually to Auto Filter frequency, resonance, drive, or rack macros inside Ableton.

If TouchDesigner values move but Ableton does not, run diagnose_tdableton_mapper. It checks the real mapper path, input CHOP, Reorder, bypass states, Min/Max ranges, and missing map1..map4 channels. This catches the common stale-target problem where a mapper looks healthy but points at the wrong track or device.

Ready-made recipes

Two browsable templates ship in the recipe gallery — ask "list recipes" or see the Recipe gallery:

• Pose Skeleton (MediaPipe) — the stick-figure skeleton from a live webcam.
• MediaPipe Body Dots — glowing dots tracking every joint.

Both expect the MediaPipe plugin; point their posein Select CHOP at its pose landmarks CHOP.