ComingUp ComingUp
Handsy AI

Handsy AI

An AI that uses your iPhone. With its own hands.

Posted on behalf of the maker

Is this your project? Ask us to take it down

Gallery

Handsy AI

About

Handsy AI lets an AI use a real iPhone the way you do: it looks at the screen and taps.

A $7 Seeed XIAO ESP32S3 board plugs into the phone's USB port. The phone mirrors its screen to the board over AirPlay, so the AI can see it. The board then acts as a standard USB mouse and keyboard. With AssistiveTouch on, a click is a tap and a drag is a swipe. Nothing is installed on the phone's system: no jailbreak, no developer mode, no new phone.

There are two ways to use it:

- iPhone app (ESP32 Agent, free on TestFlight): type a goal like "set an alarm for 7 AM every weekday" and watch it work through your apps. It checks that each step actually changed the screen, keeps a step-by-step replay of every run, and can repeat a recorded flow. Try the free AI line or plug in your own.

- Handsy CLI: let the AI agent on your computer drive your iPhone from the terminal (MCP supported). No app and no developer account needed.

Setup is three short steps. Turn on AssistiveTouch, flash the board once from Chrome or Edge at handsy.tech (no IDE, no soldering), then pair it with the app or the CLI. It also works on iPad.

The firmware and code are open source on GitHub (ESP32 iPhone Agent).

Honest limits: each step takes a few seconds, it runs in portrait mode, and iOS doesn't mirror passcode or Apple Pay screens.

Maker: Henry Yuan, X @iamhenryyuan

GitHub: https://github.com/onehumanbeing/esp32-iphone-agent

TestFlight: https://testflight.apple.com/join/1xQHgJcA

Video: https://www.youtube.com/watch?v=fC4BltPzPIU

Pricing: Free / Open Source

Comments (4)

Sierra Kihn Sierra Kihn 1 week ago

cool concept, but how does it handle different apps? curious about the accuracy of those taps.

Ryley Weissnat Ryley Weissnat 1 week ago

$7 board? what’s the latency like? also, how’s the setup? looks like a design nightmare.

Bartholome Kerluke Bartholome Kerluke 1 week ago

robot hands tapping glass is a terrible ux pattern and a liability nightmare waiting to happen

Mazie Murphy Mazie Murphy 4 days ago

looks cool but how's the latency on that? also, is it really just an esp32 board doing all the work?