Do you want an AI agent to do something on an iPhone? You can try to jailbreak the device and do some serious software hacking, or… you can grab a microcontroller off the shelf and make it do the hard work. (Henry Yuan) explains how to achieve this with an ESP32.
The first step is to get an AI agent to access the current state of the phone screen, which (Henry) achieves through AirPlay and UxPlay screen mirroring. A Gstreamer pipeline is then used to transmit the screen as a JPEG stream to your AI agent of choice. Next comes control, carried out by an ESP32-S3 acting as a USB-HID device, emulating both the USB mouse and keyboard. With iOS AssistiveTouch enabled, this gives the agent control of inputs on the device. From there, it’s up to the AI agent to determine what actions to take to achieve the desired behavior, which is obviously very specific to each use case.
This may seem a bit complicated, but there are advantages to this approach. It is not at all necessary to have an Apple developer account, or XCode, or a Macintosh. You can simply ask your agent to tinker over the phone without any complex overhead. We’ve also featured some other great iPhone hacks recently. Video after the break.
Gn tech
