Driving an iPhone from a Mac with a fake Bluetooth keyboard

When we started Opposable we did Android first, because Android lets an app read the screen and tap on it through the accessibility service, and there's nothing like that on an iPhone. iOS keeps every app in its own box, and no app is allowed to look at or touch another one. I think that's mostly a good thing, but it meant that for months the honest answer to "does it work on iPhone?" was no, and that's an awkward answer when so many of the people asking are holding one.

The usual ways around this all had something wrong with them. WebDriverAgent, the thing app developers use for automated tests, needs Xcode and a developer account, and on a free account you have to reinstall it every week. Apple's iPhone Mirroring on the Mac would be perfect, except that it's not available in the EU, and I live in Estonia. Jailbreaking was never going to be something we'd ask people to do.

What an iPhone will accept

The thing that unlocked it was noticing what an iPhone will accept without any special permission at all, which is a keyboard and a mouse. You can pair a Bluetooth keyboard with an iPhone and type into any app, and you can use keyboard shortcuts to go home or open Spotlight. If you turn on AssistiveTouch, an accessibility feature meant for people who find the touchscreen hard to use, you can also pair a mouse, and a click becomes a tap wherever the pointer is. So if a computer can pretend to be a keyboard and a mouse, it can use an iPhone the same way a person with a keyboard and a mouse would, and the phone has no reason to treat it differently.

A Mac can do that pretending. Its Bluetooth chip can advertise itself as a keyboard and pointer, and the iPhone pairs with it like it would with any other accessory. Seeing the screen turned out to be the easy half. When you plug an iPhone into a Mac with a cable, the Mac can open it as a video source, which is what QuickTime uses when you record your phone's screen. So the agent sees the screen through the cable and touches it through Bluetooth, and nothing is installed on the phone.

I should say that we didn't figure out the hard part ourselves. An open-source project called iphone-use had already worked out the trick that makes this precise, and our Mac app builds on it. The problem is that a normal mouse only reports movement, and iOS speeds the pointer up when you move it quickly, so you can never be sure where it is. You'd have to keep looking at the screen to find the pointer, which is what some earlier projects did, with a camera. iphone-use describes a second kind of pointer to the phone, one that reports a position on a grid from 0 to 32,767 in each direction, and iOS maps that straight onto the screen. The phone ignores that pointer's buttons, so the click goes through an ordinary mouse at the same time. It sounds like a hack, and it is, but in our tests every tap landed exactly where it was aimed, with no calibration.

The small things that went wrong

Most of the work after that was the same kind of work as on Android, which is finding the places where a phone does something a person handles without thinking and an agent doesn't. The first one was typing. If you send keystrokes as fast as a computer can, the iPhone's keyboard occasionally drops one, and it seemed to prefer dropping spaces, so the agent would search for "weatherin Tallinn". We now send text in short bursts with a pause of a few milliseconds between them, which is still faster than anyone types but slow enough that nothing gets lost.

The second was that Spotlight remembers what you last searched for. The agent would open Spotlight, type the name of an app, and end up with the new text stuck onto the end of the old search. Now it selects everything and deletes it before typing, which is what you'd do if you noticed. The third was more embarrassing. To make a new note, the agent pressed the keyboard shortcut for a new note, which in the Notes app on an iPhone doesn't always do what it does on a Mac, and it typed its text into the bottom of an old note of mine about a parcel. It now asks Spotlight for "New Note" instead, which works every time.

None of these are deep problems, but each one is the kind of thing that makes an agent look stupid, and I suspect a lot of building something like this is just a long list of them.

What it can't do

Because the agent is only a keyboard and a mouse, it can only do what you could do with a keyboard and a mouse. It can't look at the phone, so it can't pass Face ID, and it can't press the side button, so it can't approve an Apple Pay payment. For a lot of people that's reassuring. The agent can get you all the way to the payment screen, but the last step is your face and your finger.

It also needs the cable for now, because the screen comes through it. We're working on a version that watches the screen over AirPlay instead, so the phone can sit anywhere on the same Wi-Fi, but that isn't ready yet. If you don't have a Mac, the same idea works with a small USB stick that does the keyboard and mouse part, which you can flash from your browser, though that version is earlier still.

Trying it

The Mac app is in beta at getopposable.com/mac. You plug in your iPhone, pair it once in the Bluetooth settings, turn on AssistiveTouch, and then Claude Code, Codex, or anything else that speaks MCP can use your phone with the same tools it uses on Android. It's still rough in places, and if it gets stuck in an app you use, I'd like to hear which one.