Reval LabsBlogInvestors
Menu
AccountGet the app

Blog · · 4 min read

Computer use on the phone itself: a 2B model that operates Android apps

Erik

Opposable is an Android app that reads the screen through the accessibility service and taps and types in any other app. Claude, ChatGPT or Codex can drive it from a computer, and when none of them is connected, a 2-billion-parameter screen model runs the same loop on the phone with no server involved.

The on-device agent opening Clock and switching to the stopwatch, with no model outside the phone. Android emulator, real time, each step's timing on the right.

TL;DR

The model is GUI-Owl 1.5 2B in 4 bits on llama.cpp. On a Mac M2 CPU with 4 threads one step takes 7.3 s, down from 10.6 s, at the same accuracy on our 36-screen test set. Half of that step is the vision encoder, and on a Galaxy S25's NPU the encoder runs in 236 ms against 3.9 s on the M2.

One step of the on-device loop: encode, prefill, decode, then a tap on the phoneFilesencodescreenshot to tokens3.6 sprefilltokens plus task2.5 sdecodewrite the action1.1 sAction: Click the Files app icon. tapone step, Mac M2 CPU, 4 threads (animation runs at real speed)
One step of the loop, animated at its real speed on the Mac M2 CPU: 3.6 s to encode, 2.5 s to prefill, 1.1 s to decode, then the tap.

Model and runtime

GUI-Owl 1.5 comes from Alibaba's Mobile-Agent team. It is a Qwen3-VL fine-tune that looks at a screenshot and answers with one action. We run it in the app through llama.cpp and its multimodal library, quantised to 4 bits: 1.05 GB for the language model and 445 MB for the vision encoder. Both load once and stay in memory between steps.

The model only sees tasks the app can't do another way. About forty common ones, like setting an alarm, texting a contact or switching on Wi-Fi, run as fixed code with no model, and routines that a large agent worked out earlier replay from the text on the screen.

Step time on CPU

A step is three stages. The vision encoder turns the screenshot into image tokens, the language model reads them along with the task (prefill), and it writes about 45 tokens: a sentence saying what it intends to do, then a JSON tool call. With nothing optimised, on the M2 the encoder took 4.6 s, prefill 4.3 s and decode 1.7 s, 10.6 s per step, and the model got 27 of the 36 graded screens right.

Step time on the M2 CPU by stage, before and after the CPU optimisationsuntouched, 576 image tokens10.6 s27/36 correctcrop + prune + cache + early exit, 400 tokens7.3 s27/36 correctencodeprefilldecode
Step time on the M2 CPU by stage, before and after the CPU work. Our measurements on the 36-screen graded set.

Cropping the status bar, navigation bar and keyboard out of the screenshot cut the image before encoding. After the encoder, tokens for patches of flat background that don't border anything are dropped, and the rest keep their original grid positions, so the model sees the same layout with 25 to 40% fewer tokens. That raised accuracy to 28 of 36. The 900-token system prompt is prefilled once and its KV cache kept. At the 400 image tokens the app uses, a step takes 7.3 s and scores 27 of 36.

Image tokens for flat background are dropped before the language model reads themstatus bar: croppednav bar: croppedEach square is one image token.Grey squares are flat background with nothing next to them,so they are dropped after the encoder. Blue ones stay at theiroriginal grid position, so the layout the model sees is the same.25 to 40% fewer tokens in prefillaccuracy 28/36 (27/36 without pruning)This drawing: 139 of 200 kept. The real grid at 400 tokens is 13 by 30.
A simplified screen. Status and navigation bars are cropped before encoding, and flat-background tokens with no neighbouring content are dropped before prefill.

The model states its intent before coordinates, for example "Action: Click the Files app icon." When that sentence matches exactly one element in the accessibility tree, the app taps the element and stops generating. On the emulator those steps decoded in 0.8 to 1.3 s instead of 2 to 4 s. With all of this in place the encoder took 3.6 of the 7.3 seconds.

Grounded early exit: the model's sentence matches one accessibility element, so coordinates are never decodedmodel output, token by tokenAction: Click the Files app icon.{"action":"click","x":812,"y":388}accessibility tree, same screen[6]Button"Photos"[7]Button"Files"[8]Button"Settings"One element matches, so the app taps [7]directly and stops generating. Thecoordinates are never written.decode 0.8 to 1.3 s instead of 2 to 4 s
When the model's sentence matches exactly one element on screen, the app taps it and the coordinates are never generated.

Vision encoder on the NPU

We exported only GUI-Owl's vision tower to ONNX: a 24-layer ViT with its mergers, about 0.4 billion parameters, at a fixed input of 416 by 960 pixels. One screen at that size is 542 billion multiply-accumulates. Qualcomm AI Hub compiled it for the Galaxy S25 (Snapdragon 8 Elite) and ran it on a real S25 in Qualcomm's device lab.

Time per step of the on-device screen model, laptop CPU, and the vision part on a phone NPUone step, laptop CPU: 10.6 sseeing (vision encoder)4.6 sreading the prompt4.3 swriting the answer1.7 sthe seeing part alone, Galaxy S253.9 s on a laptop CPU0.24 s on the phone's NPU
One step on the M2 CPU by stage, and the vision encoder alone on the M2 and on the S25's NPU.

In fp16 the encoder ran in 236 ms with all 873 operations on the NPU, about 16 times faster than the M2's 3.9 s. 8-bit weights with 16-bit activations were faster at 213 ms but drifted from the reference output, cosine 0.978 to 0.985 against 0.992 to 0.996 for fp16, and the model's taps depend on small screen details, so the app uses fp16. Fully 8-bit and 4-bit versions scored 0.67 to 0.90. The first load takes 72 s while the NPU prepares the graph, once per install, and 0.39 s after that.

Vision encoder on the Galaxy S25 NPU: inference time and similarity to the reference output by precisionGalaxy S25 NPU, one screen (416×960)cosine vs reference output0.99 needed0.60.70.80.91.0fp16 (shipped)236 msw8a16213 msw8a8 / w4not keptsame encoder on the M2 CPU: 3,900 ms
Encoder precision on the Galaxy S25 NPU (Qualcomm AI Hub): inference time, and how close the output stays to the reference. fp16 is the only version above the 0.99 the app requires.

The NPU returns the same float array as llama.cpp's own encoder, so pruning, positions, the cache and the early exit work unchanged. At startup the app tries the NPU, then the GPU through LiteRT, then the CPU, and an encoder is used only if it passes a self-test on a bundled screenshot (cosine 0.97 or better against the CPU encoder) and is faster than the CPU. The 640-token input size failed to compile for the NPU, so steps that need it still encode on the CPU.

Encoder choice at startup: NPU, then GPU, then CPU, each behind a self-testNPUQualcomm, fp16failsGPULiteRT, OpenCLfailsCPUllama.cpp, always worksSelf-test on a bundled screenshot at first load: output within cosine 0.97 of the CPU encoder,and faster than the CPU. Otherwise the next one is tried. Settings shows which one won.
How a phone ends up with an encoder. Each candidate has to pass a self-test at first load, and the CPU path always works.

Accuracy on held-out tasks

On eight tasks the model had never been tuned on, run on the emulator twice each, giving it the whole task succeeded 4 and 3 times. Giving it one subgoal at a time and checking each claimed success against the accessibility tree before the next subgoal succeeded 6 times in both rounds.

Held-out tasks passed out of 8, two rounds, before and after subgoals with verificationwhole task at onceround 14/8round 23/8one subgoal at a time, each checkedround 16/8round 26/8
Eight held-out tasks on the Android emulator, two rounds each, before and after subgoals with on-screen verification.

Limitations

Only Snapdragon has an NPU build so far. Pixel phones need a separate compile for Tensor, and MediaTek and Exynos need their own, so those phones fall back to the GPU or CPU. The model is also much weaker than a cloud model on apps it hasn't seen, and the app hands those tasks to Claude, ChatGPT or Codex when one is connected.

Trying it

The Android app is free at getopposable.com. The screen model is a 1.5 GB download from its settings, and the same screen shows which encoder your phone picked and how long a screen takes to encode. I'd like to hear both from your phone, at hello@getopposable.com.

Get the app