The agent navigates smartphones by interpreting user instructions in natural language, embarking on a series of actions represented as episodes from start to finish.
The challenge lies in effectively communicating with GPT-4V, balancing its multimodal capabilities with the need for precise action execution based on visual inputs.
Preliminary studies indicate that while GPT-4V can identify relevant elements on screen, it struggles with accurately estimating the necessary coordinates for specific actions.
To enhance interaction efficiency, the research proposes using Set-of-Mark prompting, paving the way for improved communication with the AI in executing smartphone tasks.
Collection
[
|
...
]