Set-of-Mark prompting is essential for bridging the gap between text outputs from GPT-4V and executable localized actions, enhancing smartphone navigation by generative AI.
The primary challenge is developing a strategy for the agent to determine subsequent actions, considering historical context and the current state of the screen.
Computational efficiency remains a concern; feeding all historical screens increases computational load and can lead to performance reduction due to information overload.
Humans effectively use a short memory to track key information after actions, while AI struggles with vast amounts of potentially irrelevant historical data.
Collection
[
|
...
]