Microsoft’s Copilot Studio recently introduced a 'computer use' feature that allows AI agents to interact with websites and desktop applications similarly to human users. This feature enables the automation of tasks like data entry and market research in environments without direct API connections. According to Charles Lamanna, agents can execute functions by performing clicks, typing, and selecting menu options. The system adapts to changes in websites or app interfaces, maintaining functionality. Additionally, this contrasts with the existing ‘Actions’ feature which is limited to select partners, making Copilot Studio more versatile for various businesses.
"Computer use enables agents to interact with websites and desktop apps by clicking buttons, selecting menus, and typing into fields on the screen."
"This allows agents to handle tasks even when there is no API available to connect to the system directly. If a person can use the app, the agent can too."
"Copilot Studio will be able to create AI agents that can automate data entry, perform market research, or even process invoices."
"The Actions experience is limited to a number of partners though, and it sounds like Copilot Studio will be free to work with more websites and apps."
Collection
[
|
...
]