Microsoft reveals plans to allow AI agents via Copilot Studio to perform GUI operations, enabling them to click buttons, select menus, and complete forms. This will automate tasks where APIs don't exist, giving agents the ability to interact intuitively with applications like a human user. Designed for versatility, these agents can gather data from various sources and adapt to unforeseen challenges. Charles Lamanna highlights that the agents can improve efficiency, particularly in data handling and processing tasks important for businesses.
"Computer use enables agents to interact with websites and desktop apps by clicking buttons, selecting menus, and typing into fields on the screen," explained Charles Lamanna.
"This allows agents to handle tasks even when there is no API available to connect to the system directly. If a person can use the app, the agent can too."
"The new type of agents should be much more flexible. For instance, you could create an agent and prompt it to carry out a series of steps that involve browsing a previously unseen website, extracting some data, and passing that data to a desktop app."
"AI automation differs from programmed instructions in that the agent can adapt on the fly when it encounters obstacles or unexpected challenges."
Collection
[
|
...
]