Google Adds Computer Use Capabilities to Gemini’s Mac App

Associated Writers

Google Unveils Gemini’s Mac Client with Enhanced App Interaction

Google LLC has announced the release of an updated version of its Gemini Mac client, now equipped with the ability to interact with various applications on users’ computers. This new feature is currently available in the U.S. and is accessible through a Google AI subscription, which expands consumer Google accounts’ offerings with a broader array of artificial intelligence tools. The subscription also includes additional storage space along with other functionalities.

Users can now instruct Gemini to launch local applications by incorporating the command “@Computer use” within their prompts. For instance, a designer can upload a vehicle sketch and request the AI assistant to convert it into a three-dimensional animation. Gemini identifies the necessary local application, formulates a project plan, and executes the task step by step.

The assistant is capable of supporting workflows that involve numerous actions. Furthermore, users can utilize the output from a completed Gemini workflow to initiate subsequent projects. For example, a user may have Gemini generate a visual asset in a design tool and then transform it into a mini-application using Visual Studio Code.

In addition to its functionality with local applications, Gemini has the capacity to interact with web services. This enables the AI to import data from web pages into programs operating on the user’s machine, as well as facilitate data transfer between applications and cloud services.

Gemini operates within a background window, allowing users to continue their tasks without interruption during the assistant’s processing. Users can monitor Gemini’s workflows via a preview pane located in the Mac client’s chat window. Essential controls have also been integrated, enabling users to provide feedback to Gemini while tasks are in progress.

“Privacy and safety are built in: you choose which apps Gemini can access, watch every step live, pause at any time, and approve key actions before they happen,” stated Michael Friedman, Google’s group product manager for the Gemini app, in a recent blog post.

In comparison, the Mac version of Muse, an AI agent recently introduced by Meta Platforms Inc., is also capable of interacting with local applications and data. Moreover, Muse features a cloud-based virtual machine that allows it to execute tasks even when the user’s computer is offline. A competing service, Dots from OpenAI Group PBC, offers similar capabilities.

As Google looks to enhance its competitive edge, it may pursue the integration of cloud-based workflow hosting features in future updates. Additionally, Gemini could be expanded to support other operating systems and introduce more functionalities tailored for business users. OpenAI’s Dots currently incorporates features aimed at automating various business workflows, such as procurement.

[gspeech type=full]

Share This Article
Leave a comment