Anthropic's Latest Breakthrough: A New Era in AI Computer Use

Imagine an AI that can use your computer just like you do—clicking buttons, navigating software, and executing commands based on what it "sees" on your screen. It's happening now, thanks to Anthropic's latest update to their Claude AI model. With this leap forward comes a mix of excitement, skepticism, and plenty of questions from the tech community.

The Game-Changer: Coordinate Support

The real breakthrough with Claude 3.5 Sonnet? It can now interpret and act on coordinates within screenshots. Coty Rosenblath, CTO at Katalon, explains: “The big deal to me is that they have gotten x, y coordinates working in multimodal interactions. This was not the case in any systems that I had tested until now.”

For those who’ve been following the evolution of AI, this is a major step. Earlier models struggled to accurately direct clicks on a screen, but now Claude can give precise instructions—like a human sitting at the keyboard.

Beyond Basic Comparisons

Some have quickly compared this new capability to traditional tools like Selenium or early RPA solutions. But as Alex Martins from Katalon points out, those comparisons might be a bit too simplistic. While it’s true that traditional tools could easily handle automation of a predefined set of activities, AI’s ability to understand and act on visual data based only on a goal is a whole new ballgame.

This ability to analyze and act on visual data could make a huge difference in automated testing, especially when traditional methods struggle with dynamic UIs.

Navigating the Future of AI and Software Interaction

Claude’s new feature is on par with what we’ve seen from other attempts, but with a fresh twist. Instead of just relying on structured data, it can interpret screenshots—essentially treating any visible interface as a control panel. This is a major step forward.

Dharmesh Shah, CTO at HubSpot, posted on LI yesterday that: “AI will now have the ability to use computer software, just like humans can. This dramatically increases the potential use cases for AI Agents because no longer is it necessary for an API to exist for the specific functionality you need to access.”

This opens up a world of possibilities—from automating desktop workflows to solving niche problems where traditional automation can’t keep up.

It's a Computer, but Not Really

Here’s the catch: while Anthropic’s AI can act like it’s working directly on your computer, it doesn’t actually provide a virtual or cloud environment to run everything for you. As Coty puts it, “Unlike OpenAI’s Code Interpreter mode, Anthropic is not providing hosted virtual machine computers for the model to interact with. You call the Claude models as usual, sending it both text and screenshots of the current state of the computer you have tasked it with controlling. It sends back commands about what you should do next. It is up to you to determine how to implement those commands. I’m looking forward to wiring it up to our own Katalon Studio automation.”

What’s Next for AI in Automation?

As Anthropic continues to refine these capabilities, the potential applications are vast—from speeding up software testing to new forms of digital interaction. But it’s early days, and it’ll be interesting to see how developers put these tools to work, and what new challenges emerge.   

For now, Anthropic's latest breakthrough is an exciting step forward in the AI world, one that could change how we think about human-computer interaction and automation. As Dharmesh Shah summed it up: “This is a development that we knew was coming, we just didn't know when and from who.”   

Alex is really excited with this breakthrough as it opens up a whole new set of use cases for software testing that could increase the team productivity exponentially and finally remove the stigma that "Testing is the bottleneck in the SDLC".

How would you use an AI agent that can navigate your software like a human?

Share your thoughts with us on LinkedIn

About the author

Derek Weeks, a veteran marketing leader in software development, brings over 30 years of experience to his role as Chief Marketing Officer at Katalon. Known for his expertise in DevOps and open-source software, he’s driven significant growth at the Linux Foundation and Sonatype, where he pioneered software supply chain security. Co-founder of All Day DevOps, Weeks is celebrated for building communities and advancing customer-centric innovation, making him a key asset to Katalon’s mission in AI-powered software testing.

→ Check out his LinkedIn here.