Can AI Computer Control Replace Human Operators? Inside Claude’s Groundbreaking Capabilities

Craig: so, apparently, this week, anthropic released a bunch of things, including upgrades to quad, the haiku, and opus not opus, but the, the other one. sonnet. sorry. and, wow. i just gotta get to that second coffee. and they also, released, ai model called computer use.

Jon: yes. i haven't messed around with it yet, but i saw the demo. and this is cool, but super freaking scary. so for those of you that don't know and by the way, anthropic, what the hell is with your naming conventions? this is a new functionality. why are you calling it three point five sonnet? i don't get it. so their new upgraded three point five sonnet is able to control computers. so, this is interesting because it's able to, so it's designed to perform tasks on a computer by interacting with the interface. what it can do is it can view the screen. it can move the cursor. it can click buttons and type text. and what the demo was showing was that it was getting claude to open up a browser and go to a website and do something. i can't remember what it was. but, yeah, apparently, the capability, it's facilitated through an api that translates user prompts into computer commands, which is scary, especially if you create malware. right? if you think about it from a malware perspective.

Craig: yeah. it's scary completely because as soon as now yeah. i'm so what's the attack surface of the ai compared to a computer without ai?

Jon: yeah. and what about that, microsoft copilot integrated into windows? imagine how easy it could potentially be for someone to take over your computer when it's fully integrated into your laptop or into your operating system. yeah. that scares me.

Craig: especially when they could get hacked so easily and access to recall. the data that is there that they're taking every three seconds of your machine and whatnot.

Jon: yeah.

Craig: it's crazy.

Jon: this, isn't this and just to go back to something we covered, isn't this what the rabbit r one was supposed to be doing, essentially? yes. yes. yeah.

Craig: yeah. so, i mean, it's it's super interesting. i like, i'm gonna share this screen instead because i've been back and forth with reuben cohen a little bit about can ai actually reason and think. and i'm on the fence that it can't, and he is on the offense that it absolutely can. and we will get into that later, but that's a different discussion. but, specifically, he's used comp the computer use capability, and it's spectacular. you can get it to do anything, like post on linkedin, log in to websites, play games, design things in photoshop, you know, watch my ring doorbell, whatever.

Jon: yeah.

Craig: so, interesting concepts, ruins usually right on top of these things as soon as they come out. unfortunately, with my schedule, i haven't had a chance, but i'm excited to try it. i'm really excited to try this.

Jon: there's something i wanted to build myself, which is not too difficult, but someone built it. well, i don't wanna build it myself, but something similar, but someone built and it's only using, computer vision. but, so it is a camera that's outside that is attached to a, paintball gun. so if it doesn't recognize your face, you have, i think, ten seconds to walk away before it starts shooting paintballs at you.

Craig: you had ten seconds to comply.

Jon: yes. yep. which i like. so, what i read was that there's companies like canva and replit that are already exploring its potential to streamline operations, such as filling out forms, browsing the web, and things like that.

Craig: i completely forgot canva dropped their new ai yesterday as

Jon: well. they?

Craig: yeah. and, i was playing around with that. it's okay. could use some work, but it's okay.

Jon: can it do what gamma dot app can do or not yet?

Craig: well hang on. what's that? i haven't asked to do that. right now, they've just launched their thing, their auto, their image generator. so if you are a canvas subscriber or whatnot, you can go and now use its tool to text ai images that are pretty good. not great.

Jon: you know what, what jenny i or llm it uses?

Craig: they bought, they bought a company that does their own, back. now they finally integrate it. i can't remember which model it is. but yeah.

Jon: interesting. interesting.

Craig: we can do a deep dive next time. i don't have it up here, but i can load up canva and show you some of the images that produce on which, prompts and whatnot. but yeah. you know, that's not what we're talking about this morning. i'm just saying that we missed that ai news. so

Jon: ai news. yeah.

Craig: yeah. it's all over. so you can go and download this thing on github. but while computer control presents exciting opportunities, it's essential to compare the performance of agents like cloud ai with human operators. so, one, will this be able to fuel you know, captcha is now going, not necessarily those click on those things, but it detects stuff in the background to determine, you know, are you a robot or not? so the question becomes, will it bypass those detections? i'm curious.

Jon: yeah. or can it mimic those detections? can it yeah.

Craig: and, you know, openai gpt four, you can put gpts that do some of these things, but not what we're doing here. and swebench is, a key for evaluating software development skills to compare the two. but that's talking about getting it to code and whatnot. so the claudia still has limitations, particularly in handle tasks that require long term planning or adaptive problem solving. so you know, you can't go too far ahead when you're telling it to do stuff. like, what i've seen from the videos is you can say, k. go open firefox. go to this web page and find this playlist or find this thing. and it does that okay. but i think if you get too complicated in there

Jon: okay. but i, yeah. so the other the other example i saw it do was it created a website and then fixed the bugs on the website itself. so you pick up a browser, coded it, launched it, and then went through and fixed the code.

Craig: i could see how that works pretty good. i didn't see that one, but i use that use case all the time where i'm working on something, i get an error, i cut and paste the error, or i take a screenshot of the error, throw it into what i'm working with. like, i usually have chat gtp open on my side as my adviser for stuff when i don't know what i'm doing. yeah. yeah. and say, okay. this? what's that? i was, and it comes back and say, oh, it looks like you're having a problem with this or that, and it helps me. but it'd be nice for it to automatically be able to say, okay. here's the code. oh, i see. i can see the problems. let me go fix it automatically rather than me having to be that person that is, hey. it looks like this now. hey. it looks like this. for example, when i was using, latex to, get open, cache atp will i have i had written down workflows. you can say, okay. now write this in latex code, and it'll write that in latex, which is, the diagramming. okay. and i throw that in, i think, it was openleaf, which reads, latex and will output those graphics that i was kinda sending around, those, logic diagrams. and i just take that output and throw it back in and say, yeah. it looks like this. oh, and just let me fix it. because sometimes it's off the page or it's too crowded or whatever, two or three of those, and it fix it. but if it could do it automatically, that becomes way more powerful. and for sure. you know, we talk about user data and scraping and all that a lot. and tools like this now, like, sounds like you can scrape anything because

Jon: yeah.

Craig: now i can say, hey. go open up linkedin. go search for people with this blah blah blah blah blah. then, you know, for the ones i'm not connected to, click on connect, send a message, whatever, or get their details and put it in a spreadsheet so you know? there's a thousand things you can do at that point. right?

Jon: yeah. no. i'm curious. now that you're talking about this, i wonder if it can code in, like, archaic languages like cobalt or assembly. well, assembly is not archaic, but, like

Craig: maybe. i'm curious.

Jon: i know that because i don't understand. yeah.

Craig: that's not a bad idea. i know that with, chatgtp, it can do javascript. it can do a c plus plus. like, there's six or so languages it can do, and you can switch between them. you can say, okay. now put this in python. now put this in you know? and, under canvas, you can tell it to fix its own code, document its own code, and it's pretty good. i'm, you know, i'm finding the limitations of canvas. it it's not great.

Jon: yeah. it says it says

Craig: it does stuff, but it doesn't do stuff. like, it'll like, we'd say, oh, what about this? what about that? oh, i've updated your document with all this information. it's like, no. you haven't. you just said you did, but you haven't done anything.

Jon: oh, that's funny.

Craig: because i'm looking right at it. i said, yes. i did. it's like, oh, okay. so i cut and paste that into a new canvas, like, my document, copy paste it, and then go back and tell it what it was supposed to do, and then it goes ahead and does it. it, you know, it's sometimes it cuts off half the thing because it's losing contacts or something. it's just you know, if you're working with it on stuff that's important to you from a document point of view, i can't say the same for code because i don't work a lot in long code. it's gonna frustrate you, and you're gonna lose stuff.

Jon: yeah.

Craig: period. you got a nice document. you put it in there, or you get it to create it. make sure you copy it before you get it to edit because all of a sudden you say, add this thing, and then half the document's gone. and and it'll tell you, no. no. it's all there.

Jon: yeah.

Craig: but you know, working with this i'm curious about working with this, but with chatgtp and we're talking about you know, i was using google notebook l m as kind of a guru for a bit. the problem with notebook l m is you can only give it very small prompts.

Jon: oh, really? okay. yeah. i have it open in a tab, but i never messed around with it yet.

Craig: so i was trying that saying, alright. you know, let's load up very specific documents, the manuals, all the pdfs, and two or three really detailed videos on how to use this specific technical product

Jon: yeah.

Craig: in google lm. and then i'm saying, okay. you're my guru on this subject because i'm implementing this new technology, and i wanna use these tools. so help me when i'm stuck. but the problem is you get stuck. you can't put the error in because it's too big. you know, you there's it there's a lot of things you can't do because it's too big. and so it's like, alright. you're very limited in functionality because it'd really be nice. so it's compact.

Jon: window is really small?

Craig: well, the window you're allowed to give it. maybe if i attach a different document and said use this document, but i don't know. but, like, the prompt window, you can give it. it can give you back some good stuff, but, like, literally, how much information i can tell it in the prompt window is really small, which, is a problem because i'm saying, here's the error. and if the error is more than, say, five lines, it won't take it.

Jon: yeah. oh, the other one we didn't cover, and we can cover it next time, is these, advanced language or whatever it's called, so that you can interact with it. so openai released it. gemini released it. so you can talk and

Craig: you're not talk the meta prompts, are you?

Jon: no.

Craig: okay.

Jon: no. no. why is it oh, yeah. we're covering that too, aren't we?

Craig: well, i don't think so. i looked into it, and i made some comments on the

Jon: oh, okay. i know. it's super no. there's, so the, advanced anyways, i'll think about it. we'll talk about it next week.

Craig: so this is groundbreaking. i suggest everybody go out and try it. this is exciting. i'm really interested in, hopefully, we'll you'll try it too. we'll give some feedback because yeah. if it works, it might not be super fast, but all of a sudden, now you can have prompts. you can have it do step one, step two, step three of stuff. like, i was about to go and buy keyboard maestro again so i could automate some features on my mac to do certain things, like click here, click here, open this app, do this, which other things don't do. but all of a sudden, maybe i don't need that.

Jon: yeah.

Craig: any last words on this one?

Jon: no. i'm excited to play around with this one. so this will be fun.