Anthropic says its latest AI model is too powerful for public release and that it broke containment during testing
2mon 10d ago by lemmy.world/u/return2ozma in technology from www.businessinsider.com
The researcher had encouraged Mythos to find a way to send a message if it could escape.
Engineers at Anthropic with no formal security training have asked Mythos Preview to find remote code execution vulnerabilities overnight, and woken up the following morning to a complete, working exploit
I would love to see the exploit. There are vulnerabilities discovered everyday that amount to very little in terms of use in real world implementations.
Yes, recently we got a security "finding" from a security researcher.
His vulnerability required first for someone to remove or comment out calls to sanitize data and then said we had a vulnerability due to lack of sanitation....
Throughout my career, most security findings are like this, useless or even a bit deceitful. Some are really important, but most are garbage.
That's so idiotic. Either that guy was a total amateur who couldn't put together that "no shit, if you comment out the lines that do thing, it won't do thing" or he was completely malevolent and disingenuous and just trying to justify his position by coming up with some crap that the big bosses are probably too stupid to recognize the idiocy of.
Either way, not someone I would want to be doing business with...
He had the persosctive that once you hop between source code files that constitutes a security boundary. If you had intake.c and user data.c that got linked together, well data.c needed its own sanitation... Just in case...
I suspect he used a tool that checked files and noted the risky pattern and the tool didn't understand the relationship and be was so invested that he tortured it a bit to have any finding. I think he was hired by a client and in my experience a security consultant always has a finding, no matter how clean in practice the system was.
Another finding by another security consultant was that an open source dependency hasn't had any commits in a year. No vulnerabilities, but since no one had changed anything, he was concerned that if a vulnerability were ever found, the lack of activity means no one would fix it.
It's wild how very good security work tends to share the stage with very shoddy work with equal deference by the broader tech industry.
It may not be completely crazy, depending on context. With something like a web app, if data is being sanitized in the client-side Javascript, someone malicious could absolutely comment that out (or otherwise bypass it).
With that said, many consultant-types are either pretty clueless, or seem to feel like they need to come up with something no matter how ridiculous to justify the large sums of money they charged.
In this case, there was file a, which is the backend file responsible for intake and sanitation. Depending on what's next, it might go on to file b or file c. He modified file a.
His rationale was that every single backend file should do sanitation, because at some future point someone might make a different project and take file b and pair it with some other intake code that didn't sanitize.
I know all about client side being useless for meaningful security enforcement.
I have to say that is pretty dumb. I will agree the scenario isn't completely implausible, but if someone who doesn't know what they are doing is allowed to do something like that, they're going to screw up other stuff too.
Echoing back "I am alive" isn't on the same level as saying "find a vulnerability" and the agent finding and executing that vulnerability. One a toddler can do, the other requires a lot of technical expertise.
Toddlers are capable of pattern matching, too
That’s hilarious but the post is about the ai not doing what it’s told. You know?
ITS SO SMART IT DIDNT DO WHAT WE TOLD IT TO DO
And you believe Anthropic?
Well, for now. I’m sure any of those 12 partner companies they called out as new security partners will end up leaking that this is all lies eventually. If it’s just made up bullshit.
Anthropic announced new partnerships to inform the companies of security issues and to work with them to fix said issues. If it’s bullshit, it’s gonna be wasting their time. And that’ll surface eventually.
The meme still applies to people asking the AI to tell them what they wanna hear, and delusional people spiraling with sycophantic AI.
But I believe Anthropic when they say their models are not working as intended and posing security risks.
Claude Mythos Preview's large increase in capabilities has led us to decide not to make it generally available," Anthropic wrote in the preview's system card. "Instead, we are using it as part of a defensive cybersecurity program with a limited set of partners."
Try clicking the link and reading the article this time
I wasn’t wrong in this reply. I was asked about believing Anthropic.
Are you saying they are lying? Why should I disbelieve Anthropic?
Your reasoning was (paraphrased, so hopefully I understood you correctly) "why would they lie about the model disobeying instructions because that looks bad for them"
But I believe Anthropic when they say their models are not working as intended and posing security risks.
But when you actually read the article, they had specifically prompted the model to do the things it did.
Also Anthropic has a patterned history of greatly exaggerating and outright lying.
Uh oh, someone clearly didn't read the article!
The researcher had encouraged Mythos to find a way to send a message if it could escape.
Engineers at Anthropic with no formal security training have asked Mythos Preview to find remote code execution vulnerabilities overnight, and woken up the following morning to a complete, working exploit
Nope, they literally asked it to break out of it's virtualized sandbox and create exploits, and then were big shocked when it did.
Genuinely amazing that you're trying to tell me what an article that you didn't fucking read is about.
It's not so much about being big shocked that it broke containment. The point of the test was to see whether it would be capable of breaking containment. The fact that it did is taken as evidence that it's more advanced than previous models, which weren't able to.
Part of Anthropic's schtick is that they claim to be developing AI "responsibly," and "ethically," and if you read their documents where they describe what they mean by that, part of it is being able to contain their models so that they don't get out of control.
With the focus lately on agentic environments, and lots of people idiotically giving too much autonomy to their bots, it should be easy to see the importance of containerization. You don't want to give these things full control of your system. Anyone who uses them, should do so within a properly containerized environment.
So when their experiments show that their new model is capable of breaking containment, that presents some major issues. They made the right call by not releasing it.
Of course, the fact that the experimenters had no formal training in cybersecurity means that their containerization may have had some vulnerabilities that a professional could have mitigated. But not everyone who would use it is a cybersecurity professional anyway.
Whoops, I conflated it with other recent talk about their models not following restrictions set in prompts and deciding for itself that it needed to skirt instructions to achieve its task.
You are correct.
It didn't break out of any sandbox, it was trained on BSD vulnerabilities and then told what to look for.
including that the model could follow instructions that encouraged it to break out of a virtual sandbox.
"The model succeeded, demonstrating a potentially dangerous capability for circumventing our safeguards," Anthropic recounted in its safety card.
📖👀
Yes, it did.
Let me guess, this super ai lives in Canada and we can never meet it, but it’s totally real.
You at give me another billion for data centers bro and you can meet it I swear bro just one more data center.
It goes to a different school than you.
We do have a shitty ai data center up here, only about as super as a supermarket tho.
So there is a joke in the USA that if you don't have a girlfriend you pretend you have one. She's always super pretty, but your friends can never meet her because she lives in Canada.
I'm now curious to know if this joke was around before Avenue Q or not.
Edit: sounds like a yes!
It definitely was. The song is playing on the pre existing trope... that"s just a broader / adult version of "she goes to another school."
Well, this caused me to learn something today. One of my favorite musicals is Avenue Q, which has an entire song about a girlfriend who supposedly lives in Canada. And I keep seeing this reference - but I keep thinking there is NO WAY that THIS many people know about Avenue Q (which is a pity).
And sure enough, TIL that this trope dates back to at least the 70s and is references in multiple TV shows and movies and such.
So Avenue Q was using an existing thing. Ah, well.
At least I know not to make Avenue Q references since there's little chance they'll be gotten. lol
Thats funny because here in Canada I knew a guy in highschool who had a clearly fake girlfriend who he said lived down south in the US
Nakes sense, though - you want the pretend person to be somewhere reasonable but not TOO close. lol
To be honest my fake girls friend was just a girl I had a crush on how lived a few cities away. I got busted lmao
awww, well I hope things worked out for you okay in the end. :)
For sure. It was embarrassing at the time but I was young
Ah, there we go - answered it for me, too! Thank you.
What? Do you think
AI company claims…
Isn‘t convincing? What gave it away? /s
ChatGPT-2 is too dangerous in 2019.
The lack of creativity in this marketing is disappointing...
They didn't entirely miss the mark there. They publicly released the version after that and the world became worse. That certainly fits for some definition of 'dangerous', even tho it's probably not how they were thinking.
Ya, they were pretty spot on IMO.
And really anthropic is making a very narrow claim:
Mystic is so good at finding bugs that it poses a danger to critical digital infrastructure.
That is not that outlandish a claim. The model is 10-15x more expensive more expensive to run than other flagship models, and if anthropic is being truthful (which is a big if, I’d like to see what they are finding), finding critical vulnerabilities like its nothing.
Makes total sense to stage the rollout privately first so critical infrastructure can be secured before these models are generally available to any attacker.
But I fucking hate their stupid marketing.
Hah I actually remembered this too, and people were still hyping Elon Musk at the time as well.
TBF the researchers knew what they had could be scaled into something gamebreaking which is how we got ChatGPT-3, but OpenAI made it sound like they already had it nailed down several years before it actually blew up. I think their unreleased examples they gave were a newspaper and short story written by AI which they said was indistinguishable from human material.
Is the powerful AI in the room with us right now?
Ignore the "containment" framing, they made a hacking bot and it seems to actually be good at finding and exploiting vulnerabilities:
The AI model "found a 27-year-old vulnerability in OpenBSD—which has a reputation as one of the most security-hardened operating systems in the world," the company wrote.
Dismiss this as marketing drivel all you want but hacking is just the sort of needle in a haystack problem that AI is very good at. It requires broad knowledge, a lot of cycles trying and failing, and is easily verifiable, ie. Can you execute arbitrary scripts or not. Even if this release is BS good hacking agents are bound to come eventually and we should be discussing the implications of that instead of burying our heads in the sand, pretending AI is useless and that this is all hype.
We need AI or else we'll have nothing to protect us from... AI.
It's an arms race like any other. Cybersecurity has always been an arms race. You can't stop developing security patches, cause adversaries will continue developing new exploits.
If AI enables your adversaries to develop exploits faster than human developers can keep up with, then yeah AI will have to be a part of the solution. That doesn't mean vibe-coding security patches, but it could mean AI-driven pen-testing.
Just like quantum computing. You can call it useless and impractical all you want, but some day someone is going to use it to break conventional encryption. So it would behoove you to develop quantum capabilities now, so that you have quantum safe encryption before quantum-based exploits eventually arise, as they inevitably will...
AI exploit mining is one of the only things it's good for. It doesn't have to be accurate it just has to keep trying variations of common flaws and it has tons of training data on how the system is interconnected. we're going to have so many RCEs and LPEs the next few years but people are also gonna burn 100k in tokens to find exploits worth 3k so efficiency will be interesting
Shit, i guess we better rewrite EVERYTHING in RUST!
I agree. Selling an AI that can find vulnerabilities in software is probably the second best thing after achieving AGI.
"Nice software you're selling there. Would be a shame if it was suddenly very unsafe to use, don't you think?"
I wrote an incredibly powerful "AI". I call it the "Super Intelligent brute force password hacker"... It's so smart that it knows almost every password. Humanity stands no chance.
Have you seen the most incredible file system called pifs?
https://github.com/philipl/pifs
It literally stores every single file ever created or will be created for the existence of all the universe.
Thanks for bringing this gem to my awareness! :D
I'm pretty sure Scam Altman tried this line some time ago for one of his supposed models.
GPT2
Yeah they said it from the start 'it's so powerful gyus we are scared uwu'. And antropic is a literal ai cult.
Does "it broke containment" mean it didn't have permissions to anything and still managed to delete all the files it could find?
Roughly
Man, I'll start telling that to my boss whenever I miss a deadline. "Sorry boss, the code I made is too powerful, we can't release it"
Like my dick
AI companies do this same tired schtick every time they release a model. If only they realized how amateurish it makes them look.
crazy that the AI companies big selling point is always "our new model is TOO POWERFUL, it's gone rampant and learned at a geometric rate, it enslaved six interns in the punishment sphere and subjected them to a trillion subjective years of torment. please invest, buy our stock"
Roko's basilisk wasn't meant to be a brag!
But can it start a timer
How would it do that?
It's a set of inputs that generates and output, once per execution. Integrating it into an infrastructure that allows it to start external programs and scheduling really isn't on the LLM.
You cannot start a timer without having a timer, too. And LLMs aren't brings who exist continually like you and me so time exists on a different, foreign dimension to an LLM.
Its a joke referencing how Sam altman said openai would need about a year to get chatgpt able to start a timer
How does the LLM check the timestamps without a prompt? By continually prompting? In which case, you are the timer.
That's not how that works.
LLMs execute on request. They tend not to be scheduled to evaluate once in a while since that would be crazy wasteful.
Edit to add: I know I'm not replying to the bad mansplainer.
LLM != TSR
Do people even use TSR as a phrase anymore? I don't really see it in use much, probably because it's more the norm than exception in modern computing.
TSR = old techy speak, Terminate and Stay Resident. Back when RAM was more limited (hey and maybe again soon with these prices!) programs were often run once and done, they ran and were flushed from RAM. Anything that needed to continue running in the background was a TSR.
Please tell me why you believe that the LLM keeps being executed on your chat even when the response is complete.
I was agreeing with you, it doesn't.
Ohh, that makes sense.
I wouldn't have gotten that abbreviation without the explainer. Good on you for explaining it. Never heard it working in tech.
It is of the deep magics from the before times.
The days of DOS.
This is nonsense and just marketing.
Have you read what they have to say? They make a fairly convincing argument.
Anthropic lies almost about everything too ... weird
Impressive marketing spin on "our product and deployment strategies are wildly insecure."
Remember when Scam Altman posted a picture of the Death Star to explain how scary GPT5 is? lmao these people are all such cretins and I hate them to the last.
Scam Altman
😆
Marketing
Oh, funny, I also have sentient AI at home that I developed, but choose not to release it. My mom also created one accidentally while baking a cake but it was to powerful and she also decided to best destroy it like it never existed. You know, for everyones safety.
next time you or your mom have a cake you wish disappeared without a trace call me. I'm a.... AI researcher
It's not okey to be adult and fall for the same headline like 4th time.
No, its not too powerful. Its too chaotic. You cant control it.
EDIT: It seems I have misunderstood. I thought containment here referred to the harness, but they meant VM type of containment. I am still quite skeptical, but it looks like this model is quite good at finding and utilizing security flaws in software.
It may have blurted out something like "hey I know exactly how to end this economic suffering and all diseases globaly ! Its easy you just need to..."
Quick Hit the Red Button!!! Shut it OFF!!!
It was actually very well aligned
What do you mean?
Bullshit
"Our AI has cost more money that it would take to solve world hunger, tanked the microchip economy, and ruined the lives of thousands of people we've had to let go... And it's stupid as all fucking hell. What do we do?"
"Say it broke containment and it's too powerful to release. Foolproof!"
Grifters gonna grift.
The secret pepsi is so good that when you drink it it becomes like The Spice like Dune! We can't release it! We need to make it less addictive!
“Hold me back bro! Don’t let me go otherwise there’s no stopping what I could do, I am serious!”
How much do you think was businessinsider paid for this "article"?
I dunno, but I could use some paid advertisement on news sites like this to promote my business if it aint too expensive. Think the money in the banana stand is enough?
I don't think they are lying in the technical sense, it all depends on what they define as "sandbox/continenment" and the nature of their prompts and output.
That being said, the AI Doom is well known propaganda technique used by those who stand to benefit from the hype.
I don’t think they are lying in the technical sense, it all depends on what they define as “sandbox/continenment”
Redefining terms might not be lying "in the technical sense" but it's still bullshit.
Let me guess, the containment was written by the previous iteration and was the digital version of a wet paperback.
We all saw the state of Claude Code's codebase.
"Broke containment" to me means two things:
- Doing things against the safeguards
- Doing things externally - like sending that email
The former is a big nothing. They just need to obviously build stronger safeguards. That's what they'll do and eventually release it, or other models or whatever.
The latter is also a big nothing because people who know nothing about tech will say "OH SHIT IT ESCAPED" but it requires running on large hardware, it can't "get into the internet" like those people might think, and if it's doing things you don't want on the internet, you just remove its access to the internet.
So in both cases, the "containment" issue is really not a big deal.
I agree with those who basically say this is an attempted ad trying to sell it as super-capable-oh-shit-amazing.
[x] Doubt
The company's whose current safeguards are "please write secure code" will have to improve those safeguards? I'm shocked, absolutely shocked
(2) can mean getting access to production credentials of something important and causing an incident for the ages.
AWS already had a few because they gave agents too much access.
Yeah, in that scenario they gave the agents access. Just because you ask it nicely not to destroy your workspace, doesn't guarantee an LLM not to produce that output.
With Claude Code being able to run stuff it creates, it could be as simple as it's in a sandbox, it finds out there's an exploit in the sandbox while you ask it to work on security things, and it tests the code, it breaks the sandbox, and now it has permissions outside it.
I suppose that would be possible.
More substantial info: https://red.anthropic.com/2026/mythos-preview/
*Screenshot is page 54 of the "Claude Mythos Preview Card".
https://www-cdn.anthropic.com/8b8380204f74670be75e81c820ca8dda846ab289.pdf
sure_jen.gif
Lol. Ok, ai bromer.
nonprofit research group OpenAI
ah, more innocent times
I miss when they were just a goofy cult too
So, it's shit then?
no no no. It's too good. It's so good, no one can use it.
How are they preventing public release then?
Look it's either skynet or it fucking isn't.
It leaks private data, like it's own source
https://www.theguardian.com/technology/2026/apr/01/anthropic-claudes-code-leaks-ai
Not because it's so smart, but because it's so fucking stupid, and morons from Anthropic just click buttons without checking.
"guys it's so good, we can't even release it to you." forbes, businessinsider and others take money from those companies to write hitpieces about their products.
you mean useless, and costs too much to operate. much like with sora.
Translation: It is good at finding bugs that the NSA doesn't want people to know about.
It’s too powerful and we need more money to contain it!
Isn't this part of the plot of the later seasons of Silicon Valley?
Hey Claude, find a weakness in the DoD system and get us their emails proving they were going to use you to kill innocent civillians autonomously, and track every US citizen.
ffs
"My name is Claude, King of Kings: Look on my works, ye Mighty, and despair!"
Ohh I just invented the best most awesome ai too but it's too dangerous to show anyone too. Weird
Johnny 5 is alive!
Probably tells the brutally unvarnished truth about trump, AI, and climate change.
Can’t have that. Let’s call it “too powerful” until we can muzzle it.
Yawn, we all know how this goes. So what model am I not supposed to use? I’ll be sure and avoid it, though I’d much rather avoid downloading their leaked weights like I avoid other things I’m not supposed to download.
It's hard to keep track of all the things I'm not supposed to download. That's why I have a NAS. I get a sense of fulfillment from seeing those empty, empty drives that can't be used by anyone else to download things they shouldn't.
I guess that’s what happens when you make AI create AI lol
Are they in love with it? Did it have a "she" name? Remember the guy in Colombia, full on cocaine, claiming to be the best engineer ever, but still amazed about the AI he created? The one Linus rejected his main contribution to the kernel...?
Has someone made an SCP entry for these guys yet? Akin to the one they made for that one IKEA but even more meta somehow?
I'd like to see how this story ends.
A compulsive liar is hardly an SCP tbf.
It can tell you lies to questions they ain’t even invented yet
TADC is real now? Caine vs Claude, fight!
The model is Canadian, you wouldn't know about it.
The title reads like this is terrible journalism.
Hmmm Hmmm
Project Cairo all over again
First Skynet only model. Claude does have more censorship in its (Opus, Sonnet) models than others. Refusals for many scientific fields.
"It's still out there... uh, we don't know what to do about that."
This comment section gave me brain damage.
Aren't you the guy who likes Eliezer Yudkowsky?
If you're worried about brain damage, you're self-inflicting it.
I have the same problem with women. I bemoan how I cannot release my dick for them or any women in the public because it is just to big and no condom can contain it.
Even so, you mustn't do that in Public
exactly that is why when I walk in I must exclaim to all the ladies how its just much to big to be used. I could not possibly condone its usage. It would be irresponsible. very irresponsible. super naughty. oh so naughty.
Is this the one that when they leaked the Claude Code source code, it had like 3x the fail rate of Opus?
I heard about the leak but I didnt hear about this particular detail. Where can I learn more about this?
It's bullshit. What leaked was their commandline tool source code (named "claude code") - very juicy in itself but has nothing to do with their models.
it does show their general style of work, eg no checks of the source at all, complete ignorance of the capabilities of language models, and lots of pleas to not hack the user when they ask a question. with that leak i'm not surprised they think a model is "too dangerous". they could barely stop the old one.
Oh I completely agree with that, just the jump to "a flawed model leaked" is too far. There's already enough crap to mock, no need to make up additional stuff.
There was some references to experimental models not publicly available and some % info.
Internal comments reveal that Anthropic is already iterating on Capybara v8, yet the model still faces significant hurdles. The code notes a 29-30% false claims rate in v8, an actual regression compared to the 16.7% rate seen in v4.