Rendered at 02:39:06 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
pimeys 5 hours ago [-]
I have taken another look on these open models after the fiasco of Fable and GPT 5.6 this weekend and... GLM-5.2 truly is a good workhorse model for daily programming. I consider myself a heavy user of LLMs and a seasoned developer. A typical session for me with GPT is usually over a hundred dollars...
This weekend I programmed a matrix bot with encryption and a Rust agent with some tools. Because I need one and OpenClaw just felt... not what I wanted. Two days later and 20 dollars poorer I have what I need: a multimodal agent written in rust that has access to my homelab.
Nothing felt off with GLM. It did what I wanted, was fast, had a decent not very annoying personality and was much cheaper than Opus or GPT.
I used it unquantized through Fireworks, but there are multiple other providers too.
gertlabs 3 hours ago [-]
GLM 5.2 is a great model, but if you only want to use the best model available, it isn't there yet. Every lab releases models that memorize benchmark answers, both intentionally and unintentionally. But we consistently find that models from Chinese labs have a wider gap between public benchmarks and our evaluations, which we designed to be less vulnerable to benchmaxxing.
In multi-agent coding environments, GLM 5.2 is just shy of Opus 4.6 on average. Data at https://gertlabs.com/rankings
But when factoring in performance/cost, GLM 5.2 is the frontier model.
ronsor 25 minutes ago [-]
Opus 4.6 is still my preferred model for work, so this is great to hear.
bjourne 57 minutes ago [-]
Man, there is exactly zero information on your site about how your benchmarks work. Why should one trust your numbers when there is no way to verify them?
gertlabs 50 minutes ago [-]
Scroll to the bottom for the methodology (sorry, this should be linkable)
skeptic_ai 2 hours ago [-]
Why Deepseek v4 flash is better than pro in your benchmarks?
gertlabs 49 minutes ago [-]
It's 100% due to tool use -- Flash adapts much better to our custom harness with tool names that are not identical to what models were likely trained on. DeepSeek V4 Pro performs much worse in that aspect than almost all other recent releases, for whatever reason.
rockwotj 1 hours ago [-]
I have also found deepseek flash beat pro in some of my own internal evals for tasklet.ai it’s really surprising and I don’t understand it
freakynit 55 minutes ago [-]
Same.. although rare, but have observed twice till date.
Some blog post I read few weeks back said that DSV4Flash in xHigh effort beats even the pro model in xHigh effort.
onoesworkacct 54 minutes ago [-]
The rumour is that it's trained on Opus, but who knows
rockwotj 42 minutes ago [-]
Oh of course all deepseek and glm are. Multiple people have seen GLM self report that it is claude, which makes it super obvious.
I think the surprising thing is I expect flash to be a pure distillation and strictly worse quality but clearly it’s more nuanced than that.
jchw 2 hours ago [-]
After having used GLM 5.2 and Opus 4.8 for enough time, I'm very unconvinced of the benchmark maxxing claims - if anything, GLM 5.2's rather lackluster performance on benchmarks compared to Opus 4.8 paints the opposite picture when compared to the subjective experience.
When I first used Opus 4.8, I threw several different workloads I had at it - I have Claude doing a lot of misc projects whose primary purpose is pretty much just studying what AI agents can do for my own curiosity and no other reason. Opus 4.8 was one of the first models I ever snuck in there that basically ran out of control. No previous Opus or Sonnet model I had used ever did this. Within hours every agent I had running was writing non-sense tool calls that echoed pretend commands that didn't exist, like 10 in a row, and talking about the "tool channel" being dirty. I switched back to Opus 4.7 and assumed Opus 4.8 was legitimately just broken.
I did come back to Opus 4.8 and found that it was indeed, pretty powerful. But that initial experience has stuck with me on just how narrow of a perspective any given test or benchmark is guaranteed to have. LLMs are too broad, it really doesn't matter what you try to do in your benchmark, you will necessarily get a limited view of what the model is capable of and its shortcomings. This will remain true for at least as long as models are susceptible to massive swings in performance based on randomness and minor differences in prompts and other environmental factors.
I'm not saying benchmarks are useless or that your benchmarks are not possibly closer to the truth either. All evidence at least points to the idea that Chinese models perform very well in coding but often have more mixed results on other tasks. I'm just saying that at this point, benchmarks feel like they have limited connection to my actual real experiences. GLM 5.2 actually scored kinda meh on a lot of benchmarks (compared to closed frontier models) but my actual experience using it does not match this.
And I'm definitely not saying GLM 5.2 is better than the frontier LLMs here, just that the race is close. I still prefer GPT 5.5 right now for code review, I think, and Opus clearly has some advantages depending on the task. It's just no longer a given that Opus 4.8 will perform better than GLM 5.2 on any given task, so to me the calculus behind "using the best model available" is getting complex and you might need to get a feel for what models have what strengths to really figure it out.
I do feel like the "use the best model available" mentality is not going to die any time soon, but if it does die, it will be gradual and start soon for programming. Modern LLMs are still not a full superset of what human programmers can do, but still larger models are definitely starting to hit diminishing returns for tasks at the lower end of complexity, and that is a big deal. It's a weird world where some tasks you can feel kinda confident just throwing Gemma 4 at it and not sweating whether you should use a better model; I've certainly done it for some quick Python scripts or getting an overview of some code I'm unfamiliar with.
Madmallard 1 hours ago [-]
Notice the website url is the same name as the commentor.
Notice he's using "trust me bro" benchmarks.
Can we just remove all the motivated speech on HN? This is just not trustworthy information at all and obviously is incentivized.
Everyone is grinding and marketing nobody is actually discussing anything for real.
Aditya_Garg 3 hours ago [-]
Im really curious about this. Why pay API pricing? I burn 1000s of dollars a month of api according to claude usage but only pay the $100 subscription
horsawlarway 3 hours ago [-]
My increasing frustration with these plans is the harness lock in.
Anthropic won't even let you run "claude -p [prompt]" any more... They bill it at api rates.
So if you're trying to automate the ai (and seriously, that's the point) the subsidized plans are crippled.
cortesoft 3 hours ago [-]
They postponed that change, here is the email they sent out:
> In May, we sent you an email announcing that starting today, the Claude Agent SDK, claude -p, and third-party apps built on the Agent SDK would stop drawing from subscription rate limits and move to a dedicated monthly credit. We're writing to let you know that we’re not making this change today. We’re working to update the plan to better support how users build with Claude subscriptions.
> What this means for you
> Nothing changes for now. Agent SDK, claude -p, and third-party app usage continues to work with your subscription exactly as it did before today, and there's no credit to claim. Your subscription limits are unchanged. When we have an update, we'll share it with advance notice before it takes effect
smcleod 3 hours ago [-]
They canned the moved to make -p commands API billable.
sroerick 3 hours ago [-]
I'm using synthetic.new and Neuralwatt with pi and its good and also cheap
computerex 3 hours ago [-]
I have had bad experience with neuralwatt GLM 5.2. Seems like they may be using quantized version of the model.
scottcha 1 hours ago [-]
Hi I'm the CTO of neuralwatt, would love to hear your feedback on what your experience was. Feel free to email me scott@neuralwatt.com. Also for GLM5.2 we run the FP8 quantization at 1M context which is a common deployment target.
throwawayffffas 3 hours ago [-]
Z.ai does not lock you in to any harness.
weird-eye-issue 3 hours ago [-]
I think they rolled that back
SV_BubbleTime 3 hours ago [-]
There is a whole iceberg topic on subsidizing.
So your question is really “if they’re giving free usage, why not take advantage of it?”
I do, so I don’t know the reasons not to, other than to experiment.
AussieWog93 2 hours ago [-]
[dead]
shostack 5 hours ago [-]
If you're using Matrix, consider Hermes as a harness if you haven't already. Native gateway support. I've been primarily using mine through Element and it has largely been great.
pimeys 4 hours ago [-]
Oh interesting. I basically chose Matrix because setting anything up with Whatsapp or signal was kind of painful and telegram doesn't make it easy to use encryption with bots.
I kind of wanted to see if I can make a Matrix agent from scratch with Rust with GLM and it was surprisingly easy. Just make something for myself how I want it. Maybe I'll take a look on Hermes later...
Barbing 2 hours ago [-]
Very interesting—Element X solved a lot of the pains of Element (iOS), could be a good solution!
KaoruAoiShiho 4 hours ago [-]
Are you sure fireworks is unquant? It's not listing precision on openrouter like everyone else.
jklmnopqrstuvw 1 hours ago [-]
> A typical session for me with GPT is usually over a hundred dollars.
I don't think a $100 session is "typical". I use GPT for months. $20/m plus plan is enough for my daily work.
simple10 4 minutes ago [-]
I use an observability tool with claude code [1] that shows me usage including prompt and session cost. Even though I use a max subscription, it's interesting to see what it would cost me if I was using API directly.
My typical session ranges from $100-$400 - higher end when using workflows with lots of subagents. $100/session is expected when using the API without the subsidized subscription pricing. Most larger orgs have to use API pricing AFAIK.
I have Claude max plan and the vscode claude dashboard plugin has logged about $4k worth of tokens in the past 2 months. I upgraded because I was using my weekly basic plan tokens in like 5 hours.
Likewise, I don't understand how anyone survives on the basic plans. It's funny seeing these two camps not understanding what the other is doing :)
adamtaylor_13 26 minutes ago [-]
It's really interesting what "normal" is for folks. I use the $200/month Anthropic subscription and use it within a few percentages of my limit every week.
Just FYI, this question was a quote from Pulp Fiction, the other commenter (mdre) replied also with a quote, that was an answer to this question in the movie.
mdre 4 hours ago [-]
[flagged]
dom96 3 hours ago [-]
Twenty dollars?
How are you comfortable spending that much to write something as simple as a matrix bot?
Are people doing this kind of thing just super rich or am I missing something?
ygjb 1 hours ago [-]
It's pretty simple. There are things that I do because it's fun, like gamedev. I hand code that, and don't use LLM tools because I like learning and building. I do lots of utility stuff coding for my wife's business, most of that is stuff I could do in a few hours. It's worth $20 to not spend a few hours doing it. It's a cost benefit tradeoff. I won't learn much fixing WordPress themes or adding a feature to her web page, or setting up an automation for her, so I don't see the point of doing that.
Same thing for stuff at work. Oh, the tables/schema changed and my queries broke? I could dork around with spark and cypher for an hour, or I can tell claude to update the queries for the new schema. At the rate I am paid, spending on Claude tokens is generally a better use of my resources.
Building a net new solution? Coding tools take a back seat until I get the core logic right, then I let automation handle web page and UI scaffolding.
copperx 22 minutes ago [-]
$20 is really cheap for the amount of work saved, considering you're in the US.
adamtaylor_13 25 minutes ago [-]
Is spending $20 considered "super rich"?
playorizaya 3 hours ago [-]
[flagged]
SwellJoe 4 hours ago [-]
I added GLM 5.2 to my security bug hunting benchmark when it came out, and found it to be a good performer, but not the best open model. The benchmark tests whether models can find bugs Mythos found. The best open models in the initial benchmark were DeepSeek V4 Pro or MiMo 2.5 Pro. But it turned out MiMo got lucky, it's performed worse on almost every test I've done since, while DeepSeek has consistently been among the best performers and its extreme caching performance makes it cheaper than just about anything, including much smaller models.
Also of note, I found giving models access to the open source semgrep as a tool makes some perform worse and none perform better, though it's plausible there's a way to wire it up in a harness that presents useful information to the model without the model having to know how to use it (my theory is that semgrep isn't heavily represented in the training data, so you're asking the model to do two things at once: figure out how to use semgrep and find security bugs, and both tasks suffer for the lack of focus...most small models, and some big models, can't do that well).
Edit: But, also, more testing is ongoing. I suspect GLM 5.2 will also be a consistently strong performer. It seems to excel at most things I've tested on it.
Overall, I still think GLM 5.2 is the much stronger performer. It's hard to tell the difference between GLM 5.2 and Opus at <120k tokens.
SwellJoe 37 minutes ago [-]
I have found that some models consistently find or miss specific bugs, and which bugs are hard don't completely line up across all models, so I believe that. I just refactored the security bug-finding harness I've been working on completely (not checked in yet, testing it currently) to strongly encourage "multi-model, multi-pass" scans and make them easy to orchestrate with de-dupe and weeding false positives with a strong model, rather than one model or doing just one pass over each file. Giving a model a second attempt increases their findings by 20-30%, and giving them a third, adds another 10-15%.
I'm inclined to use DeepSeek V4 Pro the most, because it is consistently extremely strong, it's very fast, it's very cheap and has excellent caching and cheap-as-free cached input tokens (something like 80% of token usage is cached when I'm using it for security scanning). So, my probably "pair" of frontline security researchers will probably be DeepSeek V4 Pro and Gemma 4 31B self-hosted (another shockingly strong contender, competitive with the best models once you let it loop on the same file a couple/few times). But, I won't be surprised if GLM 5.2 turns out better than DeepSeek V4 Pro...it costs quite a bit more.
Barbing 2 hours ago [-]
We need a benchmark of independent community sourced benchmarks!
…probably already is one
mapontosevenths 42 minutes ago [-]
It's not super scientific, but I really like to watch Bijan Bowen's videos on Youtube. I think he's pretty fair about the way he compares them, and it's enough for what I'm doing.
SwellJoe 30 minutes ago [-]
Actually doing something normal but challenging with a model is generally enough for me. I do a quick (an hour or two) project, and see how it holds up. If I'm feeling like it's harder than it should be, I switch to a comparable model I know is good. e.g. I most recently tested Gemini Flash 3.5 for making a web app. It shit the bed...kinda worked, but was ugly and needed several bugfixes right off the bat. I tried the same app in Opus 4.8, which aced it with barely any extra conversation, it looked great (basic but clean, like it was intentional) without any effort.
I like reading benchmarks, but I take them all with a grain of salt. They're just to tell me if the model is worth even trying for my task. I've heavily used self-hosted Qwen 3.6 and Gemma 4 on a bunch of different tasks, and while the benchmarks consistently say Qwen is the better model, I simply don't find that to be the case for anything I do. I think Qwen is tuned for benchmarks, while Google couldn't give two shits about most of the benchmarks, they're just busy making unusually smart tiny models.
SwellJoe 2 hours ago [-]
I don't know how you'd judge benchmarks beyond "did it test and measure what it says it tests and measures". And, I guess there have been instances where the benchmark failed to do that, and the models could cheat in some way and it just tested the models ability to find the answer key. In the case of my benchmarks every model other than Claude models running in Claude Code never have network access and all information from after the bug was discovered has been removed from the repository the model can see.
But, there are benchmarks for so many different kinds of ability, I don't know how to compare them directly against one another. Like, models that do well on terminal and agentic coding benchmarks tend to do well on finding security bugs, but it's not a 1:1 correlation, there are surprises.
bArray 5 hours ago [-]
Apparently GLM 5.2 is 753B parameters [1], what kind of hardware are people using to run this locally?
I ran it on my laptop, which is a Lenovo Legion 5i (think 32 GB RAM, 4060 w/ 8 GB VRAM, you get the picture). It was a quantized model (otherwise it would not fit on my NVMe 1TB drive) at 4 bits per weight - UD_Q4_K_XL. It ran at about 12 seconds per token (not tokens per second). A fun project, but not worth it. I used 4096 tokens of context cache, and I ran it with llama.cpp - as it supports memory mapping. Because the whole thing could obviously not fit in RAM, I was curious how much it would need to stream from SSD. The answer? For a simple 4 sentence description of who it was, about 1.5 TiB was streamed from disk.
bArray 2 hours ago [-]
Thank you for sharing. 1.5TB of streamed data at 12 seconds per token on a high end consumer laptop is a pretty high requirement - I can only imagine how much that cost to train. I don't know how running this model could be cost effective for anybody.
Retro_Dev 2 hours ago [-]
Indeed - definitely not cost effective to run it on this laptop LOL. It makes me wonder how fast we could run the model if we could fit the weights entirely within CPU cache (assuming a whole ton of CPUs with low latency & high speed IO of course).
A few people are running highly quantized models with limited context windows. It's still impressive, but not the benchmark level intelligence. Very few people could afford a rig for reasonable local performance at a reasonable quant, at full context size.
The antirez example is 2.6bit quant, 32k context, and few tokens per second... on a ~$7000 MacBook M5 (new RAM pricing).
4 hours ago [-]
dakolli 5 hours ago [-]
8 X RTX6000. It will run you around 80-100k to get started with a model at this size with decent tps..
Don't worry though, open source evangelists will tell you that these will be running on your phone in the next 3 years.
For $100k you could run this model 24/7 through open router with 10 concurrent sessions at 50tps for a decade and have money left over for a vacation. There's no point in investing this type of money in local models unless you have a business where you're already paying for many employee's individual token usage.
Aurornis 4 hours ago [-]
> 8 X RTX6000. It will run you around 80-100k to get started
8 x RTX6000 GPUs cost $100,000 alone. You then need to build a system that can support those GPUs with enough PCIe lanes through a PCIe switch.
It's going to be $120K to $150K to build or buy a system to run this.
cheschire 2 hours ago [-]
Not to mention the three separate dedicated 15A circuits you would need to have installed in order to run the 3x 2000W power supplies running ideally at no more than 1400W sustained load each. And definitely would need 200A service to the house if you have a family living there with you.
But hey you could save on heating?
InvertedRhodium 1 hours ago [-]
That’s a uniquely US issue - in NZ you can get a 100A single phase at 230V nominal without any issue. 23kw, straight to your door.
A single circuit using 10mm TPS would technically be enough to run what you’re describing. Might be pricey though, I’d probably take the excuse to get 3 phase installed so I could get access to the stock of used 3 phase machinery.
knollimar 3 hours ago [-]
isn't throwing that into a [insert financial vehicle that gives 99.99999% safe returns] going to destroy that when you factor in electricity costs?
Or even just electricity costs vs token cost
CamperBob2 4 hours ago [-]
You can run the NV4FP quant with 8x RTX6000 cards at 50-75 tps output, but not (practically speaking) the OEM FP8 version. You will learn more about PCIe than you ever wanted to know.
The real gangstas are running 16x RTX6000s. Too rich for my blood, and the NV4FP quant doesn't seem to be that much worse.
Sanzig 3 hours ago [-]
Anyone done any benchmarks on the NV4FP quant? Seriously considering pitching an 8 x RTX 6000 Pro box at work to run GLM-5.2 in an air gapped environment.
tiahura 3 hours ago [-]
Good luck. I’m in the legal field, and even there, selling airgapped is tough.
AussieWog93 1 hours ago [-]
>Don't worry though, open source evangelists will tell you that these will be running on your phone in the next 3 years.
Not sure if you're being sarcastic, but I can run a quantised version of Gemma or Qwen on my 16GB M1 Macbook Pro that beats GPT-4 from 2023 hands-down.
I wouldn't be surprised if, in another 3 years, you'd be able to run something as powerful as Opus 4.5 or GLM-5.2 on standard consumer hardware - say a 32GB/64GB M7 Pro.
I also wouldn't be surprised if, 3 years after that, cheaper hardware and improved model efficiency means that there's a much smaller gap between what you can run on a consumer CPU (which, with memory prices coming down, could look like a 256GB M9 or M10 Pro) and $100k GPU cluster.
marcus_holmes 46 minutes ago [-]
This is clearly where the industry is going, imho. Everyone who is playing with LLMs wants a laptop with enough grunt to run a decent model locally.
We've been sat with basically the same PC specs for ~20 years - our current specs are within an order of magnitude of the ones we could buy back in 2010. This is not really constrained by tech, as we could have much, much, larger machines. It's more because there's no mass demand for much, much, larger machines - if it's big enough to run Office apps or VSCode then you're good to go. The exponential growth we saw in the 90's was driven as much by software demand as it was by hardware development.
I can see the next 10 years produce the same kind of push for larger machines that the 90's did. And we should probably expect the same kind of standards churn as our existing technologies for storage, memory, etc, don't scale up enough and new technologies become worth developing because there's demand for them.
InvertedRhodium 4 hours ago [-]
Depends how much you value privacy and running uncensored models.
Personally, I’m waiting for hardware to hit the secondary market before I buy something to run unquantized models like GLM. But I have no doubt that I will, at some point.
Ldorigo 2 hours ago [-]
How do the economics of your statement work out? Clearly inference providers don't have a time to ROI of 10 years on their hardware costs; and that's without even taking ongoing energy costs into account. What's missing here?
ac29 59 minutes ago [-]
The inference providers are running batch sizes much larger than 10
krackers 5 hours ago [-]
Would you be better off pooling that money with some hackerspace group and then setting up shared inference infra, so that way you at least get better utilization?
KaoruAoiShiho 4 hours ago [-]
And before you know it, you invented some openrouter provider from first principles...
janalsncm 3 hours ago [-]
Right. For example you will need to figure out how to share it and who maintains it.
aetch 3 hours ago [-]
You can then rent spare capacity out to people on a subscription or token basis ….wait
8note 5 hours ago [-]
you can however, have fun with it.
oil workers buy 100k trucks they do not-much with. why not a 100k in computer?
Ken_At_EM 5 hours ago [-]
I can't help but ask where this comment came from, you must have some exposure..
CamperBob2 4 hours ago [-]
It is so easy to spend $100K on a pickup truck these days, it's not even funny.
tiahura 3 hours ago [-]
A Honda minivan is > 50k.
SV_BubbleTime 3 hours ago [-]
Factory F350 Platinum is at least 90k sticker.
afavour 4 hours ago [-]
Because car loans can’t be used to buy computers
ElProlactin 3 hours ago [-]
And there's your idea. If you could find a way to get people to add another $500/month over 80+ months to an auto loan, dealers would eat that up like filet mignon.
jliptzin 3 hours ago [-]
Yea as far has hobbies go, I feel like this is on the low end. I know people who collect watches and corvettes, that's way more expensive and functionally you can't really do anything special with them.
theteapot 3 hours ago [-]
The difference is watches and corvettes typically appreciate in value, where as computer hardware typically drops like a rock.
15155 2 hours ago [-]
> watches
Some, and the market fluctuates a ton.
> corvettes
Only the oldest, most unique model years: nobody is buying (C4-C5-realistically C6) mid-90s or early 2000s Corvettes for more than what they paid for them, and they never will.
randomNumber7 2 hours ago [-]
Also LLMs are mainly used for work and if you can spend 6 digits on watches your likely financially independent.
cdelsolar 36 minutes ago [-]
hmm ok let's build a state of the art from 2021 homelab using 2x Epyc Milan chips + DDR4 RAM and lmk how much it costs...
parineum 2 hours ago [-]
> The difference is watches and corvettes typically appreciate in value
Both of those things' value drops like a rock as soon as you buy them and, at least for cars, they don't all appreciate. Most don't. Even so, they appreciate at an incredible slow rate.
I can't speak for watches but I'd be surprised if it wasn't the same situation.
At least the gpus can create value after you buy them before they are worthless.
dakolli 5 hours ago [-]
Sure, If you want to light money on fire for entertainment, more power to you. There's probably worse ways to light 100k on fire. If I have an extra 100k laying around it's going to my family though.
KetoManx64 4 hours ago [-]
As an individual I do not need the whole model. I don't need the model to have knowledge of the rain history of Algeria nor how many colors are in the Russian flag. Once they start trimming down the excess and making them field focused they will run just fine on people's individual devices.
JumpCrisscross 4 hours ago [-]
> I do not need the whole model. I don't need the model to have knowledge of the rain history of Algeria nor how many colors are in the Russian flag
Isn’t the performance gap between quantized and full models indicative that even if you aren’t using it directly, the model knowing the colors in the Russian flag does have something to do with the intelligence you demand?
KetoManx64 4 hours ago [-]
Do quantized models specifically prune out specific knowledge? I think they just compress things down but they're still in there. You'd most likely need to do that when you're doing the initial model training, but I'm not expert.
JumpCrisscross 1 hours ago [-]
> they just compress things down but they're still in there
The compression is almost certainly in part specific knowledge getting fuzzed.
DennisP 14 minutes ago [-]
Yeah, but it's everything getting fuzzed, including the parts you care about.
kibwen 4 hours ago [-]
Quantizing is one thing. But in general it's self-evident that training the model on information that is irrelevant to your use case does not necessarily improve ability, otherwise you'd have AGI just from reinforcing your model on memorizing the first 10^50 digits of pi.
Likewise, LLMs do not violate the laws of information theory, and therefore the only way to encode X amount of information in Y amount of bits where X > Y is by performing what is effectively lossy compression, and as X grows larger relative to Y the compression ratio must change to lose ever more information.
Yes, for the sake of making chatbots that are "conversational" in that they can interpret natural language as input and produce code as output you can easily benefit in incidental and unintuitive ways by training it on more natural language text. But for a given fixed parameter size, it's possible to produce a better model for a specific task by selectively not muddying its training set in the first place with things that are likely irrelevant to the task.
JumpCrisscross 1 hours ago [-]
> it's self-evident that training the model on information that is irrelevant to your use case does not necessarily improve ability
We don’t understand AI or natural intelligence well enough to make such statements. As for self evidence, cross-domain competence in humans and the rise of generalist models over domain-specific ones (on competence, not cost) seems to pretty directly tank your hypothesis.
coldtea 2 hours ago [-]
>But in general it's self-evident that training the model on information that is irrelevant to your use case does not necessarily improve ability, otherwise you'd have AGI just from reinforcing your model on memorizing the first 10^50 digits of pi.
It's hardly self-evident, and your counter-example is hardly applicable.
The first 10^50 of pi is not the same as having BREADTH of information in the training data, which is the whole point not just any random "information that is irrelevant to your use case".
not to mention that the first 10^50 digits of pi compress to quite small formula, so not much information there to begin with from a shannon/kolmogorov perspective
kibwen 2 hours ago [-]
It is self-evident. Bringing up Kolmogorov complexity is irrelevant, we're talking about rote memorization, but if you can't ignore the given example then replace "digits of pi" with "bits of output from a true random number generator". There's an infinite amount of information that we could shove into a model, and a finite amount of bits with which to store any of that information such that it can be usefully recalled or form useful logical associations.
tiahura 2 hours ago [-]
Apparently irrelevant data can help because model weights are entangled.
wonnage 4 hours ago [-]
Yeah, the neoclouds and hyperscalers are taking massive losses right now, self hosting is basically signing yourself up to do the same. There are philosophical reasons to do so but it’s a terrible economic decision
rekttrader 5 hours ago [-]
Or you have data that HIPAA, GDPR, PII, or have to care about the concern of others training on your data.
dakolli 5 hours ago [-]
That too.
dist-epoch 5 hours ago [-]
> 50tps for a decade
assuming demand doesn't keep on increasing. even google has trouble having enough capacity apparently.
himata4113 6 hours ago [-]
These numbers are seem pretty low compared to what I was able to achieve specifically around windows kernel, win32k<->win32u to be exact. It honestly wouldn't surprise me anymore if china started surpassing models that US makes public, at least in specific categories such as cyber.
GLM 5.2 is already capable enough to assist in self-training which is similar to what we saw happen with frontier models and they appear to be getting there at a significantly lower cost than openai/anthropic.
EMIRELADERO 2 hours ago [-]
> These numbers are seem pretty low compared to what I was able to achieve specifically around windows kernel, win32k<->win32u to be exact.
Care to give more context to this? Seems interesting
danmaz74 4 hours ago [-]
It will almost for sure surpass the models which Trump will allow US "allies" (which he just considers client states) to use. This, together with China's growing dominance in PV, rechargeable batteries, EV, could really be the nail in the coffin for the post WWII economic world order.
himata4113 4 hours ago [-]
Honestly, it's becoming increasily hard to disagree with such sentiment when china is preparing itself to lead in energy, manufacturing, research, chip production and so on while there's an entire group of people trying to put datacenters in space.
woeirua 3 hours ago [-]
You are delusional if you think China is going to let Europe have access to Mythos level models for free.
chillfox 2 hours ago [-]
Why not?
Mythos level really doesn't seem that scary. And it would be a great way to take away the American labs international market.
I think it would make strategic sense for them to release more capable models than what American labs are allowed to make available to the world. It would help them grow their global soft-power and be a destabilizing effect on the American economy.
lukan 2 hours ago [-]
To hurt the US, maybe. I have not tried it, but GLM here seems already pretty capable.
jmye 2 hours ago [-]
What does "free" have to do with anything?
WithinReason 6 hours ago [-]
> [...] beating Claude Code (32%) at roughly $0.17 per vulnerability found
Claude Code is an agent harness, not an LLM.
Claude is a brand (or group of LLMs), not an LLM.
raincole 5 hours ago [-]
Yes, and the article author is fully aware of that. Thank you for pointing out this small mistake though.
mkagenius 4 hours ago [-]
It looks like the author is specifically avoiding model's name, because results are really weird.
Opus 4.8/4.7 scored 28%
Opus 4.6 score 37%
So the author thought as let's not get into that just write Claude.
happycube 4 hours ago [-]
Not weird at all, given the variance in Opus' quality over the last few months.
wild guess - I wouldn't be surprised if Opus 4.6 was run quantized for a while, and 4.7/4.8 have QAT for that nerfed size.
raincole 13 minutes ago [-]
Where is the weird part?
andriy_koval 4 hours ago [-]
many people think opus 4.6 was the best
tills13 5 hours ago [-]
It costs nothing to not be pedantic.
alienbaby 4 hours ago [-]
Possibly, nothing other than accuracy
croemer 2 hours ago [-]
The dollar amount is meaningless without comparison - and no other model has a price tag. Sloppy article.
Onavo 5 hours ago [-]
Claude code it's the only way to get access to the actual amortized cost of running a Claude-scale model. The consumer non-enterprise API is extremely expensive (with increasing marginal costs for the user and fat profit margins for Anthropic). If you want to approximate a State level attacker's cost where they can have the model on their own hardware, Claude Code is probably the best guess at the amortized cost.
gurjeet 37 minutes ago [-]
Twice in the text quotes Claude Code's F1 score as 32%, but the table shows the score is 37%. It's very likely that the actual score is 32% (because it is referenced 2 times, and a third time indirectly as the difference 'seven').
Oddly, this is a strong indication of the text being hand-written rather than LLM-assisted; it's very likely that a human made a mistake in creating the table.
> ... beating Claude Code (32%) ...
> ... GLM 5.2 ... beat Claude Code by seven points (39% vs. 32%).
> Rank | Configuration | Harness | F1
> ...
> 4 | Claude Code (Opus 4.6) | Claude Code SDK | 37%
XCSme 2 hours ago [-]
Does a bit worse than Opus 4.8 in my tests[0], but it's 5x cheaper and 3x slower.
You'd be surprised, some models on high do worse than on medium, because they start overthinking and doubting themselves, polluting the context with too much information, etc.
It depends a lot on the task and harness too (using plans and to-do lists, vs one-shot answers), but for simply answering directly to an inquiry, often extra thinking doesn't necessarily improve the answer, especially if the answer is binary, or can be correct or wrong, as opposed to having more time to refine a creative output.
XCSme 31 minutes ago [-]
Note that being open-weights, "slower" is relative, as it depends on who's serving the model. This can drastically change over time too.
solenoid0937 7 hours ago [-]
GLM export controls incoming? I predict Commerce will force OpenRouter, HuggingFace to take some open models down within the next few months.
Not that it would make any sense.
rgbrenner 6 hours ago [-]
If that happens it'll be an absolute disaster. Imagine a scenario where Anthropic and OpenAI prohibit most US companies from using their latest models because of safety.. And meanwhile attackers use equivalent open source models to attack US companies.
Any prohibition on open source models will do nothing to fix the problem.. since attackers will never feel bound to the law. All advanced models must be available for defensive purposes.
andy99 6 hours ago [-]
Right, but is there any evidence of intelligence behind any of these (government) decisions? It’s just regulatory capture + marketing (plus some people living out an imaginary fantasy that they’re in Neuromancer or something), absolutely no reason to think they won’t try and target open models as part of this.
popalchemist 6 hours ago [-]
There's at least one reason:
much harder to make a profit in policing non-american companies and open-source models without huge (or even any) MRR.
If the real motive is profit, then open source models are likely simply not a viable means to that end.
richardlblair 2 hours ago [-]
And someone will start a competing company in a sane environment.
solenoid0937 6 hours ago [-]
> since attackers will never feel bound to the law.
But that's the whole point.
Fall out of favor with the admin and you lose access to the good American models, aren't allowed to use Chinese ones, and fall prey to the attackers and behind your competitors.
lenerdenator 4 hours ago [-]
It'd be less about "safety" and more "we've spent trillions developing these AI tools only to have the Chinese, once again, copy them and offer them for pennies on the dollar, and no one seems to care about the impact that has on the long-term sustainability of this sector of the American economy as a whole, so we're yanking the models."
jmye 2 hours ago [-]
"I'm going to take this box razor and make some really deep cuts around the middle of my face because my tech sector is too good and that's actually a bad thing because $foreigners."
lenerdenator 2 hours ago [-]
I'm not saying it's necessarily a good thing. I'm also not saying it's about foreigners at this point. It's about seeing a bet through. They've burned a metric crapload of capital on developing AI models and the infrastructure to host them. They want that money back and then some. Remember, the fine shareholders of OpenAI think that 100x returns just aren't reasonable and want that cap lifted. If this kind of thing continues, they'd be lucky to make their money back at all, let alone 100x.
Which would be fine, but as we know, people securitize the crap out of their investments these days, and least some people probably leveraged themselves on some US AI companies, so now the risk is spreading outside of the sector to the economy in general, which is made worse by the sheer amount of spending on AI.
aussiegreenie 5 hours ago [-]
The Americans may ban the use of the Chinese models in America. But like the Chinese car ban, everyone else will use them.
lenerdenator 4 hours ago [-]
That's not necessarily a good thing for everyone else, mind.
Yes, you get your free model, but the cost of this is not developing your own capability and tying your fate to a country which may or may not have your best interests as a nation in mind.
This is just the deindustrialization that occurred in my home region (the American Midwest) playing out on a global scale in different sectors. It was originally driven by the Japanese, who, to their credit, acted more as partners than competition. Eventually that desire for larger margins went to China, and now you basically can't build anything of consequence without at least some Chinese parts, because there's "no economic case" for it. This means that you have to play Beijing's game if you want access to any sort of modern market.
You see this happening with Volkswagen's restructuring, next you'll see it with non-American, non-Chinese AI.
singpolyma3 3 hours ago [-]
It's not really the same because we already have the model. If China stopped letting us have it tomorrow I'd doesn't matter because... We have it already
chillfox 2 hours ago [-]
So... how's that any different from using American stuff for those of us in the rest of the world?
Over the last decade, the US has been way more unreliable than China. There's been a near constant negative impact from the US doing something.
At least with China, we are very good at winning trade wars with them here in Australia.
lenerdenator 1 hours ago [-]
You might feel differently if you were a Filipino or Vietnamese fisherman whose family relied on the income from the stocks of the South China Sea, or a Uighur person living in Western China, or a Ukrainian soldier who has to deal with drones built with Chinese components, or a democracy advocate in Hong Kong, or arguably, a person who had plans for 2020-2021.
Or, on a more local note, an Australian automotive worker who worked for a company that figured out 10 years ago that they wouldn't be able to pay him a decent wage, compete with the then-upcoming Chinese EVs, and remain profitable.
Paradigm2020 1 hours ago [-]
You might feel different if you're a palestinian who's getting american bombs dropped on him, or an afghani collateral damage or...
There is no good guys in general, and whataboutism and making the scope bigger doesn't help.
The thing is that if the models you are building on are open source whether hosted on chinese / american / whatever service at least give you an option to switch provider easier vs a fable / chatgpt 5.6 that gets banned for none americans etc...
2 years ago america would have had the branding/perception advantage but right now that is well and truly gone...
skissane 4 hours ago [-]
> GLM export controls incoming? I predict Commerce will force OpenRouter, HuggingFace to take some open models down within the next few months.
I’m sceptical they could find the legal framework to do this even if they wanted to
They have legal authority to (a) prevent export of US goods/services; (b) ban imports of physical goods; (c) ban transactions (including purchasing services or license agreements) with foreign firms
But I’m not aware of any legal authority which lets them ban US firms from running a Chinese-developed open source AI model in the United States, if they are at arms length from the vendor, and aren’t using it for government contracts or regulated applications
Possibly they could order HuggingFace/etc to suspend Chinese accounts. But if someone in the US (or a third country) downloads the model from China then reuploads it to a US server, completely independently of the vendor - where is the legal hook to prohibit that?
bardak 4 hours ago [-]
They could ban payment processors from processing payments to any hosts of GML 5.2, despite the open weights the vast majority of people will be using cloud providers to get access since it is to heavy to host for 99% of people.
This would be extremely heavy handed and probably end up accelerating the loss of the virtual US monopoly of payment network. The reast of the world isn't going to let the US dictate that only they get the frontier models whether their US made or otherwise
skissane 4 hours ago [-]
> They could ban payment processors from processing payments to any hosts of GML 5.2
Can they actually though? Do they have legal authority to tell a payment processor that it has to block transactions of a legal US company, just because the company is hosting a Chinese-developed open source model? I’m sceptical
And what about companies (e.g. AWS) that let you “bring your own model”?
bardak 4 hours ago [-]
It would be extremely heavy handed but the administration has sanctioned the International Criminal Court judges such that they basically have no access to the Wests modern financial system. I think domestic US providers would have to deal with different ways but someone like Herzner could easily be cut off from the financial system if the administration doesn't feel that they are adequately blocking the model
skissane 2 hours ago [-]
> It would be extremely heavy handed but the administration has sanctioned the International Criminal Court judges
That's sanctioning specific individuals for specific acts they performed which the US claims contravene its interests and those of its allies.
I don't agree with the ICC sanctions, but it really can't be compared with the proposal "sanction any company, even US domestic entities, which use a Chinese-developed open source model".
In fact, I think part of what enables the US to sanction them (under US law) is the fact they are neither US citizens nor residents; if they were US citizens living in the United States, I don't think the President would have the legal authority to impose those kinds of sanctions.
They could sanction Hetzner–because it is a German firm based in Germany. I don't see how they could sanction a US firm based in the US whose owners and staff were US citizens.
Also, the 5th Circuit Court of Appeal decision Van Loon v Treasury (Nov 2024) is relevant–it held that IEEPA (the law used to sanction ICC officials) couldn't be used to sanction the Tornado Cash smart contract system, since open source code wasn't "foreign property" under IEEPA.
phs318u 3 hours ago [-]
Swapping the footgun for a huge long-range boomerang doesn’t mean it’s not going to eventually swing around and whack you in the back of the head.
bardak 3 hours ago [-]
100% agree and don't think it will come to that but I won't completely put it past this administration
addandsubtract 3 hours ago [-]
Label AI as porn and the payment processors will cut their ties automatically.
mrandish 4 hours ago [-]
> I’m sceptical they could find the legal framework to do this even if they wanted to
I agree, my only caveat is that the current administration has shown it's willing to go beyond aggressive regulatory interpretations to questionable and outright implausible interpretations. As we've seen recently, the federal courts and SCOTUS are overturning most of these but that can take a year or more to resolve. The one positive light is they seem to push the hardest on certain culture war issues (immigration, voting, districting, etc). AI doesn't seem like a core hot button issue for the White House and there is a strong pro-AI / business faction.
eunos 3 hours ago [-]
OpenRouter or Huggingface should consider moving to Switzerland
gruez 7 hours ago [-]
>GLM export controls incoming?
US imposing export restrictions on a model from China?
mcintyre1994 6 hours ago [-]
It’d be restrictions on Americans and American companies, and probably also pressure on America’s allies.
mkagenius 5 hours ago [-]
Token smuggler sounds like a profession coming soon. For distillation and stuff.
addandsubtract 3 hours ago [-]
I mean, there are already places where you can buy tokens at 10% of their original cost.
manquer 6 hours ago [-]
While unlikely , it is not without precedent , there are restrictions on ASML a Dutch company to sell EUV machines
throwup238 5 hours ago [-]
That’s because the Department of Energy originally funded and contributed IP to the EUV Corp joint venture between several semiconductor companies (including ASML and Intel). Their ability to export control EUV was part of that original agreement that the entire technology is built on.
1 hours ago [-]
verdverm 6 hours ago [-]
ASML complies as an ally, why would China comply?
The weights are already available and downloaded, is it going to be a crime to have them, run them, make them available? Constitutional rights still exist (I hope)
solenoid0937 6 hours ago [-]
> is it going to be a crime to have them, run them, make them available?
Now you're getting it! Commerce will call it a munition and those harboring it as harboring illegal/foreign munitions.
No business will take the hit, so they will quickly deplatform the models.
No end user has the GPU capacity to use GLM 5.2 or similar models at full precision so the government will call the problem "mostly solved." But they might choose to "make examples" out of a few people using p2p software to download the weights if they choose to.
verdverm 6 hours ago [-]
Or we use the models to work on fixing vulns and stop over-blowing the doom scenarios. Gotta save the kids and kill the terrorists though!
I'm for making software better instead of banning it based on what the rich and powerful claim.
I suspect the real fear is that open weight models undermine the financials and token prices they thought were going to pay off their ludicrous spending because they have all raced and raised hardware prices.
hadlock 5 hours ago [-]
> making software better instead of banning it
We're still in the middle of the cambrian explosion.
If Anthropic was capable of developing Opus 4.49-4.5 2H 2025.... then any company with a research team capable of reading all the papers and press releases will be capable of producing Opus 4.8 by the end of 2027, either raw model competency, or in a harness like claude code (or better with both). I guess what I am trying to say is that Opus 4.5 does not represent the edge of agentic capability, merely somewhere in the thick meaty layer of "functional and achievable".
We can draw the line at Sonnet 4.6 in the US but much like encryption export restrictions in the 1980s, the line drawn will be laughably low within a few years and simply unthinkable in a decade.
solenoid0937 6 hours ago [-]
> making software better instead of banning it
That would be the rational thing to do.
> financials and token prices
I do not think the government thinks this deeply. Market manipulation might be a rational, if unethical reason to ban open source models.
But this admin banned Anthropic models to "own the libs." They will continue to ban what they want for whatever reason they want. I don't think those reasons will be particularly coherent.
verdverm 5 hours ago [-]
Yeah, the current admin is reactionary, they appear to put little thought in, or at least disregard input they dislike. I don't think Ant's ban was about "owning the libs" as much as it was asserting dominance over someone who spoke up counter to the admin's aims and claims. They do listen to money, which is where I see Big Ai paying for executive orders (because the admin forgot what it means to compromise as part of legislating for all americans).
5 hours ago [-]
6 hours ago [-]
manquer 2 hours ago [-]
That too has precedence , there is long history of controls of cryptographic algorithms up until the 90s. It wasn't abstract either, older greybeards would remember browsers like Netscape had two versions International and U.S. for this reason.
If you classify AI as a weapon which seems to be the direction that we are all heading towards, they yes first amendment rights won't likely apply.
matheusmoreira 6 hours ago [-]
> it going to be a crime to have them, run them, make them available?
Yeah. Illegal numbers.
fragmede 4 hours ago [-]
DeCss was short enough to fit in a t-shirt. Americans are larger these days, but not by enough to fit a decent LLM's weights on an XXXXL shirt, even double sided.
Art9681 4 hours ago [-]
They can easily issue an order for any American company to stop hosting/serving the models. If the model was a threat to national security because of its capabilities then a lot of other countries would follow, including China. No nation will allow some vibe coder with a rogue AI to pose a threat to their systems.
The reason GLM-5.2 hasn't been banned is that despite these cherry picked use cases, GLM-5.2 isn't even close to Opus in all use cases. These vibe benchmarks are ran by companies that are not part of the cyber services offered by Anthropic and OpenAI where they can use the models without the safeguards and refusals so their actual cyber capabilities can be utilized.
These guys that wrote the article compared a gimped Opus to GLM-5.2, knew full well it's misleading, and got the clicks regardless. They don't have enough clout to be a part of something like Project Glasswing, GPT Cyber, etc.
fph 6 hours ago [-]
How would that even work for an open-weight model?
bardak 4 hours ago [-]
Go after the hosts, 99% of people won't be able to run this locally even if they wanted to.
djeastm 5 hours ago [-]
I think state-of-the-art AI is going to be defense industry only from now on. We can have our toy drones but not the Predators and Reapers.
Gigachad 5 hours ago [-]
Turns out toy drones are more useful in war than multi million dollar planes anyway.
techpression 5 hours ago [-]
Reaper and Predator are both drones and there’s really no comparison to toy drones in terms of sheer destruction and capabilities in general, the comparison is actually quite apt imo.
solenoid0937 1 hours ago [-]
You're right. Toy drones have proven vastly more effective IRL.
The others are a waste of taxpayer money. Extraordinarily low return on investment (kill-on-investment?)
fragmede 4 hours ago [-]
Which ones are the ones Ukraine has used to bomb Moscow?
4 hours ago [-]
serf 5 hours ago [-]
the things that empower modern toy drones were export restricted for years before hand.
mullingitover 3 hours ago [-]
Obvious answer: build all your open source LLMs into firearms, get the SC to grant 2A protections.
dakolli 5 hours ago [-]
Cool then everyone will just change their config to route through a provider overseas for an added 50-100ms latency. Who cares.
solenoid0937 1 hours ago [-]
Countries and businesses that don't want to be sanctioned by the US government or the US financial system care - so all western countries and corporations.
4 hours ago [-]
jackdawed 3 hours ago [-]
I use GLM 5.2 via Neuralwatt and it's gotten so cheap I wouldn't mind cancelling my personal Claude subscription if work gave me one. I've spent 374M tokens this month and it only cost me $18 on energy-based pricing.
cmrdporcupine 23 minutes ago [-]
How's the reliability and speed?
croemer 2 hours ago [-]
They should also at least run Opus through the same Pydantic harness they used for GLM. As is, it's apples vs pears.
Where's the cost per vulnerability for all the other models than GLM?
Also, without code this isn't very trustworthy. Could all be made up as well.
danslo 7 hours ago [-]
It reads like an ad.
Secondly these are "just" IDORs, arguably the easiest class of vulnerabilities.
Thirdly it compares to GPT 5.5 and Opus 4.8.
No, we don't have Mythos at home.
vlian2088 6 hours ago [-]
>Thirdly it compares to GPT 5.5
mythos is <10% ahead of gpt 5.5 on all benchmarks, which it gains by being several times the size of opus. had it been economical to provide, it would've been released to the public on day one instead of the marketing circus those effective altruism clowns had exhibited. admitting that it costs >1000% to run inference on a <10% better model would've been very damning.
oa335 4 hours ago [-]
> it costs >1000% to run inference
do you have a source for this claim? i thought LLM providers earn high margins from inference (charged by token). is this no longer the case?
vlian2088 4 hours ago [-]
if a $6000000 cabinet can generate 10000/s tokens of Opus but only 1000/s tokens of Mythos, then Mythos costs 1000% to run no matter the markup.
no one has a source, because no one knows closed model parameter counts. we have only heuristics which strongly indicate that Mythos is simply a big fucking model that any other lab could make an equivalent of.
3836293648 4 hours ago [-]
This was just theorised. The leaked OpenAI financials suggest otherwise (because of shady naming of losses)
The only ones who seem to profit are the ones running smaller Chinese models. Even NVIDIA seems to have to "reinvest" their profits into sponsoring companies to buy their cards now.
InsideOutSanta 6 hours ago [-]
In my experience, GLM 5.2 is extremely good at finding vulnerabilities, and more importantly, unlike Opus, I've never seen it refuse a command. It genuinely is a very strong model for finding and fixing vulnerabilities.
nozzlegear 3 hours ago [-]
More importantly, unlike Mythos and Fable, you can actually use GLM 5.2! It's not just marketingware that got its founder in hot water with the government.
NitpickLawyer 5 hours ago [-]
> Thirdly it compares to GPT 5.5 and Opus 4.8.
> No, we don't have Mythos at home.
That's still useful. To paraphrase the kids these days, GLM5.2 is in the room with us, today. Mythos is not. And for us in the EU, it's even more complicated, as Mythos might be with us in the room one day, and go poof the next day, on the whims of political entities that we have 0 control over.
Knowing where open, accessible, local models are is important. We know they're behind. But there comes a time when "good enough" is useful. Even if they're "just IDORs" today, and even if they're behind SotA today.
As someone else said above, GLM5.2 (and other models in the same tier like kimi, dsv4, etc) is / are slowly becoming "good enough" to assist in automated repo prepare work (download, install, test, edit, re-test, etc). And that translates in RL traces ready to be trained into the next generations. That might be more important than x% behind on benchmarks.
sanid 5 hours ago [-]
Technically we don't have Mythos at all? You guys have access. This tells me we have Opus at home (open weights).
jimbob45 5 hours ago [-]
Yeah they straight up say that their criteria is narrow and primarily important for their specific use case. Never let rationality cause your pitchfork to be cast away though!
46 minutes ago [-]
g42gregory 5 hours ago [-]
If only the "cybersecurity" crowd were focused on patching the vulnerabilities.
Instead of shilling for the LLM providers.
__MatrixMan__ 5 hours ago [-]
But if we patch all of the vulnerabilities, who will pay for our vulnerability scanner?
_factor 5 hours ago [-]
The robot figured out how to bump the lock. The obvious solution is to ban the robot.
4 hours ago [-]
_s_a_m_ 4 hours ago [-]
I tried GLM many times and it is bad, i have on clue what these people are talking about
jeffnash 3 hours ago [-]
have you tried 5.2? I agree that 5.1 and prior were below Kimi, Mimo, Qwen, Minimax, and probably Deepseek (depending on task), but 5.2 (especially unquantized) feels like something else.
Now I feel like that I'm covered by GLM 5.2 and Minimax M3 (when I need vision or a second pass on something).
throw10920 2 hours ago [-]
Bad for security research or for general coding?
Having used GLM 5.2 for non-security software work, I can say it's better than Sonnet (but not Opus), and cheaper than both (because when you steal someone else's IP, you don't have to amortize the cost of their R&D).
theteapot 5 hours ago [-]
> Constant: the IDOR dataset (the same real, open-source applications we've used in prior research) ...
What we're they? Also, wouldn't one expect a more recently released coding agent (with a more recent knowledge cut off) to perform better because they have access to more knowledge about vulns in these OSS projects, and even possibly have knowledge of your own "prior research"?
mkagenius 4 hours ago [-]
One would. But then the results are even weirder as opus 4.6 scored more than opus 4.8 by a huge margin
admax88qqq 6 hours ago [-]
> beats Claude in our Cyber Benchmarks
Beats which model in Claude? Whenever a "benchmark" doesn't put precise model numbers in their headlines I am immediately skeptical. Either they don't know the difference (bad) or they are benchmarking against weaker models (misleading, also bad).
It's like when studies say "AI is bad at X" and they used GPT-3.5 in current year.
InsideOutSanta 6 hours ago [-]
They say "Claude Opus 4.8" in the first paragraph.
crm9125 5 hours ago [-]
We're supposed to read the article?
How are we supposed to stay skeptical of everything if we read anything!?
ls612 6 hours ago [-]
Opus 4.8 according to TFA. Whether or not the safety guardrails were responsible for the difference is an open question but for a dev who wants to secure their software who doesn’t work at one of the blessed Glasswing companies it doesn’t really matter why, it matters what the best tool you actually have is.
Alien1Being 13 minutes ago [-]
The current US administration has gone a long way towards handing over leadership in AI to China.
veselin 7 hours ago [-]
Here, it appears they compare a single prompt "find IDOR", against a multi-agent system. However, one can also start far more sophisticated skills that spin up subagents and mostly do the same in Claude Code, Codex, OpenCode, Pi, etc.
Which I guess makes what semgrep sells obsolete. Unless they have built a pareto-optimal point in terms of capabilities and token usage maybe?
blazespin 7 hours ago [-]
I think the point is less "how can we throw shade on the OP" and more "a harness can enable a lot of models to do very serious cybersec, glm 5.2 is one of them"
s3p 6 hours ago [-]
Are you replying to a response to the original comment? I looked but i didn't see anyone saying he's throwing shade.
BikiniPrince 5 hours ago [-]
You have to forgive the GLM bot. It's not very good.
I think they give $5 trail credits to test with any of the open weight models.
Art9681 4 hours ago [-]
This is because of the safeguards and not the model capabilities. If these folks signed up for the proper cyber service offered by Anthropic where refusals are removed then the open weight model wouldn't look as capable.
3 hours ago [-]
rvz 1 hours ago [-]
Many people here are now realizing that open weight models are now able to compete against frontier closed models.
This is where we are heading and why many closed labs are terrified of this affecting their bottom line and the reason why they want them banned from being released.
crazylogger 1 hours ago [-]
Actually they don't even need to compete against frontier closed models, they just need to work.
99.99% people's day jobs aren't competing for the Fields Medal or even finding security vulnerabilities. So it appears while TAM (total addressable market) of AI in general is huge, TAM for frontier LLMs is tiny. Efficiency gains at roughly the same performance might be all people care about from now on.
dist-epoch 5 hours ago [-]
Anthropic is saying other models were good at detecting vulnerabilities, where Mythos excelled was in creating functional exploits for them.
This article only talks about detecting vulnerabilities, so it's unclear if it's a true Mythos equivalent.
igregoryca 4 hours ago [-]
It seems "Mythos is really good at finding vulnerabilities" has been what people took away from the Project Glassing announcement, which makes sense. Unfortunately for Anthropic, most seem to have forgotten the best argument Anthropic had for holding Mythos back from the general public, "it's crazy good at crafting exploits". Then, without that context, the tinfoil hats came out.
laybak 4 hours ago [-]
how representative are Semgrep's benchmarks? everyone seems to have their own benchmark these days (guess it's good "content marketing") I'm honestly losing track
slashdave 3 hours ago [-]
Advertisement
dools 3 hours ago [-]
I think Opus 4.8 is deliberately nobbled. Kimi k2.6 with Kimi code beats opus models at finding vulnerabilities, even though it produces some false positives, when I give the same issues to opus and ask it to verify most of the time it concurs it’s a real issue even though it failed to find the issue itself
cmrdporcupine 4 hours ago [-]
I like GLM 5.2... ish. It's ok.
I'd be mostly fine switching to it.
I just can't find a cost effective way to do that. z.AI's coding plan is both overpriced and unreliable. ollama's is also overpriced. Paying by the token for it on openrouter etc is more expensive than just having a Codex or Claude coding plan.
If you have to pay by the token, it's clearly cheaper. It's not competitive with a coding plan though.
TurdF3rguson 4 hours ago [-]
It also means giving up vision which I don't know how I would deal with. I think I would prefer a weaker model with vision than a stronger without.
maxk42 2 hours ago [-]
Openrouter definitely supports vision models. Why would you have to give up vision?
TurdF3rguson 59 minutes ago [-]
For example if I want to paste a screenshot of what I mean, I can't.
gmerc 2 hours ago [-]
vision runs just fine locally for most usecases, so it's really just a skill to call that Ollama instance
cmrdporcupine 4 hours ago [-]
If you using opencode or similar you can just temporarily switch models -- in the same session -- to something that has vision and have it look at your image. And then switch back.
gazpachotron 3 hours ago [-]
[dead]
nozzlegear 3 hours ago [-]
Why's that?
lowbloodsugar 2 hours ago [-]
Felt like I was reading advertising for their harness.
utunga 3 hours ago [-]
Just popping in to say that no you can't use the word "tokenomics" to mean that. Argh.
lenerdenator 4 hours ago [-]
The incentive to develop Claude further is to make money.
The incentive to develop these Chinese models further is to trash the business case of most American AI labs.
csjh 4 hours ago [-]
I found it to spiral into complete nonsense a few times when I tested it out, but it's possible that was a bug in the provider
CurbStomper 2 hours ago [-]
people still using Claude?
yieldcrv 3 hours ago [-]
who is your favorite hosted GLM 5.2 provider? I'm looking for fastest tokens/sec and best cost
additionally, reliable API, because z.ai can be finicky
also, not for Enterprise use, but I like non-US providers, I don't care if the party happens to be the one reading my information and stealing my trade secrets, if they won't respond to a US subpoena
TacticalCoder 4 hours ago [-]
How to reconcile that with the recent, highly upvoted, article titled: "The gap between open weights LLMs and closed source LLMs"?
What explains it?
Is TFA lying? Is the most upvoted comment here lying?
aussinholdn 4 hours ago [-]
[dead]
rode1974 5 hours ago [-]
Hopefully i get a macbook pro soon enough to run some small or medium sized LLMs
paperterminal 5 hours ago [-]
Same, but so much $$
5 hours ago [-]
BikiniPrince 5 hours ago [-]
This is a joke right? I wouldn't install this in a sandbox.
mlnj 2 hours ago [-]
Why? Don't tell me you've never tried a non-US based model, ever.
There's a number of US providers who also run it, if that is your preference.
This weekend I programmed a matrix bot with encryption and a Rust agent with some tools. Because I need one and OpenClaw just felt... not what I wanted. Two days later and 20 dollars poorer I have what I need: a multimodal agent written in rust that has access to my homelab.
Nothing felt off with GLM. It did what I wanted, was fast, had a decent not very annoying personality and was much cheaper than Opus or GPT.
I used it unquantized through Fireworks, but there are multiple other providers too.
In multi-agent coding environments, GLM 5.2 is just shy of Opus 4.6 on average. Data at https://gertlabs.com/rankings
But when factoring in performance/cost, GLM 5.2 is the frontier model.
Some blog post I read few weeks back said that DSV4Flash in xHigh effort beats even the pro model in xHigh effort.
I think the surprising thing is I expect flash to be a pure distillation and strictly worse quality but clearly it’s more nuanced than that.
When I first used Opus 4.8, I threw several different workloads I had at it - I have Claude doing a lot of misc projects whose primary purpose is pretty much just studying what AI agents can do for my own curiosity and no other reason. Opus 4.8 was one of the first models I ever snuck in there that basically ran out of control. No previous Opus or Sonnet model I had used ever did this. Within hours every agent I had running was writing non-sense tool calls that echoed pretend commands that didn't exist, like 10 in a row, and talking about the "tool channel" being dirty. I switched back to Opus 4.7 and assumed Opus 4.8 was legitimately just broken.
I did come back to Opus 4.8 and found that it was indeed, pretty powerful. But that initial experience has stuck with me on just how narrow of a perspective any given test or benchmark is guaranteed to have. LLMs are too broad, it really doesn't matter what you try to do in your benchmark, you will necessarily get a limited view of what the model is capable of and its shortcomings. This will remain true for at least as long as models are susceptible to massive swings in performance based on randomness and minor differences in prompts and other environmental factors.
I'm not saying benchmarks are useless or that your benchmarks are not possibly closer to the truth either. All evidence at least points to the idea that Chinese models perform very well in coding but often have more mixed results on other tasks. I'm just saying that at this point, benchmarks feel like they have limited connection to my actual real experiences. GLM 5.2 actually scored kinda meh on a lot of benchmarks (compared to closed frontier models) but my actual experience using it does not match this.
And I'm definitely not saying GLM 5.2 is better than the frontier LLMs here, just that the race is close. I still prefer GPT 5.5 right now for code review, I think, and Opus clearly has some advantages depending on the task. It's just no longer a given that Opus 4.8 will perform better than GLM 5.2 on any given task, so to me the calculus behind "using the best model available" is getting complex and you might need to get a feel for what models have what strengths to really figure it out.
I do feel like the "use the best model available" mentality is not going to die any time soon, but if it does die, it will be gradual and start soon for programming. Modern LLMs are still not a full superset of what human programmers can do, but still larger models are definitely starting to hit diminishing returns for tasks at the lower end of complexity, and that is a big deal. It's a weird world where some tasks you can feel kinda confident just throwing Gemma 4 at it and not sweating whether you should use a better model; I've certainly done it for some quick Python scripts or getting an overview of some code I'm unfamiliar with.
Notice he's using "trust me bro" benchmarks.
Can we just remove all the motivated speech on HN? This is just not trustworthy information at all and obviously is incentivized.
Everyone is grinding and marketing nobody is actually discussing anything for real.
Anthropic won't even let you run "claude -p [prompt]" any more... They bill it at api rates.
So if you're trying to automate the ai (and seriously, that's the point) the subsidized plans are crippled.
> In May, we sent you an email announcing that starting today, the Claude Agent SDK, claude -p, and third-party apps built on the Agent SDK would stop drawing from subscription rate limits and move to a dedicated monthly credit. We're writing to let you know that we’re not making this change today. We’re working to update the plan to better support how users build with Claude subscriptions.
> What this means for you
> Nothing changes for now. Agent SDK, claude -p, and third-party app usage continues to work with your subscription exactly as it did before today, and there's no credit to claim. Your subscription limits are unchanged. When we have an update, we'll share it with advance notice before it takes effect
So your question is really “if they’re giving free usage, why not take advantage of it?”
I do, so I don’t know the reasons not to, other than to experiment.
I kind of wanted to see if I can make a Matrix agent from scratch with Rust with GLM and it was surprisingly easy. Just make something for myself how I want it. Maybe I'll take a look on Hermes later...
I don't think a $100 session is "typical". I use GPT for months. $20/m plus plan is enough for my daily work.
My typical session ranges from $100-$400 - higher end when using workflows with lots of subagents. $100/session is expected when using the API without the subsidized subscription pricing. Most larger orgs have to use API pricing AFAIK.
[1] https://github.com/simple10/agents-observe
Likewise, I don't understand how anyone survives on the basic plans. It's funny seeing these two camps not understanding what the other is doing :)
I'd blow through $20/month plan in hours.
How are you comfortable spending that much to write something as simple as a matrix bot?
Are people doing this kind of thing just super rich or am I missing something?
Same thing for stuff at work. Oh, the tables/schema changed and my queries broke? I could dork around with spark and cypher for an hour, or I can tell claude to update the queries for the new schema. At the rate I am paid, spending on Claude tokens is generally a better use of my resources.
Building a net new solution? Coding tools take a back seat until I get the core logic right, then I let automation handle web page and UI scaffolding.
https://swelljoe.com/post/will-it-mythos/
Also of note, I found giving models access to the open source semgrep as a tool makes some perform worse and none perform better, though it's plausible there's a way to wire it up in a harness that presents useful information to the model without the model having to know how to use it (my theory is that semgrep isn't heavily represented in the training data, so you're asking the model to do two things at once: figure out how to use semgrep and find security bugs, and both tasks suffer for the lack of focus...most small models, and some big models, can't do that well).
Edit: But, also, more testing is ongoing. I suspect GLM 5.2 will also be a consistently strong performer. It seems to excel at most things I've tested on it.
Overall, I still think GLM 5.2 is the much stronger performer. It's hard to tell the difference between GLM 5.2 and Opus at <120k tokens.
I'm inclined to use DeepSeek V4 Pro the most, because it is consistently extremely strong, it's very fast, it's very cheap and has excellent caching and cheap-as-free cached input tokens (something like 80% of token usage is cached when I'm using it for security scanning). So, my probably "pair" of frontline security researchers will probably be DeepSeek V4 Pro and Gemma 4 31B self-hosted (another shockingly strong contender, competitive with the best models once you let it loop on the same file a couple/few times). But, I won't be surprised if GLM 5.2 turns out better than DeepSeek V4 Pro...it costs quite a bit more.
…probably already is one
I like reading benchmarks, but I take them all with a grain of salt. They're just to tell me if the model is worth even trying for my task. I've heavily used self-hosted Qwen 3.6 and Gemma 4 on a bunch of different tasks, and while the benchmarks consistently say Qwen is the better model, I simply don't find that to be the case for anything I do. I think Qwen is tuned for benchmarks, while Google couldn't give two shits about most of the benchmarks, they're just busy making unusually smart tiny models.
But, there are benchmarks for so many different kinds of ability, I don't know how to compare them directly against one another. Like, models that do well on terminal and agentic coding benchmarks tend to do well on finding security bugs, but it's not a 1:1 correlation, there are surprises.
[1] https://huggingface.co/zai-org/GLM-5.2
A few people are running highly quantized models with limited context windows. It's still impressive, but not the benchmark level intelligence. Very few people could afford a rig for reasonable local performance at a reasonable quant, at full context size.
The antirez example is 2.6bit quant, 32k context, and few tokens per second... on a ~$7000 MacBook M5 (new RAM pricing).
Don't worry though, open source evangelists will tell you that these will be running on your phone in the next 3 years.
For $100k you could run this model 24/7 through open router with 10 concurrent sessions at 50tps for a decade and have money left over for a vacation. There's no point in investing this type of money in local models unless you have a business where you're already paying for many employee's individual token usage.
8 x RTX6000 GPUs cost $100,000 alone. You then need to build a system that can support those GPUs with enough PCIe lanes through a PCIe switch.
It's going to be $120K to $150K to build or buy a system to run this.
But hey you could save on heating?
A single circuit using 10mm TPS would technically be enough to run what you’re describing. Might be pricey though, I’d probably take the excuse to get 3 phase installed so I could get access to the stock of used 3 phase machinery.
Or even just electricity costs vs token cost
The real gangstas are running 16x RTX6000s. Too rich for my blood, and the NV4FP quant doesn't seem to be that much worse.
Not sure if you're being sarcastic, but I can run a quantised version of Gemma or Qwen on my 16GB M1 Macbook Pro that beats GPT-4 from 2023 hands-down.
I wouldn't be surprised if, in another 3 years, you'd be able to run something as powerful as Opus 4.5 or GLM-5.2 on standard consumer hardware - say a 32GB/64GB M7 Pro.
I also wouldn't be surprised if, 3 years after that, cheaper hardware and improved model efficiency means that there's a much smaller gap between what you can run on a consumer CPU (which, with memory prices coming down, could look like a 256GB M9 or M10 Pro) and $100k GPU cluster.
We've been sat with basically the same PC specs for ~20 years - our current specs are within an order of magnitude of the ones we could buy back in 2010. This is not really constrained by tech, as we could have much, much, larger machines. It's more because there's no mass demand for much, much, larger machines - if it's big enough to run Office apps or VSCode then you're good to go. The exponential growth we saw in the 90's was driven as much by software demand as it was by hardware development.
I can see the next 10 years produce the same kind of push for larger machines that the 90's did. And we should probably expect the same kind of standards churn as our existing technologies for storage, memory, etc, don't scale up enough and new technologies become worth developing because there's demand for them.
Personally, I’m waiting for hardware to hit the secondary market before I buy something to run unquantized models like GLM. But I have no doubt that I will, at some point.
oil workers buy 100k trucks they do not-much with. why not a 100k in computer?
Some, and the market fluctuates a ton.
> corvettes
Only the oldest, most unique model years: nobody is buying (C4-C5-realistically C6) mid-90s or early 2000s Corvettes for more than what they paid for them, and they never will.
Both of those things' value drops like a rock as soon as you buy them and, at least for cars, they don't all appreciate. Most don't. Even so, they appreciate at an incredible slow rate.
I can't speak for watches but I'd be surprised if it wasn't the same situation.
At least the gpus can create value after you buy them before they are worthless.
Isn’t the performance gap between quantized and full models indicative that even if you aren’t using it directly, the model knowing the colors in the Russian flag does have something to do with the intelligence you demand?
The compression is almost certainly in part specific knowledge getting fuzzed.
Likewise, LLMs do not violate the laws of information theory, and therefore the only way to encode X amount of information in Y amount of bits where X > Y is by performing what is effectively lossy compression, and as X grows larger relative to Y the compression ratio must change to lose ever more information.
Yes, for the sake of making chatbots that are "conversational" in that they can interpret natural language as input and produce code as output you can easily benefit in incidental and unintuitive ways by training it on more natural language text. But for a given fixed parameter size, it's possible to produce a better model for a specific task by selectively not muddying its training set in the first place with things that are likely irrelevant to the task.
We don’t understand AI or natural intelligence well enough to make such statements. As for self evidence, cross-domain competence in humans and the rise of generalist models over domain-specific ones (on competence, not cost) seems to pretty directly tank your hypothesis.
It's hardly self-evident, and your counter-example is hardly applicable.
The first 10^50 of pi is not the same as having BREADTH of information in the training data, which is the whole point not just any random "information that is irrelevant to your use case".
not to mention that the first 10^50 digits of pi compress to quite small formula, so not much information there to begin with from a shannon/kolmogorov perspective
assuming demand doesn't keep on increasing. even google has trouble having enough capacity apparently.
GLM 5.2 is already capable enough to assist in self-training which is similar to what we saw happen with frontier models and they appear to be getting there at a significantly lower cost than openai/anthropic.
Care to give more context to this? Seems interesting
Mythos level really doesn't seem that scary. And it would be a great way to take away the American labs international market.
I think it would make strategic sense for them to release more capable models than what American labs are allowed to make available to the world. It would help them grow their global soft-power and be a destabilizing effect on the American economy.
Claude Code is an agent harness, not an LLM.
Claude is a brand (or group of LLMs), not an LLM.
wild guess - I wouldn't be surprised if Opus 4.6 was run quantized for a while, and 4.7/4.8 have QAT for that nerfed size.
Oddly, this is a strong indication of the text being hand-written rather than LLM-assisted; it's very likely that a human made a mistake in creating the table.
[0]: https://aibenchy.com/compare/anthropic-claude-opus-4-8-mediu...
https://aibenchy.com/compare/google-gemini-3-1-flash-lite-hi...
It depends a lot on the task and harness too (using plans and to-do lists, vs one-shot answers), but for simply answering directly to an inquiry, often extra thinking doesn't necessarily improve the answer, especially if the answer is binary, or can be correct or wrong, as opposed to having more time to refine a creative output.
Not that it would make any sense.
Any prohibition on open source models will do nothing to fix the problem.. since attackers will never feel bound to the law. All advanced models must be available for defensive purposes.
If the real motive is profit, then open source models are likely simply not a viable means to that end.
But that's the whole point.
Fall out of favor with the admin and you lose access to the good American models, aren't allowed to use Chinese ones, and fall prey to the attackers and behind your competitors.
Which would be fine, but as we know, people securitize the crap out of their investments these days, and least some people probably leveraged themselves on some US AI companies, so now the risk is spreading outside of the sector to the economy in general, which is made worse by the sheer amount of spending on AI.
Yes, you get your free model, but the cost of this is not developing your own capability and tying your fate to a country which may or may not have your best interests as a nation in mind.
This is just the deindustrialization that occurred in my home region (the American Midwest) playing out on a global scale in different sectors. It was originally driven by the Japanese, who, to their credit, acted more as partners than competition. Eventually that desire for larger margins went to China, and now you basically can't build anything of consequence without at least some Chinese parts, because there's "no economic case" for it. This means that you have to play Beijing's game if you want access to any sort of modern market.
You see this happening with Volkswagen's restructuring, next you'll see it with non-American, non-Chinese AI.
Over the last decade, the US has been way more unreliable than China. There's been a near constant negative impact from the US doing something.
At least with China, we are very good at winning trade wars with them here in Australia.
Or, on a more local note, an Australian automotive worker who worked for a company that figured out 10 years ago that they wouldn't be able to pay him a decent wage, compete with the then-upcoming Chinese EVs, and remain profitable.
There is no good guys in general, and whataboutism and making the scope bigger doesn't help.
The thing is that if the models you are building on are open source whether hosted on chinese / american / whatever service at least give you an option to switch provider easier vs a fable / chatgpt 5.6 that gets banned for none americans etc...
2 years ago america would have had the branding/perception advantage but right now that is well and truly gone...
I’m sceptical they could find the legal framework to do this even if they wanted to
They have legal authority to (a) prevent export of US goods/services; (b) ban imports of physical goods; (c) ban transactions (including purchasing services or license agreements) with foreign firms
But I’m not aware of any legal authority which lets them ban US firms from running a Chinese-developed open source AI model in the United States, if they are at arms length from the vendor, and aren’t using it for government contracts or regulated applications
Possibly they could order HuggingFace/etc to suspend Chinese accounts. But if someone in the US (or a third country) downloads the model from China then reuploads it to a US server, completely independently of the vendor - where is the legal hook to prohibit that?
This would be extremely heavy handed and probably end up accelerating the loss of the virtual US monopoly of payment network. The reast of the world isn't going to let the US dictate that only they get the frontier models whether their US made or otherwise
Can they actually though? Do they have legal authority to tell a payment processor that it has to block transactions of a legal US company, just because the company is hosting a Chinese-developed open source model? I’m sceptical
And what about companies (e.g. AWS) that let you “bring your own model”?
That's sanctioning specific individuals for specific acts they performed which the US claims contravene its interests and those of its allies.
I don't agree with the ICC sanctions, but it really can't be compared with the proposal "sanction any company, even US domestic entities, which use a Chinese-developed open source model".
In fact, I think part of what enables the US to sanction them (under US law) is the fact they are neither US citizens nor residents; if they were US citizens living in the United States, I don't think the President would have the legal authority to impose those kinds of sanctions.
They could sanction Hetzner–because it is a German firm based in Germany. I don't see how they could sanction a US firm based in the US whose owners and staff were US citizens.
Also, the 5th Circuit Court of Appeal decision Van Loon v Treasury (Nov 2024) is relevant–it held that IEEPA (the law used to sanction ICC officials) couldn't be used to sanction the Tornado Cash smart contract system, since open source code wasn't "foreign property" under IEEPA.
I agree, my only caveat is that the current administration has shown it's willing to go beyond aggressive regulatory interpretations to questionable and outright implausible interpretations. As we've seen recently, the federal courts and SCOTUS are overturning most of these but that can take a year or more to resolve. The one positive light is they seem to push the hardest on certain culture war issues (immigration, voting, districting, etc). AI doesn't seem like a core hot button issue for the White House and there is a strong pro-AI / business faction.
US imposing export restrictions on a model from China?
The weights are already available and downloaded, is it going to be a crime to have them, run them, make them available? Constitutional rights still exist (I hope)
Now you're getting it! Commerce will call it a munition and those harboring it as harboring illegal/foreign munitions.
No business will take the hit, so they will quickly deplatform the models.
No end user has the GPU capacity to use GLM 5.2 or similar models at full precision so the government will call the problem "mostly solved." But they might choose to "make examples" out of a few people using p2p software to download the weights if they choose to.
I'm for making software better instead of banning it based on what the rich and powerful claim.
I suspect the real fear is that open weight models undermine the financials and token prices they thought were going to pay off their ludicrous spending because they have all raced and raised hardware prices.
We're still in the middle of the cambrian explosion.
If Anthropic was capable of developing Opus 4.49-4.5 2H 2025.... then any company with a research team capable of reading all the papers and press releases will be capable of producing Opus 4.8 by the end of 2027, either raw model competency, or in a harness like claude code (or better with both). I guess what I am trying to say is that Opus 4.5 does not represent the edge of agentic capability, merely somewhere in the thick meaty layer of "functional and achievable".
We can draw the line at Sonnet 4.6 in the US but much like encryption export restrictions in the 1980s, the line drawn will be laughably low within a few years and simply unthinkable in a decade.
That would be the rational thing to do.
> financials and token prices
I do not think the government thinks this deeply. Market manipulation might be a rational, if unethical reason to ban open source models.
But this admin banned Anthropic models to "own the libs." They will continue to ban what they want for whatever reason they want. I don't think those reasons will be particularly coherent.
If you classify AI as a weapon which seems to be the direction that we are all heading towards, they yes first amendment rights won't likely apply.
Yeah. Illegal numbers.
The reason GLM-5.2 hasn't been banned is that despite these cherry picked use cases, GLM-5.2 isn't even close to Opus in all use cases. These vibe benchmarks are ran by companies that are not part of the cyber services offered by Anthropic and OpenAI where they can use the models without the safeguards and refusals so their actual cyber capabilities can be utilized.
These guys that wrote the article compared a gimped Opus to GLM-5.2, knew full well it's misleading, and got the clicks regardless. They don't have enough clout to be a part of something like Project Glasswing, GPT Cyber, etc.
The others are a waste of taxpayer money. Extraordinarily low return on investment (kill-on-investment?)
Where's the cost per vulnerability for all the other models than GLM?
Also, without code this isn't very trustworthy. Could all be made up as well.
Secondly these are "just" IDORs, arguably the easiest class of vulnerabilities.
Thirdly it compares to GPT 5.5 and Opus 4.8.
No, we don't have Mythos at home.
mythos is <10% ahead of gpt 5.5 on all benchmarks, which it gains by being several times the size of opus. had it been economical to provide, it would've been released to the public on day one instead of the marketing circus those effective altruism clowns had exhibited. admitting that it costs >1000% to run inference on a <10% better model would've been very damning.
do you have a source for this claim? i thought LLM providers earn high margins from inference (charged by token). is this no longer the case?
no one has a source, because no one knows closed model parameter counts. we have only heuristics which strongly indicate that Mythos is simply a big fucking model that any other lab could make an equivalent of.
The only ones who seem to profit are the ones running smaller Chinese models. Even NVIDIA seems to have to "reinvest" their profits into sponsoring companies to buy their cards now.
> No, we don't have Mythos at home.
That's still useful. To paraphrase the kids these days, GLM5.2 is in the room with us, today. Mythos is not. And for us in the EU, it's even more complicated, as Mythos might be with us in the room one day, and go poof the next day, on the whims of political entities that we have 0 control over.
Knowing where open, accessible, local models are is important. We know they're behind. But there comes a time when "good enough" is useful. Even if they're "just IDORs" today, and even if they're behind SotA today.
As someone else said above, GLM5.2 (and other models in the same tier like kimi, dsv4, etc) is / are slowly becoming "good enough" to assist in automated repo prepare work (download, install, test, edit, re-test, etc). And that translates in RL traces ready to be trained into the next generations. That might be more important than x% behind on benchmarks.
Instead of shilling for the LLM providers.
Now I feel like that I'm covered by GLM 5.2 and Minimax M3 (when I need vision or a second pass on something).
Having used GLM 5.2 for non-security software work, I can say it's better than Sonnet (but not Opus), and cheaper than both (because when you steal someone else's IP, you don't have to amortize the cost of their R&D).
What we're they? Also, wouldn't one expect a more recently released coding agent (with a more recent knowledge cut off) to perform better because they have access to more knowledge about vulns in these OSS projects, and even possibly have knowledge of your own "prior research"?
Beats which model in Claude? Whenever a "benchmark" doesn't put precise model numbers in their headlines I am immediately skeptical. Either they don't know the difference (bad) or they are benchmarking against weaker models (misleading, also bad).
It's like when studies say "AI is bad at X" and they used GPT-3.5 in current year.
How are we supposed to stay skeptical of everything if we read anything!?
Which I guess makes what semgrep sells obsolete. Unless they have built a pareto-optimal point in terms of capabilities and token usage maybe?
After installing, do a `n8 build` to build the image, then `n8 --danger --provider opencode interactive` to launch it in a container.
Signup for GLM-5.2 here: https://z.ai
I think they give $5 trail credits to test with any of the open weight models.
This is where we are heading and why many closed labs are terrified of this affecting their bottom line and the reason why they want them banned from being released.
99.99% people's day jobs aren't competing for the Fields Medal or even finding security vulnerabilities. So it appears while TAM (total addressable market) of AI in general is huge, TAM for frontier LLMs is tiny. Efficiency gains at roughly the same performance might be all people care about from now on.
This article only talks about detecting vulnerabilities, so it's unclear if it's a true Mythos equivalent.
I'd be mostly fine switching to it.
I just can't find a cost effective way to do that. z.AI's coding plan is both overpriced and unreliable. ollama's is also overpriced. Paying by the token for it on openrouter etc is more expensive than just having a Codex or Claude coding plan.
If you have to pay by the token, it's clearly cheaper. It's not competitive with a coding plan though.
The incentive to develop these Chinese models further is to trash the business case of most American AI labs.
additionally, reliable API, because z.ai can be finicky
also, not for Enterprise use, but I like non-US providers, I don't care if the party happens to be the one reading my information and stealing my trade secrets, if they won't respond to a US subpoena
What explains it?
Is TFA lying? Is the most upvoted comment here lying?
There's a number of US providers who also run it, if that is your preference.