I think it's useful to be realistic about what you can do with a local model, especially something as small as the 9B the author is using. A 9B model is around the level of Sonnet 3.6 - it can do autocomplete and small functions but it loses track trying to understand large problems.
But the are interesting and fun to play with! I do a LOT of work on local agent harnesses etc, mostly for fun.
My current project is a zero install agent: https://gemma-agent-explainer.nicklothian.com/ - Python, SQL and React all run completely in browser. Gemma E4B is recommended for the best experience!
This is under heavy development, needs Chrome for both HTML5 Filesystem API support and LiteRT (although most Chromium based browsers can be made to work with it)
It's different to most agents because it is zero install: the model runs in the browser using LiteRT/LiteLLM (which gives better performance than Transformers.js), and Filesystem API gives it optional sandbox access to a directory to read from.
It is self documenting - you can ask questions like "How is the system prompt used" in the live help pane and it has access to its own source code.
There's quite a lot there: press "Tour" to see it all.
I agree, earlier Sonnet wasn't that great, but Sonnet 3.5 is where things really came together. The difference was night-and-day. Sonnet 3.7, 4.0, 4.5, etc... didn't have as drastic of a change to me.
Not to be nitpicky, but many of the 4-12b models are somewhere between GPT-3.5 and GPT-4o-mini. It's hard to find a good comparison though, because the benchmarks people score models against change so often. For reference, Sonnet 3.6 came out about a year after GPT 3.5
Don't worry about being nitpicky! I'm going to out-nitpick you....
Actually....
I write and publish my own benchmark for this stuff. It's an agentic SQL benchmark which isn't in the training data yet and I've found can separate frontier models from close-followers (the only models to get 100% are Opus 4.6 and GPT 5.5).
I think that the more people who try local models (especially the larger ones) the better.
I sometimes get the impression that many people claiming that local models are as good as frontier models work in "token poor" environments. If you can't build large-scale programs using at least Opus 4.5+ then it's difficult to compare. They compare something like Qwen 27B with Sonnet and see that it is nearly as good, but miss that the frontier models are a lot better.
That knowledge is power, too.
I personally can help making local models more accessible. I can't make Opus cheaper.
> I sometimes get the impression that many people claiming that local models are as good as frontier models work in "token poor" environments. If you can't build large-scale programs using at least Opus 4.5+ then it's difficult to compare.
I sometimes get the impression that people posting comments on HN don't realize that LLMs do more than vibe coding.
Yeah no kidding. For instance, if you are an independent inventor trying to write a patent while keeping your patent lawyer expenses to a minimum, you want to write as much of the first draft(s) of the patent as you can yourself. (You’ll save billable hours with your patent lawyer, and you’ll end up with a better patent because you’ll communicate your innovations more clearly to your lawyer.)
However, and this is the big thing, you absolutely do not want to be asking a SOTA LLM for help with the language in your patent application. This is because describing your invention to a web based LLM could be considered a public “disclosure” of your invention, which, (after a one year grace period goes by), could put your invention in the public domain, basically—and thereby prevent you (or anyone else) from being able to ever patent the invention. Plus, you know, a random unscrupulous employer at the SOTA company could be reviewing logs and notice your great idea, and file a patent on it before you do, and remember, the United States patent office went to “first to file” in 2013.
Oh and don’t take legal advice from random people in the internet by the way.
> This is because describing your invention to a web based LLM could be considered a public “disclosure” of your invention, which, (after a one year grace period goes by), could put your invention in the public domain, basically—and thereby prevent you (or anyone else) from being able to ever patent the invention.
This is simply not true. Even if it were true (and again, it's not) you could simply use zero data retention APIs.
No one at the big model companies is trawling through your chats to steal your patents. It's not only illegal and against their own terms of service, but these people have better uses of their time.
If a competitor to your business discovers that you used a free online AI to help draft your patent 1.5 years ago, that competitor could then cause your patent to be invalidated, which could be greatly to their benefit of course.
The Terms of Service (ToS) for Open/Public AI (e.g., free consumer versions of ChatGPT, Gemini, Claude) often reserve the right to store your prompts and use them to train and refine the model.
Doing an enabling disclosure of your patent draft to another party that is not bound by a non-disclosure agreement is a big mistake, at least while the case law has not yet been settled.
My post was meant to be encouraging to people that might be considering local LLM for this specific use case, where protecting confidential information is of particular importance.
Compute doesn't necessarily linerarly follow parameters. And with how many active parameters Mythos vs Opus gets its effectivenes from? Is it 1x or 2x? We don't know. We don't even know the parameters (it's more of rumor than confirmed 10T iirc).
But even more so, who said the improvements are "exponential"? Mozilla's single metric, that doesn't even prove anything of the sort?
I know parameters don’t translate directly like that (and that linear and exponential aren’t the only types of growth) but a doubling as a go-to example of “not exponential growth” is pretty funny.
Ah yes, the marketing model that's ostensibly so powerful us mere mortals aren't allowed to use it. It's certainly led to exponential hype and speculation.
The labs are spending hundreds of millions of dollars hiring people doing many fairly random (but economically valuable) tasks to collect this tacit knowledge for RL.
> It ceases to become tacit as soon as it is collected.
I'm not sure.
It it is collected via preferences then it isn't necessarily something that can be communicated (except in the LLM's latent space).
That still feels tacit to me.
To simplify that argument, the relationship between King and Queen in the Word2Vec latent space can be easily explicitly labelled.
But the relationship between Napoleon and Tsar Alexander I also exists and encodes much of the tacit knowledge about their relationship but isn't as easily labelled (eg, Google AI Mode says "Napoleon I and Tsar Alexander I had a volatile "bromance" that shifted from mutual admiration to deep animosity, acting as a defining conflict of the Napoleonic Wars".)
Word2Vec is a very simple model. In a more complex LLM that deeper knowledge can be queried by asking questions but you can never capture it all. Isn't that what "tacit knowledge" is?
It's a good question, yeah, and a lot of these boundaries get fuzzy when they're looked at closely enough.
It's certainly the case that LLMs already are able to represent and make use of some kinds of apparently still tacit knowledge, and that the scope of that is apparently expanding. I don't question that. I question two things: whether it is always desirable for that scope to expand, and whether it is possible for that scope to ever fully cover what it seeks to cover.
Say what you like about Sam Altman, but given how Anthropic is scrambling to sign capacity deals for compute we can sure say he was right about the capcity build out needed.
Scrambling? Seems to me xAI built too much capacity (for what they can use in 2026). Does that mean OpenAI built the right amount? I don't see how this proves that just because we see one AI company willing to sell compute. We don't even know the terms/pricing.
> Anthropic CEO Dario Amodei said his company tried to plan for 10-fold growth. But revenue and usage increased 80-fold in the first quarter on an annualized basis, which he says explains why it’s been so hard to keep up with demand.
> “That is the reason we have had difficulties with compute,” Amodei said Wednesday at his company’s developer conference in San Francisco. Amodei added that the company is “working as quickly as possible to provide more” capacity and will “pass that compute on to you as soon as we can.
I think "scrambling" is a fair characterization of the CEO saying "we have had difficulties with compute" and "working as quickly as possible to provide more"
They've also signed new compute deals with Google and AWS recently.
Or the bubble he was pumping hasn't popped yet. We won't be able to say how much of this capacity was actually "needed" until 10 years in the future, if ever.
People are getting real stuff done with the internet. But there were also a whole lot of overhyped companies that rightly crashed back in '99.
Once we've gone through the AI equivalent of the dot.com crash, will Anthropic still be scrambling for more capacity, or will they have more than they can profitably use, like the dark fiber we were left with last time?
Depends when it happens. I'm sure they'll take up whatever capacity is available.
At the moment computer providers are charging more for outdated H100 capacity now than when the H100s were new. That capacity is going to the smaller labs, not the frontier labs.
That hardware has already been depreciated financially so even if all those small labs disappeared it's not sending computer providers bankrupt - they can just cut prices and so long as they can charge more than electricity and maintenance they'll just keep them running.
There were some amazing feats. But it was slow and frustrating. Like you wouldn't believe how long things took.
In the 90s most technical documentation was in actual physical books. If you wanted to learn something you had to order and buy the book (and Amazon wasn't a thing everywhere!), and it would take weeks or months to arrive. Or you did inter-library loans (which were amazing but also took weeks).
Or you relied on magazines which had a publication cycle. Writing actual physical letters about a program that was written out in the magazine was a thing.
When I got internet access in the mid-90s I remember emailing someone to ask about mirrors of their documentation project because I didn't want to use up their bandwidth.
I'd never ever want to go back. Bring on the future!
I was also there. It isn't rose-colored glasses. The tech truly was better back in the 90s and 2000s. Not perfect, certainly, but it was made by people who gave a damn and were trying to actually make good things. It wasn't like today, where most software is slop (not necessarily AI slop, but slop nonetheless) churned out by companies whose decisions are made solely by how much profit they can squeeze out no matter if the quality tanks.
> I started to refer to the process of writing software using AI assistance (soon to become just "the process of writing software", I believe) with the term "Automatic Programming"
Antirez wrote Redis. That is "production-code with critical concurrency"
To quote another of his posts:
> I fixed transient failures in the Redis test. This is very annoying work, timing related issues, TCP deadlock conditions, and so forth. Claude Code iterated for all the time needed to reproduce it, inspected the state of the processes to understand what was happening, and fixed the bugs.
...
> In the past weeks I operated changes to Redis Streams internals. I had a design document for the work I did. I tried to give it to Claude Code and it reproduced my work in, like, 20 minutes or less (mostly because I'm slow at checking and authorizing to run the commands needed).
His summarized assessment from that very post: "...state of the art LLMs are able to complete large subtasks or medium size projects alone, almost unassisted, given a good set of hints about what the end result should be. The degree of success you'll get is related to the kind of programming you do (the more isolated, and the more textually representable, the better: system programming is particularly apt), and to your ability to create a mental representation of the problem to communicate to the LLM."
He's saying you should be writing up complex, highly detailed specs for the LLM to turn into code, stressing that it's critical to work in a self-contained and "textually representable" problem domain. This is not one-shotting complete products from a vague prompt. You're still going to need software architects, and they'll still be doing much the same work. Turning fully-specified design into code has never been a "10x" task, it was always regarded as a relatively straightforward, if often tricky part of the job. And the way he worked with Redis makes it clear that you can't take what the AI delivers at face value, either: you'll have to go through it yourself, and that will take time and effort.
First he didn't write Redis with LLMs, it was way before. Second I'm not speaking of him in that comment.
Also his whole blog is about how, in order to do a task, he would need to spec it properly, then do "code inpainting" with the LLM, then fix all the issues that he could spot only because he's a senior, then repeat, etc
The author said "You know what was the biggest realization of all that?"
> For high quality system programming tasks you have to still be fully involved, but I ventured to a level of complexity that I would have otherwise skipped. AI provided the safety net for two things: certain massive tasks that are very tiring (like the 32 bit support that was added and tested later), and at the same time the virtual work force required to make sure there are no obvious bugs in complicated algorithms.
Javascript running on a page can use a feature that requires a model to be downloaded.
I have pages that use it, or other LLM models via LiteRT or HuggingFace transformers.js.
I try to warn the user, but that is my responsibility as a page author. I like that this is enabling the web platform to remain competitive.
The author is pulling a long bow by trying to claim this is some GDPR violation. Have they ever used the web? There are inefficient sites everywhere, with autoplaying video etc.
4GB isn't nothing, but if a page wants to use it then hopefully it is useful to the user!
Note that one large factor in this is Anthropic and OpenAI's lobbying to ban large open source models. They frame it as "China is stealing US intellectual property" which plays well in the current environment, but be very clear: it is just an anti-competitive play.
To make that very clear, contrast the loud public blog posts about how "Chinese Companies" were using multiple accounts to distill model data, and the complete silence when Elon Musk admitted in court they do the same for Grok.
(The side issue that LLM outputs are probably only protected under contract law, not as something that is copyrightable is a distraction here BTW)
But the are interesting and fun to play with! I do a LOT of work on local agent harnesses etc, mostly for fun.
My current project is a zero install agent: https://gemma-agent-explainer.nicklothian.com/ - Python, SQL and React all run completely in browser. Gemma E4B is recommended for the best experience!
This is under heavy development, needs Chrome for both HTML5 Filesystem API support and LiteRT (although most Chromium based browsers can be made to work with it)
It's different to most agents because it is zero install: the model runs in the browser using LiteRT/LiteLLM (which gives better performance than Transformers.js), and Filesystem API gives it optional sandbox access to a directory to read from.
It is self documenting - you can ask questions like "How is the system prompt used" in the live help pane and it has access to its own source code.
There's quite a lot there: press "Tour" to see it all.
Will be open source next week.
reply