Rendered at 14:35:27 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
stusmall 2 days ago [-]
This is so interesting to watch. For a couple minutes I was in awe of how quick and cheap it was. Then I saw just how bad the decision are and how it would get stuck in strange loops of going in and out of the same door to no end.
This seems like a technology heading in the right direction but not quiet there yet. Excited for what they are cooking up but probably won't start building around it yet.
binlog 2 days ago [-]
This entire conversation around Jev seems weird to me. Like... we started from neural nets that could do basic decision making and classifications pretty well, then trained larger and larger language models to get to where we are now. Now suddenly everyone is going crazy because someone trained a smaller model that is adequate at making decisions? We already went through the "look this AI can play pokemon terribly" phase like a decade ago.
c7b 2 days ago [-]
A pre-trained universal classifier that can replace specifically-trained ones would have been considered just as much science fiction in the 2010's as the capabilities of modern LLMs. I'm not sure Jev is actually there yet, but at least it sounds theoretically doable today.
That being said, one thing having been unrealistic 10 years ago and just about possible today doesn't mean that it's going to change the world the same way another technically related, previously-impossible thing did. The Jev hype gives me a bit of the "you're still early to crypto" vibes of some later altcoins. I really like the idea, I think it's going to open up possibilities for using classifiers where we wouldn't or couldn't have trained one before. I'm crossing my fingers for an open weights version to drop. But it's still just a classifier, people have built similar things before Jev, the one thing that really stands out about it is their ability to generate hype.
daveguy 24 hours ago [-]
> A pre-trained universal classifier that can replace specifically-trained ones would have been considered just as much science fiction in the 2010's as the capabilities of modern LLMs. I'm not sure Jev is actually there yet, but at least it sounds theoretically doable today.
That's the point. It can't. And it's not even close.
someothherguyy 2 days ago [-]
> but at least it sounds theoretically doable today
why
thornewolf 2 days ago [-]
We have a bad universal classifier now (via Jev). 0->1, one might say.
A bad universal classifier does suggest a good one later. And that is exactly what I would call "theoretically doable"
That said, I don't think that Jev is a magic breakthrough or anything. I think it is just a particularly good narrative with an easy way to try it out.
lukev 2 days ago [-]
Jev is interesting in that it's much cheaper and faster than a frontier LLM.
But I've seen nothing to indicate that the upper bound on classification tasks of a Jev-like model can exceed a frontier LLM with reasoning tokens. That seems nearly impossible even in principle (since Jev-style models are still based on LLM pretraining).
So while they're definitely on the Pareto frontier, which is valuable, they're at the "cheap" end of the spectrum more than the "good" end and I don't expect that to change.
2 days ago [-]
c7b 2 days ago [-]
LLMs are like lossy compression of ~all of written text ever produced, with useful recall. To the extent that the corpus contains labelled examples of the given classification task, it's not unreasonable to think that we'll be able to build a decoder for that, just like we already have a useful decoder for next-token prediction. Extend to image classification the same way we already have multimodal LLMs.
solidasparagus 2 days ago [-]
The cheap, fast and smart-enough LLM space has been wildly neglected. Jev is one of the few players truly targeting that space. And for a lot of people it is the first time they are asking "what could I build if llms were interaction-speed fast?". The answers are cool, the problem is that Jev is not, I think, smart-enough yet to have that many applications, but it's smart enough that you can start to see what they will look like.
Garlef 1 days ago [-]
The difference is that you don't need training here;
The decision graphs can be built on the fly by an LLM and contain instructions in plain language.
The magic moment for me from the Jev release was not that there was some system playing doom: Rather it was the moment, they just changed a part of the prompt to "don't shoot, just dodge" and the behavior changed immediately.
This means you can have a system with fast decision-making but still interact with it via language.
catlifeonmars 6 hours ago [-]
You’re kind of screwed on the latency optimization side bc your bottleneck is a language model though. Why wouldn’t you eventually just use a specialized model that isn’t constrained by running a language model under the hood?
Garlef 4 hours ago [-]
True - in the long run. But that's quite expensive. And you'd need a bigger org to even support this.
They even named the model after "Jevons Paradoxon" - They anticipate that their model lowers the cost of adopting this kind of AI significantly, unlocking a lot of use cases.
osener 2 days ago [-]
It is impressive, but all the hype and fake demos are selling it as a model that is as smart as frontier reasoning LLMs in the decisions it makes yet much cheaper and much faster, which is not true.
pancomplex 2 days ago [-]
Nothing fake here and fully open source if you wanna take a peek. It does make a bunch of mistakes, often. But it eventually recovers!
For those curious how it works, it’s essentially a script that plays the game but uses jev as a source of rng to make it stochastic
Jach 1 days ago [-]
Yeah, there's a lot hard-coded into the typescript files that make this much less impressive than the other "AI plays" versions that have come and gone that play with less help. A Claude one that only used screenshots would always get stuck in the rocket hideout...
osener 1 days ago [-]
For the record I did not mean to say that about your work, nice job.
gchamonlive 2 days ago [-]
I think that misses the point of Jev being ridiculously efficient while maintaining adequate intelligence for automation tasks. We have to train our minds to filter out branding and marketing.
1 days ago [-]
mtford 1 days ago [-]
This happens all the time in tech. A few years ago everybody got excited about static websites and server-side rendering as if we hadn't been doing that with PHP long ago.
gchamonlive 2 days ago [-]
For what it does, it classifies, orchestrates, operates and delegates tasks exceedingly well for its size and weight. It's ridiculously cheap and efficient, but if you can only see progress in terms of raw cognitive power then you'll surely miss how interesting this is.
sky2224 12 hours ago [-]
The difference to me is that the models from a decade ago would need to be trained to play the game. As far as I'm aware, Jev has had zero specific training or fine-tuning to play the game. It's simply told, "Hey, here are the rules. Play." That's it.
I work in manufacturing. I think this will be fantastic for stuff like SPC.
michaelchisari 2 days ago [-]
Performance and efficiency have been neglected as everyone threw every available GPU and trillions of dollars trying (and failing) to create AGI. Though the results have been impressive.
But good enough for pennies in an instant is very useful.
ford 2 days ago [-]
I agree it's overhyped, but the transition to a general purpose classifier (vs a narrow scope classifier) is new and noteworthy.
Ie the famous "Hotdog" clip from Silicon Valley [0]
Yea but this model wasn’t trained to play Pokémon but it can. It’s general.
daveguy 24 hours ago [-]
So is a random number generator. That doesn't mean it can play Pokemon.
azan_ 2 days ago [-]
Making decisions quickly, cheaply and without having to train your own model.
joshuat 2 days ago [-]
Math.random can make poor decisions quickly and cheaply
azan_ 2 days ago [-]
Benchmark it against jev and you'll have your answer.
catlifeonmars 6 hours ago [-]
No you have to use Math.sqrt(Math.random()) to outperform jev
zahlman 2 days ago [-]
I mean,
> get stuck in strange loops of going in and out of the same door to no end
Math.random is statistically unlikely to do this.
joshuat 2 days ago [-]
mathrandomplayspokemon.org
irregularbowels 1 days ago [-]
[flagged]
jbjbjbjb 2 days ago [-]
Jev is for single shot classification, not multi-step RL environments with delayed reward and explore/exploit. My guess is it would go through the door with high confidence every time unless you change the input to add the history.
pancomplex 2 days ago [-]
This runs entirely on Jev as the only AI with a typescript harness that feeds it selective context.
Garlef 1 days ago [-]
I think the people behind jev made their intended use clear by labeling as "system one" - so they anticipate that there's a slower "system two" mediating jev
ralusek 2 days ago [-]
The exact message I sent my friend this morning:
> the most interesting thing about this jev stuff
> is that people are seemingly like
> completely disinterested in how smart it actually is
> I haven't even heard it mentioned a single time how it actually compares to other LLMs coming up with their own classifications. Just: it's fast and cheap
After watching a few minutes of this it makes me think that maybe we should be a little more interested in how smart it is.
johnsmith1840 2 days ago [-]
They know it's not. The company actually has or had public statements that they didn't like public benchmarks for comparison.
My main wonder is the difference between it and having a small llm no thinking output a single number only as a choice. Isn't that nearly the same here?
stusmall 2 days ago [-]
There is a lot of room for a lot of different models. For many use cases, intelligence beats out all.
For me in my day job, having extremely fast low quality decision makers over noisy inputs is very valuable. I work in security and having something that can help triage alerts, classify items and group things together is extremely valuable. It doesn't need to be perfect. Just being able to take a set of inputs from deterministic tooling and to be make general priority classifications goes a long way on helping humans look at the most important items first.
refactor_master 1 days ago [-]
Why not spend a few tokens or so on making a frontier LLM build the classifier from scratch? Then you also know exactly what the classifier is capable of, and whether it’s easily learnable/informative data.
Why would you be confident in feeding garbage to a “cheap and fast” classifier with unknown domain-specific performance?
I know we kind assume omniscience for frontier models, but at this point the evidence is kind of out there.
raincole 2 days ago [-]
Because it's not that smart especially when you compare it to other LLMs. The top LLMs have completely change the baseline of being smart.
hbrn 22 hours ago [-]
You gotta admit that turning "our model is very dumb and can only make simple decisions" into "System One Model" is a pretty clever marketing move.
pancomplex 2 days ago [-]
Like others have mentioned in this post, I think a mix of models like Jev for simple stuff + a smarter reasoning model for more strategic thinking is the optimal solution. This experiment however is purely Jev. Which sometimes can be kinda dumb.
JamesSwift 1 days ago [-]
Well in a lot of ways this demo actually is a mix of jev for simple stuff + a very intelligent harness to fill in the rest. Pay attention to the "jev counter" on when api calls actually happen and the kinds of decisions its making. Its very rarely in a tight loop, and when it is (eg in an item menu) it tends to randomly walk through options.
Its very cool, but the harness is doing a _ton_ of heavy lifting.
stusmall 1 days ago [-]
Oh I want to be clear, I don't want my post to come off as negative. I really do mean things are headed the right direction and I think this harness is super cool. So don't take it that way and thank you for bringing something cool into this world.
mitxela 1 days ago [-]
feels like pre-LLM AI playing a video game but more expensive
ac2u 2 days ago [-]
Cool project, comes with a little too much guidance in the harness though IMO (pathfinding, textual milestones etc). (The author is very upfront about this in their README though)
I think if it was combined with a regular vLLM it could be really interesting, especially watching the reasoning logs.
Bonus points if it was one of the latest open models that somehow had all prior training knowledge of Pokemon abliterated so it was reasoning as an intelligent persona that had no knowledge of even the concept of Pokemon.
budoso 1 days ago [-]
I also tried to create a “Jev plays Pokémon” but with minimal additional context outside a move history and what is available in memory from the emulator, so no pathfinder, predetermined game path, etc. I can safely say that this experiment failed however, and Jev was not able to even get to Professor Oak’s lab to get a starter.
Props to OP for getting a working version, but it does not seem that this model is capable enough to play Pokémon at this point in time.
pancomplex 21 hours ago [-]
Thank you. What I realized is that Jev is incredibly powerful for the decision making part of it when presented with a scenario that gives it a little context. It looks like we’re gonna be able to beat this game for under $2 of tokens. This is what blows my mind
MitPitt 2 days ago [-]
This is kinda chill to have in the background. I wish there were livestreams showing live reasoning of top models which are currently trying to solve cancer or whatever. Imagine the pogs in chat when it does.
pancomplex 2 days ago [-]
People need to be live streaming their AI more!
tehnoslow 2 days ago [-]
Actually, yes, that would be at least interesting
someothherguyy 2 days ago [-]
then it would cost human lives, no more fun i made this thing with no real effort vibes
brenschluss 1 days ago [-]
The seminal (lol) Twitch Plays Pokémon was twelve years ago, so just posting this amazing moment of internet history/lore just in case folks don’t know or have forgotten: https://en.wikipedia.org/wiki/Twitch_Plays_Pok%C3%A9mon
testaccount28 2 days ago [-]
with such a fat harness, this is more like watching a walk thru play the game.
laszlokorte 2 days ago [-]
Yeah I would have expected it to only decide which button to press, not something abstract like the choice of "go east to lavender town" for the goal of "in lavender town, climb the pokemon tower"
pancomplex 2 days ago [-]
That was how I first implemented it. Jev sadly never left Pallet Town.
pancomplex 21 hours ago [-]
Fair point, but I’d challenge anyone to build an ai or hard coded engine that, even following a walkthrough, completes the entire game for under $2 in tokens
ArcHound 19 hours ago [-]
I'm watching this. After 6 hours it managed to solve the rock boulder puzzle and now it's trying to break the Elite Four wall with its face. I'm no pokemon expert, but I'd say that no way in hell it beats them with this team composition. It has to take a step back, re-compose or at least re-train for overall higher levels. Will see if it manages to do it, my expectations are low.
ngruhn 19 hours ago [-]
It got to Lance. It's Charizard is pretty strong. If it would just HEAL between fights I think it could do it.
I predict it will keep trying and failing without changing strategy until the team is so overleveled that brute force works.
captainbland 18 hours ago [-]
That's probably correct. Blue is going to be another wall as well, especially the blastoise since the team leans so heavily on charizard and graveler
ArcHound 17 hours ago [-]
Poor Graveler died to Dewgong in the first fight (nearly every time). Don't forget about Gyarados though, that one is still undefeated.
ArcHound 19 hours ago [-]
yea, I saw it.
Unfortunately, the only usable move is flamethrower with 15 PPs that will be exhausted.
Remember, if it starts using items, it can run out of money. And what then?
But yes, brute force might prevail in the end. Let's say it started elite four at 13k jev calls or 1.3 USD. I'd call it a "loss" if it crosses the 2 USD threshold. It's still quite cheap though.
pancomplex 18 hours ago [-]
The next version I make should take advice like this from the chat
ArcHound 5 hours ago [-]
I saw the commit log, refining the harness as it went. I don't have Jev access, but I'd like to see it try again with this new setup. Seems it could be much faster, as plenty of commits addressed some blind spots where it fell in a loop.
pancomplex 19 hours ago [-]
It has the option to go back and level up its pokemons + switch it up. Hopefully it gets the same idea as you soon!
ArcHound 19 hours ago [-]
It's quite a long term plan it has to make:
- establish a composition
- train the selected composition to an appropriate level
- ration PPs through 5 fights
- ration items through 5 fights + globally (you can run out of money)
- maybe even switch the order of pokemon on the list? Poor Graveler
I'm not sure it can follow-through on this. Would be funny if it had to re-do the rock boulder puzzle. Guess I'll keep watching!
pancomplex 19 hours ago [-]
It has continued to surprise me throughout all of this so hopefully it will again
ArcHound 19 hours ago [-]
it's a fantastic demo and promo for the product though. Great idea, well executed, I'm genuinely curious on how this will go.
nico 8 hours ago [-]
Wow, it looks like it was able to finish the game (or they stopped it), after almost 38hrs of play and a total of $1.65 in Jev requests
Very cool proof of concept!
1 days ago [-]
rickintoplace 2 days ago [-]
That's actually fun to watch. Did you experiment with nicknaming before you turned it off? I'd be a little curious to see how it behaves.
pancomplex 2 days ago [-]
Jev can't come up with original text, but I did consider giving it a list of hilarious names.
JamesSwift 2 days ago [-]
Sure it can, just ask it for next char or "done" in a loop
pancomplex 2 days ago [-]
I just took your advice and added it to the list of Jev decisions. Watch it name its next Pokemon!
pancomplex 2 days ago [-]
True!
IanCal 2 days ago [-]
Maybe letter by letter spelling?
2 days ago [-]
stusmall 2 days ago [-]
>but not fast enough to play Doom yet sadly
Did I miss something? I thought one of the demo videos was it doing pretty decent at the first level of Doom?
pancomplex 2 days ago [-]
In my experience it was too slow to do an FPS with 30 ticks per second reliably. It gets killed too fast.
dang 2 days ago [-]
Dare we have two Pokemon-playing-AI threads at the same time?
Ai pokemon red speed run chart seems like a cool benchmark
pancomplex 2 days ago [-]
The more the merrier!
lwarfield 2 days ago [-]
Looking at the diagram in the gh repo, it looks like this is entirely jev. Are there any examples of people having a big model like Fable handle high level goals?
Hm wondering what a first pass optimal setup might be - jev for overworld navigation, escalate to sonnet for easy battles, opus for medium difficulty battles, and fable for gym bosses could probably have jev also manage all the escalation / de-escalation to different models.
ViscountPenguin 2 days ago [-]
The choices Jev has here feel very railroady, it seems like something very significantly dumber could beat the game with these options.
pancomplex 2 days ago [-]
I challenge you to fork the repo and solve it faster using less tokens/cost than Jev :)
mrkaye97 1 days ago [-]
I think “significantly dumber” might include traditional RL, and thus no tokens at all
Where does the "goal" come from? Is it just known based on different positions in the game? Otherwise it seems like it would require text generation.
pancomplex 20 hours ago [-]
I wrote down a series of 33 high level goals + some sub goals for completing the game
jackbrookes 1 days ago [-]
I've been watching GPT_Plays_Games on twitch and with Astra now it can beat pretty much all of them with very few mistakes, with vision only. One downside is that when pathfinding it can only plan high level paths, wait for result, then try again if it fails. Its a ~10-30 second loop.
Has anyone tried a combined LLM + Jev? So the LLM directs the high level goal ("reach the door of the pokemon center while avoiding NPCs") then allow it to instruct Jev to do the actual movements? That seems like a good balance between the high-level, slow planning of the LLM to the fast but limited Jev. That kind of mirrors how humans work too. The harness could even allow Jev to delegate back to the LLM when it doesn't have a confident answer.
Its almost like when humans drive, we kind of let our subconscious take over once we know where to go. But if we see something unexpected in the road, we can go back to a conscious planning mode to decide what to do.
Considering it just made Charizard forget its only fire-type move "Ember" to learn "Counter", I note no signs of intelligence.
pancomplex 2 days ago [-]
Rookie mistake clearly..
djhworld 2 days ago [-]
What's not clear to me on the video is whether jev is doing the button presses for controlling the character to move around.
The "Jev calls" counter only seems to increment at junction points like battles, conversation prompts, menus etc.
Is something else moving the character around?
pancomplex 2 days ago [-]
It's connected to the ROM of the actual game, so it can see a lot of things. It has multiple tools available, including being able to move to coordinates.
That team is going to take some time to beat the elite four
singularity2001 1 days ago [-]
Jev is dumb a rock, so it's basically just a random selection of a smart model's suggested plans?
hummusFiend 2 days ago [-]
Great work!
Super cool to see it do the whole game. I spent my fable budget building something similar this week but only drove it to Brock. I like the "current focus" framing too.
pancomplex 2 days ago [-]
Thank you! And interesting data point on Fable burning out on tokens so early.
flockonus 2 days ago [-]
The decisions are quick, but look good as random w/ tons of back and forth.
Are you at least feeding back some of its previous decisions on to state?
iugtmkbdfil834 18 hours ago [-]
Only watched for a few minutes, but it did seem to backtrack a lot.
2 days ago [-]
361994752 2 days ago [-]
watched it stuck at rocket hideout for 10 mins.... let me check 1hr later to see if it can find a way out
jumploops 2 days ago [-]
It’s currently stuck at an elevator and deciding to teach Pokemon various TMs and HMs instead of progressing… pretty hilarious!
pancomplex 2 days ago [-]
it made it through!
dochaus 2 days ago [-]
He confirms what I long suspected, all you need is to cheese Charizard and flamethrower
DylanMerigaud 1 days ago [-]
Struggles with loops show complexity isn't fully mastered yet.
rubicon33 2 days ago [-]
Pretty awesome! Just curious are you vibe coding this?
theturtletalks 2 days ago [-]
Is Frigade going to use Jev to do object detection?
pancomplex 2 days ago [-]
Working on it :)
theturtletalks 2 days ago [-]
Nice. When I saw your connection to Frigade, I knew there was a connection haha
sync1117 1 days ago [-]
i need to come up with something more creative and interesting than video games as benchmarks
zaik 2 days ago [-]
Seems like it got stuck on a Ghost enemy.
avaer 2 days ago [-]
I wish jev took in images so we could do this generically for any game, without memhacks. I'm sure that's coming.
You could front this with an image -> text model but that would be much lower quality vs latency, and the whole point of doing it with a decision model is remove the latency.
Games are a really interesting testing ground for robotics; if we can solve game playing (incl 3d) we could embody "system one" intelligence into robots that have something emulating general reflexes without needing to fine tune.
pancomplex 2 days ago [-]
Agreed this would be super cool and I do see that coming in the future. But Jev-level latency just isn't there yet with full images.
fzysingularity 2 days ago [-]
We just did exactly this - added TypeSafe-compatible API (incl. websocket support) for various VLMs. Latency right now is <250ms, but will be able to get it to <150ms (p95).
Yet another absolutely fake demo where Jev is flipping the coin and harness is doing 99% of the work.
The harness reads the ROM/RAM, builds a collision grid, computes reachable paths with A-star, constructs a region graph, calculates distance to the current objective, determines whether an exit leads toward or away from the objective, identifies Pokémon Centers and Marts, detects NPCs and items, and then turns all of that into natural-language choices for Jev.
Jev is given a list of "choices", already annotated information that tells Jev whether the choice should be taken or not.
For example, Jev might receive options like:
* Enter ROCK_TUNNEL_1F — Leads toward the objective (2 areas away)
* Go west to ROUTE_9 — Leads away from the objective (5 areas away)
* Talk to NPC X — Mentioned in the current objective
* Enter POKECENTER — toward nearest Pokémon Center
Those "toward the objective" judgments are not Jev figuring out the map. The harness computes shortest path distances through its region graph and literally annotates the choices with that answer.
Once Jev says "Enter X" the harness already has the exact A* path and walks it automatically, including replanning around moving NPCs and retrying when blocked.
Puzzle solving is also done by the harness - it gives Jev a set of choices, annotated with sentences like "brings the boulder closer to the floor switch", "after this push no sequence of pushes can bring this boulder onto a floor switch anymore".
Story progression contains hardcoded instructions for Jev, e.g.: 'Go south from Cerulean through Route 5, the Underground Path and Route 6 to Vermilion City; board the S.S. Anne at the dock and get HM01 (Cut) from the captain.'
And even with all of those hardcoded decisions, it still plays terrible. I just saw it get stuck in a loop - but of course harness annotates repeated actions, so eventually Jev is steered to escape the loop.
This seems like a technology heading in the right direction but not quiet there yet. Excited for what they are cooking up but probably won't start building around it yet.
That being said, one thing having been unrealistic 10 years ago and just about possible today doesn't mean that it's going to change the world the same way another technically related, previously-impossible thing did. The Jev hype gives me a bit of the "you're still early to crypto" vibes of some later altcoins. I really like the idea, I think it's going to open up possibilities for using classifiers where we wouldn't or couldn't have trained one before. I'm crossing my fingers for an open weights version to drop. But it's still just a classifier, people have built similar things before Jev, the one thing that really stands out about it is their ability to generate hype.
That's the point. It can't. And it's not even close.
why
A bad universal classifier does suggest a good one later. And that is exactly what I would call "theoretically doable"
That said, I don't think that Jev is a magic breakthrough or anything. I think it is just a particularly good narrative with an easy way to try it out.
But I've seen nothing to indicate that the upper bound on classification tasks of a Jev-like model can exceed a frontier LLM with reasoning tokens. That seems nearly impossible even in principle (since Jev-style models are still based on LLM pretraining).
So while they're definitely on the Pareto frontier, which is valuable, they're at the "cheap" end of the spectrum more than the "good" end and I don't expect that to change.
The magic moment for me from the Jev release was not that there was some system playing doom: Rather it was the moment, they just changed a part of the prompt to "don't shoot, just dodge" and the behavior changed immediately.
This means you can have a system with fast decision-making but still interact with it via language.
They even named the model after "Jevons Paradoxon" - They anticipate that their model lowers the cost of adopting this kind of AI significantly, unlocking a lot of use cases.
https://github.com/christianmat/jev-pokemon
I work in manufacturing. I think this will be fantastic for stuff like SPC.
But good enough for pennies in an instant is very useful.
Ie the famous "Hotdog" clip from Silicon Valley [0]
https://www.youtube.com/watch?v=ACmydtFDTGs
Example (2022):
https://developers.openai.com/cookbook/examples/zero-shot_cl...
> get stuck in strange loops of going in and out of the same door to no end
Math.random is statistically unlikely to do this.
> the most interesting thing about this jev stuff
> is that people are seemingly like
> completely disinterested in how smart it actually is
> I haven't even heard it mentioned a single time how it actually compares to other LLMs coming up with their own classifications. Just: it's fast and cheap
After watching a few minutes of this it makes me think that maybe we should be a little more interested in how smart it is.
My main wonder is the difference between it and having a small llm no thinking output a single number only as a choice. Isn't that nearly the same here?
For me in my day job, having extremely fast low quality decision makers over noisy inputs is very valuable. I work in security and having something that can help triage alerts, classify items and group things together is extremely valuable. It doesn't need to be perfect. Just being able to take a set of inputs from deterministic tooling and to be make general priority classifications goes a long way on helping humans look at the most important items first.
Why would you be confident in feeding garbage to a “cheap and fast” classifier with unknown domain-specific performance?
I know we kind assume omniscience for frontier models, but at this point the evidence is kind of out there.
Its very cool, but the harness is doing a _ton_ of heavy lifting.
I think if it was combined with a regular vLLM it could be really interesting, especially watching the reasoning logs.
Bonus points if it was one of the latest open models that somehow had all prior training knowledge of Pokemon abliterated so it was reasoning as an intelligent persona that had no knowledge of even the concept of Pokemon.
Props to OP for getting a working version, but it does not seem that this model is capable enough to play Pokémon at this point in time.
I predict it will keep trying and failing without changing strategy until the team is so overleveled that brute force works.
Unfortunately, the only usable move is flamethrower with 15 PPs that will be exhausted.
Remember, if it starts using items, it can run out of money. And what then?
But yes, brute force might prevail in the end. Let's say it started elite four at 13k jev calls or 1.3 USD. I'd call it a "loss" if it crosses the 2 USD threshold. It's still quite cheap though.
- establish a composition
- train the selected composition to an appropriate level
- ration PPs through 5 fights
- ration items through 5 fights + globally (you can run out of money)
- maybe even switch the order of pokemon on the list? Poor Graveler
I'm not sure it can follow-through on this. Would be funny if it had to re-do the rock boulder puzzle. Guess I'll keep watching!
Very cool proof of concept!
Did I miss something? I thought one of the demo videos was it doing pretty decent at the first level of Doom?
Teaching a World Model to Play Pokemon - https://news.ycombinator.com/item?id=49849907
The interesting thing here imo is the cost and latency. So far we're at 4 badges for less than $0.5
However, https://x.com/TynanSylvester/status/2096965749369720970 Astra was able to beat RimWorld. So LLMs are definitely able to drive these sorts of games to completion with their current abilities.
Has anyone tried a combined LLM + Jev? So the LLM directs the high level goal ("reach the door of the pokemon center while avoiding NPCs") then allow it to instruct Jev to do the actual movements? That seems like a good balance between the high-level, slow planning of the LLM to the fast but limited Jev. That kind of mirrors how humans work too. The harness could even allow Jev to delegate back to the LLM when it doesn't have a confident answer.
Its almost like when humans drive, we kind of let our subconscious take over once we know where to go. But if we see something unexpected in the road, we can go back to a conscious planning mode to decide what to do.
1. https://www.twitch.tv/gpt_plays_games
The "Jev calls" counter only seems to increment at junction points like battles, conversation prompts, menus etc.
Is something else moving the character around?
All in the OSS repo if you wanna play around with it: https://github.com/christianmat/jev-pokemon
Super cool to see it do the whole game. I spent my fable budget building something similar this week but only drove it to Brock. I like the "current focus" framing too.
You could front this with an image -> text model but that would be much lower quality vs latency, and the whole point of doing it with a decision model is remove the latency.
Games are a really interesting testing ground for robotics; if we can solve game playing (incl 3d) we could embody "system one" intelligence into robots that have something emulating general reflexes without needing to fine tune.
Take a look at a snake demo with streaming image inputs: https://x.com/spillai/status/2103630735425089957
The harness reads the ROM/RAM, builds a collision grid, computes reachable paths with A-star, constructs a region graph, calculates distance to the current objective, determines whether an exit leads toward or away from the objective, identifies Pokémon Centers and Marts, detects NPCs and items, and then turns all of that into natural-language choices for Jev.
Jev is given a list of "choices", already annotated information that tells Jev whether the choice should be taken or not.
For example, Jev might receive options like:
Those "toward the objective" judgments are not Jev figuring out the map. The harness computes shortest path distances through its region graph and literally annotates the choices with that answer.Once Jev says "Enter X" the harness already has the exact A* path and walks it automatically, including replanning around moving NPCs and retrying when blocked.
Puzzle solving is also done by the harness - it gives Jev a set of choices, annotated with sentences like "brings the boulder closer to the floor switch", "after this push no sequence of pushes can bring this boulder onto a floor switch anymore".
Story progression contains hardcoded instructions for Jev, e.g.: 'Go south from Cerulean through Route 5, the Underground Path and Route 6 to Vermilion City; board the S.S. Anne at the dock and get HM01 (Cut) from the captain.'
And even with all of those hardcoded decisions, it still plays terrible. I just saw it get stuck in a loop - but of course harness annotates repeated actions, so eventually Jev is steered to escape the loop.
You should be ashamed of yourself.