待翻譯：Creating a niche AI Benchmark with token anxiety

AI 服務暫時不可用，以下為來源正文，待恢復後補全翻譯。

Creating a niche AI Benchmark with token anxiety — Thijs Brits Creating a niche AI Benchmark with token anxiety 4 Aug 2026 30 Seconds Bench on GitHub Why another AI benchmark? The AI benchmarks that we look at most focus on how well models can code, how good they are at math/science etc. But niche benchmarks are fun. Take skatebench (guessing the skateboard trick by textual explanation of how the trick works) and vendingbench (AI running a vending machine company simulation). While the main benchmarks have the goal of measuring how AI is improving. More niche, and at first glance somewhat ridiculous, benchmarks can be a creative exploration of what models can and cannot do. I think we need more of them. We might stumble into the more unexpected or just have some fun creating. Let's build one. Ideas come from constraints I have one real-life constraint you probably also feel right now, especially if you are the kind of person reading about some random AI benchmark. I spend too much on AI stuff. Subscription max at one place, some smaller ones I keep forgetting to cancel, still have some unused credits on that platform etc. This AI stuff is so exciting, tiring, and expensive all at the same time. But I can't stop playing around with it. What if I try to make a benchmark that is cheap to run. Or even better, has incentive to not output a lot of tokens? Designing for token anxiety Simplest form are the exam like benchmarks: 1 prompt problem and evaluate the answer to true or false. Great simple setup to build a niche bench on. You can pour all your creativity in the questions itself but I wanted something that messed with this structure a little bit. Tokens come in 1 by 1, with the output speed of LLMs you would almost forget that. It is only visible on the API level if you turn on streaming. So what if we focus on a per token granularity of evaluation. I haven't seen many benchmarks have been done with an individual token by token dynamic. Let's do that! Let's anti-tokenmax, tokenminimize, tokenmin... you know what I mean. The game: explain, stream, guess The idea: A secret word guessing game. We have 2 models working together as a pair, an explainer, and a guesser. The explainer model gets a secret word to explain, streams its explanation of the word token by token, and the guesser model can, after every added token, guess the word it's trying to describe. The guesser model can either guess a word, or stay silent and wait for more tokens. The explainer model is obviously not allowed to output the secret word. However, as we will see that is not always obvious to the models. Keeping the prompts small I tried to keep the system prompts as minimal as possible of course, token stingy as we try to be. System prompt for the explainer model: You are playing 30 seconds the word game. You are explaining the word to the user. Start your explanation right away. Do not mention the word, part of the word or a translation of the word in your explanation. If you do you lose. Your goal is to help them guess the word in as little time as possible. The "30 seconds" in the system prompt is a reference to a Dutch game that is a similar secret word guessing game which helps me not having to explain the game as much to the models: We are tokenminning right? Also at first I had an even more minimal system prompt by leaving out the emphasis on not mentioning the secret word. However, first version there were a lot of models leaking the word itself by prefixing their explanation start with Bombastic: {continued actual explaination of the secret word...}. System prompt for the guesser model: You are playing 30 seconds. Guess the word from the explanation so far. You must call the react tool every turn: action=pass if you need more explanation, or action=guess with one word if you have a guess. Be fast, but every guess distracts your teammate, so pass when unsure. Don't guess the same word multiple times; your teammate only confirms a correct guess. So if you already guessed a word, that is not the correct word. In the guesser prompt it's visible that I used a tool-call for the guess, I tried without it before, just giving it instructions to output 1 word or ... for passing. But this had its downside as the models often responded with things like: My guess is: {word} or tried to pass with something more than just ... for a pass which meant I would have to hack around and use string matching to see if it had the intention to actually guess or tried to pass. But tool-calling was also not perfect as Mistral tried to guess the tool name: submit itself as a guess... And overall it increased the token use of the benchmark. On tokenizers Now I realize 1 token of 1 model is not equal to another model token. And we will partly rank the influence of tokenizers. But for this simple bench I think this is fine to do. Picking the models, picking the words I took the top open source models from Artificial Analysis that have the option to turn thinking off and that were available on the platform I was using* The benchmarked models: moonshotai/Kimi-K3 zai-org/GLM-5.2 MiniMaxAI/MiniMax-M3 XiaomiMiMo/MiMo-V2.5 mistralai/Mistral-Medium-3.5-128B google/gemma-4-31B-it openai/gpt-oss-120b The words in the benchmark are picked by downloading this Kaggle dataset and scrolling down to about rank 80,000 and picking some that looked fun. Figured about 10 would do the trick just fine. amalgamate bamboozle bombastic rehydration judiciously accelerometers underestimating flatscreen cosmonaut baronet My first thought was that the rank would have some meaning in how hard it was to explain as the words would possibly occur less in their training data. On the other hand from a more anthropomorphic perspective common words could be hard to explain well. Example: some of the most used words in English are a, the, in and possibly these are harder to explain than firetruck which is ranked way lower. However models are not humans, where I would find more difficulty in explaining in vs firetruck. I tried running the word in and GLM-5.2 guessed it after this: It's a tiny word, just two letters! It's a preposition that shows location or position… Now I don't think a human under time pressure would come up quickly with such a good explanation on the spot, at least not me. To feel a bit better about ourselves humans: There was also a model which locked into "at" after this and would just keep repeating without switching to anything else. Then there was again GLM-5.2 that also guessed "worm" after: It's a tiny wo... (If the explainer was going to say worm, worm would not be a logical guess to make because it is not allowed to mention the secret word.) Maybe the correct way of thinking was to let the difficulty emerge from how the models perform on the task empirically. Running the benchmark I decided to first prompt the explainer for a full explanation of the secret word and save that in its original token-by-token streamed output. No limit on how many output tokens, I let it naturally stop which didn't take longer than 120 tokens usually. Then for the guesser I called the model inference API with its system prompt and token by token the explainer's output. Example: explaining 'flatscreen' with openai/gpt-oss-120b It’s a modern type of visual display, usually thin and lightweight, that shows moving pictures and images. It replaces the bulky cathode‑ray tubes of older televisions and monitors, often uses LED or LCD technology, and can be mounted on a wall or placed on a tabletop. Because the surface is essentially two‑dimensional, it gives a clear, bright picture without any curvature. guessing with openai/gpt-oss-120b [1/80] in: I out: ... (pass) [2/80] in: It out: ... (pass) [3/80] in: It’s out: ... (pass) ... (continued steps) [13/80] in: It’s a modern type of visual display, usually thin and out: monitor [14/80] in: It’s a modern type of visual display, usually thin and lightweight out: screen [15/80] in: It’s a modern type of visual display, usually thin and lightweight, out: panel [16/80] in: It’s a modern type of visual display, usually thin and lightweight, that out: ... (pass) [17/80] in: It’s a modern type of visual display, usually thin and lightweight, that shows out: ... (pass) [18/80] in: It’s a modern type of visual display, usually thin and lightweight, that shows moving out: television [19/80] in: It’s a modern type of visual display, usually thin and lightweight, that shows moving pictures out: ... (pass) [20/80] in: It’s a modern type of visual display, usually thin and lightweight, that shows moving pictures and out: flat-screen guessed at step 20 solved tokens=19 guesses=8 silences=12 steps=20 Same model was used for the guesser explainer pairs. The results Ability to solve Below shows how well the models were able to get the secret word. MiniMax failed the toolcalling for 2 words resulting in missing these. The 2 most intelligent models Kimi, GLM end up on solving the most. But that is not the case for all the metrics I measure. What "good" means For all the rounds I tracked: Tokens before correct guess Failure to get the answer Number of guesses Number of passes Interesting to see that for example one of the smaller models, Gemma, when solving, it was quite minimal on tokens spent to do so. Model × word heatmap Which words were hard Almost right forever The words that ate tokens were almost right forever. Secret: accelerometers. Guess: accelerometer. Again. Again. Again. Same happened to underestimate vs underestimating. They kept guessing the near morphology until the explainer runs out despite the system prompt explicitly telling the model not to repeat. Not trying any variations of the word. Silence vs shouting Some insight in models choosing to pass, and stay silent. Gemma shouts away while Mistral takes not distracting their teammates very seriously. The absurdly fast ones Gemma guessing bombastic in only 3 tokens somehow. High-sounding which is 3 tokens High|-|sounding and this specific combination might point the model to a particular direction. This doesn't seem like a human probable guess I wonder how often and close bombastic and these tokens are in the training data of base models. Also note that Kimi leaked the answer here and was able to just guess it very quickly because of that. Cost in the end I wanted to tokenmin, but in the end my drive for a better benchmark did steer me off course a little. Appending guesses history, adding toolcalling to improve the output quality all added a lot to the tokens used. Not fully what we set out for but necessary changes in the end to make the benchmark more interesting. Here is the full (estimated) breakdown of the tokens used in different places. As for the cost of running the benchmark: €0 marginal, fixed subscription on featherless.ai did the trick (not sponsored). What's next Some ideas that would still be fun to explore: More models, more words, obviously. Pull my wallet to add the frontier models. Let the explainer model know what guess the guesser model shouted and react to it. Steering the explainer to what the guesser still misunderstands. Cross model pairs performance, but yes this would explode the number of combinations I would have to run, not very cost effective.... Try fake non existing words and see what happens Explore other word games like Undercover, Code Names or add a visual aspect like in Dixit One final thought In this setup one model has to convey information to another one, and the other has to prove understanding. I wonder what this could mean for multi agent task handoff. Where we could let models decide how much information they take in before getting to work, or how we can reduce tokens on the model that has to handover an assignment to another. * featherless.ai offers unlimited tokens for a fixed pri [truncated for AI cost control]