2026-07-09 10:29 UTCIn-site rewrite6 min readUpdated: 2026-07-09 10:55 UTC

Show HN: Self-Hosted AI That Sees, Plays Minecraft, and Reacts

Wallie is an open-source AI streamer that watches your screen, hears audio, and now plays Minecraft autonomously. It features long-term memory, natural pacing, a full persona system, and emotion-driven Live2D avatar.

SourceHacker News AIAuthor: Alrady

Notifications You must be signed in to change notification settings

Fork 4

Star 18

BranchesTags

Open more actions menu

Folders and files

NameName

Last commit message

Last commit date

Latest commit

History

36 Commits

audio

avatar

chat

control

core

dashboard

dist/mods

docs

hearing

llm

profiles

scripts

tests

tts

utils

vision

.env.example

.gitattributes

.gitignore

CONTRIBUTING.md

LICENSE

README.md

SECURITY.md

main.py

config.py

install.bat

install.sh

requirements.txt

secrets_store.py

start.bat

start.sh

wallie.py

Repository files navigation

The open-source AI streamer that sees, hears & reacts — and actually feels alive.

Quick Start · Features · Dashboard · Architecture · Roadmap

⭐ If you think an AI that watches and hears your screen should exist — give it a star. Two seconds, and it genuinely helps a solo dev keep shipping.

Every AI streamer you've seen has the same problem.

It says two sentences. Pauses. Says two more. Pauses. Describes your YouTube homepage like a screen reader. Forgets what it said thirty seconds ago. Ends every thought with "what do you guys think?" into a chat that doesn't exist. It's not a show — it's a tech demo on loop.

Wallie is different. It's an AI that actually streams — develops thoughts across minutes, reacts to your screen like a person who's been using that computer all day, remembers what it covered an hour ago, takes opinions and sticks with them, drifts between topics the way real conversations drift, and shuts up when there's nothing worth saying.

You design the streamer. Wallie runs the show.

persona ──┐ topics ──┤ chat ──┤ vision ──┼──► Wallie ──► Voice (TTS) ──► OBS / virtual cable ──► your stream hearing ──┤ ──► Live2D avatar (VTube Studio) schedule ──┘

Pick the personality. Pick the voice. Pick the LLM. Pick the platform. Everything is swappable, everything is configurable from the browser, and the whole thing runs on your machine with your keys.

🎮 New in v2.0 — Wallie can now PLAY

Wallie doesn't just watch your screen anymore — it picks up the game and plays it. Right now that's Minecraft: it gathers, crafts, builds, fights, and explores entirely on its own, narrating every moment in character. Smooth, human-looking camera. No human at the keyboard, no script — a real playthrough with a personality.

https://www.youtube.com/shorts/NEfsmppOXJk

And here's the part that makes it believable: the commentary is grounded in what Wallie is actually doing — its real inventory, health, and current goal — not guessed from a blurry frame. So it says "heading down to mine diamonds" because it is, and brings up "that one time I dug straight into a lava pit" because that's its run.

It actually plays — a planning brain sets the goal; Baritone handles pathfinding & mining, and a custom Fabric mod handles crafting, combat, item pickup, and a buttery-smooth camera that looks human, not robotic.

Grounded commentary — the streamer core stays in sync with the playing brain, so what it says matches what it does.

One-click setup — the dashboard installs Fabric + every mod for you (and backs up your current ones first).

Any persona, any model — Play is just another mode. Vision, Hearing, and Play each toggle on/off independently.

Flip on Play in the dashboard, pick a goal, and go — or jump straight in:

python scripts/run_wallie_live.py

Why another AI streamer?

Because the existing ones are toys.

They work for a 2-minute demo and fall apart on a real stream. Here's what actually goes wrong after 10 minutes:

The problem What usually happens What Wallie does

Repetition Says "that's interesting" every 30 seconds Dedupe engine + phrase cooldown + rolling summary that tracks everything already said

Short memory Forgets the topic from 5 minutes ago Rolling summarizer compresses old turns into bullet notes, injected into every prompt

Robotic vision "I can see a YouTube homepage with several videos" First-person ownership + SKIP escape hatch — narrates nothing, reacts to what matters

Question loops Every segment ends with "what do you think, chat?" Question detector throttles after one; next prompt forces a statement ending

No personality Generic helpful assistant voice Full persona system: energy, humor style, catchphrases, backstory, opinions, taboo topics

Choppy pacing 2 sentences → long pause → 2 sentences Pipeline overlap + longer thought development eliminates dead air

Broken threads Sets up a story, never delivers it Open-thread tracker forces the next segment to pay off teases and answer its own questions

Topic whiplash Jumps from AI to cooking to space with no bridge Association-based topic drift with configurable style (rigid / natural / freeform)

These aren't edge cases. They're the default behavior of most AI streamer projects. Wallie was built by hitting every one of these problems and refusing to ship until they were fixed.

Features

Bring your own everything

Six LLM providers. Three TTS engines. Three chat platforms. Mix and match per profile.

LLM TTS Chat Avatar

OpenAI Fish Audio Twitch VTube Studio (Live2D)

Anthropic (Claude) ElevenLabs YouTube —

Google (Gemini) Piper (local, free) Kick —

Groq — — —

OpenRouter — — —

Ollama (local, free) — — —

Swap providers without changing code. Run a fully offline stream with Ollama + Piper, or go premium with Claude + ElevenLabs.

Full persona design

Not just a name and a system prompt. You design a character:

Identity — name, handle, pronouns, age range, origin, archetype, backstory

Voice — energy level (chill → unhinged), humor style (pick multiple: ironic, deadpan, absurd, observational, roast...), profanity level, formality

Flavor — catchphrases, running gags, banned words, favorite topics, taboo topics

Opinions — strong opinions toggle, admit uncertainty, break the fourth wall

Extra notes — free text for anything the structured fields miss

Save multiple personas. Switch between them with a dropdown.

Hours-long sessions without decay

The single biggest technical challenge. Most AI chatbots break down after 20 minutes because the context window fills up.

Wallie's approach:

Rolling summarizer — every ~14 segments, a background LLM call compresses older history into tight bullet notes

Session notes — that compressed memory is injected into every system prompt, so the streamer knows what it already said

Cross-session memory — key facts and viewer interactions persist across streams

Dedupe engine — paraphrase-aware similarity check (bigram + trigram Jaccard) catches the model repeating itself in different words

Organic pacing

Real streamers don't talk at a constant rate. They get excited, they pause to think, they go on tangents, they settle into a flow.

Mood engine — arousal, valence, focus, and talkativity drift slowly over the stream. High arousal = faster, warmer output. Low talkativity = silence beats. Mood state feeds into the avatar for reactive animation.

Attention engine — vision events are filtered through a probabilistic decision layer. Not every screen change gets a deep reaction. Some get a glance. Some get ignored. Some trigger a personal tangent.

Silence beats — the streamer holds natural pauses when the mood says to. Dead air handled by design, not by accident.

Pipeline overlap — next segment starts generating while current audio is still playing. No awkward 3-second gaps between thoughts.

Vision that isn't narration

wallie_minecraft.mp4

Screen reactions are the hardest part to get right. Wallie's approach:

First-person ownership — Wallie owns whatever is on screen. Gaming: "I just got bodied by that boss", not "the character is fighting a boss". Browsing: "let me pull this up", not "the user is browsing". Never third-person, never narration.

SKIP escape hatch — if there's nothing specific to name, the model outputs SKIP and stays quiet. No more narrating generic UIs.

Activity adaptation — detects scrolling, typing, app-switching, video playback and adjusts reactions accordingly. Typing → ignore. App switch → react. Rapid browsing → wait until user settles. Each activity type gets context-aware prompting.

Attention engine — not every screen change gets the same treatment. DEEP reactions (22%), quick GLANCEs (28%), personal TANGENTs (5%), deliberate IGNOREs (27%), and SILENCE beats (18%). Streak fatigue prevents reacting the same way twice in a row.

Scene memory — remembers what it last said about the current screen. Dedupe threshold at 0.65 with per-sentence comparison catches paraphrased repetition.

Hearing — it reacts to what it hears, not just what it sees

Wallie captures your system audio (WASAPI loopback) and reacts to it live, fused with vision into a single reaction — sight and sound, one voice. Just like you organize your senses, not two narrators talking over each other.

Speech & lyrics — local speech-to-text (faster-whisper) transcribes videos, voice chat, and song lyrics. Auto-uses your GPU if available, clean CPU fallback if not.

Real music understanding — pure-numpy DSP, no extra ML weight. It reads major/minor key → mood, tempo + beat strength, instrumentation texture (bass-heavy / acoustic / airy / lo-fi), and production quality (crisp / muddy / harsh). A sad song reads as melancholy; a beat drop reads as energy. It'll call a track a banger or roast a muddy mix.

Multimodal fusion — what it hears is fused with what it sees into one coherent reaction, never two competing ones.

No self-echo — content-based guard so Wallie never reacts to its own TTS bleeding back through the loopback.

Pure-hearing mode — run it with vision off and it reacts to audio the way it reacts to a screen: stays quiet, listens, then reacts to what's playing.

Enable it in the dashboard's Hearing section. Needs two extra deps: pip install soundcard faster-whisper.

Wallie plays games — not just watches them

Play mode hands Wallie the controls. First game: Minecraft, survival, live and unscripted.

Reliable autonomy — a planning brain picks high-level goals; a deterministic skill library (crafting, smelting, full tool/armour sets, mining to ore depths, hunting for food & wool, beds, combat with weapon selection, dropped-item pickup) does the actual work — so even a small/free model plays competently instead of flailing.

Human-like camera — a custom Fabric mod eases every turn at full FPS. No snapping, no teleport-cam — it reads like a person playing.

Grounded commentary — reactions come from the agent's real game state (inventory, health, threats, current goal), fused through the same single pipeline as vision and hearing. No screen-guessing, no hallucinated "desert temples".

One-click install — Fabric + all mods straight from the dashboard's Play section. Vision/Hearing/Play toggle independently, on any persona.

Live2D avatar with emotion

Not just a mouth that opens and closes. Six animation layers run simultaneously over a single WebSocket, so the avatar feels like a person — not a puppet.

Viseme lip sync — spectral analysis of PCM audio estimates mouth shape in real-time. Front vowels (A/E/I) spread the mouth wide, back vowels (O/U) round it. Combined with RMS amplitude for jaw openness, attack/release envelope, noise floor gating, and a speaking smile baseline. The result: the avatar's mouth actually forms different shapes per sound, not just open/close.

Blink — periodic natural eye blinks (~3.8s int

[truncated for AI cost control]