Spriteloom – local AI pixel-art generator plugin for Aseprite

Notifications You must be signed in to change notification settings Fork 0 Star 1 BranchesTags Open more actions menu Folders and files NameName Last commit message Last commit date Latest commit History 149 Commits 149 Commits .github .github assets assets launcher launcher plugin plugin server server site site .gitignore .gitignore .luacheckrc .luacheckrc CONTRIBUTING.md CONTRIBUTING.md LICENSE LICENSE README.md README.md TODO.md TODO.md build.spec build.spec install-plugin.bat install-plugin.bat start-server.bat start-server.bat Repository files navigation Local AI pixel-art assistant for Aseprite. Generate sprites from a text prompt, edit existing sprites with an instruction, or redraw a selected region — all running on your own GPU. No cloud, no subscription, your pixels never leave your machine. Sample output and how it works → vkarach.github.io/spriteloom Generate · Edit with AI · Inpaint Rotate / Instruct, two different turnarounds All clips sped up for the demo. Real output, one generation each. Subjects like "tall man in a dark coat", "weathered explorer in a red leather coat", "wooden cottage, thatched roof". Scaled up with hard pixel edges for display. What it is A WebSocket server that runs a diffusion model locally, plus an Aseprite extension that talks to it. You stay in Aseprite the whole time; results open in a side window and drop in as new layers on click. It never edits your existing pixels. Four tasks, one panel: Generate — a sprite from a text prompt. Edit with AI — change an existing sprite by instruction ("make the sword glow blue"); style and everything unmentioned stays put. Inpaint Selection — same, but only ever touches the selected region. Rotate + Instruct — re-view the same subject from another angle. Everything runs on a single model (FLUX.2 Klein), so there are no model swaps: after the first load, tasks respond in seconds. Hardware you need — read this first This runs a 4B-parameter diffusion model on your own machine. It is not a lightweight tool: Requirement GPU NVIDIA, 12+ GB VRAM (developed on an RTX 5080); 8 GB GPUs work too, in Legacy 8 GB mode — see below RAM 16 GB minimum. Legacy 8 GB mode holds the whole ~16 GB model in system RAM, so free RAM plus the Windows page file must comfortably clear that — see below OS Windows Python 3.11+ Aseprite 1.3+ Disk ~15 GB for the model (downloaded during Setup) No NVIDIA GPU, no Spriteloom. There is no CPU fallback and no cloud option by design — the whole point is that it runs locally. VRAM modes Setup's VRAM mode dropdown picks how the ~8 GB transformer is fit onto the GPU: Auto (default) — detects free VRAM and picks bf16 on 12+ GB cards, Legacy 8 GB mode below that. bf16 — the whole model stays resident on the GPU. Fastest; needs 12+ GB free VRAM. Legacy 8 GB mode — for 8 GB cards (e.g. a laptop RTX 4060). The model doesn't fit on the GPU all at once, so it moves one layer onto the GPU right before that layer runs, then swaps it back out to system RAM. VRAM usage stays low the whole time (one layer resident, not the whole model), and so does GPU utilization: most of the time goes to shuttling weights over PCIe, not to compute, and that is the actual reason this is slow. Output is identical to bf16, just much slower. There is no faster 8 GB path currently. An fp8-quantized mode was tried and measured no better than this, while adding a real quality risk, so it was dropped. Legacy 8 GB mode keeps the whole ~16 GB model in system RAM (that is where layers are swapped from). On a 16 GB machine that only fits with a generous Windows page file, so leave it enabled and sized (or set it to system-managed). If free RAM plus page file cannot cover the model the load fails; the server now checks this at startup and prints why instead of crashing. Install Download the latest build from itch.io and unzip it anywhere. It has Spriteloom.exe, server/, and plugin/ together; nothing else to fetch first. Windows will likely show a "Windows protected your PC" SmartScreen warning — Spriteloom.exe isn't code-signed, so it has no reputation yet. Click More info, then Run anyway. Press Setup. It shows what is missing: the environment, PyTorch, the dependencies, the plugin, the model (~15 GB), a Start Menu shortcut. Tick everything and press Install selected. It builds the .venv, installs the packages and the plugin, downloads the model, adds a Start Menu shortcut, and prints a live log. Restart Aseprite once the plugin is in. Start stays disabled until every required piece — including the model — is in place; there is no lazy first-run download. Setup in action, sped up. If you would rather do it by hand: py -3 -m venv .venv .venv\Scripts\python -m pip install -r server\requirements.txt .venv\Scripts\python -m pip install torch --index-url https://download.pytorch.org/whl/cu128 install-plugin.bat, then restart Aseprite. With the .venv set up this way, start-server.bat starts the server in a console window instead of through the launcher. To build the exe yourself from a full clone of this repository: .venv\Scripts\python -m pip install -r launcher\requirements.txt, then .venv\Scripts\python -m PyInstaller build.spec --distpath . -- it builds straight into the project root as Spriteloom.exe, about 15 MB: the model and PyTorch stay outside it. Use Run Spriteloom.exe, press START and leave the window open. The dot turns green once the model is resident, about 25 seconds after a warm start. Closing the window stops the server. In Aseprite: Sprite → Spriteloom... (or press F1). Pick a task, fill the fields, press Run. Results open in a separate window; click a variant to insert it as a new layer. Generate understands full sentences: pick a View preset, name the Subject ("closed book with dark brown leather cover"), add Extra details if needed — the panel shows the exact text it will send. Edit / Inpaint take instructions, not a strength slider: say what to change and how much. Inpaint only touches the selection. Rotate / Instruct: name the subject explicitly ("four-legged brown horse", not "character"). Optional Mirror symmetry forces the result left/right symmetric, which can help a front/back view hold together - but it mirrors everything, including a one-handed weapon or an asymmetric pose, so it can also make those look wrong. Try it, don't assume it's always the better result. Advanced... opens a separate window with Background, Palette, Seed, and Extra (appended to the prompt for Generate and Rotate/Instruct; Edit and Inpaint ignore it). Background: Auto detects and strips a uniform background, Remove strips the dominant border color, Keep leaves it fully opaque. Palette: Auto derives colors per result; Current palette pins output to the open sprite's whole palette; Selected colors pins to only the swatches highlighted in the palette bar; Palette file pins to a .gpl/.pal/.png file so a batch of sprites shares one set of colors. History browses past generations (stored in output/), newest first; click a run to see its variants, click a variant to insert it. How it works +-----------------------+ +---------------+ +--------------------+ | Aseprite plugin (Lua) | | Python server | --> | FLUX.2 Klein (GPU) | | dialogs, results, | | protocol, | | single resident | | history, layer insert | | postprocess | | model, no swaps | +-----------------------+ +---------------+ +--------------------+ One model, one warm-up. Every task hits the same FLUX.2 Klein pipeline, so there is no per-task load/unload. On 12+ GB cards it stays fully resident; on 8 GB cards (Legacy 8 GB mode) each layer streams to the GPU per task instead, which is where that mode's extra time goes. WebSocket protocol with request validation at the boundary and streamed progress messages back to the plugin (server/protocol.py, server/main.py). Postprocess pipeline turns raw diffusion output into a clean sprite: crop to subject, palette quantization, background removal, mirror symmetry, fit-into-canvas (server/postprocess.py). The plugin is modular Lua with pure, unit-tested prompt assembly and a UI layer tested against a stubbed Aseprite API — see below. Quantization note 8-bit quantizing the Klein transformer produces pure noise in text-to-image (edits work fine). The shipped setup sidesteps this: 8-bit text encoder + bf16 transformer, fully resident. See TODO.md. Development Server tests: .venv\Scripts\python -m pytest server/tests/ --ignore=server/tests/smoke.py Prompt tuning without Aseprite (writes raw + postprocessed variants to output/): .venv\Scripts\python -m server.tests.smoke "demonic sword" --size 64 Plugin tests (needs scoop install lua luacheck, user-scoped): luacheck plugin\*.lua plugin\tests\*.lua lua plugin\tests\test_prompt.lua lua plugin\tests\test_panel.lua luacheck bundles Lua 5.4 (the version Aseprite runs); .luacheckrc declares the API globals it injects. test_panel.lua loads every module against a stubbed Aseprite API and repaints the status canvas in each server state — this catches broken cross-module calls without launching the editor. Layout and ghosting still need a real Aseprite. Plugin layout file holds main.lua entry point, registers the menu command dialogs.lua the control panel results.lua results window (fresh variants) history.lua history list and single-run windows ui.lua theme colors, checkerboard, variant grid, prompt preview sprite.lua frame/mask export, inserting variants as layers prompt.lua prompt assembly and key maps (pure Lua, unit-tested) client.lua WebSocket client base64.lua base64 codec Launcher layout file holds launcher/app.py the window, the JS bridge, window sizing launcher/ui/index.html markup and styles, main and setup screens launcher/server_proc.py the server subprocess, port probing, health launcher/plugin_install.py copying the plugin, version comparison launcher/paths.py finding the root, Python, Aseprite and the model launcher/setup_checks.py what the setup is missing, detection only launcher/setup_steps.py running the install steps in order server/config.py the settings, shared by the launcher and the server The launcher owns the settings file %APPDATA%\Spriteloom\config.json: the port, the VRAM mode, and the setup paths all live there, and every write merges so one key never erases another. The port defaults to 8765, and Install stamps it into server.json next to the plugin, so both ends agree. The server runs inside a Windows job object that dies with the launcher. That is what keeps a crashed or killed launcher from leaving a server behind holding your VRAM. Troubleshooting If the server says it crashed or never finishes loading, run the built-in loader check from the install folder: .venv\Scripts\python -m server.diagnose It prints package versions, memory headroom, and whether the model files are complete, then loads the transformer on its own. TRUNCATED means the download was cut short — delete the model folder and re-download via Setup. A crash only in the full run, with the isolated load passing, is memory: close other apps or raise the Windows page file. The server runs the same file and memory checks at startup, so this is mainly for a closer look. License Apache 2.0. The model, FLUX.2 Klein 4B, is licensed separately by Black Forest Labs under Apache 2.0 and downloaded at runtime, not redistributed here. Resources Readme Apache-2.0 license Contributing Contributing Activity Stars 1 star Watchers 0 watching Forks 0 forks Report repository