Lessons from a weekend building local AI workflows
After building a multi-agent video editor over a weekend, the author shares three key lessons: the lost-in-the-middle problem, the bias compound problem, and that Whisper is not a silver bullet.
Like everyone and their grandmother, these days I am into Agents! I finally got to spend some time learning more about multi-agent workflows: I came up with a simple use case, built a first iteration and watched it shatter against the messy reality. Then I learned a few things.
This post shares three things learned: lost-in-the-middle, the bias compound problem, and that Whisper isn’t a silver bullet.
The tool I built sort of works and is available on GitHub. The whole thing is a multi-agent video editor which takes a video and outputs a shortened down version by removing all the fluff so just the juicy parts remain.
Don’t expect production ready magic, but I find it pretty entertaining :).
Naive solution
The first naive solution that came to my mind is the following:
graph TD A[Initial Video] -->|"raw video"| B[Speech To Text] B -->|"full transcript"| C[Editor Agent] B -->|"full transcript"| D[Reviewer Agent] C -->|"proposed cuts"| D D -.->|"❌ Rejected: retry"| C D -->|"✅ Accepted: cut list"| E[Video Editing Agent] E -->|"stitched video"| F[Final Video]
The plan: take a video, run it through a speech-to-text model to get the transcription, feed the full video transcript into an editor agent that decides what the most important segments are, then feed the full transcript and the selected segments to a Reviewer Agent tasked with deciding whether the selected sections of the video actually preserve the message. In this plan, the editor agent and the reviewer agent would go back and forth until the reviewer agent agrees with the selection made by the editor agent. Finally, FFmpeg stitches the final video together.
On paper? Flawless. In reality? The output looked terrible 🥹.
You can look at it yourself:
Original