Lessons from Building AI Agents for Financial Services

Lessons from Building AI Agents for Financial Services

2026-01-24 · 20 min read

I've spent the last two years building AI agents for financial services. Along the way, I've accumulated a fair number of battle scars and learnings that I want to share.

Here's what I'll cover:

The Sandbox Is Not Optional - Why isolated execution environments are essential for multi-step agent workflows

Context Is the Product - How we normalize heterogeneous financial data into clean, searchable context

The Parsing Problem - The hidden complexity of extracting structured data from adversarial SEC filings

Skills Are Everything - Why markdown-based skills are becoming the product, not the model

The Model Will Eat Your Scaffolding - Designing for obsolescence as models improve

The S3-First Architecture - Why S3 beats databases for file storage and user data

The File System Tools - How ReadFile, WriteFile, and Bash enable complex financial workflows

Temporal Changed Everything - Reliable long-running tasks with proper cancellation handling

Real-Time Streaming - Building responsive UX with delta updates and interactive agent workflows

Evaluation Is Not Optional - Domain-specific evals that catch errors before they cost money

Production Monitoring - The observability stack that keeps financial agents reliable

Why Financial Services Is Extremely Hard

This domain doesn't forgive mistakes. Numbers matter. A wrong revenue figure, a misinterpreted guidance statement, an incorrect DCF assumption. Professional investors make million-dollar decisions based on our output. One mistake on a $100M position and you've destroyed trust forever.

The users are also demanding. Professional investors are some of the smartest, most time-pressed people you'll ever work with. They spot bullshit instantly. They need precision, speed, and depth. You can't hand-wave your way through a valuation model or gloss over nuances in an earnings call.

This forces me to develop an almost paranoid attention to detail. Every number gets double-checked. Every assumption gets validated. Every model gets stress-tested. You start questioning everything the LLM outputs because you know your users will. A single wrong calculation in a DCF model and you lose credibility forever.

I sometimes feel that the fear of being wrong becomes our best feature.

Over the years building with LLM, we've made bold infrastructure bets early and I think we have been right. For instance, when Claude Code launched with its filesystem-first agentic approach, we immediately adopted it. It was not an obvious bet and it was a massive revamp of our architecture. I was extremely lucky to have Thariq from Anthropic Claude Code jumping on a Zoom and opening my eyes to the possibilities.

At the time the whole industry, including Fintool, was all building elaborate RAG pipelines with vector databases and embeddings. After reflecting on the future of information retrieval with agents I wrote "the RAG obituary" and Fintool moved fully to agentic search. We even decided to retire our precious embedding pipeline. Sad but whatever is best for the future!

People thought we were crazy. The article got a lot of praise and a lot of negative comments. Now I feel most startups are adopting these best practices.

I believe we're early on several other architectural choices too. I'm sharing them here because the best way to test ideas is to put them out there.

Let's start with the biggest one.

The Sandbox Is Not Optional

When we first started building Fintool in 2023, I thought sandboxing might be overkill. "We're just running Python scripts" I told myself. "What could go wrong?"

Haha. Everything. Everything could go wrong.

The first time an LLM decided to rm -rf / on our server (it was trying to "clean up temporary files"), I became a true believer.

Here's the thing: agents need to run multi-step operations. A professional investor asks for a DCF valuation and that's not a single API call. The agent needs to research the company, gather financial data, build a model in Excel, run sensitivity analysis, generate complex charts, iterate on assumptions. That's dozens of steps, each potentially modifying files, installing packages, running scripts.

You can't do this without code execution. And executing arbitrary code on your servers is insane. Every chat application needs a sandbox.

Today each user gets their own isolated environment. The agent can do whatever it wants in there. Delete everything? Fine. Install weird packages? Go ahead. It's your sandbox, knock yourself out.

The architecture looks like this:

┌─────────────────────────────────────────────────────────┐
│ User Sandbox │
├─────────────────────────────────────────────────────────┤
│ /private (read/write) │ User's personal files │
│ /shared (read-only) │ Organization files │
│ /public (read-only) │ Global resources │
└─────────────────────────────────────────────────────────┘

Three mount points. Private is read/write for your stuff. Shared is read-only for your organization. Public is read-only for everyone.

The magic is in the credentials. We use AWS ABAC (Attribute-Based Access Control) to generate short-lived credentials scoped to specific S3 prefixes.

User A literally cannot access User B's data. The IAM policy uses `${aws:PrincipalTag/S3Prefix}` to restrict access. The credentials physically won't allow it. This is also very good for Enterprise deployment.

We also do sandbox pre-warming. When a user starts typing, we spin up their sandbox in the background. By the time they hit enter, the sandbox is ready. 600 second timeout, extended by 10 minutes on each tool usage. The sandbox stays warm across conversation turns.

So sandboxes are amazing but the under-discussed magic of sandboxes is the support for the filesystem. Which brings us to the next lesson learned about context.

## Context Is the Product

Your agent is only as good as the context it can access. The real work isn't prompt engineering it's turning messy financial data from dozens of sources into clean, structured context the model can actually use. This requires a massive domain expertise from the engineering team.

### The heterogeneity problem

Financial data comes in every format imaginable:

- **SEC filings**: HTML with nested tables, exhibits, signatures
- **Earnings transcripts**: Speaker-segmented text with Q&A sections
- **Press releases**: Semi-structured HTML from PRNewswire
- **Research reports**: PDFs with charts and footnotes
- **Market data**: Snowflake/databases with structured numerical data
- **News**: Articles with varying quality and structure
- **Alternative data**: Satellite imagery, web traffic, credit card panels
- **Broker research**: Proprietary PDFs with price targets and models
- **Fund filings**: 13F holdings, proxy statements, activist letters

Each source has different schemas, different update frequencies, different quality levels.

Agent needs one thing: clean context it can reason over.

### The normalization layer

Everything becomes one of three formats:

- **Markdown** for narrative content (filings, transcripts, articles)
- **CSV/tables** for structured data (financials, metrics, comparisons)
- **JSON metadata** for searchability (tickers, dates, document types, fiscal periods)

Raw Source → Parser → Normalized Format → Chunking → Metadata Extraction → Index

### Chunking strategy matters

Not all documents chunk the same way:

- **10-K filings**: Section by regulatory structure (Item 1, 1A, 7, 8...)
- **Earnings transcripts**: Chunk by speaker turn (CEO remarks, CFO remarks, Q&A by analyst)
- **Press releases**: Usually small enough to be one chunk
- **News articles**: Paragraph-level chunks
- **13F filings**: By holder and position changes quarter-over-quarter

The chunking strategy determines what context the agent retrieves. Bad chunks = bad answers.

### Tables are special

Financial data is full of tables and csv. Revenue breakdowns, segment performance, guidance ranges. LLMs are surprisingly good at reasoning over markdown tables:

| Segment | Q1 2024 | Q1 2023 | YoY Growth | |--------------|---------|---------|------------| | iPhone | $45.9B | $51.3B | -10.5% | | Services | $23.9B | $20.9B | +14.3% | | Mac | $7.5B | $7.2B | +4.2% |

But they're terrible at reasoning over HTML tags or raw CSV dumps. The normalization layer converts everything to clean markdown tables.

Metadata enables retrieval

The user asks the agent: "What did Apple say about services revenue in their last earnings call?"

To answer this, Fintool needs:

Ticker resolution (AAPL → correct company)

Document type filtering (earnings transcript, not 10-K)

Temporal filtering (most recent, not 2019)

Section targeting (CFO remarks or revenue discussion, not legal disclaimers)

This is why meta.json exists for every document. Without structured metadata, you're doing keyword search over a haystack. It speeds up the search, big time!

Anyone can call an LLM API. Not everyone has normalized decades of financial data into searchable, chunked markdown with proper metadata. The data layer is what makes agents actually work.

The Parsing Problem

Normalizing financial data is 80% of the work. Here's what nobody tells you.

SEC filings are adversarial

They're not designed for machine reading. They're designed for legal compliance:

Tables span multiple pages with repeated headers

Footnotes reference exhibits that reference other footnotes

Numbers appear in text, tables, and exhibits (sometimes inconsistently)

XBRL tags exist but are often wrong or incomplete

Formatting varies wildly between filers (every law firm has their own template)

We tried off-the-shelf PDF/HTML parsers. They failed on:

Multi-column layouts in proxy statements

Nested tables in MD&A sections (tables within tables within tables)

Watermarks and headers bleeding into content

Scanned exhibits (still common in older filings and attachments)

Unicode issues (curly quotes, em-dashes, non-breaking spaces)

The Fintool parsing pipeline

Raw Filing (HTML/PDF) ↓ Document structure detection (headers, sections, exhibits) ↓ Table extraction with cell relationship preservation ↓ Entity extraction (companies, people, dates, dollar amounts) ↓ Cross-reference resolution (Ex. 10.1 → actual exhibit content) ↓ Fiscal period normalization (FY2024 → Oct 2023 to Sep 2024 for Apple) ↓ Quality scoring (confidence per extracted field)

Table extraction deserves its own work

Financial tables are dense with meaning. A revenue breakdown table might have:

Merged header cells spanning multiple columns

Footnote markers (1), (2), (a), (b) that reference explanations below

Parentheses for negative numbers: $(1,234) means -1234

Mixed units in the same table (millions for revenue, percentages for margins)

Prior period restatements in italics or with asterisks

We score every extracted table on:

Cell boundary accuracy (did we split/merge correctly?)

Header detection (is row 1 actually headers, or is there a title row above?)

Numeric parsing (is "$1,234" parsed as 1234 or left as text?)

Unit inference (millions? billions? per share? percentage?)

Tables below 90% confidence get flagged for review. Low-confidence extractions don't enter the agent's context. Garbage in, garbage out.

Fiscal period normalization is critical

"Q1 2024" is ambiguous:

Calendar Q1 (January-March 2024)

Apple's fiscal Q1 (October-December 2023)

Microsoft's fiscal Q1 (July-September 2023)

"Reported in Q1" (filed in Q1, but covers the prior period)

We maintain a fiscal calendar database for 10,000+ companies. Every date reference gets normalized to absolute date ranges. When the agent retrieves "Apple Q1 2024 revenue," it knows to look for data from October-December 2023.

This is invisible to users but essential for correctness. Without it, you're comparing Apple's October revenue to Microsoft's January revenue and calling it "same qu

[truncated for AI cost control]