待翻譯：How to extract meaning from charts and tables in PDFs

AI 服務暫時不可用，以下為來源正文，待恢復後補全翻譯。

Intro​ If you have ever tried to put a stack of investor decks, scientific papers, or annual reports through a RAG pipeline, then you know the drill: set up an Optical Character Recognition (OCR) or text-extraction step, pick a chunking strategy, embed text, and finally retrieve information. After going through all this, someone may ask a question about revenue in Q2 FY25, and your retrieval would return three pages of unrelated bullet points because the actual answer is in a bar chart that was invisible to your index. This used to be what people had to do — just leave out the interesting parts of a PDF from RAG such as bar charts with trends, comparison tables, architectural diagrams, and things in general that make PDFs so much more valuable than mere text. In this article, we'll show you a better way that not only retrieves the rich information embedded in charts, but also eliminates complex processing steps. We'll cover: Classic RAG: Why OCR and text embedding works great for some (structured and unstructured) data, but not for rich PDFs. Late Interaction RAG: What late-interaction multi-vector models are, and why they let you skip text extraction entirely. Drag-N-Drop and Done: How to ingest multiple PDFs in Weaviate Cloud with just a few clicks. Real Examples: Example queries against NVIDIA's FY2026 quarterly earnings, where each top result is the exact chart that answers a question. Complex Agentic Reasoning: How to wrap the same data with the Weaviate Query Agent for synthesised answers with page-image citations. Codified and Deployment-Ready: The same ingestion pipeline in roughly 50 lines of Python for when you need to deploy this in production. If you'd rather just see the demo first and read the explanations later, skip to the queries section below. How traditional RAG falls short​ Let's first look at what most people do today when it comes to building a RAG pipeline for PDFs: OCR (or text-extract) the file (using either a python library or third-party tooling). Chunk the text. Embed the chunks with a text embedding model. Retrieve, (maybe) rerank, generate. There's nothing inherently bad about this workflow, but there's a naive underlying assumption that the page can be reduced to a sequence of text tokens without losing the richness of the content. For a press release or a Wikipedia article, this is mostly fine, but for a slide deck full of charts, a 10-K with comparison tables, or an academic paper with illustrative figures, it's simply not enough. You could build a complex ETL pipeline to extract charts and vectorize them separately, but that means another model in the pipeline and introduces more complexity into the stack as well as merging problems at query time. With a late-interaction multi-vector model, you don't need any of those: You embed the page, and not the text within it. The model sees the chart the way you do. Yes, you heard that right, there is not even a chunking step — the page is the chunk. Late-interaction multi-vector retrieval, in two paragraphs​ Before we get to the demo, let's take a quick detour on what makes this work. If you already know your way around ColPali / ColBERT, feel free to skip or skim. Traditional dense embedding models compress an entire document (or chunk) into a single vector. Late-interaction multi-vector models do something different: they encode the document as a set of vectors, typically one per token (or for a vision model like the one we're about to use, one per image patch). At query time, your query is also encoded as a set, and the relevance score is the sum of best matches between query tokens and document tokens (a quantity called MaxSim). Instead of asking "is this whole page about my whole question?", the model can ask "is the part of this page that talks about Q4 FY26 a good match for the part of my question about 'change over time'?". For a chart-heavy page, this is exactly the granularity that's needed. The model doesn't have to summarise an entire slide into a single vector, but can keep one vector per region of the page, and the query can pick out the regions that matter. So really, there are two distinct advantages here: Because it's a visual model you keep the layout, the charts, the tables, etc. and By using MaxSim over a set of vectors, you eliminate the need for chunking. Weaviate offers multi2multivec-weaviate, a vectorizer module that runs a hosted late-interaction multi-vector model on Weaviate Cloud, so you don't have to host or manage a model in order to generate multi-vectors from your PDFs. Ingesting PDFs by Drag-and-Drop​ The fastest way to try this is the drag-and-drop importer in Weaviate Cloud. Open your cluster in the Console. Go to Collections and create a new collection. Select the upload from file option and drop in your PDFs. And that's the entire ingestion process. Behind the scenes, Weaviate is doing three things: Rendering each PDF page to a high-resolution image. Storing that image as a BLOB property on a new collection. Vectorizing it with the multi2multivec-weaviate module, producing many vectors per page. A few things are worth noting: One object equals one page. The unit of retrieval is the page, which is also the unit a human navigates a document by. No OCR. The model never sees the text as text. It sees the page as an image. That's why a chart with no caption is just as searchable as a paragraph. The vectors are compressed. Late-interaction models can produce hundreds of vectors per page, which would be expensive to store naively. Weaviate uses a multi-vector encoding scheme that keeps the index compact. (More on that also in the trade-offs section below.) For this demo, we've imported NVIDIA's four FY2026 quarterly investor presentations (Q1 through Q4). They cover the financial year ending in January 2026, contain wall-to-wall charts and tables, and total 92 pages. The whole import took about a minute and a half. Interrogating NVIDIA's Quarterly Earnings​ Before we do anything fancy such as agentic reasoning on the data, we want to show you the raw retrieval results, as they are already impressive on their own. With ingestion complete, you can query the data directly in the Console (or with any Weaviate client). The query is in plain English, and the result is a ranked list of pages with page images inline. Let's walk through three queries and show you what results are returned. Query 1: "how did automotive revenue change over time?"​ The top result is a single page from the Q2 FY26 deck, titled Automotive. The left half is a bar chart showing five quarters of revenue ($346M → $449M → $570M → $567M → $586M, +69% Y/Y). The right half contains three bullet points about Thor SoC and DRIVE AV. The phrase "over time" doesn't appear anywhere on this page. Neither does the word "change". The model didn't match against text semantically, but rather the image of the page. What it saw was five bars of increasing height with quarterly labels, and that was enough to identify the page as a match for a question about a temporal trend. Query 2: "gross margin trend"​ The top result here is the Q2 FY26 Financial Summary page. On the left is a combination chart: revenue bars and a non-GAAP gross margin line over five quarters. On the right is a GAAP/non-GAAP KPI table with Y/Y and Q/Q deltas. Again, nothing on this page literally says "gross margin trend", but there is a line chart that visualises the gross margin dipping from 75.7% to 61.0% in Q1 FY26 and recovering to 72.7% in Q2 FY26. That's what a trend looks like, and it's what the model retrieved. A side note: the same page also contains a detailed financial table. That makes it a particularly useful retrieval target if you're then going to ask follow-up questions like "by how many basis points did gross margin recover Q/Q?" The answer is sitting on the page the model already returned. More on that below when we introduce the Query Agent. Query 3: "how did data center revenue change over time?"​ The top result is the Q4 FY26 Revenue page — a Y/Y bar chart ($39.3B → $68.1B) with a callout that data center revenue is up 13x since the emergence of ChatGPT. This is a great example that the model still respects text when it's the better match. In this case the chart is less relevant, but the box stating the exact answer is what returned the highest similarity (MaxSim) on this page. So you get the best of both modalities. The runner-up is the dedicated Data Center page from a different quarter, which splits the segment into Compute and Networking: Notice how the second-best match isn't simply "another page about data center", but a different kind of answer — the same revenue, broken down differently. That's a useful property for an agent that wants to triangulate across multiple views of the same underlying number. Speaking of which... From search to answers: the Weaviate Query Agent​ Vector search returns results, but sometimes you want an answer. The Weaviate Query Agent is a managed agent that wraps vector retrieval with multi-step reasoning, source citations, and inline page images. It is available out of the box for any Weaviate Cloud cluster. Simply point it to a collection and ask a question. If we ask "How did automotive revenue change across FY26 quarters? What's driving it?" about the same collection, the agent comes back with a synthesised answer (the numbers from the bar chart, plus the bullet points about Thor SoC and DRIVE AV adoption) and the underlying page images as citations. The agent decided which pages to retrieve, looked at them visually, and quoted directly from the slide. Open the Sources panel and you'll see exactly which pages backed the answer. Among the citations is the Automotive page from the Q1 FY26 deck — the same kind of bar chart we surfaced in the raw vector search earlier, just for a different quarter. Every numerical claim in the response is anchored to a specific page in a specific PDF, with the page image right there for verification. You don't have to blindly trust the agent, but can read the contents of the slide for yourself. Building a deployable pipeline with Python​ The drag-and-drop UI is the fastest path to creating a POC, but most production pipelines need code, and here is the equivalent in roughly 50 lines. First, install the dependencies: pip install "weaviate-client>=4.21" PyMuPDF Then, render each PDF page to a 2000-pixel PNG and store it as a BLOB in a collection vectorized with multi2multivec-weaviate: import os from base64 import b64encode from pathlib import Path import fitz # PyMuPDF from weaviate import connect_to_weaviate_cloud from weaviate.classes.config import Configure, DataType, Property def page_to_b64(page, long_edge: int = 2000) -> str: scale = long_edge / max(page.rect.width, page.rect.height) pix = page.get_pixmap(matrix=fitz.Matrix(scale, scale)) return b64encode(pix.tobytes(output="png")).decode() client = connect_to_weaviate_cloud( os.environ["WEAVIATE_URL"], auth_credentials=os.environ["WEAVIATE_API_KEY"], ) if not client.collections.exists("PDF"): client.collections.create( name="PDF", properties=[ Property(name="pdf_name", data_type=DataType.TEXT), Property(name="page_number", data_type=DataType.INT), Property(name="page_image", data_type=DataType.BLOB), ], vector_config=Configure.MultiVectors.multi2vec_weaviate( image_field="page_image", ), ) col = client.collections.get("PDF") for pdf_path in Path("pdfs").glob("*.pdf"): with col.batch.fixed_size(batch_size=2) as batch, fitz.open(pdf_path) as doc: for i, page in enumerate(doc, start=1): batch.add_object(properties={ "pdf_name": pdf_path.name, "page_number": i, "page_image": page_to_b64(page), }) client.close() A few notes on the above: The Python call MultiVectors.multi2vec_weaviate(image_field="page_image") configures the multi2multivec-weaviate module. This is the same vectorizer used by the Cons [truncated for AI cost control]