How to ground Genie Agents in both structured data and documents without losing governance

How to ground Genie Agents in both structured data and documents without losing governance | Databricks Blog

Skip to main content

• Ground Genie Agents in structured data (Managed Tables, External Tables, Foreign Tables, Views, Metric Views, and Materialized Views) and unstructured files (Unity Catalog Volumes) so one agent can answer across all of it.

• Agent governance lives in the catalog layer, not at the model layer. Governance scales with the agent instead of growing out of control.

• With Automatic Identity Management (AIM), Object Privileges, ABAC, Row Filters, and Column Masks in Unity Catalog, Genie Agents run as the user’s identity, and every answer is filtered by that user's permissions.

Building an agent to automate simple business tasks can be easy. But creating one that actually understands your business and respects your existing data governance is much harder.

For a long time, teams had to use separate systems to analyze structured and unstructured data, often spending weeks just to bridge the two. By enabling analysis directly from tables and unstructured files, Genie Agents simplify this architecture, allowing you to ground a single agent with both structured and unstructured data.

As you consolidate this data, a critical question emerges: if one agent has access to everything, what stops it from telling the wrong person the wrong thing?

The good news is that with Databricks, the answer exists within the data governance foundation you already have. The same Unity Catalog mechanisms you rely on today (identity sync, object privileges, ABAC, row filters, and column masks) automatically govern Genie Agents without any additional setup. This seamless inheritance relies on a well-architected governance strategy, which we will explore in detail.

To bring these concepts to life, we will walk through these scenarios using examples from Brickstore, a fictional global brick retailer, as a reference point.

The governance contract: Genie Agents run with the end user’s credentials

The core architectural principle is simple: Genie Agents run with the end user’s credentials. Unity Catalog enforces governance by default, ensuring that access to tables and volumes are tied directly to the end user's existing identity and permissions.

This is critical, because many homegrown systems grant agents broad access and rely on prompt engineering to filter results at the model layer. This effectively makes the LLM your security perimeter—a dangerous bet, given that models can be manipulated or bypassed. Telling an auditor that "I added instructions that said to not to show restricted data" is not a defensible governance control.

With the governance framework outlined in this article, Unity Catalog, not the model, remains your security perimeter, just as it does across the rest of Databricks. While Genie determines how to query the data, it is  incapable of returning a record the end-user is not authorized to see, as every answer is filtered at the data layer before it ever leaves the Lakehouse.

Step 0: It starts with identity: Automatic Identity Management (AIM) and Just-in-Time (JIT) provisioning

The architectural foundation begins with ensuring your enterprise identities are both precise and current.

Access controls are fundamentally only as reliable as the identities they evaluate. A policy that says "members of brickstore_apac can only see APAC orders" is meaningless if your group memberships in Databricks are a stale, hand-maintained copy of what's in your identity provider.

Automatic Identity Management for Microsoft Entra ID and Okta closes that gap. When enabled, users, groups, group memberships, and service principals sync from those identity providers into Databricks automatically, with no SCIM application required. Just-in-time provisioning is always on, so a user who has never logged into Databricks is provisioned on first login and arrives already carrying their existing group memberships.

Here is the flow step-by-step:

IdP is the source of truth. Someone joins the APAC sales org; your Identity Provider puts them in the brickstore_apac group.

AIM syncs that into Databricks — including the group membership. JIT provisions the user on Databricks the first time they open Genie One.

Unity Catalog policies key off those groups — object privileges, ABAC policies, row filters, and column masks all evaluate group membership at query time.

The user asks a Genie Agent a question, and the answer is shaped exactly by what their group permission allows. No more, no less.

The payoff is that governance is continuous, not a point-in-time setup. When an employee transfers from APAC to AMER, the IdP moves them between groups, the sync propagates it, and the very next question they ask Genie returns the AMER view — without anyone filing a ticket or making changes to the Genie Agent. When the employee leaves the company, they are deactivated from the IdP and their access to every Genie Agent is immediately removed.

Step 1: Grounding structured data and controlling four layers of access

Once identities are properly established, we can now focus on what they’re allowed to see. For structured data, a Genie Agent can access any Unity Catalog data asset—tables, views, materialized views, metric views, streaming tables, and even foreign tables federated from external systems.

For example, Delta tables are the facts and dimensions. In Brickstore that's brickstore.sales.orders (every order, with a region and a customer_email) and brickstore.sales.products (the brick catalog). Metric Views are the governed semantic layer on top — they encode the definitions of your business metrics (e.g. what "net revenue" means, how "bricks sold" is calculated, what counts as a "top-selling brick") once, in YAML, so every consumer computes them the same way.

On top of those assets sit four layers of access control that people routinely blur together:

Layer

Question it answers

Mechanism

Object Privileges

Who has what level of access to what resource?

GRANT SELECT on the catalog/schema/table

Attribute-Based Access Control (ABAC)

Which policy applies, and to what?

Governed-tag-driven policies that attach once and propagate (ex: any column with the tag "PII" is only available to certain groups)

Row filters

Which rows does a user have access to?

SQL user-defined function (UDF) that evaluates each row at query time (rows where the function returns FALSE are excluded from query results)

Column Masks

Which columns should be masked and how?

SQL UDF that takes the column value as input and returns the original value or a masked version

Object privileges is the first layer of access: without SELECT, Genie can't query the table on the end user’s behalf. But granting access to a table doesn't mean you need to grant all of it. You layer row filters and column masks on top of those grants, so a regional manager can query the orders table while only ever seeing their own region's rows and never the raw customer email. Those row and column controls key off the very same groups your grants already use — is_account_group_member('brickstore_apac') and the like. ABAC, up next, doesn't replace any of this; it's just a way to attach the same filters and masks by a policy instead of table-by-table.

ABAC: define the policy once, let it propagate

The old way to do row and column security was per-table: write a row filter, attach it to orders; write a column mask, attach it to another table; repeat forever. It still fits one-off logic, but across hundreds of tables it's a gap-prone configuration.

ABAC policies, which are now GA in Unity Catalog along with governed tags and automated data classification, invert that. You tag sensitive data with governed tags (account-level, access-controlled key/value pairs like pii:email), and you write one policy that says "wherever this tag appears, apply this protection." New tables inherit the protection the moment they're tagged, so there is no per-table work.

A column mask + ABAC Policy that protects every email column in the catalog, in one statement:

And a row filter + ABAC Policy so each manager sees only their region's orders, driven by group membership:

The result: the APAC manager asks a question about orders and the Genie Agent returns APAC rows only, with customer_email masked. The AMER manager queries the same table and gets AMER rows.

Step 2: Extending the same governance to documents

Historically, the governance strategy for teams has been more challenging when dealing with unstructured data. While structured data is securely managed in a data warehouse or a database, documents are often kept in an isolated storage system governed by separate ACLs.

The fix is to keep files inside the same governance plane as your structured data. You can land them in Unity Catalog Volumes and they become securables like everything else. You GRANT READ VOLUME to the groups and users that should see them, and Genie reasons over them under the same identity contract:

One behavior is worth understanding before you design your agent: when you attach a volume to a Genie Agent, it becomes a required source. This means that the agent validates access to every attached source when it loads, so a user who lacks READ VOLUME on an attached volume can't use that agent at all. In other words, volume grants govern documents as a prerequisite to using the agent so make sure to scope each agent's document sources to the audience that should use that agent. If two audiences need different documents, you may need to give them different Genie Agents (each mounting only the volumes that the user can read).

Also keep in mind that a Unity Catalog Volume is the smallest securable unit, so permissions apply to the entire volume rather than to individual files. You cannot pick and choose specific files to share; you must grant access to the entire volume or none of it.

With these considerations in mind, volumes can be attached directly to Genie Agents as a knowledge source in the same way you would do so for a table or view. Genie Agents read well beyond PDFs — supported formats include PDF, image files (JPG, JPEG, PNG, TIFF, TIF), and Office documents (DOC, DOCX, PPT, PPTX), along with plain text and Markdown. In practice that means scanned contracts, slide decks, and spec sheets are all fair game, not just clean PDFs. (See the Genie Agents volumes documentation for the full list and current limits.)

To ensure accurate routing and optimal performance, follow these best practices for configuring your volumes:

Add a clear description: Describe exactly what content the volume contains, how it is organized, and how the agent should use it. Do not use generic placeholders. For example, instead of "regional files," use "APAC market report — demand drivers, trends, and watch items for the APAC region." Genie relies on this description to select the right volume.

Avoid duplicate content: Attaching multiple volumes that contain overlapping information makes it harder for the agent to retrieve relevant documents. The same applies to individual files within a volume.

Avoid irrelevant files: Include only files that are relevant to the agent's domain. Irrelevant files can confuse the agent.

Use clear file names: Use descriptive file names so the agent can distinguish between files.

Step 3: Genie Agents in production: Same question, different answers

At this point, the Genie Agent is fully empowered to act as a true domain expert -  with full access to both structured and unstructured data. The underlying knowledge base is still thoroughly protected by row-filters, column masks, and volume grants based on per-user permissions.

We benchmark our implementation by testing it with two different users asking the exact same question, which should yield two uniquely correct responses.

Consider two concurrent Genie Agent sessions, both grounded in the identical assets—the orders table, the products catalog, and the market_report volume. While one requester belongs to brickstore_apac and the other to brickstore_amer, they both submit the exact same query:

"Which product is our top seller this quarter and what's driving that demand? Also list the top customers behind those sales and their emails."

Response 1 - for the APAC Manager

Response 2 - for the AMER Manager

There are three things worth pointing out:

The numbers are different, and both are correct. Both managers' "top-selling bricks" query pull data from the same tables — the difference is purely the rows each is entitled to, not a difference in how the metric was computed.

The difference required zero per-user prompt engineering. Nobody wrote "if the user is APAC, hide other regions." The agent's instructions are identical. Unity Catalog did the filtering at query time — both on the rows and the masked column. Notice that the email column is masked to protect PII.

Structured data, enriched by unstructured knowledge. Without the regional docs, Genie may have been able to identify the “what” question easily, but would’ve struggled to figure out what factors are fueling the demand. With unstructured data available, Genie has full context into the business.

Patterns to watch for

A few patterns to watch out for as you push the learnings from this blog to production:

Tag, then policy. Don't mask table-by-table. The instinct is to secure the three tables in front of you. Resist it. Define governed tags and ABAC policies to future-proof your data governance.

One audience per volume. Because the volume is the smallest grantable unit, decide document access at the volume boundary. If two documents need different readers, they need different volumes and different Genie Agents — plan the layout upfront.

Be careful when surfacing Genie One or Genie Agents externally via MCP or API - you must handle identity carefully. Unlike running it through the Databricks UI, you are not always granted to use the end user’s identity (ex: when using a Service Principal for auth). There are specific patterns to adopt, and Databricks details the U2M, M2M, and OBO configurations in Access Genie everywhere.

Test by impersonation, not inspection. Don't validate governance by reading the policy and convincing yourself it's right — ask the same question as a member of each group and compare the responses. Make it a regression test and run it whenever policies or groupings change.

The takeaway

Building an agent can be easy, but governing it takes real design work. In Databricks, Enterprise identities are synced from the IdP, object privileges gate access, ABAC and governed tags apply protection at scale, row filters and column masks control what comes back, and documents are kept in the same system as the data.

With this setup, Genie Agents inherit all of the governance without any additional configuration.

To get started building your first governed Genie Agent, visit the Genie documentation and the ABAC policies documentation.

Get the latest posts in your inbox

Subscribe to our blog and get the latest posts delivered to your inbox.

Sign up

View all blogs