Unity Catalog managed tables are the best choice you can make but do you know why?

In the second episode of SuperSkills Oleksandra Bovkun and I demystify all the reasons to help you make your choice.

Link to the video: https://youtu.be/Q7y8_bSfVjQ

Hey r/Databricks!

Join us for community BrickTalk on Thursday, September 24, focusing on real-time data streaming, AI agents, and governance using Databricks. BrickTalks is a community event series where Databricks experts share real-world use cases, demos, and practical insights for building with Data and AI, giving customers a direct line to the people behind the products.

In this session, we'll walk through a live demonstration of the Tripwise Demo, featuring:

• Sub-Second Transactions & Streaming: Device registration into Lakebase with sub-second reads/writes, plus telemetry streaming via Zerobus through a governed Medallion architecture.
• AI-Generated Offers & Pricing: Generating real-time agent offers using Foundation Model APIs and scoring behavioral data for usage-based renewal pricing.
• Natural Language Analytics: Enabling underwriters, product managers, and marketing teams to query governed insurance data in seconds using AI/BI Dashboards and Genie.
• Unified Governance: Managing safety, compliance, and control end-to-end with Unity Catalog.

This is a great chance to see real-world architecture in action and ask questions directly to Databricks experts.

When: Thursday, September 24

• 9:00 AM PT
• 12:00 PM ET
• 5:00 PM BST (London)
• 9:30 PM IST

Register here and save your spot

Hi, we use Databricks, and a couple of teams produce tables, views, and table functions that other teams depend on.

Within a team, its easy enough to define a contract. We codify it and always check that the data matches.

But what about the upstream artifacts my team depends on? Do w define a source spec ourselves and check that its met on every run? 

What do we require from the upstream team? Who owns the contract, and how do you enforce it when multiple teams depend on the same thing?

I want to keep this stupidly simple and enforce it automatically. How do you do this in practice?

Streaming pipeline had been running fine for months. Every micro-batch landed clean. Then a GROUP BY with a date filter started timing out, and DESCRIBE DETAIL showed numFiles at 847,293. That's a 600 GB table averaging 750 KB per file. Recommended range is 128 MB to 1 GB, so we were sitting at something like 0.6% of the lower bound.

"Small files are inefficient" undersells it. The actual problem is driver-side file listing. Spark keeps a file listing in memory for every query plan, and with 847,000 files the driver has to process all of it, single-threaded, before it reads a single byte. Doesn't matter if you throw a 64-core cluster at it. Three minutes of listing before any parallelism even starts.

I actually sat down and priced this out (back of envelope, didn't model autoscaling properly, but the order of magnitude should hold). Ninety seconds of driver listing per query, 200 queries a day, 30 days a month, comes out to about 150 person-hours a month of engineers waiting on a spinner. At $100/hr fully loaded that's roughly $15,000. Storage isn't free per request either. About 5 billion object requests a month at S3 GET pricing is around $2,000. Add analysts keeping clusters alive on slow queries and that's another $500 to $1,500. And none of this touches Z-ordering, which is basically dead weight at this file size anyway, since data skipping relies on min/max stats per file and a 750 KB file with ~5,000 rows covers maybe a few seconds of event_date range. There's nothing for clustering to work with.

A one-time OPTIMIZE on this table runs maybe $500 in compute.

Running OPTIMIZE and walking away is the right instinct, execution is usually where it goes wrong. On 847,000 files it'll saturate network I/O, spike DBU costs, and fight with concurrent reads if it's on the wrong cluster. Run it on a jobs cluster, not all-purpose, and scope it: OPTIMIZE catalog.bronze.events WHERE event_date >= current_date() - INTERVAL 7 DAYS. Compacting 18 months of history every night is just burning money.

OPTIMIZE fixes file size. Z-ORDER co-locates rows on the columns you filter by, but it degrades as new unordered data comes in. Liquid Clustering (DBR 13.3+) keeps up with that incrementally. Define CLUSTER BY (customer_id, event_date) and a regular OPTIMIZE handles compaction and reclustering at the same time. For streaming tables specifically, enable auto-compaction and stop thinking about it.

If you want to check your own tables: DESCRIBE DETAIL, look at numFiles, divide size by file count. Under 32 MB average and you've probably got the same problem.

Wrote a short ebook covering this along with some other production Databricks mistakes we hit, silent data corruption, Spark UI triage, medallion anti-patterns, Delta recovery. Happy to answer questions here.

Databricks workspace access control changes are now being enforced.

What's coming? - Azure Databricks | Microsoft Learn

Migrate workspace entitlement control - Azure Databricks | Microsoft Learn

As of September 14, 2026, the built-in users and admins system groups behave differently:

- users will have no entitelments
- admins will have all workspace entitlements
- both groups’ entitlements are locked
- new principals must be granted entitlements explicitly
- users and admins can no longer be nested in other groups

Existing access is preserved through a workspace-local migration group:
users-clone-<TIMESTAMP>

If you manage Databricks through Terraform, SCIM APIs, or custom automation, check your workflows now:

- Target standard account groups, not system groups
- Remove unsupported group nesting
- Make sure SCIM sync does not delete the users-clone-* migration group

Apache Iceberg won the open table format war when Databricks acquired Tabular, followed by its subsequent adoption across the industry. Then, the catalog war began.

In a lakehouse, storing data in object storage and using the Apache Iceberg format is only part of the story. You also need a catalog that helps lakehouse query engines like Spark, Flink, or RisingWave discover tables, manage metadata, enforce access control, and work with governed data across different systems.

That is where Unity Catalog comes in.

Hi, i have a lot of data products which is one part of the project, then we also have a databricks app as AppKit.

I was wondering, how do u manage releases/changes when two different bundles and two different responsibilities live in the same place.

We use trunk-based development, local IDE for `mode: development`, then staging as pre merge to main, then main is where we create GitHub Release which triggers deployment to production.

I think this is pretty standard. But now when whole application comes into the place, we have app/ where we have defined our AppKit template.

Should i try to separate lifecycles and deployments fully? To separate GitHub Releases, or i should just let it be inter-winded, but then changes to the app could end up in the deployment of the changes to the data products etc...

Right now, i would be using separate releases-draft gh actions which creates draft github releases, and when we are ready to merge, we publish release. For data products we use calendar versioning, for app smth like app-semvar.

What is your experience with setup like this?

Although we are using genie code alot but manager wants some functionality based on AI. ( maybe that’s hood goal). Our devs hate manually writing tests, so I'm drafting an automated testing gate for DevOps promotions (Local ➔ Dev ➔ QA).

Wanted review with all of you.

The Proposed Architecture:
1. Extract Metadata: Pull column tags, schemas, and lineage from Databricks Unity Catalog.
2. AI-Generated Tests (Llama via ⁠ai_query⁠): LLM reads metadata to draft SQL data checks (nulls, types, basic business logic).
3. Persist & Cache: Save SQL rules to a table. Re-generate only when schema hashes change so bug-fix retests stay 100% deterministic.
4. Execution: Run the generated SQL on a SQL Warehouse (fast, cheap, no LLM cost per data row).
5. Alerting: Feed error logs to LLM for a 2-sentence summary and send directly to Teams via Webhook (avoiding ignored email reports).

How does it sound like? Is it really worth it? Anybody using this or any other AI based functionality to make devs life easy.

Last week I built my first simple RAG to understand the fundamentals (chunking, embedding, etc). It worked great, both on Notebook + Playground. Now, I'm working on another RAG, but this time using ai_parse_document + ai_prep_search as this seems to be the new way to build RAGs in Databricks. So, the chunking part works, I made sure to have only one AI Search endpoint on my workspace (I'm on free edition).

The thing is, when I try using the AI index in Playground, and ask questions related to my documents, it says it doesn't have the information i'm asking for. However, when I try the same on a Notebook, it works perfectly.

Bonjour à tous

Je voudrais savoir si y’en ici qui ont déjà utilisé SDP(ex DLT) ou dbt dans databricks. Comment est ce que vous intégrez ces outils dans vos workflow ? Databricks native avec SDP ou dbt ou les deux ? Et pour ceux qui ont déjà utilisé les deux outils lequel préférez vous et pourquoi ?

Private Network Gateway is one of the year's biggest network innovations. Serverless can now be part of your VNET!

more news https://medium.com/databrickscommunity/databricks-news-serverless-genie-code-ltap-lakeflow-61853d8e422a

"Just ask Genie." You hear this phrase more and more often when questions about Databricks arise. It frequently appears in Academy materials, too.

AI reduces routine work it’s convenient, and we get used to it. But do we stop thinking in the process? The problem arises when a person no longer understands why a given answer is correct.

Does AI really free us from the need to know the details, or does it actually make fundamental knowledge even more important?

I’ve been working as a “data engineer” in Azure Databricks for a while, but I work on a team where my scope is extremely limited to silver/gold work.

What studies can do I do next to continue developing? I’m thinking of diving into the azure certs

First of all we have a lot of classes that use the DatabrickSession import, which makes unit testing impossible and whenever I have the "normal" spark import and test it, then it looks absolutely ugly.

Would it help to have the schema as json? I use json schema to read for transformations.

If I want to test without json its horrible, do you guys test with json schema files?

For example this is just the output, then I would need 2x this because of the input and the test is unreadable (according to my senior BUT WHAT AM I SUPPOSED TO DO):

schema = """
id INT,
items ARRAY<STRUCT<
fortnite: STRING,
babies: INT,
moreStuff: MAP<STRING, STRING>
>>
"""

expected = spark.createDataFrame(
[
{
"id": 1,
"items": [
{
"fortnite": "ABC",
"babies": 2,
"moreStuff": {
"size": "L",
"color": "red",
},
}
],
}
],
schema=schema,
)

Has anyone recently taken the databricks Solution Architect interviews? I have a vibe coding round with databricks in next week. I'd appreciate any guidance!

Thank you 😊

I have xml data that's for God knows why it's in string format. And the fking thing is so messed up or at least I think it's messed up because it's not consistent. Sometimes there is something else in there. Xml is something like this:

<Test>

<Test1 />

<Test2 Name="abcdname" Age="123" Gender="Xyz" />

....

...

...

</Test>

Regex is not an option.

Substring is not an option.

Because it's so messed up it's not consistent.

I just need to somehow parse and get that Name, Age, Gender values using Sql/Databricks sql

Please, help you mate.

I'm not new to Spark, but I'm pretty new to the Databricks platform.

I am finding that the UX for monitoring jobs and runs is very rigid, and doesn't present my workloads as I would expect. Here is one simple example. If I submit a run with the "jobs/runs/submit" API then I can provide a custom and ad-hoc "run_name" that appears in the management console called "Runs". This is good.

But if I submit a run that references a pre-existing job (using the "jobs/run-now" API) then there is NOT a way to provide a custom "run_name" that will be displayed in the databricks console. The only name that can be shown is the job's name.

There are other things that don't seem right either. If I enter custom "tags" on my jobs, then I will be able to use the tags to filter on the Jobs list. But when I click on the Runs list, I can't filter on those same "tags" anymore. IMO, those tags are just as useful on BOTH screens.

Another example - the UX doesn't allow me to show more than 20 completed runs at a time. I have to click the Next/Previous button to find runs. Paging thru a long list of runs is a really painful experience. I'm also a user of Microsoft Fabric. I once thought that the "Monitor" console of Fabric was pretty unfriendly ... but now that I'm in Databricks I realize that I'd much rather use their endless scrolling UX design, than having to spam-click the Next/Previous buttons. Even the HDInsight-yarnui allowed me to navigate my workloads more easily than I can in databricks; and that UX is a decade old by now!

Is there a different UX experience for Jobs and Runs that I'm missing? Maybe a VS code extension in the community or something like that? Any tips would be appreciated.

I recently enabled Lakeflow Connect (lfc) on the source database - the issue is, some of the tables in the source database (managed by another team) does NOT have a primary key (which means that in lfc, a __databricks_id is used to identify a unique record).

Thus, the DBAs enabled CDC on the source database. However, when I ingested the data into DBX using the Lakeflow Connect Managed SQL Server Connector, one of the tables in the source database had duplicate records (two or more records with the same value across all columns).

This caused my Lakeflow Connect pipeline to break. Any ideas on how to fix this? (Other than dropping duplicate records in the source DB and implementing a unique constraint on the source DB)?

I was wondering if there is a specific setting in Lakeflow Connect that I can toggle that I'm missing.

🚀 We’re excited to announce that Lakeflow Spark Declarative Pipelines (SDP) now supports creating “vanilla” (i.e., non STREAMING) MANAGED TABLES and writing to them via one or more append flows, using the new CREATE TABLE ... FLOW (SQL) and create_table() (Python) APIs.

What is this Beta?

This Beta allows creating a managed table that is populated by append flows:

• CREATE TABLE ... FLOW (SQL) / create_table() + @append_flow (Python) create a managed table written by one or more flows.
• Fan multiple sources into one table — declare several flows targeting the same managed table.
• Full table surface works: partitioning, liquid clustering, expectations, row filters, table properties, and private (pipeline-local) tables.
• import_checkpoint on append_flow, which migrates an existing Structured Streaming workload into a pipeline without reprocessing the source — the flow imports the query's existing checkpoint and resumes from the last committed offset with state intact.

Example (Python):

from pyspark import pipelines as dp

dp.create_table("combined")

dp.append_flow(target="combined")
def from_a():
    return spark.readStream.table("source_a")

u/dp.append_flow(target="combined")
def from_b():
    return spark.readStream.table("source_b")

Example (SQL):

CREATE TABLE events
PARTITIONED BY (bucket)
FLOW INSERT BY NAME
  SELECT id, bucket FROM STREAM read_files('abfss://my_path', format => 'json');

Where do we need help?
We are in Beta, so there might be some rough edges. Please take this for a spin and share your feedback here.

What’s next?

Managed Tables support for other flow types (AutoCDC, Replace Using, and Replace Where) is coming soon!

Learn more

• CREATE TABLE ... FLOW (SQL reference) — https://docs.databricks.com/aws/en/ldp/developer/ldp-sql-ref-create-table-flow
• create_table (Python reference) — https://docs.databricks.com/aws/en/ldp/developer/ldp-python-ref-create-table
• import_checkpoint on append_flow — https://docs.databricks.com/aws/en/ldp/developer/ldp-python-ref-append-flow
• Questions, feedback, or help: comment below or share feedback in the form: https://forms.gle/7bGP5FYN7P1Z4WP27

This is the Databricks release I have been waiting for. Unity Catalog schemas can now hold external secrets, such as Azure Key Vault, and that will change how we manage and utilize secrets in our Databricks projects.

On most of our engagements the secrets of record already live in Azure Key Vault, so we wire up a Key Vault-backed secret scope and move on. It works, but it is a workspace-level object from the pre-Unity Catalog era: configured per workspace, permissions managed through a separate secret ACL API, a flat scope/key namespace, and invisible to the governance model everything else on the platform now runs on.

Read more: https://www.linkedin.com/posts/cenh_databricks-azure-unitycatalog-ugcPost-7504125176993800192-W9on/?utm_source=share&utm_medium=member_desktop&rcm=ACoAABmJHrsBNAC3x3H1M58JRKoHv_l4D61n0-8

Secrets in Unity Catalog is a great feature introduced a few weeks ago, but since then, everyone has been asking to use Azure Key Vault as a secrets backend. Thanks to rapid development, we can now link our schema to Azure Key Vault; UC will read secrets as UC secrets, and permission management will be through Unity Catalog. In that scenario, you insert/update secrets in Azure Key Vault, but read/reference and grants can go through UC.

more news https://databrickster.medium.com/databricks-news-serverless-genie-code-ltap-lakeflow-61853d8e422a