Unity Catalog: Open, Multimodal Catalog for Data & AI
Unity Catalog is the industry’s only universal catalog for data and AI.
- Multimodal interface supports any format, engine, and asset
- Multi-format support: It is extensible and supports Delta Lake, Apache Iceberg and Apache Hudi via UniForm, Apache Parquet, JSON, CSV, and many others.
- Multi-engine support: With its open APIs, data cataloged in Unity can be read by many leading compute engines.
- Multimodal: It supports all your data and AI assets, including tables, files, functions, AI models.
- Open source API and implementation - OpenAPI spec and OSS implementation (Apache 2.0 license). It is also compatible with Apache Hive's metastore API and Apache Iceberg's REST catalog API. Unity Catalog is currently a sandbox project with LF AI and Data Foundation (part of the Linux Foundation).
- Unified governance for data and AI - Govern and secure tabular data, unstructured assets, and AI assets with a single interface.
The current roadmap is available at Unity Catalog Roadmap.
Vibrant ecosystem
This is a community effort. Unity Catalog is supported by
Unity Catalog is proud to be hosted by the LF AI & Data Foundation.
Quickstart - Hello UC!
Let's take Unity Catalog for spin. In this guide, we are going to do the following:
- In one terminal, run the UC server.
- In another terminal, we will explore the contents of the UC server using a CLI.
An example project is provided to demonstrate how to use the UC SDK for various assets
as well as provide a convenient way to explore the content of any UC server implementation.
If you prefer to run Unity Catalog in Docker use docker compose up. See the Docker Compose docs for more details.
Prerequisites
You have to ensure that your local environment has the following:
- Clone this repository.
- Ensure the
JAVA_HOME environment variable your terminal is configured to point to JDK17.
- Compile the project using
build/sbt package
Run the UC Server
In a terminal, in the cloned repository root directory, start the UC server.
bin/start-uc-server
For the remaining steps, continue in a different terminal.
Operate on Delta tables with the CLI
Let's list the tables.
bin/uc table list --catalog unity --schema default
You should see a few tables. Some details are truncated because of the nested nature of the data.
To see all the content, you can add --output jsonPretty to any command.
Next, let's get the metadata of one of those tables.
bin/uc table get --full_name unity.default.numbers
You can see that it is a Delta table. Now, specifically for Delta tables, this CLI can
print a snippet of the contents of a Delta table (powered by the Delta Kernel Java project).
Let's try that.
bin/uc table read --full_name unity.default.numbers
Operate on Delta tables with DuckDB
For operating on tables with DuckDB, you will have to install it (version 1.0).
Let's start DuckDB and install a couple of extensions. To start DuckDB, run the command duckdb in the terminal.
Then, in the DuckDB shell, run the following commands:
install uc_catalog from core_nightly;
load uc_catalog;
install delta;
load delta;
If you have installed these extensions before, you may have to run update extensions and restart DuckDB
for the following steps to work.
Now that we have DuckDB all set up, let's try connecting to UC by specifying a secret.
CREATE SECRET (
TYPE UC,
TOKEN 'not-used',
ENDPOINT 'http://127.0.0.1:8080',
AWS_REGION 'us-east-2'
);
You should see it print a short table saying Success = true. Then we attach the unity catalog to DuckDB.
ATTACH 'unity' AS unity (TYPE UC_CATALOG);
Now we are ready to query. Try the following:
SHOW ALL TABLES;
SELECT * from unity.default.numbers;
You should see the tables listed and the contents of the numbers table printed.
To quit DuckDB, press Ctrl+D (if your platform supports it), press Ctrl+C, or use the .exit command in the DuckDB shell.
Interact with the Unity Catalog UI
To use the Unity Catalog UI, start a new terminal and ensure you have already started the UC server (e.g., ./bin/start-uc-server)
Prerequisites
How to start the UI through bun
cd /ui
bun install
bun run start
CLI tutorial
You can interact with a Unity Catalog server to create and manage catalogs, schemas and tables,
operate on volumes and functions from the CLI, and much more.
See the cli usage for more details.
This is the beginning of the README. Read the rest at the original. Read the full README ↗
