MuviaDevelopers
Guides: Iceberg tables

Iceberg tables

Muvia can keep a copy of your projects' data as Apache Iceberg tables (Parquet files) and let your own tools read it: Spark, Trino, DuckDB, Snowflake, PyIceberg and any other engine that speaks the Iceberg REST catalog protocol. The copy is read-only for you. It refreshes on its own, so you query it the way you would query a table of your own, with no export job to run.

Export is a company-wide setting that an administrator turns on in Muvia, and it needs the data export module. Each exported Muvia project appears as one namespace; each exported contract, query, sink or event type is one table in it.

What you need

Three values, shown on the export settings page in Muvia:

Value Placeholder in this page
Catalog URL https://<lake-host>/catalog
Token URL https://<lake-host>/lake/oauth/token
Warehouse <warehouse>

and a credential: a client id and a client secret, issued by an administrator on the same page. A credential is limited to the projects it was issued for and can only read. The secret is shown once, when the credential is issued; copy it then, because it cannot be shown again. If you lose it, revoke the credential and issue another.

Your engine signs in with the client credentials flow: it posts the client id and secret to the token URL and receives a short-lived token (15 minutes), which it sends to the catalog. Every recipe below does this for you.

In the recipes, <client_id> and <client_secret> are your credential. Keep the secret in your engine's secret store or an environment variable, never in a notebook or a repository.

Spark

Works with any Spark, Databricks classic clusters included. Add the Iceberg runtime for your Spark and Scala version (iceberg-spark-runtime-<spark>_<scala>) and the Iceberg AWS bundle, then configure the catalog:

spark.sql.catalog.muvia=org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.muvia.type=rest
spark.sql.catalog.muvia.uri=https://<lake-host>/catalog
spark.sql.catalog.muvia.warehouse=<warehouse>
spark.sql.catalog.muvia.credential=<client_id>:<client_secret>
spark.sql.catalog.muvia.oauth2-server-uri=https://<lake-host>/lake/oauth/token
spark.sql.catalog.muvia.header.X-Iceberg-Access-Delegation=vended-credentials
SHOW NAMESPACES IN muvia;
SELECT * FROM muvia.<project_namespace>.<table> LIMIT 10;

Snowflake

CREATE CATALOG INTEGRATION muvia
  CATALOG_SOURCE = ICEBERG_REST
  TABLE_FORMAT = ICEBERG
  REST_CONFIG = (
    CATALOG_URI = 'https://<lake-host>/catalog'
    CATALOG_NAME = '<warehouse>'
    ACCESS_DELEGATION_MODE = VENDED_CREDENTIALS
  )
  REST_AUTHENTICATION = (
    TYPE = OAUTH
    OAUTH_TOKEN_URI = 'https://<lake-host>/lake/oauth/token'
    OAUTH_CLIENT_ID = '<client_id>'
    OAUTH_CLIENT_SECRET = '<client_secret>'
    OAUTH_ALLOWED_SCOPES = ('catalog')
  )
  ENABLED = TRUE;

Then create a catalog-linked database over the integration, or one table at a time with CREATE ICEBERG TABLE … CATALOG = 'muvia' …. If your company stores the export in a bucket of its own, use EXTERNAL_VOLUME_CREDENTIALS over your own external volume instead of vended credentials.

DuckDB

INSTALL iceberg;
LOAD iceberg;

CREATE SECRET muvia_secret (
  TYPE iceberg,
  CLIENT_ID '<client_id>',
  CLIENT_SECRET '<client_secret>',
  OAUTH2_SERVER_URI 'https://<lake-host>/lake/oauth/token'
);

ATTACH '<warehouse>' AS muvia (
  TYPE iceberg,
  ENDPOINT 'https://<lake-host>/catalog'
);

SHOW ALL TABLES;

PyIceberg

from pyiceberg.catalog import load_catalog

catalog = load_catalog(
    "muvia",
    type="rest",
    uri="https://<lake-host>/catalog",
    warehouse="<warehouse>",
    credential="<client_id>:<client_secret>",
    **{"oauth2-server-uri": "https://<lake-host>/lake/oauth/token"},
)
print(catalog.list_namespaces())
table = catalog.load_table("<project_namespace>.<table>")
print(table.scan(limit=10).to_pandas())

Databricks

Read the tables from Databricks on a classic cluster with the Spark recipe above. If your company stores the export in a bucket of its own, you can also register the tables in Databricks from their location in that bucket.

How fresh the data is

The copy follows your data with a delay of up to about 15 minutes on the default refresh. Each refresh is one new Iceberg snapshot; a query sees one snapshot, so it never reads half a refresh.

When access ends: the revocation bound

Revoking a credential, or the export stopping being allowed (the module removed, the licence lapsed, your bucket disconnected), closes access at once: the credential can get no new token and the catalog refuses its next call. What an engine already holds keeps working for a short while: the storage access the catalog handed out lasts at most 15 minutes.

After a revocation, no new read through the catalog is possible; reads with storage access already issued end within 15 minutes.

If the export is in your own bucket, your engines read it with your own access, which Muvia's revocation does not touch.

When data is deleted: the deletion bound

A row that Muvia deletes (retention, or a source removed) leaves the table's current snapshot within one hour, and is physically gone from storage within snapshot keep time + 1 hour, which is 25 hours by default. Until then an older snapshot may still hold it, and an engine that asks for an old snapshot by id can still read it. A table that is dropped (removed from the export, or its source deleted) has its files deleted when it is dropped. A deleted project's tables are dropped 30 days after the deletion.

If the export is in your own bucket the same holds while the export is connected; once it is disconnected, the copy is yours and stays.

Versions tested

Each recipe was run against Muvia and read the same rows as the source. The versions are recorded here at each release.

  • Spark: verified on «date» with «version»
  • Snowflake: verified on «date»
  • DuckDB: verified on «date» with «version»
  • PyIceberg: verified on «date» with «version»
  • Databricks classic cluster: verified on «date» with «version»