# Iceberg tables

Muvia can keep a copy of your projects' data as **Apache Iceberg** tables
(Parquet files) and let your own tools read it: Spark, Trino, DuckDB,
Snowflake, PyIceberg and any other engine that speaks the Iceberg REST catalog
protocol. The copy is **read-only** for you. It refreshes on its own, so you
query it the way you would query a table of your own, with no export job to run.

Export is a company-wide setting that an administrator turns on in Muvia, and
it needs the data export module. Each exported Muvia project appears as one
**namespace**; each exported contract, query, sink or event type is one table
in it.

## What you need

Three values, shown on the export settings page in Muvia:

| Value | Placeholder in this page |
| --- | --- |
| Catalog URL | `https://<lake-host>/catalog` |
| Token URL | `https://<lake-host>/lake/oauth/token` |
| Warehouse | `<warehouse>` |

and a **credential**: a client id and a client secret, issued by an
administrator on the same page. A credential is limited to the projects it was
issued for and can only read. **The secret is shown once**, when the credential
is issued; copy it then, because it cannot be shown again. If you lose it,
revoke the credential and issue another.

Your engine signs in with the **client credentials** flow: it posts the client
id and secret to the token URL and receives a short-lived token (15 minutes),
which it sends to the catalog. Every recipe below does this for you.

In the recipes, `<client_id>` and `<client_secret>` are your credential. Keep
the secret in your engine's secret store or an environment variable, never in a
notebook or a repository.

## Spark

Works with any Spark, Databricks classic clusters included. Add the Iceberg
runtime for your Spark and Scala version (`iceberg-spark-runtime-<spark>_<scala>`)
and the Iceberg AWS bundle, then configure the catalog:

```text
spark.sql.catalog.muvia=org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.muvia.type=rest
spark.sql.catalog.muvia.uri=https://<lake-host>/catalog
spark.sql.catalog.muvia.warehouse=<warehouse>
spark.sql.catalog.muvia.credential=<client_id>:<client_secret>
spark.sql.catalog.muvia.oauth2-server-uri=https://<lake-host>/lake/oauth/token
spark.sql.catalog.muvia.header.X-Iceberg-Access-Delegation=vended-credentials
```

```sql
SHOW NAMESPACES IN muvia;
SELECT * FROM muvia.<project_namespace>.<table> LIMIT 10;
```

## Snowflake

```sql
CREATE CATALOG INTEGRATION muvia
  CATALOG_SOURCE = ICEBERG_REST
  TABLE_FORMAT = ICEBERG
  REST_CONFIG = (
    CATALOG_URI = 'https://<lake-host>/catalog'
    CATALOG_NAME = '<warehouse>'
    ACCESS_DELEGATION_MODE = VENDED_CREDENTIALS
  )
  REST_AUTHENTICATION = (
    TYPE = OAUTH
    OAUTH_TOKEN_URI = 'https://<lake-host>/lake/oauth/token'
    OAUTH_CLIENT_ID = '<client_id>'
    OAUTH_CLIENT_SECRET = '<client_secret>'
    OAUTH_ALLOWED_SCOPES = ('catalog')
  )
  ENABLED = TRUE;
```

Then create a catalog-linked database over the integration, or one table at a
time with `CREATE ICEBERG TABLE … CATALOG = 'muvia' …`. If your company stores
the export in a bucket of its own, use `EXTERNAL_VOLUME_CREDENTIALS` over your
own external volume instead of vended credentials.

## DuckDB

```sql
INSTALL iceberg;
LOAD iceberg;

CREATE SECRET muvia_secret (
  TYPE iceberg,
  CLIENT_ID '<client_id>',
  CLIENT_SECRET '<client_secret>',
  OAUTH2_SERVER_URI 'https://<lake-host>/lake/oauth/token'
);

ATTACH '<warehouse>' AS muvia (
  TYPE iceberg,
  ENDPOINT 'https://<lake-host>/catalog'
);

SHOW ALL TABLES;
```

## PyIceberg

```python
from pyiceberg.catalog import load_catalog

catalog = load_catalog(
    "muvia",
    type="rest",
    uri="https://<lake-host>/catalog",
    warehouse="<warehouse>",
    credential="<client_id>:<client_secret>",
    **{"oauth2-server-uri": "https://<lake-host>/lake/oauth/token"},
)
print(catalog.list_namespaces())
table = catalog.load_table("<project_namespace>.<table>")
print(table.scan(limit=10).to_pandas())
```

## Databricks

Read the tables from Databricks on a **classic cluster** with the Spark recipe
above. If your company stores the export in a bucket of its own, you can also
register the tables in Databricks from their location in that bucket.

## How fresh the data is

The copy follows your data with a delay of up to about 15 minutes on the
default refresh. Each refresh is one new Iceberg snapshot; a query sees one
snapshot, so it never reads half a refresh.

## When access ends: the revocation bound

Revoking a credential, or the export stopping being allowed (the module
removed, the licence lapsed, your bucket disconnected), closes access at once:
the credential can get no new token and the catalog refuses its next call.
What an engine already holds keeps working for a short while: the storage
access the catalog handed out lasts at most **15 minutes**.

> After a revocation, no new read through the catalog is possible; reads with
> storage access already issued end within 15 minutes.

If the export is in your own bucket, your engines read it with your own
access, which Muvia's revocation does not touch.

## When data is deleted: the deletion bound

A row that Muvia deletes (retention, or a source removed) leaves the table's
current snapshot within one hour, and is physically gone from storage within
**snapshot keep time + 1 hour**, which is **25 hours** by default. Until then
an older snapshot may still hold it, and an engine that asks for an old
snapshot by id can still read it. A table that is dropped (removed from the
export, or its source deleted) has its files deleted when it is dropped. A
deleted project's tables are dropped 30 days after the deletion.

If the export is in your own bucket the same holds while the export is
connected; once it is disconnected, the copy is yours and stays.

## Versions tested

Each recipe was run against Muvia and read the same rows as the source. The
versions are recorded here at each release.

- Spark: verified on «date» with «version»
- Snowflake: verified on «date»
- DuckDB: verified on «date» with «version»
- PyIceberg: verified on «date» with «version»
- Databricks classic cluster: verified on «date» with «version»
