SourceLace Docs
FAQOpen the app

Lake tables: Iceberg, Delta and Parquet

How to let people query Apache Iceberg, Delta Lake and Parquet tables in your Amazon S3 storage with read-only SQL, when you have no query engine (such as Athena, Trino, Snowflake or Databricks) for SourceLace to ask. SourceLace reads the table files itself, so an admin must turn this on, and only for tables that have no row or column rules.

Status: Pilot. New or partial, offered to pilot customers. Built from the Iceberg and Delta specifications and AWS's documentation, and covered by automated tests with real table files in a simulated S3 bucket; not yet used with a customer's live account.

At a glance

Connects through Sign-in Network Status First test question
Lake tables Amazon S3 API, with AWS Glue Data Catalog or an Iceberg REST catalog Shared account Nothing to open Pilot "Describe the orders table in lake:warehouse."

Add each source with the usual steps. The network answers are explained in What your network needs. If the first test question fails, the error message tells you what to fix: look it up under When something goes wrong below.

When to use it

If your tables already have a query engine, connect that instead (for example Snowflake or Databricks): the engine applies each person's own permissions, including row and column rules, and does the heavy work.

Use a lake source when there is no engine, for tables everyone allowed to use the source may see in full. It reads the files directly, which skips any row filters, column masks or per-person grants that AWS Lake Formation, Unity Catalog, Ranger or a query engine would apply. That is why it only works once an admin sets the direct_reads option to on.

Like Amazon S3, a lake source has no per-person sign-in: everyone who may use it reads with your organization's own AWS access, set on the source. To narrow who in your organization may use it, use access by group.

What people can do

  • search_schema lists the tables the admin declared; describe_object lists a table's columns and types, from the table's own metadata.
  • query (language sql) takes one SELECT or WITH ... SELECT statement in a PostgreSQL-like SQL dialect over the declared tables, by name, such as SELECT region, sum(amount) FROM orders WHERE order_date >= DATE '2026-01-01' GROUP BY region. Joins across the source's tables work, whatever their format.
  • Anything else is refused before any file is read: changes, more than one statement, other tables, file paths or web addresses, and functions that read files or settings (read_parquet, read_csv, glob, query and the like).
  • SourceLace reads only the columns a query uses and skips the parts of files, and whole files, that a filter rules out (for example a filter on a partition column), so name the columns you need and filter on partition columns.
  • Results are capped like other queries (2,000 rows kept, 50 shown at a time), and held for at most 30 minutes and never written to a database (see How long things are kept).
  • Lake sources never accept changes.

Supported tables:

Format How to name it Notes
Iceberg, by its folder or metadata file orders = iceberg sales/orders/ The newest metadata file is found from version-hint.text, or the highest vN.metadata.json. Renamed and added columns are followed.
Iceberg, in AWS Glue Data Catalog orders = iceberg glue:sales_db.orders SourceLace asks Glue (GetTable) for the table's current metadata file.
Iceberg, in an Iceberg REST catalog orders = iceberg rest:sales.orders Needs the catalog_uri option, and usually catalog_token.
Delta Lake customers = delta crm/customers/ Read from the latest checkpoint and the commits after it. Column mapping is supported.
Parquet clicks = parquet raw/clicks/ Every .parquet file in the folder (or one file). key=value folders become extra text columns. The files must share one schema.

Not supported yet: Iceberg tables with row-level deletes (delete files or deletion vectors), Delta tables with deletion vectors or V2 checkpoints, ORC and Avro data files, time travel, and storage other than Amazon S3. Such a table is refused with a message saying so.

What your admin does in AWS (Lake tables)

Once per source. About 15 minutes.

Who: an AWS admin who may create IAM policies and roles (and, for KMS-encrypted buckets, edit the key policy); with Lake Formation, a data lake administrator.

Before you start, write down the bucket and folder where the tables are (such as s3://corvanta-lake/warehouse/), the bucket's region (its Properties tab in the S3 console, such as eu-west-1), and, for Glue tables, the Glue database. Also confirm with the tables' owners that no row or column rules apply to them (Lake Formation filters, or a query engine's policies): reading the files directly skips those rules.

1. Create the read-only policy. SourceLace needs to list and read the objects under the tables' location, and, for tables in Glue, to read their definitions. Create an IAM policy like this one, with your bucket and folder:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::corvanta-lake",
      "Condition": { "StringLike": { "s3:prefix": ["warehouse/*"] } }
    },
    {
      "Effect": "Allow",
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::corvanta-lake/warehouse/*"
    },
    {
      "Effect": "Allow",
      "Action": "glue:GetTable",
      "Resource": [
        "arn:aws:glue:us-east-1:123456789012:catalog",
        "arn:aws:glue:us-east-1:123456789012:database/sales_db",
        "arn:aws:glue:us-east-1:123456789012:table/sales_db/*"
      ]
    }
  ]
}

Leave out the Glue statement if no table is in Glue. If the bucket uses a customer-managed KMS key, also allow kms:Decrypt on that key, and allow the role in the key's own key policy.

2. Give SourceLace the policy in one of two ways, exactly as for an Amazon S3 source:

  • A role (recommended). Create an IAM role with the policy, trusting SourceLace's AWS account, with a trust policy condition that requires the external ID SourceLace shows for your organization on the source (once saved). SourceLace assumes the role for each query and never stores the short-lived credentials.
  • Access keys. An IAM user with the policy and an access key. The secret is stored encrypted and never shown again.

Check that it worked: after your SourceLace admin adds the source, connect it and ask for 5 rows of a table. AWS CloudTrail shows the role session reading the bucket.

Amazon's own guides: Amazon S3 documentation, AWS Glue documentation and IAM documentation.

Add the source (SourceLace admin)

On Connections, add a source of kind Lake tables (lake), such as lake:warehouse, with:

Option What to enter
location Where the tables are, such as s3://corvanta-lake/warehouse/. SourceLace never reads outside it, even if a table's metadata points elsewhere.
region The bucket's AWS region, such as eu-west-1, from its Properties tab in the S3 console.
tables The tables, separated by semicolons (or new lines), each name = format where, as in the table above. Folders are relative to the location, or a full s3:// address inside it. Table names are lowercase letters, digits and underscores.
role_arn, or access_key_id and secret_access_key Your organization's AWS access from step 2.
direct_reads (Read files directly) Required. On confirms these tables have no row or column rules. Until it is On, the source reads nothing.
catalog_uri For rest: tables: the catalog's https address, such as https://catalog.corvanta.com/api/catalog.
catalog_warehouse For rest: tables, if your catalog needs one.
catalog_token For rest: tables: a bearer token with read access to the tables' metadata. Stored encrypted, never shown again, and dropped if catalog_uri changes.
max_bytes_read, max_files_read Optional: lower this source's caps (below).

Then connect it once like any other source (there is no sign-in page) and try describe_object on a table.

Limits and safety

  • Bytes and files. One query may read at most 2 GB and 2,000 files (data and metadata together) unless your limits say otherwise; a source's max_bytes_read and max_files_read options can only lower them. SourceLace counts every read as it happens and stops the query at the cap, so the cap is never exceeded.
  • Time. Queries have the usual query time limit and are stopped when it runs out.
  • Kept apart. Each query runs on its own and shares nothing with other queries, people or organizations. It reads only the files of the tables you declared, inside the location, and cannot reach the internet or other buckets.
  • Working space. A very large query may use temporary working space while it runs. That space is encrypted with a key made for that query and never written down, and it is deleted when the query ends; where it cannot be encrypted, it is not used at all and such a query is refused instead. Nothing else is written down.

When something goes wrong

What you see What it means What to do
"SourceLace: the database could not run this statement. Check the names and the comparison values, and ask again." On a source with data protection rules, row rules or access by group, the lake could not run the question. Its own reason is not shown, because it could quote protected data. The AI usually asks again another way. If it keeps happening, check the names and values in the question, and that the sign-in may read the tables.
"... skips any row or column rules ..." Read files directly is not On. Turn it On, but only if the tables have no such rules.
"...'s bucket is in region ..., not the source's region ..." The bucket is in another region than region. Set region to the region named.
"Amazon S3 refused ... access ..." The role or keys cannot read that table's files. Check the policy (and KMS key) covers the whole location (step 1).
"AWS refused to let SourceLace assume the source's role (...)..." The role's trust policy or role_arn is wrong. See the Amazon Redshift table for each AWS code.
"... was not found ..." The table's entry is wrong. Check it in tables (folder, Glue database and table, or catalog namespace).
"... names a file outside the source's location ..." The table's metadata points outside location. Widen location if that is expected.
"This query would read more than ..." The query passes the bytes or files cap. Select fewer columns, filter on partition columns, or ask an admin to raise the limit.
"The query on ... needed more memory and temporary disk than allowed ..." A very large sort or join. Narrow it, or aggregate before sorting or joining.
"... cannot read directly yet ..." The table uses a feature listed above as not supported. Query it through your query engine.
"The Iceberg catalog refused ...'s token ..." catalog_token expired or lacks access. Enter a new catalog_token.
"...'s storage or catalog could not be reached. Try again shortly." S3 or the catalog did not answer. Try again shortly.
"... did not finish the query in time, so it was stopped. Narrow it down." The query ran longer than SourceLace allows. Select fewer columns or filter on a partition column.
"...'s catalog_uri (...) could not be found." or "...'s catalog_uri is a private network address, and this SourceLace server only connects to catalogs on the internet." The catalog address is misspelt, or points inside a private network. An admin corrects the Iceberg catalog address.