Semantic Extraction
Semantic extraction is the intelligence layer between raw tabular data and a user's natural language question. It is the process that makes it possible for Lakehousecat to understand what your data means — not just its structure, but its relationships, hierarchies, and content.
Two Levels of Semantic Processing
Semantic processing happens at two distinct levels. Understanding this distinction is important for knowing what gets reused and what gets rebuilt.
Data Source Level — Semantic Extraction
When you run semantic extraction on a data source, a multi-agent process analyzes the loaded data and produces:
- Semantic metadata in PostgreSQL — descriptions, classifications, relationships, and structural understanding of the data source (tables, columns, hierarchies, data types, master vs. transactional data)
- Intermediate semantic structures in ClickHouse — supporting structures used to understand and work with the data source
During this process, the agents:
- Identify and classify data types (numeric, categorical, temporal, textual)
- Distinguish master data (reference entities like customers, products, regions) from transactional data (events, orders, measurements)
- Detect relationships between tables and columns
- Identify hierarchies (e.g., region → country → city)
- Analyze content patterns to understand what each column represents
This work is done once per data source and can be reused across multiple Custom Models.
Custom Model Level — Semantic Model Creation
When you trigger semantic model creation on a Custom Model (after connecting one or more trained data sources), the system builds the Star Views — denormalized analytical entities in ClickHouse that connect data from the linked data sources. These are the structures that charts and sessions actually query.
No data is loaded at the Custom Model level. It is a pure semantic layer that combines and cross-references the semantic knowledge from the connected data sources. Because the heavy DS-level extraction work has already been done, Custom Model training is significantly faster — a complex data source may take ~15 minutes to extract; a Custom Model built from it typically trains in ~5 minutes.
What Model Is Used
Semantic extraction uses the Default Backend Model — a Provider Model designated by the Administrator in Workspace → Models → Provider Models → Default Backend Model.
This is distinct from the Provider Model assigned to a Custom Model for chat queries. The Backend Model is a system-level configuration used for all extraction operations.
Where to Trigger Each Process
Data Source Semantic Extraction
Where: Data Source → Operations tab Who: Administrator or Builder
Enable the semantic extraction toggles and trigger the process. The system creates Job Definitions in the Operations section and runs the process as a background Airflow job.
Custom Model Semantic Training
Where: Custom Model → Operations tab Who: Administrator or Builder
After connecting trained data sources to a Custom Model, trigger the semantic model creation process. This builds the Star Views and completes the model so sessions can use it.
You can monitor progress and check logs for both processes in Workspace → Operations → Job Runs.
Duration
Duration depends on the complexity of the data source:
- Number of tables
- Number of columns per table
- Complexity of relationships
A small data source (a few tables with clear structure) may complete in minutes. A large, complex schema may take significantly longer. Resource availability in your Airflow workers also affects speed.
Data Quality Principle
Your data quality determines your semantic quality.
Lakehousecat is not an ETL tool. It does not clean, transform, or repair data before processing. If your source data has:
- Inconsistent naming conventions
- Missing or misleading column names
- Poorly populated columns
- Ambiguous relationships
...the semantic layer will reflect those issues, and query quality will suffer.
Best practice: Before running semantic extraction, review your data source. Add descriptions to tables and columns that might be ambiguous — these descriptions are used by the agents during extraction to improve understanding.
Best Practices
Run sparingly
Semantic extraction is a resource-intensive operation. Run it once after initial setup, then re-run only when the underlying data structure changes significantly (new tables added, schema changed, major data quality improvements).
Use domain-based models
Do not connect too many unrelated data sources to a single Custom Model. Semantic complexity grows with each additional data source, and mixing unrelated domains (e.g., HR data and sales data in one model) reduces query quality. Build separate Custom Models for separate domains.
What Can Go Wrong
| Issue | Likely Cause | Resolution |
|---|---|---|
| Extraction job fails or times out | Airflow worker resource limits | Increase CPU/memory for Airflow workers in the Operator configuration, or increase the job timeout |
| Poor query quality after extraction | Low data quality or ambiguous column names | Improve data quality at the source, add descriptions to tables/columns, re-run extraction |
| Extraction completes but model behaves unexpectedly | Too many unrelated data sources connected to one Custom Model | Split into domain-specific Custom Models |
Logs and Debugging
All extraction jobs run as Airflow DAGs. Logs are available in:
Workspace → Operations → Job Runs
Select the relevant job run to view the Airflow task logs. These logs show which agents ran, what they processed, and any errors encountered.
For low-level infrastructure logs (e.g., pod crashes, memory issues), check the Airflow worker pod logs via kubectl.
Reusability and the Two-Stage Architecture
Trained data sources are reusable building blocks. Once a data source has completed semantic extraction, it can be used as a component in multiple Custom Models — without re-running the extraction.
Example: You have three data sources (Sales, HR, Finance). You can create:
- A Sales Custom Model using the Sales data source
- A Combined Model using Sales + Finance data sources
- An HR Model using only the HR data source
Each Custom Model must still go through its own semantic training (to build the Star Views for that specific combination), but this is much faster than re-running data source extraction — because the semantic understanding of each data source is already available.
| Process | Level | Duration (example) | Reruns when |
|---|---|---|---|
| Semantic Extraction | Data Source | ~15 min (complex DS) | Data structure changes significantly |
| Semantic Model Creation | Custom Model | ~5 min | DS combination changes, or DS is re-extracted |
The expensive DS-level work is done once and reused. The CM-level work is lightweight and creates the final analytical layer.
See Quickstart for the full setup flow.