Federated retrieval and the catalog¶
Status: mostly built. Checked against main on 2026-11-19. Changes 1 to 5
have landed; change 6 has not. "Current boundary" describes the system before
these changes.
What is built¶
Numbers are the items of Changes. Each was read in code on main.
Nothing was run for this list.
| # | Change | State | Evidence |
|---|---|---|---|
| 1 | Hybrid search: the embedder slot, the Vertex embedder, derived vectors and fusion | Built | #132; pkg/dense, internal/app/embedder.go, pkg/knowledge/vectors.go |
| 2 | Catalog core: pkg/catalog, catalog_search, GET /api/catalog, publish and withdraw |
Built | #154, #156; pkg/catalog/catalog.go, internal/platform/catalog.go |
| 3 | BigQuery and Snowflake connectors | Built | #155, #197; pkg/catalog/bigquery, pkg/catalog/snowflake |
| 4 | GCS and S3, then OneDrive | Built | #197, #284; pkg/catalog/objectstorage. A collect capability constant: not confirmed, pkg/catalog/connector.go declares catalog and live only |
| 5 | Console: a Catalog view | Built | #202; Catalog mounted in apps/web/src/App.tsx. Grouping of New Source connectors by capability: not confirmed |
| 6 | Opt-in indexing of agent-written files | Not built | No code found on main |
Goal¶
An agent can find and use the right information wherever it lives (Google Drive, OneDrive, GCS, S3, Snowflake, BigQuery, Notion, Slack, uploads, its own files) without Humagen copying every system into its own index. Search has to hold at the size of a company's data, and stay inside each reader's own access.
Current boundary¶
- Retrieval.
pkg/knowledgeis what runs are grounded in. A source is cut into passages of about 800 bytes and ranked with BM25 only: lexical, no embeddings. Its logic and passages live in memory, written throughpkg/state(status). - Hybrid search exists, unused. fiber's knowledge package (its embedders
are now
pkg/densein Humagen) has anEmbedderinterface (a deterministicHashingembedder and an OpenAI-compatibleHTTPone), a dense arm, reciprocal rank fusion and a pgvector backend. The platform does not use it. - Providers. Manual upload (UTF-8 text, Markdown, CSV, JSON), Google Drive
and Notion, as a collected copy, a live query or both
(sources). Retained external text is withheld
until the current reader's upstream access is verified; live queries use the
reader's own credential through the
source_searchtool. - Warehouses and chat. Snowflake, BigQuery and Slack are reachable only as MCP tools an agent is granted. Nothing records what tables or channels exist, so an agent must already know where to look.
Decision¶
Three rules decide where each kind of information is searched:
| Information | Where it is searched | Humagen stores |
|---|---|---|
| Content in an external system (files, pages, rows, messages) | Live, in that system, with the reader's own credential: its search API, its SQL | Nothing, unless an owner opts into a collected copy |
| What exists in those systems (tables, columns, folders, file titles) | The catalog, Humagen's own index | Metadata only, never row data or file text |
| Content Humagen owns (uploads, files agents write, opted-in copies) | Hybrid index in pkg/knowledge |
Passages and their vectors |
The catalog is what makes live access work at scale. An agent asks the catalog
"where is churn by region", gets back analytics.billing.churn_monthly with its
columns and their descriptions, then queries that table live. It never needs a
copy of the warehouse, and it never guesses a table name.
Hybrid retrieval in pkg/knowledge¶
pkg/knowledge keeps its API, lifecycle, variants and access rules. It gains a
dense arm beside BM25. As built, the embedders are pkg/dense and the fusion
is in pkg/knowledge/vectors.go, in Humagen, not composed from another module:
- Embedder slot. A System spec field
knowledge.embedder, a slot:none(the default, today's behavior),hashing,openai(any OpenAI-compatible/embeddingsendpoint) orvertex(Vertex AI, with Application Default Credentials). The Vertex kind defaults togemini-embedding-001at 768 dimensions. - Vectors are a derived index. A passage's vector is written under its own key beside the passage, tagged with the embedder's model. They are computed off the write path by a bounded background indexer, so an upload never waits on a model and a model outage never fails a write. At start, passages whose vector is missing or from another model are queued again.
- Fusion. Search embeds the query (bounded by a short timeout, outside the
store lock), ranks the selected passages by cosine similarity, and fuses that
list with BM25 by reciprocal rank fusion (
pkg/knowledge/vectors.go). A passage without a vector still ranks lexically. If the query cannot be embedded, search is lexical, and the hit says so. - Same access rules. Fusion runs over exactly the passages the lexical arm may see: the same tenant, sources, exclusions, variants and upstream-read checks. A vector is never served on its own.
- Limits. Vectors are compared in memory, exhaustively, like the passages
are today. At 768 dimensions in float32 that is about 3 KB a passage, so the
current caps (
MaxImportDocuments,MaxImportBytes) bound it. Moving to the pgvector backend is the step afterpkg/statestops loading whole namespaces (limits at scale).
Connectors¶
A connector is one kind in a spec slot, declaring what it can do:
| Capability | Meaning | Example |
|---|---|---|
live |
Search or read in place with the reader's credential | Drive search, a BigQuery query |
collect |
Fetch selected resources into a collected copy | Drive folder, Notion page, S3 prefix |
catalog |
List what exists, with schema and descriptions | BigQuery INFORMATION_SCHEMA, Snowflake SHOW, GCS object listing |
The existing Provider (Fetch) becomes the collect capability. Drive and
Notion keep working unchanged. New kinds, in order of value: BigQuery and
Snowflake (catalog, live), GCS and S3 (collect, catalog), OneDrive
(live, collect). A system that already has a good MCP server can be a
connector whose live capability is that server's tools, so Humagen does not
reimplement a client it can compose.
Credentials follow sources: a reference, resolved
at the provider boundary, never a workspace fallback for a person's read. A
connector may also name a workspace credential (env:, a service account)
for catalog only, when an owner chooses to publish metadata (below).
The catalog¶
pkg/catalog is a tenant-scoped store of entries, each one an object in a
connected system:
| Field | Example |
|---|---|
system, kind |
bigquery, table |
path |
analytics.billing.churn_monthly |
title, description |
from the system, else empty |
schema |
columns with names, types, modes and descriptions |
stats |
row count, size, last modified, when the system reports them |
connector, seen_at |
which connector listed it, and when |
- Filled by a crawl. A connector's
catalogcapability is run on demand or on the source's refresh cadence. A crawl replaces its connector's entries atomically, so a deleted table disappears and a partial crawl publishes nothing. - Searched hybrid. Each entry is rendered to one short text (path, title,
description, column names and descriptions) and indexed lexically and with
the configured embedder, the same way
pkg/knowledgedoes. - Used by agents through a tool.
catalog_search, granted explicitly likesource_search, returns entries with their schema. The agent then queries the system live through its connector or MCP tool. The console gets a Catalog view to browse and search the same entries. - Never data. Entries hold structure and descriptions. Sample values are off, and an owner can turn them on per connector.
Who sees an entry¶
Table and column names can themselves be sensitive. An entry is visible to:
- the person whose credential crawled it, by default;
- the members of the connector's project, only when its owner publishes the connector's metadata. Publishing is explicit, recorded with who and when, and can be withdrawn, which hides the entries at once.
Seeing an entry never grants reading the object. A live query still runs with the reader's own credential and fails if they lack access.
What gets vectors¶
| Content | Vectors |
|---|---|
| Uploaded documents and pasted text | Yes, through the embedder slot |
| Files agents write (artifacts) | Yes, opt-in per project, indexed as a native corpus |
| Collected copies of external content | Yes, when an owner chose a copy |
| Live external content | No: searched in place |
| Catalog entries | Yes: one vector per entry, metadata only |
PDF and Office extraction for uploads is a separate change; it adds formats to the same pipeline.
Alternatives¶
- Ingest everything into one vector store. Rejected: it duplicates data Humagen should not hold, goes stale between syncs, multiplies cost, and needs every system's permissions mirrored to be safe. The existing rule that collection authority never grants external access already rules out serving most of it.
- MCP tools only, no catalog. Rejected as the whole answer: an agent cannot find what it does not know exists, and listing a warehouse in every turn costs tokens and time. The catalog is the index MCP lacks.
- pgvector now. Deferred: the platform's stores load whole namespaces into memory today, so the in-memory dense arm matches how every other store works. The vector index is derived and rebuildable, so moving it later loses nothing.
Changes¶
- Hybrid search. The embedder slot, the Vertex embedder, the derived
vector index with its background indexer, and fusion in
pkg/knowledge. Spec reference and status table updated. - Catalog core.
pkg/catalogwith hybrid search, thecatalog_searchtool,GET /api/catalog, and the publish and withdraw rules. - Warehouse connectors. BigQuery and Snowflake with
catalogandlive. - Storage connectors. GCS and S3 (
collect,catalog), then OneDrive. - Console. The New Source catalog groups connectors by capability, and a Catalog view browses and searches entries.
- Agent files. Opt-in indexing of artifacts as a native corpus.
Each change is its own PR and keeps the previous behavior when its settings are left out.
Open questions¶
- Whether Vertex's task types (
RETRIEVAL_QUERYandRETRIEVAL_DOCUMENT) should become part of theEmbedderinterface, which today does not tell a query from a passage. - How often a published catalog should be recrawled by default, and whether a connector that supports change feeds should push instead.
- Whether column-level descriptions written in Humagen (a data dictionary) should be stored as catalog annotations that survive a recrawl.