Multi-Modal Retrieval (Hybrid Search)

Real applications often need to search multiple modalities at once — a product with text + image embeddings, a video clip with audio + frame embeddings, a document with dense + sparse representations. The MultiModal layer adds first-class support for multiple named vector fields per record plus Reciprocal-Rank-Fusion (RRF) hybrid search, while keeping PistaDB's zero-dependency, single-machine character.

What you get

• Multi-vector schema — up to 16 named fields per record, each with its own dim, metric, and index algorithm. Mix HNSW for text with IVF for image freely; every field is internally a standard .pst file, so all 8 index types are available.
• Atomic multi-field writes — mm.insert(id, vecs={...}) either commits every field, payload, and catalog entry, or rolls every one of them back. Powered by a dedicated MM-layer WAL that survives os._exit / power loss.
• Hybrid search with RRF — parallel per-field k-NN, fused by score = Σ 1 / (k + rank_i), with configurable rrf_k (default 60).
• Binary payload per record — bring your own caption / URL / tags as a blob; readers borrow the bytes from an internal read buffer.
• Backward-compatible — does not change the .pst file format or any existing API. Existing single-modal .pst files keep working unchanged.

On-disk layout

A multi-modal bundle is a directory:

mybundle.pmm/
├── pmm.manifest          (128-byte header + schema)
├── pmm.catalog           (64-byte rows, id → flags/mask/payload_off/label)
├── pmm.payload           (append-only blob log; compacted at checkpoint)
├── pmm.wal               (canonical multi-field WAL; CRC32 + torn-tail safe)
└── fields/<name>.pst     (one standard PistaDB file per field)

Each field's <name>.pst is a regular PistaDB file — you can even open it with the single-modal pistadb_open() API if you ever need to. The manifest, catalog, payload log, and MM WAL are the only new on-disk artifacts.

Python example

import numpy as np
from pistadb import MultiModal, FieldSpec, Metric, Index, Params

mm = MultiModal.create("products.pmm", [
    FieldSpec("text",  dim=384, metric=Metric.COSINE, index_type=Index.HNSW),
    FieldSpec("image", dim=512, metric=Metric.COSINE, index_type=Index.IVF),
])
mm.train_field("image")     # IVF needs training before inserts

mm.insert(
    id=1,
    label="red leather wallet",
    payload=b'{"sku":"W-1042","price":49.99}',
    vecs={
        "text":  text_emb,     # numpy float32, shape (384,)
        "image": image_emb,    # numpy float32, shape (512,)
    },
)

# Search both modalities in parallel and fuse with RRF:
hits = mm.hybrid_search(
    {"text":  (query_text_emb,  20),     # per-field top-k before fusion
     "image": (query_image_emb, 20)},
    top_k=10, rrf_k=60, parallel=True,
)
for h in hits:
    print(h.id, h.score, h.label)
    print("  payload:", mm.get_payload(h.id))

mm.checkpoint()             # snapshot + compact + truncate WAL
mm.close()

How RRF fusion works

For each field query, PistaDB runs a regular k-NN search on the matching child index. Then the fused score for an id is:

$$\text{score}(\text{id}) = \sum_{i \in \text{fields}} \frac{1}{k_{\text{rrf}} + \text{rank}_i(\text{id})}$$

where $\text{rank}_i(\text{id})$ is 1-indexed (the top hit gets rank 1). Ids that don't appear in a field's top-k contribute zero for that field. The default rrf_k = 60 is the value from the original RRF paper — higher values flatten the contribution of top ranks, lower values amplify them.

C API

The full surface lives in src/pistadb_mm.h — 14 entry points following the same opaque-handle pattern as pistadb_batch.h / pistadb_txn.h / pistadb_cache.h:

Category	Functions
Lifecycle	pdb_mm_create · pdb_mm_open · pdb_mm_close · pdb_mm_save · pdb_mm_checkpoint
CRUD	pdb_mm_insert · pdb_mm_update · pdb_mm_delete · pdb_mm_get
Search	pdb_mm_hybrid_search
Maintenance	pdb_mm_train_field · pdb_mm_count · pdb_mm_schema · pdb_mm_last_error

#include "pistadb_mm.h"

PdbMmFieldSpec specs[2] = {
    { .name="text",  .dim=384, .metric=METRIC_COSINE, .index_type=INDEX_HNSW,
      .params=pistadb_default_params() },
    { .name="image", .dim=512, .metric=METRIC_COSINE, .index_type=INDEX_IVF,
      .params=pistadb_default_params() },
};
PdbMmSchema schema = { .n_fields = 2 };
schema.fields[0] = specs[0];
schema.fields[1] = specs[1];

PistaDBMM *mm = pdb_mm_create("products.pmm", &schema, NULL);

PdbMmVecEntry vecs[2] = {
    { .field_name = "text",  .vec = text_emb  },
    { .field_name = "image", .vec = image_emb },
};
PdbMmRecord rec = {
    .id          = 1,
    .label       = "red leather wallet",
    .payload     = jpg_bytes,
    .payload_len = jpg_len,
    .n_vecs      = 2,
    .vecs        = vecs,
};
pdb_mm_insert(mm, &rec);

PdbMmFieldQuery qs[2] = {
    { .field_name="text",  .vec=q_text,  .k=20 },
    { .field_name="image", .vec=q_image, .k=20 },
};
PdbMmQuery q = {
    .n_field_queries = 2,
    .field_queries   = qs,
    .top_k           = 10,
    .ranker          = { .kind = PDB_MM_RANKER_RRF, .rrf_k = 60 },
    .parallel        = 1,
};
PdbMmResult results[10];
int n_out = 0;
pdb_mm_hybrid_search(mm, &q, results, &n_out);

Crash safety

The MM-layer WAL is the canonical source of truth for committed writes. Child .pst files are opened with their own WAL disabled — the MM WAL coordinates all of them. Every logical operation is a sequence of records under one monotonic txid:

BEGIN_INSERT(txid, id, label, payload_len, payload_crc, n_vecs)
PAYLOAD_WRITE(txid, off, len)
FIELD_INSERT(txid, field_idx, dim, vec[dim])    ×N
COMMIT(txid)

On replay (after pdb_mm_open on a previously-crashed bundle):

1. The WAL is scanned forward; records group by txid.
2. A txid is applied only if a COMMIT record was seen before a torn tail.
3. In-flight (un-committed) txids are rolled back: orphan payload bytes are truncated, partial child inserts are deleted.
4. The torn tail (if any) is truncated and a fresh WAL append is opened.

Tested end-to-end with os._exit(1) inside a child process and with manually-corrupted WAL tails — both cases recover cleanly.

What's not in this release

The MVP intentionally ships a focused feature set. The following are planned for future releases:

• Scalar-metadata filtering (category == 'shoes' AND price < 100)
• Sparse vector fields for BM25 / SPLADE hybrid retrieval
• Late-interaction multi-vector fields (ColBERT / BGE-M3 with MaxSim aggregation)
• WeightedRanker as an alternative to RRF (weighted sum with score normalisation)

Vote on these with a GitHub issue if they matter to your workload.