Vector Index (Overview)

Level 4 — Schema Definition & Constraints The specialized index type in SurrealDB designed for AI and semantic search, storing high-dimensional vector embeddings and using spatial clustering algorithms (like M-Tree or HNSW) to perform fast nearest-neighbor lookups.

1. Prerequisites

• DEFINE INDEX (Deep Dive) — The parent index context.

2. Term Category

Performance / Operations (vector similarity embedding index): - Database Structure / Paradigm

3. Explanation

(1) Design Motivation — "Why did we design this?"

Traditional search indexes (B-Tree, Full-Text) match exact words.

However, they fail at Semantic Search (searching by meaning):

• If a user searches for "feline", a full-text search will miss documents containing the word "cat" if the letters don't overlap.
• In modern AI workflows (like Retrieval-Augmented Generation / RAG), text, images, and audio are converted by machine learning models into lists of numbers called Vector Embeddings.
• To find similar items, the database must calculate which vectors are closest in high-dimensional space.

In PostgreSQL, you must install the pgvector extension.

In MongoDB, you must use Atlas cloud-specific indexes.

We designed native Vector Indexing in SurrealDB to support AI workloads out of the box.

By building multi-dimensional indexes (using M-Tree or HNSW clustering algorithms) directly into the engine, SurrealDB allows you to store embeddings and run semantic similarity searches alongside your graph and document data.

(2) Vector Index Parameters

When defining a vector index in SurrealQL, you must configure three key parameters:

1. Algorithm Type: Choose the indexing method:
   • MTREE: Multi-way tree structure (default, balanced speed/accuracy).
   • HNSW (Hierarchical Navigable Small World): High-speed, approximate nearest neighbor graph (ideal for massive databases).
2. Dimension: The number of values in the vector array (e.g., OpenAI's text-embedding-3-small outputs vectors with 1536 dimensions).
3. Distance Metric: The formula used to calculate proximity:
   • COSINE (Cosine Similarity): Measures the angle between vectors (standard for text embeddings).
   • EUCLIDEAN (L2 Distance): Measures straight-line distance.
   • MANHATTAN (L1 Distance): Measures grid-based distance.

(3) Reality Metaphor (The 3D Topic Room)

Imagine organizing books in a library:

• Full-Text Search: Books are indexed alphabetically by title words.
• Vector Search: Placing books in a 3D Floating Room based on meaning.
  • Books about "dogs" and "puppies" float right next to each other because their meanings are close.
  • Books about "quantum physics" float in a distant corner.
  • When you toss a book about "canines" (the search vector) into the room, you measure which books float closest (shortest distance) to it, finding "dogs" instantly.

(4) Code Examples

Creating Vector Indexes in SurrealQL

Let's build a movie recommendation schema:

DEFINE TABLE movie SCHEMAFULL;

-- 1. Define a field to store the vector embeddings array
-- (For a model outputting 4-dimension arrays for simple illustration)
DEFINE FIELD embedding ON movie TYPE array<float>;

-- 2. Define the Vector Index using the MTREE algorithm and Cosine similarity
DEFINE INDEX movie_vector ON movie COLUMNS embedding
  MTREE DIMENSION 4 DISTANCE cosine;

-- 3. Insert records with vector embeddings
CREATE movie:1 SET title = "Space Adventures", embedding = [0.1, 0.9, 0.0, 0.2];
CREATE movie:2 SET title = "Romantic Comedy", embedding = [0.8, 0.1, 0.7, 0.0];

-- 4. Semantic search queries are run using vector distance helpers (covered in Level 10)

4. Common Mistakes & Pitfalls

Mistake 1: Defining a vector index with a dimension size that does not match the embedding array output size of your AI model

The mistake: Configuring the index with DIMENSION 768 to save space, while passing 1536-dimension vectors generated by OpenAI's API in your write queries.

Why it's wrong: The vector index requires the coordinate space size to match exactly.

If the dimension sizes do not match, the database engine will throw validation errors and reject the insert writes.

Fix: Verify the output size of your embedding model, and configure the DIMENSION parameter to match it exactly.

Mistake 2: Querying Vector Indexes Without Exact Array Dimension Matching

The mistake: Inserting 1536-dimension embeddings into a vector index defined for 768-dimension vectors.

Why it's wrong: Vector similarity indexes (HNSW / MTREE) require exact dimension vector matching. Mismatched vector lengths cause index insertion failures.

Incorrect:

DEFINE INDEX vec_idx ON TABLE doc FIELDS embedding MTREE DIMENSION 768;
CREATE doc SET embedding = [0.1, 0.2 ... 1536 items]; // ❌ Dimension mismatch!

Fix:

DEFINE INDEX vec_idx ON TABLE doc FIELDS embedding MTREE DIMENSION 1536; // Matching dimension count

Mistake 3: Confusing Vector Distance Metrics (EUCLIDEAN, COSINE, MANHATTAN)

The mistake: Using EUCLIDEAN distance search for normalized AI text embeddings trained on COSINE similarity.

Why it's wrong: Matching vector index distance metrics to embedding model training specifications (e.g. OpenAI embeddings use Cosine) is essential for retrieval accuracy.

Incorrect:

DEFINE INDEX vec_idx ON TABLE doc FIELDS embedding MTREE DIMENSION 1536 DIST EUCLIDEAN;

Fix:

DEFINE INDEX vec_idx ON TABLE doc FIELDS embedding MTREE DIMENSION 1536 DIST COSINE;

5. Practice Exercises

Exercise 1: Defining HNSW Vector Search Indexes

Scenario: Configure an HNSW vector index doc_embedding_idx on table document for 4-dimensional vector embeddings using Cosine distance.

Requirements:

1. Define field embedding as array<float>.
2. Write DEFINE INDEX doc_embedding_idx ON TABLE document COLUMNS embedding HNSW DIMENSION 4 DIST COSINE.

DEFINE TABLE document SCHEMAFULL;
DEFINE FIELD embedding ON TABLE document TYPE array<float>;

-- Define HNSW vector index
DEFINE INDEX doc_embedding_idx ON TABLE document COLUMNS embedding HNSW DIMENSION 4 DIST COSINE;

Exercise 2: K-Nearest Neighbor Vector Similarity Queries

Scenario: Query the top 2 documents most similar to target query vector [0.1, 0.2, 0.3, 0.4] using vector distance ordering.

Requirements:

1. Write a SELECT query sorting by vector::distance::knn().
2. Apply LIMIT 2.

CREATE document:d1 SET embedding = [0.1, 0.2, 0.3, 0.4];
CREATE document:d2 SET embedding = [0.9, 0.8, 0.7, 0.6];

-- Vector K-Nearest Neighbor similarity search
SELECT *, vector::distance::knn() AS dist 
FROM document 
WHERE embedding <|2,COSINE|> [0.1, 0.2, 0.3, 0.4];

Exercise 3: Distance Metric Selection (Euclidean vs Cosine)

Scenario: Compare EUCLIDEAN vs COSINE distance metrics for vector index configuration.

Requirements:

1. Describe when to use EUCLIDEAN vs COSINE metrics.

COSINE: Ideal for normalized text embeddings (e.g. OpenAI / Cohere embeddings) measuring angle direction.
EUCLIDEAN: Ideal for spatial coordinate vectors and unnormalized magnitude distance measurements.

6. Related Terms

• DEFINE INDEX (Deep Dive) — The parent index context.
• Vector Search Index (ML/AI) — Querying vectors.
• DEFINE INDEX ... HNSW (Approximate Vector Search) — HNSW vector indexing.

7. Key Takeaways

• Vector indexes enable semantic and AI similarity search in SurrealDB.
• Replaces separate vector databases (like Pinecone) in RAG architectures.
• Store embeddings in standard array<float> or array<decimal> fields.
• Supported algorithms include MTREE and HNSW.
• Configure the index DIMENSION to match your embedding model size exactly.
• Supported distance metrics include COSINE, EUCLIDEAN, and MANHATTAN.
• Mismatched dimension sizes will cause database write failures.