vortexa 🧠

Codebase indexing and semantic search engine

Dense + sparse hybrid retrieval · AST-aware chunking · LMDB persistence · MCP server

Overview

vortexa is a standalone codebase indexing and semantic search engine designed for AI agents and developers. It builds a persistent, hybrid search index over source code using:

Dense retrieval via VortexEmbedderV4 (native LF2/LF4 dequant, SIF+PC, Matryoshka) or static embeddings (Model2Vec / SentenceTransformers)
Sparse retrieval via BM25 keyword scoring
AST-aware chunking that respects function and class boundaries via tree-sitter
LMDB-backed storage for fast, persistent vector and chunk storage

The result: natural language code search that understands intent, not just keywords.

results = indexer.search("authentication middleware that validates JWT tokens", top_k=5)
# → Finds the right files even if they use "auth", "verify", "token" instead of "authentication"

vortexa can run as a standalone Python library, be embedded into any agent, or serve as an MCP server for LLM tools.

Features

Semantic search	Find code by describing what it does in natural language — no exact-string matching needed.
Hybrid retrieval	Combines dense embeddings (semantic meaning) with BM25 (keyword precision) using adaptive alpha weighting.
AST-aware chunking	Splits source code at function/class/block boundaries using tree-sitter when available, falls back to line-based splitting.
Incremental indexing	Content-hash memoization means only changed files are re-indexed. Full re-index avoids redundant embedding computations.
Persistent storage	LMDB-backed vector store survives restarts. Embedding cache avoids recomputing identical content.
Live watch mode	Background thread polls for file changes and auto-re-indexes with configurable debounce.
MCP server	Expose `search`, `resolve`, and `explain` tools for MCP-compatible agents (Claude Code, Cursor, etc.)
Zero mandatory heavy deps	Core requires only `numpy`, `lmdb`, and `pathspec`. Model2Vec and tree-sitter are optional extras.

Quick Start

Installation

# Core (BM25 + line-based chunking)
pip install vortexa

# Full (Model2Vec embeddings + tree-sitter AST chunking)
pip install "vortexa[full]"

# With MCP server support
pip install "vortexa[mcp]"

Index a codebase

from vortexa.core.indexer import CodebaseIndexer

indexer = CodebaseIndexer(root=".")
stats = indexer.index()

print(f"Indexed {stats.indexed_files} files, {stats.total_chunks} chunks")
print(f"Languages detected: {stats.languages}")

Search with natural language

results = indexer.search("CSV parser implementation", top_k=5)

for r in results:
    print(f"{r.chunk.file_path}:{r.chunk.start_line}  score={r.score:.3f}")
    print(f"  {r.chunk.content[:150].strip()}")
    print()

Output:

src/parsers/csv_parser.py:42  score=0.892
  def parse_csv(filepath: str, delimiter: str = ",") -> list[dict]:
      """Parse a CSV file into a list of dictionaries."""
      with open(filepath, "r") as f:

tests/test_csv_parser.py:15  score=0.756
  def test_parse_csv_with_header():
      result = parse_csv("test.csv")
      assert len(result) == 3

Python API

Indexing

from vortexa.core.indexer import CodebaseIndexer
from vortexa.core.types import ChunkConfig

# Default chunking (aim for 50-line chunks, 5-line overlap)
indexer = CodebaseIndexer(root="/path/to/project")
stats = indexer.index()
# → IndexStats(indexed_files=127, total_chunks=843, languages={"python": 45, "typescript": 32, ...})

# Custom chunk configuration
indexer = CodebaseIndexer(
    root=".",
    chunk_config=ChunkConfig(chunk_size=100, chunk_overlap=10),
)
stats = indexer.index(force=False, include_text_files=True)

# Force full re-index
stats = indexer.index(force=True)

Searching

# Hybrid search (auto-weighted semantic + BM25)
results = indexer.search("error handling", top_k=10)

# Pure semantic search
results = indexer.search("database connection pool", top_k=5, alpha=1.0)

# Pure BM25 keyword search
results = indexer.search("parse csv", top_k=5, alpha=0.0)

# Symbol lookup (find definitions by name)
results = indexer.find_symbol("ConnectionPool", top_k=5)

# Related chunks (find chunks similar to a given chunk index)
results = indexer.find_related(chunk_idx=3, top_k=5)

Each result is a SearchResult with:

Field	Type	Description
`chunk.file_path`	`str`	Relative file path
`chunk.start_line`	`int`	Start line number
`chunk.end_line`	`int`	End line number
`chunk.content`	`str`	Code snippet (up to 500 chars)
`chunk.language`	`str`	Detected programming language
`chunk.lineage`	`Lineage`	Source path + byte offsets
`chunk.chunk_hash`	`str`	Content hash for memoization
`score`	`float`	Relevance score (0–1)
`source`	`str`	`"semantic"`, `"bm25"`, or `"hybrid"`

Watch Mode

from vortexa.interfaces.watcher import IndexWatcher

watcher = IndexWatcher(indexer, poll_interval=3.0)
watcher.start()   # Background thread, polls every 3s, debounces 2s
# ... files change on disk, auto-re-index happens ...
watcher.stop()

Management

# Index statistics
stats = indexer.stats()
# → {indexed_files: 127, total_chunks: 843, languages: {...}, memo_hits: 42, memo_misses: 15}

# Reset
indexer.clear()   # Delete the persistent index

CLI Search

The installed vortexa command can also search a codebase directly:

# Search the current working directory
vortexa -q "authentication middleware that validates JWT tokens"

# Search a specific codebase root
vortexa -q "CSV parser implementation" /path/to/project

# Pass Kilo-style environment details; `Working directory` is used as the root
vortexa -q "error handling" "Working directory: /path/to/project
Workspace root folder: /"

Useful flags:

Flag	Description
`-q`, `--query`	Search query. Quote multi-word queries.
`--root`	Codebase root to index and search. Overrides `environment_details`.
`--top-k`	Maximum number of results to return. Default: `10`.
`--alpha`	Semantic weight from `0.0` to `1.0`; defaults to adaptive weighting.
`--include-text`	Include text files such as `.md`, `.json`, and `.yaml` in the index.
`--force`	Force a full re-index before searching.
`--no-index`	Search the existing index only.
`--plain`	Print human-readable results instead of JSON.
`--model`	Embedding model ID or alias (`mini`, `nano`; both use LF2 by default). Use `mini-full` or `nano-full` for LF4.
`-f`, `--fast`	Explicitly select LF2 for a full-model alias. The `mini` and `nano` aliases already use LF2.

By default CLI output is JSON:

[
  {
    "file": "src/auth/middleware.py",
    "lines": "12-48",
    "score": 0.892,
    "source": "hybrid",
    "content": "def validate_jwt(token: str) -> User: ..."
  }
]

The vortexa command still starts the MCP server when no query is provided. You can also start the server explicitly:

vortexa serve
# or
vortexa-serve

Model Configuration

vortexa supports configurable embedding models. The default is mini (VTXAI/vtx-embed-7M-lf2), using the native LF2 2-bit checkpoint. Use mini-full or nano-full to select the original LF4 checkpoints.

Available Models

Alias	Model ID	Description
`mini`	`VTXAI/vtx-embed-7M-lf2`	Default native 2-bit 7M model. 256-dimensional embeddings.
`nano`	`VTXAI/vtx-embed-1M-lf2`	Default native 2-bit 1M model. 64-dimensional embeddings.
`mini-full`	`VTXAI/vtx-embed-7M`	Original LF4 checkpoint for maximum quality.
`nano-full`	`VTXAI/vtx-embed-1M`	Original LF4 lightweight checkpoint.

CLI Usage

# Use the nano model (smaller, faster)
vortexa -q "authentication" --model nano /path/to/project

# Use mini explicitly (same as default)
vortexa -q "authentication" --model mini /path/to/project

# Use a custom HuggingFace model ID
vortexa -q "authentication" --model VTXAI/vtx-embed-1M /path/to/project

Python API

from vortexa.core.indexer import CodebaseIndexer

# Use nano model (LF2 by default)
indexer = CodebaseIndexer(root="/path/to/project", model_id="nano")

# Use mini model (LF2 by default)
indexer = CodebaseIndexer(root="/path/to/project", model_id="mini")

# Opt into the original LF4 checkpoint
indexer = CodebaseIndexer(root="/path/to/project", model_id="mini-full")

Alternative Embedders

The embedding.py module also provides SentenceTransformerEmbedder for transformer-based models (requires the sentence-transformers package).

Inference API

vortexa can be used as a standalone embedding inference engine for VTXAI models, similar to how sentence-transformers works but purpose-built for Vortex-Embed models.

Python API

from vortexa.core.inference import embed

# Encode a single string (default: mini model)
vec = embed("India is a diverse country")

# Encode multiple strings with the nano model
vecs = embed(["Indian cricket team is strong", "Chennai is a major city"], model="nano")

# Use any HuggingFace model ID
vecs = embed(["Indian agriculture output"], model="VTXAI/vtx-embed-7M")

# Shape: (1, D) for single text, (N, D) for batch
print(vec.shape)

CLI

# Encode text with the mini model (default)
vortexa embed "India is a diverse country"

# Encode with the nano model
vortexa embed "Indian cricket team is strong" --model nano

# Encode multiple strings
vortexa embed "India has 28 states" "Chennai is in Tamil Nadu" --model nano

# Use a custom model ID
vortexa embed "Indian monsoon patterns" --model VTXAI/vtx-embed-7M

MCP Server

# Auto-indexes current directory, serves on stdio
python -m vortexa.interfaces.mcp_server

# Or via the installed entry point
vortexa serve

On startup it indexes the current working directory and prints stats to stderr:

[vortexa] Indexing C:\projects\my-app ...
[vortexa] Ready: 127 files, 843 chunks
[vortexa] Auto-reindex watcher started (polling every 3s)

The server exposes three tools:

Tool	Description	Arguments
`search`	Semantic + BM25 hybrid code search	`query` (str), `top_k` (int, default 10)
`resolve`	Feature-level search with graph context, tests, imports, callers, and callees	`query` (str), `top_k` (int, default 5)
`explain`	Explain a file, `file:line` location, or symbol	`location` (str)

Use search for discovery, resolve when you need feature context, and explain when the file or symbol is already known. The server indexes the current directory and auto-reindexes it while running.

Usage with Claude Code / Cursor

Add to your MCP configuration file (~/.cursor/mcp.json or Claude Code's mcp_servers config):

{
  "mcpServers": {
    "vortexa": {
      "command": "python",
      "args": ["-m", "vortexa.interfaces.mcp_server"],
      "cwd": "/path/to/your/project"
    }
  }
}

The agent will now have access to semantic code search — it can find functions, classes, and patterns by describing them in natural language. This is significantly more effective than grep or rg for exploratory queries.

Agent Skill

Vortexa ships with an Agent Skills-compatible skill at skills/vortexa/SKILL.md. It teaches compatible agents when to prefer Vortexa, how to choose between search, resolve, and explain, and how to interpret returned file paths, line ranges, scores, and graph context.

The skill follows the standard SKILL.md structure: YAML frontmatter with a lowercase hyphenated name and trigger-oriented description, followed by workflow instructions. Copy or link the skills/vortexa directory into the skills directory used by your agent environment.

---
name: vortexa
description: Use Vortexa to index and semantically search local codebases...
---

Use the installed MCP server when available; otherwise the same workflow is available through the vortexa search, vortexa resolve, and vortexa explain commands.

Architecture

Directory Layout

vortexa/
├── core/
│   ├── indexer.py       # CodebaseIndexer — main orchestrator
│   ├── chunking.py      # AST-aware (tree-sitter) + line-based chunking
│   ├── embedding.py     # Embedding model wrappers (Model2Vec, SentenceTransformers, LF4)
│   ├── language.py      # Language detection & file extension mapping
│   └── types.py         # Shared types (Chunk, ChunkConfig, IndexStats, SearchResult, ...)
├── storage/
│   ├── vector_store.py  # LMDB-backed persistent vector store
│   ├── bm25.py          # BM25 keyword index with persistent storage
│   └── walker.py        # File system walker with .gitignore support
├── search/
│   ├── search.py        # Hybrid search orchestrator (dense + sparse)
│   ├── ranking.py       # Result ranking & symbol query detection
│   └── tokens.py        # Identifier tokenization (camelCase, snake_case)
└── interfaces/
    ├── cli.py           # Command-line search entrypoint
    ├── mcp_server.py    # MCP server (stdio transport)
    └── watcher.py       # Live file poller with debounced auto-reindex

Data Flow

sequenceDiagram
    participant User as User Code
    participant Indexer as CodebaseIndexer
    participant Walker as File Walker
    participant Chunker as Chunking Engine
    participant Embedder as Embedding Model
    participant Store as LMDB Vector Store
    participant BM25 as BM25 Index
    participant Search as Search Engine

    User->>Indexer: index()
    Indexer->>Walker: walk_files(root, extensions)
    Walker-->>Indexer: file_paths
    loop Each file
        Indexer->>Chunker: chunk_source(source, language)
        Chunker-->>Indexer: list[Chunk]
        Indexer->>Embedder: embed(chunks)
        Embedder-->>Indexer: vectors
        Indexer->>Store: store(vectors, chunks)
        Indexer->>BM25: index(chunks)
    end
    Indexer-->>User: IndexStats

    User->>Search: search(query)
    Search->>Store: query(vector)
    Search->>BM25: query(tokens)
    Search->>Search: hybrid_fusion(results)
    Search-->>User: list[SearchResult]

Indexing Pipeline

graph LR
    A[Source Files] --> B[File Walker<br/>.gitignore aware]
    B --> C[Language Detector]
    C --> D{AST Available?}
    D -->|Yes| E[Tree-sitter Parser<br/>Function/class boundaries]
    D -->|No| F[Line-based Splitter<br/>Configurable size/overlap]
    E --> G[Chunk Set]
    F --> G
    G --> H[Embedding Model<br/>VortexEmbedderV4 (LF4) / Model2Vec / SentenceTransformer]
    G --> I[BM25 Tokenizer]
    H --> J[(LMDB Vector Store)]
    I --> K[(BM25 Index)]
    J --> L[Content Hash Memo]
    K --> L
    L --> M[Skip unchanged files]

Module Dependencies

graph TD
    subgraph "Public API"
        Indexer["core.indexer<br/>CodebaseIndexer"]
        Search["search.search<br/>search_hybrid()"]
    end

    subgraph "Core"
        Chunking["core.chunking<br/>chunk_source()"]
        Embedding["core.embedding<br/>Embedder"]
        Language["core.language<br/>detect_language()"]
        Types["core.types<br/>Chunk, ChunkConfig, ..."]
    end

    subgraph "Storage"
        VectorStore["storage.vector_store<br/>LMDB Vector Store"]
        BM25["storage.bm25<br/>BM25 Index"]
        Walker["storage.walker<br/>walk_files()"]
    end

    subgraph "Interfaces"
        CLI["interfaces.cli<br/>Command-line search"]
        MCP["interfaces.mcp_server<br/>FastMCP server"]
        Watcher["interfaces.watcher<br/>IndexWatcher"]
    end

    Indexer --> Chunking
    Indexer --> Embedding
    Indexer --> Language
    Indexer --> Types
    Indexer --> VectorStore
    Indexer --> BM25
    Indexer --> Walker
    Indexer --> Search

    Search --> Embedding
    Search --> VectorStore
    Search --> BM25
    Search --> Types

    CLI --> Indexer
    MCP --> Indexer
    MCP --> Watcher
    Watcher --> Walker

Dependencies

Package	Required	Used For
`numpy`	Yes	Vector operations, embedding inference
`lmdb`	Yes	Persistent vector and chunk metadata storage
`bm25s`	Yes	Fast BM25 keyword index and persistence
`pathspec`	Yes	`.gitignore` pattern matching in file walker
`model2vec`	Optional	Alternative static embeddings
`huggingface-hub`	Yes (default model)	Loading LF2 defaults (`VTXAI/vtx-embed-7M-lf2`, `VTXAI/vtx-embed-1M-lf2`)
`tokenizers`	Yes (default model)	HF tokenizer for embedding model
`safetensors`	Yes (default model)	Safe tensor loading for 4-bit weights
`sentence-transformers`	Optional	Transformer-based dense embeddings
`model2vec`	Optional	Alternative static embeddings
`tree-sitter-language-pack`	Optional	AST-aware code chunking
`fastmcp`	Optional	MCP server for LLM tool integration

Install optional groups:

pip install "vortexa[full]"     # model2vec + sentence-transformers + tree-sitter
pip install "vortexa[full, mcp]" # everything including MCP server

License

Copyright 2025 VortexAI

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

Name		Name	Last commit message	Last commit date
docs		docs
skills/vortexa		skills/vortexa
src/vortexa		src/vortexa
tests		tests
.gitignore		.gitignore
CHANGELOG.md		CHANGELOG.md
LICENSE		LICENSE
MANIFEST.in		MANIFEST.in
README.md		README.md
pyproject.toml		pyproject.toml
uv.lock		uv.lock

Navigation Menu

Repository files navigation

vortexa 🧠

Table of Contents

Overview

Features

Quick Start

Installation

Index a codebase

Search with natural language

Python API

Indexing

Searching

Watch Mode

Management

CLI Search

Model Configuration

Available Models

CLI Usage

Python API

Alternative Embedders

Inference API

Python API

CLI

MCP Server

Usage with Claude Code / Cursor

Agent Skill

Architecture

Directory Layout

Data Flow

Indexing Pipeline

Module Dependencies

Dependencies

License

About

Releases

Packages

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

vortexa 🧠

Table of Contents

Overview

Features

Quick Start

Installation

Index a codebase

Search with natural language

Python API

Indexing

Searching

Watch Mode

Management

CLI Search

Model Configuration

Available Models

CLI Usage

Python API

Alternative Embedders

Inference API

Python API

CLI

MCP Server

Usage with Claude Code / Cursor

Agent Skill

Architecture

Directory Layout

Data Flow

Indexing Pipeline

Module Dependencies

Dependencies

License

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages