How to use local embeddings with ChromaDB for semantic search

Why build a fully local vector search engine?

When interacting with artificial intelligence assistants, we often need them to know our own private documents, codebases, or personal notes. Sending this sensitive data to external cloud APIs can violate privacy and security policies. That is exactly where local semantic search comes in, allowing you to process everything on your own machine.

Simply put, embeddings are numerical representations of words or phrases that capture the actual meaning of the text rather than just exact keyword matches. When a system understands meaning, it can find relevant documents even if the user phrases the query differently. To deepen your understanding of processing and scaling this approach, it is worth exploring how to manage embeddings in vector databases to avoid performance bottlenecks.

The great advantage of keeping this architecture on your own computer or private server is absolute control over your information. You do not depend on an internet connection, you do not pay for API requests, and you ensure confidential data never leaves your security perimeter. In practice, this means running compact models directly on your hardware using open-source tools.

Understanding the core concepts of vectors and similarity

For a computer to understand text, it must turn letters into numbers. This process is handled by specialized language models called encoders, which read a sentence and output a long list of decimal numbers known as a vector. Texts with similar meanings generate lists of numbers whose points sit very close together in a multi-dimensional mathematical map.

Imagine a giant chart where the word 'dog' sits close to 'cat' but far away from 'airplane'. The search engine calculates the mathematical distance between the user's question and the stored documents, finding the most relevant excerpts in milliseconds. This approach solves the classic flaw of traditional keyword search, which fails if you forget the exact terminology.

To put this theory into practice every day, many developers combine vector databases with local inference engines, as we explore in our guide on how to run local AI with Llama.cpp and REST API, creating ecosystems completely independent of external services.

Preparing the Python environment and installing libraries

The first practical step to get the search engine running is setting up an isolated Python environment. Using virtual environments prevents version conflicts between packages installed on the operating system. We will install ChromaDB, the lightweight vector database that will store our data, and the transformer library to generate embeddings.

Open your terminal and run the commands below to create your project folder, activate the virtual environment, and install the required dependencies:

python3 -m venv venv
source venv/bin/activate
pip install chromadb sentence-transformers

With these tools ready, your computer has everything it needs to turn text into numbers and index it locally without relying on any commercial API keys.

Creating the vector database and inserting documents

Now that the environment is configured, let's write the Python code responsible for initializing ChromaDB and adding some sample documents. ChromaDB automatically manages data persistence on disk, saving everything into a local folder so you do not lose data when closing the script.

Create a file named indexer.py on your computer and paste the functional code below, which reads short sentences, converts them into vectors, and saves them to the database:

import chromadb
from chromadb.utils import embedding_functions

# Initialize ChromaDB client pointing to a local folder
client = chromadb.PersistentClient(path='./my_vector_db')

# Use an efficient and lightweight default model for embeddings
emb_fn = embedding_functions.SentenceTransformerEmbeddingFunction(
    model_name='all-MiniLM-L6-v2'
)

# Create or retrieve a document collection
collection = client.get_or_create_collection(
    name='internal_documents',
    embedding_function=emb_fn
)

# Add sample documents to the local database
collection.add(
    documents=[
        'The database server experienced slowdowns due to high IOPS usage.',
        'Security policy requires passwords to be at least twelve characters long.',
        'Daily backups run automatically every morning at three o'clock.'
    ],
    ids=['doc1', 'doc2', 'doc3']
)

print('Documents successfully indexed in local ChromaDB!')

Run the script in your terminal using python indexer.py. In practice, ChromaDB will download the lightweight model on the first run, process the texts, and write the vector files into the my_vector_db folder created in your project directory.

Executing semantic searches and retrieving context for LLMs

With data indexed, the next step is querying the database by asking a question in natural language. The engine will turn your question into a vector, compare it against all saved documents, and return the most relevant excerpts. This retrieved context can be fed directly into an artificial intelligence to generate accurate answers based on real facts.

Create a file named search.py and insert the code below to test semantic retrieval:

import chromadb
from chromadb.utils import embedding_functions

client = chromadb.PersistentClient(path='./my_vector_db')

emb_fn = embedding_functions.SentenceTransformerEmbeddingFunction(
    model_name='all-MiniLM-L6-v2'
)

collection = client.get_collection(
    name='internal_documents',
    embedding_function=emb_fn
)

# Natural language question that does not use exact document words
question = 'How is disk performance?'

results = collection.query(
    query_texts=[question],
    n_results=1
)

print('Question:', question)
print('Most relevant document found:', results['documents'][0])

When running this script, notice that even without using the word 'IOPS' or 'slowdowns' in the question, the system understood the meaning and located the correct document about the server. This is the core of context retrieval for LLMs. To explore these architectural concepts further in production, see our related technical analysis on How to manage concurrency in Rust with Tokio and Channels.

Maintenance best practices and final considerations

Building a local vector search engine transforms how we handle private data and artificial intelligence automation. To keep the system fast as your database grows, it is important to monitor persistence folder sizes and update embedding models as newer, more efficient versions emerge in the open-source community.

In short, combining Python, ChromaDB, and local models eliminates recurring API costs and shields your information from external leaks. Adopting this architecture ensures complete data sovereignty and opens doors to building truly useful, secure intelligent assistants within your own infrastructure.