Semantic Search with Sentence Transformers and Qdrant

Semantic Search Architecture

Semantic search understands meaning rather than just matching keywords. It uses embedding models to convert text into vectors and stores them in a vector database for similarity search.

Installing Components

# Install libraries
pip install sentence-transformers qdrant-client

# Run Qdrant vector database
docker run -d --name qdrant \
    -p 6333:6333 -p 6334:6334 \
    -v /opt/qdrant/storage:/qdrant/storage \
    --restart unless-stopped \
    qdrant/qdrant

Building the Search Engine

from sentence_transformers import SentenceTransformer
from qdrant_client import QdrantClient
from qdrant_client.models import VectorParams, Distance, PointStruct

# Initialize
model = SentenceTransformer("all-MiniLM-L6-v2")
client = QdrantClient(host="localhost", port=6333)

# Create collection
client.create_collection(
    collection_name="documents",
    vectors_config=VectorParams(size=384, distance=Distance.COSINE)
)

# Index documents
documents = [
    "How to set up a VPS server",
    "Configuring Nginx reverse proxy",
    "Docker container management guide",
    # ... more documents
]

points = []
for i, doc in enumerate(documents):
    embedding = model.encode(doc).tolist()
    points.append(PointStruct(id=i, vector=embedding, payload={"text": doc}))

client.upsert(collection_name="documents", points=points)

# Search
query = "How do I configure a web server?"
query_vector = model.encode(query).tolist()
results = client.search(
    collection_name="documents",
    query_vector=query_vector,
    limit=5
)
for r in results:
    print(f"{r.score:.3f}: {r.payload['text']}")

Best Practices

• Choose embedding model based on your language and domain
• Use batch encoding for large document sets
• Implement hybrid search (vector + keyword) for best results
• Monitor Qdrant memory usage and configure disk-based storage for large collections