Understanding pgvector Postgres extension, neighbour gem and vector embeddings in Rails

Conversation between AI and I to understand step 1 of working with AI APIs in Rails applications

Q: What does the line has_neighbors :body_embedding in my model do?

A: The line has_neighbors :body_embedding is a method provided by the `neighbor` gem that sets up vector similarity search functionality in Rails using the pgvector PostgreSQL extension. Here's what it does:

1. Establishes a Relationship: It tells Rails that the :body_embedding column (a vector data type) will be used for finding similar records based on vector similarity.

2. Adds Query Methods: It adds methods like nearest_neighbors to your model, which you can use to find similar records based on vector similarity.

3. Configures Distance Calculations: It sets up the necessary database query mechanisms to calculate distances between vectors (using metrics like cosine similarity).

4. Enables Practical Usage: With this single line of code, your model can now perform queries like:

# Find similar emails based on semantic content 

similar_emails = email.nearest_neighbors(:body_embedding, distance: "cosine") .where(account: account) .first(10) .select { |state| state.neighbor_distance < 0.5 }

The :body_embedding refers to a vector column in the database (typically defined as :vector(256) or similar) that stores numerical vectors representing semantic embeddings of content.

Q: Which gem enables the has_neighbors method?

A: The project is using the neighbor gem to enable the has_neighbors functionality for pgvector integration with Rails. This gem serves as a Ruby wrapper for pgvector that:

• Adds ActiveRecord integration for vector columns

• Provides methods like has_neighbors to set up similarity search

• Handles the complexity of vector distance calculations

• Offers a clean Rails-like API for vector operations

gem "neighbor"

Q: What is the pgvector PostgreSQL extension?

A: pgvector is a PostgreSQL extension that adds support for vector similarity search to PostgreSQL databases.

A PostgreSQL extension is a module that extends the functionality of the PostgreSQL database system without modifying the core code. Extensions:

• Are packaged as libraries that can be loaded into the database

• Can be installed, upgraded, and removed using SQL commands

• Are isolated from the core PostgreSQL code, making them safer to use

• Can be maintained independently of the main PostgreSQL release cycle

pgvector specifically introduces:

• A new vector data type for storing embeddings

• Vector similarity search operators and functions

• Indexing methods for efficient vector search

Q: How do I install and use pgvector?

A: Here's how to set up pgvector:

1. System-level Installation

# For Ubuntu/Debian
sudo apt install postgresql-14-pgvector  # Replace 14 with your PostgreSQL version

# For macOS with Homebrew
brew install pgvector

# From source
git clone --branch v0.5.1 https://github.com/pgvector/pgvector.git
cd pgvector
make
make install

2. Database-level Installation

Enable the extension in your database:

CREATE EXTENSION vector;

In Rails, create a migration:

class EnablePgvectorExtension < ActiveRecord::Migration[7.0]
  def change
    enable_extension "vector"
  end
end

3. Adding Vector Columns to Existing Tables

class AddEmbeddingToEmails < ActiveRecord::Migration[7.0]
  def change
    add_column :emails, :embedding, :vector, limit: 384
  end
end

The limit: 384 specifies the dimensionality of your vectors.

Q: How is pgvector stored in my database?

A: pgvector doesn't maintain separate tables - it integrates with your existing database structure. Vector columns are stored alongside regular columns in your tables.

When you install pgvector:

• It adds a new vector data type to your PostgreSQL instance

• It adds new functions and operators for vector operations

• It adds new index types for vector similarity search

• Your existing data and tables remain unchanged

Q: How do I set up complete Rails integration with the neighbor gem?

A: Here's a complete setup process:

1. Setup

# Gemfile
gem "neighbor"
gem "pgvector"

# Migration
class SetupVectorSearch < ActiveRecord::Migration[7.0]
  def change
    enable_extension "vector"
    add_column :emails, :body_embedding, :vector, limit: 384
    add_index :emails, :body_embedding, using: :hnsw, opclass: :vector_cosine_ops
  end
end

2. Model Configuration

class Email < ApplicationRecord
  has_neighbors :body_embedding
  
  # Optional: Embed text when it changes
  after_save :generate_embedding, if: -> { body_changed? && body.present? }
  
  private
  
  def generate_embedding
    EmbedEmailJob.perform_later(id)
  end
end

3. Background Job for Embedding Generation

class EmbedEmailJob < ApplicationJob
  queue_as :default
  
  def perform(email_id)
    email = Email.find(email_id)
    return if email.body.blank?
    
    embedding = OpenAiService.generate_embedding(email.body)
    email.update_column(:body_embedding, embedding)
  end
end

4. Service for API Interaction

class OpenAiService
  def self.generate_embedding(text)
    client = OpenAI::Client.new(access_token: ENV["OPENAI_API_KEY"])
    
    response = client.embeddings(
      parameters: {
        model: "text-embedding-3-small",
        input: text
      }
    )
    
    response.dig("data", 0, "embedding")
  end
end

5. Using Vector Search in Your Application

class EmailsController < ApplicationController
  def similar
    @email = Email.find(params[:id])
    @similar_emails = @email.nearest_neighbors(:body_embedding, distance: :cosine)
      .where.not(id: @email.id)
      .limit(5)
  end
  
  def search
    if params[:query].present?
      query_embedding = OpenAiService.generate_embedding(params[:query])
      @emails = Email.nearest_neighbors_for_vector(:body_embedding, query_embedding, distance: :cosine)
        .limit(20)
    else
      @emails = Email.none
    end
  end
end

Q: How is pgvector used in AI applications?

A: pgvector enables several AI features in applications:

• Semantic Search: Find documents by meaning, not just keywords

• Recommendation Systems: "Users who liked X also liked Y"

• Duplicate Detection: Find similar or duplicate content

• Classification: Group items by semantic similarity

• Personalization: Match user preferences to content

• Question Answering: Find relevant context for generating answers

In the Cora email client example, pgvector is used for:

1. Email Processing:

# When a new email arrives
email = EmailProcessingState.create!(
  account: current_account,
  body: email_body,
  # other attributes...
)

# Schedule embedding generation
EmbedEmailBodyJob.perform_later(email.id)

2. Finding Similar Emails:

# When generating a response
def generate_response(email)
  # Find similar emails to learn from
  similar_emails = email.nearest_neighbors(:body_embedding, distance: "cosine")
    .where(account: email.account)
    .where.not(actual_response: nil)
    .limit(5)
  
  # Use similar emails as examples for AI response generation
  example_responses = similar_emails.pluck(:actual_response)
  
  # Generate response using AI with examples
  # ...
end

This approach enables learning from past email interactions and generating more personalized, contextually appropriate responses.

Q: What's the cost of using OpenAI embeddings with pgvector?

A: OpenAI's text-embedding-3-small costs $0.02 per 1M tokens. Here's what that means in practical terms:

Token Understanding

• About 4 characters per token for English text

• About 0.75 tokens per word (on average)

• A typical email of 200 words ≈ 150 tokens

Cost Scenarios for an Email System

For 1,000 Active Users

• Assuming each user receives 50 emails per day

• Each email averages 200 words (≈ 150 tokens)

• Daily: 1,000 users × 50 emails × 150 tokens = 7.5M tokens

• Daily cost: 7.5M tokens × $0.02 per 1M tokens = $0.15 per day

• Monthly cost: $0.15 × 30 days = $4.50 per month

For 10,000 Active Users

• Monthly cost: approximately $45 per month

For 100,000 Active Users

• Monthly cost: approximately $450 per month

Cost Optimization Strategies

• Selective Embedding: Only embed important emails

• Truncate Long Emails: Cap token count for very long emails

• Batch Processing: Process emails in batches to reduce API calls

• Caching: Store embeddings for reuse in similar contexts

Cost Comparison

• Embedding costs for 10,000 users: $45/month

• GPT-4 costs for generating responses: ~$1,500-3,000/month

• Database hosting costs: ~$200-500/month

• Developer salary: ~$5,000-15,000/month

Embeddings are one of the most cost-effective components of an AI system, providing exceptional value for their cost by enabling semantic search and similarity matching that dramatically improve AI-generated responses.