Below you will find some suggested materials on these topics, which we believe may help you complete the final activity in the lesson. Please note that using these resources is entirely optional, and you are welcome to explore any other sources or courses at your own discretion.

Introduction (2 Minutes)

Opening Statement:

• Welcome to Lesson 3 of Module 3! In this session, we’ll explore how to build and query a vector database using FAISS, an industry-standard tool for efficient similarity searches.

• By the end of this lesson, you’ll understand how to store embeddings, perform similarity searches, and retrieve meaningful results.

Agenda:

1. What is FAISS, and why use it?

2. Key concepts: similarity metrics and vector databases.

3. Hands-on: Build a vector database and perform queries.

Part 1: What Is FAISS and Why Use It? (4 Minutes)

Definition:

• FAISS (Facebook AI Similarity Search) is an open-source library designed for efficient similarity search and clustering of dense vectors.

Why Use FAISS?

• Efficiency: Handles millions of embeddings with low latency.

• Scalability: Optimized for large-scale datasets.

• Flexibility: Supports different similarity metrics and indexing methods.

Use Cases:

• Recommendation Systems: Retrieve items similar to a user’s preference.

• Question Answering: Fetch contextually relevant documents for user queries.

• Clustering: Group similar data points for analysis.

FAISS Workflow Overview:

1. Index Construction: Store embeddings in a vector database.

2. Querying: Find the closest vectors to a given input using similarity metrics.

3. Updates: Add or remove embeddings dynamically.

Part 2: Key Concepts (4 Minutes)

1. Similarity Metrics:

• Metrics used to compare vectors in embedding space:

  • Cosine Similarity: Measures the angle between two vectors.

  • Euclidean Distance: Measures the straight-line distance between points.

• Example:

  • High cosine similarity or low Euclidean distance indicates closer embeddings.

2. Index Types in FAISS:

• Flat Index:

  • Stores all vectors and performs exhaustive search.

  • Best for small datasets.

• Approximate Indexes (e.g., IVF, HNSW):

  • Partition data for faster searches.

  • Suitable for large-scale datasets.

3. Vector Normalization:

• Ensures embeddings are on the same scale.

• Example:

  • Normalize vectors for cosine similarity searches.

Code Example: Vector Normalization

python

Copy code

from sklearn.preprocessing import normalize

# Normalize embeddings

normalized_embeddings = normalize(embeddings, norm='l2')

Part 3: Hands-On: Building and Querying a Vector Database (10 Minutes)

Step 1: Install FAISS

bash

Copy code

pip install faiss-cpu

Step 2: Create a Vector Database

Code Example: Building a Flat Index

python

Copy code

import faiss

import numpy as np

# Load embeddings

embeddings = np.load("embeddings.npy")

# Create a flat index

dimension = embeddings.shape[1]

index = faiss.IndexFlatL2(dimension)

# Add embeddings to the index

index.add(embeddings)

print(f"Number of vectors in the index: {index.ntotal}")

Step 3: Perform Similarity Searches

Code Example: Querying the Index

python

Copy code

# Example query vector (normalized)

query_vector = embeddings[0].reshape(1, -1)

# Search for the top 5 most similar vectors

k = 5

distances, indices = index.search(query_vector, k)

print("Distances:", distances)

print("Indices:", indices)

Explanation:

• distances: Measures how close each result is to the query.

• indices: Returns the row numbers of the closest embeddings.

Step 4: Advanced: Use Approximate Index

python

Copy code

# Create an IVF index for larger datasets

nlist = 100  # Number of partitions

quantizer = faiss.IndexFlatL2(dimension)

index_ivf = faiss.IndexIVFFlat(quantizer, dimension, nlist)

# Train the index

index_ivf.train(embeddings)

index_ivf.add(embeddings)

# Perform a search

distances, indices = index_ivf.search(query_vector, k)

print("Distances:", distances)

print("Indices:", indices)

Step 5: Save and Load Index

Save the index for reuse:
python
Copy code
faiss.write_index(index, "vector_index.faiss")

Load the saved index:
python
Copy code
loaded_index = faiss.read_index("vector_index.faiss")

Conclusion and Key Takeaways (4 Minutes)

Recap:

1. FAISS is a powerful tool for storing and querying embeddings efficiently.

2. Flat indexes work well for small datasets, while approximate indexes scale for larger ones.

3. Similarity metrics like cosine similarity or Euclidean distance are key to retrieving meaningful results.

Next Steps:

• Use the vector database to power applications like recommendation systems or retrieval-augmented generation.

• Experiment with different index types and similarity metrics to optimize performance for your dataset.

Objective

In this activity, you will:

1. Install and configure FAISS.

2. Build a vector database using the generated embeddings.

3. Perform similarity searches on the vector database to find the most relevant vectors.

Step 1: Install and Configure FAISS

FAISS is an open-source library for efficient similarity search and clustering of dense vectors.

Installation:

Install the FAISS CPU version (suitable for most systems):
bash
Copy code
pip install faiss-cpu

If you are using a GPU and have the necessary drivers installed, you can install the GPU version:
bash
Copy code
pip install faiss-gpu

Verify the installation:
python
Copy code

import faiss

print("FAISS version:", faiss.__version__)

Step 2: Load the Generated Embeddings

Ensure you have your embeddings saved in a file (embeddings.csv).

Code Example:

python

Copy code

import numpy as np

# Load the saved embeddings

embeddings = np.load("embeddings.csv")

# Display shape of embeddings

print("Embeddings Shape:", embeddings.shape)

Step 3: Build a Vector Database

FAISS provides different index types. For this example, we will use a Flat Index for exact nearest neighbor searches.

Code Example:

python

Copy code

import faiss

# Get the dimensionality of the embeddings

dimension = embeddings.shape[1]

# Create a FlatL2 index

index = faiss.IndexFlatL2(dimension)

# Add embeddings to the index

index.add(embeddings)

# Verify the number of vectors in the index

print("Number of vectors in the index:", index.ntotal)

Explanation:

• FlatL2 Index: Uses exact nearest neighbor search with Euclidean distance.

Step 4: Perform Similarity Searches

Search the vector database for the most similar vectors to a given query.

Code Example:

python

Copy code

# Select a query vector (e.g., the first embedding)

query_vector = embeddings[0].reshape(1, -1)

# Perform a similarity search for the top 5 nearest neighbors

k = 5

distances, indices = index.search(query_vector, k)

# Display the results

print("Distances:", distances)

print("Indices:", indices)

Explanation:

• distances: The Euclidean distances to the nearest neighbors.

• indices: The indices of the nearest neighbors in the original dataset.

Step 5: Optimize for Larger Datasets (Optional)

For large datasets, you can use an IVF Index to partition the data for faster searches.

Code Example:

python

Copy code

# Create an IVF index

nlist = 100  # Number of partitions

quantizer = faiss.IndexFlatL2(dimension)

ivf_index = faiss.IndexIVFFlat(quantizer, dimension, nlist)

# Train the index with the embeddings

ivf_index.train(embeddings)

ivf_index.add(embeddings)

# Perform a similarity search

distances, indices = ivf_index.search(query_vector, k)

# Display the results

print("Distances:", distances)

print("Indices:", indices)

Advantages of IVF Index:

• Faster searches for large datasets.

• Trades off some accuracy for speed.

Step 6: Save and Reload the Index

To avoid rebuilding the index every time, save it to a file.

Code Example:

python

Copy code

# Save the index

faiss.write_index(index, "vector_index.faiss")

# Reload the index

loaded_index = faiss.read_index("vector_index.faiss")

# Verify the number of vectors in the reloaded index

print("Vectors in reloaded index:", loaded_index.ntotal)

Step 7: Interpret the Results

1. Use the indices returned by the search to retrieve original data points.

Example for retrieving metadata:
python
Copy code

import pandas as pd
# Load the metadata

metadata = pd.read_csv("embeddings.csv")

# Display the closest neighbors

print("Closest Neighbors:")

for i in indices[0]:

    print(metadata.iloc[i])

Expected Outcomes

1. • Configured FAISS:

     1. Installed and ready for use.

   • Vector Database:

     1. Embeddings are stored in a FAISS index, ready for efficient similarity searches.

   • Search Results:

     1. Identified the most similar vectors to the query vector with distances and indices.

Objective

This activity will guide you through writing a Python script to:

1. Load the saved embeddings from a file.

2. Build a FAISS index for storing the embeddings.

3. Perform similarity queries to find the closest embeddings for a given input.

Step-by-Step Instructions

Step 1: Load the Saved Embeddings

Ensure you have the embeddings.csv file saved from previous activities.

Code Example:

python

Copy code

import numpy as np

# Load the embeddings from the saved file

embeddings = np.loadtxt("embeddings.csv", delimiter=",")

# Display the shape of the embeddings to confirm successful loading

print("Loaded Embeddings Shape:", embeddings.shape)

Expected Output:

• The shape of the embeddings, e.g., (number_of_vectors, embedding_dimension).

Step 2: Build a FAISS Index

We will use the FlatL2 Index for exact nearest neighbor searches. For larger datasets, approximate indexes can be used (e.g., IVF).

Code Example:

python

Copy code

import faiss

# Determine the dimensionality of the embeddings

dimension = embeddings.shape[1]

# Create a FAISS FlatL2 index

index = faiss.IndexFlatL2(dimension)

# Add the embeddings to the index

index.add(embeddings)

# Verify the number of vectors in the index

print("Number of vectors in the FAISS index:", index.ntotal)

Explanation:

• IndexFlatL2 is a basic index type in FAISS that performs exhaustive search using Euclidean distance.

Step 3: Perform Similarity Queries

To perform a similarity search, you need a query vector. Here, we’ll use one of the embeddings as a sample query.

Code Example:

python

Copy code

# Select a query vector (e.g., the first embedding)

query_vector = embeddings[0].reshape(1, -1)

# Number of closest neighbors to retrieve

k = 5

# Perform a similarity search

distances, indices = index.search(query_vector, k)

# Display the results

print("Distances to nearest neighbors:", distances)

print("Indices of nearest neighbors:", indices)

Explanation:

• distances: The Euclidean distances between the query vector and its nearest neighbors.

• indices: The indices of the closest embeddings in the dataset.

Step 4: Retrieve Metadata Using Indices

If you have metadata associated with your embeddings (e.g., employee names or roles), you can use the indices to retrieve this information.

Code Example:

python

Copy code

import pandas as pd

# Load the metadata file

metadata = pd.read_csv("metadata.csv")

# Display the closest neighbors using indices

print("Closest Neighbors:")

for idx in indices[0]:

    print(metadata.iloc[idx])

Step 5: Save and Reload the Index

Save the FAISS index for reuse without rebuilding it.

Code Example:

python

Copy code

# Save the FAISS index to a file

faiss.write_index(index, "faiss_index.bin")

# Reload the index from the file

loaded_index = faiss.read_index("faiss_index.bin")

# Verify the number of vectors in the reloaded index

print("Number of vectors in the reloaded FAISS index:", loaded_index.ntotal)

Step 6: Full Script

Here’s the complete Python script combining all the steps:

python

Copy code

import numpy as np

import faiss

import pandas as pd

# Step 1: Load embeddings

embeddings = np.load("embeddings.csv")

print("Loaded Embeddings Shape:", embeddings.shape)

# Step 2: Build a FAISS index

dimension = embeddings.shape[1]

index = faiss.IndexFlatL2(dimension)

index.add(embeddings)

print("Number of vectors in the FAISS index:", index.ntotal)

# Step 3: Perform similarity queries

query_vector = embeddings[0].reshape(1, -1)  # Example query

k = 5  # Number of neighbors

distances, indices = index.search(query_vector, k)

print("Distances to nearest neighbors:", distances)

print("Indices of nearest neighbors:", indices)

# Step 4: Retrieve metadata

metadata = pd.read_csv("metadata.csv")

print("Closest Neighbors:")

for idx in indices[0]:

    print(metadata.iloc[idx])

# Step 5: Save and reload the FAISS index

faiss.write_index(index, "faiss_index.bin")

loaded_index = faiss.read_index("faiss_index.bin")

print("Number of vectors in the reloaded FAISS index:", loaded_index.ntotal)

Expected Outcomes

1. FAISS Index:

   • A fully built FAISS index containing your embeddings.

   • Ability to save and reload the index for future use.

2. Query Results:

   • List of nearest neighbors for a given query vector, along with their distances.

   • Metadata retrieved for the nearest neighbors.