Master Full Stack AI Semantic Search

Master Full Stack AI Semantic Search

Table of Contents:

1. Introduction
2. What is Semantic Search?
3. Applications of Semantic Search
4. Setting Up the Environment
5. Creating the Pinecone Index
6. Uploading Data to the Pinecone Index
7. Querying the Pinecone Index
8. Building the User Interface (UI)
9. Testing the Application
10. Conclusion

Introduction

In this article, we will learn how to build a full stack AI-enabled semantic search application using Next.js, Pinecone, LangChain, and ChatGPT. Semantic search is an advanced search technique that aims to understand the intent and meaning behind a user's search query, rather than relying solely on keyword matching. By incorporating natural language processing and machine learning techniques, semantic search enhances the search experience by providing more Meaningful results that Align with the user's intent.

What is Semantic Search?

Semantic search refers to a search technique that aims to understand the intent and meaning behind a user's search query, rather than relying solely on keyword matching. It goes beyond traditional language-Based search methods by incorporating natural language processing and machine learning techniques to comprehend the Context, relationships, and concepts associated with a query. Semantic search enhances the search experience by providing more meaningful results that align with the user's intent, making it a valuable tool for information retrieval and knowledge discovery.

Applications of Semantic Search

Semantic search has a wide range of applications across various industries. Some of the notable applications include:

1. Developer Documentation: Semantic search can be incorporated into developer documentation sites, allowing developers to easily find answers to their queries by simply asking the question in natural language, regardless of the specific syntax or keywords used in the documentation.

2. E-commerce: Semantic search can improve the user experience on e-commerce platforms by providing more accurate and Relevant search results based on the user's intent. It can understand synonyms, related terms, and context to deliver more precise product recommendations.

3. Customer Support: Semantic search can be leveraged in customer support systems to enable users to find answers to their queries faster and more effectively. By understanding the user's intent, semantic search can provide relevant support articles or direct the user to the most appropriate resources.

4. Content Discovery: Semantic search can optimize content discovery platforms by providing users with more personalized and relevant content recommendations based on their interests, preferences, and previous browsing behavior.

5. Data Exploration: Semantic search can be used in data exploration applications, allowing users to query large datasets using natural language. This enables more intuitive and user-friendly interactions with the data, without the need for complex SQL queries or predefined schemas.

Setting Up the Environment

Before we start building our application, we need to set up our development environment. Here are the steps to follow:

1. Obtain Pinecone API Key: Sign up for Pinecone and obtain an API key. Pinecone is a vector database that we will be using to store and query our data.

2. Obtain OpenAI API Key: Sign up for OpenAI and obtain an API key. OpenAI will provide us with the capabilities to embed our text into vectors and perform language processing tasks.

3. Install Node.js: Make sure You have Node.js installed on your machine. Node.js will be used to run our application.

4. Install Dependencies: Once you have obtained the API keys and installed Node.js, navigate to your project directory in the terminal and run the command npm install to install the required dependencies.

Creating the Pinecone Index

The first step in building our semantic search application is to Create the Pinecone index. The index is where we will store our data for efficient querying. Here is the function to create the Pinecone index:

async function createPineconeIndex(client, indexName, Dimensions) {
  console.log("Creating index:", indexName);
  const existingIndexes = await client.getIndexes();

  if (existingIndexes.includes(indexName)) {
    console.log("Index already exists!");
    return;
  }

  await client.createIndex(indexName);
  console.log(`Index '${indexName}' created successfully!`);

  // Wait for index initialization (approx. 80 seconds)
  await new Promise((resolve) => setTimeout(resolve, 80000));
}

Pros:

• The function creates the Pinecone index, making it ready to store our data.
• It checks if the index already exists before creating it to avoid duplication.
• The function waits for the index to be initialized before moving forward.

Uploading Data to the Pinecone Index

Next, we need to upload our data to the Pinecone index. This involves splitting the data into chunks, embedding each chunk, and uploading the embeddings to the Pinecone index. Here is the function to upload data:

async function uploadDataToPinecone(client, indexName, documents) {
  console.log("Uploading data to Pinecone...");

  for (const document of documents) {
    console.log("Processing document:", document.path);

    const { text } = document;
    const chunkSize = 1000; // Split text into chunks of 1000 characters

    console.log("Splitting text...");
    const textSplitter = new TextSplitter(chunkSize);
    const chunks = await textSplitter.createDocuments(text);
    console.log("Splitting text completed!");

    console.log("Creating embeddings...");
    const embeddings = await openAIEmbeddings.embedDocuments(chunks.map((chunk) => chunk.pageContent.replace(/\n/g, " ")));
    console.log("Embeddings created!");

    console.log("Uploading vectors to Pinecone...");
    const vectors = [];

    for (let i = 0; i < chunks.length; i++) {
      const chunk = chunks[i];
      const embedding = embeddings[i];

      const vector = {
        id: `${document.path}_${i + 1}`,
        values: embedding,
        metadata: {} // Additional metadata if needed
      };

      vectors.push(vector);

      if (vectors.length === 100 || i === chunks.length - 1) {
        await client.upsertVectors(indexName, vectors);
        vectors.length = 0;
      }
    }

    console.log("Data upload completed for document:", document.path);
  }

  console.log("Data upload to Pinecone completed!");
}

Pros:

• The function splits the text data into chunks for efficient processing and uploading to Pinecone.
• It creates embeddings using the OpenAI API to convert text into vectors.
• The embeddings are then uploaded to the Pinecone index in batches for optimal performance.

Querying the Pinecone Index

Once we have created the Pinecone index and uploaded the data, we can perform queries to retrieve relevant results based on the user's search query. Here is the function to query the Pinecone index:

async function queryPinecone(client, indexName, query) {
  console.log("Querying Pinecone...");

  const queryEmbedding = await openAIEmbeddings.embedDocuments([query]);

  const { results } = await client.query(indexName, {
    topK: 10,
    query: queryEmbedding
  });

  console.log(`Found ${results.matches.length} matches from Pinecone!`);

  if (results.matches.length) {
    const response = await llm.call(new document(responses.result[0].values, query));
    console.log(response.text);
    return response.text;
  } else {
    console.log("No matches found.");
    return null;
  }
}

Pros:

• The function queries the Pinecone index using the user's search query.
• It retrieves the top 10 matches and logs the number of matches found.
• If matches are found, it uses the LLM chain from OpenAI to generate a response based on the retrieved documents.
• The function returns the response for further use in the application.

Building the User Interface (UI)

To Interact with our semantic search application, we need to create a user interface (UI) for users to enter their search queries and view the results. Here is an example UI using React:

import React, { useState } from "react";

const App = () => {
  const [query, setQuery] = useState("");
  const [result, setResult] = useState("");
  const [loading, setLoading] = useState(false);

  const sendQuery = async () => {
    if (!query) return;

    setLoading(true);
    setResult("");

    const response = await fetch(`/api/read`, {
      method: "POST",
      body: JSON.stringify({ query })
    });

    const data = await response.json();
    setResult(data.result);
    setLoading(false);
  };

  const createIndexAndEmbeddings = async () => {
    await fetch(`/api/setup`, {
      method: "POST"
    });

    console.log("Index creation and data upload completed!");
  };

  return (
    <div className="container mx-auto p-4">
      <h1 className="text-3xl">Semantic Search Application</h1>
      <div className="my-4">
        <input
          type="text"
          value={query}
          onChange={(e) => setQuery(e.target.value)}
          className="p-2 border border-gray-300 rounded mr-2"
        />
        <button
          onClick={sendQuery}
          className="py-2 px-4 bg-blue-500 text-white rounded"
        >
          Ask AI
        </button>
      </div>
      {loading && <p>Asking AI...</p>}
      {result && <p>{result}</p>}
      <button
        onClick={createIndexAndEmbeddings}
        className="py-2 px-4 bg-green-500 text-white rounded mt-4"
      >
        Create Index and Embeddings
      </button>
    </div>
  );
};

export default App;

Pros:

• The UI allows users to enter their search queries and click a button to send the query to the server.
• It displays a loading message while waiting for the response.
• The response is displayed on the screen once it is received from the server.
• The UI includes a button to create the Pinecone index and upload the data.

Testing the Application

To test our semantic search application, follow these steps:

1. Ensure that you have the Pinecone and OpenAI API keys.
2. Install the required dependencies by running npm install in the project directory.
3. Start the development server by running npm run dev.
4. Open your browser and navigate to http://localhost:3000.
5. Enter a search query in the input field and click "Ask AI" to perform the search.
6. The results will be displayed on the screen. You can try different queries to see different results.

Conclusion

In this article, we learned how to build a full stack AI-enabled semantic search application using Next.js, Pinecone, LangChain, and ChatGPT. Semantic search is an advanced search technique that aims to understand the intent and meaning behind a user's search query, providing more meaningful and relevant search results. By leveraging natural language processing and machine learning techniques, we can enhance the search experience and enable more intuitive interactions with data. This application has various applications, including improving developer documentation, enhancing e-commerce search functionality, and facilitating content discovery. With the knowledge gained from this article, you can now implement semantic search in your own projects and explore the possibilities of AI-powered search.