Azure AI Search - 自定义GPT-4视觉技能

Find AI Tools

No difficulty

No complicated process

Find ai tools

Home AI News TW Azure AI Search - 自定义GPT-4视觉技能

Azure AI Search - 自定义GPT-4视觉技能

Table of Contents:

Introduction
Using Azure AI Search to Get Image Metadata
Creating a Data Source in Azure AI Search
Building a Skill Set in Azure AI Search
Creating an Indexer in Azure AI Search
Custom Web API Skill
Azure Open AI Embedding Skill
Running the Indexer
Searching for Images with Text Vector
Deploying the Custom Skills

Introduction

In this article, we will explore how to use Azure AI search to extract metadata from images and Create an image index. We will learn how to upload images to Azure Blob storage, create a data source in Azure AI search, define a skill set, build an indexer, and run searches using text vectors. Additionally, we will discuss the creation and deployment of custom skills to enhance the image metadata extraction process.

..... Using Azure AI Search to Get Image Metadata

To begin, we need a container in Azure Blob storage where we can upload the images. Once the container is set up, we will create a data source in Azure AI search to specify the location of the image data. With the data source created, we can then focus on creating a skill set, which is a collection of skills used to extract information from the images. The skill set will include a custom web API skill that calls a GPT-4 Vision model to generate descriptions of the images and an Azure Open AI embedding skill to create text vectors from the descriptions. .....

Creating a Data Source in Azure AI Search

In this section, we will create a data source in Azure AI search. The data source acts as a link between Azure AI search and the container in Azure Blob storage where the images are stored. The data source provides information on how to access the images, such as the connection STRING and container name.

To create the data source, we need to make a REST API call and provide a JSON definition for the data source. The JSON definition should include details such as the name of the data source, its description, the Type of source (Azure Blob in this case), and the connection string. Once the data source is created, it will appear in the Azure AI search resource and can be used in the next steps.

.....

Building a Skill Set in Azure AI Search

After creating the data source, we can move on to building a skill set in Azure AI search. A skill set is a collection of skills that will be applied to the images in the data source to extract desired information. In our case, the skill set will include a custom web API skill and an Azure Open AI embedding skill.

The custom web API skill is a skill that we build ourselves. It calls a GPT-4 Vision model to generate descriptions of the images in the data source. We provide the URL to the GPT-4 Vision model as an endpoint to the custom web API skill. The skill will take an input JSON structure containing the URL of an image and return an output JSON structure containing the description of the image.

The Azure Open AI embedding skill is a built-in skill provided by Azure AI search. It takes the description generated by the custom web API skill and creates a text vector from it. The skill set combines both skills, ensuring that the description field generated by the custom web API skill is used as an input for the Azure Open AI embedding skill. Once the skill set is defined, it can be used in the next step to create an indexer.

.....

Creating an Indexer in Azure AI Search

Now that we have the data source and skill set in place, we can create an indexer in Azure AI search. The indexer is responsible for fetching the images from the data source, applying the defined skill set to extract metadata, and populating an index with the extracted information.

In the indexer configuration, we specify the data source, the skill set, and the target index where the metadata will be stored. We also define field mappings to map the metadata fields to the fields in the index. For example, we can map the name of the image file to the "name" field in the index. Additionally, we specify output field mappings to map the fields generated by the skill set (such as description and image vector) to the fields in the index.

Once the indexer is created, it can be scheduled to run at specific intervals to automatically update the index with new images from the data source. The indexer will run the defined skill set on the new images and populate the index with the extracted metadata.

.....

Custom Web API Skill

In this section, we will Delve into the custom web API skill that we included in the skill set. The purpose of this skill is to fetch image descriptions using the GPT-4 Vision model. The skill takes the URL of an image as input and returns the description of the image as output.

To implement this skill, we need to develop a custom skill as a container app. The container app will expose an endpoint that can be called by the custom web API skill. The web API skill will send the URL of the image to the custom skill, which will use the GPT-4 Vision model to generate the description. The skill will then return the description to the web API skill, which will pass it on to the indexer.

.....

Azure Open AI Embedding Skill

The Azure Open AI embedding skill is a built-in skill provided by Azure AI search. It takes a text input and creates a text vector representation from it. In our case, we will use this skill to create a text vector from the description of each image.

The skill is already available in Azure AI search, so we don't need to develop any custom code for it. We just need to include it in our skill set and specify the input and output fields, ensuring that the input field matches the output field of the previous skill in the skill set.

.....

Running the Indexer

After creating the indexer, we can run it to populate the index with the extracted image metadata. The indexer will fetch images from the data source, Apply the defined skill set, and populate the fields in the index.

The indexer can be scheduled to run at specific intervals, such as every night or every few hours, to automatically update the index with new images from the data source. The scheduler will ensure that only new images are indexed, avoiding duplication of metadata.

.....

Searching for Images with Text Vector

Once the index is populated with image metadata, we can start searching for images using text vectors. The text vectors are created from the descriptions of the images and can be used to find similar images Based on their textual Context.

We can use the Python SDK for Azure AI search to perform a search using a text vector query. The query will take a text input and vectorize it using a pre-trained model. The search will return the most Relevant images based on the similarity of their text vectors to the query vector.

.....

Deploying the Custom Skills

In the final section, we will discuss the deployment of the custom skills used in the image metadata extraction process. The custom skills need to be built as container apps and deployed to a container registry. Once deployed, they can be used in the skill set by providing the endpoints of the custom skills.

We provide detailed instructions on how to deploy the custom skills using Azure Container Apps. The process involves building a Docker image, pushing it to a container registry, and deploying it as a container app. The environment variables required for the custom skills are set using the N file.

.....

FAQ:

Q: Can I use Azure AI search to extract metadata from images stored in Azure Blob storage? A: Yes, you can use Azure AI search to extract metadata from images stored in Azure Blob storage. By creating a data source, defining a skill set, and running an indexer, you can extract information such as image descriptions and image vectors from the images.

Q: How often does the indexer update the index with new images? A: The indexer can be scheduled to run at specific intervals, such as every night or every few hours, to update the index with new images. This ensures that the index is always up-to-date with the latest image metadata.

Q: Can I search for images using text queries? A: Yes, you can search for images using text queries. By using text vectors generated from image descriptions, you can perform text-based searches to find images that are similar in nature or context.

Q: Can I deploy my own custom skills for image metadata extraction? A: Yes, you can deploy your own custom skills for image metadata extraction. By building container apps and deploying them to Azure Container Apps, you can enhance the capabilities of Azure AI search to extract specific metadata from images.

.....

Highlights: