Experience Dynamic Voice Conversations with OpenAI and ElevenLabs in Unity!

Table of Contents

1. Introduction
2. Demo of the New Feature
3. Setting up the Scene
4. Recording Voice and Transcription
5. Sending the Text to Open AI API
6. Generating Voice with 11 Labs API
7. Adding Personality to the AI
8. Improving the Prompt for a More Interesting Response
9. Code Breakdown: Voice Recorder Script
10. Code Breakdown: Open AI Demo Script

Article

1. Introduction

In this article, we will explore a new feature that has been added to the GPT AI integration asset in version 1.3. This feature allows You to communicate with Open AI or chat GPT using your voice and receive responses in AI-generated voices. This capability can be a game-changer in creating dynamic and unscripted interactions with NPCs in your game.

2. Demo of the New Feature

To better understand how this feature works, let's walk through a demo scene. The scene is broken down into five steps, showcasing the underlying processes involved in this feature. However, in a real game, these steps would be chained together into one function for seamless integration.

3. Setting up the Scene

The demo scene consists of a canvas with a text box for AI responses and a white box for transcriptions. Additionally, there are buttons for recording voice, transcribing, sending text to Open AI API, and generating voice with the 11 Labs API. In the scene, you will find an Open AI Completer and an AR Controller, which handle the settings and functionality of the feature.

4. Recording Voice and Transcription

To begin the demo, you need to Record your voice. Clicking on the "Record Voice" button triggers the recording process, which saves the audio file. Once the recording is complete, you can transcribe the audio by clicking on the "Transcribe" button. The audio clip is sent to the Open AI Whisper API, which transcribes it into text.

5. Sending the Text to Open AI API

With the transcribed text, you can now send it to the Open AI API for an AI response. Clicking on the "Send to GPT" button triggers the sending process. The Open AI API uses the text prompt to generate a response from the AI model. The response is received and displayed in the text box.

6. Generating Voice with 11 Labs API

Once you have the AI response, the next step is to generate a voice for the response. Clicking on the "Generate Voice" button sends the response to the 11 Labs API. The 11 Labs API uses AI to generate voices. The demo scene uses a basic voice, although you have the flexibility to Create and customize your own voices.

7. Adding Personality to the AI

To make the interactions more interesting, you can give the AI a specific personality. In the demo, we use an included prompt generator tool to give the AI the personality of a medieval era woman speaking in Old English dialect. The tool generates a high-quality prompt, which is combined with the user's message to create a unique conversation.

8. Improving the Prompt for a More Interesting Response

You can further enhance the prompt to Elicit more engaging and accurate responses from the AI. By tweaking the instruction prompt, you can Shape the AI's behavior and personality. This adds depth to interactions with NPCs in the game, making the conversations more immersive and realistic.

9. Code Breakdown: Voice Recorder Script

Let's take a closer look at the code that powers the voice recording functionality. When the "Start Recording" button is clicked, the script checks for an available microphone and starts recording at the specified recording length. Clicking on the "Stop Recording" button ends the recording and saves the audio file.

10. Code Breakdown: Open AI Demo Script

The Open AI demo script manages the integration with the Open AI API. It includes settings for the AI model, such as the temperature, and handles the sending and receiving of text Prompts. The script differentiates between two types of API requests, depending on the model used: GPT 3.5 and GPT 4. It also shows how to format the messages for chat-Based interactions.

By following the steps outlined in this article, you can leverage the GPT AI integration asset's new voice interaction feature to create dynamic and immersive conversations in your game. Feel free to explore and experiment with different personalities and prompts to make your NPC interactions more engaging than ever.

Pros:

• Dynamic and unscripted conversations with NPCs
• Immersive and realistic gameplay experience
• Flexibility to create and customize voices

Cons:

• Complex setup and integration process
• Limited language support for audio translation
• Processing limitations with special characters

Highlights

• Experience dynamic conversations with NPCs using your voice
• Unscripted interactions for realistic gameplay
• Customize AI voices and personalities for immersive experiences
• Enhance AI responses with tailored prompts and instructions

FAQs

Q: Can I use different AI voices for different characters in my game? A: Absolutely! The 11 Labs API allows you to generate and customize voices to match specific characters in your game. You can create unique personalities for each character.

Q: Is audio translation supported for languages other than English? A: Currently, the audio translation feature only supports English. However, future updates might include support for additional languages.

Q: Can I save and recall past conversations with NPCs? A: Yes, by following the chat message structure and maintaining conversation context, you can enable the AI to remember and recall past conversations, creating a more continuous and personalized experience.

Q: Are there any limitations in terms of processing special characters? A: Yes, the 11 Labs API may not handle special characters like quotes or line breaks effectively. It is advisable to filter out these characters to ensure a smooth response from the API.

Q: How can I improve the AI's responses? A: By refining the prompt and providing clear instructions, you can guide the AI to provide more accurate and engaging responses. Experiment with different prompts to achieve the desired outcomes.