Unlock the Power of Speech-to-Text with OpenAI Whisper in Unity

Table of Contents

1. Introduction
2. Installing the OpenAI Whisper API Package
3. Exploring the Sample Scene
4. How the Whisper API Works
5. Code Explanation
6. Using Different Microphones
7. Recording and Transcribing Audio
8. Translating Audio
9. Language Support and Limitations
10. Conclusion

Introduction

In this article, we will explore the Whisper API in OpenAI and learn how it works. We will start by installing the OpenAI Unity package and exploring the provided sample scene. Then, we will dive into the code and understand the underlying process of the Whisper API. We will also discuss how to use different microphones, Record and transcribe audio, and even Translate audio into different languages. Lastly, we will look at the language support and limitations of the Whisper API and conclude with some key takeaways.

👉 Installing the OpenAI Whisper API Package

To get started with the Whisper API in OpenAI, we first need to ensure that the OpenAI Unity package is installed. You can easily update the package by clicking on the update button within your Unity project. If you haven't installed the package yet, you can download it from the GitHub URL provided in the description. Once you have the package installed and updated, we can move on to exploring the sample scene.

👉 Exploring the Sample Scene

Within the OpenAI Unity package, you will find a folder named "Samples". Inside this folder, locate the "Whisper" folder. In the Whisper folder, you will find a scene file that we will open to access the sample scene. This scene is designed to showcase the functionality of the Whisper API.

Upon opening the sample scene, you will Notice a basic user interface (UI) consisting of a microphone selection, a fill bar, a text screen, and a button. The microphone selection allows you to choose the microphone you want to use for recording audio. This is particularly useful when you have multiple microphones connected to your device. The fill bar visually represents the recording progress, while the text screen displays the transcribed text. The button initiates the recording process.

Now that we have familiarized ourselves with the sample scene, let's dive into the workings of the Whisper API.

👉 How the Whisper API Works

The Whisper API enables us to transcribe spoken audio into text using the power of OpenAI. Once the audio is transcribed, we can further process the text for various purposes, such as text completion or chat functionalities. Let's take a closer look at the code and understand how the Whisper API functions.

Within the code, the start method performs several important tasks. Firstly, it checks for available microphone devices and populates them into a drop-down selection. This allows us to choose the desired microphone for recording. Additionally, it sets up the UI elements, such as the button, to initiate the recording process.

Once the recording begins, the selected microphone captures the audio, which is then stored as an audio clip. The audio clip is converted into a byte array using a third-party script called "SaveWav". This conversion is necessary as the Whisper API expects the audio data in the form of a byte array. By using this script, we can handle the audio data in memory without the need for file-saving operations.

After the recording ends, we make a request to the OpenAI API using the byte array data. The Whisper API's endpoint receives this audio data and processes it to transcribe the spoken words into English text. However, it's worth mentioning that the Whisper API also supports audio translation. In the next section, we will explore how we can leverage this feature.

👉 Recording and Transcribing Audio

The sample scene provides a straightforward way to record and transcribe audio. Once you have selected the desired microphone, simply click the "Record" button, speak into the microphone, and wait for the Transcription to appear on the text screen. The Whisper API will process the audio in real-time and convert it into English text.

It's important to note that the Whisper API supports a wide range of languages, not just English. In the next section, we will discuss how to leverage the audio translation feature of the Whisper API.

👉 Translating Audio

Apart from transcribing audio into English, the Whisper API also allows us to translate audio into different languages. To do this, we simply need to modify the code and specify the desired target language.

By creating an audio translation request, we can translate the spoken audio into the target language. The request structure is similar to the transcription request, but without specifying the language. The Whisper API will automatically detect the source language and provide the translation accordingly.

For example, if we wanted to translate Turkish audio into English, we would modify the code to create an audio translation request without specifying the language. The Whisper API will automatically detect Turkish as the source language and provide the English translation.

It's worth noting that while the Whisper API performs well with languages like English and Turkish, it may not be as accurate with certain languages. In the next section, we will discuss the language support and limitations of the Whisper API.

👉 Language Support and Limitations

The Whisper API offers support for multiple languages, allowing you to transcribe and translate audio in various linguistic contexts. You can refer to the OpenAI documentation to see the full list of supported languages.

While the Whisper API demonstrates impressive performance with popular languages such as English and Turkish, it may not deliver the same level of accuracy with less commonly used languages. For instance, during our exploration, we found that the Whisper API struggled to accurately transcribe Estonian audio.

When working with the Whisper API, it's advisable to stick to widely supported languages to ensure the best possible results. Experimentation and further exploration with different languages can provide valuable insight into the performance and limitations of the Whisper API.

Conclusion

In this article, we delved into the workings of the Whisper API in OpenAI and explored its functionality within Unity. We started by installing the OpenAI Whisper API package and navigating the provided sample scene. Then, we examined the code to understand how the Whisper API handles audio recording, transcription, and translation.

We learned that the Whisper API allows us to transcribe audio into English and even translate it into various languages. While the Whisper API performs well with languages like English and Turkish, it may not be as accurate with less commonly used languages.

By leveraging the power of the Whisper API, developers can integrate Speech-to-Text and audio translation capabilities into their Unity projects, opening up a whole new world of possibilities.

Highlights

• Learn how to use the Whisper API in OpenAI
• Install the OpenAI Unity package and explore the sample scene
• Understand the code behind the Whisper API's functionality
• Record and transcribe audio in real-time
• Translate audio into different languages
• Discover the language support and limitations of the Whisper API
• Unlock the potential of speech-to-text and audio translation in Unity projects

FAQ

Q: Can the Whisper API handle multiple microphones? A: Yes, the Whisper API allows you to choose from multiple microphones connected to your device.

Q: Which languages are supported by the Whisper API? A: The Whisper API offers support for a wide range of languages, including but not limited to English and Turkish. Refer to the OpenAI documentation for the complete list of supported languages.

Q: Does the Whisper API provide accurate translations for all languages? A: While the Whisper API performs well with popular languages, it may not deliver the same level of accuracy with less commonly used languages. It's advisable to stick to widely supported languages for optimal results.

Q: Can I use the Whisper API for text completion or chat functionalities? A: Yes, you can utilize the transcribed text from the Whisper API for various purposes, including text completion and chat functionalities in your Unity projects.

Q: Is the Whisper API suitable for real-time audio processing? A: Yes, the Whisper API is designed to handle real-time audio processing, allowing for seamless integration into applications that require live transcription or translation capabilities.