Creating a Script to Translate Japanese Voice Chat

Table of Contents

1. Introduction
2. Creating a Script to Translate Japanese Voice Chat 2.1. Recording Game Audio 2.2. Limitations of Recording Game Audio 2.3. Real-Time Transcription Code 2.4. Editing the Code for Game Audio Translation 2.5. Displaying Text on Screen
3. Enhancing the YouTube Watching Experience 3.1. Introducing Hollow Songs Chrome Extension 3.2. Features of the Chrome Extension 3.3. Customization Options
4. Collaboration and Acknowledgments 4.1. Assistance from Foreign for Documentation 4.2. Support from Minchoko for Coding
5. Whisper ai Transcription Models 5.1. Testing Different Whisper AI Models 5.2. Small Model vs. Big Model 5.3. Challenges with Hallucinations
6. Concluding Thoughts and Appreciation 6.1. A Journey of Building an AI System 6.2. Overcoming Code Issues and Improvements 6.3. Running Docker Containers and Apex 6.4. Limitations with Available Computing Resources 6.5. Reflecting on the Project and Future Options

Creating a Script to Translate Japanese Voice Chat

In the last video, we explored how to communicate with other Japanese players in APEX as an anime waifu. However, this was only one-way communication. In this article, we will delve into creating a script that can translate Japanese voice chat in games for those who don't understand the language. We received an interesting comment in the last video suggesting Recording the game audio whenever the speaker icon lights up. While it sounds like a plausible idea, there are some issues with it.

Recording Game Audio

The idea of recording game audio when the speaker icon lights up seems like a convenient solution. However, there are a few limitations to consider. Some users have microphones with low audio thresholds, causing the speaker icon to light up even for small noises like typing on the keyboard. This unreliable behavior makes it challenging to record game audio accurately.

Moreover, focusing solely on Apex and not being compatible with other applications like Discord is another drawback. We want to create a script that can be utilized across various applications. Lastly, considering my expertise, which doesn't extend to coding complex object or edge detection techniques, it's best to explore alternative solutions for compatibility reasons.

Real-Time Transcription Code

To overcome the limitations of recording audio, we will be utilizing real-time transcription code written in Java. This code is capable of constantly listening to audio in the background and displaying a transcript that updates as you speak into your microphone. By modifying this code, we can redirect it to listen to our game audio and print the latest translation.

However, simply printing the translation to the console isn't practical, especially when playing full-screen games like Apex. Constantly alt-tabbing to view the translation would be inconvenient. To address this, we can implement code from Stack Overflow that displays the text on the screen in a manner similar to subtitles used by Neurosama. The only challenge at this point is adjusting the position of the displayed text to ensure it doesn't hinder the gaming experience.

Stay tuned as we explore TKinter's documentation to find a solution to this positioning issue.

• Pros:
  • Real-time transcription allows for immediate translation of Japanese voice chat in games.
  • Displaying text on the screen instead of relying on console printing enhances user experience.
  • The script can be compatible with various applications, not limited to Apex.
• Cons:
  • Recording game audio may not be reliable due to mic sensitivity and unwanted noises triggering the speaker icon.
  • The script requires modifications and adjustments to ensure optimal positioning of the displayed text.

Highlights

• Creating a script to translate Japanese voice chat in games.
• Exploring the limitations of recording game audio for translation purposes.
• Utilizing real-time transcription code to overcome audio recording challenges.
• Displaying translated text on the screen for a seamless gaming experience.
• Acknowledging foreign assistance and collaboration with Minchoko.
• Evaluating Whisper AI transcription models and their effectiveness.
• Reflecting on the journey of building an AI system and overcoming code issues.
• Discussing limitations with available computing resources and future options.

FAQ

Q: How accurate are the whisper AI transcription models for Japanese voice chat? A: After testing different whisper AI models, we found that the small model provides the best accuracy for transcribing Japanese. Larger models may cause delays in translation due to increased computing requirements.

Q: Why do whisper AI models hallucinate and transcribe random noise? A: Whisper AI is trained on YouTube videos and their transcripts. It may recognize common phrases like "thank you for watching" even when they're not spoken. This can result in the model transcribing unrelated noise as if it were spoken language.

Q: Can the script run alongside Apex without overloading computing resources? A: Running Apex and the whisper AI transcription simultaneously can stress the GPU, leading to request timeouts. To avoid this issue, it may be necessary to sacrifice translation accuracy or consider upgrading the graphics card.

Q: What improvements were made to the code from the previous video? A: The code was optimized to reduce input latency and prevent crashes. Basic error handling was added to address issues like empty audio files being sent to whisper AI.

Q: How can the Hollow Songs Chrome extension enhance the YouTube watching experience? A: The Hollow Songs Chrome extension adds useful features to the YouTube player, such as the ability to view song setlists, loop songs, and skip talking sections. It aims to improve the user experience for watching YouTuber streams.

Resources:

• Hollow Songs Chrome extension: Website