Accelerating ML Lifecycle with Large Language Models

Table of Contents

1. Introduction
2. Machine Learning at Glassdoor
   1. Mission and Pillars
   2. Legacy NLP Architecture
   3. Case Study: Fishbowl Insights
3. Speeding up the Machine Learning Lifecycle
   1. Challenges and Pitfalls
   2. Model Checklists for Foreseeable Issues
   3. Robust Guardrails for Hard-to-Foresee Issues
   4. System Design for Efficiency and Maintainability
4. Conclusion
5. FAQs

Machine Learning at Glassdoor and Speeding up the ML Lifecycle

Machine learning (ML) is playing a significant role in the success of Glassdoor, a platform dedicated to helping people find their dream jobs and ideal companies. The ML team at Glassdoor focuses on four key pillars: salary transparency, fraud detection, content moderation, and personalization. By leveraging ML, Glassdoor aims to achieve radical transparency and provide accurate insights to its users.

Legacy NLP Architecture

Glassdoor extensively uses natural language processing (NLP) algorithms to analyze unstructured text data, such as user reviews. These reviews are processed using techniques like aspect extraction and sentiment analysis, allowing Glassdoor to categorize them into predefined categories like compensation, benefits, and diversity.

However, Glassdoor realizes that its legacy NLP architecture presents several challenges. The architecture relies on sequential processing, with one model's output often serving as another model's input. This increases the risk of cascade errors and poses limitations in terms of scalability and language support.

Case Study: Fishbowl Insights

Glassdoor recently acquired Fishbowl, a social networking app for professionals. To integrate Fishbowl's content and Scale up ML-powered services, Glassdoor embarked on a case study using large language models for its new product, Fishbowl Insights. This product aims to provide employers with digestible reports and actionable insights by analyzing the discussions happening in company-specific bowls.

Fishbowl Insights utilizes a new ML architecture that leverages large language models. This architecture allows for better deployment and utilization of models, eliminating the limitations of the legacy sequential approach. Glassdoor also emphasizes responsible design by incorporating a human-in-the-loop module, where human experts review and override model predictions to ensure accuracy and reduce bias.

Speeding up the ML Lifecycle

Glassdoor acknowledges the need to accelerate the ML lifecycle to improve efficiency and adopt state-of-the-art algorithms. The traditional ML lifecycle at Glassdoor involved months of data gathering, offline analysis, experimentation, and adoption. However, this process often encountered issues, such as resource constraints, production bugs, and data quality problems. These challenges resulted in delays and hindered the Timely adoption of ML models.

To address these challenges, Glassdoor implemented a model checklist approach for foreseeable issues and established robust guardrails for hard-to-foresee issues. The model checklist ensures that vital procedures are followed, covering aspects like data labeling, latency SLAs, and human-in-the-loop processes. It helps reduce common failure Patterns and facilitates collaboration with other teams.

Additionally, Glassdoor emphasizes proper system design to minimize ambiguity, account for external dependencies, and reduce the risk of delays. Allocating ample time for system design, maintaining a balance between complexity and coverage in checklists, and involving ML engineers in the design process have proven beneficial in driving efficiency.

Conclusion

Glassdoor's ML team has made remarkable strides in utilizing ML techniques to enhance its platform. The adoption of large language models, responsible design practices, and a focus on speeding up the ML lifecycle has resulted in improved accuracy, scalability, and efficiency. By leveraging the power of ML, Glassdoor continues to fulfill its mission of helping people find fulfilling careers and companies.

FAQs

Q: How long has Glassdoor been using machine learning? A: The exact timeline for Glassdoor's adoption of machine learning may vary, but significant progress has been made in the past few years. The changes described in this article occurred within the three years prior to its writing.

Q: Can You provide more details about the ML model checklist? A: While the model checklist used by Glassdoor is not publicly available at the moment, there are plans to potentially share it on the Glassdoor Engineering Blog in the future. Stay tuned for updates on this valuable tool.

Q: How does Glassdoor ensure data quality in ML projects? A: Glassdoor emphasizes the involvement of knowledge engineers, product managers, and annotation teams to ensure accurate and unbiased training data. They work together on data labeling guidelines, establish baselines, and periodically perform human evaluation to enhance data quality.

Q: What are the benefits of the new ML architecture used by Glassdoor? A: The new ML architecture leverages large language models and replaces the legacy NLP architecture. It enables better deployment, scalability, and language support. The modular design reduces cascade errors and allows for the adoption of state-of-the-art algorithms.

Q: Is the ML model checklist applicable to all ML projects? A: The ML model checklist can be a valuable tool for most ML projects, but it is essential to tailor it to the specific needs and requirements of each project. Consider the maturity of your ML team and find the right balance between complexity and coverage in the checklist.