Unforeseen Influences on AI Personality Prediction in Hiring
Table of Contents
- Introduction
- Background
- Automated hiring systems
- Algorithmic personality tests
- Validity and reliability concerns
- DISC and the Big Five models of personality
- DISC model
- Big Five model
- Examining Humantic AI and Crystal
- Presence in the hiring market
- Machine learning-based personality profiling
- Scoring and ranking of candidates
- Methodology for auditing algorithmic personality predictors
- Collecting preliminary information
- Identifying key facets of measurement
- Creating an input corpus
- Estimating stability
- Results of the external audit
- Instability on key facets of measurement
- File format and its impact on personality scores
- Source context and its effect on stability
- Participant time and its influence on results
- Implications and limitations
- Conclusion
AI Personality Prediction in Hiring and its Unexpected Influences
🔍 Introduction
In the era of automated hiring systems, algorithmic personality tests have gained significant popularity as tools to predict job seekers' future success. However, the reliability and validity of these tests are subjects of immense debate. In this article, we will explore the surprising influences and consequences of AI personality prediction in the hiring process. By conducting an external audit of two commercial systems – Humantic AI and Crystal – we aim to shed light on the stability of predictions made by algorithmic personality tests.
🔍 Background
Automated hiring systems have become pervasive in the modern recruitment landscape. Among these systems, algorithmic personality tests play a crucial role in evaluating job applicants. These tests claim to estimate an individual's personality based on their Resume or social media profile. However, the fundamental question remains: do these tests actually work?
Validating the assumptions made by the vendors of algorithmic personality tests is at the core of our methodology. We aim to develop an audit informed by psychometric testing literature to assess the stability of predictions made by these tests. Before diving into the details of our audit, let's briefly understand the two major models of personality used today – DISC and the Big Five.
🔍 DISC and the Big Five models of personality
The DISC model assesses four personality traits: dominance, influence, steadiness, and conscientiousness. On the other HAND, the Big Five model consists of five traits: openness, conscientiousness, extraversion, agreeableness, and neuroticism. While the practice of using personality tests in hiring is not new, skepticism surrounding their validity and reliability has persisted. Scholars have questioned the meaningfulness and measurability of personality constructs, especially in the case of DISC.
🔍 Examining Humantic AI and Crystal
Humantic AI and Crystal are two major players in the algorithmic personality testing market. According to their websites, Humantic AI boasts clients like Apple and PayPal, while Crystal claims that ninety percent of Fortune 500 companies utilize their products. Both systems leverage machine learning techniques to extract personality profiles from job candidates' resumes and LinkedIn profiles. In addition to DISC scores, Humantic AI also provides scores for the Big Five model.
Employers rely on these systems to build personality profiles of job applicants and rank them based on their match scores. Crystal assigns a job fit score by comparing candidates' profiles to a benchmark candidate or a specified ideal personality profile. Similarly, Humantic AI assigns a match score by comparing candidates to an ideal candidate or a LinkedIn URL.
🔍 Methodology for auditing algorithmic personality predictors
To assess the stability of algorithmic personality predictors, we developed a comprehensive auditing methodology. It involves collecting preliminary information about the system's context, identifying key facets of measurement, creating an input corpus representative of the tool's intended use, and estimating stability across each facet. We leverage statistical tests and stability metrics from IO psychology literature to determine the reliability of the systems.
🔍 Results of the external audit
The audit revealed substantial instability on key facets of measurement for both Humantic AI and Crystal. Notably, personality profiles generated by both systems varied significantly based on whether they were computed from resumes or LinkedIn profiles. This violates the assumption that algorithmic personality tests should be stable regardless of the input source. Additionally, Crystal displayed different personalities for the same resume in different formats, further undermining its stability.
🔍 Implications and limitations
The findings of our audit raise serious concerns about the reliability and validity of algorithmic personality tests in pre-hiring assessments. Employers relying on these tests may unknowingly introduce biases and inconsistencies into their hiring processes. Moreover, the lack of clear guidance on time periods during which test results remain valid poses further challenges. However, it is important to note that our study has limitations, and further research is needed to explore additional aspects of these systems.
🔍 Conclusion
In conclusion, our audit of algorithmic personality predictors for hiring shows that both Humantic AI and Crystal exhibit substantial instability on key facets of measurement. This instability undermines the reliability and validity of these tests as pre-hiring assessment instruments. Consequently, caution should be exercised while relying solely on algorithmic personality predictions in the hiring process. For a detailed analysis and additional results, please refer to the published paper.
Highlights:
- Algorithmic personality tests in hiring are widely used but raise concerns about reliability and validity.
- DISC and the Big Five models are commonly used to assess personality traits.
- Humantic AI and Crystal are major algorithmic personality testing systems in the market.
- Our auditing methodology reveals substantial instability in predictions made by these systems.
- Employers should exercise caution when using algorithmic personality tests for pre-hiring assessments.
FAQ:
Q: What are algorithmic personality tests?
A: Algorithmic personality tests are tools used in hiring to predict job seekers' future success based on their resumes or social media profiles. These tests leverage machine learning algorithms to extract and analyze personality traits.
Q: What is the DISC model?
A: The DISC model is a behavioral psychology test that assesses four personality traits - dominance, influence, steadiness, and conscientiousness.
Q: What is the Big Five model?
A: The Big Five model consists of five personality traits - openness, conscientiousness, extraversion, agreeableness, and neuroticism. It is widely used in personality assessment.
Q: What is the reliability of algorithmic personality predictors?
A: The reliability refers to the consistency of measurements made by algorithmic personality predictors. It is an essential aspect in determining the validity of these tests.
Q: What are the key findings of the external audit?
A: The external audit revealed substantial instability in the predictions made by both Humantic AI and Crystal. Personality profiles varied significantly based on the input source, file format, and even participant's time, raising concerns about the reliability and validity of these systems.
Q: What are the implications for employers using algorithmic personality tests?
A: Employers should exercise caution when relying solely on algorithmic personality predictions in the hiring process. The instability and biases introduced by these tests can undermine the fairness and effectiveness of pre-hiring assessments.
Resources:
