Spark Hire’s Predictive Talent Assessment offers a range of built-in predictive profiles based on key competencies for various jobs to help you find the best candidates. This article covers how the Predictive Talent Assessment was developed and tested for validity and fairness. For a general overview of Predictive Talent Assessment, read our Predictive Talent Assessment overview article.
Predictive Talent Assessment development
Origins and purpose of the Predictive Talent Assessment
The Predictive Talent Assessment was originally developed to address the need for a legally compliant and predictive hiring tool for law enforcement roles. Traditional assessments, such as aptitude tests, failed to predict future performance reliably. Spark Hire’s Predictive Talent Assessment bridged this gap by focusing on job-specific performance metrics and evidence-based validation.
Research and methodology
- Job analysis: The Predictive Talent Assessment is built on data from over 600,000 assessments, correlating behavioral patterns with job success.
- Actuarial approach: Researchers used statistical modeling to identify item responses that predict successful outcomes, enhancing accuracy.
Assessment tools and competency framework
To capture core competencies across roles, the team developed the Motivational Analysis Profile (MAP) for motivational drivers and the Self-Descriptive Index (SDI) for temperamental traits. Further scales were added for specialized job categories like sales, technical, and managerial roles. Profiles were validated with Subject Matter Experts (SMEs) to ensure alignment with role-specific requirements.
Continuous validation and adaptability
The Predictive Talent Assessment has expanded into a framework of over 130 competencies tailored for specific jobs. Profiles are customized through either current validation studies or existing benchmarks, providing consistent reliability and adaptability. This continuous refinement sustains the assessment’s accuracy and fairness, meeting industry standards for predictive and equitable employee selection.
Validity of the Predictive Talent Assessment
When evaluating the validity of the Predictive Talent Assessment, it's important to understand how well it measures job-related traits and predicts job performance. Validity refers to the degree to which the Predictive Talent Assessment measures relevant traits and predicts job performance.
Spark Hire adheres to recognized standards from the Uniform Guidelines and the American Psychological Association, employing three main types of validity:
- Content validity: This ensures the assessment is representative of all important aspects of the job. It confirms that the test items reflect the tasks, knowledge, skills, and abilities needed for successful job performance.
-
Criterion-related validity: This shows how well the test scores relate to actual job performance, providing statistical evidence that the assessment is predictive of success. It’s split into two types:
- Concurrent validity: Measures the relationship between test scores and current job performance using existing employees (incumbents).
- Predictive validity: Measures the relationship between test scores and future job performance, often using applicants to gauge how well the assessment predicts success for new hires.
- Construct validity: This demonstrates that the assessment accurately measures specific traits or characteristics it purports to assess.
These three types of validity provide a clear picture of how well the Predictive Talent Assessment works in hiring and employee development. High validity means the Predictive Talent Assessment can offer meaningful insights into whether a candidate will succeed in a job, making it easier for companies to make informed hiring decisions. Validity also ensures that the assessment results are reliable and that the conclusions drawn from them are based on solid evidence. A valid assessment, like the Predictive Talent Assessment, is a powerful tool for improving hiring accuracy and supporting workforce development.
Content validity processes
To validate the accuracy of the assessments and make sure they’re measuring the right things, we can conduct thorough content validity studies for each of the Predictive profiles offered by Spark Hire. This involves evaluating the alignment between the competencies measured by the assessment and the actual requirements of the job.
What does this mean?
- Working with subject matter experts (SMEs): We collaborate with SMEs who have detailed knowledge of the specific roles for which the profiles are created. SMEs rate the importance of each competency based on how critical it is for job performance.
- Identifying key job skills and ensuring the assessment measures them: Using the ratings provided by SMEs, we calculate the Content Validity Ratio (CVR) to determine which competencies are considered essential for success in the role.
- Reviewing the results to confirm that the assessment focuses on the most critical competencies: Employers select 3-5 SMEs to ensure the evaluation reflects real-world job requirements.
This process guarantees that the assessments are valid and aligned with the actual job requirements.
Criterion-related validity demonstrated
A summary of representative findings from criterion-related validity research is presented below.
The validity coefficients in the table reflect the correlation between the Predictive Talent Assessment profile scores and job performance criteria, based on data from client organizations for specific roles. This overview covers a broad range of clients and industries where validation studies are typically conducted, though it is not exhaustive. These coefficients offer strong evidence of the Predictive Talent Assessment's ability to predict job performance. By aligning candidate scores with actual job outcomes, the Predictive Talent Assessment helps organizations make more effective and data-driven hiring decisions.
Summary of validity studies
To illustrate the range of industries, positions, and correlations, here is a sampling from the Predictive Talent Assessment's validity studies, showing the predictive power of the assessment across various roles:
Industry | Position | Sample size | Validity coefficient |
---|---|---|---|
Advertising | Acquisition Sales | 100 | 0.57 |
Automotive | Metro District Manager | 101 | 0.57 |
Banking | Acquisition Analyst | 59 | 0.43 |
Business Services | Recurring Revenue Sales Rep | 111 | 0.33 |
Computer Distribution | Account Executive | 133 | 0.49 |
Manufacturing | Furniture Sales Specialist | 89 | 0.51 |
Media | Account Executive (Radio) | 52 | 0.55 |
Retail | Corporate Account Manager | 130 | 0.35 |
Telecommunications | Account Executive | 274 | 0.52 |
Pharmaceuticals | Sales Representative | 402 | 0.26 |
The table below compares the predictive validity of the Predictive Talent Assessment (a Behavioral Assessment) with other commonly used tests and selection methods. The validity coefficients shown are uncorrected, representing the observed correlations from previous validation studies. These results highlight the Predictive Talent Assessment’s strong performance compared to other selection tools, emphasizing its effectiveness as a predictive tool in personnel selection.
Selection measure | Validity coefficient |
---|---|
Behavioral Assessment | 0.43 |
Assessment Centers | 0.36 |
General Mental Ability | 0.31 |
Biodata | 0.30 |
Situational Judgment Tests | 0.26 |
Structured Interviews | 0.24 |
Personality Tests | 0.23 |
Reference Checks | 0.23 |
Integrity Tests | 0.23 |
Unstructured Interviews | 0.18 |
Criterion-related validity and the Workplace Motivation Score
The Predictive Talent Assessment includes a Workplace Motivation Score, which measures an individual’s potential for high motivation and engagement at work. This score is based on three key factors:
- Advocate: How likely employees are to speak positively about the organization.
- Belong: How much employees feel a sense of belonging and commitment to the organization’s goals.
- Contribute: How much employees go beyond their expected duties to help the organization succeed.
Criterion-related validity for the Workplace Motivation Score was established by analyzing the correlation between employees’ scores on these factors and their self-reported engagement outcomes. The analysis showed correlations between each Propensity to Engage score and its corresponding Engagement Outcome ranged from 0.35 to 0.39 in a U.S. incumbent sample, and from 0.38 to 0.52 in an online sample. These results indicate that the Workplace Motivation Score consistently predicts engagement outcomes across different sample groups. All observed correlations were moderate to large and statistically significant at a p < .001 level, confirming the score’s validity as a predictor of workplace engagement.
Local predictive validity studies
The Predictive Talent Assessment is built to predict job performance by correlating assessment scores with real-world job success. While general validity studies offer a broad overview, local validation studies focus on how well the Predictive Talent Assessment aligns with an organization’s specific performance criteria. These studies allow businesses to tailor the Predictive Talent Assessment to their unique metrics, making the assessment more relevant and effective in predicting success for their particular needs. By conducting local validation studies, organizations can enhance the Predictive Talent Assessment’s predictive power, ensuring it aligns closely with company goals and performance standards.
Construct validity described
The Predictive Talent Assessment is based on the IPIP, ensuring strong construct validity through well-established personality dimensions. While the Predictive Talent Assessment doesn’t directly report the Five-Factor Model (FFM), its IPIP items align with these key traits, providing a scientifically sound basis for assessing personality and behavioral traits relevant to workplace success. Construct validity for the Predictive Talent Assessment is demonstrated through two key components:
- Convergent validity: The Predictive Talent Assessment’s factors align closely with other established personality assessments, like the NEO PI-R, which measures constructs similar to the FFM. This consistency supports the Predictive Talent Assessment’s reliability in assessing essential personality traits, even if it doesn’t directly report FFM factors.
- Divergent validity: Structured within the IPIP framework, the Predictive Talent Assessment’s factors are designed to measure distinct personality traits, showing low correlation with unrelated constructs like cognitive ability. This separation confirms that the Predictive Talent Assessment assesses specific personality traits without overlap, ensuring a focused evaluation of characteristics tied to job performance.
Supporting research
Extensive studies have validated the Five-Factor Model (FFM) across diverse cultures and settings, confirming that assessments based on this framework, like the Predictive Talent Assessment, are scientifically robust and effectively capture workplace-relevant personality traits. This research foundation further reinforces the Predictive Talent Assessment’s construct validity, as it reliably measures targeted traits aligned with the IPIP model, even without directly reporting FFM traits.
In summary, the Predictive Talent Assessment’s construct validity is established through convergent and divergent validity, backed by a solid research foundation. Together, these factors ensure that the Predictive Talent Assessment accurately assesses relevant personality traits, making it a reliable tool for predicting workplace success.
Fairness and adverse impact analysis
Adverse impact occurs when an assessment disproportionately affects certain groups. When creating predictive profiles, Spark Hire prioritizes minimizing adverse impact and focuses on making sure the assessments are fair and don’t favor one group over another.
For this, Spark Hire conducts an adverse impact analysis to check for any potential bias. Specifically, we use the 4/5 rule, which means that the selection rate for any group must be at least 80% of the highest rate.
How does the adverse impact analysis work?
- The selection rates are evaluated across different groups to identify any potential disparities.
- The selection rate for any group must be at least 80% of the rate for the group with the highest selection rate. Predictive profiles that don’t meet this requirement are ruled out.
Through this analysis, Spark Hire makes sure that only predictive profiles that meet this requirement are offered in the platform. It ensures that every candidate, regardless of their background, has an equal chance of being selected based on their skills and fit for the job.
Related to
Comments
0 comments
Article is closed for comments.