Shaker Consulting Group Logo Virtual Job Tryout Logo

Posts Tagged ‘validity’

April 8, 2014

Nature’s Way of Saying Don’t Touch – Language from Assessment Provider?

Gary Larson – the Far Side cartoonist had a great panel a number of years ago that I remember to this day.

There was a picture of an individual with a boot on his head, rubber floatie, trench coat and a shot gun. The caption read “Nature’s way of saying don’t touch.”

Enter at your own risk.

I received an email today to try an assessment.  One of the final lines in  the  message read as follows:

“[Company] rely on psychological investigations that support the results of the tests based on scales and personality traits and are supported by the standard reliability and validity coefficients required by the EEOC, the Labor Department and the Justice and Civil Service Commission.”

I was not sure what it meant. But my reaction was just like seeing the character in Gary Larsen’s cartoon.

Call those various agencies and ask them what personality traits are supported by their required standard reliability and validity coefficients.  Then let me know how those conversations went.

Sometimes it is better to be cautious,  enter at your own risk was my thinking.

October 11, 2012

Computer as Recruiter? – They Lack Data, Analysis, and Judgment

Charles Handler wrote a very thought provoking article about the future of computers and hiring decisions on ERE.

Charles – thanks for continuing to invite us forward.

The decision to hire will most likely always be an act of personal judgment.  However, better data regarding the variables that impact the quality of the decision is what differentiates down-stream outcomes.  And in the case of hiring, that means on-the-job performance.

Data-loops provide a means to manage outcomes

His first and last bullets are the ’sit up and take notice’ elements to embrace.

  • Algorithms must be fed quality post-hire performance data to be useful.
  • Our concept of validation will need to be expanded.

All the writing on Big Data is capturing the imagination of business and working its way into the business process called staffing.  To extract value from Big Data requires rigor and discipline.  This is the work of HR Analytics.

The discipline of capturing and feeding back post-hire performance data requires a system and resources.  Lack of an infrastructure to capture, analyze and report objective performance metrics is a huge barrier in many organizations.  There are many jobs where companies just do not measure performance.  The commitment of resources, i.e., manager completion of behaviorally anchored ratings, is often prevented from an attitude of ‘Our managers don’t have time for that.’, or ‘That seems like a lot of work.’

The science of servo-feedback has been used to manage process consistency, a feedback loop is used to modify or control the outcome.  By default, organizations unwilling to set up a data loop eschew true learning from experience and evidence-based process improvement. They leave a lot of unrealized potential on the table.

The concept of validation must indeed change.  Too many assessment publishers assert “our test is valid.”  This perpetuates a static-state mind-set of validation analysis, and invites the practitioner to believe there is a universal value in validation.  In fact, using a pre-employment test under claims of it being ‘valid’ is a ‘me too’ tactic which makes the user more like other companies instead of being a driver of their competitive differences.  It might be referred to as striving for vanilla.  The low value of  ’me-too’ approaches to HR practices is discussed well in the The Differentiated Workforce.  (A book well worth the read.)

Validation may be viewed as an academic term for calibration.  Assessment can be viewed as measurement rigor for hiring process outcomes.  Without in-house or local validation, your quality of raw goods measure (candidate characteristics)  is calibrated to the performance variables of other companies – their finished goods (on-the-job performance).  And, I am not sure I have ever heard a staffing or recruiting executive state; “We are just like everybody else.”  In fact, the opposite is true.  The assertion is more like we are different and unique.

So, why then do so many companies rely on off-the-shelf ‘validated’ assessments?  I believe it is because of the barriers posed by the first and last bullets in Charles’ article.  Evidence-based staffing process improvement is work and it requires specialist skills, such as those of I/O psychology.  Not every company has in-house lawyers, yet they hire one when the need expertise to solve a business problem.  Companies hire I/O Psychologists for the same reason – to solve a business problem.

Find a job where staffing process improvement will add value and commit some time, resources and dollars to collecting post-hire data and conducting validation analysis as an on-going business practice.  It’s a great way to document business impact, drive competitive differentiation, and justify resources via ROI reporting.

Some related articles are here:

Alchemy and Algorithms – Recruiting by Ego or Evidence
Validation of a Pre-employment Assessment and Crowdsourcing
Moneyball and Selection Science – Pre-employment Testing

July 21, 2011

Gaining Management Acceptance for Assessment Tools

This spring over Easter dinner, I was told by a highly successful, MBA grad from an Ivy League university that he could tell, without a doubt, whether a candidate was right for a job in 5 minutes flat. Perhaps he could. After all, Babe Ruth could hit 60 home runs while breaking every training rule in the book.

AssessmentUnfortunately, few of us are the recruiting equivalents of Babe Ruth. We need all the help we can get to accurately assess and select the best candidates for our jobs. But, how can we convince the would be Babe Ruth’s that there is a better (read: more valid, reliable, legally defensible and fairer) way to assess candidates?

My experience, working as both an external and internal consultant in candidate assessment, points to 3 important factors in gaining assessment tool acceptance:

  1. Assessment Fidelity, how close the assessment looks like the job, is assessment‘s secret weapon. It not only has a stronger track record in prediction of success for most jobs, but more easily gains acceptance from hiring managers and candidates alike. They “get it” because they can readily see and experience the job through the assessment process. Fidelity enhances the candidate experience.
  2. Have a Champion, Mid-way through a recent assessment project, we lost our executive champion. At that point we needed to re-sell the project and find a new champion. We didn’t; leading to acceptance challenges. Lesson learned. Visible leadership endorsement is a must.
  3. Ease of Use impacts  2 audiences: candidates and hiring managers. Hard to find candidates may not always be willing to take your best thought out assessment tools (though fidelity helps a lot here too). This in turn can lead to hiring managers blaming the tool for sourcing problems. Recruiters need to have a clear candidate message to convert those sourced into applicants. Know the organizational culture you’re working with. Are hiring managers and candidates patient, analytic and detail oriented or, fast paced, go with the gut types? Tailor your tool to meet your audiences’ appetite and realistically balance assessment style and time against tool acceptance.

A valid tool with strong psychometric properties and eye popping utility numbers is still only as good as management’s willing acceptance of it. The key to assessment success goes beyond validity. It is achieved by gaining the support of hiring managers through fidelity, a visible champion and making it easy to use.

Administrator’s Note:

John Miraglia is our first external contributor to the blog, a former client, and professional colleague. He has worked on the implementation of the Virtual Job Tryout for professional positions in the financial services industry. His insight and experience on implementing assessment is highly valued.

October 13, 2010

Candidate Experience: A Waste of Time? NOT!

In his recent blog, Tim Sackett suggests HR may be wasting its time with focus on the candidate experience.  I beg to differ.

Tim suggests great companies to work for are hard to get into because they hire the best.  Being able to hire the best is a result of not only a great candidate experience, but also a great recruiter experience.

Both the candidate and the recruiter need an experience that improves their ability to decide if this job opportunity is a good fit.  Companies that do not pay attention to the candidate experience are more challenged to make the best hiring decision because they have not designed a process that obtains the best data to make the most informed decision.

The candidate experience, may not be the main focus of recruiting process design, but it does offer an opportunity to differentiate.  A recent comment one of our clients received from a candidate completing their Virtual Job Tryout tells some of the story: “You guys need to talk with (competitor), their application process is terrible, this is really cool!”

Candidates in the job market compare and contrast the companies they are considering.  The candidate’s experience can be neutral,  act as a repellant, or an attractant.

Great companies to work for are great companies because they manage their entire Brand: product, service and employment.  In many cases, a candidate can be or might be a customer as well.  The candidate experience can impact the nature of the customer relationship.  A well designed candidate expereince can leave the candidate with a more favorable inclination toward the company.

As for uncomfortable with being measured, that cannot be more true when it comes to the candidate experience. In the Staffing Metrics and Performance Benchmarks Survey from First Resource it is reported that fewer than 3% of companies have a candidate experience measurement in place. 

Tim writes about measurement like its a bad thing. Without measurment, a process cannot be managed.  And the absence of measurement obscures accountability.  The important thing is to measure that which drives business results. 100% of companies with a Virtual Job Tryout measure the candidate experience.  However, it is the validation analysis based upon data from the candidate experience that documents how effective the candidate experience is at improving the quality of  hire.

Many great companies want and deliver a candidate experience as unique as their brand.  However, not all great companies want to be the target for resume spam or too many poor fit candidates.   Companies in general find they are not interested in 90 to 95% of the individuals who apply.  As such, they build an engaging, educational experience that both adds an element of challenge while obtaining better candidate data that provides internal decision support.  Great companies realize the candidate experience and the recruiter experience go hand-in-hand.

If the candidate experience is all about attraction, then it may be a waste of time.  However, if the candidate experience is about smart decision making, great accountability and the competitive differentiation, I think it deserves a great deal of attention.

September 23, 2010

Dissertating and Assessement Development – Similar Scientific Methods

I am Lei Qin, (sounds like: ‘chin’) the newest member of the Shaker Consulting Group team. 

I have recently completed the defense of my dissertation. It was a mix of feelings: relief and anxiety. Why was it a relief? I don’t need to make more endless revisions. This is really a relief.  But as the defense date approached, anxiety accumulated, slowly but robustly.  How did I deal with it? I took a simple strategy: not thinking about it too much until it comes.

I am not blogging to just expose my feelings about dissertating. My experience as an intern with Shaker provided a window of time in which building employee selection tools and dissertating overlaped in my life. This leads me to an interesting observation. A similar scientific process is shared by building a pre-employment assessment and dissertating.

 The first step of building an assessment tool for hiring is job analysis. The primary purpose of job analysis is to understand the targeted position and find out what knowledge, skills and abilities and other variables are essential for superior performance in the job. A competency model of the position is developed based on the job analysis. The first step of dissertating is a literature review and proposal. This step helps to define and understand the research progress of a specific area. After thoughtful consideration you can make a hypothesis about what is missing, unanswered and critical to advance this research area. A research proposal is developed based on the findings of literature review.  In my case, I chose Trust in Leadership as the focus for my research.

The second step of building a hiring assessment tool is content development. A story board, measurement hypothesis, item content, and computerized deployment system are developed based on the competency model and the job analysis results. The developers need to draft or choose appropriate testing components to assemble the pre-employment test. Similarly, the second step of dissertating is research design and material development. A researcher needs to choose appropriate designs for the study and develop study materials such as instructions and survey items. For my research I created three experimental conditions to activate different mindsets in anticipation of influencing down-stream thinking and responses.

The third step of building a assessment tools for hiring is validation analysis. Typically hundreds of participants are invited to take the selection tool. Predictor (test results) and criterion (job performance) data are collected. HR analytical tools are used to explore the relationship among the response patterns on the test and on-the-job performance. This type of HR analytics determines the power of the test to predict on-the-job performance. The third step of dissertating is data collection. Subjects will participate in the study. Data of independent variables and dependent variables are collected. Statistical analysis is conducted to test the hypotheses. My research explored the differences in how concrete versus generalized knowledge about an individual impacts how we establish trust with them.

The final stage of building an employee selection tool is rollout.  After a thorough review of the results, the selection system needs approval from the selection practice leaders prior to rollout  The pre-employment assessment system will be tested in the real recruiting environment. Similarly, the final step of dissertating is defense. The success of a dissertation comes from the approval from a dissertation committee, prior to rolling out a new Ph.D.  Well, I did it! My research was determined to contribute to the body of knowledge on leadership. And now I have been rolled out with a Ph.D. and Shaker has hired me to continue my work and research in the real recruiting environment.

The comparison of building a selection tool and dissertating demonstrated they follow the same rigorous scientific process. At Shaker, the development of Virtual Job Tryout® has one more component; that is, managing a feedback loop which makes the Virtual Job Tryout® adaptive to the real world, getting more accurate.  Data will be collected over time and periodic validation analysis will be conducted to improve the predictive power and document the return on investment (ROI) of the pre-employment assessment 

Earning my Ph.D. has been just one step along my path of continuous learning from experience.  And just like an assessment can get smarter over time, my skills and ability to contribute to a body of knowledge will be tested time and time again.

April 6, 2010

Shaker Consulting Group to Present and Exhibit at SIOP

The Society for Industrial and Organizational Psychology will convene for its 25th annual conference this week in Atlanta, GA.  Industrial-organizational (I-O) psychology is the scientific study of the workplace. Rigor and methods of psychology are applied to issues of critical relevance to business, including talent management, coaching, assessment, selection, training, organizational development, performance, and work-life balance.

This event brings together a diverse group of professionals with deep interest in measuring a wide range of factors from the world of work.  We at Shaker are particularly pleased to be participating in five sessions this year.  Topics covered will range from improving the candidate experience to the additional science power of multi-media in pre-employment testing to assessment design to reduce faking.

In addition to conference sessions, we are also a conference sponsor and exhibitor.  Visit us in Booth 808.

Stop by and ask for a demo of a Virtual Job Tryout.  You will see why brand conscious companies have come to expect more from assessments.

Please review the topics and times  listed below.  We would enjoy having you present to engage in the dialogue.

Cool Assessment Tools Symposium –
Marriott Vacation Club presenting on Virtual Job Tryout®
THURSDAY AFTERNOON, 4/8
12:30 – 1:50pm
Salon B

Interactive Multimedia Simulations: Criterion-Related and Incremental Validity
THURSDAY EVENING, 4/8
6:00-6:50pm
Grand Ballroom A
Top Rated Posters

Faking it well: Effects of surface acting on task performance
SATURDAY AFTERNOON, 4/10
1:30 – 2:20pm
Galleria

Leveraging Technology to Engage Candidates and Deepen Assessments
SATURDAY AFTERNOON, 4/10
2:00-2:50pm
Room 201

There’s More to Selection than Correlation Coefficients: r You Serious?
SATURDAY AFTERNOON, 4/10
3:30-4:20pm
Room 202

RSS Feed LinkedIn