Shaker Consulting Group Logo Virtual Job Tryout Logo

Posts Tagged ‘HR Analytics’

November 7, 2011

Blinded By Star Gazing and the A Player Myth

Writing in the recruiting space has generated a lot of attention on strategies for hiring A Players, Top Talent and Star Performers. While that sounds great, I think all that star gazing has blinded a few recruiters. In part, having a poorly calibrated candidate evaluation process in place is to fault.

There is no team roster filled with the likes of Lebron James, Michael Jordan or Barry Bonds, (Although Miami tried). There is no company executive committee completely staffed with the likes of a Warren Buffet. The reason can be largely explained by population statistics. A Players or bright stars only make up a small percent of the available population.

As such, it is more of a myth to make all A Player hires. The size of the candidate population might have to be enlarged exponentially to create a finalist pool of only A Players to choose from. That could be a monumental task. The organization might not have the appetite for the time requirements nor the budget to complete such an undertaking. There is another approach to contributing to organization performance with each hiring decision.

Dim Stars Get Hired Too
Making the right hiring decision requires complex reasoning. To put this into perspective requires that you wrestle with another concept within population statistics known as variation. This can be best understood by looking at your hiring track record. You hired your best, and you hired your worst. When you examine the performance differences between those hired into one job, the variation in decision quality is revealed. Using a process improvement tool called Pareto Analysis (80-20) the impact of low end variation can be revealed.

Obtain a data set of performance variables from a group. Sales performance is easy place to look. Obtain territory revenue per sales rep in a spread sheet.  Calculate the average sales per territory. Next calculate the average for the top 80% and the bottom 20%.   Look at the gap.  After you stop shaking your head, you have to admit, “Yes our process hired those bottom 20% folks too.”  You can explore the impact of this with our ROI Calculators.  You can perform this same form of HR Analytics on any dimension of performance. It is pretty revealing.

This analysis reflects the current nature of the population from which you draw, and the decision quality variation that allows in your poorest performers. In your Shining Star hiring program, Dim Stars get hired too.

Scale of Magnitude
Hipparchus , the ancient Greek astronomer created a the six point magnitude scale to calibrate the relative brightness of stars. Since then the scale has been expanded, revised and refined to better describe the difference observed in the brightness of heavenly bodies.  Hipparchus uses analytical models to refine his conclusions. Your process hired the dim stars because of the calibration of your brightness scale.   Shining stars and dim stars looked more alike than different. The evaluation process was unable to see the difference.  Using HR Analytics your candidate evaluation can be refined and your hiring decisions improved. Better candidate data can improve the yield of your staffing process.  Maybe your recruiter was blinded by star gazing.

With a well calibrated candidate evaluation process, you get better data, which can support more effective hiring decisions.

Here are a series of examples of on-the-job performance differences that were identified by score ranges during validation analysis of a Virtual Job Tryout. Each chart depicts the performance gap between individuals who scored in the top 80% versus those who scored in the bottom 20% of a Virtual Job Tryout created specifically for the job they hold.

Top 80% Achieve 47% Higher Territory Revenue

Individuals who scored in the bottom 20% on the Virtual Job Tryout, on average produced $1.6 million LESS in sales revenue. In other words, about $1.6 million of performance variation was at risk with every hiring decisions. Recruiters letting soft glowing heavenly bodies into the sales organization.

Top 80% Achieve 8% Higher Closing Ratio

Individuals who scored in the bottom 20% on the Virtual Job Tryout, on average closed the deal on 8% FEWER opportunities.   In this case, millions of dollars of revenue are at stake with a lower closing ratio.  Candidates with less effective skills and attributes for bringing in new business were entering the sales organization.

Top 80% Scores Achieve 70% Higher Level of Products Booked

Individuals who scored in the bottom 20% on the Virtual Job Tryout, on average produced 70% FEWER transaction per month.  The previous candidate evaluation process confidently advanced less capable  individuals  into the sales organization.

Top 80% Scores Achieved 21% Higher Commission Levels

Individuals who scored in the bottom 20% on the Virtual Job Tryout, on average earned$21% LESS  in sales commissions. The Virtual Job Tryout is well equipped to discern underlying traits and characteristics that drive performance differences.

When you see the order of magnitude and the insight into performance provided by candidate results, ask yourself; “What would my workforce look like if I could hire from the top 80 %, or even the top 50% of the candidate pool?”

When you calibrate your candidate assessment process to on-the-job performance, you can better distinguish the difference between stars and black holes.

Call us for more information on calibrating your candidate assessment process to reduce low-end performance variation.  And, remember, the sun is a star. If you stare at it, you can go blind.

September 22, 2011

Validation of a Pre-employment Assessment and Crowdsourcing

Find the truth about predicting on-the job success

This week in the journal Nature Structural & Molecular Biology, it was reported that gamers contributed to a scientific breakthrough.  The problem-solving was achieved by a form of crowdsouring with a focused purpose. Earlier this year, I wrote about Jane McGonigal and her view that gamers can make significant contributions to solving significant world problems.  This is one more piece of evidence that her theory is on track.  Validation of a pre-employment assessment can be viewed as a form of crowdsourcing to solve a complex staffing problem.

Read this quote from an article about the breakthrough.

“Games provide a framework for bringing together the strengths of computers and humans. The results in this week’s paper show that gaming, science and computation can be combined to make advances that were not possible before.”

Gaming, science and computation are at the core of the Virtual Job Tryout.  We crowdsource from two groups to solve the question of what it take to be successful in a job.

When you ask ten recruiters or ten incumbents about what it takes to be successful in a job, you get ten opinions.  Of course there will be some overlap, but the overlap will contain both true and false assertions.  Humans can describe the same experience in many different ways.  People perform the same task differently as well.  What is needed to solve complex staffing problem is better candidate data.

The Virtual Job Tryout is a bit of a game.  It is work-sample and problem solving activity.  We crowdsource a large group of existing employees to complete the sample activities in the validation process.  When hundreds of people complete the same tasks, we obtain a robust data set on different approaches used to address the same issues.  We also crowdsource a comprehensive data set of on-the-job performance by asking the supervisors and managers to document productivity and rate competencies of the existing employees.

Computers are good at collecting information in a standardized format.  The web makes it easy to deliver an engaging, multi-media experience that can mimic certain aspects of a job.  In addition, how people navigate web experiences allows us to collect far more data than just a specific response.  Think about it like solving a math problem.  Sometimes the teacher wants to only see the answer, however, sometimes seeing ‘the work’ is more insightful.  The web allows us to collect ‘the work’ as well.

Industrial-Organizational psychologists are scientists.  One specific skill set of these scientists is developing algorithms to drive insightful outcomes from HR analytics.  The development of the correct algorithm is critical.  Using algorithms based upon validation from other companies delivers ‘vanilla candidates’ at best.  At worst, you hire candidates just like your competitors, thus reducing your differentiation in the talent aspect of your business.  Using data from your company, your employees, and your candidates is what makes pre-employment assessment work most effectively.

If you want to use crowdsource data to create  a highly effective solution to your complex staffing problem, give us a call.

The discovery may not be as significant as learning more about the HIV virus.  However, a better way to define what it takes to be successful in your company can improve the health of your bottom line.

July 25, 2011

Shaker Consulting Group Hires Dr. E. Daly Vaughn to Support Virtual Job Tryout Design

To meet client growth and expanding market demands, Shaker Consulting Group is proud to announce the hiring of Dr. E. Daly Vaughn as Virtual Job Tryout Design Scientist.

“His experience in HR analytics, pre-employment assessment design coupled with the use of social media in hiring will prove invaluable as we expand our service offering,” said Joseph P. Murphy, vice president of Shaker Consulting Group.

Vaughn, a native of Texas, with a Ph.D. in Industrial/Organizational Psychology, brings a unique mix of capabilities to the firm.

For more information, read the full release: Shaker Consulting Group Hires Dr. E. Daly Vaughn to Support Virtual Job Tryout Design and Enhancement of the Candidate Experience.

March 28, 2011

Alchemy and Algorithms – Recruiting by Ego or Evidence

Alchemy attempts to take common materials and transform them into something rare and valuable. I don’t think anyone has succeeded in this endeavor to date.

Algorithms Can Be Derived from HR Analytics

Unlike alchemy, algorithms can turn raw goods into gold.   The raw goods can be candidate evaluation data and the gold is on-the-job performance.  However, many recruiters have not invested in the data collection and analysis required to create an algorithm.  As such, they make decisions based upon anecdote and conjecture.

Stock traders want to predict future prices and values of individual companies and broader indices.  Recruiters want to predict future behaviors and on-the-job results of candidates. Algorithms are used by the best-in-class of both of these disciplines. And the results they achieve are documented by superior outcomes.

The reason both of these professions use algorithms is to identify meaningful relationships among complex data sets.

Variables that drive company performance and market fluctuations are complex. And, there is likely no doubt in your mind that variables which drive people’s performance are complex, very complex. In fact you might assert people are unpredictable. If that was really the case the workplace would be chaos. And that is just not true. There are some predictable elements.

Algorithms are special equations, expressly for the purpose of teasing out insights and conclusions from complexity.

When used well, the outcome of algorithms increases the probability of making a correct decision more often than not. An algorithm based upon pre-employment testing brings a sophisticated level of HR Analytics that can dramatically improve your quality of hire.

Algorithms were used to determine the premium for your auto insurance, your credit score, the offer you received for a vacation package, and the books recommended for you in on-line shopping. In each case two or more large data sets were analyzed to determine the nature and significance of relationship that exist between and among the variables.

Big Bucks for Equations.

In a current algorithm competition $3 million is being offered for the equation that takes large data sets of health care and lifestyle information and calculates the likelihood of an individual being hospitalized sometime in the future. The underlying assumptions are two-fold. You could be charged a higher insurance premium based upon your probable path to the hospital, or you could be given a specific preventative intervention to reduce or eliminate the necessity of being admitted for medical care.

Why a competition?  The analysis and mental energy required to derive the equation is significant. Asking one individual to undertake the work may take a long time. A competition can attract the intellectually curious and competitively driven statisticians. Having a solution sooner than later is valuable.

How much would your organization pay for an algorithm that predicted your customer’s behavior?  Or possibly a more accurate question is how much has your organization already paid in an effort to better understand and predict your customer’s behavior. Go ask your chief marketing officer.

Ego or Evidence?

Best-in-class recruiting professionals use algorithms.  (We can introduce you to some of them.) Each hiring decision is supported with evidence.  But, just like the challenge in the competition, developing algorithms require thoughtful effort.  When I describe the process of developing a recruiting algorithm, I get two reactions.  One says,”That seems like a lot of work.” The other states. “That seems like it can add significant value to our process.”

Algorithms are derived from analyzing large data sets. Three data sets are required for transforming recruiting raw goods into job performance gold:

  1. Candidate Evaluation data – pre-employment assessment
  2. Behavior/Competency Evaluation data – supervisor ratings
  3. Productivity Evaluation data – objective metrics of on-the-job performance

Recruiting professionals working at the leading edge of candidate evaluation capture 200 to 300 data points from candidate evaluation. The data encompasses work history, work style and work samples.

Similarly, job performance, as defined by 100 to 200 data points from ratings and metrics for each individual provides a robust description of the complexity inherent in any job and the company culture in which it occurs.

When a recruiting professional embarks on capturing this level of data on their staffing process and its outcome as job performance they have the raw goods for the algorithm that predicts the future and answers the essential question – which candidates are more likely to be successful on the job.  Working with this type of information delviers a very powerful recruiter experience, adding both efficiency and effectiveness.

Differentiated Workforce

And, that ability to differentiate among candidates is competitive advantage. Michael Porter the strategy guru at Harvard states competitive advantage comes from business processes which are difficult to replicate.  In their book The Differentiated Workforce, authors Beatty, Becker and Huselid assert competitive differentiation comes from efforts that align jobs with strategic capabilities. (see page 10).

Using an off-the-shelf assessment, and generalized validity is defined as a ‘Me Too” strategy, one that is easy to replicate.  An algorithm which predicts candidate performance in your organization is impossible to replicate. Call us to explore what it might take to transform your candidate experience into competitive advantage and a strategic business driver.

It’s not alchemy, it’s algorithms. And they really do turn raw goods into gold. Employees who perform at gold star levels.

February 10, 2011

Do You Have A Talent-matician?

Kevin Wheeler wrote a great article on ERE asking about selection science and measurement.  His is suggesting staffing professionals adopt better methods for candidate evaluation or assessment and make more effective use of HR analytics to link candidate evaluation data to business outcomes.

Here are a few questions around measurement discipline, the answers to which may be revealing.

  1. Ask your CFO – “How much has been invested in the data capture and analysis system you use to report EBITA?”
  2. Ask your EVP of Sales – “How much has been invested in the data capture and analysis system you use to report daily sales performance?”
  3. Ask your EVP of Manufacturing ; “How much has been invested in the data capture and analysis system you use to calculate process yield?”
  4. Then ask your EVP of HR (self) – “How much has been invested in the data capture and analysis system you use to create a differentiated work force?”

In every case, for Fortune 1000 companies, the answer to the first three will be hundreds of thousands and in some cases millions of dollars.  Unfortunately the answer to #4 typically pales by comparison.  Why? 

I have never sat with an executive who stated their organization was just like their competition.  In fact, great pride is expressed in how their people, their products, their services are different than others.  The work that true talent-maticians (I just invented that) do is using HR analyitics in quantifying, to the degree possible, the human variables that contribute to those differences.  That requires, rigor, discipline, experiment design, and time.

Michael Porter of Harvard suggests competitive advantage comes from business processes which are difficult to copy.  Authors Becker, Beatty, Huselid, in The Differentiated Workforce present a similar framework for evaluating HR practices that put forth a ‘Me Too’ or a Differentiated outcome.  An example of this is the use of off-the-shelf assessments without local validation.  By default the user states, we are willing to use a measurement tool developed for and by someone else and calibrated by another organization to provide data on our talent decisions.  Sounds like a Me Too tactic.  One path to a differentiated workforce is at least conducting a validation analysis on how the measurement tool (pre-employment test) is adding value to your decision process.  The underlying premise is that a good assessment provides a degree of better data and therefore, better decisions.   With in-house validation, you document the relationship between assessment results and business outcomes. 

Without an in-house validation, the test is not calibrated to performance in your organization and outcomes are anecdotal.  The practice that gives assessment a poor reputation is poor implementation.

In an earlier work by the three authors above The Workforce Scorecard, they document those organization hiring a higher percentage of employees with validated evaluation methods achieve higher levels of financial performance.  Aon and SHRM conducted a significant piece of research in the mid 1990s that included a glimpse at staffing process outcome (out of print but avaiable from the research dept).  Survey participants stated the most lacking qualities in new hires were defined as work style, and basic reasoning.  Those traits or attributes can be objectively evaluated with a variety of pre-employment tests.  Companies stating they were most satisfied with staffing process outcomes were using the most comprehensive candidate evaluation methods.

  • Companies hire engineers to solve complex measurement problems.
  • Companies hire actuaries to solve complex measurement problems.
  • Companies hire statisticians to solve complex measurement problems.
  • Companies that know their competitive advantage comes from their people hire industrial organizational psychologist to solve complex measurement problems in staffing.  These folks are the talent-maticians.

Even if you do not measure variables that provide insight to performance potential, performance variation exists.  In fact you hired your best performer and your worst performer with the same evaluation process.  In manufacturing terms that is known as performance variation and is marked by upper and lower limits.  You see, staffing is a business process with a yield to measure and manage.  To do that requires data capture and analysis.

However, enter another piece of data.  It has been known for some time that a structured interview extracts better candidate evaluation data than an unstructured interview.  In a survey on Use of Objective Candidate Evaluation Methods I conducted with SHRM (write for a copy), very fascinating evidence of interview practices emerged.  Only 55% of respondents stated they use behavioral interviews with questions written in advance (an intentional discovery process).  When asked if the interviews were supported with behaviorally anchored rating scales (a method to discern an effective response from an ineffective response), only 24% of respondents stated this practice was used.  Staffing practitioners are largely ignoring known practices which at the simplest level produce better outcomes.  Implementing assessments requires the same rigor the CFO expects from data capture and analysis in financial matters.

In some jobs, learning more about what factors contribute to retention can add signnficant value.  However,most companies do not even  measure and track the cost of early turnover.  In a survey on Staffing Waste I conducted with SHRM (write for a summary), only 8% of 636 respondents stated they track and report the costs of what I call False Starts – new hire turnover that occurs in less than 120 days.  The analogy would be a head of manufacturing that does not measure defects and scrap rates.  Manufacturing is held accountable for managing the yield of that process.  In my paper Staffing Waste: Identify it, Measure it, Reduce it, a range of examples for applying measuremen- based process improvement to staffing is offered. You can read it here

Yes Kevin, the future of staffing practices will include more measurement, more science, more accountability for understanding and managing process yield.  There are exceptional methods to evaluate candidate-job fit.  It can be measured, it can be analyzed and it can contribute to the bottom line.  However, the practice leaders are already out there, doing the work right now. 

For one example kook at the 2010 ERE Award winner KeyBank.  They reduced staffing waste in one position by over $1.7 million in one year by bringing science and measurement rigor into their staffing process.  They were able to add objective candidate evaluation in a manner that measured candidate-job fit.  The retention and gains in a range of job performance metrics are impressive.

We have many more examples of how talent-maticians drive economic impact from staffing process improvement.  To explore the scope of opportunity you might have, see our ROI calculators.Call me.  We can discuss your opportunity.

January 30, 2011

Do We Need Internal Recruiting? Ask the CFO.

Kevin Wheeler posted an article on ERE that got the recruiting community fired up.  He asked, “Do we need Internal Recruting at all?”  His premise seems to rest with effectiveness, accountability and differentiation that a recruiting function may or may not deliver.

With 32 comments as of this post, it ranks near the top of the charts for getting folks riled up.

Here are my two cents, with a few more details than what I posted on ERE.

The dialogue is all good.  It may be like the question about cars, is it better to buy or lease?  And the answer is: It depends.

Kevin’s main point may really be rooted in economics.  When an internal team has the same mandate to measure, track and report economic impact that an external provider does, there is most likely performance parity.

Unfortunately, the issue lies with the fact that many CFOs and CEOs do not hold internal recruiting teams accountable to document contribution and deliver continuous staffing process improvement.  And without a mandate for economic accountability, the accounting infrastructure to document contribution is often lacking.  A vice president of sales or manufacturing would never be allowed to operate with the poor economic reporting and accounting infrastructure that is deployed for the business process of recruiting.  As such, it is common for internal recruiting teams to use ATS based reporting, thus relying on activity based measures instead of economic measures.

Henry David Thoreau gives us words to ponder for this situation: “It is not enough to be busy, so are the ants.  The question is, what are we busy about?”

One gauge we use to explore the economic accountability of a recruiting team is how literate they are about job-specific performance metrics and how quickly they can access data sets of performance metrics.  Ask a staffing professional, internal or external, if they measure and report on the cost of time to proficiency (total investment from sourcing to self-sufficient performance) for the position with the highest hiring volume.  Ask who owns the budget for staffing waste.  The answers to those questions reveal a great deal about the accountability expectations set by the CFO and CEO for recruiting.

Reporting on days to fill, requisitions open, requisitions per recruiter, and opinion-based quality of hire while good to know are a bit like busy ant metrics.  Recruiters with economic accountability use HR analytics to document and report reductions in staffing waste and rework, increased yield in new hire productivity, reduced time to proficiency, increases in job family average performance metrics and the like. 

From my experience, corporate resources flow to those who build a good business case and then document return on investment.  Outside providers have to do this to earn repeat business.  The best internal providers do so as well. Here is an example of how Key Bank documented high ROI from using pre-employment testing as a form of measurement rigor to reduce staffing waste.

November 24, 2010

More Value from Your Social Media

Kevin Wheeler wrote about Social Media on his ERE post Nov 23.

Kevin offers an excellent invitation to have a strategy and metrics for social media sourcing.  Each one of the social media sources offers a different front end to the candidate experience.  Each social media has a user base of potential candidates with similarities and differences.  The use of exceptional HR analytics can help identify the meaningful differences.

To optimize social media it must tracked by source through various stages and filters such as number of candidates who engaged in the application process by source, number of hires by source and quality of hire by source. 

Sources can vary significantly in overall yield. That means more objective understanding of the value stream is essential.  An example of a firm doing it well is here.  This client case study has lessons to leverage.

November 15, 2010

All Referrals Are Not Created Equal – Quality of Hire = Quality of Referral

John Sullivan points out in his ERE post on referral programs that numbers are available for those who want to invest in measurement discipline and operate at the level of evidence versus opinion.  There is plenty of data that suggests referrals work and make sound business sense.  And, just like the issue of diversity, there are other dimensions of referral process effectiveness that can be quantified.

In a hiring environment using pre-employment assessment, it is possible to examine the relationship between quality of hire and quality of referral.  In one client analysis, we were able to document that individuals scoring higher on the assessment tended to refer individuals who also performed well.  And as one might also conclude, those who scored less well referred candidates with similar performance results.  It is important to create referral behaviors from those more likely to generate high value candidates. 

The anomaly, and there are always some, was the cluster of individuals with modest assessment results but with a high level of referring activity. There was no pattern to the quality of the candidates they put forth.  We called these the ambassadors.  They are just out piping a ‘follow me tune’ attracting all comers.  Even a blind squirrel finds an acorn.  It is important not to get referrals for referrals sake.

As such, using HR Analytics it is possible to target referral behaviors more selectively.  But first, you need better candidate data.

November 9, 2010

Simulations and Selection Science: Interview with Mike Hudy, Ph.D. Part Two

In Part One of the Interview with Mike Hudy, he discussed the demands and opportunities I/O Psychologist face in developing simulation for pre-employment testing.  In this conclusion, Mike offers a few suggestions on how to determine if a simulation may be appropriate for staffing process improvement in your organization.

What considerations should a company examine in deciding if a simulation would be appropriate for one of their jobs?

There are several factors to consider when examining if a simulation makes sense.  If you have jobs with more than 100 incumbents, building a business case for simulations is typically pretty easy.  Another factor is hiring volume. If you will hire more than 100 people into the same job in a year, simulations can make a significant contribution. 

An additional factor would be the complexity of the job itself.  This variable is often under-valued prior to a thorough job analysis.  The more complex the job, the more complex the demands are on the pre-employment assessment. 

The last and a very important factor to consider in the use of simulations is the candidate experience.  As general rule, candidates find simulations engaging, a more valuable way of presenting their capabilities and companies who use simulations stand out in a positive way from other places the candidate may be applying. 

In short, simulations such as the Virtual Job Tryout add selection science value across a range of factors that have a positive impact on staffing process improvement.

 Part One

September 30, 2010

Social Media and Quality of Candidate | Candidate Competencies Vary by Source (Part 2)

A few months ago, I posted a blog on social media and quality of candidate. In the post, I suggested that we need to use HR analytics to evaluate this source of candidates not only by the volume of candidates generated but also by the quality of candidates produced.  We conducted some preliminary analysis using assessment scores from the client’s Virtual Job Tryout and candidate conversion rate (what percentage of candidates that actually hired from a source) as quality of candidate measures.  Results were somewhat mixed, but suggested that social media was generating a quality of candidate that was less than other sources used by the organization (e.g., referrals, job boards, etc.).

Candidate hiring rate varies by social media source

Well, we dug a little deeper into this data and a very interesting picture emerged.  When we looked at the data by the various social media sites used by recruiters, two surfaced as being particularly effective:  LinkedIn and Facebook.   Candidates sourced via LinkedIn performed much better on the pre-employment assessment than candidates sourced through other channels.  In addition, these candidates were hired at a higher rate than the typical candidate.  This pattern held true for Facebook as well, but the results were not as impressive.

Candidate quality varies by social media source

We also compared pre-employment assessment results for candidates surfaced from LinkedIn versus Facebook and found some differences that at first glance seem to make sense.  Candidates sourced through LinkedIn performed better on professionally oriented competencies such as Leads Courageously, Develops Others, and Achieves Results.  Conversely, candidates sourced via Facebook performed better on more socially oriented competencies such as Customer Focus and Works Well with Others.  Source can impact quality of hire.

While we have only scratched the surface here, these results from detailed HR analytics show that there is great promise and potential value to evaluating social media, as well as other recruiting sources, on the quality of its yield.  Further, the data suggests that different social media channels generate different types of candidates with unique competencies and characteristics.  Recruiters can use this kind of information to drive more strategic sourcing efforts by placing their bets on the channels that are best aligned with the type of candidate they’re looking to source.

Part 1

RSS Feed LinkedIn