Shaker Consulting Group Logo Virtual Job Tryout Logo
Blog home page

Archive for the ‘I/O Psychology’ Category

February 13, 2012

Industrial Organizational Psychologists – Uncommon Career, Uncommon Value

Kaitlin Madden, a CareerBuilder writer recently posted a list of the 18 least common jobs according to the Bureau of Labor Statistics (BLS).

Number 6 on the list is Industrial Organizational Psychologist (IOP).  Kaitlin also shares a BLS stat that there are only about 1,420 of them employed across the country.  (This does not include the self employed IOP.)

Society for Industrial Organizational Psychology

Also on the list was the occupation ‘model’ .  A colleague of mine, (an IOP), wondered: “How often do IOP and Model show up on the same list?”  Great question.  This might be the only occurrence.    As it is reported, there are even fewer Models than IOPs.  Both rare, but both have an impact on large audiences.  Models are used to create engaging views FROM a large audiences – namely consumers.  IOPs are used to create insightful views ABOUT a large audience – namely candidates.

Advertising professionals use a model in conjunction with consumer research to optimize market potential .  Recruiting professionals use an IOP to conduct candidate research to optimize hiring decisions for performance potential.

IOPs are the experts who design and validate simulations for pre-employment testing and assessments.  They are the professionals behind selection science.  IOPs are the scientists that create the scoring algorithms for candidate evaluation and document the economic impact of staffing process improvement.  In his recent post, Bill Kutik of HR Executive states HR professionals fear analytics.  IOPs are the professionals with a Ph.D. in HR analytics.

Have no fear, we are here.

OK, so this is stretching an obscure labor stat into a blog post.  I am as guilty as Kaitlin.  However, according to the BLS, Ohio is one of the top five job markets for IOPs.  And we have over 10% of that career market here at Shaker.

So, if you would like to talk with an individual who has chosen an uncommon career that can deliver uncommon value,  give us a call. 888.485.7633

When it comes to HR analytics our figures are impressive.

Posted by: Joseph P. Murphy | February 13th, 2012
Permalink, Comments Off
, Tags: | Posted in I/O Psychology

December 13, 2011

Measure Twice, Cut Once – It is all about job-fit

Sage advice to the trades suggests an accurate fit can be achieved by taking the time to measure, and then to take a second measure to verify, before making the cut. Two measures increase the confidence in and accuracy of the cut. Following that guidance helps reduce waste and rework when crafting a fine object. The same holds true building a workforce that achieves superior results. Using a multi-method pre-employment assessment allows you to measure twice or even seven times within one candidate experience, to help determine job-fit.

Measure Twice, Cut Once

I have worked in the trades and the joke about the advice above is – I cut it twice and it is still too short! Well intended and skillful recruiters sometime take a pipeline full of candidates, cut it twice and still make job-fit hiring decisions that miss the mark. Measurement to support job-fit decisions is critical.

Job-fit is complex. I have never sat across the desk from someone who said, “Our jobs are simple, people don’t need to bring much to be successful here.” In fact, even in entry level jobs, the variables that drive success are complex and can be difficult to objectively measure. Jobs with complex demands require rigorous evaluation methods, methods that measure twice and cut once.

Measure Twice

A common practice in the use of assessment is to administer a combination of a personality or work style questionnaire and a reasoning test. This is a simple form of two measures. The unfortunate and common outcome is poor accuracy and the ‘cut’ can be ‘off the mark.’ Candidate job-fit is far more complex than a test score and diverse high-to-low ratings on a number of personality traits.

Multi-method pre-employment assessment integrates an assortment of evaluation types to deliver a whole-person examination of diverse knowledge, skill, traits, characteristics required by the job demands. Multi-method assessment makes it possible to obtain two or more measurements or evaluations of each job relevant performance domain. When attempting to predict candidate behavior across six to eight competencies, a well developed multi-method assessment can evaluate each competency with multiple measures, thus delivering a confident job-fit measure.

Multi-Method Measurement

Here are some common assessment types that can be integrated into a multi-method assessment.

Situational judgment – choosing among options on how one might respond to common interactions with customers or co-workers

Problem solving – accessing and considering information to address questions, resolve issues

Idea generation/brainstorming – recalling or synthesizing options for a given scenario

Work history – identifying job relevant career experiences, achievements, work habits and career management behaviors

Data analysis – computations, trend analysis, comparisons and drawing conclusions from various information sources

Diagnostic reasoning – applying rule-based logic to system analysis

Prioritization – evaluation and ranking of relative importance and potential consequences of work flow demands

Delegation – discerning appropriateness of and approach to assigning work to others

Multi-tasking – splitting attention among competing demands while performance complex tasks

Work style- comparative description of preferred behavior patterns

The elegance afforded by many of these assessment methods is the ease by which the content can be created to reflect or mimic the actual demands of the job. For example, a day-in-the-life of a manager may include working with operating statements to identify issue that need attention, coming up with a variety of ways to handle the action items, selecting action items to delegate to team members and being prepared to handle a variety of team member responses.   A multi-method assessment can combine a series of exercises that present that entire sequence; Data analysis, idea generation/brainstorming, delegation, situational judgment.

Whole Person Job-Fit Profile

Whole Person Job-Fit Profile

The complexity and diverse range of job-fit attributes measured with this approach allows candidate results to be presented across a job specific competency model.  This is done through the use of HR analytics and a scoring algorithm that weights and values candidate responses according to their relationship with actual on-the-job performance.

In addition to obtaining the evaluation information, the candidates are invited to step into the job and get a glimpse of what it is like to handle the work flow. The assessment can become a form of realistic job preview.

Pre-employment testing has evolved a great deal in the past few years. The web has provided a format for delivering a highly engaging and robust multi-method assessment experience. If you value accurate job-fit, it may be time to explore how a multi-method pre-employment assessment could support your recruiting and hiring process.

Call (888) 485-7633 or write to set up a demonstration.

Measure twice, cut once to reduce staffing waste from your hiring decisions. The result is a workforce that delivers superior results.

Posted by: Joseph P. Murphy | December 13th, 2011
Permalink, Comments Off
, Tags: | Posted in Candidate Experience, HR Analytics, I/O Psychology

September 22, 2011

Validation of a Pre-employment Assessment and Crowdsourcing

Find the truth about predicting on-the job success

This week in the journal Nature Structural & Molecular Biology, it was reported that gamers contributed to a scientific breakthrough.  The problem-solving was achieved by a form of crowdsouring with a focused purpose. Earlier this year, I wrote about Jane McGonigal and her view that gamers can make significant contributions to solving significant world problems.  This is one more piece of evidence that her theory is on track.  Validation of a pre-employment assessment can be viewed as a form of crowdsourcing to solve a complex staffing problem.

Read this quote from an article about the breakthrough.

“Games provide a framework for bringing together the strengths of computers and humans. The results in this week’s paper show that gaming, science and computation can be combined to make advances that were not possible before.”

Gaming, science and computation are at the core of the Virtual Job Tryout.  We crowdsource from two groups to solve the question of what it take to be successful in a job.

When you ask ten recruiters or ten incumbents about what it takes to be successful in a job, you get ten opinions.  Of course there will be some overlap, but the overlap will contain both true and false assertions.  Humans can describe the same experience in many different ways.  People perform the same task differently as well.  What is needed to solve complex staffing problem is better candidate data.

The Virtual Job Tryout is a bit of a game.  It is work-sample and problem solving activity.  We crowdsource a large group of existing employees to complete the sample activities in the validation process.  When hundreds of people complete the same tasks, we obtain a robust data set on different approaches used to address the same issues.  We also crowdsource a comprehensive data set of on-the-job performance by asking the supervisors and managers to document productivity and rate competencies of the existing employees.

Computers are good at collecting information in a standardized format.  The web makes it easy to deliver an engaging, multi-media experience that can mimic certain aspects of a job.  In addition, how people navigate web experiences allows us to collect far more data than just a specific response.  Think about it like solving a math problem.  Sometimes the teacher wants to only see the answer, however, sometimes seeing ‘the work’ is more insightful.  The web allows us to collect ‘the work’ as well.

Industrial-Organizational psychologists are scientists.  One specific skill set of these scientists is developing algorithms to drive insightful outcomes from HR analytics.  The development of the correct algorithm is critical.  Using algorithms based upon validation from other companies delivers ‘vanilla candidates’ at best.  At worst, you hire candidates just like your competitors, thus reducing your differentiation in the talent aspect of your business.  Using data from your company, your employees, and your candidates is what makes pre-employment assessment work most effectively.

If you want to use crowdsource data to create  a highly effective solution to your complex staffing problem, give us a call.

The discovery may not be as significant as learning more about the HIV virus.  However, a better way to define what it takes to be successful in your company can improve the health of your bottom line.

Posted by: Joseph P. Murphy | September 22nd, 2011
Permalink, 3 Comments »
, Tags: | Posted in HR Analytics, I/O Psychology

July 21, 2011

Gaining Management Acceptance for Assessment Tools

This spring over Easter dinner, I was told by a highly successful, MBA grad from an Ivy League university that he could tell, without a doubt, whether a candidate was right for a job in 5 minutes flat. Perhaps he could. After all, Babe Ruth could hit 60 home runs while breaking every training rule in the book.

AssessmentUnfortunately, few of us are the recruiting equivalents of Babe Ruth. We need all the help we can get to accurately assess and select the best candidates for our jobs. But, how can we convince the would be Babe Ruth’s that there is a better (read: more valid, reliable, legally defensible and fairer) way to assess candidates?

My experience, working as both an external and internal consultant in candidate assessment, points to 3 important factors in gaining assessment tool acceptance:

  1. Assessment Fidelity, how close the assessment looks like the job, is assessment‘s secret weapon. It not only has a stronger track record in prediction of success for most jobs, but more easily gains acceptance from hiring managers and candidates alike. They “get it” because they can readily see and experience the job through the assessment process. Fidelity enhances the candidate experience.
  2. Have a Champion, Mid-way through a recent assessment project, we lost our executive champion. At that point we needed to re-sell the project and find a new champion. We didn’t; leading to acceptance challenges. Lesson learned. Visible leadership endorsement is a must.
  3. Ease of Use impacts  2 audiences: candidates and hiring managers. Hard to find candidates may not always be willing to take your best thought out assessment tools (though fidelity helps a lot here too). This in turn can lead to hiring managers blaming the tool for sourcing problems. Recruiters need to have a clear candidate message to convert those sourced into applicants. Know the organizational culture you’re working with. Are hiring managers and candidates patient, analytic and detail oriented or, fast paced, go with the gut types? Tailor your tool to meet your audiences’ appetite and realistically balance assessment style and time against tool acceptance.

A valid tool with strong psychometric properties and eye popping utility numbers is still only as good as management’s willing acceptance of it. The key to assessment success goes beyond validity. It is achieved by gaining the support of hiring managers through fidelity, a visible champion and making it easy to use.

Administrator’s Note:

John Miraglia is our first external contributor to the blog, a former client, and professional colleague. He has worked on the implementation of the Virtual Job Tryout for professional positions in the financial services industry. His insight and experience on implementing assessment is highly valued.

Posted by: John Miraglia | July 21st, 2011
Permalink, 1 Comment »
, Tags: | Posted in Client Case Studies, General, I/O Psychology

February 10, 2011

Do You Have A Talent-matician?

Kevin Wheeler wrote a great article on ERE asking about selection science and measurement.  His is suggesting staffing professionals adopt better methods for candidate evaluation or assessment and make more effective use of HR analytics to link candidate evaluation data to business outcomes.

Here are a few questions around measurement discipline, the answers to which may be revealing.

  1. Ask your CFO – “How much has been invested in the data capture and analysis system you use to report EBITA?”
  2. Ask your EVP of Sales – “How much has been invested in the data capture and analysis system you use to report daily sales performance?”
  3. Ask your EVP of Manufacturing ; “How much has been invested in the data capture and analysis system you use to calculate process yield?”
  4. Then ask your EVP of HR (self) – “How much has been invested in the data capture and analysis system you use to create a differentiated work force?”

In every case, for Fortune 1000 companies, the answer to the first three will be hundreds of thousands and in some cases millions of dollars.  Unfortunately the answer to #4 typically pales by comparison.  Why? 

I have never sat with an executive who stated their organization was just like their competition.  In fact, great pride is expressed in how their people, their products, their services are different than others.  The work that true talent-maticians (I just invented that) do is using HR analyitics in quantifying, to the degree possible, the human variables that contribute to those differences.  That requires, rigor, discipline, experiment design, and time.

Michael Porter of Harvard suggests competitive advantage comes from business processes which are difficult to copy.  Authors Becker, Beatty, Huselid, in The Differentiated Workforce present a similar framework for evaluating HR practices that put forth a ‘Me Too’ or a Differentiated outcome.  An example of this is the use of off-the-shelf assessments without local validation.  By default the user states, we are willing to use a measurement tool developed for and by someone else and calibrated by another organization to provide data on our talent decisions.  Sounds like a Me Too tactic.  One path to a differentiated workforce is at least conducting a validation analysis on how the measurement tool (pre-employment test) is adding value to your decision process.  The underlying premise is that a good assessment provides a degree of better data and therefore, better decisions.   With in-house validation, you document the relationship between assessment results and business outcomes. 

Without an in-house validation, the test is not calibrated to performance in your organization and outcomes are anecdotal.  The practice that gives assessment a poor reputation is poor implementation.

In an earlier work by the three authors above The Workforce Scorecard, they document those organization hiring a higher percentage of employees with validated evaluation methods achieve higher levels of financial performance.  Aon and SHRM conducted a significant piece of research in the mid 1990s that included a glimpse at staffing process outcome (out of print but avaiable from the research dept).  Survey participants stated the most lacking qualities in new hires were defined as work style, and basic reasoning.  Those traits or attributes can be objectively evaluated with a variety of pre-employment tests.  Companies stating they were most satisfied with staffing process outcomes were using the most comprehensive candidate evaluation methods.

  • Companies hire engineers to solve complex measurement problems.
  • Companies hire actuaries to solve complex measurement problems.
  • Companies hire statisticians to solve complex measurement problems.
  • Companies that know their competitive advantage comes from their people hire industrial organizational psychologist to solve complex measurement problems in staffing.  These folks are the talent-maticians.

Even if you do not measure variables that provide insight to performance potential, performance variation exists.  In fact you hired your best performer and your worst performer with the same evaluation process.  In manufacturing terms that is known as performance variation and is marked by upper and lower limits.  You see, staffing is a business process with a yield to measure and manage.  To do that requires data capture and analysis.

However, enter another piece of data.  It has been known for some time that a structured interview extracts better candidate evaluation data than an unstructured interview.  In a survey on Use of Objective Candidate Evaluation Methods I conducted with SHRM (write for a copy), very fascinating evidence of interview practices emerged.  Only 55% of respondents stated they use behavioral interviews with questions written in advance (an intentional discovery process).  When asked if the interviews were supported with behaviorally anchored rating scales (a method to discern an effective response from an ineffective response), only 24% of respondents stated this practice was used.  Staffing practitioners are largely ignoring known practices which at the simplest level produce better outcomes.  Implementing assessments requires the same rigor the CFO expects from data capture and analysis in financial matters.

In some jobs, learning more about what factors contribute to retention can add signnficant value.  However,most companies do not even  measure and track the cost of early turnover.  In a survey on Staffing Waste I conducted with SHRM (write for a summary), only 8% of 636 respondents stated they track and report the costs of what I call False Starts – new hire turnover that occurs in less than 120 days.  The analogy would be a head of manufacturing that does not measure defects and scrap rates.  Manufacturing is held accountable for managing the yield of that process.  In my paper Staffing Waste: Identify it, Measure it, Reduce it, a range of examples for applying measuremen- based process improvement to staffing is offered. You can read it here

Yes Kevin, the future of staffing practices will include more measurement, more science, more accountability for understanding and managing process yield.  There are exceptional methods to evaluate candidate-job fit.  It can be measured, it can be analyzed and it can contribute to the bottom line.  However, the practice leaders are already out there, doing the work right now. 

For one example kook at the 2010 ERE Award winner KeyBank.  They reduced staffing waste in one position by over $1.7 million in one year by bringing science and measurement rigor into their staffing process.  They were able to add objective candidate evaluation in a manner that measured candidate-job fit.  The retention and gains in a range of job performance metrics are impressive.

We have many more examples of how talent-maticians drive economic impact from staffing process improvement.  To explore the scope of opportunity you might have, see our ROI calculators.Call me.  We can discuss your opportunity.

Posted by: Joseph P. Murphy | February 10th, 2011
Permalink, Comments Off
, Tags: | Posted in General, HR Analytics, I/O Psychology, Surveys

November 9, 2010

Simulations and Selection Science: Interview with Mike Hudy, Ph.D. Part Two

In Part One of the Interview with Mike Hudy, he discussed the demands and opportunities I/O Psychologist face in developing simulation for pre-employment testing.  In this conclusion, Mike offers a few suggestions on how to determine if a simulation may be appropriate for staffing process improvement in your organization.

What considerations should a company examine in deciding if a simulation would be appropriate for one of their jobs?

There are several factors to consider when examining if a simulation makes sense.  If you have jobs with more than 100 incumbents, building a business case for simulations is typically pretty easy.  Another factor is hiring volume. If you will hire more than 100 people into the same job in a year, simulations can make a significant contribution. 

An additional factor would be the complexity of the job itself.  This variable is often under-valued prior to a thorough job analysis.  The more complex the job, the more complex the demands are on the pre-employment assessment. 

The last and a very important factor to consider in the use of simulations is the candidate experience.  As general rule, candidates find simulations engaging, a more valuable way of presenting their capabilities and companies who use simulations stand out in a positive way from other places the candidate may be applying. 

In short, simulations such as the Virtual Job Tryout add selection science value across a range of factors that have a positive impact on staffing process improvement.

 Part One

Posted by: Joseph P. Murphy | November 9th, 2010
Permalink, Comments Off
, Tags: | Posted in HR Analytics, I/O Psychology

November 4, 2010

Simulations and Selection Science: Interview with Mike Hudy, Ph.D. Part One

Mike Hudy is an Industrial/Organizational (I/O) psychologist and principal of Shaker Consulting Group.  He began designing custom simulations for pre-employment testing in 1997.  His work is marked by innovation in developing high-fidelity, on-line work samples and interactive evaluation experiences that expand the science and art of the profession. 

In what ways have simulations for pre-employment assessments changed the way I/O psychologists think about measurement science for the hiring process?

Psychologists have to apply traditional psychometrics to a more complex playing field.  In developing a simulation you have to capture core elements of the job in a manner that is not overly complex yet still accounts for traditional psychometric principles.  Now I/O psychologists have an opportunity and challenge to be better at balancing art with selection science.

Tell me more about the Art.

The art is the process through which we gain an understanding of a job and devise a way to represent or recreate aspects of the job in an internet delivered simulation.  Simulations collect a work sample through an informative and interactive candidate experience. This method captures a level of data a traditional Likert scale or multiple-choice assessment can never achieve.  The art is to capture some of the complexity without making it overly intricate.  The candidate needs to be able to proceed with minimal instruction to complete the exercises.  And the exercise needs to be clearly job relevant.

Is that where the power of face validity comes into play?

Yes, it is the goal is to invite the candidate to step into the role and perform elements of job which measure attributes critical for success and do job.  We create and deliver candidate evaluation in a way that the individual does not feel like they are being tested.  They know what is going on, however,the link to the job is so strong and clear.  Feedback we get from candidates strongly suggests they appreciate being afforded the opportunity to complete the Virtual Job Tryout.  They come away with a better understanding of the career opportunity they are considering.  Exposure to the role through well balanced realistic job preview and concrete elements of job demands puts the candidate in better position to decide if the job is right for them.  When we accomplish that, we know the art has achieved its purpose. 

The psychometric challenge is to still get good reliable measurement of the construct you are trying to tap into without introducing too much noise into the exercise.  What I mean by that is simulations can introduce many more moving parts into the measurement experience.  With that the risk is the moving parts or elements of the simulation could have an unintended impact on what it is you are actually trying to measure.

Can you give me an example of this?

A good example is we developed simulations for two different call center jobs.  One of them more closely resembled the actual problem solving on the job.  It simulated searching for, finding and using information to solve problems by looking for information in a multi-layered data base.

The second problem solving simulation was much simpler. It eliminated the need to search for and find information and dealt exclusively with the ability to use technical information to address customer issue and resolve problems.

While the first simulation more closely resembled the actual job, we achieved better results predicting on-the-job performance with the simpler, second simulation. 

By introducing the searching and identification task, it became a distracter and we limited our precision in assessing the actual problem solving ability.

How does that difference in complexity impact the way the candidate responds?

Candidates appreciate engaging, interesting and interactive exercises.  Not all applicants appreciate increased complexity in their candidate experience.  And, they let us know about it in the feedback.

So, how do you determine the level of complexity that is appropriate?

That is the intersection of Art and Science.  The key is to constantly take off your I/O hat and view it from the candidate’s perspective, through the test takers eyes.  At Shaker we do this through defined roles in our project teams.  It includes peer review, end-user advocate review and then a significant population of incumbents during the validation phase.  We learn more from each perspective and refine the exercises.  In developing a Virtual Job Tryout, at least four I/O psychologists will critically evaluate the experience through the eyes of the candidate.  Our programming team has over 20 years of experience designing graphically rich user interfaces and technology based training.  Each layer of feedback impacts the design.  Ultimately, the data from our HR analytics will tell us if we have it right or not.

In what ways do simulations increase the power of the selection science?

Human behavior is complex.  What defines success in any given job is complex.  Simulations allow us to measure a range of capabilities that do not lend themselves to be readily measured with traditional evaluation tools.  For example, let’s consider multi-tasking. That is the ability to split attention between numerous competing tasks.  

Measures such as personality, cognitive ability, and biodata are not able to accurately assess this construct.  Thus we developed a multi-tasking simulation that places candidates into situations where they must divide their attention between a variety of tasks that simultaneously compete for their attention.  Individuals who perform well in this exercise perform better in environments that truly demand those skills.  In call center agents, proficiency in this construct correlates to more efficient after call work and better handle times.

With a simulation we are able to capture more robust work samples such as speed accuracy, latency of response, navigation accuracy, and learning from repetition in one exercise.  Traditional and static measures such as personality and critical thinking are just not able to zero in on the subtle complexities of certain job performance domains.

Part Two

Posted by: Joseph P. Murphy | November 4th, 2010
Permalink, Comments Off
, Tags: | Posted in HR Analytics, I/O Psychology

RSS Feed LinkedIn