Applicant Questionnaires and Gaming the System

Only 36 percent of employees who responded to the 2017 Federal Employee Viewpoint Survey said promotions in their work unit are based on merit. There are many reasons for the lack of employee and hiring manager confidence in promotion programs. Concerns about “preselection” and favoritism are high on the employee list. Those reasons assume that managers and HR folks are deliberately gaming the system to get to favored applicants.

Maybe those reasons drive some of the problems with merit promotion and competitive hiring programs, but they are not the only problems. In fact, I believe they are not the biggest problem. That honor goes to the assessment process most agencies use for their job announcements. Anyone who has applied for a job in recent years has come across an applicant questionnaire. Let’s just call it an AQ for short.

The AQ is exactly what it sounds like – a lot of questions that purport to distinguish the best candidates from the rest of the pack. In theory, they are a great idea. Simple questions that tell a manager who is among the best qualified (and who is not), without putting a large burden on applicants, should make the hiring process work better. Except when they don’t.

Before we get too far into the AQ, let’s get in the wayback machine and go back in time, to the era before we had the AQ. Then we had KSAs. Remember those? KSAs are knowledge, skills and abilities, and job applicants had to write lengthy narratives describing how their education, training and experience matched the KSAs for the job. Virtually no one liked KSA narratives. They were time-consuming to write, miserable to read, and many folks believed they encouraged applicants to lie embellish their qualifications. The AQ was intended to eliminate those problems and improve the hiring process for HR, applicants and hiring managers. It is safe to say that it did not work out quite like it was expected.

What happened? Why did what seemed like a great idea not make the kind of difference everyone expected? I believe there are three reasons.

  • Hiring managers are not involved in the process as much as they should be. A good AQ requires input from someone who has in-depth understanding of the job being filled. That means an experienced supervisor or subject-matter expert (SME). Too often they are done by HR alone. Even the best HR specialists are not SMEs. The Office of Personnel Management (OPM) recognized the problem when it created its hiring excellence program, which is intended to bring HR and hiring managers together to fill jobs. Unless agencies expand their use of SMEs, the problem with poor quality AQs will not go away.
  • People lie. This one should not be a surprise. The people who lied when they wrote KSA narratives are still lying. The difference is that it is easier and faster now. No need to write 500 words to inflate your experience – just check the box and go on to the next question. Sadly, the number of people who lie on job applications seems to be going up. When we started using the AQ (about 25 years ago), lying in response to the questions did not appear to be a big problem. Now it is.
  • Many of the AQs are just terrible. Even if we solved the problems with lying and lack of SME involvement in the process, the AQs we see for many jobs have to be better. There are two big problems with them. The first is that they take a basic format – five answers ranging from “I don’t know anything” to “I am an expert” and use them for every question. The second is that there are far too many questions. Here is an example of the first problem (from a current job announcement):
Research, collect data, and provide advice and recommendations on research techniques to determine the most efficient and cost effective means of accomplishment.
  1. I have not had education, training, or experience in performing this task.
  2. I have had education or training in how to perform this task, but have not yet performed it on the job.
  3. I have performed this task on the job. My work on this task was monitored closely by a supervisor or senior employee to ensure compliance with proper procedures.
  4. I have performed this task as a regular part of a job. I have performed it independently and normally without review by a supervisor or senior employee.
  5. I am considered an expert in performing this task. I have supervised performance of this task or am normally the person who is consulted by other workers to assist or train them in doing this task because of my expertise.

Finding an example of this type of AQ took 30 seconds. They are everywhere. From the perspective of simplicity, they seem great. Come up with a job requirement, then drop in these responses. The first three answers mean you have never done the work, or you have only done it with close supervision. You have to get to answer #4 to find someone who actually has decent experience. Even #4 does not mean you did it well. It did not take long for a people to figure out that selecting answer 5 would get the most points. It is also hard to prove an applicant who selects answer #5 lied about it. “Considered an expert” is vague, as is “supervised performance.” Almost anyone can say they are “consulted by other workers.” The fact that the hiring and promotion processes are not highly regarded adds to the problem, making it easier for people who would never consider themselves to be liars to pick answer #5, even if they are not objectively what anyone would call an expert. People have told me that they pick the top answer because “It is a way to get to the interview process.” With this type of pattern, we should not be surprised when hiring managers complain that they get lists of people who cannot do the job.

The second part of the problem with AQs is the number of questions. It is common to see 20 or more questions per job. Many of those are junk questions that do not differentiate the great applicants from the average ones. If there are five questions that address the core requirements of the job, and 15 questions that are more general, such as “ability to communicate orally,” it is likely that people who have the necessary skills will not make the cut.

If we want to make the hiring process better, and we want to build employee trust in promotion programs, we have to stop misusing the AQ. They must be tailored to the job and they must have something other than the same five answers. They need to differentiate between good candidates and bad candidates in a way that the five standard responses cannot. And finally, we have to recognize that the AQ is only one part of the evaluation process. When OPM got rid of KSAs, they told agencies they could not use them in the initial screening process. They also said agencies could use multiple assessments. What happened was only the first part. We got rid of the KSAs, but we did not use additional assessments. We may not see comprehensive hiring reform any time soon, but we can and must improve the assessment process now.


2 thoughts on “Applicant Questionnaires and Gaming the System

  1. ES says:

    Where to begin?! Yes the hiring process is equally awful for those that implement it and those who have to apply within it. However, the remedy isn’t making the multi choice questions and answers better – it’s actually educating people on why and how make more effective hiring decisions. You’ve said it many times before – govt supervisors are selected for their individual contributor success and asked (not assessed or developed) to make fully informed and effective long term consequences decisions about hiring and developing people. This practice sets the expectations for successive positions in management and leadership to carry on with individual contributor success and single perspective decision making. It’s a long term recipe for diminishing returns in leadership, management, supervisors, and every aspect of talent management. The really unfortunate aspect is the people who experience the brunt of this have no voice in challenging the authorities who establish the rules of this gamesmanship. And the only way to get employed and promoted is to play the game. Applicants are absolutely right to check the expert box because that’s the only way their resumeIt is a game that no one really wins.


  2. Helene Mikes says:

    You have definitely hit on part of the problem, and in a previous column you talked about the haphazard screening process of the results of these applications. However, there is a third component: the interview process. I’ve been a supervisor for 25 years, and we traditionally used interview questions that were open-ended, designed to elicit non-formulaic responses. If we got vague responses, we would ask follow-up questions. We would try to tailor questions to clarify portions of the applicant’s application that were not clear.

    Now I am running into interviews that are designed to avoid lawsuits, not learn about the applicants. The identical questions are asked of everyone, no follow-up questioning is allowed (lest you allow one person to explain better than another). Why bother with the interviews at all, if that’s how it is going to be?


Comments are closed.