Frequently Asked Questions
This page contains answers to questions about pilot selection, screening, and hiring that our associates frequently encounter. New information will be continuously added to the page. Questions that can be answered succinctly will be posted on this page. Questions that require more extensive answers will be listed on this page with a hyperlink to a file with the reply. Information has been posted here as a general service to the aviation community. Please feel free to download it. If you have a question you would like answered, please feel free to send it to us. We will then post the reply on this page.
-
A screening system usually serves to eliminate applicants who do not meet the minimum qualifications specified by the carrier and normally precedes the selection system. An applicant must complete the screening process successfully before beginning the selection process. For example, assume an air carrier indicates that an applicant must have 1200 total hours to be considered. Despite this specification, some pilots with less than 1200 hours apply. The screening system identifies these individuals immediately and eliminates them from further consideration.
-
IATA’s Guidance Material and Best Practices for Pilot Aptitude Testing (2010) is an excellent summary document that describes the standard approach to developing a pilot selection system. Pilots and Human Resource personnel who have limited formal exposure to selection system development will find this a useful document. However, it does have three major limitations. First, Guidance Material and Best Practices for Pilot Aptitude Testing was meant for an international audience. For this reason, it cannot address the development and use of a pilot selection system within the legal context of a specific country. Air carriers incorporated in countries with extensive legal issues surrounding selection and hiring, such as the United States and South Africa, will need to consult other sources. Second, the document does not describe any of the statistical methods needed to validate individual tests or the selection system as a whole. Validation is necessary to ensure that the selection system is working as expected, i.e. it is identifying the best candidates. It also ensures that the costs of developing and running the system can be justified. Much information is available on validation strategies for selection systems, including several recent books. Third, the underlying science is never really explained. The most significant lack concerns models of human abilities and skills. These models help identify the skill and abilities that should be assessed in a pilot selection system.
Damos Aviation Services, Inc. offers a 2-day course on pilot selection that can fill in the gaps of this IATA report. Please see the Pilot Selection Course under the “Short Courses and Workshops” menu.
Guidance Material and Best Practices for Pilot Aptitude Testing is available for free from IATA. It may be download from this page: http://www.iata.org/publications/Pages/pilot-testing.aspx
-
No. A selection system is different from a recruiting system. However, a good selection system may decrease the number of pilots who fail training or are dismissed during the probationary period. Because more pilots may be successful, fewer candidates must be recruited.
-
A selection system consists of five major elements. The first, and the most obvious, element is the selection instruments–such as written tests, computerized tests, simulator evaluations, and interviews. The second element, the procedures used to administer the instruments, is closely related to the first. These procedures include such things as the order in which the instruments are administered, the time allotted to complete the instrument, the number of people on the interview boards, the media used to present the instruments, and the selection model.
The third element is the job performance criterion. The criterion represents the behavior (s) the selection system is designed to predict. All selection systems must have a criterion that is explicit (clearly identified), comprehensive, and quantifiable. The criterion is arguably the most important element of the selection system and, interestingly, is also usually the most neglected.
The fourth element is the statistical techniques. These techniques are used to relate or correlate scores on the instruments to the criterion. Statistical techniques also are used to develop cut-off points and maximize the predictive utility of the instruments.
The feedback loop is the fifth element of the system. The purpose of the feedback loop is to monitor the predictive validity (the correlation between the scores on the instruments and the scores on the criterion) of the selection system. The predictive validity of all selection systems will decrease over time without intervention. This decrease may be caused by changes in the applicant pool, changes in the criterion, or compromising of the instruments. The feedback loop allows the company to determine when the predictive validity is decreasing and to take the appropriate countermeasures. In countries where employment litigation is likely, a feedback loop establishes “intent to improve” and provides an additional measure of protection against litigation.
-
In the United States, Title VII of the Civil Rights Act of 1964 prohibits discrimination based on race, ethnicity, national origin, religion, or gender. The Equal Opportunity Commission (EEOC) monitors the hiring practices of air carriers to ensure that they comply with Title VII. The EEOC uses several guidelines to assess hiring procedures and their outcomes. One of these guidelines is the “four-fifths (4/5)” or ” 80%” rule.
The four-fifths rule might be applied in the following way. Assume that 15% of the qualified white male pilot applicants are hired and 9% of qualified Hispanic male applicants are hired. To meet the four-fifths rule, at least 12% of the qualified Hispanic male applicants should have been hired (15% x 80%; .15 x .8 = .12 or 12%). Because the hiring rate for the protected class is less than four-fifths that of white males, the EEOC may decide that the hiring process creates “adverse impact.” Note that the rule applies to the percentage of qualified applicants, not to the percentage of the specific protected class in the population.
The EEOC may apply the four-fifths rule to each part of the hiring system as well as to the system as a whole. For example, let’s assume that an air carrier has a selection system consisting of a computerized aptitude test, a simulator evaluation, and an interview with a senior check airman. Also assume that 20% of the qualified white male applicants are hired and 18% of the qualified women are hired. At first glance, this system appears to have no adverse impact for women because they are hired at more than 80% of the rate at which men are hired. However, when we examine each of the three components of the selection system separately, we find that 30% of the white male applicants pass the simulator evaluation while only 20% of the female applicants pass. In this case, the EEOC may decide that the hiring process has adverse impact because women are passing this stage of the selection process at less than 80% of the rate of the white male applicants.
-
Most practitioners believe that the best way to detect bias in a test is to develop regression equations for the majority case (usually White males in the case of pilots) and for each minority in question (i.e. White females). Typically, test scores are regressed on some measure of performance (i.e., score on the final checkride at the end of training). The slope and the intercept of the equations are then compared. Differences in slope indicate that the test is not equally valid for all of the groups and should be dropped.
This question is addressed more completely in the document entitled “Legal Issues in the Design of Selection System.”
-
The time required depends on 1) whether the existing system is being revised or a new system developed 2) the management structure of the air carrier and 3) the number of pilots who will enter the selection system. Generally, a small carrier that is revising an existing system should allow at least 3 months for the revision and validation process. A large carrier can require up to 2 years. Some of the work should be done before the carrier begins hiring, so planning ahead is vital.
-
No. The structure, content, and development process of a selection system is affected by the legal requirements of the country, the language of country, and pilot experience. The legal requirements vary widely from country to country. This affects the types of questions that may be asked in the interview and the documentation describing the development and validation process. Nevertheless, all systems should be developed in accordance with internationally recognized standards.
Language causes other differences. Large numbers of written/computerized tests assessing many different types of knowledge, abilities, skills, and traits are available for English-speaking countries. Fewer tests are available for other languages. The lack of tests may have a direct effect on the structure of the selection system.
Pilot experience also affects the selection system. Most ab initio pilot selection systems do not include a simulator test, whereas the majority of experienced pilot selection systems do. The types of questions asked in an interview also vary between an ab initio selection system and an experienced pilot selection system.
-
The cost depends on at least four factors. The first factor is the size of the carrier. The greater the number of pilots currently flying and the greater the number of individuals involved in pilot hiring, generally the higher the cost. The second factor concerns the company’s record keeping. If the carrier has not retained information pertaining to previous hiring, the selection system may have to be built “from scratch.” If the company keeps few records of their pilots’ performance during training or during evaluation, performance measures may have to be developed and new databases constructed. This requires more time and usually more money. The third factor is the Human Resources Department. If the Human Resources Department does not use structured interviewing techniques, they must receive additional training in interview techniques and scales must be developed. Again, interview training and scale development increase the cost of selection system development.
The final factor, and the most frequently overlooked, is the legal environment. Companies operating in countries like the United States and South Africa must have extensive records of selection system development and validation. Typically, three to four large reports will be generated during the development of the selection system. These reports document the development process and the outcomes of each stage. Although these reports add to the cost of system development, they provide the foundation for the company’s defense against legal actions and are well worth the associated cost.
-
An air carrier should assess the flying skills of an experienced pilot who is applying for a position. However, obtaining reliable, sensitive, and valid skill assessments is surprisingly difficult. One of the problems concerns “inter-rater reliability,” which refers to how well two or more raters agree when they observe and rate the same behaviors. Recent research has demonstrated that inter-rater reliability is surprisingly poor even among experienced check airmen. Low inter-rater reliability leads to serious questions about what is being assessed and is a source of legal vulnerability in countries like the United States and South Africa.
Objective measurement of basic flight parameters–such as airspeed, heading, and altitude–may help improve inter-rater reliability by allowing the raters to concentrate more on some of the “softer” aspects of performance. Interested parties may want to examine the scoring system described on the products page of Frasca International, www.frasca.com.
-
Small- to medium-sized carriers should start the process about 3 to 4 months before they plan to begin hiring. Larger carriers need somewhat more time and should plan on starting the process about 6 months before the actual hiring begins. Although selection systems can sometimes be developed in less time, starting the development process several months before hiring begins reduces the stress on the personnel directly involved in hiring. It also allows management sufficient time to identify appropriate selection tests and negotiate user fees or purchases. Additionally, this lead time allows management sufficient time to train any employees who are new to selection and provide any training required to administer new tests.
-
We have encountered this question frequently. The use of a selection system in civilian ab initio training depends on the purpose of the training. If the training is part of a cadet program that is run for or by an air carrier, then selection may be appropriate and should be conducted in consultation with the sponsoring air carrier. If the training is self-funded and the student has some general interest in flying as a career, then testing for the purposes of career counseling may be appropriate.
-
A “best practices” system is one that conforms to standards set by relevant professional organizations and conforms to all legal standards. It is different from “common practice.” We have found that many air carriers compare their selection system to other carriers. Unfortunately, many air carriers have pilot selection systems that do not meet best practices standards. Comparison to other carriers, therefore, may be misleading.
-
Three major types of benefits may occur from the use of a “best-practices” system. First, by definition, using a best-practices system reduces the company’s legal exposure.
Second, a best-practices system usually reduces selection and training costs. Each selection test in a best-practices system (including the interviews) must be validated. Validation ensures that the test actually helps identify the best applicants and reduces unnecessary selection expenses. In a best-practices system, success in training usually becomes an explicit goal of the selection system. As a result, the percentage of new hires who successfully complete training and probation increases. Every new hire who completes training and probation translates into one less person to be recruited and trained.
The third benefit concerns the quality of the new hires. To set up a best-practices system, the company must define clearly what it means by a “good pilot.” The selection system then is designed specifically to identify those applicants who have the highest probability of meeting those definitions. This improves the overall quality of the new hires.
-
In aviation, “validity” usually refers to the relation between a set of test scores and scores on a criterion measure. For example, assume that a carrier decides to purchase a personality test to use as part of its selection system. Also assume that the selection system is designed to predict scores on the check ride at the end of training. The carrier must determine the relation between scores on the personality test and scores on the check ride before the scores on the personality test can be used to make hiring decisions. If the carrier finds that the relation (correlation) between scores on the personality test and scores on the check ride is low, then the personality test predicts check ride performance poorly and should be discontinued. In this case, the personality test is said to have low validity. If, however, the test scores correlate highly with performance on the check ride, then the personality test is said to have high validity and may be used in hiring decisions.
-
Buying pilot screening tests can sometimes be time-consuming and frustrating. Selection tests in the United States are rated as A-level, B-level, and C-level. The pilot screening tests are C-level tests and B-level tests. To buy these tests, one person in your company must be designated as the responsible party. This person must have 1) advanced training in tests and measurements and an advanced degree in an associated field, 2) prior supervised testing experience or 3) completed a recognized testing certification course. This person will have to agree to certain test secruity requirements and take responsibility for ensuring that the test is not reproduced. The responsible party needs to be the person overseeing the test use. It is not necessarily the person actually administering the test.
Buying a pilot screening test may require several steps. The responsible party must contact the vendor and provide background information. Usually the responsible party must complete a form and may be asked to submit a request on compoany letterhead.
Remember–selection tests often require years of development and a large investment of money before the developer can offer the test for sale. The developers and vendors simply are trying to ensure that the tests are purchased by responsible individuals who will use the test correctly.