An air carrier should assess the flying skills of an experienced pilot who is applying for a position. However, obtaining reliable, sensitive, and valid skill assessments is surprisingly difficult. One of the problems concerns “inter-rater reliability,” which refers to how well two or more raters agree when they observe and rate the same behaviors. Recent research has demonstrated that inter-rater reliability is surprisingly poor even among experienced check airmen. Low inter-rater reliability leads to serious questions about what is being assessed and is a source of legal vulnerability in countries like the United States and South Africa.

Objective measurement of basic flight parameters–such as airspeed, heading, and altitude–may help improve inter-rater reliability by allowing the raters to concentrate more on some of the “softer” aspects of performance. Interested parties may want to examine the scoring system described on the products page of Frasca International,  The scoring system is described as part of PASS.