Standardized Test Battery and FCE "Systems"
Posted on Wed, Sep 14, 2011 @ 03:23 PM
Question
“I am a DPT and have been an independent provider of FCE, job analysis, and safety consulting for the past 10 years. I was trained in Isernhagen 13 years ago, but I have abandoned the name since I don't really use Isernhagen now. Working primarily in Voc Rehab, I have developed an approach that deviates from this standardized format.
I don't identify my FCE with any particular brand or producer of standardized tests. I also have a Masters degree in workplace environmental health and safety affairs where I add a section on safe environmental exposure estimates (and other areas) in my FCE, as derived from my insight from coursework in industrial hygiene and safety engineering. I have also been teaching job analysis for Human Resource Recruitment and Selection program development in the Psychology department at a local university.
In performance assessment and measurement, the use of a standardized test battery is not advocated by the APA (American Psychology Association) and others in measurement, because this leads to adverse impact - especially where employment decisions may be made. It is contentious that the data gathered and results of these tests should only be applied to the group (population) to which the test was standardized on. Given the great heterogeneity and diversity of the disability population, what is your position on this?”
Answer
As way of introduction to my writing style, let me state that I tend to be very careful about my use of words when I know their intent impacts a technical discussion. As work performance in the United States is now very much a legal issue, I will comment on seemingly small points so as to be clear with readers who may not understand the impact of these words. No disrespect to your background is intended.
Having said this, standardized tests do not necessarily lead to adverse impact. It all depends on the application to the job and the individual. Instead of saying they lead to adverse impact I would say they may lead to adverse impact for the individual seeking employment. A small point, but important for others that may not have your background.
To your questions: anti-work discrimination law in the United States is very clear about the need for employers to do two things:
1.) Monitor their hiring and transfer actions in order to reveal disparate impact. (The guidelines for this are laid out in the Uniform Guidelines on Employee Selections Procedures. The importance of the APA and Industrial and Organizational Psychologists [Division 14 of the APA] figure greatly in this document.) This begins with the first occurrence of adverse impact. (Adverse impact meaning an applicant was not hired or an internal transfer, often referred to as a ‘candidate’, was rejected). Assuming an employer has a large enough work force to qualify for monitoring, the employer needs to keep track of each person that applies for a position whether that person is hired or not. When analyzing disparate impact it is important to “calibrate the barometer” correctly by including those that self-select out of the process. To not do this places the employer at a statistical disadvantage when calculating disparate impact.
2.) Employers must also monitor their testing instruments (paper and pencil tests, physical tests) and practices (interviews, written applications) for adverse impact and eventual disparate impact. The key to this prong of an employer’s responsibility is job-relatedness and test validity.
Let’s look more deeply into this prong as it greatly impacts how physical therapists practice today:
It sounds like you are very familiar with psychometric instruments. As that is an area in which I am not strong I will leave that one alone and focus on the area of physical testing. My brother, Leonard, would be the one to discuss issues related to the field of human performance testing from the point of view of the APA as well as physical testing. I can put you in touch with him should you desire. (www.epicrehab.com).
Recent state and federal rulings in cases involving both Functional Capacity Evaluation and Post-Offer testing are the first step in setting a legal standard for testing. Until now FCEs in the U.S. were not challenged in any area except their “readability”. No one looked under the covers to determine if an FCE was based on any form of science or structure. The activity in the federal courts is already impacting FCE vendors: at least three, in my opinion, have gone out of business or have been sold because they would not withstand the scrutiny of an informed legal body. For the first time in my 20 plus years in this area of interest, the courts are finally separating good practice from practice for the sake of selling an FCE system.
Having made the above statement, I will now answer the second part of your question which is restated here:
“Do you recommend that I adopt a commercially available standardized methodology (name such as KEY or Blankenship) even though I deviate from these approaches? Do you believe that standardization is what gives FCE credibility in the eyes of the courts?”
No, you should not adopt a system, particularly those mentioned above (including Isernhagen, which you mentioned earlier). The recent scrutiny of the courts has not been friendly to “FCE systems” or, for that matter, to physicians using instruments that do not speak to Federal Rule of Evidence 702. (The Matheson “system” would also not do well if the evaluator did not stay true to our credo of “The Thinking Evaluator”.) The courts very clearly want evaluators who can serve as expert witnesses assisting the trier-of-fact to understand the truth of the case.
My comment is backed by a series of federal cases: Gillen v Fallon Ambulance service is a case wherein two physicians did not follow protocol when dealing with an applicant. They made decisions about her status of being disabled (“considered as”) and refused to allow her to test for the job. This case points out that it is important to understand ADA and the new ADAAA law as it regards protocol.
Next, in Indergard vGeorgia-Pacific an occupational therapist and a physical therapist conducted a job analysis and then, to the best of my knowledge, administered a two-day Isernhagen FCE. The incumbent (candidate) sought a right to sue under the ADA contending adverse impact, use of an invalid job analysis (see Uniform Guidelineson Employee Selections Procedures for guidelines on job analysis) and use of an invalid (not job-related) FCE protocol. After trial in the 9th district Court of Appeals, her case was remanded back to the lower court for consideration of a $250,000 settlement. Without knowing the details of the final settlement, I estimate that the actions of the two therapists cost Georgia-Pacific about $400,000.
This case was the wake-up call for FCE practitioners. Any evaluator using a standardized test battery, as opposed to a battery of standardized tests, could have walked into the validity trap. And “interview” job analysis techniques are simply not acceptable to the courts or to the Social Security Administration (see the Occupational Information Development Advisory Panel’s Findings and Report: A Review of theNational Academy of Sciences Report - A Database for a Changing Economy: Review of the Occupational Information Network (O*NET)).
A Functional Capacity Evaluation is NOT a "Checklist"!
If you want to read a positive opinion about what the courts are looking for in an FCE and a job analysis, read Paul James v. Goodyear Tire and Rubber.
The next point I want to make is based on your skills as an evaluator versus the mechanism of an FCE system: Baker v. Jo Anne Barnett, Commissioner of Social Security ripped the use of Blankenship’s Validity Profile. I have seen this repeated on two other occasions in the British Columbia Supreme Court.
The obvious point of Baker is Rule 702, Testimony by Experts which states,
“If scientific, technical, or other specialized knowledge will assist the trier of fact to understand the evidence or to determine a fact in issue, a witness qualified as an expert by knowledge, skill, experience, training, or education may testify thereto in the form of an opinion or otherwise, if (1) the testimony is based upon sufficient facts or data, (2) the testimony is the product of reliable principles and methods and (3) the witness has applied the principles and methods reliably to the facts of the case.”
This is followed by Rule 703, Bases of Opinion Testimony by Experts which says,
“The facts or data in the particular case upon which an expert bases an opinion or inference may be those perceived by or made known to the expert at or before the hearing. If of a type reasonably relied upon by experts in the particular field in forming opinions or inferences upon the subject, the facts or data need not be admissible in evidence in order for the opinion or inference to be admitted.”
The issue here is that the methods used in the Validity Profile are not accepted by peers in the field. Further, the method behind the profile has not been published in a way that is open to peer review. (A key point of liability in the trial occurred when the therapist stated, in effect, that the material had been published. He then corrected himself to say that it had been submitted for publication. His jeopardy here is that if it has been submitted for publication there would be a paper trail of submission. If he can’t produce that trail then he has a professional credibility problem as in the voir dire of the evaluator in the Supreme Court of British Columbia case of Forstved v Penner (2009 BCSC 1625).
All of this points to your professional background and the preparation you have done to become an evaluator (in every sense of the term). Your thought process, coupled with your skills, are what the court seeks.
One of the most important aspects of your future is to stay current with the court’s interpretation of what comprises a valid evaluation.
Stay in touch with us on: