UK: Testing the tests - PSYCHOLOGICAL TESTING.

UK: Testing the tests - PSYCHOLOGICAL TESTING. - How can you tell the good psychological test from the less good - or the positively harmful? Malcolm Starkey has some of the answers.

by
Last Updated: 31 Aug 2010

How can you tell the good psychological test from the less good - or the positively harmful? Malcolm Starkey has some of the answers.

On a typical week day a little over a year ago one of Britain's heavier national newspapers carried some 270 job advertisements. (It was, remember, more than a year ago.). In addition to qualifications and experience requirements the majority of these ads specified that applicants should also possess certain more personal attributes.

As might be expected, most of the advertisers were looking for ambitious, dynamic and competitive people with good leadership and interpersonal skills. A single brave employer struck a markedly different note: He wanted someone with a "quiet, unassuming manner". But, in all, 168 of the ads made some reference to personality traits. Sixty-six of them mentioned other personal characteristics as well, as such as intellectual ability.

Personal factors are evidently taken very seriously by recruiters. It doesn't really need an analysis of the appointments pages to reach that conclusion, although the figures certainly help to make the point. But the question then arises: how good are employers at measuring the personal traits of their employees - either actual or potential?

Although estimates vary, it's very likely that around 80% of the bigger businesses in the Uk - represented by The Times 1000 - now use psychological testing for one purpose or another. Only half-a-dozen years ago the incidence would have been more like 40%. Clearly, this is a growth sector to which big companies, in particular, are increasingly committed. Peter Horncastle, of Austin Knight Consulting which acts for some of the best known businesses in the country, says that "clients often cite instances of where they made hasty or ill-conceived decisions about people to the detriment of their organisations ... They see testing as a reasonably low-cost option to help them prevent similar mistakes in the future."

Almost everyone, these days, is familiar with the use of psychological testing in personnel selection. Dr Mike Smith and his team at the University of Manchester Institute of Science and Technology have shown that proper use of the technique can easily save 15% to 25% of the salary related to a job. However, tests can equally well be used in a whole host of other situations where managers need objective information on which to base judgments about people. Examples include management development advice, the identification of training needs and of management potential, succession planning, the composition of teams, redemployment, and redundancy selection.

Horncastle believes that the benefits of testing are, basically, four-fold: (1) tests provide objective information about a candidate which (2) is not accessible via the traditional interview; (3) that tests are highly cost-effective and (4) that their use makes a statement about the professionalism of the recruitment process. Nevertheless, there are good tests and poor tests, and even the good ones are vulnerable to poor interpretation. Whether one is considering tests or testers, the trick is to distinguish the good from the less good - and from the positively harmful.

The twin characteristics of a good test are that it is (comparatively) valid and (comparatively) reliable. A valid test is one which actually measures what it purports to measure. Some don't. For example, one questionnaire was developed for the simple purpose of comparing people's attitudes before and after training. However, the authors decided that it looked like a personality inventory, and they started to sell it as such. Psychological tests are susceptible to this kind of deception because it is almost impossible to tell a good test from a bad one just by looking at it. After all, anyone can compile a list of questions and call it a psychological test.

So how do you recognise a poor test? An independent review of a particular (deductive reasoning) test offers a clue: "No data relating the test scores to performance are reported in the test manual. There is no indication that ... high scorers on this test did better than low scorers on anything other than taking the test." The key point is that there must be a correlation between what the test measures and the skill or trait that it sets out to sample.

The validity of a test is indicated by its "validity coefficient" and this can be measured. For selection work, it is likely to be of the order of 0.4 to 0.55. Although these figures are low in absolute terms, they are still some three times better than is likely to be achieved by interviews alone. Validity information should always be given in the manual which accompanies a test. If it does not appear, or if it's given cursory treatment, the test should be avoided.

The second main check on the worth of a test is for reliability. Since assessments based on psychological testing obviously carry a considerable weight of authority, a test which produces unreliable results is much worse than useless. The way to check reliability is to see what degree of consistency exists between the results turned in by the same individuals on different occasions. It is measured by a "test-re-test correlation". This should also be quoted in the manual, and should be 0.7 or better.

Testers pose different problems altogether. Most psychological tests are based on the premise that if several people are asked the same set of relevant questions under consistent conditions then their answers will reflect differences in individualability, aptitude, personality, etc. Consequently, it's vital that the test must be administered in the same way each time. Consistency of administration, however, is not usually a problem. Interpreting the results is a different matter.

Competence in test interpretation is roughly proportional to length of experience times breadth of familiarity with different types of test in a range of situations. Yet people without any qualifications or relevant experience can - and do - offer testing services. Similarly, companies eager to get "into" psychological testing sometimes send personnel officers away on short course in the expectation that they will acquire, almost overnight, the ability to probe the deepest and most private corners of a person's psychological make-up. Since recommendations based on psychological tests can affect the rate - and even direction - of career development, the need for high levels of skill and integrity can hardly be overstated.

Poor interpretation can be difficult to recognise, as the following experiment shows. A group of personnel managers was asked to complete a questionnaire. A few days later, they were all sent written reports ostensibly describing their individual personalities. When, subsequently, each manager was asked to make an assessment of his own report, most replied that it was either "accurate" or "very accurate". The researcher then revealed that all the managers taking part in the exercise had been given precisely the same profile. The reason why they had all be able to recognise themselves was that all the reports contained statements like "you sometimes don't feel as confident as you like to appear" - which few people would deny. If these managers were not able to distinguish their own personalities from a collection of plausible - sounding phrases, what chance is there for someone trying to reach a judgment about a comparative stranger? The answer is not much. But there are certain pointers to watch out for.

The first safeguard is that reputable test are supplied by reputable publishers. Organisations such as ASE/The NFER-Nelson Publishing Company, Saville and Holdsworth, The Test Agency, Science Research Associates, the Psychological Corporation and a few more, can be counted upon for their commitment to high standards. For many years, they have voluntarily restricted their sales by adhering to guidelines laid down the British Psychological Society (BPS). These require that test materials should be sold only to properly qualified individuals. Unfortunately, it doesn't follow that anyone with a reputable test in his or her possession is properly qualified. Nor does it follow that the only good tests are those published by established houses.

Most of the publishers listed above supply test from a variety of authors. More recently, some organisations have started to write their own tests, selling them direct to users. Such publishers do not always stick to the BPS guidelines (Saville and Holdsworth is an honourable exception here), and seem to pay more attention to marketing and selling than to the construction.

These suppliers often employ fairly hard-selling techniques and frequently make conspicuous use of computers to convey a high-tech image. They are also apt to claim that the test requires little or no training and/or that it can assess an individual in as little as 10 minutes. The suppliers may further imply that the test offers a universal solution, without reference to the requirements of different applications.

Once again, most of the information which indicates that a test has been soundly constructed will be found in the manual. If there is no accompanying manual, or if attempts to obtain a copy are discouraged, this probably means that the test has been inadequately researched.

The manual accompanying any good test will contain a full explanation of how it should be administered. There will also be sections on its development and construction. It will say when the test was first developed, and when it was last up-dated. That a test has been around for a long time may well be a measure of its worth, but it should have been updated within the last few years to exploit advances in psychological and statistical knowledge. The section on construction should show how the test dimensions were carried at and how the questions relate to them. There should be evidence of large-scale trials and subsequent refinement. The statistical methods used should also be covered in some detail. Tables showing normative data allow the relative strength of a characteristic to be assessed, and are of vital importance. However, norms should be relevant. The user may have to ask whether female sophomores in a mid-western state university really are a suitable yardstick against which to assess a British production manager.

Research should have been carried out to assess possible race or sex bias. London Underground recently found itself facing a charge of race discrimination for using selection tests, which, it was claimed, assessed higher level skills than those needed for a particular job. The general view is that if proper testing procedures are followed, and the test can be shown to be predictive of suitability for the job, then no problem should arise. Incidentally, the Equal Opportunities Commission advises that male and female applicants should both be assessed using the same norms.

Finally, there will ideally be correlations with similar dimensions in other tests. These will enable a judgment to be made on the worth of anew test as compared with an established instrument.

As already mentioned, anyone can offer psychological services. Indeed, anyone can call himself a psychologist. How, then, is the business user - or the public - to know whether or not someone who claims to be a psychologist is actually trained and qualified? A few years ago the BPS produced a register of psychologists whose qualifications and training satisfied strict criteria. Those limited are entitled to call themselves Chartered Psychologist and use the letters C Psycol after their names. They agree to be bound by a code of conduct, and can have their names removed from the register by a disciplinary committee (on which non-psyhologists are in the majority) if they contravene its provisions. Copies of the Register are available for inspection at main libraries, or can be obtained from the BPS. Alternatively, the potential client may ask to see a current practising certificate, which all active chartered psychologist should have.

The introduction of a Certificate of Competence in Occupation testing was another BPS initiative. With this document, the BPS recognised that psychological testing at work raises particular issue of ethics and standards - and requires specific skills which are not necessarily common to all types of psychological testing. It should be mentioned in passing that not all chartered psychologist are competence to conduct test in the work context. For many people, a psychologist is a psychologist. But training as a clinical psychologist no more prepares a person to interpret selection test (for example) than proficiency as an occupational psychologist qualifies someone to treat mental illness.

Find this article useful?

Get more great articles like this in your inbox every lunchtime

Subscribe

Get your essential reading delivered. Subscribe to Management Today