Interested in what the utility of psychological measurements might be outside the lab It's interesting to consider that from an ethical perspective too, so if you're hiring people you have two conundrums that lay themselves in front of you and one is the Ethical necessity to give each person a fair chance and the other is the ethical necessity to place the proper person in the proper position, and you might say well You could do that randomly and like there are countries in Europe for example that have a quasi random approach To the selection of university students, so Holland is like that they have a pretty Practical side I'm often interested in what the utility of psychological measurements might be outside the lab Practical side I'm often interested in what the utility of psychological measurements might be outside the lab It's interesting to consider that from an ethical perspective too, so if you're hiring people you have right two conundrums that lay themselves in front of you and one is the Ethical necessity to give each person a fair chance and the other is the ethical necessity to place the proper person in the proper position and you might say well you could do that randomly and Like there are countries in Europe for example that have a quasi random approach to the selection of university students So Holland is like that they have a pretty Open admission policy now what that means is that everyone has a chance to go to college But the downside of it is that the failure rate in the first year for example is extraordinarily high Now you might say that's a perfectly reasonable price to pay for the open admission and for the opportunity to give everyone a chance But you could also say well That's a hell of a waste of time for the people who go into the first year of college or university and fail It's not a pleasant experience for them It's it's very expensive in terms of times and time and resources and perhaps it would have been better for them and for broader society had they'd been able to determine beforehand whether they had the sets of qualities that were necessary to Increase the probability of success because if I could tell you well You know you have a 80% chance of success in this domain and only a 20% chance in this domain You still might want to take the risk in the 20% success domain, but you might also think well I might as well go off and function where my particular combination of proclivities has the best opportunity to manifest itself and and Because why not position yourself for success rather than failure now. I'm not saying that the ethical Conundrum between those two alternatives is something that's easy to What would you say an easy thing to map your way through It's by no means an easy way to map your way through because there's our strong arguments to be had on both sides of the equation so but There's still the scientific question that remains which is to what degree can you accurately predict people's performance and to what degree does that Reflect positively or negatively on your psychological concepts, and then there's the actual practical utility of Potentially offering to schools to universities and to workplaces in general the probability of selecting People with an above-average chance of succeeding It's even more complicated say if you're selecting not so much entry-level employees in a company Because maybe you would tilt yourself there more towards The possibility of bringing more people in and letting them fail or succeed in the job, but let's say you're Replacing a management team at a medium to large size corporation You know if you bring in managers that are incompetent not only are they going to fail which is obviously not very good for them And the failure rate among managers is very high it exceeds it exceeds 65 They figure I think it's 65 percent of managers. I believe that's correct add Zero or negative net value to the company's it's something like that the failure rate in managerial positions It's overwhelming and the problem with bringing someone into a managerial or an executive position even worse Who isn't competent to play that role is that they can wipe out the careers of everyone that they're supervising and in the case of say large companies they can bring the whole damn company down and so it's not like the the the ethic of People deserve an equal chance Let's say to fail and succeed isn't a very practical ethic when you're putting someone in a high Demand position where the consequences of failure are can be overwhelming not only for that person But for all the people that they happen to be Whose destiny they happen to be involved in determining and so then you might say If you could come up with a selection process that would increase the probability of hiring an above-average Manager from 50 50 to 60 40 or 70 30 Maybe you're actually ethically compelled to use that and in fact by law. This is particularly the case in the United States You're required by law to use the most effective valid reliable and non-discriminatory Selection process that currently exists in order to select your employees and one of the things that's going to happen to employers in the next Ten years is they're going to get a very nasty shock for using interviews because the data on unstructured interviews indicates quite clearly that a that they're discriminatory partly because they they for example if you're tall and Assertive and good-looking and charming then you're much more likely to do well in an interview but that doesn't necessarily have any bearing whatsoever on the probability that you'll succeed in the position and so And not only that the the predictive validity of unstructured interviews is very low. It's about 0.12 It's something like that, which means that if you just pick people randomly for success or failure Let's say you'd have a 50 50 chance of predicting whether someone was going to be a success or a failure It's just a coin toss and then if you used an unstructured interview, you'd get that up to 56 44, which is slightly better But it's by no means It's not even close to the accuracy that you could get for example If you just used a standard test of conscientiousness, which would give you a correlation of about 0.25 with with say managerial With managerial productivity and there's this interesting. This is an interesting thing to know. It's called the binomial effect size display Effect size is the magnitude of an effect, right? and and it's not an easy thing to get a handle on all that you really need to if you're going to be a Psychologist because in any study there's an effect size indicator correlation off often or an R squared Which is the correlation squared or a cones D? Which is the effect size expressed in standard deviations or something like that? but you kind of have to understand that at a basic level to understand what statistics actually do and there's this phenomena called the Binomial effect size display that can help you understand What like in an embodied sense what the magnitude of a correlation means? So here's how it works Imagine that you have a predictor of 0.20. So the correlation is R equals 0.20 between Phenomena one will say conscientiousness and phenomena to workplace performance 0.20 correlation. The question might be well how much How much would you improve your predictive capacity over chance levels? If you applied that predictor and the answer is that the R is the difference between the odds ratio So, let me explain that so point five zero point five zero if you subtract one from the another you get zero So the predictive validity of selection by chance is zero point five zero minus point five zero equals zero that's the predictive validity of chance if you have a Predictor of point two zero, which is approximately that's sort of a low-end estimate for conscientiousness Then that would change your odds ratio from point five zero point five zero Right random to point six zero point four zero because point six zero minus point four zero is point two zero and so the correlation coefficient turns out to be the Difference between the odds between the odds so and so it gives you a quick rule of thumb So for example, so if you have a point two zero predictor that gives you sixty forty if you have a point three zero predictor that gives you sixty five thirty five because point six five minus point three five is Point three zero and if you have a point six predictor, which is really up on the high end, right? You're really starting to push your the limits of statistical predict prediction validity at that point That gives you point eight zero minus point two zero. And so what you've done if you use a predictive Predictor that has a correlation coefficient of point six zero, which you could get for example, if you took conscientiousness and Combine that with a good test of IQ for predicting complex jobs You might be able to get up to point six that moves your odds ratio of selecting an above-average Person for the position from point five zero point five zero to point eight zero point two zero So it cuts your your failure rate by more than half right brings it down from point five zero to point two zero Because point eight zero minus point two zero is point six zero. So that's a really good thing to know That's called the binomial effect size display. It's really good thing to have in your mind. It's very simple it's just a it's just subtraction and it gives you some sense of the power of of Statistical prediction now the question might be well, let's say you had a predictor of point two zero the conscientiousness You might say well if you square the R that gives you four percent of the variance who the hell cares four percent of the variance you've left ninety five percent of the Variability between people in terms of their performance unexplained. You might say well, why even bother? Well, the answer to that question is how much difference in productive output is there between people because if there's a tremendous degree of productive of Difference in productive output between people then increasing your ability to predict someone's performance even by some Relatively small increment might have massive economic utility, you know If let's say the top 10% of your people are 50 times as productive as the bottom 10% of your people then shifting your ability to predict up so that you have more of those Extremely high performing people or less of the extremely low performing people might more than pay off Might more than pay for itself from an economic perspective Even though your prediction your predictor isn't doesn't have that massive amount of power Well, and that actually happens to be the case. So back in 1968, there was a guy named Walter Mischel and he had reviewed He's a social psychologist he reviewed the personality literature up to that point and concluded that the typical personality measure only predicted the Typical performance measure at about point two and that's actually remained relatively stable I would say it's a little higher than that's probably point two five Especially if you do things like correct for measurement error and so forth and what Mischel said was because it's only point two five Let's say you square that that's five percent of the various you leave 95% of the phenomena unexplained you might as well not even bother measuring personality and so that actually killed the field of personality from a Psychometric perspective for about 25 years really until about the early 1990s when people woke up and thought wait a minute What are the typical effect sizes in other domains of prediction and then they found out that well the point two zero Correlation that was typical of personality prediction was actually pretty damn good by social sciences or health sciences standards Like it doesn't sound good when you just think about it as a absolute measure Because it leaves 95% of the phenomena unexplained But when you compare it to other things that people consider of reasonable magnitude, then it turns out that personality Psychologists are doing just fine. And then also in the 1990s and I'll show you some of this their economic calculations done And so one of the calculations would be well imagine that you have maybe you took 20 companies and you did a Distribution of the productivity of their employees. It's a hard thing to do because you have to measure their productivity It's like how the hell do you do that right with salespeople? You can measure sales. That's pretty straightforward with lawyers You can measure hours billed like there are some there are some occupations where the performance measure is sort of built into the job But if you're a manager in the mid level of a large corporation, how the hell can you tell how productive you are? it's it's so there's a measurement problem on the Productivity measurement and as well as the performance prediction end and it's a very intractable problem And the way that people often do that is by saying well, let's say you're a manager in the mid level of a corporation How do we determine how productive we are you are? Well, we might ask you to compare your your work productivity with your peers Maybe construct up a questionnaire asking about your efficient use of time and so forth And then we might get your peers to do the same thing to you we might get your supervisors to do the same thing to you and we might get your your subordinates to do the same thing to you and then aggregate across all those measures and Infer that that aggregate Opinion actually constitutes a valid measure of productivity You actually don't know right because that assumes that what you're doing that your peers and your supervisors and so forth Our rating is actually Related in some positive manner to the bottom line of the company and you actually don't you actually can't figure that out This is actually I think why large companies start to become unstable is because if there's enough layers between the operations of the people in the tiers of the corporation and The real outcome measure which is basically profit because that's what we've got Then the the relationship between your activity as a manager and the productivity of the company starts to become Increasingly blurred and that might mean that you're working as hard as you can on something that's actually going to cost the company money So you would actually be much more productive From a profit perspective if you just didn't go to work at all and that that happens a lot in large corporations because you'd never know Especially if there's a lot of steps that have to be undertaken in a process Before you can test the product in the market. You have no idea if you're wasting time and resources You just can't tell so the performance measurement issue is a very very complicated one We haven't talked about it that much but I give you a kind of a brief overview of it now what you really want to do is have multiple sources of Information about performance and aggregate across them and if you can use real-world measures that are tied to to income generation So much the better because you have to use something as your gold standard, right? You have to say at some point Well, we're going to define this as reality when it comes to performance and in a free market economy Roughly what you do there is you say that what profit is the proxy for productivity? And that isn't the same thing saying is that it isn't the same thing as saying that profit is Productivity that's not the same thing It's saying that at some point you have to decide what you're going to accept as a measure of productivity Because otherwise there's no point in even talking about it and you you can't just not talk about productivity if you're running an organization Because the organization doesn't exist unless it produces something that will keep it going and generally that happens to be money So anyways, it's quite it's very complicated all of this. I was also curious Because because I'm curious I guess is to find out What would happen if I took a measure that was derived in the lab and then tried to launch it out in a in the? Actual real-world environment tried to market and sell it and now that was very informative because I presume that we develop tests Which I'll talk to you about they were actually pretty good at predicting performance managerial performance, for example administrative performance We got ours of upwards of 0.6, which is you know, really bloody impressive So we could we could tell employers look if you use our tests we can increase the probability that you'll hire an above-average employee from 50 50 to 80 20 and the economic benefit of that will be staggering Staggering and I'll show you the calculations that enable that sort of that sort of prediction to be made and you might think well And this is what you do think if you're naive about producing something of value you might think well if you can produce something That's of self-evident economic value selling it will be a snap and that is so wrong You just cannot believe it. So one of the things we found which was really mind-boggling to me was that You could make a case that the probability that a company will use a test that predicts performance the Probability that they will use the test is inversely related to the accuracy of the test Which basically means that the less accurate tests are easier to sell and you think well Why the hell would that be why in the how in the world would it possibly be? the corporations would rather buy tests that don't work than tests that do work and That is what they do because really what they do buy is the Myers-Briggs Right that sells about a million units a year and the Myers-Briggs has zero predictive utility with regards to performance prediction So why do people use it? Oh, here's one reason it doesn't hurt anybody's feelings Everybody wins right and so then you think well do corporations really care Whether or not everybody wins when they're being tested and the answer to that is yes much more often than you would think so So we hit all sorts of barriers that was one The problem with tests that work is that most of the people who take them don't do very well on them And then the other problem is is that people have people aren't good at statistical reasoning at all They're really really bad at it. And so for example, they don't know the difference between a percentage and a percentile So a percentage is you know, if you get 40% on a test It means you've got 40% of the questions, right? If you if you are at the 40th percentile in the distribution of test scores It means that you perform better than 40% of the people that's actually not too bad, right? But you'll think no that's not 40 percentile That's 40% and then you'll think that you failed and so one of the things we found for example Was that when we were when we were marketing the tests to mid-level managers who had? Some say at least on whether or not they would be used the first thing they would say is well I want to do the test and The thing you say about that is no you don't because this is derived statistically You can't validate the test on the basis of your opinion about its applicability in your case But you can't have that conversation that isn't going to go anywhere because they say well I'd never give a test to my employees that I hadn't taken myself. It's like okay. So then you think well You're a typical manager. You're gonna score at the 50th percentile you are not going to be happy about that because you want to score at the 90th percentile because you confuse percentiles and percentages and also because You don't notice that if you're doing better than 50% of the managers, that's actually pretty damn good question