Understanding your PMP Score

– Updated –

PMP, CAPM  and PMI-SP scores are all based on how well you answer multi-choice questions in your exam.

Unfortunately, the information provided to candidates at the end of their PMI exam is limited. You are told if you have passed or failed based on the number of questions correctly answered (but the passing score is a secret) and you are told how well you have performed in a general sense but little in the way of specific data, even with the vastly improved feedback rolled out in 2017. In reality, the only option is to be happy if you are in the 70% to 80% who pass and talk to your training provider if you have failed. You cannot get anything additional out of PMI.

During your training, the situation is (or should be) very different!

Almost every training organisation offers its PMI exam candidates sets of questions to help consolidate their learning – we build our courses around lots of regular tests.  If the tests are well designed they use the standard 4-option multi-choice question format used by PMI and provide feedback to the trainee on the number of questions correctly answered and detailed information about the answers to all of the questions. We do, and so does everyone else I talk to.

However, what many people don’t fully appreciate is the effect of the four options in the PMI multi-choice format and why a score of 55% is definitely in the ‘fail’ category! The cause is the fact that one of the options will always be correct, therefore a set of random guesses will score 25% on average, and the more questions you answer the more the 25% will come into play. The challenge is working out what proportion of the 55% can be attributed to ‘knowing the right answer’ and what proportion is likely to be consequence of the ‘lucky 25%’.

As a starting point, any score you achieve will have a certain number of questions answered correctly because you knew the correct answer and a certain number of questions that you answered incorrectly but fluked the right answer for the wrong reason (the ‘lucky 25%’ effect). One way to assess how many of each are included in your score is to assume a known number of answers are correct because you ‘know the right answer’ and then add 25% of the rest as likely to be correct based on the ‘lucky 25%’ factor. Therefore in a set of 100 questions, if you ‘know the right answers’ to 40 questions and add 25% of the remainder (60 / 4) = 15 you get a score of 55 (or 55%). So whilst superficially a score 55% may seem to be ‘not too bad’ knowing this represents a real score of around 40% gives a very different picture. And the reason increasing your score is quite hard work is because as you real score increases, the remaining number of questions subject to the ‘25% luck’ factor decreases

This chart is one way of assessing what you really know from any score achieved in a 4-option multi-choice question exam. If you don’t know the answer to any question and guess the lot, should still score around 25%. Conversely if you know the correct answer to every question, there are none left to be affected by the ‘25% luck’ factor.

In the chart, your total score is represented by the Pale Blue line and the questions you can assume you ‘know the correct answer to’ is represented by the Dark Blue line.

Between these two lines, there are a steadily decreasing number of questions that you did not know the right answer to but got a correct answer to based on the ‘25% luck’ factor. This 25% factor is represented by the Green Line.

To use the chart

See what your score is and convert it to a percentage. This is your ‘Total Score’ on the vertical axis, and is represented by the Light Blue line. Look down the chart to see your real score on the horizontal axis, and the Dark Blue and Green lines will give you an approximate split. For example, if you scored 62.5% in total, you probably know the answer to around 50% of the questions and the random factor added another 12.5%.

Taking into account the possible distribution of results, the ‘luck factor’ is probably somewhere between 10% and 15% (at ± 1SD) meaning there is a 65% probability you actually scored somewhere between 47.5% and 52.5% (but the effect in any specific test may be greater).

Limitations & Observations

This chart explains why PMI has set a passing score above 60% in the past (when we knew this information) and seem to have maintained this policy into the present time. A score of around 65% would assure PMI the candidate probably achieved a score of 50% or better from knowledge. But there are a number of issues with the chart.

• The first major problem is that on average there will be a ‘25% luck’ factor affecting the questions you don’t know the right answer to, but some people will be luckier than others.  Based on a series of Monte Carlo analyses by Jesus Carrillo we can assume 1 Standard Deviation is a bit less than 5 (at the starting point), therefore the ‘luck factor’ in around 65% of all tests will fall somewhere within the two dotted green lines (see comments below).
• The number of questions subject to the 25% luck factor decreases as the number of questions you answer correctly increases and because we are dealing with whole numbers this introduced some distortions (skew) into the assessment, as does the 3:1 ratio between wrong and right answers.  The chart is a simplification of quite complex data which hopefully does not limit its usefulness.
• The next major problem is very few questions will be answered randomly. Most trainers tell there candidates to make sure you answer every question, but the majority of questions will have a considered answer, some considered correctly and other considered incorrectly. This factor is impossible to graph.
• There is no way of knowing which specific questions were answered correctly for the ‘right reason’ and which were in the lucky 25%. You need to check the answers from you training organisation to see if you were ‘right for the right reason’.
• The last consideration is the potential to remove some variability by correctly deleting ‘obviously wrong’ answers. If you can remove one option the probability of your guess being correct rises to 33%, if you can knock out two options it is up to 50%. This shifts the odds into your favour but reduces the ‘known to be correct for the right reason’ value. Again impossible to graph.

Summary

Next time you test you take a practice test use this chart to asses your real level of knowledge but make sure you check all of the answers you got right so you can identify the questions you got right for the wrong reason, the ‘25% luck’ factor.

One of the reasons we set our trainees a benchmark score of 75% in our coursework is to help overcome the factors discussed in this post. By the time you are scoring around 75% the effect of the 25% random variability has reduced well below 10% and we know you know enough to score around 65% in the exam without the luck factor. This means most people will pass most of the time; score below 75% and you are starting to rely on the 25% luck factor and there is always a chance you may not have an average amount of luck on exam day.

To help you practice your PMP, CAPM  and PMI-SP answering skills we Tweet a new question every day, for today’s see: http://www.mosaicprojects.com.au/Training-PMP-Q-Today.html