Inside the Q

Intrepreting Your Score Report

Your score report can provide good diagnostic information for you if you take the time to understand it thoroughly. Everyone always wants more explanation about why they failed a section or a particular exercise on the exam and what they can do to remedy the situation. The answer is actually right in front of you.  

Remember, unlike exams from college, we are not testing comprehensive recall of knowledge; we are assessing your entry-level competence to protect the public.

To fully understand what your score report is telling you, you should first read the blog posts about

• Scoring the Multiple-Choice Exams
• Grading the Practicum Exams

MULTIPLE CHOICE

On the score report we mail you for Sections 1 and 2 (the multiple-choice sections), the percentage of questions answered correctly in each content area is provided. As we’ve mentioned before, each multiple-choice exam contains 25 experimental questions; those are not included in the scoring. Scores are reported on a range from 200 to 800, with the passing point anchored at 500. Because the scores are based on sophisticated psychometric principles, it isn’t as easy as saying “500/800 = 62.5, so I need to get 63% of the questions right to pass.”

Let’s use this sample score report for illustration. Carol Testtaker took both multiple-choice sections. This is what we mailed her.

On Section 1, you can see that she got 70% of the questions right in the first content area (“Application of code requirements, laws…”). Using the content breakdown on our web site, she can extrapolate how many questions she got right or wrong, and that can help her determine areas she might want to focus on for the next exam.

On Section 2, you can see that although Carol answered only 53% of the programming questions correctly, she still was able to earn a passing score overall. We see a lot of chatter about how to figure out how many questions you have to answer correctly in order to pass, but the truth is that the number of questions you need to get right to pass the test may vary from administration to administration. We explain why here.

You can use these percentages and the content outline to determine the areas in which you may need further study and experience. If you want to drill a little deeper, you can always reference the Practice Analysis, which is essentially the blueprint for the exam.

We keep blogging with this same message, but it’s a universal truth about the NCIDQ Exam: the best way to ensure success is well rounded experience. Rather than spend your energy on how many questions you have to get right, concentrate on getting exposure to all those content areas.

PRACTICUM

Now let’s talk about Section 3, the practicum section. On this section, we report a pass/fail score for each of the seven exercises. Obviously, a quick glance at the failing exercises can identify areas of weakness.

To understand how to interpret your Section 3 score report, you need to understand how Section 3 is graded. Each exercise receives two scores ranging from 0 to five. The scores are added together and then multiplied by a weighting factor. This weighting factor is based on the criticality of the skill in the profession of interior design. It isn’t necessary to pass all seven exercises to pass this section. Depending on how high your scores are in one exercise might be enough to overcome failing scores in another exercise.

Let’s use another example. John Examcandidate took Section 3 and here is his score report. John failed the Lighting Design, Restroom, and Millwork exercises. Obviously, then John needs additional study and experience in these areas.

It’s important to understand that it is the combination of scores for each exercise that determines passing or failing, not the overall percentage of that exercise to the exam. For instance, let’s say John got a 4 from each of the graders on the systems integration exercise (meaning, both agreed it was a borderline pass). We take the sum of the two grades (8) and multiply it by the weighting factor. 

But, if John had been able to demonstrate his competence just a little more clearly, the graders might have decided it was a clear pass (score of 5) rather than a borderline pass. In that case, we would have multiplied 10 times the criticality and that might have been enough to pass the entire section.

This is why it is so important to do your best on each exercise. All of those points can add up to success.

After each exam, we publish the practicum exam as a study tool for candidates to use for practice. This is an ideal way to simulate exactly what will be required on exam day. We also publish solutions online with graders’ comments, so you can score your own exam based on their comments. Many people who took the exam use the online solutions to see where their mistakes might have been before they retake Section 3.

You can see which sample practicums are available here.

Reporting Your Exam Scores--How it Works

Waiting for the results of the NCDIQ exam can be excruciating, we realize. Other blog posts have talked about how we score the exams to ensure fairness to all candidates. In this post, we outline the process of getting the results to you.

There are two steps involved:

1. Mailing the score reports.

The mailed score reports contain numeric scores for Section(s) 1 and 2 and pass/fail reports for each exercise in the practicum exam (Section 3) and an overall pass/fall for Section 3.

2. Uploading the results into your MyNCIDQ account.

We only upload whether you passed or failed a section, not numeric scores for Sections 1 and 2 or results of each exercise in Section 3.

Before we mail the score reports, you will get an Exam Update e-newsletter from us letting you know that the score reports are ready. Make sure your email is current in MyNCIDQ.

We mail all the score reports on the same day. We don’t mail one location’s results first or all the passing scores first. Everybody’s mail goes out on the same day. Depending on where you live, it may take longer for your mail to arrive. If your state board or provincial association made you eligible for the exam (meaning you did NOT apply to us), then your scores get mailed to your board that day.

Please allow a week from the date given in that newsletter before you call us to request a copy. If you have not received your score report in the mail and you applied to NCIDQ, you can request another copy, and we will mail you another one to the address in your MyNCIDQ account. For your security, we will not:

• mail to an address not listed in your account;
• email your results;
• fax your results;
• give you the results over the phone; or
• provide results directly to you if you applied to a state board or provincial association.

About a week after we mail everything, we upload the results into your MyNCIDQ account.

If you have passed all three sections of the exam AND you applied to NCIDQ, you will have a new status that says Active Certificate Holder. This status is valid for one year. We will email you when it’s time to renew it. The first year is free, so you can start taking advantage of your Certificate holder benefits right away!

If you still need to take exam sections, your status will say Approved Exam Candidate. We will send you the registration reminders when it’s time to register for the next administration.

Now that you know the process, let’s talk about how to interpret your score report.

Grading the Practicum Exam

Grading the practicum examinations is a complicated process. There is considerable work done by both staff and volunteers before the exams are even graded. Since everything is anonymous, we spend days immersed in control numbers of exam takers and numbers assigned to graders. Everything we do builds on our commitment to fairness to the exam candidates.

Our test consultant, ACT, provides us with a list of control numbers of people who took the examination. We reconcile this list against our own registration and no-show lists to make sure the lists match. Then, we reconcile this complete list of actual exam-takers against each section for each examination to make sure that we do, indeed, have all three sections of everyone’s exam. That’s why we ask you to write your control number on every page.

Often, we uncover inconsistencies in the control number on the cover sheet of the examination and the vellum sheets. We understand that in the stress of the moment, candidates transpose digits in their control number, so we have to investigate that and correct it.  Sometimes, we find that a candidate left his or her control number off one or more pages. When that happens, we have to go back and verify the control number and write that number on all the pages. Sometimes, we can’t read a number clearly, so we go back and verify it and overwrite the digit(s) clearly so that there are no errors in the process of entering scores.  You’d be surprised at how much an “8” can look like a zero in the heat of the moment.

TIP: Write your correct control number clearly and cleanly in the cover sheet and drawing pages.

One of the other things the staff does during this physical inventory is remove any evidence of a candidate’s name. Our grading process is completely anonymous. Graders are not allowed to grade any examination where they recognize a person’s control number or work. What we often find is that, contrary to instructions, some candidates write their name on their cover sheet and/or drawing pages. We erase or obscure the name or anything other than a control number that could identify the person.

TIP: Do not write your name anywhere on your examination.

Following that, we physically inventory the contents of each exam packet to make sure we have all the contents for each person. Each page of each exam for every section for every candidate. It takes some time and concentration in order to be fair to everyone. We often find mistakes that must be corrected on the spot. For instance, in the inventory, an exam may be out of numerical order, so we have to find it to ensure that no one’s section is missing.

During the exam, candidates are given parts B and C of the exam at the same time. Most candidates get all the B sheets back in the B envelope and all the C sheets in back in the C envelope—but not all. When we come across a missing sheet in the B packet, for instance, we have to stop our work, find the corresponding C packet and locate the missing drawing sheet and re-order it in its proper place. If we did not do this, the grader for part B would assume that the candidate did not complete the entire part B and assign a failing grade. Our extra effort in the inventory process ensures everyone gets an opportunity to demonstrate his or her competence.

TIP: Make sure the proper section pages are placed in the correct exam sleeve.

As we go through the physical inventory, we are also clearing out the contents of the exam sleeve. Graders only review what is on the actual vellum drawing sheets, so any extra tracing paper, Post-it® notes or note paper is disposed of in a secure trash bin. The graders do not ever see it, and we do not review it. If a candidate drafts a solution on a separate page and tapes it to the vellum drawing sheet, we remove it.

TIP: Draft your solution ONLY on the vellum sheet.

After a couple of days of this sort of inventory and cross-checking, we begin the process of randomizing the exams. We actually use a double-randomizing technique to ensure that graders do not grade exams from all first-time takers or from one city, for instance.

While the staff and clerks are doing all this work, our grading coordinators are meeting to lay out the grading criteria and prepare the training for the graders. Graders spend about 4-5 hours in intense training before ever grading an examination. Each grader specializes in one part of the exam, so they become “experts” in that part of the exam. They discuss each exercise in depth and review how they will handle any discrepancies that candidates encountered on exam day. This ensures that no matter which grader grades the exercise, it is evaluated consistently.

TIP: Do not write notes to graders on your solutions. Graders are trained to disregard extraneous notes.

The grading coordinators monitor the graders’ work and provide on-the-spot coaching when a grader has questions about the criteria. Although graders may apply their professional judgment in their evaluations, they must adhere to a standard set of criteria.

Graders indicate a unique grader number on each exam they grade, and we input this data along with scoring data on site. This allows us to track grader performance and to provide additional training if required.

Every exercise is graded twice, and no grader knows the score of the other grader. If both graders give the exercise a passing score, that exercise passes. If both graders give that exercise a failing score, that exercise fails. In the event that the two graders’ scores diverge, the grading coordinator grades it a third time, and that score replaces the divergent score. This “automatic appeal” ensures that an additional level of fairness is built into the process.

We have quite a bit of information on how the scoring process works and how each exercise is weighted in the final scoring. Read about it here.

Once all the grading is complete, we re-inventory everything, put everything back in order, inventory all the score sheets and begin our quality control of the scores. ACT runs more analysis on the scores to make sure that every exam got a fair and impartial grading and that every exam matches up with the registration files. Then we can merge the scores with the candidate file and we transfer the data to your MyNCIDQ file.

We understand that having to wait for scores – especially 14 weeks – is grueling. We want to cut that reporting time down for you and are working on ways we can do that with our testing partner, ACT. Please understand, though, that what is going on behind the scenes is done to ensure that the examination continues to be the benchmark for interior design professionalism through its validity, reliability and fairness.

Scoring Multiple Choice Exams

We are often asked why it takes so long to release the grades for the examination. The truth is that there is much more to “grading the exam” than just marking the right or wrong answers. Most of what takes so long is done to offer additional levels of fairness to the candidates.

After the right and wrong answers are scored mechanically for each answer sheet, the multiple-choice exams are put through rigorous statistical analyses to ascertain each question’s performance. Granted, some of the questions you answer are not scored; we’re testing them for future exams. They do not count toward your score. So we filter those out and score the remaining questions.

Analyzing each candidate’s performance on each question against everyone else’s performance on each question takes some time. We do this to identify if there are questions that might not have worked in the way they were intended to. For instance, if every candidate misses question 65, that tells us that question 65 isn’t a good question. So we don’t score that question. We actually remove that question from our pool of questions and send it back for more work and more pre-testing on future exams before we use it again. Sometimes, we throw it out completely.

Analyzing data also helps us identify potential testing irregularities (commonly known as “cheating”). For instance, if every person in a certain testing center gets the same exact answers to every question, that’s a statistical improbability, so we have to go back and review that center. 

Then we go back to “equate” each section against the “anchor” test to ensure it is equivalent in its level of difficulty. That’s why we can’t tell you exactly how many questions you need to get right in order to pass. We have all these checks and balances to give you every opportunity to pass the test.

Want to know more about the content of Section 1 and 2? Click here. You'll be amazed at what it takes just to get a question onto a test. Each question has to meet a whole checklist of standards just to get pre-tested. See the whole process here.

Next up: Grading the Practicum Exams

 

 

 

Fairness to Exam Candidates—More than just A) B) C) D)

By Andrew Stafford

Andrew is NCIDQ’s Exam Manager, who oversees everything about the program from the first draft of an exam question to the publishing of the final scores. In this role, he answers a lot of questions about the exam. We asked him to share with us the questions that he hears all the time and provide some advice related to those questions. He should know, he’s a Certificate holder, and he tells us he remembers vividly what the exam experience was like. 

NCIDQ monitors blogs and discussions and talks to a lot of exam candidates who take outside prep classes for the exam. What they always ask me is, “How am I supposed to pass the exam when the questions on the actual exam were not anything like the ones I got in my prep class?”

It’s important to understand that NCIDQ develops and administers the exam. It is the “NCIDQ Exam.” Other organizations develop preparatory programs. These are two separate activities, and knowing that will help you understand who does what in this process.

NCIDQ volunteer committee members who develop the NCIDQ Exam are not permitted to be involved in exam prep activities, and likewise, people who develop and teach prep courses are not involved in the NCIDQ Exam. NCIDQ volunteers must attend a comprehensive training in the science of creating questions (“psychometrics”) so that the questions are all fair and relevant to the practice analysis that is the exam blueprint. In order to comply with industry standards, NCIDQ does not review outside companies’ test questions or prep materials to see how (or if) they relate to the actual exam.

We often hear complaints about the “trick” questions we put on the exam, and like all professional licensing examination programs, it’s absolutely false. First of all, it would violate the testing standards and render our exam invalid or indefensible, and second, the public is not protected by a “trick” question. The public is only protected when an interior designer is able to adequately demonstrate his or her competence. That is the purpose of the NCIDQ Exam. It is 100% health, safety and welfare. No tricks or hidden agendas.

Multiple-choice questions are given a trial run as unscored questions on exams before they are first scored on an exam. Some never make it. After each exam, our testing consultant, ACT, performs a series of statistical studies on the performance of each question as it relates to each candidate. If a question—whether scored or unscored—performs poorly, it is removed from the pool of test questions, and if necessary, it is not scored. For example, let’s say that a disproportionate number of people miss question 18 on Section 1. We know that is statistically impossible if the question were doing its job. So we throw out that question completely and we do not factor it into the final score. This ensures that we maintain fairness to the candidate and reliability to the public.

Many things and many hours of work go into a question that you see on the examination. If what you see on the exam seems different from what you saw when you were practicing, it is probably due to the differences in how NCIDQ develops exam questions and how others prepare them. If you want to see NCIDQ’s sample multiple-choice questions, you can take a practice test.

The practice test will tell you what the answers are, but it won’t tell you why one answer is right and why is wrong. Want to know why? We’ll tell you here.