Grading the practicum examinations is a complicated process. There is considerable work done by both staff and volunteers before the exams are even graded. Since everything is anonymous, we spend days immersed in control numbers of exam takers and numbers assigned to graders. Everything we do builds on our commitment to fairness to the exam candidates.
Our test consultant, ACT, provides us with a list of control numbers of people who took the examination. We reconcile this list against our own registration and no-show lists to make sure the lists match. Then, we reconcile this complete list of actual exam-takers against each section for each examination to make sure that we do, indeed, have all three sections of everyone’s exam. That’s why we ask you to write your control number on every page.
Often, we uncover inconsistencies in the control number on the cover sheet of the examination and the vellum sheets. We understand that in the stress of the moment, candidates transpose digits in their control number, so we have to investigate that and correct it. Sometimes, we find that a candidate left his or her control number off one or more pages. When that happens, we have to go back and verify the control number and write that number on all the pages. Sometimes, we can’t read a number clearly, so we go back and verify it and overwrite the digit(s) clearly so that there are no errors in the process of entering scores. You’d be surprised at how much an “8” can look like a zero in the heat of the moment.
TIP: Write your correct control number clearly and cleanly in the cover sheet and drawing pages.
One of the other things the staff does during this physical inventory is remove any evidence of a candidate’s name. Our grading process is completely anonymous. Graders are not allowed to grade any examination where they recognize a person’s control number or work. What we often find is that, contrary to instructions, some candidates write their name on their cover sheet and/or drawing pages. We erase or obscure the name or anything other than a control number that could identify the person.
TIP: Do not write your name anywhere on your examination.
Following that, we physically inventory the contents of each exam packet to make sure we have all the contents for each person. Each page of each exam for every section for every candidate. It takes some time and concentration in order to be fair to everyone. We often find mistakes that must be corrected on the spot. For instance, in the inventory, an exam may be out of numerical order, so we have to find it to ensure that no one’s section is missing.
During the exam, candidates are given parts B and C of the exam at the same time. Most candidates get all the B sheets back in the B envelope and all the C sheets in back in the C envelope—but not all. When we come across a missing sheet in the B packet, for instance, we have to stop our work, find the corresponding C packet and locate the missing drawing sheet and re-order it in its proper place. If we did not do this, the grader for part B would assume that the candidate did not complete the entire part B and assign a failing grade. Our extra effort in the inventory process ensures everyone gets an opportunity to demonstrate his or her competence.
TIP: Make sure the proper section pages are placed in the correct exam sleeve.
As we go through the physical inventory, we are also clearing out the contents of the exam sleeve. Graders only review what is on the actual vellum drawing sheets, so any extra tracing paper, Post-it® notes or note paper is disposed of in a secure trash bin. The graders do not ever see it, and we do not review it. If a candidate drafts a solution on a separate page and tapes it to the vellum drawing sheet, we remove it.
TIP: Draft your solution ONLY on the vellum sheet.
After a couple of days of this sort of inventory and cross-checking, we begin the process of randomizing the exams. We actually use a double-randomizing technique to ensure that graders do not grade exams from all first-time takers or from one city, for instance.
While the staff and clerks are doing all this work, our grading coordinators are meeting to lay out the grading criteria and prepare the training for the graders. Graders spend about 4-5 hours in intense training before ever grading an examination. Each grader specializes in one part of the exam, so they become “experts” in that part of the exam. They discuss each exercise in depth and review how they will handle any discrepancies that candidates encountered on exam day. This ensures that no matter which grader grades the exercise, it is evaluated consistently.
TIP: Do not write notes to graders on your solutions. Graders are trained to disregard extraneous notes.
The grading coordinators monitor the graders’ work and provide on-the-spot coaching when a grader has questions about the criteria. Although graders may apply their professional judgment in their evaluations, they must adhere to a standard set of criteria.
Graders indicate a unique grader number on each exam they grade, and we input this data along with scoring data on site. This allows us to track grader performance and to provide additional training if required.
Every exercise is graded twice, and no grader knows the score of the other grader. If both graders give the exercise a passing score, that exercise passes. If both graders give that exercise a failing score, that exercise fails. In the event that the two graders’ scores diverge, the grading coordinator grades it a third time, and that score replaces the divergent score. This “automatic appeal” ensures that an additional level of fairness is built into the process.
We have quite a bit of information on how the scoring process works and how each exercise is weighted in the final scoring. Read about it here.
Once all the grading is complete, we re-inventory everything, put everything back in order, inventory all the score sheets and begin our quality control of the scores. ACT runs more analysis on the scores to make sure that every exam got a fair and impartial grading and that every exam matches up with the registration files. Then we can merge the scores with the candidate file and we transfer the data to your MyNCIDQ file.
We understand that having to wait for scores – especially 14 weeks – is grueling. We want to cut that reporting time down for you and are working on ways we can do that with our testing partner, ACT. Please understand, though, that what is going on behind the scenes is done to ensure that the examination continues to be the benchmark for interior design professionalism through its validity, reliability and fairness.
Comments