Technical Review

Facts

During the item writing process, typically just one or two people (the author and sometimes a collaborator) see each item. To strengthen the validity of a test item and bolster its technical accuracy for all possible interpretations, a group technical review meeting is recommended. In this meeting, a group of subject matter experts reviews every test item for congruence to the objectives, technical accuracy, proper difficulty, and relevance. The group reworks items until they arrive at a consensus of agreement that each item is acceptable.

Technical Review sessions generally follow these steps:

  • A CMS facilitator displays a test item and its reference to the review SMEs.
  • The SMEs read through the testing objective and then the item.
  • The SMEs confirm item alignment to the testing objective and identify the key. The facilitator also asks if the item is written at the appropriate difficulty level for the minimally qualified candidate (MQC).
    • If the item does align with the testing objective, the facilitator proceeds to the next step.
    • If it does not align or is not at the appropriate difficulty level, the facilitator will send the item back to the writer with SME notes on how to improve/correct the item so that it can align to the objective or is at the appropriate difficulty level.
  • After the SMEs state what they believe is the correct answer, the facilitator then displays the key, the item rationales, and the item reference(s).
  • The facilitator then asks the SMEs to identify changes or corrections, if any, to the item to ensure the item is technically accurate. They also review the item rationales and confirm the accuracy of the reference(s).
    • If changes to the item can be accomplished quickly (3-4 minutes), the facilitator will make them at the time of the review.
    • If changes are extensive, then the facilitator will send the item back to the item writer with notes and direction for changes and rewrites.
  • Finally, the facilitator will ask if all the SMEs approve the item and then move it to the next workflow step in the test development process.

What outcomes should you expect from a Technical Review service?

Outcomes from a Technical Review service normally include:

    1. Fully vetted test items seen by multiple domain experts to ensure accuracy, relevance, and appropriate difficulty
    2. An item bank ready for beta testing by a select group of individuals

How can CMS help your organization?

Contact Us