Test Development

Melissa Fein

Description

Test Development: Fundamentals for Certification and Evaluation provides practical guidelines for the classical approach to test development – specifically for certification and credentialing. Most of the existing high-quality books on the topic of criterion-referenced test (CRT) development are written for professional statisticians and psychometricians. And the books written for the general reader tend to lack accurate, user-friendly coverage of some of the most critical topics. This book serves as an accessible resource on test development that does not require any technical background or expertise. Part I, Conceptual Principles, covers the process of CRT development in narrative form. Some of the topics covered include:

defining content domain as related to job analysis and assessment objectives

item writing and scoring

understanding validity and reliability

conducting item analysis

defining and differentiating the major standard setting method

preparing for and understanding issues with test administration.

Part II, the Technical Appendix, contains the computational instructions and supplementary background material. This do-it-yourself manual is a valuable resource for training program developers, credential program developers, as well as other relevant human resources personnel and upper management who need to be on board for change management related to testing programs.

Sujets

• Loisirs
• Arts Numériques
• Computers
• Graphics
• Design
• Media

Informations

• Publié par: Association for Talent Development
• Date de parution: 09 octobre 2012
• Nombre de lectures: 4
• EAN13: 9781607286769
• Langue: English
• Poids de l'ouvrage: 1 Mo

Extrait

© 2012 the American Society for Training & Development
All rights reserved. Printed in the United States of America.
No part of this publication may be reproduced, distributed, or transmitted in any form or by any means, including photocopying, recording, or other electronic or mechanical methods, without the prior written permission of the publisher, except in the case of brief quotations embodied in critical reviews and certain other noncommercial uses permitted by copyright law. For permission requests, please go to www.copyright.com , or contact Copyright Clearance Center (CCC), 222 Rosewood Drive, Danvers, MA 01923 (telephone: [phone], fax: [phone]).
ASTD Press is an internationally renowned source of insightful and practical information on workplace learning and performance topics, including training basics, evaluation and return-on-investment, instructional systems development, e-learning, leadership, and career development.
Ordering information (for print edition): Books published by ASTD Press can be purchased by visiting ASTD’s website at store.astd.org or by calling 800.628.2783 or 703.683.8100.
Library of Congress Control Number (print edition only): 2012940624
Print edition ISBN: 978-1-56286-825-3 PDF eBook edition ISBN: 978-1-60728-676-9
ASTD Press Editorial Staff:
Director: Glenn Saltzman Community of Practice Manager, Learning & Development: Juana Llorens Manager, ASTD Press: Ashley McDonald Editorial, Design, and Production: ASTD Press Cover Design: Ana Foreman
Contents
Preface
Part I: Conceptual Principles
Chapter 1: Introduction
Purpose of This Book
Content Coverage of This Book
References and Resources
Chapter 2: Credentialing
Introduction
Motivations Underlying the Development of Credentialing Programs
Differentiating Between Types of Credentials
Components of the Credentialing Process
Accreditation
Summary
References and Resources
Related Topics in the Technical Appendix
Chapter 3: Evaluation & CRT Development
Introduction
Formative and Summative Evaluation
Designs for Measuring Program Effectiveness
Kirkpatrick Model
Logic Modeling
Integration of Evaluation Frameworks: Hybrid Models
Relevance of CRTs to Evaluation in HPT/HPI
CRT Development and Monitoring Training Vendors
Summary
References and Resources
Chapter 4: Approaches to Test Development
Introduction
Classical Test Theory
Item Response Theory
Computerized Adaptive Testing
Computerized Testing
IRT Resources
Sample Size
Using Consulting Services vs. In-House Developers
Summary
References and Resources
Chapter 5: Defining Content Domain (Part I)—Job Analysis
Introduction
Stage I: Initial Task Inventory
Stage II: Survey Tool Development
Stage III: Survey Administration
Stage IV: Instructional Design and CRT Content Weighting
Summary
References and Resources
Chapter 6: Defining Content Domain (Part II)—Assessment Objectives and the Table of Specifications
Introduction
Domains and Taxonomies
Writing Assessment/Learning Objectives
Table of Specifications
Summary
References and Resources
Chapter 7: Item Writing and Scoring
Introduction
Fidelity and Authenticity
Item Types
General Guidelines for Item Writing
Multiple-Choice Items
True/False Items
Traditional Matching Item Sets and Extended Matching Items
Testlets
Performance Assessment
Scoring Keys
Rater Training and Sources of Bias
Item Shells and Item Banks
Additional Item Format Issues
Organizing the Item-Writing Team: Item Writing and Reviews
Summary
References and Resources
Related Topics in the Technical Appendix
Chapter 8: Validity
Introduction
Face Validity
Content Validity
Decision Validity (Criterion-Related Validity)
Survey Fatigue and Decision Validity
Summary
References/Resources
Related Topics in the Technical Appendix
Chapter 9: Reliability
Introduction
Coefficient-α
Mastery Decision Consistency
Equivalent/Alternate Forms
Rater Agreement
Reliability of a Two-Component Test
Strategies for Enhancing Reliability Measurement
Summary
Additional Notes on Reliability
References and Resources
Related Topics in the Technical Appendix
Chapter 10: Item Analysis
Introduction
Item Difficulty
Item Discrimination
Distractor Analysis
Instructional Sensitivity
Examinee Sample and Item Statistics
Summary
References and Resources
Related Topics in the Technical Appendix
Chapter 11: Standard Setting
Introduction
Process Issues
Bookmark Procedure
Angoff Variations
Additional Standard Setting Methods: Contrasting Groups and Borderline Group
Standard Setting in Rubric Development
Standard Setting and Program Evaluation
Imperfection of Cut Scores and the Standard Error of Measurement
Summary
References and Resources
Related Topics in the Technical Appendix
Chapter 12: Test Administration Issues
Introduction
Test Directions
Compiling and Piloting the Test
Time Limits
Accommodations
Reading Level
Test Security
Testing Formats
Adverse Impact
Score Reporting
References and Resources
Related Topics in the Technical Appendix
Chapter 13: A Conclusion of Caveats
Part II: Technical Appendix
Chapter 14: Levels of Measurement
Overview of Levels of Measurement
Levels of Measurement (LOM) and Item Scores
LOM and Test Scores
References and Resources
Chapter 15: Mathematical Tools and Microsoft Excel
Mathematical Tools
LOM and Correlation Coefficients
References and Resources
Chapter 16: Classical Item Analysis
Item Difficulty
Item Discrimination
Distractor Analysis
References and Resources
Chapter 17: Reliability
Computing Coefficient- ά (alpha)
Computing Rater Agreement
Computing Parallel Forms Reliability
Computing Mastery Decision Consistency
Creation of a Composite Score
Reliability Computation of Two-Part CRTs
References and Resources
Chapter 18: Standard Error of Measurement (SEM)
Computation
Error Bands
Implications
Chapter 19: Validity
Item Labels: Coding Items
Item Development Worksheet
Computing Decision Validity
Level 2 Foundations: Computing Predictive Validity and Kirkpatrick Level 3
References and Resources
Chapter 20: Scaled Scores
Chapter 21: Standard Setting
Ordering Items by Difficulty
Median Computation: Borderline Groups
SME Credentials Form
Standard Setting Facilitator’s Worksheet
Confidentiality Agreement Components
Credential Team Leader’s Worksheet
Chapter 22: Job Analysis
Hints for Running Focus Groups for J/PA
Hints for J/PA Interviews
Job Analysis, DIF Scores, and Content Weighting
Conclusion: Overview of the Criterion-Referenced Test (CRT) Development Process
Publishers Note
About the Author
Index
Preface
This handbook is an expansion, revision, and hopefully an improvement on Measuring Merit , bits and pieces of which I had been working on for some years, and which came to a completed form in 2005. I say a completed form because as soon as I spiral-bound a draft copy for use in workshops, I wanted to make changes. The first change I wanted to make was to split the book into two parts, one being narrative, and the other computational and technical. The motivation for the split is that many users had no interest —make that a strong disinterest —in being burdened with mathematical details. It was boring to them and they had no intention of personally using the detailed computational information. They wanted mostly a conceptual grounding in the material, but they also needed a detailed set of instructions that could be passed on to a quantitative support person on staff.
A second change I wanted to make had nothing to do with the structure of the book, and everything to do with rounding out my perspective as a credentialing professional. I felt that I needed to turn the tables on myself and become a participant and an examinee in some of the diverse certificate and certification programs available. So, I did, perhaps to an addictive degree.
A list of acronyms and symbols is included because I used to belong to two organizations that used the acronym ASA, the American Statistical Association and the American Sailing Association. Within the field of test development, the acronym DIF is used for two different purposes, and the acronyms SEM and SME might be easily confused.
Disclaimers
This book is not a substitute for legal advice. Proper legal advice and advice on changing human resource policies in the workplace should be sought from qualified personnel. Psychometric trends in test development are not always perfectly aligned with the Uniform Guidelines . It is up to the reader to keep current. In addition, all test development consultants do not agree on every point of test development. The opinions expressed in this book are my opinions, based on my experience helping clients balance different priorities and constraints.
Acknowledgments
I wish to express special appreciation to Dr. Robert W. Lissitz for his ongoing help, encouragement, and support in my career. A substantial amount of the material in this book grew out of seminar materials and consulting projects that we worked on together over many years, as well as discussions we have had. I could not ask for a better mentor.
Thank you to Juana Llorens and her colleagues at the A