National Survey of Student Engagement

Brendan Hennessey

Description

National Survey of Benchmark Recalculation 2007
Student Engagement University of Hawaii at Hilo



In our continuing efforts to provide institutions with the best information possible, changes were
made in 2004 in the way we calculate the NSSE benchmarks of effective educational practice. These
changes allowed us to produce student-level benchmark scores, enhancing the usability of the
information for intra-institutional comparisons. For example, institutions can now examine
benchmarks at the school, college, or department level, or can compare particular subgroups of
students (e.g., men and women or seniors from two different years). The changes in the calculation
require that benchmarks prior to 2004 be recalculated to more accurately compare institutional
performance over the years using the same metric (Table 1).

Another change made to the survey in 2004 affects the information in Table 1. Response options for
the ‘enriching’ items (question 7 on the survey) were altered in 2004 making it untenable to compare
newer results on these items with those of 2003 and earlier. For this reason, the Student-Faculty
Interaction benchmark is recalculated without one item and the Enriching Educational Experiences
benchmark is not recalculated.


Table 1
aRecalculated Benchmarks for NSSE Participation since 2001
bBenchmark Class 2001 2002 2003 2004 2005 2006 2007
Level of FY 46.9 47.5 48.8 47.1 46.6
Academic
Challenge SR 57.0 54.9 54.5 52.8 55.2 ...

Sujets

• Linternaute.com
• Ittoqqortoormiit
• Fyrkat
• Studentersamfundet i Trondheim
• Sexe du locuteur en japonais parlé
• Population des provinces italiennes

Informations

• Publié par: Asso
• Nombre de lectures: 37
• Langue: English

Extrait

National Survey of Student Engagement

Benchmark Recalculation 2007 University of Hawaii at Hilo

In our continuing efforts to provide institutions with the best information possible, changes were made in 2004 in the way we calculate the NSSE benchmarks of effective educational practice. These changes allowed us to produce studentlevel benchmark scores, enhancing the usability of the information for intrainstitutional comparisons. For example, institutions can now examine benchmarks at the school, college, or department level, or can compare particular subgroups of students (e.g., men and women or seniors from two different years). The changes in the calculation require that benchmarks prior to 2004 be recalculated to more accurately compare institutional performance over the years using the same metric (Table 1). Another change made to the survey in 2004 affects the information in Table 1. Response options for the ‘enriching’ items (question 7 on the survey) were altered in 2004 making it untenable to compare newer results on these items with those of 2003 and earlier. For this reason, the StudentFaculty Interaction benchmark is recalculated without one item and the Enriching Educational Experiences benchmark is not recalculated. Table 1 a Recalculated Benchmarks for NSSE Participation since 2001 b Benchmark 2002 2003Class 20012006 20072004 2005 Level of FY 46.947.5 48.847.1 46.6 Academic Challen eSR 57.054.9 54.552.8 55.2 Active and FY 38.239.6 38.340.9 39.8 Collaborative Learnin SR50.5 47.549.6 50.250.7 Student FY 31.336.0 33.237.1 34.9 Faculty c Interaction SR 49.4 45.247.4 46.546.1 Supportive FY 58.755.0 58.458.5 58.9 Campus Environment SR 62.4 60.5 61.655.7 57.5 For more information about benchmark construction and to download syntax that calculates studentlevel scores, visit the NSSE 2007 Institutional Report Web site: www.nsse.iub.edu/2007_Institutional_Report.

IPEDS: 141565

Page 1

National Survey of Student Engagement

Benchmark Recalculation 2007 University of Hawaii at Hilo

How comparable are benchmark scores from improve the survey’s validity and reliability. Most notably, response options for the ‘enriching’ items yeartoyear? d (question 7 on the survey) were revised in 2004.Our This report is a brief introduction to comparing analysis shows that these items are not comparable institutional performance over time, not an exhaustive with prior years.For most institutions, this change treatment of all the pertinent issues that need to be will produce a substantially lower Enriching considered. We recommend that you do further analysis to Educational Experiences score since 2004 compared better understand the changes within your institutional to prior years, particularly for firstyear students. context. It is important to keep in mind three issues before comparing benchmark scores from yeartoyear: What constitutes a real change in a 1)Drawing a random sample from a population results in benchmark score? a certain amount of sampling error – an estimate of the One way to estimate the magnitude of change in a degree to which the characteristics of the sample do benchmark score over time is to combine your institutional not match those of the population. Smaller samples data from all participating years and run statistical analyses relative to the size of the population risk larger between students from the respective years. For example, sampling errors. Thus, relatively small benchmark ttests can be computed between firstyear students in 2003 differences could be attributed to random sampling and firstyear students in 2006 to see if the differences fluctuation. between benchmark scores are statistically significant. 2)In addition to sampling error, you should examine the Effect sizes can also be computed by dividing the demographic characteristics of the samples to be sure difference of the benchmark scores by the standard that similar groups of students are represented among deviation of the entire distribution. The ttests can also be the respondents in various years. If respondent weighted according to statistical weights provided by characteristics are different, and these differences NSSE (based on gender and enrollment status), or could likely affect engagement scores, they should be institutions can create their own weights based on school acknowledged and taken into account when attributing records. reasons for benchmark differences. A more Institutions can also conduct regression analyses using the sophisticated approach would be to weight the multiyear data and include a dummy variable for the year samples so they more closely resemble the student of participation as an independent variable. With this population, and then recalculate the benchmark scores approach, the regression model could control for student using the formulas provided by NSSE. However, keep demographic variables or other independent variables to in mind that all of your recalculated benchmarks are b see what the unique effect of the year of administration weighted by gender and enrollment status. might be. 3)Some questions and response options were changed over the years based on psychometric analyses to Notes a. Scoresfrom NSSE 2000 are not includedtaken from institutional population filesprevious year reports, or on your 2007 because several significant changes were rather than IPEDS.Benchmark Comparisons report. made to the survey instrument after that c. Allitems in question 7 on the current NSSEd. All items in question 7 on the 2004 year, thus making yeartoyear comparisons instrument were rescaled in 2004. One ofinstrument were rescaled in 2004. The old less suitable. these items, “Work on a research projectresponse set (NSSE 20002003) was ‘yes,’ b. Student weights prior to 2004 werewith a faculty member outside of course or‘no,’ or ‘undecided’ whereas the new (NSSE computed exclusively using the most recentprogram requirements,” contributes to the20042007) response set is ‘done,’ ‘plan to IPEDS data available. Starting with 2004,StudentFaculty Interaction benchmark. Seedo,’ ‘do not plan to do,’ or ‘have not institutional population files were used fornote ‘d’ for more details. Therefore thedecided.’ Our analysis shows that these class rank and gender because these filesStudentFaculty Interaction scores on thisitems are not comparable across years. provide more recent and accurate data.report do not include the ‘research’ item.Therefore, it is not possible to compare the Beginning in 2005, enrollment statusThis also means that the score on this report20042007 Enriching Educational information (fulltime/parttime) was alsowill not match benchmarks reported onExperiences benchmark with prior years (20012003).

IPEDS: 141565

Page 2