2006 Benchmark results

Me

Description

??2006 IAS Benchmark Results Background Stevens Group, Inc. conducted Benchmark Scope a benchmark of their Integrated Assessment System (IAS) The scope of the benchmark was: software product in May 2006. • Stage 1 Independent Review • creation of a Stage 1 Scorebook to be The benchmark was comprised of examiners who submitted to the Baldrige office as part of would be: the pre-work for training sessions in May • attending National Baldrige training sessions. 2006. • using IAS to create their Scorebook for the Note: At the request of the Baldrige office none of 2006 Case study Arroyo Fresco. the responses for Arroyo Fresco were entered in IAS. All the examiners worked off a hard copy of the application. Summary of Participants Each examiner used IAS to develop their individual • Benchmark participants had National Baldrige review of the case study which included the and State examiner experience creation of the following: • Some examiners were first time National examiners, others had 5-10 years experience • Key Factors • Average experience: • Strengths and Opportunities for Improvement National examiners - 4 years • Key Factors linked to each Item and State examiners - 5 years Comment • Total of 13 examiners from California, Iowa, • Scoring North Carolina, Florida and Illinois. • Key Themes • Some of the benchmark participants have • Consolidated Scorebook used IAS in the North Carolina Awards for Excellence (NCAfE) ...

Informations

• Publié par: Zyon
• Nombre de lectures: 18
• Langue: English

Extrait

2006 IAS Benchmark Results

Stevens Group, Inc.

www.stevensgrpinc.com

Integrated Assessment System – A new way of doing assessments!

Background

Stevens Group, Inc. conducted

a benchmark of their

Integrated

Assessment

System

(IAS)

software product in May 2006.

The benchmark was comprised of examiners who

would be:

•

attending National Baldrige training sessions.

•

using IAS to create their Scorebook for the

2006 Case study Arroyo Fresco.

Summary of Participants

•

Benchmark participants had National Baldrige

and State examiner experience

•

Some examiners were first time National

examiners, others had 5-10 years experience

•

Average experience:

¾

National examiners - 4 years

¾

State examiners - 5 years

•

Total of 13 examiners from California, Iowa,

North Carolina, Florida and Illinois.

•

Some of the benchmark participants have

used IAS in the North Carolina Awards for

Excellence (NCAfE) program and provided

results beyond Stage 1 (including additional

insights from having applicant data in IAS).

Training & Support

Each examiner received a 1 hour training session

via WebEx which was designed to provide

information on how to get started with creating a

Scorebook in IAS.

Additional information on

specific features of IAS was provided through the

online help system with email support from Stevens

Group, Inc.

Benchmark Scope

The scope of the benchmark was:

•

Stage 1 Independent Review

•

creation of a Stage 1 Scorebook to be

submitted to the Baldrige office as part of

the pre-work for training sessions in May

2006.

Note: At the request of the Baldrige office none of

the responses for Arroyo Fresco were entered in

IAS. All the examiners worked off a hard copy of the

application.

Each examiner used IAS to develop their individual

review of the case study which included the

creation of the following:

•

Key Factors

•

Strengths and Opportunities for Improvement

•

Key Factors linked to each Item and

Comment

•

Scoring

•

Key Themes

•

Consolidated Scorebook

Results

At the end of the benchmark each examiner was

asked to complete a survey on their overall

experience using IAS and on specific features.

The

chart below shows: 69% of the examiners would

highly recommend (++), 23% would recommend (+)

and 8% would slightly recommend (-) IAS.

+

23%

-

8%

++

69%

2006 IAS Benchmark Results

Stevens Group, Inc.

www.stevensgrpinc.com

Integrated Assessment System – A new way of doing assessments!

Results

The chart below shows the average of all the ratings for each survey question.

The scale for the chart is:

•

1 --

double minus – Needs major improvement

•

2 -

single minus – Needs some improvement

•

3 +

single plus – Feature is a strength

•

4 ++

double plus – Feature is strong strength

Rating of IAS Features

1

2

3

4

Install

WebEx Training

Support

Copy and paste Criteria

Copy and paste Responses

Key Factors to an Item

Key Factors to comments

Scoring

Key Themes

Generation of Scorebook

Other Reporting

Comment Writing

Online Help

Related criteria questions

Import and Export of data

Automatic spell checking

Overall ease of use

Recommend

Overall Experience

2006 IAS Benchmark Results

Stevens Group, Inc.

www.stevensgrpinc.com

Integrated Assessment System – A new way of doing assessments!

Time Saved

Each examiner was asked for the percentage of time saved using IAS for Stage 1.

¾

Average time saved in completing the Stage 1 Scorebook – 24%

The results for Stage 2 & 3 are based on feedback from examiners that have used

IAS beyond Stage 1.

¾

Average time saved in Stage 2

– 40%

¾

Average time saved in Stage 3 – 23%

The results for Report Generation are based on the time saved in generating the

Scorebook, Key Themes, Key Factors worksheets, Scoring summary and Feedback

report.

¾

Average time saved generating reports using IAS – 35%

Percentage of Time Saved by Stage

Stage 1

Stage 2

Stage 3

Report

Generation

0

5

10

15

20

25

30

35

40

45

50

1

Willing to Pay

Each examiner was asked if they would pay to use IAS with either their State or

National Baldrige program and if so what amount would they pay.

¾

61% of the examiners would be willing to pay on average $100

for the

use of IAS.

2006 IAS Benchmark Results

Stevens Group, Inc.

www.stevensgrpinc.com

Integrated Assessment System – A new way of doing assessments!

Examiner Feedback

Each examiner was asked to provide feedback on the following:

•

Identify what benefits they experienced using IAS during their case study pre-work

•

The impact of using IAS at the State or National level would have on their decision to volunteer

•

Which IAS features were most helpful to them

•

Overall comments on their experience with IAS

•

Recommended improvements that would IAS more valuable

Below are the quotes from each examiner.

Benefits

•

Less administrative work, more time to work on the content of

comments.

•

Having everything in one place and being able to print out to

proof

•

Faster.

Like having the criteria right above the comments for

continual referencing.

Kept me more on track.

•

The IAS software saved time and allowed me to more fully

comprehend the material. Since this was my first time as an

examiner, the IAS benefit goes way up in subsequent trips back

to the software because of the framework learned at the

National training.

•

Shortened overall assessment time a little

•

Completed work in shorter period of time

•

Great to print out scorebook!!!!

•

Time savings on the case study; even more so when doing

consensus with others; and even more so when the applicant’s

application is in the software.

•

Time savings and not as much material sitting around because

of the Criteria being on the page.

•

Saves time in re-keying information in from the criteria or the

applicant. Know exactly where to go without having to scroll

through the scorebook, sometimes that seemed odd to leave

what you’re doing to see what you wrote elsewhere, but better

than scrolling through the scorebook.

•

The software produces a scorebook in less time with more

accuracy and provides a valuable document for the customer. It

also helps drive the process in stage 2 and 3.

The amount of

time saved in producing the consensus document is incredible.

2006 IAS Benchmark Results

Stevens Group, Inc.

www.stevensgrpinc.com

Integrated Assessment System – A new way of doing assessments!

Volunteer Impact

Examiners were asked if using IAS would have an impact on their volunteering for either

their State or the National Baldrige program.

The most impact was found from those

examiners that have used IAS past Stage 1.

•

In North Carolina I can't imagine having the process without it. Don't KNOW that it

would impact my decision since we are in the application process and still learning.

•

Significant

•

BIGGIE, esp. now with Consensus functionality

•

I would continue regardless..but this would ease the process.

•

From an examiner point of view--its is a significant time saver for the state level especially for the

consensus; don't know that it would make or break my participation--it just helps to facilitate

•

Could have a play in the decision but probably not in the final.

•

Passion and persistence in advancing Baldrige are not related to volunteering, although it definitely

makes it more enjoyable.

Most Helpful Features

•

I think the key factor report is exemplary!

It really integrates the

connection between what is important to the organization and the criteria

comments.

It also helps with the focus on key themes because you can

look over the +/- related to each comment/key factor.

I also think the

assigning of key factors, cross cutting, and ADLI is much more precise

because of the ease of working through each comment.

The scoring is

much more accurate because it gives you the band range and you can

click and see your comments or the comments in a crosscutting category

with speed and ease

•

Cut and paste from the Criteria.

Automated Key Factor linkage, ADLI,

and ++ or cross-cutting.

•

Criteria on the page.

Loved it with the real applicant to be able to cut and

paste. Liked how to link the key factors.

Having the pop up screen for key factors

•

Quick assistance via e-mail.

•

KF linkage, Key Theme draft generation, ability to keep KF as pop up while crafting responses, linkage

to questions

•

Multiple screens and views of windows allowed for ease compared to the "traditional approach".

•

Spell check, viewing of criteria while assessing

•

Criteria readily available, easy selection of Key Factors.

•

Key factors linkage, ADLI linkage, and reporting

•

Significantly cuts down the time of aligning key factors to the comments; formats for you when printing;

spell check; pulling of cross cutting and ++ or -- comments to the key themes worksheets

•

I would love to see this product through Consensus.

That's where I expect the most help!

•

Key factor linkage, Criteria visibility and connections, tabs for strength and OFI, I have not used it but

consolidation of remarks for consensus sounds awesome

•

Cut and paste was a great help!

If there was real applicant content in it, it would have been

tremendous.

2006 IAS Benchmark Results

Stevens Group, Inc.

www.stevensgrpinc.com

Integrated Assessment System – A new way of doing assessments!

Overall Comments

•

This is a tool that will assist examiners in producing a higher quality

scorebook - Key factor and Key theme generation easier, examiner time

focused on quality of evaluation not wrestling with logistics.

Will not

replace examiner gray matter, but certainly organizes the work so that I

can use it!

•

I can see no

dis

advantage to using IAS to enable professional facilitation

of the awesome responsibilities involved in the awards process at state

and national levels.

•

Very helpful with maximizing time spent evaluating.

Thank you.

•

It would be great to incorporate the software into a training module during

new examiner week..some 1/2 or full day training would have been useful

instead of the first day orientation which really did not work until you had

the benefit of the full program.

•

This product does save time in the mechanics of developing a scorebook.

I found it difficult to keep

track of all the comments I was writing, unless I generated a report to review.

•

Testimonials of those with experience in the national process.

Forward thinking for how this could

impact the applicant and the process value if done appropriately.

•

Very helpful, terrific support

•

I also appreciate the support from Stevens Group in answering any questions, comments, or concerns.

They exceeded my expectations and truly provided service that delighted the customer.

Thank you!

Recommended Improvements

Examiners recommended several improvements to IAS that would make it easier

for future users.

Below are the improvements that have ALREADY been

implemented along with those planned for future releases.

1) Implemented Improvements

•

Scoring should be on same sheet as items

•

Counts of strengths and OFIs

•

A count of comments for an entire item (to keep track no matter what area

to address you may be working on).

Scoring screen: ability to see ALL

comments for each item not just what fits on a single page.

•

An option to set up dialog boxes as reminders of the National process. For

example, if you begin to type in a comment before selecting Key Factors for an item, then dialog box

pops up to alert you to that fact.

Another example, is the ADLI dimensions, let's you leave the

comment without identifying those and in some cases I simply forgot and it would have been nice to

have a reminder. If it is made as a preference option then more seasoned examiners who may get

annoyed with it can opt to turn that option off.

•

Reminder to add the ADLI and KF

•

Bigger window to see the full text of selected key factors.

2006 IAS Benchmark Results

Stevens Group, Inc.

www.stevensgrpinc.com

Integrated Assessment System – A new way of doing assessments!

2) Planned Improvements

•

Ability to select and edit Key Factors per item to pinpoint specific KFs.

•

Not having the application on line in the Arroyo Case was definitely a negative for that process.

•

Printed reference materials regarding screen/font size, and linking key factors to individual comments.

•

Difficult to look at comments in other items during assessment, can't make comments that apply to

more than one area at a time, can't easily see other comment for the item

•

Being able to see the full comment in the right side; ability to see both strengths and OFI comments for

the same item at the same time to ensure that they are not in conflict--can only do after printing out.

•

To make the screen larger so you don't have to scroll down to read criteria, comments, and the

application.

It would be nice to be able to pull the screen down like in word.

For additional information on IAS and how it can improve your assessment process

Contact

Stevens Group, Inc. at [phone]

Email

[e-mail]

Visit

www.stevensgrpinc.com/iasindex.html