Technology Training for Digital Projects (Southeastern NY Library Resources Council)

Outcome-Based Evaluation
Best Practices in New York State

(Example illustrating the use of a Complete OBE Package, including a logic model)

This entry includes the entire second year LSTA project report. All aspects of an OBE plan are in evidence. The outcomes are stated in terms of what participants will do with their learning and it reaches the furthest level of impact to include how library users will ultimately benefit. While there are confidence indicators of success, they are backed up by requirements that the participants demonstrate understanding on post-tests and in a Mid-Term Review that served as a "Show and Tell" of what the participants achieved after the workshops. This was an outstanding way to assess what participants could do with what they learned and reflects true program impact.
The seventh indicator of the first outcome captured how many participants could continue to use what they learned working independently. The use of a rubric to hold the learners to a pre-specified standard is excellent. The indicators for the second outcome were ambitious and laudatory. The attempt was to learn how patrons ultimately benefited from the post-workshop efforts of the participants. While the data are incomplete for this outcome, the plan to ultimately learn the impact on patrons is being built into the ongoing program with a plan to concentrate on an effective way to communicate with student users. Even though the project originates at the system level, there was a plan from the beginning to collect data at the local direct service-to-patron level.
The project is a good model for systems to set similar goals and enlist the support of the people they train to report impact on patrons. The project has only two overarching outcomes with multiple indicators. It has realistic targets for achievement and interesting ways to get feedback. The report includes both outputs and outcomes and interprets the data so that impact is clear. It is a fine example of OBE.

1. Describe the project's accomplishments to date in relation to its objectives and the target population.

The objectives for Year Two of the project, Technology Training For Digital Projects, were to continue to teach staff from libraries and cultural heritage organizations the fundamental skills and issues necessary to create and promote a digital project. SENYLRC hosted six workshops, starting in April, 2006, on various topics related to digitization. SENYLRC administered the training and provided support and assistance as part of this LSTA project. SENYLRC also provides the infrastructure for the organizations to host their digital collections and make them accessible on the Internet. We are pleased to report that 18 organizations are independently digitizing historical materials as a direct result of the training, and hands-on support that they received from the Digital Project Trainer, which they received from this LSTA funded project.

The target audience for the project is researchers and students who will use the site, Hudson River Valley Heritage (www.hrvh.org), where the participants in the training will have their collections available. While the usages statistics for HRVH are excellent, we reached about half of the researchers that we planned to target, and few educators.

2. To what extent were project activities completed within the project timeline?

The timeline in the application for Technology Training for Digital Projects stated that the activities in Year Two would begin in April, 2006. The second year of the project began on schedule with a workshop on the application platform and software, CONTENTdm, on April 5, 2006. After the workshop on How to Create and Manage Digital Collections using CONTENTdm, the Digital Project Trainer began to make site visits to the participating organizations for them to begin creating digital objects using CONTENTdm. We found it critical for the training that there was follow-up in the use of the software and the concepts that they learned in the first year of the project (Project Management, Copyright, Scanning/Imaging and Metadata). At this point, we focused on the hands-on training and there was not a formal program until June 20, 2006, when we held a Mid-Term Review. At this program, all of the participants came to SENYLRC and presented the digital objects that they had created up until this point. Each organization, 20 in all, had 20 minutes to discuss their project. The next program was on July 12, 2006 when we had two half-day programs on Digital Preservation Strategies and Sustainability: Business Planning for Digital Initiatives. On September 13, 2006 we held a program on Promotion and Use of Digital Projects. A Final Program for Training for Digital Asset Management through the Hudson River Valley Heritage Program was held on March 29, 2007.

We provided the participants with six months to create their digital projects and found that this was not enough time for some people to create a digital project, while others were committed to completing a project in the allotted time. We also discovered at the workshop on promotion that we needed a Communication Plan for HRVH.  A small group of people held three meetings and completed a Communication Plan for HRVH.  Unfortunately, we were not able to carryout all of the promotional efforts in the timeframe outlined in the original proposals. Also, the participating organizations were at various stages of their projects, while some were still digitizing, others were putting out press releases announcing their projects.

3. Briefly summarize any quantitative evaluation results for this project and attach the Quantitative Measures page, noting to what extent you reached the target population. 

Twenty-four organizations completed all of the workshops in Year One of the project and continued into Year Two. In the second year, a total of 138 people attended the six workshops. There were 20 organizations that digitized items and contributed them to HRVH. There are 20 organizations with a webpage as part of HRVH. Among them they digitized 2,347 digital objects from June, 2006 until March, 2007. SENYRLC also created several documents and publications as part of their second year of the project that included three press releases, a Metadata Style Guide 2.1, a Communication Plan and a bookmark to distribute to users.

The target audience for the project was researchers and students who used Hudson River Valley Heritage. There have been a total of 18,408 unique visitors to the collections in HRVH that were created from the organizations who took part in this training.  We received comments from 50 researchers and one educator about the items in these collections.

4.Briefly summarize the qualitative evaluation results, providing information in relation to progress toward expected outcomes

There are two outcomes for Technology Training for Digital Projects. The first outcome is: Staff from libraries and cultural heritage organizations enter objects independently into Hudson River Valley Heritage. There are seven indicators for this outcome involving metadata, ownership/copyright, management, digital imaging, digital preservation, and successful creation of digital objects and the uploading of multiple digital objects. Four of these indicators were covered in Year One of this project and three in Year Two. The three indicators that were evaluated in this project year involve the fundamental skills and issues that are necessary to create a digital project as well as the actual proof that they can digitize.

Four of the workshops included a post-test for participants to complete. We received 65 replies from all four workshops, on which 60 participants or 92% of them indicated that they knew less than 76% of the content covered in these classes: CONTENTdm, Digital Preservation Strategies and Sustainability: Business Planning for Digital initiatives, and Promotion and Use of Digital Projects, as indicated on the chart below. This helps to show that the majority of the content covered in these workshops was appropriate and new to them.

After the workshop on CONTENTdm, we received 20 post-tests on which an average of 91% of participants indicated that they felt highly confident in ten areas of the software application, CONTENTdm after the program, as indicated on the chart below. An exercise in class involved having participants use the software to upload digital images and metadata to test collections. A combination of in-class exercises and the confidence level of participants after the program indicated that 18 participants (an average of 91% of participants) could use CONTENTdm.

The fifth indicator states that “x number and x% of participants can demonstrate an understanding of the issues involved with digital preservation.”  After the workshop on Digital Preservation Strategies, we received 14 post-tests on which an average of 19% of participants indicated that they felt highly confident in five areas of digital preservation before the program and an average of 43% of participants indicated that they felt highly confident in the same categories after the program, as indicated on the chart below. The confidence level of participants after the program indicated that six participants (an average of 43% of participants) felt confident with the issues related to digital preservation.

 

After the workshop on the Sustainability: Business Planning for Digital Initiatives, we received 14 post-tests on which an average of 11% of participants indicated that they felt highly confident in five areas on digital imaging before the program and an average of 55% of participants indicated that they felt highly confident in the same categories after the program, as indicated on the chart below. The confidence level of participants after the program indicated that 8 participants (an average of 85% of participants) felt confident in the five issues related to sustainability of digital projects.

After the workshop on the Promotion and Use of Digital Projects, we received 15 post-tests on which an average of 5% of participants indicated that they felt highly confident in three areas on promotion before the program and an average of 69% of participants indicated that they felt highly confident in the same categories after the program, as indicated on the chart below. A combination of in-class exercises and the confidence level of participants after the program indicated that 10 participants (an average 69% of participants) felt confident in the issues related to promotion.

The average confidence level involving CONTENTdm, Digital Preservation, Sustainability and Promotion after the workshops was between 43% and 91%, up from between 5% and 19% before the workshops, as indicated on the chart below. 

The sixth indicator states that “20 and 83% of participants in the program attend a Mid-Term Review and successfully created 5 digital objects with metadata in CONTENTdm as part of HRVH.” There were 20 organizations represented at the Mid-Term Review on June 20, 2007, where they each demonstrated their progress with their digital collections. 

The seventh indicator states that “18 and 90% of participants continue to upload digital objects to HRVH with the guidance and assessment by the Digital Project Trainer.” This is the final, cumulative, indicator for the outcome: Staff from libraries and cultural heritage organizations enter objects independently into Hudson River Valley Heritage. In order to assess this last indictor we created a rubric to evaluate the skills of the participants in independently uploading digital objects to HRVH.  The rubric included a list of five skills necessary for digitization with five levels (1=low to 5=high ) for each category: Selection, Scanning/Image Quality, Metadata, Copyright, and Project Management. The average on the Evaluation Matrix for each skill is: Selection: 4.5; Scanning/Image Quality: 4.5; Copyright: 4.6; Metadata: 4.4; and Project Management: 3.3. The range for overall scores on the rubric ranges from 3.2 to 5.0 and the average for all 5 criteria for all 18 organizations is: 4.3. The fact that 18 organizations continue to digitize and add digital objects to HRHV with little or no assistance from the Digital Project Trainer, demonstrates that we successfully met the first outcome for this project.

The second outcome is: Researchers and students use Hudson River Valley Heritage to find historical materials about the Hudson Valley. There are three indicators to show if we met this outcome.  The first indicator is that “7 and 35% of staff at contributing organizations who promote and direct researchers and students to HRVH.”  Since several organizations continued to digitize past the six months that we initially provided to them, their promotional efforts were postponed, although seven organizations successfully distributed press releases or held events to promote their digital collections to researchers and students.  The Communications Plan for HRVH and bookmark for distribution were not complete in enough time to provide assistance to them with their promotional efforts.

The second indicator is that “47 and 31% of researchers who report using the digital images in HRVH” We used the comments section of the metadata record to record the number of researchers who used the digital images contributed by contributors who took part in this project. 

The third indicator is that “1 and 10% of students who use HRVH for school projects.”  Unfortunately these numbers do not indicate that students used HRVH.  The next LSTA project that we are implementing focuses 100% on students. 

The second outcome for this project was more difficult to accomplish. We learned, in the workshop on Promotion and Use of Digital Projects, that we needed a Communication Plan for HRVH in order to coordinate our efforts to get the message out about the content in HRVH and how useful it will be to researchers and educators and their students. We also learned that if we want educators and their students to use HRVH for school lessons, then we need to provide lesson plans. The next LSTA project, Learning Activities using Historical Materials, for April, 2007 – March, 2008, focuses on creating lesson plans based on historical materials in HRVH and testing them in the classroom.

  1. Please provide one or two anecdotes or stories about how this project affected people who received services.

  This program was an excellent, clear, well organized presentation which taught me how to do everything I had hoped to learn and more about mounting digital images on the website; information about metadata (very difficult), image editing, and project management was presented with great patience; excellent theory presentations explained valuable information about purpose, ownership of information and sustainability; I feel well equipped to continue my museum’s project especially with the well prepared printed information that was provided.
     This was a wonderful project, both in its vision and its execution; my personal expectations were well met by the entire program, but more importantly, HRVH has enabled me to organize a strategy for meeting the needs/expectations of the Bard College community; I used HRVH as my
starting point for organizing archival collections for presentation.
     I had no idea in the beginning that this was going to be such a massive and exciting project; I’ve learned a lot, starting from near zero on scanning, technical stuff; I’m still trying to put it all together so that it flows – rather than dragging with fits and starts.
     Not having a clear understanding of what I was going to learn, I did not have specific expectations; however, the quantity of every workshop was superb; all were quite knowledgeable, professional and accessible; overall the results are quite gratifying as we can all see our result/projects; to have had the standards presented to us all was excellent; now I have the goals!

 Is there anything else you want to tell us? 

The staff at SENYLRC observed a strong bond and camaraderie that formed among the participants in this training project. Most of them came to this training with similar levels of experience with digital projects and a commitment to digitizing the holdings from their organizations. They used each training program to catch up on the events at their respective organizations and to discuss issues related to their digital projects both one-on-one and as a group. The staff at SENYLRC held a final training program for the group in March 2007, where we reviewed the entire two year project, the results of their training and their digital projects to date.  We provided them with 1.5 hours to discuss their projects, including success stories or roadblocks that they were faced with.  I was impressed with how they used their time and for how long they wanted to talk about these issues. We reassured them that this final training program was to put closure to the grant funded portion of their training, but that their participation in the Hudson River Valley Heritage was just beginning. 

Quantitative Measures

1. Number of users served 

Unique visitors to digital collections implemented as part of this project           18,408

Number of comments made by researchers and educators                                     50

2. Number of bibliographies, publications, lesson plans, documents

or publicity prepared                                                                    6

(Specify) three Press Releases, one guideline, one Communication Plan, and one bookmark

3. Number of workshops/programs                                                          6

Total number in attendance                                                             138

4. Number of Websites or web pages developed                                            20

5. Other: Number of organizations with trained staff that contributed to HRVH           20

6. Number of digitized objects from the organizations with trained staff	            2,347



      

OBE Activities

Training Manuals

Certified OBE Trainers

OBE Training Plan

Links

OBE Best Practices home

OBE home

Go to Library Development home page


Last Updated: June 3, 2009