Making it REAL! Recruitment, Education, And Learning:
Creating a New Generation of Librarians to Serve All New Yorkers
IMLS Grant Partners Program Evaluation Workshops
Presentations, June 1-2, 2005
Overall NYSL IMLS Evaluation Plan as of June 1, 2005
REAP Change Consultants
June 2005
[Text version of presentation; also available in .PDF]
Background
- The New York State Library Request for Proposals to evaluate the overall IMLS Grant project required a specific
overall evaluation plan to be delivered by September 2005.
- This presentation represents current REAP Change Consultants evaluation planning and is subject to change.
Focuses of Overall Evaluation
- Scholarship student recruitment
- Students at entry
- Program during student education and teaching library experiences
- Partnering of library systems/library schools
- Student graduation and competency outcomes
- Project web site monitoring and user satisfaction
- Grant partner participation in annual conference meetings, learning, and satisfaction
- Publication by partners of “best practices” for replication
- Library board and library changes in recruitment-related activities
Scholarship Student Recruitment
- Partner questionnaire
- Telephone Interviews with partners about recruitment process
- Statistical analysis of past several years and current recruitment data (diversity focus)
- National literature review of diversity recruitment practices
- Now (June 2005) or when scholarship awards are done.
- May to June 2005 as needed.
- Library schools ONLY provide data in June/July 2005. REAP Change staff analyze. Whom should we contact?
- REAP Change staff, May to July 2005
- REAP Change recruitment report due August 31, 2005
Scholarship Student Recruitment Overall Project Evaluation Criteria
Teaching Libraries
- 15 MLS students receive scholarships
- 50 percent or more are diverse students (across all Teaching Library partners)
- Identification of best practices for recruiting for diversity
Library Schools
- 29 MLS students receive scholarships
- 82 percent or more are diverse students (across all Library Schools)
- Identification of changes in recruitment practices and of best practices for recruiting for diversity
Students at Entry
- Survey students at entry using quantitative and qualitative questions.
- This sets a "pre-intervention" baseline for each student.
- It also provides information from the students’ viewpoint about the recruitment process and recruitment targets.
- Partners provide student contact information after scholarships are given.
- September 2005 survey (Summer 2005 starts?).
- REAP Change analysis and short report by October 31, 2005.
- Updates in September 2006 annual report and in final report.
Students at Entry Evaluative Criteria
- See scholarship student recruitment output criteria.
- Evaluation design assumes a "self as control" approach – student progress later can be evaluated
against individual starting levels of knowledge, experience, opinions, attitudes, and reasons for entry.
- To protect individuals, only a summary of overall entry characteristics and later changes will be provided
in evaluation reports.
Programs During Student Education
- Annual interviews with partners in Summer 2005, 2006, 2007 about planned and actual support and activities.
- Annual surveys of students (at ends of academic years 2005/2006 and 2006/2007).
- Student focus group at NYLA 2006.
- Review by evaluators of written grant partner annual reports to the state coordinator and of program evaluation
reports done by the partners.
- REAP Change ongoing review of national literature on library education, librarianship, and internships.
Programs Evaluative Criteria
Teaching Libraries: "Students will demonstrate the appropriate competencies in a work situation related
to the needs specified by the sponsoring Teaching Library or system to the satisfaction of the monitoring library."
Library Schools: None specified by NYSL except…
Both: "Libraries, library systems, and library schools will report more frequent partnering to provide
internships and support independent learning projects to achieve specific skills and competencies in prospective
librarians."
Comments on Programs Evaluative Criteria
- A multi-site grant project is by its very nature expected to be somewhat experimental. General parameters are
set and people do their own thing to head in those directions.
- Your individual program evaluation criteria that you set are important. NYSL, IMLS, and REAP Change aren't
trying to set those criteria for you. If you need help with criteria setting, REAP Change can offer some advice
and technical assistance, but you need to set criteria for yourselves.
- The overall evaluation is formative in approach. Just be honest in reporting what is happening – we're trying
to learn and identify "best practices," and learn from one another. We cant learn if people aren't willing
to report problems as well as progress.
Partnering of Library Systems and Library Schools
- Overall grant expectations are not well-defined yet, so the evaluation plan for the partnering is not well-defined.
- Current evaluation plans are to use the annual interviews of partner staff and selected items in the student
surveys to gather information.
- This will be supplemented by participant observation or focus groups (or both) at the NYLA 2006 meeting.
Student Graduation and Competency Outcomes
- REAP Change reviews individual program evaluation plans, competency measures, and annual reports of Teaching
Libraries and Library Schools that will be made to the project coordinator.
- Library Schools provide student progress information (in aggregate or, after resolution of FERPA issues, for
individuals – e.g., transcript review?) Timing: June or July 2006 and 2007.
- REAP Change performs exit interviews with program dropouts when Library Schools or Teaching Libraries report
that a student has dropped out of school.
- REAP Change performs exit interviews with graduating MLS/MLIS students after they graduate.
Overall Student Graduation and Competency Evaluative Criteria
Teaching Libraries
- 15 scholarship students (100%) complete the MLS within the grant period (i.e., by August 31, 2007).
- Also see programs evaluation criteria.
Library Schools
- 80% (24 or more) of 29 scholarship students complete the MLS within the grant period (i.e., by August 31, 2007),
and
- 6 of 6 (100%) MLS degree students demonstrate competencies for library service to disabled persons according
to pre-established standards (SUNY Buffalo?).
- Also see programs evaluation criteria.
Graduation Outputs and Outcomes
Project Web site Monitoring and User Satisfaction
- Track web site usage statistics monthly once the web site is designed and operational.
- Results trended by month, analyzed and reported annually and in final report.
- Online survey to capture web site satisfaction with a random selection of users asked to complete it. Results
reported annually.
Project Web site Monitoring and User Satisfaction Evaluative Criteria
- Evaluative criteria are to be decided, preferably collaboratively.
- REAP Change proposes a 75 percent or more satisfaction rate with the web site as a "success" measure
of web site satisfaction.
- Alternative evaluation approaches and specific outcome criteria might be proposed later following further discussions
with NYSL staff and the web site consultant.
Grant Partner Participation in Annual Conference Meetings, Learning, and Satisfaction
- Online and paper surveys (in October or November) of students, faculty, Teaching Library and State Library
staff who have attended NYLA 2005 and 2006 about their learning and satisfaction.
- Participant observation and focus groups by REAP Change staff at NYLA 2006 meeting.
Grant Partner Participation in Annual Conference Meetings Evaluative Criteria
- At least one new concept learned (from the meetings) by each person.
- Proposed criteria for outcome success: 75% or more of meeting participants report satisfaction on each learning
opportunity question (to be developed) and 90% or more are able to write down at least one new concept learned
at the meetings.
- Output criteria (how many participate) to be decided. Expectation of 100% participation over the course of
the grant, or at all meetings?
Publication by Partners of "Best Practices" for Replication
- This multi-site grant is partly to try and model out "best practices" in diverse student recruitment
and MLS/MLIS training of diverse students, as well as new or innovative ways to deliver better library service
to diverse communities.
- Both IMLS and NYSL expect publications and conference presentations to result from the grant.
- Faculty, Teaching Library staff and students are all expected to participate in publishing and/or presentations
at conferences.
- REAP Change will distribute a form annually in November or December of 2005 and 2006 and in June or July 2007
asking partners to provide details on any presentations or publications related to the grant that were done since
the last report.
Publication by Partners of "Best Practices" Evaluative Criteria
- No evaluative criteria are set at present.
- REAP Change currently expects to report lists, counts and trends in annual evaluation reports and the final
evaluation report.
- Do NYSL or the partners want to set output and outcomes criteria?
Library Board and Library Changes in Recruitment-related Activities
- NYSL expectation of increases in library and Board recruitment activities related to diversity. No specific
criteria set.
- REAP Change will review copies of Library Board minutes from one year before the grant started through July
2007. Request will go out once a year, probably in July 2005, 2006 and 2007.
- Questions in annual staff/faculty partner interviews and at one focus group at NYLA 2006 also will seek evidence
of changes.