Making It REAL! Final Report

Making It REAL! Evaluation

August 16, 2007

Stephen C. Maack
Owner and Lead Consultant
REAP Change Consultants

consultant@reapchange.com
Telephone: (310) 384-9717

EAP Change Consultants
2872 Nicada Drive
Los Angeles, CA 90077-2024
Telephone: (310) 384-9717
FAX: (310) 474-4161
E-Mail: consultant@reapchange.com
Web Site: http://www.reapchange.com

This report also available in .PDF format .

Table of Contents Executive Summary Grant Overview Outcomes Based Evaluation Plans Overall Making It REAL! Outcomes Based Evaluation Plan Partner Outcomes Based Evaluation Plans Lessons Learned about Outcomes Based Evaluation Planning Scholarships Awarded, Enrollment, Progress Toward Degree and Graduation Outcomes Recruitment Phase Outcome 1 Lessons Learned Mid-Project Implementation Targets for Academic Progress Outcome 4 Mid-Project Implementation Lessons Learned End of Project Outcome: Library School Graduation University at Buffalo MIR Student Graduation Goals Lessons Learned about End of Grant Graduation Outcomes Diversity Goals Diverse Student Recruitment and Graduation Organizational Changes in Relation to Diversity Recruitment Making It REAL! Partner Best Practices Related to Diversity Recruitment Developing Competencies of Making It REAL! Scholars to Serve Ethnically Diverse, Disabled, and Otherwise Diverse Communities in New York State Organizational Best Practices in Preparation to Serve Diverse Communities Meeting Library Needs of Diverse Communities Meeting Needs of Diverse Communities by Focusing on Training Librarians in Specific Specialties of the Diverse Library Field	Teaching Library Goals Teaching Library and University Library School Interaction Expectations Mixed Teaching Library Results Feedback Loop 1: Practical – the library to the student and the student to the library Feedback Loop 2: Principles – the library school to the student and the student to the library school Feedback Loop 3: Collaboration – the library school to the teaching library and the teaching library to the library school Mentoring in Making It REAL! Teaching Library Best Practices Presentation and Publication Goals Notable Presentations or Publications Outside the Partners’ Systems Career Website Unexpected Consequences Financial Issues Partner Communication with Students About Scholarship Terms Not Enough Time To Do Everything Lack of Communication Among Partners Organizational Preparation and Capacity for Entrepreneurial Creativity Future Making It REAL! Impacts More Quality Professional Librarians in New York State Developing Organizational Impacts Current and Future Lasting Community Impacts Footnotes [NOTE: Footnotes appear in page when called, and are also shown at end of document.]
Figures Final Making It REAL! Grant Partners Methods Used to Evaluate Project Start-Up and Recruitment Phases Methods Used to Evaluate Project Mid-Implementation Phase Methods Used to Evaluate the Project End Phase Number of Making It REAL! Scholarships Awarded Annually per Grant Partner Outcome 1: Students Receive Library School Scholarships Outcome 4: Students Make Expected Progress Toward a Library Degree Self-reported Overall Grade Point Averages by Spring 2006 and Spring 2007 or at Graduation Full-time, Part-time and No Attendance by Partner Type and Term Comments on Working, Online Courses, and Going to Library School University at Buffalo Special Program Outcome Outcome 10: Teaching Library Scholars Complete Library Degrees Outcome 10: University MIR Scholars Complete Library Degrees Outcome 10: University at Buffalo Scholars Complete Library Degrees Outcome 12: Diverse Scholarship Student Recruitment Targets Other Kinds of Diversity among Making It REAL! Students Proposed Outcome 8: Teaching Libraries and Library Schools Report Increases in Activities Related to Recruiting for Diversity Degree of Student Preparation for Providing Library Service to Diverse Populations in New York State	Student Comments on the Relationship of Participation in the Making It REAL! Program to Preparation for Providing Library Services in Diverse New York Communities Outcome 9: MIR Partners Involve Students in Training and Activities Designed to Improve Service to Diverse Communities Outcome 7: NYLA Conference Workshops Outcomes Outcome 9: Partners Develop New Ways to Serve Diverse Communities Specialties of Making It REAL! Scholarship Students Types of Making It REAL! Grant Teaching Library Partners Partner Library School Requirements for an Internship or Practicum Library Schools or Departments Attended by Teaching Library and University MIR Scholars Faculty Mentoring MIR Student Satisfaction with Academic Advising MIR Student Comments on Academic Advising Did Students Have a Librarian Mentor that They Didn’t Have Before Receiving a MIR Scholarship? Satisfaction of MIR Students with Any Mentoring They Have Had Since Receiving a MIR Scholarship Student Comments about Any Mentoring Received Since Receiving MIR Scholarships Making It REAL! Student Satisfaction about Fiscal Matters Student Satisfaction about Communication with the Organization Awarding the MIR Scholarship Concerning Its Terms Student Comments about Fiscal, Payout, and Scholarship Terms Problems

Executive Summary

In 2004 the New York State Library (NYSL) embarked on an ambitious program titled Making It REAL! Recruitment, Education and Learning: Creating a New Generation of Librarians to Serve All New Yorkers. The Institute of Museum and Library Services (IMLS) funded the grant in 2004 under its Librarians for the 21st Century Program ^{Note 1} with $995,630 and New York State and grant partners provided matching funds of $645,058. The New York State Library, 12 library systems, and six of New York State's schools of library and information science formed a partnership to carry out the grant. Together they awarded scholarships to recruit and educate professional librarians, cultivated a diverse new workforce that is well prepared to serve community needs, especially those of diverse groups and special populations, and tested out “teaching library” approaches to provide applied experiences to students. Three years later the project is coming to a close and this final overall evaluation details experiences and outcomes of the project.

The Executive Summary is organized around outcomes identified in the Making It REAL! (MIR) grant project narrative under the “Evaluation” and “Impact” sections. ^{Note 2} The project narrative statements are in boldface, followed by the outcomes. The grant had many positive outcomes but the results are mixed. The main report provides greater detail including discussion of other outcomes indicators proposed in the REAP Change overall final evaluation plan to ensure a rigorous assessment.

Outcomes of Teaching Library Component

Fifteen students will enter programs sponsored by Teaching Libraries and complete the MLS within the grant period. Reasons for both success and failure will be gathered for publication.

This outcome was exceeded. Including replacements, 20 students entered programs sponsored by Teaching Libraries but three left the programs, leaving 17 scholarship recipients. Of those 17, seven have already graduated or will have graduated by the end of summer 2007, and another nine are expected to graduate by December 2007 (i.e., within the grant period). With 16 students who entered Teaching Library sponsored programs expected to graduate within the grant period, the target of 15 has been exceeded. The last student also expects to graduate, but after the grant period, in spring 2008.

Students will demonstrate the competencies in a work situation related to the needs specified by the sponsoring Teaching Library or system to the satisfaction of the monitoring library according to pre-established standards.

Based on Teaching Library partner Outcomes Based Evaluation (OBE) plan reports all students still in the program have met or are demonstrating competencies in work situations. Several partners report that scholarship students are far exceeding their expectations, particularly in regards to exhibiting leadership within public or school library systems. However, the monitoring library did not always explicitly state competency standards in OBE plans and evaluative evidence provided in reports was only sometimes, not always, gathered in a rigorous, systematic fashion. The two school library system (BOCES) did jointly design rubrics and evaluation data collection instruments to measure student progress according to pre-established standards.

Teaching Libraries will meet or exceed a target of 50 percent for diversity.

Using a broad definition of “diversity” appropriate to the New York State context the grant exceeded its target by selecting 71% diverse scholars. Of the final 17 students, four are ethnically diverse (24%), one is Gay/Lesbian/Bisexual/Transexual (6%), one is male (6%), and six (35%) are from rural or remote small town library areas that they intend to serve after graduation. All three of the students who left the program were also diverse (two ethnic minorities, one male, and one from a rural/small town area).

Both teaching and learning participants will report best practices for publication and replication.

This outcome was not well defined in the grant narrative and was for the most part not met if the intent was to report nationally for publication and replication. Only 8 of 22 partners interviewed (36%) had produced any presentations or publications related to the Making It REAL! program. Those were primarily presented within NY library systems or Councils, or at NY State conferences or venues such as Board or advisory committee meetings. About 29% of the Teaching Library MIR scholars had done a presentation about Making It REAL! by July 2007. Most of the student presentations were also done in internal settings, at library system meetings or events, or students published in system paper or electronic newsletters. Those students who had presented or published had often done so multiple times as part of partner OBE plan requirements. The purposes of student presentations included encouraging paraprofessionals to consider going to library school, and communicating selected best practices of librarianship, that the students had learned in their studies, to librarians or others (e.g., Board members) who are not professional librarians.

Outcomes of Scholarship Component

Twenty-nine MLS degree candidates will participate. Library schools will meet or exceed recruitment targets of 82 percent for student diversity.

These targets were not met but the outcome was positive. Only 28 degree candidates accepted scholarship recipients and entered library school, with 25 still enrolled or graduated as of July 27, 2007. When the diversity target is applied to 28 students who received scholarships and started library school, 21 (75%) are diverse. Of the 25 left in MIR, 18 (72%) are diverse. All of the University MIR scholars who left library school are diverse. The library schools often gave the scholarships to already admitted students who were ethnically diverse, but sometimes used the scholarships to recruit ethnically diverse students to their Master’s programs. Some of the private library schools that only had one or two MIR scholarships available to award were able to award other scholarships, or arrange for part-time work at the University for diverse students to whom they did not give a MIR scholarship. Therefore the existence of the scholarship had a positive impact on diversity of those schools even without award of MIR scholarships to all newly enrolled students. While not meeting the grant target, the diversity of the MIR scholarship recipients far exceeded a combined 12% ethnic minority enrollment rate or a 28% diverse student enrollment rate (including all males and ethnic minority females) posted by these six universities from 2000 to 2002.

Eighty percent of the students will complete the MLS degree requirements within the grant period. Exit interview data for students who do not complete requirements will be analyzed (e.g., some students who attend library school part-time will require a longer time to complete the MLS requirements).

Neither outcome target was met. Only 28 scholarships were awarded and three students dropped out, leaving 25 possible graduates. Of those 25, 16 graduated by spring 2007, two are expected to graduate in summer and three in fall 2007, totaling 21 graduates by December 2007 for a rate of 21/28 = 75%, below the 80% target. Three additional students may graduate after the grant period, and graduation expectations for the other are unknown.

Among both University and Teaching Library MIR scholarship recipients, those who did not complete requirements did not make adequate academic progress to stay in school or had personal reasons for dropping out (e.g., marriage in a foreign country, having a baby, or health related reasons). Several prospective scholarship recipients turned down scholarship offers for personal reasons (e.g., the loss of health benefits and income if they dropped back from full-time to part-time work while going to school), or because they did not like the scholarship terms (notably the requirement to work in New York State for two years).

Teaching Library MIR scholars attended library school part-time and in summers. University MIR scholars were more likely to go full-time in fall and spring terms and not at all in summer. University MIR scholars accumulated more units toward graduation in a shorter period of calendar time than did Teaching Library MIR scholars. Larger proportions of University than Teaching Library MIR scholars had graduated by the end of spring 2007.

Reasons for delayed graduation included lack of sufficient funds leading to part-time attendance and childbirth leading to a temporary stopping out of school. University MIR scholars often had partial scholarships. Sometimes Universities found additional scholarships or part-time jobs at the University to supplement their incomes. In contrast, Teaching Library MIR scholars tended to work full-time or longer hours while attending school part-time (often online). MIR scholars of either type generally agreed that working for pay while attending school was difficult or very difficult. Those trying to work full-time, go to school part-time, and take care of families or maintain a personal life reported extreme time stresses. Some Teaching Library MIR scholars went to extraordinary lengths to continue in their schooling and complete their degrees.

Changes in recruitment practices to improve diversity in recruitment will be documented.

Recruitment practices used were documented in a recruitment survey and in interviews. Partners were not always successful in improving diversity recruitment. There was confusion among some partners at the beginning of the project concerning the diversity goal of the grant and some Teaching Library partners had completed scholarship recruitment without considering ethnic diversity at all, focusing on seeking librarians who would fill certain librarianship specialties in short supply in their areas.

Universities often awarded scholarships to ethnically diverse students who had already been admitted, but two used the grant program to actively recruit students who would otherwise not attend. Palmer School was able to help one student “stack” scholarship aid from other sources, including an ALA Spectrum scholarship, on top of her partial tuition MIR scholarship to give her a “free ride” at a private school. Syracuse University was able to find other scholarship and part-time job resources to enroll two African American students who had been attracted to apply by the MIR scholarship, but were not awarded it. After success with two diverse MIR students, St. John’s University is developing new ways to recruit ethnically diverse students from within its University undergraduate population and in ethnically diverse neighborhoods, as well as from diverse Queens Library paraprofessional ranks. Advertising and placing stories about library services to diverse communities in the many small community newspapers in the diverse community around St. John’s is one approach.

Teaching Libraries often used recruitment strategies designed to seek candidates from paraprofessional or administrative staffs within their systems or Councils. Some advertised for candidates from the outside community. In less ethnically diverse parts of the state often no ethnically diverse candidates applied for scholarships. Two Teaching Libraries sought advice from the University at Albany Dean or the Syracuse University MLS program Director about scholarship criteria to use, and one had the Syracuse Master’s program Director sit on the scholarship selection committee. Since Teaching Libraries are not experienced in running scholarship award programs, several that did not seek University advice recommended that in the future Universities work more closely with them to set criteria and approaches for identifying quality candidates for scholarships.

Six MLS degree students will demonstrate the competencies for library service to disabled persons to the satisfaction of library school faculty according to pre-established standards.

This target was not met, although not for wont of trying. The target applies only to the specific program of the University at Buffalo to educate students in how to create web sites that give disabled people better access to library services. The program included creation and offering of a pilot course in library web design for disabled access. The course was created and offered in spring 2006. Two students failed it and one subsequently left the MIR program for personal reasons. The other should obtain a MLS. However, only four of the original six students who obtain the MLS will have demonstrated the competencies for library service to disabled persons. Also, because of a lack of libraries volunteering their websites for students to practice what they learned, only two students have so far completed a University at Buffalo program expectation that they would demonstrate application of skills learned by providing or recommended improvements in library websites for better disabled access. The other two students are still seeking to meet the faculty pre-established standard of demonstrating their new skills in an applied setting.

Outcomes of Support Activities to Strengthen Overall Library Education

Best practices for recruiting students for diversity will be identified.

Best practices identified include:

• personal contact;
• provision of fiscal and other support;
• presentation of diverse role models to potential recruits in print and in person;
• enrolling a group of diverse students and forming them into a cohort for self-support;
• providing examples of service to diverse communities by librarians (preferably diverse librarians);
• being aggressive in talking to potential diverse candidates about librarianship as a career;
• presenting a positive image of librarians as service providers;
• advertising in local community newspapers in diverse neighborhoods; and
• encouraging paraprofessionals and non-degreed librarians to obtain a MLS for career advancement.

Many of the approaches tried by partners are not original and were outlined in an article that was available before the MIR scholarship recruitment period started. It is not clear how many partners consulted that article.

In regards to providing financial support, Palmer School leveraged $30,000 of IMLS grant funds with a $15,000 match over two years into a Long Island University promise of $80,000 of scholarship funds annually for increasing diversity of Palmer School students. St. John’s obtained a new IMLS grant to recruit 40 new students in two cohorts from low-income, international, and other under-represented groups, including those with marginal academic qualifications, to its library Master’s program. St. John’s modeled its application on MIR.

Tracking of students recruited through the project will demonstrate project success through employment in library careers.

This outcome cannot be completely measured since the overall evaluation is ending before the December 2007 grant deadline for graduation. NYSL plans to track this outcome over the next five years. Of 21 MIR scholars who have graduated, or are about to do so in summer 2007, 16 (76%) had looked for library jobs in New York State. Three of the five who hadn’t were already employed by NY State libraries and were continuing with the employers they had before going to library school. Of the graduates, 12 (60%) had found a library job somewhere, two in the NY City boroughs, four in the Greater NY City area, two in a major urban area of upstate New York, two in upstate or western New York suburban areas, and only one outside New York. Those graduates with professional jobs obtained them either within three months of graduation or before graduating.

Both potential employers and potential employees have choices regarding obtaining library jobs and advancing in library careers. Although the plurality (43%) of graduated students consider it “neither difficult nor easy” to find a librarian job in New York State, 38% think it will be difficult or very difficult, compared to 19% who consider it easy or very easy. Some students described the New York librarian market as “competitive.” Several factors may interfere with employment in library careers that have little to do with the grant or preparation of MIR students for successful library careers. One of the complications includes jobs not being available. However, some Teaching Library partners in system level positions in more remote or rural parts of New York State detailed exactly how many current and expected librarian job opportunities would become available over the next few years as people reached retirement age. Another issue is that MIR candidates can choose which jobs they wish to seek and where, and some may chose not to apply for certain kinds of available jobs. Finally, employers may choose not to hire even well-qualified library school graduates who had received MIR scholarships.

Libraries, library systems, and library schools will report more frequent partnering to provide internships and support independent learning projects to achieve specific skills and competencies in prospective librarians.

This target was not met for most partners. Only one Teaching Library sought out one University MIR partner to discuss and negotiate on selection of appropriate internship sites for its MIR scholar. Queens Library sent its MIR students to Queens College (not a MIR grant partner), and did communicate with faculty there about applied experiences it was giving its scholars, as well as academic progress of its scholars.

Two rural public library systems originally planned to talk with Syracuse University about more course coverage concerning rural or small libraries, but delayed the discussion until after their students could provide feedback on courses that were different from those the partners had taken in library schools decades ago. Teaching Libraries also usually did not communicate amongst themselves. Howevere, the two BOCES SLS systems did write and implement a joint OBE plan, created progress measurement rubrics, and communicated some during the grant. The students sponsored by these two Teaching Libraries progressed through their programs at different rates so the partnership cooled later in the grant period.

Often Teaching Library partners and library schools did not communicate at all except about financial aspects of scholarship payments, University regulations or other purely procedural matters. Teaching Libraries in rural or small town parts of the state sent their students to three out-of-state universities that offered web-based online programs as well as to MIR partner universities. Universities did not always require internships for MLS degrees. Universities such as Syracuse, University at Albany, and St. John’s that had their own MIR scholarships and also had Teaching Library MIR scholars in their programs only wrote OBE plans and made special efforts concerning their own students and did not include the Teaching Library MIR scholars in special program related actions.

Web site activity will be monitored and user satisfaction assessed.

After considerable discussion concerning the content and focus of a planned website, the New York State Department of Education awarded a contract ^{Note 3} for website creation and librarycareersny.org was announced as open to the public in mid-April, 2007. An online survey was used to assess user satisfaction. A MIR student also provided disabled user accessibility tests of the site. Results were generally positive, with some improvement suggestions, but are not fully discussed in this report. For details on the evaluation of the website, see Cook, S. (August 6, 2007). Website Evaluation: Evaluation of librarycareersny.org. Los Angeles: REAP Change Consultants.

Participants in annual meetings will report satisfaction with learning opportunities and will be able to identify when asked at least one new concept learned.

NYSL put together three grant funded diversity-related workshops at NYLA 2005, and one diversity-related workshop, one diversity-related session, and one session concerning “teaching libraries” at NYLA 2006. Participants were surveyed for satisfaction on several measures immediately after participation and were satisfied with most aspects of all but one session. Most could report at least one new concept learned.

Libraries and library boards will report increases in recruitment-related activities.

After a meeting with partners on June 1 and 2, 2005, NYSL decided not to measure this vaguely stated outcome since a) not all of the partners have library boards, and b) recruitment opportunities varied depending on funding variability that was outside the control of the grant or the partners. Many Teaching Library partners reported limited or no recruitment during the two year grant period. Others indicated in final interviews that they recruited with attention to diversity as they always had. One Teaching Library partner reported that the Board Chairperson of a small local public library had gained a new appreciation for what a professional library degree offered after the Director of that small library became a scholarship recipient and began applying at her library knowledge learned in library school and in applied experiences elsewhere.

Impact

One library school will develop a program to recruit MLS students who will serve people with disabilities. It will develop competencies for the coursework and identify what learning experiences have potential to achieve these competencies.

The University at Buffalo developed such a program and piloted a special course on developing library websites with improved access for disabled populations. The University program director attempted to locate students with disabilities to enter the program, but had little success. Similarly, when she sought out library or non-profit sites at which the course students could demonstrate their new skills in designing and testing websites for better access by the disabled, only one library offered its site. Also, agencies that serve the disabled were contacted locally and nationally about possible joint projects but did not respond. The reasons for the lack of response are not clear.

The Teaching Libraries, in addition to recruiting for diversity, address a specific local need (e.g., the need for school library media specialists in urban school libraries and for librarians to serve in remote rural areas).

Teaching libraries frequently awarded scholarships under their control only to individuals interested in specific specialties, including school library media and children’s or Young Adult librarianship. They uniformly succeeded in choosing candidates for such specialties, and MIR scholars with specialties at the beginning of library school typically did not change their specializations. Many of the new librarians who have or will soon be entering the profession are doing so with specialties that are currently in short supply in New York State, including:

• Children’s Services (5 new professionals)
• Young Adult Services (3 new professional)
• School Library Media Specialists (10 new professionals)
• Digital Librarians (3 new professionals)
• Rural or Small Libraries (1 new professional plus several of the 9 interested in “Public Libraries” who plan to end up serving in rural, small town, or remote areas of New York).

The statewide career web site will help to recruit students from diverse backgrounds to fill the library service needs of diverse populations.

The librarycareersny.org website was created later than anticipated and has only been advertised since April 2007. However, by August 2007 Queens Library, New York Public Library, library schools at St. John’s University and University at Albany, and the Palmer School were already using the site to help recruit students from diverse backgrounds, educate them about library careers and how those could serve the library needs of diverse populations. A usual practice is to link the website to the organization’s intranet and place the URL on its own web pages about library careers and jobs.

The project will establish Teaching Library models that benefit library students by enabling and broadening their learning experiences while the students serve the needs of the communities served by the libraries.

The teaching library concept was not well defined in the grant. During the grant process, Assistant Professor Suzanne M. Stauffer of Louisiana State University developed a theoretical model of components of a “teaching library,” based on the history and experiences of library education in the United States, that has three interacting feedback loops:

• Loop 1: Practical – the library to the student and the student to the library;
• Loop 2: Principles – the library school to the student and the student to the library school;
• Loop 3: Collaboration – the library school to the teaching library and the teaching library to the library school

The model was discussed by MIR partners and students at NYLA 2006 and felt to have value.

The overall evaluation determined that the main MIR innovations in the teaching library model came in loop 1, in the types and depths of practical experiences given to MIR students as individuals in libraries, and the number of different types of libraries in which some students were placed. When asked to compare what happened with MIR scholars to what normally happened with an “intern,” grant Teaching Library partner representatives spoke of giving the scholars more responsibilities than typically expected of interns, and providing closer mentoring. There were also a variety of models in how the grant partner Teaching Libraries (which were generally public library systems, school library systems, or Regional Resources and Reference Councils) selected and worked with individual library sites to provide the MIR students with good practical experiences tailored to the individual students.

There were few innovations or new models in feedback loop 2, although some partners who were Deans or Directors of library schools or MLS programs reported more mentoring involvement with MIR scholars than they typically held with library school students.

Feedback loop 3 also had very few innovations by partners, and was the one that those present at the NYLA 2006 session by Dr. Stauffer were least likely to have experienced. Those collaborations that did occur came about as the result of pro-active behavior by Teaching Library partners. They also elaborated on pre-existing individual friendships between Teaching Library staff members and library school Deans, Directors or faculty. The existence of the grant acted as an occasion for further collaboration among friendly colleagues, but did not lead to new such collaborations with library school faculty or staff who were not already personal friends of the Teaching Library partners.

The project will help to fill library service needs in the communities served by New York's library systems and will address issues facing system member libraries.

The report lists specific current and expected continuing impacts of the grant on local community services by libraries. The list includes awards to SENYLRC for its Hudson River Valley Heritage project and to NYPL-branches for its continuing “Anti-Prom” event. Rochester Public Library has a new local author’s database as a result of the grant. Other examples include professional advice regarding small library collection development and innovative programming related to diversity at several local libraries. New York Public Library – SIBL has developed a “Getting Down to Business Toolkit” that highlights its innovative business library service model.

The Teaching Library component will provide models for other State Library Agencies and for schools of library and information science.

One model demonstrated by the outcomes of this grant is that Teaching Libraries can be as successful, or more successful, than library schools in locating and encouraging diverse individuals to attend and graduate from library school. While it can be time consuming, and takes longer to complete a degree overall, using web-based, online distance learning on a part-time basis while working full-time or part-time is a viable option for rural or remote scholarship recipients who are too far away from any library or information science school to attend in person. However, the Teaching Libraries indicated that they are not experienced with choosing scholarship recipients and could have benefited from closer cooperation and sharing of advice from schools of library and information science about how to select successful students, and what criteria the schools consider important for academic success in particular specialties. Some Teaching Libraries were critical of library school practices in placing students for internships and in making sure that local library sites provided a good applied learning experience for interns. Teaching Library partners would have liked a closer consultation with library schools about placement and experiences expected of their scholarship students, including attention to what the public and school library systems or regional councils hoped their scholarship students would gain from experiences.

Partners expect a State Library to take more of a pro-active and directive leadership role if similar grants are done again. In particular, the State Library might require that partners, especially Teaching Library and University partners, but also similar Teaching Library partners, communicate and work more closely together. Key times for Teaching Library-University library school collaboration are before and during recruitment and at internship placement times, although sometimes opportunities arise for a partner library site to be used as a “case example” for class projects. ^{Note 4} The State Library also should design and fund occasions to bring partners and students together, as a grant requirement. Students, who value the experience of attending a professional conference, might meet at a professional conference such as NYLA. However, many partners have multiple responsibilities at professional conferences that make their meeting at them impractical.

Grant overview

This is the final overall evaluation report to the New York State Library (NYSL) of the program Making It REAL! Recruitment, Education and Learning: Creating a New Generation of Librarians to Serve All New Yorkers. The Institute of Museum and Library Services (IMLS) funded the NYSL program in 2004 under its Librarians for the 21st Century program (Award Number RE-01-04-0053-04), with $995,630 of IMLS funds and a state match of $645,058 (press release of July 13, 2004). New York State Librarian and Assistant Commissioner for Libraries Janet Martin Welch announced receipt of the grant on July 23, 2004 as Making It REAL! Recruitment, Education and Learning: Creating a New Generation of Librarians to Serve All New Yorkers, and designated Mary Linda Todd, Library Development Specialist in the Division of Library Development at NYSL as the Project Director (retrieved July 17, 2007 from http://www.nysl.nysed.gov/libdev/imls/announce.htm). This grant proposal originated as a step toward realizing recommendation nine of the Regents Commission on Library Services New Century Libraries legislative initiative (retrieved July 17, 2007, from http://www.nysl.nysed.gov/rcols/finalrpt.htm#Recommendation9 and http://www.nysl.nysed.gov/libdev/imls/rec9.htm).

The purpose of the Making It REAL! (MIR) program has been to recruit and educate a next generation of librarians in such a way that diversity in the profession will be enhanced and new librarians will be well prepared to serve diverse populations. The New York State Library, library systems and Councils throughout the state (referred to as “Teaching Libraries”) and New York State schools of library and information science, along with the New York Library Association (NYLA), formed a partnership to carry out such a plan. Together the Teaching Libraries and library schools have awarded scholarships funded by the grant to recruit and educate professional librarians, and cultivate a diverse new workforce that is well prepared to serve community needs, especially those of diverse groups and special populations. The Teaching Libraries and library schools were expected to build strong partnerships and develop new means of collaboration to develop new strategies and alternatives for library education.

As planned, the original project components, to be carried out over three years, were as follows:

• Thirteen library systems and councils, partnering with six library schools, would collaborate in Teaching Libraries across the state. The library systems would provide practical, hands-on experience for librarian recruits in a Teaching Library selected from among the systems’ member libraries.
• Six of New York’s ALA-accredited library schools would participate in a scholarship component. The scholarships would enable the schools to recruit students who will represent diverse groups and acquire competencies for serving diverse populations.

In addition, the project partners were to initiate activities to strengthen library education overall, enabling all MLS degree candidates to benefit. Other anticipated activities included construction of a career website; implementation of a public-relations campaign to reach out to diverse populations and raise statewide awareness of the project; annual meetings of recruits and project participants; and publication of project experiences and outcomes.

The original plans were that the grant would provide financial support to 44 students. ^{Note 5} It would contribute to developing and testing model Teaching Libraries and support a rigorous evaluation process that would complement the Outcome-Based Evaluation Implementation Plan of the State Library. It would also assist in building statewide resources to enhance workforce recruitment, including the career website, the public-relations initiative, and events for participants.

The grant funded annual scholarships in the range of $5,000 to $25,000 (if awarded through Teaching Libraries), or $5,000 to $30,000 (if awarded through library schools). ^{Note 6} Scholarship recipients signed contracts that they would work in a professional librarian or informational science position in New York State for at least two years of the five following their graduations with MLS or similar Master’s level degrees in library or information science. ^{Note 7}

The New York State Library and an Advisory Group representing different types of libraries and regions of the state were to oversee project activities and administration. They were to support the activities outlined above and develop plans and alternatives for continuing the project after the three-year grant period.

After one of the original Teaching Library grant partners ^{Note 8} dropped out before awarding any scholarships, the final set of partners that implemented the grant are shown in Figure 1 in alphabetical order.

Although in their original proposals to NYSL, the partner Teaching Libraries had proposed library school partners, neither Teaching Libraries nor universities were required by NYSL to team up together in any particular way. NYSL did require that scholarship recipients attend a library school that was accredited by the American Library Association. Part of the reason for not restricting where the students could go to school was that good universities in various parts of the country are now offering online Master’s degree programs that might be of interest to some of the New York based MIR scholars who live in more remote parts of New York State and have no library school near them. Some Teaching Libraries required that students to whom they awarded scholarships be admissible at and attend specific library schools, while others left the final choice of library school attended up to the scholarship recipient but offered advice or guidance on possible library schools to attend, and helped obtain information about those schools. As will be discussed, some scholarship recipients attended other library schools than those directly involved with the Making It REAL! program, including three library schools outside the state of New York that offer completely online Master’s degree programs and the only library school in New York State that was not a Making It REAL! partner.

Figure 1. Final Making It REAL!Grant Partners

Outcomes Based Evaluation Plans

The original IMLS proposal required two kinds of outcomes based evaluation (OBE) plans. The first consisted of a set of outcomes based evaluations to be planned and completed by each of the teaching library and library school grant partners concerning their own specific programs. The second consisted of an overall outcomes based evaluation of the entire grant project by an external consultant, which REAP Change Consultants (REAP Change) was contracted to complete under New York State, Department of Education Contract C007802. This report comprises the final evaluation report of the overall REAP Change evaluation of the grant.

The Making It REAL! grant was the first competitive grant of this size that New York State Library, Office of Grant Development had obtained and it was the first time that NYSL attempted an outcomes based evaluation for this kind of a grant. NYSL has more experience handling pass-through funding programs with more specific reporting and evaluation criteria, including IMLS funded technology training projects that now require Outcomes Based Evaluations. This grant-funded project has considerably fewer specific IMLS and NYSL expectations and required program actions than either NYSL or the grant partners (especially the Teaching Libraries) are used to managing. It was also the first time that NYSL has used an outside evaluator, such as REAP Change Consultants, to evaluate a program of this size.

Overall Making It REAL! Outcomes Based Evaluation Plan

REAP Change is responsible for the overall Making It REAL! Outcomes Based Evaluation. The REAP Change team originally included Dr. Stephen C. Maack, owner and lead consultant of REAP Change Consultants as team leader; Dr. Clara M. Chu, a tenured Assistant Professor in the UCLA Information Studies Department and an internationally recognized expert on diversity in librarianship, as a sub-contractor; and Dr. Suzanne M. Stauffer, a librarian with extensive experience in public, academic, and special libraries in New York and California, as Senior Research and Evaluation Specialist with REAP Change. During summer 2006 Dr. Stauffer left REAP Change employment to become a tenure-track Assistant Professor at Louisiana State University, School of Library and Information Science. Lead evaluator Maack took on more evaluation responsibility and eventually sub-contracted with Ulia Gosart and Sam Cook to help complete the last two months of the evaluation work.

The proposed evaluation approach included a multi-methods (quantitative and qualitative), multi-site, multiple source evaluation that would provide outcomes based evaluation results, as well as a process evaluation for formative and summative use. It was meant to provide an evaluation for the intended use of intended users. Evaluation reports, including this one, were to include “Lessons Learned” sections with evaluator comments and occasional recommendations for consideration by NYSL and IMLS, especially in the event of similar future grant applications. The general evaluation approach was laid out in the bid for contract, and then partially detailed further for presentation at workshops held on June 1 and 2, 2005 (http://www.nysl.nysed.gov/libdev/imls/workshop/evalplan.pdf, retrieved July 25, 2007). The evaluation plan was revised and presented to NYSL by the contracted deadline of September 21, 2005, then revised again following comments from NYSL and program changes (http://www.nysl.nysed.gov/libdev/imls/evalplan.pdf, retrieved July 25, 2007).

While the written overall evaluation plan was not revised again after February 15, 2006, the overall evaluation was implemented differently after Dr. Stauffer left REAP Change employment. Before the change in staffing, implementation had proceeded essentially as planned at the beginning of the overall evaluation, as shown in Figure 2.

Figure 2. Methods Used to Evaluate Project Start-Up and Recruitment Phases

REAP Change experienced some delays in implementing planned data collection in spring 2006, and New York State Library agreed to collect progress reports that year. Also, it turned out to be very difficult to get the few program leavers to talk to the evaluators, so some of the information on program leavers came from partners or NYSL staff interviews or student e-mails to the partners or NYSL staff or to the evaluators. The evaluation methods used for looking at what was happening in mid-implementation of the Making It REAL! program are shown in Figure 3.

Figure 3. Methods Used to Evaluate Project Mid-Implementation Phase

The originally planned and actual evaluation methodologies for the end of the Making It REAL! project are shown in Figure 4. By spring 2007 the three-person evaluation team was in practice down to the lead evaluator, Dr. Maack. Since Dr. Maack would have been unable to accomplish all of the remaining evaluation work by himself, the REAP Change team was augmented by adding sub-contractor Ulia Gosart to expand, revise, and reorganize a bibliography on “Diversity” that Dr. Chu had completed in summer 2005, and to expand and reorganize the bibliography from Dr. Stauffer’s NYLA 2006 presentation on “Teaching Libraries.” These were to provide additional context and summative evaluation deliverables, with only the “Diversity” bibliography originally promised. Sam Cook, who had graduated from the University at Albany Library School at the end of the spring 2006 term and left NYSL employment related to the MIR project at the end of August 2006, was sub-contracted to evaluate the grant funded Library Careers New York website at http://librarycareersny.org/ (retrieved July 25, 2007) that had finally been completed in late Winter 2006/early spring 2007. He also has provided invaluable logistical support and report writing/editing services for this final report. As planned, Dr. Chu has provided excellent review and editing assistance for this final report.

Dr. Maack thanks the REAP Change team members for their many contributions to this evaluation for their excellent work throughout this project. The overall evaluation could not have been completed without their help, although any mistakes in this final report are the fault of the author. Dr. Maack also thanks Making It REAL! Project Director Mary Linda Todd of NYSL, Division of Library Development, for her patience, input, feedback, and cooperation during a long evaluation. Finally, this final overall evaluation report would not have been possible without the frank comments and contributions of the contact people at the 19 Making It REAL! partner organizations, and the Making It REAL! scholarship students.

Figure 4. Methods Used to Evaluate the Project End Phase

Partner Outcomes Based Evaluation Plans

On November 6, 2003 an e-mail was sent out over the NYLINE e-mail network inviting library system, Council and library school proposals to NYSL for an eventual IMLS grant proposal. There was about a two month time period between the original e-mail solicitation of partner proposals and submittal of the Making It REAL! grant proposal to IMLS. All 14 projects proposed by 13 library systems/councils ^{Note 9} and six proposals from public and private library schools in New York, and the NYLA proposal were accepted and included in the NYSL grant proposal submitted to IMLS on January 9, 2004. IMLS funded the project in July 2004. Scholarship student recruitment proceeded primarily in summer and fall, 2004 but also into spring and summer 2005, with additional grants awarded in fall 2005 to recycle scholarship money from one partner who withdrew, ^{Note 10} and additional recruitment activity to replace students who turned down scholarships that they had at first accepted. The REAP Change team was contracted on April 19, 2005 to do the overall evaluation and provide technical assistance to partners who would produce their own OBE plans for their own programs.

NYSL felt that all partners would want or benefit from developing their own OBE plans since they might have their own particular expectations of their own involvement in and contributions to the program. While most of the grant partners had participated in NYSL funded training in IMLS OBE approaches in spring 2005 or earlier, REAP Change included a brief refresher session ^{Note 11} on outcomes based evaluation as part of two one-day workshops that were held on June 1 and 2, 2005. ^{Note 12} Ten of the 12 teaching libraries, five of the six library schools, NYLA and NYSL sent representatives to these workshops. ^{Note 13} On each day teaching libraries as well as library school representatives were present. At the end of each day partners started from their original grant proposals to NYSL and worked in small groups with REAP Change team members to begin fleshing out their own program OBE plans. The REAP Change team members rotated around among the groups so that the partner representatives present would get the perspectives from different REAP Change team members with different library, library school and evaluation backgrounds and expertise.

After the workshops the REAP Change team was divided up among the partners so that each partner had a designated overall evaluation team member available to provide technical assistance, telephone and e-mail consultation, review and feedback on the developing specific partner project evaluation plans. Dr. Maack acted as the general evaluation expert, available for additional consultation by any partner and either of the other two members of the REAP Change team, who were less experienced in program evaluation. The partners were held responsible for creating their own Making It REAL! program OBE plans using the forms that had been used in NYSL provided OBE training earlier in the year. They had to specify the anticipated program inputs, outputs, outcomes, indicate how and when these would be measured, what the indicators were and what target measurement level would need to be reached to declare the program a success. The partners were also instructed that they would be responsible for tracking and gathering evidence on progress toward meeting the outcomes of their own individual programs. The REAP Change team would be primarily responsible for the overall grant program evaluation, although it offered to provide technical assistance to the partners, if asked, on developing or perfecting evaluation data collection instruments. ^{Note 14} NYSL asked the REAP Change team to take responsibility for approving individual partner program OBE plans as the best that the partners could produce, before these were forwarded to NYSL as the partner individual OBE plans. The intention was to have individual partner OBE plans completed by July 15, 2005. While most of the partners had their plans done by the target date, some took as long as December 2005 to finish.

The New York State Library collected progress reports from partners, including progress on their OBE plans, in spring 2006. REAP Change Consultants collected final OBE plan progress reports in June and July 2007. General comments about the entire set of partner OBE plans are made here.

The REAP Change evaluation team found it difficult to get some of the partners to look at Making It REAL! as much more than a scholarship award project. Even though REAP Change had presented a Making It REAL! program overview in the workshops that included mention of a possible focus on organizational changes in relation to diversity and teaching libraries as opportunities for grant program participation, ^{Note 15} and the REAP Change team brought up the possibility of including organizational activities, outcomes, and targets in the partner OBE plans, most partners did not do so. During OBE plan technical support discussions partners often referred the REAP Change Team back to the original individual partner proposals that referenced recruiting library school students to meet specific library staff shortage needs in the local areas, or recruiting ethnically diverse students. Both teaching library and university partners therefore felt it was sufficient to produce outcomes based evaluation plans that focused on what was happening with the scholarship students, and frequently on outcomes that the students themselves would achieve, more than on what the organizations would achieve to assist the students in their education. Universities in particular, but also some Teaching Libraries, frequently set outcomes and outcome indicator targets that simply involved typical measurements of progress toward the Master’s degree at different stages of any student’s library education, without specification of any organizational activities other than one might normally find in a library school graduate program. The overall quality of the set of partner OBE plans suffered as a result.

Another problem with partner OBE plans is that 11 of the 12 teaching library partners and two of the six University partners had only one or two Making It REAL! scholarships available to award (see Figure 5). These partners viewed the grant program as small from their perspective, both in terms of dollars expended and people affected. Many therefore felt that the time needed to create and track OBE plans just for their small programs was a burden given the size of their individual programs. ^{Note 16}

Figure 5. Number of Making It REAL! Scholarships Awarded Annually per Grant Partner

The distribution of scholarships was based on the original proposals from the partners to NYSL and included considerations such as the higher cost of awarding scholarships to cover tuition and fees at private university partners (i.e., Palmer, Pratt, St. John’s and Syracuse University) and whether partners intended to offer scholarships covering full or partial tuition. In addition, the University at Buffalo, a publicly funded (and thus less expensive) State University of New York institution, started with six scholarships designed for use in educating librarians who could provide web accessible websites for libraries, and then during mid-fall 2005 received another five scholarship awards following the pull-out of a teaching library partner freed up scholarship funds that could be moved to another partner.

When Teaching Libraries and universities only had one or two scholarship recipients the implications for OBE plans that focused on what those students did was that there was no variation or almost no variation possible in outcomes. The students would either succeed or not succeed. Despite OBE training and continual encouragement and coaching from the REAP Change team to distinguish outcomes from outputs, it was much more difficult for the partners to distinguish “outcomes” from “outputs” with OBE plans focused on only one or two participants, and attribute those outcomes to the program rather than to unique characteristics of the individuals in the program. Some partners and REAP Change team members discussed ways to set measurable outcomes rather than outputs of activities that students were expected to undertake, but even then, accomplishing those outcomes for the partner would be dependent on the success of the one or two scholars sponsored by that partner. Teaching Libraries in particular often indicated to the outside evaluators during the process of writing OBE plans that the evaluation approach seemed like overkill with so few scholarship students involved. In exit interviews Teaching Libraries would rave about their scholarship recipients and how well they were doing, then note how lucky they were to have recruited those particular students, or comment on some individual student characteristic that helped that student succeed. REAP Change agrees that there were too few scholarships awarded per partner to do a good outcomes based evaluation for most of the partners.

Also, most partners had finished scholarship recipient recruitment before they finished their specific OBE plans. A few Teaching Library partners noted in final interviews in June or July 2007 how the success of their OBE plans had hinged on their success in attracting and picking out exceptional Making It REAL! scholarship recipients, and wondered out loud if their programs would have gone as well if they had chosen less adept and appropriate scholarship recipients. Some had included OBE plan measures for a recruitment process that was finished before the OBE plans were written, and reported out their indicators and measures. However, only one or two partners provided clear evidence of a thorough “logic model” for how the scholarship selection process might yield potentially successful library school and program candidates that had been developed before recruitment. ^{Note 17} This was more of an issue for “Teaching Libraries,” who had little or no experience in recruiting and selecting among scholarship candidates, than for the partner Library Schools that handle selection of students and awarding of scholarships annually.

Many of the Teaching Library staffs involved with the Making It REAL! project were also writing OBE plans for the first time. Some appreciated the personalized coaching provided by the REAP Change team, and worked hard to produce two or three drafts before their liaison would accept an OBE plan as sufficient, while others resisted the process. Frequently OBE plans would include outcomes that could not even be measured until well into the grant project (e.g., what happened during student’s library school internships) or even after the grant project was over (e.g., obtaining a job in the sponsoring library system, schools, or Library Council, and working several years while demonstrating leadership). This kind of an OBE plan was accepted as at least meeting the spirit of OBE planning, although the partners were warned that they would have nothing to show as progress during the grant period if their plan only included such outcomes and measures. For one or two partners, then, the time period for gathering any outcomes data hasn’t arrived yet or is just starting as this final overall evaluation report is being written. There is a similar problem for the overall evaluation that relates to the evaluation contract. The overall evaluation report is due by August 31, 2007 yet grant scholarship recipients have until December 2007 to complete their library school degrees. The external evaluation therefore also has to move forward with incomplete information or projected results concerning legitimate overall OBE plan targets.

Finally key staff turnover between when partners wrote their original proposals to NYSL and the writing of OBE plans, or after the OBE plans were written and when they were evaluated, created problems for the grant program. Monroe County Library System (Rochester Public Library) had director turnover and turnover of leadership and other staff within the Rochester Public Library Division that was actually implementing the plan. This created confusion and communication related issues about the purpose of the grant, the Rochester Public Library proposal, and the OBE plan. New York Public Library – Branches had a key staff person who had been involved with the proposal and OBE plan leave employment. Pratt Institute, St. John’s University and the University at Albany all went through changes of Deans or program Directors, and the University at Buffalo was being reorganized while its OBE plans were created and implemented. While some staff turnover might be expected over the course of a multi-year, multi-site evaluation, it was disruptive for OBE plan creation, program implementation, and outcomes measurement for about one-third of the grant Teaching Library and University partners.

Lessons Learned about Outcomes Based Evaluation Planning

• Outcomes based evaluations are not the appropriate approach for all program evaluations. Outcomes based evaluation training materials developed for NYSL, Division of Library Development that are based on materials developed by Performance Results, Inc. for IMLS recommend in PowerPoint Slide 8 that one select another evaluation method when: a) one can’t predict user benefits, and b) one is evaluating something other than user benefits. That appears to have been the case with this grant project for such expected outcomes as “best practices” for being a “teaching library.” The more appropriate overall evaluation model for this evaluation would be that used for a multi-site evaluation, including a close look at processes and variations in levels of outcomes achieved in relation to input and program variations as well as with respect to program or community context.
• If there are a sufficient total number of program participants and specific anticipated results are measurable, it is appropriate to specify program outcomes, indicators, measures, and success criteria and perform an outcomes based evaluation. However, it is not appropriate to expect an outcomes based evaluation approach from partners with only one or two program participants unless programs outcomes are specified that look at factors other than what happened with those individuals, such as specific, measurable expected organizational changes.
• Outcomes based evaluations (and many evaluations) have the greatest benefit if they are closely matched to a logic model of how program actions will lead to expected outputs and outcomes. This is harder to do after the fact, after programs are designed and started.
• Evaluation training for partners should occur before programs are started and technical assistance in OBE planning is best done as the programs are being planned. In similar future grant efforts it would be better to make sure that partners receive OBE training and logic modeling assistance before they even start trying to identify scholarship recipients. It would be helpful to provide technical OBE planning assistance from an outside expert evaluator sooner in the grant period. It would be ideal for NYSL to seek input from an evaluation expert even as it is developing a grant proposal to IMLS, so that the anticipated overall evaluation process can be more closely related to the anticipated program process, activities, and outcomes.
• Some outcomes are time dependent. When a program logic model specifies a maximum time in which an outcome can occur, it is appropriate to arrange for a final outcomes based evaluation report that follows rather than proceeds the final time point for measuring that outcome.
• The grant program administrators should set clear criteria for partner OBE plans that require that at least some outcomes be measurable during the grant period and the writing of the overall evaluation report.
• Clear and repeated communication between the grant administrator, partners and external evaluators about expectations, responsibilities and reporting requirements for OBE plan progress might help overcome unavoidable key staff turnover problems that affect evaluations and evaluation activities as well as planned program actions.

Scholarships Awarded, Enrollment, Progress Toward Degree and Graduation Outcomes

The overall Making It REAL! grant program had several specific overall measurable outcomes that relate to numbers of scholarships awarded, enrollment, progress toward degrees and graduation. These are evaluated in this section of the report. The Making It REAL! program achieved most, but not all, of its scholarship outcome targets, and came close on others.

Recruitment Phase

As explained in the partner report on the recruitment phase, ^{Note 18} the number of students anticipated in the program was a moving target because of limited funds available awarded at library schools with different tuition and fees costs, and partner decisions to award full or partial scholarships. During the course of the grant the target became further complicated for the following reasons:

• A Teaching Library partner pulled out before awarding any scholarships, and the funds were eventually redistributed to the University at Buffalo, which awarded five scholarships in addition to the six it had originally awarded. Unlike the first six scholars, the additional five scholars had no special expectations other than completing a Master’s degree and working for two of the five years after graduation.
• A scholarship recipient at Queens Library encountered legal problems during a trip back to his home country of Bangladesh to see family, and on his return withdrew from the program before starting library school.
• Other students at both Teaching Libraries and universities, who are not counted in Figure 6, were offered scholarships and turned them down before starting library school. Reasons included prospective recipients not liking scholarship terms, full-time paraprofessional employees in Teaching Libraries not wanting to give up health benefits and income if they worked part-time, and personal factors.
• Two scholarship recipients started library school but were unable to maintain adequate academic progress and had to leave library school. These are counted above as having started library school. Their scholarship funds were recycled to replacement scholarship recipients, also counted in Figure 6, one in the Teaching Library and one in the University counts. A third University scholar did not pass a key course, left the program and was not replaced.
• Two other scholarship recipients started library school but left for other reasons. One returned to her native Chile to marry, and the other encountered serious health issues that caused her to leave library school and library employment. One was a University MIR scholar, the other a Teaching Library MIR scholar, and both left in a time frame that did not allow their scholarships to be recycled to other students.
• Timing issues related to mismatches of multiple calendars (those of IMLS grant awards, NY State budget and accounting years, and pre-set University admissions calendars) and unavoidable delays in actual release of funds to partners led in some instances to rushed or truncated scholar recruitment periods and processes.
• Key staff turnover at unfortunate times also contributed to delayed recruitment for some partners as new staff learned about and set out to act on the grant program.

The number of scholarships actually in place therefore totaled 42 at the end, 17 (or 40%) awarded by Teaching Libraries and 25 (60%) awarded by Universities, and reached as many as 44 scholarship students in graduate school in spring 2006. The program therefore exceeded its outcomes target of 15 scholarships for Teaching Libraries and fell one short for University scholarships by spring 2006, but 4 short of 29 at the end. This was somewhat surprising since library schools have considerably more experience with running scholarship programs and placing scholarships than Teaching Libraries do, as verified in early evaluation surveys of the partners, but the outcome was predicted in the recruitment evaluation report concerning that survey. ^{Note 19} Around recruitment time, three of the Library Schools had experienced turnover of Deans, Directors or key staff who had written the original University proposals and scrambled to place their scholarships for students starting in fall 2005, reserving some for starts in spring 2006. A fourth university (the University at Buffalo) had to offer its scholarships to different students several times before being able to place them. While some of the Teaching Libraries also experienced problems around recruiting as key staff turned over, each Teaching Library had fewer scholarships to award, so there was less of an impact on the grant project as a whole.

Figure 6. Outcome 1: Students Receive Library School Scholarships

Outcome 1 Lessons Learned

• Overall this kind of a scholarship grant program can meet or exceed its targets for number of scholarships awarded. However, targets need to be set clearly based on both total numbers of scholarships and total scholarship amounts awarded so as to be measurable in outcomes based evaluations.
• Teaching Libraries can be as or more successful than Universities in awarding scholarships to appropriate candidates.
• Grant partners and NYSL should give advance consideration to how to handle instances when applicants offered scholarships decide to turn them down and to how to structure and present scholarship opportunities to maximize the number of applications and the likelihood of acceptance.
• If possible, longer recruitment periods should be built into the scholarship recipient selection process and additional consideration given to recruitment timing issues related to multiple pre-set annual scheduling patterns, including those of University admissions offices and IMLS grant award processes. Teaching Library staff and prospective scholarship applicants might need advice or assistance regarding library school entry criteria and application timing.
• Partners should apply appropriate communication strategies internally and between NYSL and partners if key library or university staff turnover occurs just before or as student scholarship recruitment should be occurring.
• Outcome targets might be adjusted or new outcomes targets set to specify the number or percentage of full or partial scholarships, the number or percentages of scholarship students attending library schools in New York, or the number of percentages of scholars attending library schools with different costs.

Figure 7. Outcome 4: Students Make Expected Progress Toward a Library Degree

Mid-Project Implementation Targets for Academic Progress

Based on careful review of the original grant proposal and partner OBE plans, REAP Change identified six possible mid-implementation outcomes for the grant. REAP Change recommended that the indicators and target criteria for these overall outcomes be discussed further with the Project Director and partners The discussion of indicators and criteria never took place. Nevertheless, outcomes relevant to academic progress toward library school degrees, those proposed by the overall evaluator, are presented, starting with the indicators for Outcome 4 in Figure 7.

The criteria for Teaching Library and University awarded scholar success were based on the graduation criteria specified in the grant narrative, ^{Note 20} in which Teaching Libraries were expected to have all 15 of their anticipated scholarship recipients move forward to graduation while only 80% of the University sponsored scholarship recipients were expected to graduate within the grant period.

Grades Average B or Better. The logic model behind these targets is not clear and may have been faulty. Unless there were reasons to expect that Teaching Libraries would be able to identify students with superior academic ability better than faculty with more experience in such matters, it might have been reasonable to expect that Teaching Library scholars and University awarded scholars would have equivalent academic ability coming into the program. If scholarship students are admissible to accredited library schools (a requirement for Making It REAL! scholarships), and have equivalent abilities at admission, then it would be a reasonable assumption (based on retention and graduation theory) that the same proportions should move forward no matter the source of the scholarship (i.e., a Teaching Library or a University). However, program differences among partners in help toward graduate school success might change the proportions of successful students. Tested retention and graduation hypotheses have shown repeatedly that, while a good predictor, the quality of incoming academic preparation is not the sole predictor of academic success. A better logic model would have specified which elements of project implementation (e.g., mentoring programs) were expected to contribute to academic success and why.

Since a grade of “B” or better is usually considered the minimum for graduate student academic progression, in measuring the first indicator all scholarship recipients who entered graduate school at any point in the grant program were used as the base for percentage calculations and those who were eligible to advance normally each term were counted as successful even if they chose to drop library school for non-academic reasons. Over two years, teaching Library and University awarded scholars had equivalent academic success by this measure. Since there is insufficient information on incoming academic ability uneven academic ability at entry cannot be ruled out as a cause of this result. However, neither do the results suggest sufficient differences in grant program contributions to library school academic success to indicate that scholars from Teaching Libraries did better or worse than their counterparts in academic progress. For example, it is also impossible from data gathered to rule out differences in standards of admission and academics of the different library schools attended as a possible cause of lack of success.

There were no criteria in the grant concerning online courses. However, the availability of online programs at Syracuse University, Southern Connecticut State University, Clarion University, and Texas Woman’s University made it possible for ten of the Teaching Library MIR scholars from more remote areas to attend library school. Teaching Library MIR scholars were more likely than University MIR scholars to take more online courses. Therefore, Teaching Library MIR scholars were more likely than University MIR scholars to be acquiring their “B” or better averages in wholly or partially online programs.

Figure 8 provides a closer look at grade success for students who replied to web-based surveys done in August 2006 and June/July 2007, and most respondents ^{Note 21} reported their grade point averages (GPAs). Figure 8 shows that most Making It REAL! students not only made adequate progress -- many thrived academically in library school. By the end of spring 2007 about two-thirds of the Making It REAL! students had self-reported Grade Point Averages of 3.50 or higher. That includes almost three-quarters (74%) of the Teaching Library scholarship recipients, and six out of ten (61%) Making It REAL! students awarded scholarships by partner Universities. ^{Note 22}

Figure 8. Self-reported Overall Grade Point Averages by Spring 2006 and Spring 2007 or at Graduation

Library School Attendance Patterns. The original grant proposal had no particular expectations of full-time or part-time attendance by Making It REAL! scholars. It had, however, allowed at least two and a half years (from fall 2005 through fall 2007) to complete 36 to 42 unit degrees that might be completed in two years without summer terms if a student had been able to attend full-time. The grant program therefore allowed for part-time study, so the evaluator proposed target outcomes for “progress through library school” accordingly. Although Teaching Library scholarship recipients held their own academically, they progressed through library school at a different pace than University scholars. This is shown in the two charts of Figure 9.

Figure 9. Full-time, Part-time and No Attendance by Partner Type and Term

In the first student survey two-thirds of the Teaching Library scholars indicated that they expected to go to library school part-time every semester including summers, compared to 23% of the University awarded scholars. About 27% of the University scholars expected to attend graduate school full-time including summers, 23% a mix of full-time and part-time plus summers, and 14% full-time except summers. In fact, Teaching Library scholars were significantly more likely than University scholars to be part-time students in both the fall 2005 and spring 2006 terms, ^{Note 23} and also in fall 2006. ^{Note 24} The difference was not statistically significant in spring 2007, but the direction was the same – 69% of Teaching Library MIR scholars in graduate school that term took a part-time course load and 56% of the University MIR scholars took a full-time course load. Also, 94% of Teaching Library MIR scholars attended summer school in summer 2006 compared to 77% of University MIR scholars. ^{Note 25}

REAP Change also determined from Academic Year (AY) 2005-2006 data that even though they were more likely to take summer school courses, Teaching Library MIR scholars accumulated fewer courses by the end of a calendar year of study than University MIR scholars. Since attendance patterns for AY 2006-2007 were expected to be the same, ^{Note 26} REAP Change predicted that a greater proportion of University than Teaching Library students would graduate well before the end of the grant period (i.e., by the end of spring 2007). As will be discussed later in this report, this did turn out to be the case.

Part of reason for the greater likelihood of part-time attendance by Teaching Library scholars was that somewhat larger proportions of these students worked longer hours per week for pay than their University MIR scholar counterparts. The latter also worked while going to library school, just not as often or as many hours per week. ^{Note 27} Most students found it “somewhat difficult” or “very difficult” to attend school and work at the same time, especially if working 21 hours a week or more, or going to school full-time and working any number of hours. The reasons for working for pay while going to library school relate to both the cost of library school and the need to support a family or oneself.

Scholarship students from the Teaching Libraries were especially likely to be taking online courses. In fact, Teaching Library scholarship students attending Clarion University, Southern Connecticut State University, and Texas Woman’s University took all of their courses online, as did most of the Teaching Library students who were attending Syracuse University. The experience of these students was that online courses often take more effort than regular face-to-face classroom courses, even though the students often have flexibility as to when to do the work. The stress of having to work, attend school, and maintain some semblance of a family or personal life cannot be overstated. Most of the students were continually juggling time and priority issues as shown in Figure 10 comments. ^{Note 28}

Figure 10. Comments on Working, Online Courses, and Going to Library School

University at Buffalo Mid-Implementation Outcomes. Since the University at Buffalo (UB) of the State University of New York was mounting a special program to increase the number of librarians able to make websites accessible to people with disabilities, it was picked out in the grant narrative and so in the overall evaluation as having its own expected outcomes. These special outcome goals apply only to the original six scholarship recipients, not to the additional five who received Making It REAL! scholarships that were recycled from the partner that withdrew.
While the six students took a special new web accessibility course, LIS 501, in spring 2006, only four of the six students passed it. The two who failed were not required to give up their MIR scholarship. One later left the MLS program as of fall 2006 for other reasons. ^{Note 29}

An unanticipated problem encountered by the University at Buffalo was that libraries did not respond to an offer to have the students make library websites more accessible to clients with disabilities. The program director noted in her final report:

We were able to find only one site to support a practicum for one of the six students to apply her accessible web design skills. In January 2006, the project director sent an email to several listservs announcing this opportunity. One library asked to participate. In March 2006, Making It Real web site developer Polly-Alida Farrington announced the opportunity to all the Making It Real library partners. None responded to the offer.

Figure 11. University at Buffalo Special Program Outcome

The lack of response will be discussed more fully later in relation to the goal of new Teaching Library models and University-Teaching Library partner interactions.

Outcome 4 Mid-Project Implementation Lessons Learned

• Teaching Library scholars should be expected to progress at the same rate as University scholarship recipients.
• Future grants might specify in advance procedures for replacing scholars if some of the original scholarship recipients who start library school have to stop due to unforeseen circumstances (e.g., serious illness, death in the family, spouse getting a job in another state), or lack of adequate academic progress.
• Future grants might specify potential uses for returned scholarship funds if replacement scholars cannot be identified in a timely fashion or it is too late in the grant period to award new scholarships to replacement candidates.
• Teaching Library scholarship recipients may be expected to attend library school mostly on a part-time basis and take fewer courses per calendar year even if they go to school in the summer, while University scholarship recipients may be expected to attend library school mostly on a full-time basis, with some attending part-time in the summer. Setting a grant expectation of two and a half years time to degree is reasonable for 36 unit library school Master’s degrees that might ideally be completed in two years (four semesters) if a student attended full-time (nine units per semester) and didn’t take summer school. The longer time period allows for schedule flexibility for scholarship recipients and particularly for the part-time attendance patterns typically followed by Teaching Library scholars.
• The combination of library school, work, and often caring for a family is very stressful for the scholarship students even if going to school part-time or taking online courses. Future grant programs might consider making resource materials on time management, handling online courses, juggling priorities, and stress reduction approaches available to students who desire them. Supplementary non-academic materials have proven successful in increasing retention and graduation in undergraduate programs and so are worth considering in graduate programs for unconventional students such as might be attracted by Making It REAL! grants.

End of Project Outcome: Library School Graduation

The graduation target varied by the type of partner that awarded the scholarship.

Teaching Library Graduation Outcomes. The standards for the Teaching Library graduation outcome targets started with the grant narrative and were modified from “15 students” to read “at least 15 students” to take account of more Teaching Library scholarships being awarded than expected. The original outcome target was 100% graduation by December 31, 2007, but the final evaluation report is due by August 31, 2007 and so has to rely on projected as well as actual graduation data. Therefore the outcome goal language was modified. The outcome target percentage failed to take into account the possibility of replacement scholarships, which creates confusing about the number to use in the denominator of the percentage. Given the reality of limited budgets, the base for the graduation rate might be considered 19. One of the 20 scholarship students started library school dropped out for lack of sufficient academic progress after one term and was replaced by another scholarship recipient in spring 2006. On that basis the successful graduation rate projected would be 16 out of 19 or 84% graduating by December 31, 2007.

Figure 12. Outcome 10: Teaching Library Scholars Complete Library Degrees

Only if one also removes from consideration the additional two Teaching Library scholars who left the program (one for lack of sufficient academic progress and one because of serious health problems encountered while in school) would one consider the base to be 17 and even with that lower base, only 94% are expected to graduate by December 31, 2007.

The grant program therefore met its overall target of at least 15 graduates but not its percentage target of 100% graduation. As discussed above in the mid-project implementation section, the target outcome standard might have been poorly formulated or not clearly related to a solid logic model. It did not allow for either academic failure or unforeseen circumstances leading to withdrawal from the scholarship program.

University Making It REAL! Scholar Graduation Rates. As shown in Figure 13, the University awarded Making It REAL! scholars met neither the numeric nor the percentage targets for graduates. Since only 28 total scholarships were awarded in the first place, the university partners did not meet the goal criteria of awarding 29 scholarships and the reasons for this are not entirely clear. Three of the 28 scholarship recipients left the program, one for reasons of lack of academic progress in the library school, and two for other reasons. This left 25 scholarship students who might progress toward graduation by December 31, 2007.

A greater proportion of University MIR scholars graduated by spring 2007 (16 of 28 = 57% or 16 of 25 = 64%) than did Teaching Library MIR scholars (4 of 19 = 21% or 4 of 17 = 24%) for reasons of different full-time/part-time course-taking patterns discussed in the mid-implementation evaluation section above. This is as the evaluation predicted based on results of the August 2006 student survey. However, three of the remaining University MIRS scholars are not expected to graduate by December 31, 2007 and the fourth has been incommunicado, so her expected graduation date is unknown. Two students are expected to graduate at the end of summer 2007, and three at the end of fall 2007, which means a total of 21 University MIRS scholars are expected to graduate by December 31, 2007. This is only a graduation rate of 75% (21 of 28), which is also short of the 80% target. Problems have occurred with University MIRS scholars throughout the grant: not enough scholarships were awarded; three people left the program; and too many did not proceed toward graduation at an adequate pace despite the fact that more than half graduated well before the deadline. Reasons for late graduations include late starts in the program, serious health problems, and the birth of a child.

University at Buffalo MIR Student Graduation Goals.

The University at Buffalo program also did not meet its numeric or graduation rate targets. For the reasons explained in the mid-implementation section above, the University only had five rather than all six students advance toward potential graduation. Only four of these (4 of 6 = 67%) are expected to graduate by December 31, 2007, rather than 100% (or all six).

Because of the problems in locating library websites to improve toward ADA standards, only two students (33%) graduated at the end of spring 2007, which is well below the overall University scholars graduation rate at that point in time. One more student is expected to graduate at the end of summer 2007 and another at the end of fall 2007, bringing the total to four or 67% of the original six scholars graduating within the grant period deadline.

Figure 13. Outcome 10: University MIR Scholars Complete Library Degrees

Figure 14. Outcome 10: University at Buffalo Scholars Complete Library Degrees

The fifth student eligible for graduation is also expected to graduate, but not until the end of spring 2008. The final graduation rate for the University at Buffalo should be 5 of 6 or 83%, which is more than the overall University scholar target, but not achieved in a timely fashion.

Lessons Learned about End of Grant Graduation Outcomes

• Although greater proportions of University MIR scholars may graduate earlier in the grant period, greater proportions of Teaching Library scholars may graduate by the final target graduation date.
• Teaching Libraries are more likely than Universities to meet or exceed their numeric targets. However, the percentage graduation rate target of 100% is set too high since it allows for no margin for error or unforeseen circumstances. In the future the percentage target might be set at about the same level as those of the Universities (80%). No individual University should be expected to have a 100% graduation rate.
• Meeting end outcome targets depends on meeting both scholarship award and mid-implementation targets of the grant. Future grants might want to set reasonable targets for numbers and percentages of students left at the end after assuming a reasonable amount of program attrition and individual student lack of success.
• The base to be used for calculating percentage targets needs to be clearly set and understood. If graduation rate is used, then the percentage of those in a starting cohort is the usual denominator and a decision must be made as to whether to include or not include replacement scholars.
• Clear outcome goals or targets depend on a well thought through, clear logic model of how the program is expected to operate that is based on theory or practical experience. This first-time, more experimental grant did not have practical experience for Teaching Library scholars on which to base its Teaching Library targets, but one would have expected Universities with their greater experience base to have been able to meet their lower targets.
• The grant is expected to fall short of its overall target of graduating 42 to 44 new librarians or information scientists with professional credentials by December 31, 2007 primarily because the University partners did not meet their targets. The grant can expect to have graduated 37 new library or information science professionals by it target end date of December 31, 2007 and 41 within a year after that.
• It would be preferable to set the reporting date for the final evaluation report after the end of the grant program implementation period so that the overall evaluation can measure actual rather than projected graduation outcomes.

Diversity Goals

According to the grant narrative: ^{Note 32}

The grant narrative further comments:

Diversity

New York State is one of the most diverse areas in the U.S., with more than 100 ethnic groups within its population of more than 19 million people. It is home to the largest city in the nation, and is also home to vast rural areas of small hamlets and farms (Encyclopedia of New York State, 2003). Preparing a new generation of librarians to serve this extraordinary population is the focus of this grant.

All the Teaching Libraries respond to clearly identified community needs. For example, the Teaching Libraries in the Queens Borough Public Library, the Monroe County Library System, and The New York Public Library respond to community” “needs for ethnically diverse librarians as well as local needs for specialty librarians. The library systems' plans demonstrate a knowledge of their communities and a strong commitment to enhancing diversity.”

Diversity within the library community in New York State can range from the bustling Science, Industry and Business Library of The New York Public Library to small rural libraries in northern New York State that are widely separated, poorly equipped, and often without professional librarians. The state's urban public libraries serving disadvantaged, multiethnic neighborhoods are usually overcrowded and understaffed; lack adequate collections, facilities, and resources; and cannot meet the increasing demand for library services. Each strategically located Teaching Library provides an important first step toward improving library service and addressing community needs.

The project's scholarship component will provide opportunities to reach out to potential students in underserved groups. In addition, rather than concentrating only on the demographics of ethnic and racial diversity in the recruits, the scholarship component will focus on helping all students develop competencies for serving and responding to the needs of diverse populations. The Buffalo program will study competencies for serving the disabled, another underserved community.

Part of the evaluation process of each project will be an assessment of the competencies acquired in serving special populations. A fundamental change in recruitment philosophy and practices is a likely long-term outcome.”

The broad view of “diversity” adopted for this grant therefore refers to:

• Recruitment and graduation of diverse individuals to receive the scholarships;
• Institutional changes in recruitment philosophy and practices of Teaching Library organizations and library schools regarding diversity;
• Developing competencies of Making It REAL! scholars to serve ethnically diverse, disabled, and otherwise diverse communities in New York state; and
• Meeting needs of diverse communities by focusing on training librarians in specific specialties of the diverse library field.

As will be documented below, the Making It REAL! grant was very successful in meeting the target for recruiting and training more diverse individuals as library or information science professionals and largely successful in focusing on training librarians in library specialties that currently are experiencing a shortage of trained librarians in New York State. The grant program had considerable success in educating students to serve ethnicity diverse, disabled, and otherwise diverse communities in New York State, but with some mixed results. Finally, during the grant period there were only limited and scattered changes in Teaching Library organizations and library school recruitment and practices regarding diversity. However, a few exciting long-term impacts surfaced during final partner interviews which indicate some “best practice” long-term impacts in regards to diversity related organizational changes.

Diverse Student Recruitment and Graduation

Adding diverse people to the ranks of professional librarians is a key goal of the Making It REAL! program that had specific expected outcomes. The results were different for Teaching Libraries and University library school partners (see Figure 15). What happened at the beginning of the grant writing process regarding diversity expectations is disputed, with different versions coming from New York State Library and the grant partners. ^{Note 33} However, the situation was resolved after the NYSL Program Manager obtained a clarification from Tracie Hall, then Director of the American Library Association’s Office for Diversity and the Spectrum Initiative on what “diversity” might mean in the New York State context. ^{Note 34} The definition goes well beyond ethnicity to include gender and diversity among types of libraries.

Figure 15. Outcome 2: Diverse Scholarship Student Recruitment Targets

Caucasian females will not be considered to be diverse unless they fall in one of the “diversity in context” categories. Caucasian females in rural areas, slated to serve rural partners, will be counted as “diverse in context.” All males, including Caucasian males, will be considered to be diverse, as will all physically or mentally disabled people. The recruitment targets are assumed to apply to scholarships actually awarded rather than to scholarship applications. ^{Note 35}

The Teaching Libraries exceeded their diversity outcome targets by awarding scholarships to five ethnic minority students (one of whom left the program due to lack of adequate academic progress), one GLBT, one male, and six Caucasian females from rural areas intending to serve rural areas (one of whom left the program for health related reasons). Without counting Caucasian females from rural areas, diversity among the Teaching Library MIR students would have been 37% based largely on ethnic minority representation, which is still impressive. Queens Library, which is in the most diverse county in the country, started with two ethnically diverse immigrant scholars and had to replace both after one didn’t start library school and the other didn’t make sufficient academic progress. While the replacements were both Caucasians, one is a male, so the University still made its target. The Queens Library partner contact person indicated that other ethnic or immigrant minority candidates were interested but were not willing to give up the benefits or income from full-time paraprofessional jobs.

An earlier evaluation report indicted that the University partners had also exceeded their diversity recruitment target, primarily by awarding scholarships to “traditionally under-represented minority” students. ^{Note 36} However, the five additional students added late to the University at Buffalo MIR scholarship recipients all happened to be Caucasian females. As a result, with 75% diverse scholarship recipients, the University MIR scholarship recipients fell ten percentage points below the diversity target of 85%.

Even though the University library schools did not in the end meet their outcome target, 75% diversity is still very impressive compared to the typical library school enrollment. For example, in the three fall terms from 2000 to 2002 the six New York University MIR partners posted a combined 12% ethnic minority enrollment rate or a 28% diverse student enrollment rate including all males and ethnic minority females (cf. to 71% for this kind of diversity among Making It REAL! students). Not enough Making It REAL! scholarships were available to offset the overall makeup of any of the University library and information science schools or departments, ^{Note 37} but the grant program is certainly contributing strongly to the diversity of library school students.

It is less desirable that all three University MIR scholars who left the program were ethnic minorities (two African Americans and one Latina). Their departure left the library schools with 72% diversity toward the end of the grant period.

The Making It REAL! scholars were also diverse in other ways as can be seen by exploring http://www.nysl.nysed.gov/libdev/imls/awardees/index.html (retrieved July 31, 2007), the list of scholarship recipients, or the profiles of twelve of them on the Library Careers NY website at http://librarycareersny.org/profiles/index.cfm (retrieved July 31, 2007) that grew out of the Making It REAL! project. Figure 16 indicates some of the ways.

Figure 16. Other Kinds of Diversity among Making It REAL! Students

Graduation Outcomes. As discussed earlier in this report, not all of the Making It REAL! scholarship recipients will graduate, and not all will graduate on time. However, as of July 31, 2007 REAP Change expects that Making It REAL! will have assisted 42 scholars in obtaining library school Master’s degrees.

Of these 42 scholars, 71% are of diverse backgrounds and include:

• 20 ethnically diverse individuals (48%);
• 5 men (12%)
• 1 GLBT person (2%)
• 2 disabled people (5%)
• 6 librarians serving remote or rural areas of New York (14%)

Organizational Changes in Relation to Diversity Recruitment

Although the partners recognized that the Making It REAL! program would allow them to recruit diverse candidates for the grant program scholarships, many did not expect that the grant would lead to overall organizational change in diversity recruitment. Indeed, that expectation was not stated in the grant proposal to IMLS in a way that lent itself clearly to measurement. REAP Change recognized this early and proposed in its overall evaluation plan that it might evaluate organizational change in relation to a model put forward in 1996 by Thomas and Ely, ^{Note 38} or an elaboration on that model published in 2001 by Selden and Selden. ^{Note 39} Such a proposal would have required additional resources and additional participation from MIR program members which were not in the original proposal. Therefore this intensive and intrusive evaluation approach was not pursued.

Figure 17. Proposed Outcome 8: Teaching Libraries and Library Schools Report Increases in Activities Related to Recruiting for Diversity

REAP Change did propose a possible way to measure changes in organizational diversity recruitment, shown in Figure 17. However, the consultation with New York State Library and library school or Teaching Library partners never occurred, so neither NYSL nor the grant partners concurred on the outcome targets suggested in Figure 17. Nevertheless, REAP Change was able to gather some information relevant to organizational changes in diversity recruitment approaches in response to direct questions in a four-person partner focus group at NYLA 2006, in-person interviews with three other partners at NYLA 2006, and telephone or in-person interviews with all partner representatives during late June and mid-July 2007. In addition, the partner OBE plans themselves, partner progress reports in spring 2006 and final OBE outcomes reports on partner plans surfaced several recruitment approaches that were new or innovative for those partners.

When partner representatives were asked directly in final interviews in June or July 2007 “What did you do differently as an organization in relation to diversity as the result of the grant participation?” many were at a loss for what to say, beyond talking about their scholarship recipients or in some cases a service to the community that came out of their specific grant related efforts. When probed further and specifically asked about recruitment, several gave responses similar to those reported in the original recruitment report:^{Note 40}

• Some Teaching Library partners had small staffs with little turnover and so rarely recruited (notably the Library Councils);
• Some Teaching Library partners had not even been able to recruit any new employees of any sort for several years because of state or local library district fiscal shortages;
• Teaching Libraries always recruited with attention to diversity (i.e., legal requirements for non-discrimination) and so did nothing different as a result of grant participation;
• University library schools also always recruited students with attention to diversity, and 83% had reported in 2005 that they had prior experience specifically with student diversity recruitment programs. ^{Note 41}

Nevertheless, a few were able to offer specific new “best practices” related to recruiting diverse librarians into the professions. An exploration of the individual OBE plan progress reports and final outcomes reports indicated that some Teaching Libraries were implementing in practice elements of what Adkins and Espinal (2004) ^{Note 42} had proposed as diversity recruitment best practices for students beyond offering financial incentives and substantial assistance (which the Making It REAL! scholarships did). In particular, the partners modeled the “Best Practices in Diversity Recruitment” shown below either during or in response to participation in the grant.

Making It REAL! Partner Best Practices Related to Diversity Recruitment

Offer financial incentives and substantial assistance

• All partners offered Making It REAL! scholarships.
• The Palmer School leveraged $30,000 in Making It REAL! scholarships (over two years) into a promise from Long Island University of $80,000 in scholarship funds annually to be awarded to diverse students.
• St. John’s University on July 26, 2007 announced the funding of 40 full scholarships for MLS students through an IMLS Laura Bush Grant, 20 to start in spring 2008 and another 20 in summer 2008. St. John's program will emphasize engaging underserved children and their families and preparing students for employment in public libraries, school library media centers, and museums in the New York City metropolitan area. Designed with the working student in mind, this two-year part-time scholarship program will offer evening, weekend, and online classes. The application was modeled on the Making It REAL! program.
• The Rochester Public Library staff support their multiracial MIR scholar so much that one staff member is going above and beyond the call of MIR program support by providing the now graduated scholar with a place to live while she hunts for her first professional library job.

Enroll a cohort of students from a particular group.

• Syracuse University has a one month “boot camp” for its online students that pre-dates Making It REAL! and helps orient students to the culture of the University. The “boot camp” experience deliberately crosses School specializations and includes diverse people from around the world, since that is the student body attracted to Syracuse. Making It REAL! students participating in a NYLA 2006 MIR evaluation focus group thought that this boot camp might be used in the future as an occasion to start forming library school students from particular groups into cohorts. These online program Syracuse students otherwise would not have a chance to meet face-to-face during their courses.
• Making It REAL! scholars from the Teaching Library programs of Clinton-Essex-Franklin Library System and Northern New York Library Network in different parts of the state met in online Clarion University classes and had formed their own supportive cohort by the time of NYLA 2006.
• Making It REAL! scholars who attended NYLA 2006 enjoyed meeting one another at the professional conference and several recommended that future programs provide more opportunities for scholarship cohort interaction at professional conferences.

Present librarianship as a helping profession in which one can give back to one’s community

• See examples later in this report of how Making It REAL! students and partners have provided or plan to provide service to their communities with new attention to diverse communities.

Allow potential recruits to meet diverse librarians or library school students with whom they can relate

• Teaching Libraries highlighted their Making It REAL! scholars in paper or online newsletters transmitted throughout their systems. Students often wrote stories for the staff newsletters about the grant program and their experiences in library school. Partners taking this approach include the Clinton-Essex-Franklin Library System and Northern New York Library Network.
• As part of their MIR programs and OBE plans Teaching Libraries required that their MIR scholarship recipients participate in sessions specifically set up so that they could meet and talk with paraprofessionals in the larger systems. For example, one of the Northern New York Library Network (NNYLN) scholars met with the Library Assistants Committee during the spring 2007 NNYLN Conference and answered many questions about her studies. Several people in attendance have since then expressed an interest in obtaining a M.L.S. and another employee in a library where the scholar is working has applied to several schools for admission in a M.L.S. program. NYPL-SIBL is currently debating the best format for its African American MIR scholar to interact with a group of support and technical staff.
• The Clinton-Essex-Franklin scholarship recipient successfully led two training sessions for her peers (primarily librarians without MLS degrees in small libraries) per year, one of which was on multicultural resources and services.
• Librarians not directly responsible for the Making It REAL! program asked permission for diverse (e.g., African American) new paraprofessional staff who are interested in librarianship to e-mail, telephone, or meet directly with a diverse (e.g., African American) Making It REAL! student to talk about library school and librarianship as a career. This has occurred at NYPL-SiBL and NYPL-Branches. As the NYPL-SIBL Director wrote in an e-mail “It’s like viral marketing!”
• As part of the Making It REAL! program, NYSL funded a website at http://librarycareersny.org/ that includes a “Profiles” section. This section (http://librarycareersny.org/profiles/index.cfm, retrieved August 1, 2007). includes not only profiles (and photos) of diverse Making It REAL! students but also links to profiles or stories about other diverse librarians.

Be aggressive or assertive. Talk about librarianship careers with patrons, at library events, and at other social or business events

• The NYPL-SIBL Director reports that she is speaking more with diverse people about librarianship as a career, including SIBL patrons, at library events, and at other organizations to which she belongs.
• St. John’s University, Division of Library and Information Science reports that the School thought it would be “difficult” to recruit diverse students. However, after a successful Making It REAL! experience with two students led to a “consciousness raising” the faculty have been pleasantly surprised at how dialogue, ideas, and actions have developed around new ways to recruit students and to let community residents know about library services to local communities. St. John’s has a geographic advantage of being located in Queens, the most diverse county in the United States. The ideas include creating a library science minor in an Associates degree program, recruiting more actively among St. John’s baccalaureate seeking undergraduates, advertising the library school in the many local community newspapers, and identifying and placing stories in the local community newspapers about faculty and local library projects that serve the community.
• The Rochester Regional Library Counsel MIR scholarship recipient plans to provide new information about School Library Media Services careers to high school career and guidance counselors and speak with prospective students.

Share information about the variety of library work.

• Based on her experience with sharing information about what is going on in NYPL with her Making It REAL! student, the NYPL-SIBL Director has added more junior staff to her e-mail lists. This has the effect of more broadly sharing information about the variety of work that goes on in a large library system with people who might not otherwise have been aware of it.

Recruit from paraprofessionals in your library System

• Almost half (46%) of the Teaching Libraries characterized their scholarship student recruitment efforts as primarily internal.
• Before and during the Making It REAL! program Queens Library and New York Public Library have had different programs that help their paraprofessionals go to library school and move into professional librarianship positions. At these large public library systems the Making It REAL! program constituted a variation on a theme of continuing diversity recruitment efforts.
• One Teaching Library partner created a “short list” of potential recruits who might be interested in attending library school in case other scholarship opportunities such as Making It REAL! surface or come to the attention of the Director.

Problems related to Forming Cohorts as a Recruitment Tool. Making It REAL! partners and students also reported a few problems in implementing the recruitment related suggestion of admitting cohorts of students. One problem is that the diversity of professional library careers can inhibit the formation of cohorts of students even when admitted to the same library school if, as happened at one MIR partner library school, students with the same scholarship happen to be interested in different specialties. At that library school the School Library Media Specialist students tended to have their own clique or cohort that tended not to interact much with students in other specialties. One of the Making It REAL! students was interested in becoming a School Library Media Specialist and the other was not, which contributed to their never gelling together as their own small cohort.

Another issue relates to diversity in the age range of students. As shown earlier in this report, Making It REAL! students with both Teaching Library and University MIR scholarships tended to cover a wide range of ages at the time of library school entrance. As such they were contending with different life stage experiences outside their common University experiences that made it appear somewhat less likely that they would bond together in a cohort.

In a different case, two Teaching Libraries sponsored MIR students in the same specialty who were attending different universities started at the same time but did not form into a cohort even though their sponsoring Teaching Libraries had coordinated their programs and OBE plans. The problem was that the one was attending a local University in her town more on a full-time basis and the other was taking classes in an online program at a different University, so their program progress got out of synchronization and the one taking courses locally graduated much quicker than the other, who is still attending school. Similarly, students given scholarships by the same Teaching Library may be less likely to form a cohort based on the common origin of their scholarship if they are attending different library school programs.

Finally, some partners reported hesitating to treat Making It REAL! scholars differently than other students or employees, which would tend to inhibit cohort formation. Both Teaching Libraries and library schools may need to think carefully about what to do when and how to help diverse students form into a cohort for positive mutual self-support while in library school, and why the effort is being made. Cohorts don’t always form by themselves.

Developing Competencies of Making It REAL! Scholars to Serve Ethnically Diverse, Disabled, and Otherwise Diverse Communities in New York State

Making It REAL! scholars were asked in the final student survey “How prepared are you to provide library service to diverse populations in New York State?” As shown in Figure 18, about 87% felt “very prepared” (28%) or “prepared” (57%) to provide such services, and no one felt “unprepared” or “very unprepared.” There was no statistically significant difference by partner type or by library school where the student studied.

Figure 18. Degree of Student Preparation for Providing Library Service to Diverse Populations in New York State

When asked to elaborate, the students indicated that only part of the preparation came from the Making It REAL! program itself. As shown by the comments in Figure 19, the strongest diversity impact of the program for Teaching Library scholars was on those who valued the combination of practical experiences in libraries, including in several libraries in a system, and their formal library school education. Two or three Teaching Library scholars noted that they were already attuned to service to diverse communities but participation in the program kept that in the forefront of their minds, or particular experiences in courses or in libraries made them more sensitive to diversity issues. ^{Note 43} Three felt that prior life experiences, including some working in libraries, had helped prepare them more than any library school coursework made possible by the Making It REAL! scholarship paying for their education. The wide age range of the MIR students is relevant in this regard. The Teaching library students included not only the social worker identified in Figure 19 comments, but also a school teacher and a school nurse, all with years of practical experience with diverse people already under their belts. Two or three students mentioned that service to diverse communities was stressed in multiple classes in library school. Two, who were viewing “diversity” primarily as “ethnic diversity,” felt that the question was not applicable to them because of where they lived or because of special circumstances surrounding their entry into the MIR grant program.

Fewer University MIR scholars felt that the grant program itself, at least that implemented by the University partners, had contributed much to their preparation to provide library services in diverse New York communities. Two or three mentioned a general awareness of diversity by virtue of being a scholarship recipient, and a University at Buffalo student specifically spoke of the contribution of the special program concerning web services to the disabled.

Figure 19. Student Comments on the Relationship of Participation in the Making It REAL! Program to Preparation for
Providing Library Services in Diverse New York Communities

Of special importance to three of the University MIR scholars, and missed with regret by one, were the grant sponsored diversity-related workshops and sessions at NYLA 2005 in Buffalo and NYLA 2006 in Saratoga Springs. These appeared to provide practical knowledge and self-reflection not available for those students from their Universities. Most of the University MIR scholars indicated in their comments that their preparation to serve diverse New York communities came from their previous work experiences, or course work plus internship experiences that they felt they would have received anyway simply by virtue of being in library school. Several University MIR scholars pointedly commented that the Making It REAL! program provided nothing beyond scholarship money for their education.

Examining the same set of comments by library school where all the Making It REAL! scholars were receiving their education indicated that some of the New York library schools are doing well in covering diversity in their curriculum. Also, Clarion University offered a course on Multicultural Diversity that sensitized one scholar to her own prejudices. Queens College, which was not a Making It REAL! partner, seemed to give New York City area students who were already immersed in diverse communities a better sense of how to work as a professional librarian in those communities. Students at Syracuse University, all but one of whom was a Teaching Library MIR scholar, indicated that providing service to diverse communities was covered in more than one course and is infused throughout most of the curriculum. Palmer School has at least one diversity-related course taught by a professor that was highly regarded by one of the MIR scholars. Three students at St. John’s mentioned that diverse and special populations were covered in several classes, but one noted a lack of day-to-day experience in a working library and another noted that precisely that kind of experience contributed the most to one’s ability to work with diverse populations. The University at Albany students apparently gained classroom experience with diversity and had valuable paid or volunteer work opportunities. The University at Buffalo students gave a mixed review in regards to preparation to serve diverse communities. Those in the accessible web design program cohort felt prepared and engaged in relation to diversity in communities. Those in the cohort of five added later depended more on experiences outside the classroom, including work and the NYLA conferences, for their preparation to serve diverse communities. The one very negative community about University at Buffalo was from the latter group, from a student who mistakenly thought she should have been trained in web design for the disabled by virtue of her participation in the MIR program.

Students evaded the question in their responses about their online courses at Southern Connecticut State University and Texas Woman’s University, by referring either to their prior life and work experiences with diversity or saying that the question did not apply in the remote, small town North Country of New York state. None of the eight MIR scholars at Pratt Institute identified any specific or special contribution of Making It REAL! to their preparation to serve diverse communities. Nor did they specifically point to courses at Pratt that focused on diversity.

Figure 20. Outcome 9: MIR Partners Involve Students in Training and Activities Designed to Improve Service to Diverse Communities

Partners often did not have specific goals for their MIR scholars to serve diverse communities so in many cases this outcome was unmeasurable. Looking at both the student comments and partner OBE plan reports, the following “best practices” in preparation to serve diverse communities emerged, with examples highlighted from the Making It REAL! program.

Provide practical library experience in diverse settings to students in library school along with classroom education related to serving diverse communities

• The Clinton-Essex-Franklin Library System serves 34 libraries in northeastern New York in an area not considered very ethnically diverse. Nevertheless, the Clinton-Essex-Franklin Library System is giving its MIR scholar training experiences in three public libraries, including the Akwesasne Native American Library.
• Queens Library, whose MIR scholars are all taking courses at Queens College, serves the most diverse county in the country. Its MIR scholars are being given experiences not only at the Central Library, including its Youth Services Division, but also in a branch, with the bookmobile, and in library outreach at community venues.
• Rochester Regional Library Council is providing its scholar learning experiences in three public libraries, three K-12 school libraries and a University library in a variety of communities, including diverse high need communities, suburban communities and the Central Library of Rochester Library. The scholar is expected to become a school library media specialist at a “high needs” school on graduation.
• The University at Buffalo special program to educate students in providing accessible web design for libraries incorporated practical experience in implementing what was learned in the classroom into its program. There were problems in locating libraries willing to allow students to work on their websites, but the approach is sound and one student was able to provide direct service to one library.

Develop, offer and encourage students to take library school courses specifically related to library service to diverse communities or spread consideration of diversity issues across the curriculum.

• Clarion University, which is not a Making It REAL! partner, has a course on Multicultural Diversity that can sensitize even online students to become more aware of their own prejudices. As UCLA Professor Clara Chu indicated in her Making It REAL! sponsored presentation at NYLA 2006, individual awareness of prejudice is an essential first step for dealing with it in response to diverse individuals and communities.
• The University at Buffalo developed and taught a new course, LIS 501 Accessible Web Design, specifically to educate students in how to provide accessible web services in libraries.
• St. John’s University and Syracuse University were reported to have incorporated approaches and concepts related to library service to diverse communities in many courses in the library school curriculum.

Focus Faculty Efforts on Library Service to Diverse Communities

• Identify and assign professors with relevant expertise, such as Dr. Amy Spaulding at Palmer School, to teach courses that specifically focus on diversity.
• Advocate that faculty cover diversity related issues and practical approaches in their courses.

Encourage and arrange for library school students to participate in special diversity related opportunities outside the classroom to gain experience or training

• A University at Albany sponsored Making It REAL! scholar participated in a three day institute to train librarians on how to serve Spanish-speaking patrons called the “Spanish Language Outreach Program.” He spoke with pride about this experience to other MIR students during an evaluation focus group at NYLA 2006 and mentioned that the program will be producing a variety of Spanish language materials for libraries throughout the state to use. The NYPL-Branches MIR student has also participated in a training workshop for this program.
• NYSL set up special diversity-related workshops and sessions at both NYLA 2005 and NYLA 2006. These were generally well received by workshop or session participants, ^{Note 44} and were specifically mentioned in favorable comments made by both Teaching Library and University MIR scholars.

Figure 21. Outcome 7: NYLA Conference Workshops Outcomes

The NYLA 2005 and NYLA 2006 workshops and sessions were evaluated separately by REAP Change using survey questionnaires passed out immediately after each session and, in the case of NYLA 2006, through participant observation. The outcomes based evaluation above is based on comments and criteria in the grant narrative.

Meeting Library Needs of Diverse Communities

Figure 22. Outcome 9: Partners Develop New Ways to Serve Diverse Communities

The conversations needed to set overall grant targets for this measure never took place, so the evaluation of Outcome 9 depends on looking at individual project OBE plans. Many of the partners viewed their primary responsibility as recruiting scholarship students (diverse, if possible) and supporting their education in librarian specialties in short supply, so that after graduation the students would be able to better meet the needs of diverse communities. Many partners included no proposals for changing their service to diverse communities in either their original proposals or final OBE plans.

The spirit of the grant, however, suggested that the partners themselves might improve service to diverse communities in their area during the course of the grant program. The REAP Change evaluation team pointed this out at the workshops that most partners attended on June 1 and 2, 2005, during which the partners worked at fleshing out their specific programs and started writing their individual program OBE plans. The evaluation team continued to encourage partners to include actions and measures related to service to diverse communities in their OBE plans, but it was never required that partners do so. Some had already specifically included activities related to service to diverse communities in their original proposals to NYSL (e.g., the University at Buffalo program specifically designed to assist disabled people with provision of better web access) and others thought of ways that they could do so during the program, drawing on help from their MIR scholars in the implementation, and incorporated those ideas in their final OBE plans. If successful, those OBE plans would give the students direct experience in serving diverse communities (shown above to be desirable for student education), and in some cases the scholars would be role models as diverse librarians providing such service. Most importantly, such OBE plans, if implemented, would deliver tangible library products or services to diverse communities, perhaps in new ways. Successful projects would therefore be a win-win-win situation for the scholarship students, the organizations sponsoring the scholarships, and the communities.

We highlight here the specific new service to diverse communities plans attempted in conjunction with the Making It REAL! grant program and what became of them.

• New York Public Library – Branches planned for its MIR scholarship recipient to help make the Gay, Lesbian, Bisexual and Transgender (GLBT) community organizations more aware of the library’s services and expand those services. The MIR student worked out of Teen Central, the Nathan Straus Young Adult Center of NYPL’s Donnell Library Center in midtown Manhattan, using class visits and direct communication with community organizations serving GLBT youth. She took a leadership role in organizing Teen Central’s Third Annual “Anti-Prom,” an event of special interest to NY City’s GLBT youth. The “Anti-Prom” received recognition in an “Excellence in Library Services to Young Adults” award from the Young Adult Library Services Association (YALSA). Also, the scholar served on several Young Adult (YA) Services committees, including the YA Book Selection Committee, and attended a focused workshop on “Outreach to the Spanish Speaking Population.” The summer 2007 YA summer reading program, “You Never Know @ Your Library” includes many Spanish titles.
• NYPL-SIBL provides a variety of library services to businesses and entrepreneurs, large and small, from around the world. It takes a unique one-stop business support services approach. It decided as part of the Making It REAL! grant efforts, and in celebration of its tenth anniversary as a NYPL library, that it would create a “Getting Down to Business” toolkit, consisting of resource guides, training class outlines, and two specially created videos, one of which features entrepreneurs describing the resources they used to start businesses. The SIBL staff is compiling the toolkit and intends to use it to expose library school students and graduates, academic institutions and libraries in general to best practices in business librarianship.
• The Monroe County Library System Making It REAL! project runs out of the Rochester Public Library, Local History and Genealogy Division. The Local History and Genealogy Division decided to take advantage of the presence of the MIR scholar to create a database indexing information about local authors, to be made available to Division libraries serving the general public. Because of the volume of material to be indexed, the MIR scholar designed and completed the Local Author database in Microsoft Access over two summers, rather than the one summer planned. A staff person has been trained in ACCESS to maintain it, while other staff members are obtaining information from the database using Word. The MIR Scholar is currently fielding calls from smaller local area libraries about how to set up such a database for their local authors. The Local History Division had also hoped to create a list of items pertaining to African American history in local collections that might become available for use in future exhibits. A small Frederick B. Douglass Museum has opened to little fanfare. However, since the state of the local history community in Rochester has been in a state of flux for some time this additional project has not moved forward during the grant period.
• The Southeastern New York Library Resources Council (SENYLRC) Making It REAL! project needed staff trained in digital librarianship in order to advance the new Hudson River Valley Heritage (HRVH) service. The MIR scholar was educated in digital librarianship and was responsible for creating over 6,100 digital objects from 25 contributing institutions. These digital objects are now accessible to libraries and the public through HRVH. The last collection that she worked on, in Vassar College’s Archives and Special Collections Library in January 2006, contained the Harriot Stanton Blatch Papers. Harriot Stanton Blatch was the daughter of Elizabeth Cady Stanton and continued her mother’s work in the women’s rights movement after her mother died. The MIR scholar graduated in May 2007 and expects to continue expansion of HRVH as a librarian. Among possible future collections to digitize are papers and materials related to African American history.

Other than the special program focusing on disabled library patron web access, which has been discussed, none of the library schools planned to mount special programs to serve their diverse communities as part of their Making It REAL! activities. The University at Buffalo program ran into an unexpected block when only one library responded to a request to provide a free check and update by the Buffalo student of web accessibility of their websites. One library did benefit.

Students themselves also found creative ways to increase diversity related library services to local communities.

• When blocked in his efforts to find a practicum site at which to use his web accessibility training, a Making It REAL! student at the University at Buffalo identified a directed study in which he would co-author a web accessibility study of libraries and Library and Information Science schools across the country. The results of this study are about to be published nationally. The coauthor is Alex Schmetzke, guest-editor for a special issue on Library Hi Tech that is focused on accessibility and tentatively scheduled for publication in late 2007. The draft article is available online at http://www.djcomeaux.com/lis/comeaux_accessibility_study_LIS598.doc (retrieved August 2, 2007).
• A MIR scholar at St. John’s University gave a presentation at the Bayside Historical Society of New York. She made puppets to tell a story of an Indian kingdom beset by negative circumstances. The cooperation of the people helped to change the environment of the kingdom. The use of this artwork is associated with Indian village traditions of special decorations created by women. There is now government funding to train women in this tradition of Mithila Painting so that they can make a career in advertising, weaving or creating special exhibitions of this unique artwork. Students and children who attended this special presentation were given arts and crafts materials to make representations of the diversity of the different cultures presented.
• As a result of her participation as a Making It REAL! grant program, a North Country Library System MIR scholar was focused on diversity in not very ethnically diverse Oswego, NY. She was coordinating the children’s room at the Oswego Public Library while being educated to become a School Library Media expert. She came up with the idea of holding a Chinese New Year’s party open to all children and families wanting to attend it at the children’s room of Oswego Public Library. Since the Chinese Year of the Pig was about to start, she sought out and found a miniature pot-bellied pig to attend the party, along with her owner. The owner happened to be an English teacher at the high school where the MIR student was doing school library media practicum work. Speck, the pig, and the party were a great success, complete with reading of two children’s books, “The Three Pigs” by David Wiesner and “The Chinese New Year” by Cheng Hou-tien, and a snack table with egg rolls and fortune cookies donated by a local Chinese restaurant near the library. Not only did the event bring people to the library on February 16, in the middle of a very snowy winter of 2007, it was covered by the local Oswego Daily News. The online version of the article caught the attention of American Library Association staff. A reference to it was put in the “Seen Online” column of the March 14 and April 2007 issues of ALA Direct, the online publication that goes out to all ALA members with e-mail addresses (who haven’t opted out).

While not spread across all partners or students, the Making It REAL! program has had some very positive intended and unintended outcomes related to library service to diverse local communities, modeled to the “community” of librarians and librarians across the country.

Meeting Needs of Diverse Communities by Focusing on Training Librarians in Specific Specialties of the Diverse Library Field

In terms of being a scholarship program designed to add diverse library professionals in specialties with qualified personnel shortages, the Making It REAL! project has certainly met the spirit of the grant expectations. For Teaching Libraries it has also largely met the letter of their specific grant funded programs. Most of the Teaching Library partners deliberately sought to recruit or required as a condition of scholarship receipt that students be interested in library specialties with a shortage of qualified personnel in their areas. Except for the first cohort of University at Buffalo MIR scholars, the University library schools or departments were not prescriptive about library specialties in their recruitment efforts.

Figure 23 shows a complete list of library school specialties of the Making It REAL! students who have graduated or are about to graduate. Many of the new librarians who have or will soon be entering the profession are doing so with specialties that are currently in short supply in New York State, including:

• Children’s Services (5)
• Young Adult Services (3)
• School Library Media Specialists (10)
• Digital Librarians (3)
• Rural or Small Libraries (1 plus several of the 9 interested in “Public Libraries” who plan to end up serving in rural, small town, or remote areas of New York).

Figure 23. Specialties of Making It REAL! Scholarship Students

Teaching Library Goals

The Making It REAL! grant narrative (http://www.nysl.nysed.gov/libdev/imls/narative.htm, retrieved August 2, 2007) provides this context for evaluating the Teaching Library Goals.

“The target population, prospective librarians, will be served by this project through interrelated components:
• Teaching Library Component: MLS degree candidates will be offered experience-based learning opportunities tied to pre-identified career opportunities. Each student will receive teaching, mentoring, and work experience in the Teaching Library to gain specific knowledge and acquire competencies needed for successfully filling local community needs in the identified librarian position(s).
• Scholarship Component: MLS degree candidates will receive scholarships tied to experience-based learning and diversity with a twofold purpose:”
1. To increase the numbers of candidates representing diverse groups who enter the profession.
2. To develop the capabilities of future librarians to serve diverse populations. In addition to recruiting students from diverse backgrounds, one participating library school will identify competencies required to serve patrons with disabilities and provide the learning experiences needed to attain competency.
The project partners will also institute activities to strengthen overall library education. MLS degree candidates will benefit from grant activities aimed at…
1. Strengthening the recruitment process and developing improved recruitment strategies.
2. Building Teaching Library and library school partnerships that yield graduates who both represent diverse populations and are prepared to serve diverse populations.
3. Building a website of value to recruitment and career placement.
4. Providing professional opportunities, such as programs at New York Library Association conferences, where recruits and others can learn about serving diverse populations.
5. Increasing awareness of recruitment needs among members of the library community, including library professionals, library boards, library associations, and patrons.
Teaching Libraries: This project component is based on programs of major libraries that identify high-quality candidates and assist them in their efforts to become library professionals. A variety of work experiences, mentoring, and assistance in balancing work with study characterize these programs, which draw heavily on the resources of the libraries to enrich the learning experience.
Internships are frequently part of degree requirements in library schools. However, this project’s Teaching Library component enhances conventional internships because it is based on partnerships between accredited library schools and New York State’s library systems. The library schools will provide the coursework necessary to achieve the MLS degree. The library systems will identify available career opportunities, supply the facilities of a system member library to serve as the Teaching Library, and provide professional development opportunities for students. For example, the Queens Borough Public Library has identified career openings for children’s or young-adult librarians; it will recruit two MLS candidates, with preference to diverse groups, to receive appropriate training, mentoring, and experience in its central and member libraries. Together, the library schools and the library systems will create a plan tailored to provide a rich educational experience, including the academic content, the practical experience, and the mentoring needed for the student to succeed in the available career position.”

The term “Teaching Library” itself, lacking a clear definition, has to be defined in order to ensure appropriate evaluation. First, as shown in Figure 24, the grant used the term to refer to grant partners that were in fact usually public library systems, school library systems, or Regional Reference and Resources Library Councils, ^{Note 46} as defined under New York State law (except for NYPL-branches and NYPL-SIBL that are in traditionally separate branch and Research Library Divisions of the New York Public Library System).

Figure 24. Types of Making It REAL! Grant Teaching Library Partners

These grant “Teaching Library” partners are not structurally or legally the same kinds of entities. Furthermore, New York State’s nine Reference and Research Library Resources Councils are State-funded regional library systems chartered by the New York State Board of Regents and designated to support improved access to information for the people of New York through resource sharing among the 23 public library systems, 41 school library systems and over 900 academic, hospital, law, business, large public and special libraries. Therefore there were “overlap” situations among grant partners, such as the North Country Library System being served by the Northern New York Library Network.

Second, the term “teaching library” was also used in the grant as usually understood by both grant partners and MIR students to refer to individual libraries, or units of larger systems or Councils, in which library school students apply their librarianship skills. The grant proposal itself was therefore vague about what was meant by a “teaching library.” There were no expected outcomes for the “teaching library” component of the grant other than trying out “best practices” and that students would eventually graduate from library school. Only the latter is measurable. The search for “best practices” from an evaluation viewpoint requires a multi-site evaluation perspective for this kind of a grant, for which the evaluation focuses on processes of grant implementation as well as outcomes. Often in a multi-site evaluation there is an effort to determine which of several kinds of approaches produces better or more positive outcomes in similar kinds of organizations serving different clientele, perhaps with somewhat different programs.

It was not clear if the overall evaluation should look for “best practices” in how grant partner organizations (“Teaching Libraries”) work with the individual libraries or units of the larger systems (“teaching libraries”) to arrange for and provide applied experiences for the scholarship students. Relationships between libraries within a system are themselves subject to legal constraints in New York State. Regional Library Councils, such as Northern New York Library Network, have no legal authority over the libraries that they serve – a Director of a Regional Library Council cannot require one of the libraries that it serves to do anything. The school library systems involved in the grant were both operated by the headquarters staff of Boards of Cooperative Education Services (BOCES), and primary roles of headquarters units are to enable the school libraries and school library media programs in individual school districts to participate in system-wide development of databases and library materials. The headquarters units are coordinating services related to regional resource sharing, professional development and other specialized activities. They do not themselves provide school library and library media services to school library users – the member school libraries do that. Finally, legally there are three types of public library systems recognized by the State Legislature and Governor, and all three were represented in the grant project:

• Consolidated systems chartered as a single entity under a board of trustees (e.g., The New York Public Library and Queens Library);
• Federated systems created by action of the board or boards of supervisors or legislature of the county or counties involved while member libraries retain their own charters (e.g., Clinton-Essex-Franklin Library System, Monroe County Library System, and Onondaga County Library System);
• Cooperative systems created by agreement of boards of chartered member libraries, which retain their autonomy (e.g., Mid-York Library System, North Country Library System).

The REAP Change team considered early in the evaluation whether it would be fruitful to look closely at questions concerning how having one or another of these types of partners running the grant program might result in ‘best practices” for the grant. However, in the end it decided to consider the legal arrangements of the different public library systems, school library systems, and Regional Reference and Resources Library Councils as a given part of the context of program operation. What type of system or council is best suited to running this kind of a grant program will not be an evaluative question addressed in this report.

Rather the overall MIR evaluation focused on looking for “best practices” surrounding what happens with the applied library education of students during their involvement with individual “teaching libraries” in which they are gaining experience as well as with the grant “Teaching Library” partners. In particular, the overall evaluation looks for individual partner and overall grant evidence, outputs, and outcomes related to this sentence from the grant narrative:

“Each student will receive teaching, mentoring, and work experience in the Teaching Library to gain specific knowledge and acquire competencies needed for successfully filling local community needs in the identified librarian position(s).”

Teaching Library and University Library School Interaction Expectations.

The grant narrative also speaks of interactions between universities and teaching libraries being a component of possible “best practices” for “teaching libraries.” That interaction is expected to go beyond that which typically takes place between a university library school and a library practicum site during the placement and implementation of internships required for graduation from some library schools. The ideal relationship is laid out in this sentence of the grant narrative about the practical experience students are to gain during the program:

Therefore, according to the grant narrative, the library schools would work with and through the partner Teaching Libraries (i.e., library systems, BOCES SLS, and Regional Library Councils) to set up the best practical experiences for the MIR students so that the students can have rich educational experiences. This means that there should be some kind of a collaborative effort and interaction going on between library schools, grant partner Teaching Libraries, and on-the-ground “teaching library” practicum/internship sites.

It should be noted that not all library school students are even required to take an internship or practicum in order to obtain a Master’s degree in librarianship. The federally recognized accrediting agency for library schools, the American Library Association, does not require that a library school have an internship program to be accredited. The State of New York does not require it for award of any Master’s degrees in library or information science. The State of New York does require that School Library Media Specialists gain specified kinds of practical experience in order to be certified as individual librarians, but has no requirements for internships in library schools or departments in the publicly financed State University of New York system. Therefore, while library schools in New York typically offer an internship or practicum course, not all require students to take one in order to graduate.

Figure 25 shows the graduation requirements for the library school MIR grant partners in regards to Internships or practicums. The three university MIR partners that do not usually require an internship or practicum for graduation with a Master’s degree, or only do so for Media Specialists, would have to go outside the parameters of their required curriculum to involve students to whom they awarded MIR scholarships with “teaching libraries.”

Figure 25. Partner Library School Requirements for an Internship or Practicum

Figure 26. Library Schools or Departments Attended by Teaching Library and University MIR Scholars

It would not be impossible for these three universities to do that. If there is not already an “internship” or “practicum” course in their curriculum, such practical experience might be handled as part of an independent study course.

It would be logical to expect that the impetus for teaching library-university cooperation in the case of university sponsored MIR scholars would come from the university. The university might work either with a grant Teaching Library partner or with another “teaching library” (perhaps one not even involved with the grant program other than as an internship site) to help their MIR students have a practical experience that included “best practices” above and beyond the average university internship or practicum. When MIR scholars sponsored by grant Teaching Library partners went to University library school grant partners, one might look for initiation of interactions from either grant partner, centered on good applied librarianship experiences for the Teaching Library MIR scholar.

The original proposals from the grant partner Teaching Libraries to NYSL specified the library schools with whom they expected to partner, but even at that point not all of the library schools involved were part of the grant program. In addition, when it came time to actually enroll students in library school, some students were not eligible to be admitted to some library schools with rigorous admissions requirements (e.g., Syracuse University), or deliberately chose other library schools than had been specified in the original partner proposals because of availability of certain specialties, online only educational programs, or simply because they were able to get admitted to those schools in time to meet grant deadlines. Therefore, in addition to library schools or departments that are part of the Making It REAL! program, Teaching Library MIR scholars attended four other library schools that were not grant program partners at all (see Figure 24). Two of the Teaching Library MIR scholars attended Clarion University in Pennsylvania (one of whom left the grant project), one attends Southern Connecticut State University, and one attends Texas Woman’s University – all four in online only programs. In addition, four other Teaching Library MIR scholars are getting their degrees by attending regular face-to-face classes at Queens College, the only New York State library school that did not participate in the MIR program.

Since these are all Teaching Library sponsored scholars, the question of whether the University they are attending requires an internship is moot for the students since the Teaching Library partner might arrange for applied library training in one or more of its system or Council libraries anyway as part of the Making It REAL! experiences it provided. , However, the question of whether the University the student attends is or is not in New York and does or does not require an internship for graduation is directly relevant to Teaching Library-University interactions and creation of “best practices” for “Teaching Libraries” and on-the-ground “teaching library” practicum/internship sites. REAP Change would expect that the grant Teaching Library partners that have or had students taking online courses at Clarion College or Southern Connecticut State University might have a challenge engaging these out-of-state library schools in a New York State grant program involving internships when these Universities do not require internships for graduation and the universities get no direct benefit from participation beyond the normal tuition and fee of one or two students. The teaching libraries might be asking faculty at those Universities to act outside department cultural norms and expectations concerning students getting practical experience before obtaining their degrees. Franklin-Essex-Hamilton School Library System (BOCES) would be dealing with Texas Woman’s College, an out of state school but one that had an internship/practicum requirement. That might make Texas Woman’s University more amenable to cooperation with grant goals about “teaching library” practical experience, but coordination between the online program of a university and an out-of-state teaching library might prove challenging, especially if the MIR student is trying to meet New York State (rather than Texas) media library specialist certification requirements for experience. Without the incentive of receiving any grant funds an out of state university might not even be interested in participating in this New York State focused program.

Queens College is the only New York State library school not involved with the Making It REAL! grant and was already in frequent contact with NYPL and Queens Library because of other long-standing programs those public library systems had for educating their staffs. It might see a benefit to cooperating with the grant both because of its location in New York and its long-standing relationship with MIR Teaching Library grant partners who were sending their students to it. Syracuse University received 7 MIR scholars from 6 Teaching Library partners, primarily in its online Master’s program. Syracuse University and St. John’s University are in New York State and are partners that awarded their own MIR scholarships and also taught Teaching Library sponsored MIR scholars. Syracuse University, Queens College, and St. John’s University, all New York library schools, would on the surface seem to be the most likely places for innovative Teaching Library-university relationships to develop as a result of grant participation centered on the Teaching Library MIR scholars.

Mixed Teaching Library Results

The potential confusions and issues described above were essentially built into the grant by lack of specificity about what is meant by a “teaching library,” lack of restriction on where students could enter library school, and lack of restrictions on whether library schools that Teaching Partner MIR scholarship recipients attended had to be within New York State, or had to be MIR grant partner library schools. In addition, the original partner proposals for MIR scholarship programs were of mixed quality in creativity or completeness and depth of development regarding what was expected to actually happen in Teaching Library partner/University, Teaching Library/teaching library and University/teaching library interactions to give students applied library experiences.

Often the Teaching Library partners fleshed out more of the Teaching Library–teaching library relationships, roles and expected student outcomes during specific program development and OBE plan writing. ^{Note 47} When University MIR partners included internships or applied experience in their OBE plans for their scholars, all but one university partner merely mentioned participation in internships or applied experiences and all were either vague about what was involved, or presented no specific expected activities, outputs, or outcomes different from expectations of any of their students engaged in internships. The exception was the University at Buffalo, which expected its MIR students to actively engage in reviewing and recommending improvements to a library website as an applied experience.

What actually happened in relation to development of new models for Teaching Libraries in the first year was very limited, based on what the overall evaluation team was hearing from students and from partner self-reports by August 2006. Both students and partners reported few to no interactions between the university library schools the students were attending and Teaching Library partners. Some University MIR scholars couldn’t even figure out what a “teaching library” was, or simply said in the August 2006 survey that questions about “teaching libraries” were “not applicable” since they themselves hadn’t gotten to the stage of their library school education during which they would be undertaking required internships. During the first year and a quarter of the evaluation, the REAP Change team had found so few instances of innovation of any sort in regards to “teaching libraries” that it changed the focus of a NYLA 2006 grant session called “It Takes a Village: Partners in Creating a New Generation of Librarians” away from what was originally expected to be a formative evaluation review of “best practices” in teaching libraries discovered to date in the Making It REAL! grant program. All that it had were plans. Instead, former REAP Change team member Dr. Suzanne M. Stauffer, who had become a Louisiana State University tenure-track library school professor, presented a paper at the NYLA conference in November, 2006 that looked at the “teaching library” concept historically and in a theoretical model. The floor was then opened for a general discussion among the partners, students, and others present about the “teaching library” concept and University library school-library interactions. ^{Note 48}

The paper presented by Assistant Professor Stauffer set forth a specific theoretical model of three interlocking feedback loops around what ideally might happen with educating library school students. ^{Note 49} Lacking clear criteria or guidance from the Making It REAL! grant proposal or from partners about what might constitute innovations in “teaching libraries” we will use this model to guide what we look for in this overall evaluation.