This report also available in .PDF format.

Grant Background

This plan concerns overall evaluation of the multi-site grant funded program Making It REAL! Recruitment, Education and Learning: Creating a New Generation of Librarians to Serve All New Yorkers. It was originally completed on October 31, 2005, per contract, and has been revised to its present form following unanticipated delays in program implementation and clarifying feedback from the program coordinator. The grant proposal by the New York State Library (NYSL) to the federal Institute of Museum and Library Services (IMLS) originated as a step toward realizing recommendation nine of the Regents Commission on Library Services New Century Libraries legislative initiative. IMLS funded the program in 2004 under its Librarians for the 21st Century Librarians program (Award Number RE-01-04-0053-04), with $995,630 of IMLS funds and a state match of $645,058.

The purpose of the grant program is to recruit and educate a next generation of librarians in such a way that diversity in the profession will be enhanced and new librarians will be well prepared to serve diverse populations. The New York State Library (NYSL), 12 library systems throughout the state[1] (referred to as "Teaching Libraries") and six (6) New York State schools of library and information science[2], along with the New York Library Association (NYLA), have formed a partnership to carry out such a plan. Together the Teaching Libraries and library schools have awarded scholarships funded by the grant to recruit and educate professional librarians, and cultivate a diverse new workforce that is well prepared to serve community needs, especially those of diverse groups and special populations. The annual scholarships range from $5,000 to $25,000 (if awarded through Teaching Libraries), or $5,000 to $30,000 (if awarded through library schools). Scholarship recipients agree to work in professional positions at a library, library system, or library organization in New York State libraries for at least two of the five years after earning their MLS degrees. The library systems and library schools are expected to build partnerships and to develop new means of collaboration to develop new strategies and alternatives for library education.

[1]Thirteen library Systems and Councils in the State of New York originally submitted 14 proposals to New York State Library for participation in this project: Capital District Library Council (Albany), Capital Regional Board of Cooperative Educational Services School Library System, Mid-York Library System (Utica), Monroe County Library System (Rochester), New York Public Library -- Branches and Science, Industry and Business Library (New York), Northern New York Library Network, North Country Library System, Clinton-Essex-Franklin Library System, Franklin-Essex-Hamilton School Library System, Onondaga County Public Library (Syracuse), Queens Borough Public Library (Jamaica), Rochester Regional Library Council (Fairport), and the Southeastern NY Library Resources Council (Highland). Capital District Library Council withdrew from the partnership on May 27, 2005.

[2]The six library schools partners are: Division of Library and Information Science, St. John's University; Graduate School of Information and Library Science, Pratt Institute; Palmer School of Library and Information Science, C. W. Post Center of Long Island University; School of Information Science and Policy, University at Albany; School of Information Studies, Syracuse University; and School of Informatics, Department of Library and Information Studies, University at Buffalo.

REAP Change Consultants (REAP Change), a sole proprietorship firm of Stephen C. Maack engaged in research, evaluation, assessment, planning and organizational change services, has contracted with the New York State Department of Education, New York State Library under the terms of contract C007802 to undertake the program evaluation. Dr. Maack, author of this plan, is the lead evaluator on the project. Dr. Suzanne M. Stauffer, REAP Change Senior Research and Evaluation Specialist, and Dr. Clara M. Chu, a REAP Change Associate and independent contractor, compromise the rest of the evaluation team. The contract covers the period from April 21, 2005 though August 31, 2007.

Evaluation Focus and Overview

This will be a multi-site evaluation conducted using multiple quantitative and qualitative methods that gather and triangulate data from multiple viewpoints. The focus of the evaluation is multi-layered.

• A formative evaluation of implementation at the beginning and middle of the project will provide information useful for modifying the project as it develops, and for planning new grant proposals and projects.
• An outcomes based evaluation will focus on evaluating early, intermediate, and end-of-grant period outcomes that primary stakeholders identified and proposed in the NYSL grant proposal to IMLS.
• A summative evaluation of the overall project will be made in the final report.

Stakeholder Needs for Information and Confidentiality of Evaluation Data

The evaluation addresses evaluative information needs of the stakeholders involved with this project as summarized in Table 1. It will not specifically address other information needs of stakeholders, such as legislative needs to determine if the benefits of the grant program are worth matching funds costs.[3]

Several of the stakeholders are in important hierarchical or dependency relationships with one another (e.g., funding relationships -- IMLS - Legislators - NYSL - Partners - students; hierarchical dependency relationships -- partners and library school students). As seen in Table 1 there are multiple interests and both unanticipated events and unintended results could affect how the grant plays out.

The evaluation will proceed with continual attention to issues of confidentiality and anonymity, since confidentiality and anonymity are especially important both to gather honest and accurate information and to avoid retributions that might be imposed by one stakeholder on another before the three-year project has a fair chance to develop. Each survey and interview will be prefaced with a statement as to confidentiality and use of the replies. Since the first part of the evaluation is to be formative, the evaluators will mask identity when reporting out individual quotations from the evaluative data, and provide summary totals in quantitative analyses. While it may be important in the summative evaluation to identify individual partners (library systems/councils and library schools) and evaluate their relative successes and problems encountered in this project, a constructive, appreciative inquiry approach will be used - highlighting individual successes and best practices in texts and mentioning when appropriate rather than dwelling on problem cases.

[3]Table 1 is based partly on a draft document produced by Mary Linda Todd, NYSL Making It REAL! project manager, on December 7, 2004 that sets out a logic model and outcomes for the entire project. Some items in that document are left out of Table 1, such as cost-benefit of matching funds, since the evaluation request for proposal did not charge the evaluator with investigating those matters.

Table 1. Stakeholder Interests

Students are the most vulnerable stakeholder group in the project, and the overall evaluation intends to protect any confidential information they provide to the overall evaluation as much as possible, and in ways that do not result in adverse consequences for individual students. For example, the evaluators will ask to self-report on their progress toward library and information science Masters degrees. However, only aggregate statistics about the students will be reported. Students will not be individually identified in evaluation reports. When students are quoted a general category such as "Teaching Library" will be used to frame their responses, rather than naming the specific Teaching Library or the library school that awarded the scholarship, or the recipient. This should help encourage frank student responses to questions in overall evaluation surveys, interviews and focus groups.

Using generic groupings in reporting of student responses also helps to prevent adverse consequences for project partners, notably Teaching Libraries and library schools that have awarded only one or two scholarships each, should there be negative student comments about them. The project partners will be reporting separately to NYSL regarding their own evaluations of their specific programs, and students could contact NYSL directly in case of problems they encounter with the organizations that awarded their scholarships. NYSL will have sufficient opportunity to address concerns about specific programs based on those reports, without the overall evaluation reporting out individual results.

Evaluation Data Breakouts

The 19 partners other than NYSL and NYLA operate in widely varying organizational contexts and it is important to consider those contexts during the evaluation, summarizing across similar contexts when possible. During the formative, outcomes and summative parts of the evaluation REAP Change will always report out results broken into categories of "Teaching Libraries" and library schools. During the evaluation of the recruitment phase, REAP Change began searching for other ways of summarizing results that provide evaluative insights (e.g., public/private library schools; school/federated/ cooperative/consolidated library systems plus library councils; school and public libraries; metropolitan New York City area/upstate New York), although so far mainly the metropolitan New York City/upstate New York distinction has proven to be somewhat helpful in interpreting evaluation data findings. The nature of a multi-site evaluation is to be exploratory and to consider implementation in different contexts, so throughout the evaluation the search for appropriate contexts will continue.

Statistics to be used in this evaluation will be primarily descriptive (e.g., counts, percentages, proportions), with non-parametric statistical analysis and trend analysis for significant changes added when appropriate. The primary reason for this statistical analysis approach has to do with small sample sizes, whether one is looking at partners (N=19) or students (N=36 to 42). Several of the partners also have no statistical variation possible, since they have awarded only one scholarship. In such circumstances parametric statistical techniques based on random, independent sampling from a population of partners or students won't work, and complex modeling approaches such as Hierarchical Linear Modeling (HLM) do not have enough N to apply statistically valid approaches. Non-parametric statistical methods can, however, be used to determine whether differences are statistically significant or not, as long as there are not too many categories in play so that there is the possibility of some variation in individual data cells. As noted above, the best way to break out an categorize data will be a matter of empirical investigation as well as development of a better understanding of which theories best apply to the situations being observed and cannot be fully specified at this time.

Diversity Focus in the Overall Grant Evaluation

The stress in this grant on diversity provides special challenges, both because the definition of "diversity" adopted for this grant is broad, and because "diversity" needs to be evaluated at the levels of students and organizations. In August 2004 Project Director Mary Linda Todd sent an e-mail to the partners with these comments by Tracie Hall and clarification about "diversity in context" that will provide clarification on how to define "diversity" in the evaluation when evaluating individual student outcomes.

"I recognize the often amorphous nature of the term "diversity." As part of our own Diversity Action and Inclusion Plan at the Association we've acknowledged the following facets of diversity: 'The American Library Association recognizes that in addition to race, creed, color, religion, gender, disability and national origin, there are a multitude of differences (language origin, regional and geographic background, economic class, education, learning and communication styles, sexual orientation and personal lifestyle) that individuals bring to the workplace.' The "protected classes" according to the 1964 Civil Rights Act and one of its enforcement agencies, the Equal Opportunity Commission (EEOC), are: race, creed, color, religion, gender, disability and national origin, all of which are included in the larger category of those protected by Affirmative Action. I am sure that you will hold to the definition of "diversity" that you defined in your IMLS grant application and in doing so, it is important to pay attention to "context." For example, the gender make-up of librarianship being nearly 82% women might mean that focusing on female recruitment is not a diversity priority, but the recruitment of librarians from rural communities who will go back and serve in rural libraries (many of which are staffed by non-degreed library staff) could be critical to both regional diversity as well as equity of access. Also, geographical location needs to be tossed around a bit in the name of context. The question again, is what kind of "diversity" will recruitment of people from upstate New York bring to the profession? Looking at the demographics for New York, which is now according to the Census, a slow growth state (meaning that demographics may shift but there are not dramatic increases in population -- an example while the "Black" population in NYC between 1990-2000 stayed pretty static much of that was a result of new African immigrants from Senegal, Guinea, Ivory Coast replacing African-Americans who moved from NYC to places like Georgia and the Carolinas) warrants looking at diversity in terms of ethno-linguistic community, nationality, race and ethnicity. There may also be a case for intensive recruitment in high-density, economically disadvantaged areas (i.e., Buffalo, Poughkeepsie, some of NYC's boroughs), etc. Take a look at the CensusScope's dissimilarity index to start.

Tracie Hall expands this further to note in a contextual sense, men would increase diversity in the female-dominated library profession, and recruitment of librarians from rural communities who will go back and serve in rural libraries could improve regional diversity and equity of access. She suggests also looking at diversity in terms of ethno-linguistic community, nationality, and economical disadvantage.

The evaluation is charged with looking for "best practices" and organizational changes in regards to diversity in organizations and in library relationships with diverse communities. When looking at organizational change, REAP Change intends to use concepts derived from a research based theoretical article, Thomas, D.A., & Ely, R.J. (1996) "Making differences matter: A new paradigm for managing diversity," Harvard Business Review, 74, 74-91 to help it understand and evaluate organizational responses to diversity. Thomas and Ely identified a "discrimination-and-fairness," an "access-and-legitimacy" and a "learning-and-effectiveness" paradigm that organizations follow in regards to dealing with diversity in the workplace, and discuss organizations transitioning from one paradigm to another. This is a constructive approach for understanding and evaluating what happens in organizational responses to diversity in the Making It REAL! program as it is being implemented in different Teaching Library and university settings. The evaluators are also considering using another article that expands on Thomas and Ely's article, which is Selden, S.C., & Selden, F. (2001) "Rethinking diversity in public organizations for the 21st Century: Moving toward a multicultural model," Administration & Society, 33(3), 303-329. For understanding community diversity, the evaluation will depend on secondary data sources (e.g., U.S. Census Bureau dissimilarity index, written descriptions of communities) as well as interviews with partners and (as necessary) community leaders on matters related to partner-community relationships in regards to the Making It REAL! program (only).

Report Formats, Lessons Learned, and Report Dissemination

REAP Change will include graphs and tables as well as text and an Executive Summary in major full reports to NYSL. Since the evaluative approach uses multiple methods and seeks to gather evaluative data from multiple perspectives, major reports may include extensive quotes in the words of respondents, often placed in Appendices. Out of respect for the integrity of the responses, the quotes will be verbatim, including any spelling or punctuation errors made by the respondents.

In a multi-site evaluation the evaluator does not act as the arbiter of any singular "truth," but rather as the elucidator, illuminator and analyst of a variety of approaches being tried out in a variety of settings by a number of organizations working on their own individual projects within a broad theme. The reports may express multiple viewpoints about the same subject, not all of which will be in agreement with one another. The evaluators will make a concerted effort to distinguish who contends what, to distinguish factual evidence and to identify evaluator analysis. Because of the many different organizational types and environments in which the project is taking place, REAP Change considers it important to triangulate evidence -- qualitative evidence from different lenses or perspectives and from written texts, as well as quantitative evidence from counts, proportions, and tabulation of opinion data -- before drawing evaluative conclusions. In this regard the reader is cautioned that "truth" may differ depending on who one asks about a situation, and that what can be demonstrated to have actually happened may differ from planned or ideal behavior.

In addition to its evaluative conclusions, REAP Change will include "lessons learned" sections in its reports to NYSL, and its understanding is that these full reports or a summary of them may also be transmitted to IMLS. These "Lessons Learned" are to be viewed as evaluator summaries, evaluative comments, and professional identification of some opportunities for different future actions rather than direct recommendations on which NYSL, partners, or IMLS should or must act. They are put forward for NYSL and IMLS consideration as those organizations plan or implement future policies and actions.

REAP Change recognizes that NYSL, with whom it has contracted, owns all the evaluation data and reports made by REAP Change. Since the project is funded with public moneys, any of the reports and data should be available, as a matter of public record, under the Freedom of Information Act after they are submitted and accepted. However, given the many stakeholders directly involved, to increase report usefulness REAP Change recommends to NYSL that the overall evaluator be approached to prepare multiple reports or summaries on the same topics geared to different audiences. For example, the following audiences would be appropriate to consider for separate reports or report summaries:

• New York State Library (full reports)
• IMLS
• Making It REAL! partner agencies
• Making It REAL! scholarship students
• The media and general public (Making It REAL! website of NYSL, and/or media releases)

To the extent that it can be done within the estimated time and dollar limits for the overall evaluation contract, REAP Change is willing to negotiate its role in producing reports or summaries geared toward any or all of these audiences.

In order to keep reproduction and travel costs down, REAP Change recommends that electronic distribution and possible discussion/feedback on overall evaluation reports be used as an approach to reaching Making It REAL! partner agencies and scholarship students. The Making It REAL! project already has a Yahoo Groups website on which evaluation reports could be placed for partners, for example, and e-mail addresses for partners and students that could be used to distribute findings. It might be possible to encourage comments of the overall evaluation reports through messages to the partner Yahoo Group! Website, or set up a Webinair style presentation/discussion of reports that partners and/or students could log into over the Web and telephone in questions. REAP Change recommends negotiating the extent and feasibility of producing multiple reports for multiple audiences with NYSL as soon as possible.

NYSL is interested in broad dissemination of findings from the evaluation and REAP Change Consultants staff are interested in making presentations of work or results of this project at professional conferences or in publishing in professional journals. REAP Change staff are interested in collaborative publication or presentation possibilities done with NYSL or partner staff or faculty members. REAP Change staff will always seek permission of NYSL for any professional presentations or publications related to the Making It REAL! overall evaluation. For example, REAP Change Consultants' project staff of three are all planning to attend the New York Library Association (NYLA) 2006 conference to collect additional evaluation data from partners and students, and are willing to participate in one or more presentations at that conference if there were an opportunity to do so. Other than NYLA 2006, however, REAP Change has not included any budget in its contract with NYSL for presentations at other conferences, and has no corporate budget earmarked for conference attendance, so in-person presence of REAP Change staff at conferences would require negotiation regarding coverage of costs.

Finally, per prior agreement with Mary Linda Todd, Stephen C. Maack, the lead REAP Change consultant, has obtained permission to produce a special report on the recruitment phase of the Making It REAL! project as part of his participation in the 2005 Howard University Evaluation and Training Institute (HU/ETI) that focused on mid-career training in culturally and contextually competent evaluation. This report will be based on the partner and student surveys regarding recruitment, and the findings of implementation of the Making It REAL! recruitment phase that have already been reported to NYSL in the REAP Change report of October 7, 2005 titled "Making It REAL! Student Recruitment:: Library School and Teaching Library Survey Results." NYSL will be given an opportunity to review Maack's final report to HU/ETI that is due in April 2006.

Organizing and Retaining Evaluation Information

During the evaluation REAP Change Consultants will maintain file copies of its surveys and reports at its main office, 2872 Nicada Drive, Los Angeles, CA 90077-2024. It will also maintain back-up copies of computerized data on CD-ROMs and may store these at a secure remote location. Computer software at the main office of REAP Change Consultants to be used in the data collection and analysis currently includes a Dell Latitude C800 laptop computer loaded with SPSS, Word, EXCEL, and ArcView 9.1 software, an HP 135d four-in-one machine and an HP 4p printer, a wireless system to link to a cable modem plus a back-up landline telephone modem, a landline fax/answering machine, a cell telephone, and additional personal computers, hardware, and software owned by staff.

REAP Change Consultants has also set up a free Yahoo! Group called REAPNYSL, for sharing and exchange of files, reports, documents, and e-mails among the staff assigned to the project. The Yahoo! Group site, which is hidden from the general Yahoo! Group list and only accessible to REAP Change staff, also acts as a remote back-up site for reports and electronic documents in the event of catastrophic destruction of source data at the REAP Change Offices. REAP Change has opened a Survey Monkey (web-based) subscription for delivery, completion, and data collection from online surveys and is charging New York State Library for costs of this subscription during months in which it is used from Making It REAL! evaluation data collection (as specified in its cost proposal to NYSL).

REAP Change will return all of the original data collected to NYSL at the conclusion of the project or if removed earlier from further evaluation responsibilities (whichever comes first), and not maintain copies of evaluation data after that point at its offices, the Yahoo! Group web site or the Survey Monkey web site. REAP Change will have no responsibility after terminating its involvement with the project and returning original data for maintaining copies of the data used in the evaluation and the reports it makes. REAP Change will retain paper and/or electronic copies of all its project reports, blank questionnaires and other data collection instruments for its own records and legal protection for a minimum of five years after completion of its project involvement.

Evaluation Team

REAP Change is using a highly qualified team to perform outcomes based and implementation evaluation that fosters diversity and change in New York State libraries and library schools.

The three key REAP Change team members have provided effective results from outcomes based evaluation, have evaluated library programs, have New York library experience, and have had careers focused around diversity issues. The members of the team have proven research and program evaluation experience in library or higher education arenas based on a foundation of empirical qualitative and quantitative methods.

Dr. Stephen C. Maack, the Founder and Lead Consultant of REAP Change Consultants, has over 25 years of evaluation, research and planning experience in the public sector and higher education. A former university Director of Institutional Research with a Columbia Ph.D. in Anthropology, he is skilled in both qualitative and quantitative evaluation and research methods. He has worked in an African-American led organization, two Hispanic serving institutions of higher education, supervised deaf and ethnically diverse employees, and researched first generation in college students. Dr. Maack is the evaluation team leader and key contact person during the evaluation.

Dr. Suzanne M. Stauffer has experience as an MLS certified public librarian in New York and California with diverse populations, and in academic library management. Having recently completed her PhD in Information Studies at UCLA, she is a skilled researcher and evaluator. She has experience providing public library services to Hispanic and African-American clients, working with immigrant Asian students and cataloging library materials for developmentally disabled persons. Dr. Stauffer is employed by REAP Change for this project as a Senior Research and Evaluation Specialist.

Dr. Clara M. Chu is a nationally recognized expert on diversity in libraries who is interested in bringing transformative change to the library profession. An Associate Professor in the UCLA Department of Information Studies, Dr. Chu teaches courses on library planning, research and evaluation - and practices what she teaches. She has dedicated 20 years to working with multicultural communities and over 10 years to conducting empirical research on multicultural librarianship and equity of information services. She actively recruits people of culturally diverse backgrounds into the information profession and the professorate. She received an IMLS grant at UCLA focused on diversity and has completed IMLS training in outcomes based evaluation. She is subcontracting with REAP Change as an independent consultant providing expert advice, guidance and specific evaluation services.

The evaluation team also includes a graduate student researcher already employed by New York State Library using Making It REAL! grant funds not charged to the evaluation. The current graduate student researcher has been maintaining contact with scholarship students and partners, updating contact information, helping distribute and collect surveys and other evaluative materials, and providing general help for the evaluation. NYSL is also using the graduate student researcher as the list owner of a Making It REAL! Yahoo! Group at http://groups.yahoo.com/group/makingitreal_group/ that was set up following a formative recommendation of REAP Change Consultants.

Specific Evaluation Plans

REAP Change will use various evaluation approaches for different phases of the project.

Project Start-up and Recruitment Phase

A formative evaluation in progress as of this plan is addressing the following evaluative questions about the creation, implementation and recruitment phase of the project.

• What is the background of the Making It REAL! project? How, when, and why did it develop? Did it start up as planned or were there unexpected obstacles?
• How did the project process in relation to scholarship student recruitment? Did it proceed as planned? If not, why not? What unexpected obstacles and opportunities were encountered?
• How did the Teaching Libraries and the Universities go about doing recruitment? What recruitment "best practices" did they use or develop?
• How might the recruitment process or program be improved in terms of different actions, policies, grant funding terms, etc.?

The table below shows the evaluation methods with responsible staff (lead staff are in boldface).

There will also be an outcome evaluation component for this part of the evaluation.

The definition of "diverse students" is found in program documents and given above. The determination of nationally known diversity student recruitment "best practices" will be made in comparison to the results of the library school and library diversity recruitment literature review described above.

Evaluation results will be reported through:

• A full report[4] to NYSL based on the partner recruitment survey due by September 2005, with Executive Summary.
• A full report to NYSL based on the student survey due by February 28, 2006, with Executive Summary[5].

[4]REAP Change Consultants submitted an advanced draft of Making It REAL! Student Recruitment: Library School and Teaching Library Survey Results on September 21, 2005 and after discussing it with New York State Library staff on September 30 and reviewing additional project documentation, revised and resubmitted a final report of the same name on October 7, 2005.

[5]Because five scholarship students were awarded scholarships in late October 2005 by SUNY, Buffalo after funds released by the withdrawal of Capital District Library Council were transferred to that University, the deadline for closing the student survey was extended and an extension of the report date was requested to that shown above.

As discussed above, REAP Change Consultants will negotiate with NYSL regarding possible preparation of additional reports or report summaries for the following audiences: Making It REAL! partners; student scholarship recipients; the media and general public. Dr. Maack will be primarily responsible for report writing, with Drs. Stauffer and Chu contributing evaluative comments based on partners with whom they are working, and when reviewing report drafts.

Making It REAL! Mid-project Implementation Phase

Once Making It REAL! scholarship students actually begin library school the project is conceived for evaluative purposes as moving into a mid-project implementation phase. This period will run from late August/early September 2005 through July 2007 and is overlapping to some extent and at some locations with the recruitment phase. Based on the original project design, several things are expected to happen during the mid-project implementation phase:

• Students are expected to move successfully through graduate school and progress steadily toward Master's degrees in library or information science;
• Teaching Library and library school partners are expected to provide mentoring and other planned program services to students and as part of their implementation of the Making It REAL! project, and to report to NYSL on their progress based on their own outcomes based evaluation (OBE) plans;
• Teaching Library and library school partners involved with the Making It REAL! program are expected to develop new ways of interacting and collaborating in the education and training of the scholarship students.
• All partner staff and students are encouraged to communicate about the project to colleagues and others through professional meeting presentations, papers, and publications.
• NYLA is to hold diversity related sessions or workshops at its 2005 and 2006 annual meetings.
• NYSL is to contract with a website developer who will set up a website designed to communicate about librarianship and library careers.

In its response to the RFP for the overall evaluation, REAP Change identified some of these expectations as either "outcomes" or "outputs" of the project. However, then and to date, the criteria for successfully achievement of these "outputs" or "outcomes" are poorly defined by the parties responsible for implementing the programs - NYSL and the partners. REAP Change anticipates that the evaluation plan presented here will be further refined as it works with NYSL and perhaps a group of partners to more clearly define expected outcomes for the overall evaluation. However, REAP Change expects that most of the evaluation of the mid-project implementation will be an evaluation of process and implementation, and a multi-site comparative evaluation of a formative nature, rather than an outcomes-based evaluation.

As part of its contractual obligations, REAP Change staff have been working with the partners other than NYSL and NYLA to develop good OBE plans for the individual partner projects. Because of delays both in partner completion of their OBE plans and in REAP Change reviews this work is not yet completed as of the writing of this evaluation plan. If the partners follow through on implementing and reporting on their individual project OBE plans, then that reporting can become one of several data sources for evaluation of program implementation. However, the overall evaluation depends on multiple sources from multiple sites, so will also gather information from REAP Change staff interviews or focus groups, as well as possible surveys of partners and review of the OBE plan reports. In addition, the overall evaluation is proceeding by looking at the project from the varying perspectives of difference stakeholders. Again, the overall evaluation will seek out student opinions not only about their own progress, but also of how the Teaching Libraries and library school programs are working out for them.

Finally, the overall evaluation will need to gather separate evaluative information about the website development and use, the NYLA sessions, as well as the overall funding environment for the New York State library organizations.

Evaluation of the mid-implementation phase of the project will address the following questions in the ways shown below each.

• How and how well are the scholarship students progressing through library school? Is there any variation in progress among diverse and other students? Are there unexpected obstacles?

• What kind of mentoring or other support services, if any, are partners (Teaching Libraries or library schools) providing to the scholarship students?

• How are partners progressing in relation to their own implementation and OBE plans? What obstacles and opportunities are they encountering? What approaches or which kinds of partners are doing better or worse in implementation and why?

• What kinds of cooperative efforts did Teaching Libraries and library school partners mount? What new ways of interacting and collaborating in the education and training of the scholarship students have these partners developed, if any? Which of these might be considered "best practices" and why?

• In what ways and how often are partner staff, faculty, and students communicating to others about the project?

• Are the NYLA diversity related sessions or workshops at its 2005 and 2006 annual meetings successful in general? Do Making It REAL! stakeholders attend these sessions, what do they learn from them or do with new information gained?

• How can a website best be used to advance library careers and communicate with diverse potential students and the public about librarianship? How is the new career web site developed? How good is it? How often is it used? What is the response of website users? Does it make a difference in communication of information about library careers and librarianship?

• What new recruitment approaches, if any, do Making It REAL! partners develop or try during the project, particularly in regards to attracting diverse applicants to library school or open positions, without having additional Making It REAL! scholarship funds available? What impact does the fiscal environment have for employment or scholarship recruitment in these regards?

• What changes, if any, occur in the partner organizations as a result of participation in the Making It REAL! project? What new understandings or approaches to diversity, working with diverse staff or providing library services to diverse communities can be traced back to the Making It REAL! project? What "best practices" in regards to diversity, if any, are identifiable?

As shown in the timeline below, there are fewer data collection instruments and methodologies than might appear from the tables above. For example, focus groups at NYLA 2006 will cover several evaluation question areas; document reviews of OBE reports will look for several kinds of information; interviews with partners and students will have multiple foci to them. This approach is taken to minimize evaluation costs, provide parsimony in data collection efforts, and minimize the number of times that busy partners and students need to be contacted for overall evaluation information.

Consolidated Timeline for Data Collection in Making It REAL! Mid-Project Implementation Phase

Mid-project Implementation Outcomes Evaluation

The expected outcomes for the mid-project implementation phase of the project are not very well defined in the original grant proposal or the RFP for the overall evaluation. There has been some further clarification of partner expectations of mid-implementation outcomes in their OBE plans, but these are not yet approved for all partners. The following outcomes are based on current REAP Change understanding of the project as it has developed and are subject to review by NYSL and partners. In particular, REAP Change has proposed some target achievement levels that should be reviewed and agreed to or modified by NYSL and other partners. In some cases the target achievement levels and criteria will be those specified in individual partner OBE plans.

REAP Change has identified six possible mid-implementation outcomes, which are presented one per page on the next six pages in the standard outcome statement format being used for this project. They and their criteria are subject to further discussion with the project coordinator and partners. These are shown as Outcomes 4 to 9 because there were three recruitment-related outcomes and the numbering scheme is being continued.

Mid-project Implementation Evaluation Reports

There will be at least a brief report provided after each data collection specified in the tables on pages 17 and 18, including monthly statistical reports with trends on website use. Later reports will specify changes from earlier reports of the same type.

By mid-March 2006 and January 2007 REAP Change will produce annual reports summarizing everything learned from all sources to date, organized according to the evaluative questions specified above and with information and analysis related to the outcomes specified below. These annual reports will be supplemented with shorter updates submitted to NYSL by mid-September of 2006 and mid-March of 2007 to provide information for April 1 and October 1 narrative reports that NYSL must submit to IMLS.

Relevant to the evaluation, but not strictly a part of it, but per contractual obligations, REAP Change will produce updates in August 2006 and August 2007 of newly published bibliographic information on diversity in libraries and library recruitment, to supplement a report on this subject prepared by Professor Clara Chu in July 2005. These two reports will include diversity related materials that REAP Change staff may have consulted when drawing their evaluative conclusions. REAP Change will use appropriate bibliographic citations to such literature when it specifically informs comments in its other reports.

Summative Evaluation

A summative evaluation will take place at three time points:

• When a scholarship recipient drops out of school;
• When a scholarship recipient completes a Master's degree in library or information science and seeks work as a professional librarian;
• Approximately one month before the end of the evaluation period (i.e., July 15 to August 15, 2007).

By gathering data when a scholarship recipient either completes a Master's degree or drops out of the program by dropping out of library school REAP Change expects to be able to provide summative evaluation comments about both project successes and failures.

The evaluation is scheduled to end on August 31, 2007 even though the grant period ends in December 2007 and the scholarship criteria require recipients to serve for at least two years in a library in the state of New York during the five years following their graduation. Therefore, the summative evaluation that REAP Change will provide cannot be considered to be a truly summative evaluation of the whole grant project.

Summative Evaluation Questions

The questions to be addressed in the summative evaluation are these:

• How successful was the implementation? What can be learned from problems as well as successes to guide future grants? What unintended obstacles were encountered? How could one correct for or overcome those obstacles in future projects with similar goals?
• What was learned about recruitment and education of diverse students? What worked and didn't work in regards to diversity and why?
• What new or improved approaches to serving diverse communities were developed as a result of this grant project?
• Teaching Library model -- what works and what new education approaches are used? What are "best practices" for Teaching Libraries?
• How can Teaching Libraries be integrated into library school education programs? How and to what extent should library schools pay attention to the practitioner needs and concerns of working librarians?
• What intended and unintended library-community and library school-Teaching library partnerships developed? What was the focus and what were the results of the partnerships?
• From the student perspective, are the scholarships and studies worth the effort and work obligations on completion of the degree?
• From the partner perspective, was the scholarship program worth the effort needed to mount and administer it?
• What are the benefits of the grant project for constituents (individuals and communities)? Are the benefits worth the costs?
• How can the project be sustained, and what would be the best form in which to sustain it?

Summative Evaluation Methods and Approaches

The summative evaluation will take place using the methods and approaches specified in the following table. In the references to "pre-test/post-test" evaluation designs regarding student stakeholders, the "pre-test" information will come from selected questions in the student survey done as part of the recruitment phase formative evaluation. This survey includes questions regarding student reasons for entering the Making It REAL! scholarship program, expectations of it, reasons for entering librarianship, and career plans. While many of these "pre-test" questions were qualitative, it may be possible to do some pre-test/post-test statistical comparisons on certain questions.

The final two evaluative questions will be addressed in the reports as summary comments. Fiscal costs may be considered to the extent that NYSL makes available project cost-to-date information and estimates can be made of community benefit values. However, it will not be possible to undertake a full cost-benefit analysis, given the timing of the completion of the evaluation before the end of the project, and the resources made available for performing the evaluation.

Summative Outcomes Based Evaluation

The basic summative outcomes based evaluation questions are:

• Did the grant meet its targets? How many new librarians entered the profession in New York State, and what proportion were from diverse groups?

The data needed to answer these questions will be gathered through careful record keeping throughout the project, and using the methods detailed above. Since the final evaluation report is due on August 31, 2007, which is before the grant period ends in December, 2007, both actual graduation as of August 15, 2007 and anticipated estimated graduation figures as of December 31, 2007 will be report. Continuing the numbering scheme, these are shown below as Outcomes 10 and 11.

At this point the Making It REAL! grant stakeholders have not specified outcomes-based targets or criteria for graduation of the diverse scholarship recipients. Rather than imposing outcomes-based targets or criteria for Outcome 10 or a new Outcome, REAP Change proposes two possible approaches for discussion that appear to be consistent with the intent of the grant and possible later modification of this evaluation plan:

• Diverse scholarship recipients as a subgroup might be expected to graduate at the same rate as non-diverse scholarship recipients as a subgroup;
• Diverse scholarship recipients as a subgroup might be expected to graduate at a higher rate than non-diverse scholarship recipients as a subgroup.

This distinction is most important for library school scholarship recipients, and is in fact moot for Teaching Library scholarship recipients if they meet their graduation target of 100%. Since one of the purposes of the grant is to increase the number of diverse Master's degree librarians working in New York libraries, library systems, or library organizations, it would not seem to be consistent with grant purposes to expect diverse scholarship recipients as a subgroup to graduate at lower rates than the non-diverse scholarship recipients, even if any graduating diverse recipients could be viewed as increasing the number of diverse librarians in the state. Since the diverse scholarship recipients have been and can be identified, it would be simple to do a data breakout and check for additional indicators of this sort.

Similarly, the grant documents do not specify what proportion of the graduating students, or graduating diverse students should obtain professional jobs by the end of the evaluation or the end of the grant period. In fact the students have up to five years after graduation to work two years in librarian jobs in New York State. Therefore, the evaluation will simply check the proportion of graduating students and of all students in the program who have completed jobs by the end of the evaluation period in August 2007. The target has been set in discussions with the program coordinator and will be evaluated in relation to the general job market for librarians in spring and summer 2007.

The grant and the scholarship stipulations are that each student "shall work at a library, library system, or library organization in New York State for at least two years after he/she graduates". This could be stated as an Outcome 12, but this has not been done since the REAP Change contract does not include funds to track graduates or perform a follow-up survey or count two or more years after graduation (i.e., by December 2009). NYSL may wish to do this tracking and reporting as an impact (long-term outcome) of the Making It REAL! project.

Final Report

The final evaluation report produced by August 31, 2007 will include summaries of findings from the earlier formative evaluation reports and summative evaluative comments about them based on the final round of data collection to take place in Summer 2007. It will also report out the results of the summative data collection outlined above, in relation to implementation, process, and summative outcomes. It will identify costs and detail benefits, but fall short of being a full-fledged cost-benefit analysis. The summative evaluation will also include identification of any "best practices" that have surfaced during the evaluation in regards to student education, Teaching Library-library school partnering, training and involvement of diverse librarians in school and in the Teaching Libraries. It will summarize what has been learned in regards to recruiting, educating, and including diverse people as librarians, and discuss organization changes (if any) in regards to diversity within the Teaching Libraries and library schools that can be traced to participation in this project. It will detail unexpected obstacles as well as unintended consequences (good and bad) of the project. Finally, at the option of NYSL, the summative report will include a section that briefly reviews and summarizes results and key aspects of the experience of each of the 19 partner Teaching Libraries and library schools.

Last modified on March 1, 2006 -- asm
For further information, please contact Mary Linda Todd.
URL: http://www.nysl.nysed.gov/libdev/imls/evalplan.htm