Evaluation of the LSTA Program in New York State; final report, December 2001
Final Evaluation Report
The Library Services and Technology Act Program in New York State: October 1997 December 2001
Report prepared by
Dr. Kathleen Toms
CDA Corp.
for
The State Education Department
The New York State Library
December 2001
| Go to Table of Contents | This document in .PDF format (960K) |
Executive Summary
This report presents the findings of the evaluation of The New York State Library Services and Technology Act (LSTA) Program by CDA Corp.. The evaluation was carried out between November 2000 and December 2001. The LSTA Program includes both Grants and Statewide Services components and the evaluation of both of these component types are covered in this report. An overview of the evaluation design, presented in the introduction, identifies the six key informant groups which were targeted in the acquisition of data. They are; Reference and Research Library Resource System Directors, Public Library System Directors, School Library System Directors, Central and Co-Central Library Directors, LSTA Grant Project Directors and members of the staff of the New York State Library.
Section One of this report presents an overview of the use of LSTA funds and the positive impacts that have been felt as a result of that funding. Survey results and focus forum participant responses indicated that LSTA funds facilitated and supported the objectives of the New York State Library Services and Technology Act Five Year Plan October 1, 1997 September 30, 2002. A majority of grant project directors agreed that their system had learned a great deal through the LSTA supported opportunity to innovate. Both the LSTA Grants Program and the development of the state virtual library network in the EmpireLink, and more recently, NOVEL strategies, were easily recognized by most key informants as important to the future quality of the library services in New York State. Reporting on the impact of Statewide Services, focus forum participants reported that the impact is on the ability of their system (Reference and Research Library Resource System, Public Library System or School Library System) to help their system member libraries to accommodate the dynamic needs for library services. They also noted that it is their use of Statewide Services that helps the system directors to support strategies which will stimulate change and transform services in their regions.
Section Two of this report presents information regarding the achievement of the two objectives stated in the New York State LSTA Five Year Plan October 1, 1997 September 30, 2002. The first objective, developing electronic doorway library services was explored with regard to both Electronic Content and Training for Technology. Library system directors reported that LSTA grants have been used to partially fund electronic conversion of bibliographic records. Linkage of these bibliographic records through system and regional catalogs to create a statewide virtual library was also examined, both in terms of its present status and future needs. When asked which areas of library development will need the most support during the next five years, access to commercial databases was chosen most frequently by the directors of each of the three library system types in New York. In regard to the use of LSTA Grants for Training in Technology, it was reported that there is a predicted need for professional development in all areas, with high levels of increase in future need compared to past use of professional development support from LSTA Grants. Resolution of telecommunications issues was also reported to be extremely important to the future technology development of the library systems by the system directors.
Study of the second objective in The New York State LSTA Five Year Plan, Encouraging Information Empowerment Through Special Services to Increase Access, is also reported in Section Two. The LSTA Information Empowerment Through Special Services Grants Program in New York emphasizes the role that public libraries and public library systems play in promoting Adult and Family Literacy and Economic Opportunity to help all New Yorkers achieve more independent lives. A majority of the LSTA Grant Project Directors and Public Library System Directors indicated that their system had used LSTA funds to enhance or expand services for individuals. Open-ended responses on the Statewide Survey indicated that some of the library system directors believe that outreach to new populations can be enhanced through networking and collaboration with other community agencies.
A discussion of specific issue areas within the changing context of library services and of advocacy and policy information dissemination as a form of change management and support is presented as part of this section of the report and focuses specifically on the role of Statewide Services in supporting responsive programming in libraries. Three issues related to the changing context of library services; collaboration within the library community, collaboration outside the library community and changes in professional practices among librarians in the State is also included. The report indicates that there is a great deal of collaboration in New York State both within library system types and across library system types. All three types of system directors also agree that collaboration is a powerful tool in bringing libraries to the table at all levels of policymaking. Library system directors and the directors of member libraries indicated that there had been a significant change in services for users over the five-year period covered by the present LSTA Five Year Plan. The professional practice of librarians has changed considerably as well. A high magnitude of need for professional development designed to address issues of changes in library practice was noted, as well as the need for the development of an infrastructure which will support the changes which are now taking place in professional practices and the role of libraries.
The evaluation found that the New York State Library is seen as the source of policy level information for the library service delivery system in the State. A majority of the directors from all library system types indicated that they contact the New York State Library for information for planning and advocacy and that it aids in their system level planning.
Findings regarding operations and management of the LSTA Grants Program are presented in Section Three of this report. System directors and project directors were asked about possible barriers to grant participation and questions about funding and institutionalization of LSTA grant funded projects. They expressed the opinion that the way the five year plan is put in place to manage the Library Services and Technology Act program in New York causes some problems. They noted that in a rapidly changing environment such as they find themselves in at present, the five year plan should be revisited at least every two years, in order to provide more flexibility which would allow for proactive programming by libraries and library systems.
Regarding the identification of best practices, the focus forums revealed some themes. They expressed the belief that if the State is to move forward in the innovative environment, the program developers at library system and member library level have to receive clearer information regarding what innovations have been tried and tested and with what result.
Focus forum participants discussed the quality of the communication of grant requirements and funded program information from the New York State Library to them, and the within system communication networks that do and do not exist in our State. Actual management logistics of the grant program are considered to be clear and easy to follow by a majority of the Grant Project Directors. Issues which arose around the communication of the nature of the type of proposals which would be funded, and of changes to State level priorities from year to year were described. It was noted from the focus forum that the use of a ListServ to disseminate information to them is not efficient. Those participants expressed the belief that the State Library wants to communicate with them but that using more cutting edge technology to reformulate communication strategies would be helpful.
With regard to Change Management, focus forum participants noted that the next phase of change in the State will have to be more focused, more closely managed and therefore implemented with more open, clear and regular communication among the concerned parties.
The evaluation found that the time restrictions on the grant projects is seen by many participants as having a negative impact on the quality of the programs that are delivered. The other restriction which is seen by many of the system directors as a barrier to institutionalization of LSTA funded innovations is the inability to use the grant funds to pay existing staff to work on grant funded activities.
The nature of innovative funding and continuation of support is also discussed in Section Three of this report. Library system directors indicated a strong belief in the uses of innovative funding to support only innovation and expressed their commitment to encourage a restriction of the use of funding such as that provided by LSTA to support ongoing programming. It was reported that LSTA funds are often a small part of the overall funding for one of these projects with additional support being provided by local budgets. This means funds are leveraged prior to the implementation of the program being funded, thus insuring marked rates of institutionalization of innovative practices.
Findings from this evaluation indicate that the use of LSTA funds has made a positive contribution to the development of the system for library service delivery at the local, regional and State levels in New York State. For a complete report of the conclusions and recommendations of this evaluation, please see Section Four of this report.
Table of Contents
Section One: Library Services and Technology Act Funds in New York: Use and Effect
- Uses of the Funds
- Reported Positive Effects of Participation in the LSTA Grant Program
- Reported Positive Effects of EmpireLink
- Reported Positive Effects of Other Statewide Services
Section Two: Achieving the Objectives in the New York State LSTA Five Year Plan
- Objective 1: Developing Electronic Doorway Library Services
- LSTA Technology Grants Program in New York
- LSTA Grants to Support Electronic Content
- Future Trends and Perceived Needs: Electronic Content
- LSTA Grants for Training in Technology
- Future Trends and Perceived Needs: Professional Development
- Future Trends and Perceived Needs in Other Areas
- Statewide Services Supporting Objective One
- Leadership in Planning and Development
- Development of NOVEL
- Objective 2: Encouraging Information Empowerment Through Special Services to Increase
Access
- LSTA Information Empowerment Through Special Services Grants Program in New York
- Services to New Populations: Present and Emerging Needs
- Statewide Services Supporting Responsive Programming in Libraries
- The Changing Context of Library Services in New York State
- Collaboration Within the Library Community
- Libraries and Community Coalitions
- Changes to Professional Practice
- Advocacy and Policy Information as Support
Section Three: Findings Regarding Management and Operations
- Management of the LSTA Grants Program in New York
- Limited Categories as Barriers to Participation
- Identification of Best Practices and Replication
- Need for Improved Communications
- Barriers to Institutionalization of Grant Funded Innovations
- The Nature of Innovation Funding and Continuation of Support
Section 4: Conclusions and Recommendations
- Evaluability Questions
- Conclusions Evaluability Questions
- Recommendations Evaluability Questions
- Implementation Evaluation Questions
- Conclusions Implementation Evaluation Questions 1 & 2
- Recommendations Implementation Evaluation Questions 1 & 2
- Conclusions Implementation Evaluation Questions 3 & 4
- Recommendations Implementation Evaluation Questions 3 & 4
- Conclusions Implementation Evaluation Question 5
- Recommendations Implementation Evaluation Question 5
- Impact Evaluation Questions
- Conclusions Impact Evaluation Question 1
- Recommendations Impact Evaluation Question 1
- Conclusions Impact Evaluation Question 2
- Recommendations Impact Evaluation Question 2
- Conclusions Impact Evaluation Question 3
- Recommendations Impact Evaluation Question 3
- Conclusions Impact Evaluation Question 4
- Recommendations Impact Evaluation Question 4
- Conclusions Impact Evaluation Question 5
- Recommendations Impact Evaluation Question 5
- Conclusions Impact Evaluation Question 6
- Recommendations Impact Evaluation Question 6
- Conclusions Impact Evaluation Question 7
- Recommendations Impact Evaluation Question 7
- Conclusions Impact Evaluation Question 8
- Recommendations Impact Evaluation Question 8
Appendices
List of Tables
- System Directors Perception of Support by LSTA to Development of High Quality Library Services
- Use of Statewide Services by System Director Type
- Impact of Statewide Services on System Member Libraries
- Number and Percent Library Systems Linked to State, Regional and/or System Catalog by System Type
- Percent Library System Directors Agree to Importance of Records from Each Library Type to NYS System
- Areas of Professional Development Supported by LSTA Grant Funds
- Progress on Development of Tracking System of Public Use of Electronic Library Resources by System Type
- Resolution of Telecommunications Issues by Library System Type
- Project Directors Reported Use of LSTA to Support Program Areas
- Public Library System Directors Reported Use of LSTA to Support Program Areas
- Extent of Participation in Library Collaborations by System Directors
- Linking of Library System Member Libraries to Information Services Outside of the Library Community
- System Directors Report of Need for Professional Development Targeting Changes in Library Practice
- New York State Library Services Aid in System Level Planning
- Percent System and Project Directors Reporting Limited Grant Categories a Barrier to Participation
- Grant Project Directors Report of LSTA Grant Management Paperwork
- Restriction on Use of LSTA Grant Funds to Pay Current Staff as a Barrier to Participation
- Public Library System Directors (PLS) and Project Directors (PD) Reported Use of LSTA to Support Special Program Area by Magnitude of Support
- Areas of Professional Development and Other Services: All Systems by Magnitude of Support by LSTA Funds
Introduction
Evaluation of the Library Services and Technology Act in New York
New York is a large, populous and geographically diverse state, with a land area of 47,225 square miles and a population of 19.0 million. The state has both large urban areas with highly concentrated populations and vast geographic areas with low populations per square mile, all needing diverse library services. In any given year, three in five New Yorkers use a public library. Yet, there are still over one million residents of the State who do not reside in any public library district.
For administrative and library service delivery purposes, New York State is divided into three types of library systems. Seven hundred and fifty public libraries in the State are members of the Public Library Systems, of which there are 23. The public library systems also include twenty-six central and co-central libraries, which are public libraries that provide targeted services throughout their respective public library system areas. Nine Reference and Research Library Resources Systems have academic and special libraries, as well as public library systems and school library systems as members. A total of 4,100 public school libraries and 450 nonpublic school libraries participate in the forty-two School Library Systems in the State.
The transition from Library Services and Construction Act (LSCA) activities to the Library Services and Technology Act (LSTA) activities in 1997 provided the New York State Library and the statewide library community with the opportunity to redefine the federal role in support of library services in the Empire State. In the area of increased access to information and library services for all citizens of the State, the New York State Library has concentrated its attention on developing new ways of providing services in a period in which all public services are being re-examined while addressing the public policy issues around the recognition of the role of libraries in a learning society.
The evaluation reported here measured the LSTA progress to date in helping New York to achieve this increased access. The findings of the study can also be applied to next steps to be considered by the library community and its leaders in New York. This report is divided into four sections. Section One presents a summary of the areas of use of Library Services and Technology Act Funds in New York and a summary of the positive effects of these areas of use is presented. The second section of this report presents a detailed analysis of the states progress in achieving the two objectives of their LSTA Five Year Plan 1987-2002. Because New York uses a significant amount of the LSTA funds to support local grant programs, the evaluation addresses findings regarding management of the LSTA Grant Funded Program in the third section of this report. Finally, Section Four details conclusions and recommendations based on the data collected during the one year of this evaluation study.
Evaluation Design
The evaluation study described in detail in this report is an evaluability[1] study of the New York State Library Services and Technology Act (LSTA) Program in New York State. The study was designed to meet the requirements of the Library Services and Technology Act (P.L. 104-208) and the funding agency, the Institute of Museum and Library Services (IMLS). The LSTA Program includes both Grants and Statewide Services components and the evaluation of both these component types are covered in this report. See Appendix I of this report for a copy of the Evaluation Plan which was developed with the staff from the New York State Library and approved by the LSTA Advisory Councils Evaluation Committee.
[1Evaluability is a type of evaluation which is done to determine if more evaluation is necessary by determining if the program is at a stage where a more extensive evaluation would contribute to the program's effectiveness.]
The evaluators undertook six evaluation activities during the one year evaluation reported here. First, the New York State Library staff and the evaluator met and developed a Program Logic Model for the LSTA program in New York. As part of this process, the evaluators presented the model to the Evaluation Committee of the New York State LSTA Advisory Council. The evaluators then used this logic model in combination with group and one-on-one interviews to develop a set of surveys for ascertaining information from key informants in the State. The two Statewide Surveys which were developed were reviewed and commented on by staff at the New York State Library and by members of the LSTA Advisory Councils Evaluation Committee. The final surveys were sent to five stakeholder groups in the State. A Survey of Library System Directors was sent to all directors of the three library system types in the State and to all directors of central and co-central libraries. A separate survey was developed for LSTA Grant Project Directors, with some questions in common with the System Directors survey, and other questions specifically about their experience directing an LSTA Grant funded program. Response rates for these surveys are presented below.
| Stakeholder Type | Number Sent | Number Returned | Percent Returned |
|---|---|---|---|
| Reference and Research Library Resource System | 9 | 9 | 100% |
| Public Library System | 23 | 16 | 69.6% |
| School Library System | 43 | 19 | 44.2% |
| Central and Co-Central Library Directors | 22 | 14 | 63.6% |
| LSTA Grant Project Directors | 101 | 61 | 60.4% |
In addition, the evaluators interviewed key informants at the New York State Library as well as designing a web-based log for State Library staff to record their work related activities on identified random days during the month of May 2001. Following the analysis of the survey data and a discussion with the LSTA Advisory Councils Evaluation Committee regarding the initial results of the surveys, the evaluators held a series of six focus forums with a total of seventy-six library system directors and the directors of system member libraries at four locations around the State. In addition a representative number of LSTA Grant Project Directors attended and participated in the discussion at each forum. See Appendix II of this report for examples of all evaluation instruments.
Section One -- Library Services and Technology Act Funds in New York: Use and Effect
Uses of the Funds
The New York State Library has used Library Services and Technology Act support in combination with other State, federal and private funding to move the State towards the goal of ensuring that all New Yorkers have access to library resources and services that advance and enhance their lives as workers, citizens, family members and lifelong learners. The electronic doorway library is a metaphor adopted in 1988[2] to represent the fundamental changes in the delivery of library services which where then taking place in library service delivery in New York. In this concept, all library service delivery entities in the state have the potential to simultaneously operate both as portals into the automated system and as resource components of the network. The network thus constructed offers a state such as New York the ability to maximize the use of existing resources while doing so with all the value-added of seamless interoperability.
[2 See Appendix III of this report for a brief overview of statewide automation and the electronic doorway library in New York.]
The effects of the changes in the delivery of library services have been noted throughout this evaluation. When asked to reflect on the changes in library services in the period from 1997 to the present, participants in the focus forums held as part of this evaluation indicated that the changes to their practice and the services provided by their system and system member libraries in the State have been both far reaching and rapid. In general, participants identified the following four things as the largest changes in library service delivery in the past five years:
- A greater focus on technology;
- The need to address difficulties resulting from increased staff turnover, and difficulty finding and hiring qualified staff;
- The need to learn how to attract and serve new populations; and,
- Adjusting to the increasing concern and anxiety about the future, particularly the long-term relationship between library services and the Internet.
Library Services and Technology Act (LSTA) funds are used to facilitate and support innovation and change in three ways in New York State. First, part of the funds are used to support two types of local grants programs: Information Access through Technology Grants and Information Empowerment Through Special Services Grants. Second, part of the funds are used to support the development of what began as the Electronic Doorway Library, evolved into EmpireLink, and has now developed into the New York Online Virtual Library (NOVEL) in the State. And finally, funds are used to support other Statewide Services provided by the New York State Library. All three uses of the funds have facilitated and supported the two objectives of the New York State Library Services and Technology Act Five Year Plan October 1, 1997 September 30, 2002. These two objectives are:
- Assist libraries in New York State, including the NY State Library, to enable librarians and other staff to provide the highest possible level of electronic doorway library services; and,
- Emphasize special library services which contribute to improved access to information and library services for all the people of the State.
Reported Positive Effects of Participation in the LSTA Grant Program
It is important to note that a majority of those responding to the surveys sent out as part of this evaluation, and all of the participants in the focus forums held as part of this evaluation, were from systems that had received LSTA grant support during the period between 1997 and the dates of this evaluation. Most of the system directors responding to the statewide Library System Directors Survey reported that their system had applied for and received an LSTA Grant. That is, all of the nine reference and research library resource system directors (100%), ninety-four percent (15 of the 16 directors responding to the survey) of the public library system directors, and seventy-nine percent (15 of the 19 directors responding to the survey) of the school library system directors reported having received at least one LSTA grant.
Grant project directors were asked if their system had learned a great deal through the LSTA supported opportunity to innovate. Seventy-five percent agreed that they had.
The statewide System Directors Survey then asked the system directors that reported having received LSTA Grant support to consider the support that these grant monies had given to the ongoing development of high quality library services to their member libraries and thereby to the clients of those member libraries. Respondents were offered three choices and asked to choose all that applied to their experience with LSTA Grant support in their service delivery area.
Focus Forum participants told the evaluators that the two most important things that they had done with LSTA Grant funds had been to provide technology training to member libraries and to expand services using outreach projects and new initiatives which involve member libraries and foster collaborations. These same participants indicated that among the most important things done by the State Library using LSTA funds has been the development of EmpireLink.
Reported Positive Effects of EmpireLink
Focus forum participants were strongly positive about the impacts of the development of EmpireLink/NOVEL on their systems. As one participant said, "EmpireLink is one of the best things to come out of LSTA". Participants reported their opinion that the equal access afforded to any library of a free full-text database such as EmpireLink actually functions as a quality equalizer which helps all libraries meet a minimum standard of service. Participants were clear that the availability of the database is positive for large systems for different reasons than for small, rural or special libraries, however. In the case of large systems and libraries, focus forum participants noted, the money saved by using a state supported database allows them to allocate their limited resources to support other services and programs. Small special libraries and rural libraries represented at the focus forums indicated that without the state supported database they would not be able to afford any database at all.
Both the LSTA Grants Program and the development of the state virtual library network in the EmpireLink and more recently NOVEL strategies, were recognized by most key informants to this evaluation as important to the future quality of the library services in New York State.
Less obvious, but considered by the New York State Library as no less important for their value and contribution to the development of the library systems in the State, are the other statewide services supported by LSTA. Measurement of the effect of these services presented a challenge to the evaluation.
Reported Positive Effects of Other Statewide Services
Systemic change is complex for many reasons. One of the chief elements of this complexity is the difficulty experienced in communicating about change within the changing system. This should not be surprising. Systems operate, by definition, at a number of levels so that the same system is viewed by people working within it from any of a number of vantage points. These differing vantage points offer unique and often very different perspectives on the system and the progress of its change process. In this evaluation, the differing perspectives operating within LSTA funded activities were apparent in all sets of data collected. The evaluators noted a difference in the way in which key informants conceptualized the role which some of the services and initiatives implemented by the State Library were viewed. This was obviated by the pattern of response on a number of questions in the statewide System Director Survey. In order to clarify these response patterns, clarifying questions were subsequently addressed in the Focus Forums.
On the statewide surveys, library system directors and central and co-central library directors were asked two sets of questions related to the Statewide Services provided by the New York State Library. First, they were asked to indicate which of a set of Statewide Services their system or library had used. Reponses to these questions are presented on Table 2, with percent of all survey respondents choosing each option reported.
As can be seen from the information on Table 2, it is clear that library systems as reported by their directors and central and co-central library directors do use the Statewide Services provided by the New York State Library. What was unclear from the second set of questions on the survey was how these directors saw these Statewide Services fitting into the big picture of library service development in their region or service delivery area.
In the second set of questions on the statewide surveys, system directors were asked to indicate whether Statewide Services had contributed to the libraries in their systems in any of the following ways with the indicated results presented here as Table 3. What the two tables seem to indicate is a high level of use of Statewide Services but with no perceived impact of that use on the ability of libraries in these systems to develop and implement change sensitive library services.
These two sets of responses on the Statewide Directors Survey caused the evaluators to include questions about the use and value of Statewide Services in the focus forums. The answer was quite simple. The first set of questions on the survey regarding the use of Statewide Services asked the directors about their use of these services, as directors of library systems. The second set of questions asked about the impact of Statewide Services on their member libraries. Focus forum participants reported that from the perspective of the library system directors the impact of these Statewide Services is on the ability of their respective systems (Reference and Research Library Resource Systems, Public Library Systems or School Library Systems) to help their system member libraries to accommodate the dynamic needs for library services. And, focus forum participants noted, it is their use of Statewide Services that helps the system directors to support strategies which will stimulate change and transform services in their regions. The Statewide Services were seen as mediating the ability of the systems to support their member libraries and not as directly affecting member libraries as the survey question asked.
This interpretation of the data is supported by the Activity Logs completed by members of the New York State Library staff during May 2001, and by the information collected by the evaluators during interviews of the New York State Library staff in April 2001. 67% of the log entries reported interaction with a system director or library system staff person. The effect of the Statewide Services is to support the work of the library systems in New York as they work closely with their member libraries to meet the needs of the residents of the state.
Section Two -- Achieving the Objectives in the New York State LSTA Five Year Plan
This section of the report combines information gathered through individual interviews, statewide surveys, logs of staff activity and focus forums of key informants to present evaluation findings regarding the achievement of the two objectives stated in the New York State LSTA Five Year Plan October 1, 1997 September 30, 2002. These two objectives are:
- Assist libraries in New York State, including the NY State Library, to enable librarians and other staff to provide the highest possible level of electronic doorway library services; and,
- Emphasize special library services which contribute to improved access to information and library services for all the people of the State.
Objective 1: Developing Electronic Doorway Library Services
There is no question that the introduction of technology into the library service delivery system has meant immense changes in the demands of the citizens on libraries and that these changes have required changes in the professional work of librarians. As is the case in all other states, New York has had to work hard to meet the challenges which a rapidly changing environment for service delivery presents. These challenges include maintaining high quality service delivery while attending to the technology and human resource infrastructure development which are required. Of primary concern has been the efficient combination and use of multiple resources to support simultaneously ongoing service delivery and the management of change, such that the clients of the system do not notice any lapses in service magnitude or quality, while experiencing the timely introduction and delivery of emerging services. The New York State Library Services and Technology Act Five Year Plan October 1, 1997 September 30, 2002 described the states intended use of the LSTA funds to support a Technology Grants program in New York State and some Statewide Services provided by the staff of the New York State Library, both of which have provided important support to these efforts.
LSTA Technology Grants Program in New York
The intent of the LSTA Technology Grants Program is to allow all library systems to help all of their member libraries or branches to become contributing member electronic doorway libraries. To do so, the plan divides the grants awarded under this section of the Plan into two categories: Electronic Content and Training for Technology. This section of the report will discuss the findings of the evaluation relevant to these two grant categories.
LSTA Grants to Support Electronic Content
The purpose of this category is to enable libraries to provide library resources in electronic format for local, regional, statewide and global access, and to create value added information products which package information available in libraries, or link the user to other electronic sources. This grant category targets its support towards the achievement of a statewide automation of libraries in New York with seamless interoperability that utilizes both collections resident in libraries across the State and resources external to the present New York State system.
The concept of a statewide library network, which is realized through the development of an infrastructure using the mechanism of the electronic doorway library, has not changed over the period from 1987 through 2001. However, the nature of the technology which will make that system a reality in New York has changed considerably. See Appendix III of this report for a summary of the development of this New York State Library initiative.
Debates which were seen as important in the early stages of this development have ceased to be so, largely due to developments in the capabilities of technology to address the issues. The evaluators did hear echoes of these earlier issues throughout the data collection activities associated with this study from all stakeholder groups. This information was not ignored; however, it is not reported in the main body of the evaluation and is addressed in the Conclusions and Recommendations section of this report.
The rapid development of cross application interfaces and the most recent advent of widely available web-based system management and data sharing applications and software platforms have rendered some of the earlier debate moot. The New York State Library and library system directors in the State have remained current in this changing landscape so that the most recent planned development of NOVEL, the New York Online Virtual Electronic Library, which will connect libraries to databases, shared catalogs and other electronic information, subsumes previous work into the final development of statewide library technology activities which will achieve the goal of seamless interoperability among libraries across the state.
Library Services and Technology Act funds have been used by New York State to further both the digitization of bibliographic and of unique and historical records and to encourage broad access to information. The number of New Yorkers with electronic access to information resources has grown at a strong pace over the past five years. On the survey distributed as part of this evaluation, library system directors reported the following average percent of libraries in their respective systems as having as much of their bibliographic records as necessary converted.
| Reference and research library resource Systems | 73.5% |
| Public Library Systems | 78.1% |
| Central and Co-Central Libraries | 98.2% |
| School Library Systems | 62.1% |
Library system directors reported that LSTA grants have been used to partially fund electronic conversion of bibliographic records. Discussion of the conversion of bibliographic records during focus forums indicated that these conversions are considered important and will continue until completed.
The second stage in this development of a statewide library technology is to link these digitized bibliographic records through system and regional catalogs to create a statewide virtual library.
Respondents to the LSTA Library System Director Survey in June 2001 indicated that the converted records in their system are linked to regional catalogs or system catalogs. These converted records in both reference and research library resource systems and school library systems are more often linked to regional catalogs. In public library systems, the converted records are more often linked to system catalogs.
One method for gauging the progress towards this type of outcome is to elicit information from respondents on future need. The Statewide Survey asked system directors to indicate their expected need for support during the next five years in a number of key areas. Shared electronic catalog development, an element of the New York Online Virtual Electronic Library, was chosen by fifty-eight percent of the school library system directors as one of four most critical areas for future support. However, only twenty-five percent of the public library system directors and none of the reference and research library resource system directors indicated a future need for support in this area. In combination with other information from this evaluation, it would seem that shared electronic catalog development has been completed in many areas of the State and within two of the three library system types.
Throughout the period of this evaluation, the evaluators found evidence of the rapid nature of the changes taking place in the area of library service delivery. One example of the manner in which the change in technology during the five year period of the present LSTA Five Year Plan has impacted the use of LSTA grant funds by the library community in New York is in the need to expand the use of commercial databases. Focus forum participants explained to the evaluators that as the development of the shared electronic catalog progressed, they noted an increased demand for information which is contained in commercially available databases. As information use increased, so too the need for better, more easily searchable and continuously updated sources of that information increased.
In response to a question on the Statewide Surveys which asked if the restriction on use of LSTA funds to purchase commercially available databases is a barrier to the use of LSTA grant funds by their system, the following percent of responding system directors indicated that it is a barrier.
| Reference and Research Library Resource Systems | 100% |
| Public Library Systems | 50% |
| School Library Systems | 42% |
The State Library, in consultation with the LSTA Advisory Council, when beginning to develop EmpireLink in 1998, decided not to fund the purchase of commercial databases through LSTA grants. The rationale for this decision was reported to the evaluators to be that a coordinated, state level effort would be more cost effective than supporting individual, possibly duplicative, local purchases.
Future Trends and Perceived Needs: Electronic Content
When asked which areas of library development will need the most support during the next five years, access to commercial databases was chosen most frequently by each of the three system type directors. Sixty-seven percent of the reference and research library resource system directors, sixty-nine percent of the public library system directors and ninety-five percent of the school library system directors indicated that this area will need the greatest amount of support during the next five years.
Library systems that have completed conversion are now turning their attention to other tasks related to the electronic medium, for example, the purchase of electronic full-text journals and reference materials, the digitization of historical documents and exploring the issues surrounding e-books. Indeed, four of the reference and research library resource system directors and six of the public library system directors reported already using LSTA grant funds to digitize local history collections for inclusion in the statewide system. Four (forty-four percent) of the reference and research library resource system directors chose digitizing of full text resources when asked for areas in which they will need support over the next five years, making this the second most frequently indicated option. The school library system directors chose digitizing of full text resources as their third choice of area in need of support during that timeframe. In addition, thirty-three percent of the reference and research library resource system directors, fifty percent of the public library system directors, and fourteen percent of the school library system directors reported using LSTA grant funds in their efforts to develop new technologies for distance learning.
The evaluation was also interested in ascertaining from system directors their judgment of the importance of including the collections resident in each of the three types of library system in the state into the State electronic information network that will be the State virtual library. The survey presented library system directors with two types of materials, bibliographic and unique and historic documents. They were asked to consider a set of six statements which described the material in each of the three types of library in which they might be held. The statements read as follows: Unique and historic documents in academic, research and special libraries, Unique and historic documents in public libraries, Unique and historic documents in school libraries; Academic, research and special libraries bibliographic records, Bibliographic records from public libraries, and, Bibliographic records from school libraries.
Directors were then asked to rate how important the inclusion of each of these six alternatives would be in continuing to develop a system to deliver high quality information access to all New Yorkers using a four point Likert-type scale ranging from Strongly Agree to Strongly Disagree. As can be seen from the information presented on Table 5, system directors indicated a slightly higher rate of importance to the unique and historic documents and the bibliographic records in the reference and research library resource systems and the public library systems than they did for the collections in school library systems.
When asked about this at the focus forums participants pointed out that the it depends caveat applies here. Some libraries have valuable collections of unique documents which will have to be preserved, while others do not, and in the opinion of the focus forum participants the importance of the collection is not necessarily connected to the type of library in which the collection is housed. There was a general consensus that school library systems may not have as much to offer that is unique because of the nature of their collections and their mission, which is to support curriculum delivery in schools and school districts. To the focus forum participants the question is more who will decide what is worth including, rather than where the sources of the bibliographic records or the unique and historic documents are found.
LSTA Grants for Training in Technology
It is not enough to build a new technology infrastructure in the state in order to deliver high quality electronic information access to all New Yorkers. The professional staff of both the library systems and the library systems members have to receive targeted professional development in the use and application of technology to their professional practice. For libraries to provide excellent library services to their users in a rapidly changing technological environment, the library staff must have access to an ongoing training program in all aspects of technology.
All of the reference and research library resource directors and seventy-five percent of the public library system directors reported using LSTA Technology Grants program funds to support strategies that stimulate change in professional practice among librarians in their systems. The rate among school library system directors is smaller, thirty-seven percent. This is probably due to the fact that school library systems have only recently been allowed to apply for LSTA funds and have been using this source of support for extensive retrospective conversion of their bibliographic records.
Library Services and Technology Act Technology Grants have contributed to the training of system library staff and system member library staff in order to support and facilitate changes in professional practice among librarians across the state. On the statewide surveys, library system directors were asked to report on areas of professional development that they have used LSTA Grant funds to support over the years between October 1997 and the present. In addition, they were asked in which of those same professional development areas they believe their system still has a need for professional development. Shaded fields on Table 6 indicate a higher future need than past use.
Future Trends and Perceived Needs: Professional Development
As can be seen from Table 6, there is a predicted need for professional development in all areas, with high levels of increase in future need compared to past use of professional development support from LSTA Grants. Interviewees prior to the development of the surveys, and questions to focus forum participants following initial analysis of the survey data both indicated that there are two reasons for this increase in the need for technology related professional development. The first is that the change in library services due to the growing technological infrastructure has increased the need for system member librarians to be able to use technology applications and to help the public to use them. The second reason for this need is the high rate of turnover in library personnel, requiring basic instruction in these critical areas to be repeated regularly. These roles for professional development parallel two of the three areas which the research literature on professional development identifies as part of a good professional development plan. Those three areas are:
- The use of professional development as a mechanism for introducing innovations and innovative practices.
- The use of professional development to upgrade existing professional skills and abilities in order to ensure basic standards of practice.
- The use of professional development in order to provide the skills necessary to become active participants in systemic change.
Of note here is the fact that during focus forum discussions at three of the sites the provision of more support for the work of change and improvement as a future need included the need for professional development in how to participate in systemic change processes.
Future Trends and Perceived Needs in Other Areas
One of the questions regarding the future development of library services in the State which arose during the pre-survey interviews was that of tracking public use of electronic library resources. In the opinion of many of the interviewees, this has become an issue for libraries because the traditional methods of measuring the magnitude of library services to their constituencies, i.e., tracking circulation of resident collections, logging of inter-library loan requests, counting of reference searches, and similar measures, no longer capture the true magnitude and extent of modern library services. Quality and magnitude of services delivered by any level of the library services system in the State have to be measured in ways which are considered fair, rigorous and representative.
Failure to attend to the issues surrounding the measurement of the new and emerging profile of library service delivery in each of the three library system types and in the member libraries of each of these systems would result in a serious undermining of the system as a whole. This is so for two reasons. First, key informants to this evaluation regularly pointed out the local basis of support, both fiscal and ideological, for library services. Failure to report on quality and magnitude of services in ways which make sense to local constituencies and their policy structures, they reported, would have serious consequences. Second, the operation of services within a complex and intertwined system of systems, as is the case in New York, makes services difficult to report on in ways which make sense to the state constituencies and their policy structures. The first step in the development of an acceptable accountability system for use in this new and emerging context was reported to the evaluators to be the development of methods for tracking public use of electronic library resources. The Statewide Library System Directors Survey, therefore, included a question about present progress and perceived future need for technical support in this area. Results of that question are presented with the rate of response reported in the columns of Table 7 below. What can be seen in this table is that the majority of the system directors believe that this is an important area for future attention and that they need help in the development of these tracking methods.
The resolution of telecommunications issues was the subject of a separate question on the Statewide Survey, as well as being included in the list of possible library development areas for attention in the next five years. This question had arisen during the survey development interviews conducted by the evaluators so that the decision was made to elicit specific information regarding telecommunications issues for each of the three library system types in New York on the Statewide Surveys. Respondents were asked to indicate if the resolution of telecommunications issues is very, somewhat or not at all important to the future development of technology in their context. Of those responding to this question, eight of the nine reference and research library system directors (89%); fifteen of the fifteen public library system directors (100%); and, seventeen of the eighteen school library system directors (94%) indicated that they consider the resolution of telecommunications issues important to the future technology development in their systems. Focus forum participants noted two things here. First, that it is this area in which they need the most technical assistance, and it is this area in which expert help is unlikely to be resident at the New York State Library, and where consultant experts are extremely expensive. Second, that the details of the telecommunications issues differ according to the stage in technology development of the library system in question, and so an approach to this concern will have to be flexible.
The increasing use of web-based applications by libraries was noted by focus forum participants as the reason why telecommunication has become an issue. Indeed, one central library director interviewed during the pre-survey development stage of this evaluation reported that the cost of connectivity in his system was immense, thus creating a real barrier to service delivery to his client population.
The evaluation treated the issue of telecommunications development as different from issues of telecommunications access. This is so because it was made clear to the evaluators during the pre-survey interviews that questions of access are not as important in most library systems as are the more complex issues of telecommunications technology now facing the interviewees. The Statewide Survey did, however, include a question about improving telecommunications access. When responding to the question regarding the library development areas which will require the greatest attention during the next five years, eleven school library system directors (58%) chose improvement of telecommunications access of the nineteen directors completing the survey. This was not an option chosen by any of the reference and research resource library system or public library system directors.
Statewide Services Supporting Objective One
The New York State Library has three primary functions in their support of the development of the technology and human resources infrastructure needed to provide high quality electronic information access to all New Yorkers. First, the New York State Library provides leadership and guidance for the planning and coordinated development of library services using a networked electronic environment for the people of the State. Second, the New York State Library has supported the ongoing creation of NOVEL (New York Online Virtual Electronic Library) to make electronic information freely available to all library systems and libraries in the state. And third, the New York State Library manages the LSTA Grants Program for the state. This third function is described in Management of the LSTA Grants Program in New York beginning on page 48 of this report.
Leadership in Planning and Development
The use of results-based planning in order to effect systemic change has been the topic of much discussion in New York. In the area of library system development, the use of planning at the library, library system and state system levels has received much attention. Indeed, system directors were asked on the Statewide Survey through which means the State Library contributes to planning in their system. Thirty-three percent of the Reference and research library resource directors; sixty-three percent of the Public Library system directors; seventy-nine percent of the school library system directors; and, fifty percent of the central library directors indicated it was done by coordinating strategic planning at a statewide level, e.g., Regents Commission on Library Services, NOVEL Planning Team, Third Statewide Automation Plan. When reporting which Statewide Services they access on behalf of their systems, "Information for planning and advocacy" was indicated by:
66% of Reference and Research Library Resource System Directors
63% of the Public Library System Directors
95% of the School Library System Directors
36% of the Central Library Directors
Development of NOVEL
The development of EmpireLink and the plans to expand it into NOVEL met with a general positive response by both survey respondents and participants at the focus forums held as part of this evaluation. In addition, when asked on the statewide survey of library system directors to indicate the areas in which they contact the New York State Library for help, Obtaining cost-free access to full-text electronic databases was cited by:
78% of Reference and Research Library Resource System Directors
88% of Public Library System Directors
90% of School Library System Directors
21% of Central Library Directors
The reported use of electronic interlibrary loan on the same survey was reported at 78% for reference and resource library system directors, 13% for public library system directors, 79% for school library system directors, and 43% for central library directors. Information from the New York State Library based on sample data from the State Library Inter-Library Loan Unit indicates that all 23 of the public library systems used the State Library for Inter-Library Loan in 2001.
The following data looks at the response pattern from the central and co-central library directors only. Central and co-central libraries are one type of public library and reflect public library use of interlibrary loan from the New York State Library.
| 98% of own bibliographic records digitized | 36 % of the records of these central libraries are linked to the regional catalog. | 79% of these records are linked to the system catalog. | 43% of these directors say they use State electronic interlibrary loan. |
The low percent of interlibrary loan contact for public library systems reported on the Statewide Directors Survey was addressed at the focus forums at which time the evaluators were told that this is due to the successful implementation of the use of full-text online electronic databases and links with system and regional catalogs. In addition, it should be noted that the New York State Library has encouraged the development of point-to-point interlibrary loan among all library system types in the State.
The focus forum participants also noted that the use of the State Library electronic interlibrary loan system entailed heavy fines for non-returned materials, a concern of public libraries especially since their customers may not return the materials for a number of reasons, thus causing the local library to entail the expense. Materials borrowed through the regional or system catalog, for example, would not entail this level of potential expense. In addition, it may be that public library system directors do not use the States electronic interlibrary loan, but that their member libraries do. The 43% of reported use by central library directors is lower than that of any of the other type of system directors, 78% for reference and research resource systems and 79% for school library systems, but is higher than the 13% reported by the public library system directors.
Objective 2: Encouraging Information Empowerment Through Special Services to Increase Access
Libraries empower people so that library services must be dynamic and responsive to the changing needs of people of all ages and all abilities. A diverse state such as New York, which has both large numbers of people to be served in densely populated urban areas, as well as large geographic areas with smaller numbers of people, requires a complex network of library services to assist all citizens of the State to locate and use information and services that will help to advance their lives in many ways. Some focus forum participants voiced the opinion that "too much money has been spent on technology, meaning that other special services and programs supported by LSTA are also important.
This part of the report presents information gathered through the evaluation on the achievement of the second objective of the New York State LSTA Five Year Plan. As with the first objective, the activities here are divided between those supported through a local grant program and Statewide Services provided by the New York State Library.
LSTA Information Empowerment Through Special Services Grants Program in New York
Grants in this area emphasize the role that public libraries and public library systems play in promoting adult and family literacy and economic opportunity to help all New Yorkers achieve more independent lives. Programs providing adult and family literacy services, and economic opportunity services are included in the grant program supporting this objective. For that reason, the purpose of this program is to enable public libraries and public library systems to provide programs and services that promote the improvement of literacy skills for people of all ages; to assist individuals to develop job-readiness skills; and, to help small businesses to find the business information resources they need to prosper.
LSTA Grant Project Directors were asked if their system had used LSTA funds to enhance or expand services for
individuals. Forty-two of the sixty-one respondents (69%) indicated that their system had used LSTA funds to enhance
or expand these services. The grant project directors were then asked to indicate specific program areas where
LSTA grant support had been used. Respondents were asked to indicate all of the areas in which their system had
used LSTA funds in services for individuals, with the result reported here as Table 9.
Public Library System directors were asked the same question on their statewide survey. Fourteen of the sixteen survey respondents, or 88%, indicated that they had used LSTA funds to enhance or expand services for individuals in these categories.
As can be seen from the data reported in Tables 12 and 13, the use of LSTA grant funds to support special services is widespread among those that answered the surveys sent out as part of this evaluation, as reported by both grant project directors and public library system directors.
Choices of library development areas which will need support over the next five years differed for each type of library system. Three of the reference and research library resource system directors indicated adult literacy services, as well as services to new populations, making both of those areas the third most chosen by this group of directors. Eleven of the public library system directors indicated family literacy services as among those that will be the most important library development areas over the next five years. These directors also chose services to new populations as their third most frequently indicated area which will need support.
Services to New Populations: Present and Emerging Needs
The seventy-six focus forum participants were clear in their reporting that their library systems and system member libraries provide extensive services to new populations. Public library system members and central libraries, especially those in large cities (New York, Buffalo, Rochester, Syracuse and Yonkers) and small cities (e.g., Utica, Binghamton, Troy, Schenectady), typically provide both adult literacy and family literacy programs. Adult Basic Educators have long considered libraries as the ideal place to meet with adult learners who will not be embarrassed if seen entering a public place that provides voluntary services to adults. This legacy has meant that libraries are increasingly perceived by other community agencies as a neutral environment in which information for job seekers, and training in work-readiness skills can be delivered without undue stress to the information seekers.
Some of the new populations referred to by the reference and research library resource system directors and the public library system directors on the Statewide Survey were the same: new Americans and non-Native English speakers were noted in general, along with underserved areas and US Army personnel. Specific reference was also made to Hispanic and Chinese ESL populations.
All data collection activities carried out under this evaluation yielded information regarding the expansion of library services to include more new populations of library service users. In general, each library system type defined new populations somewhat differently. Reference and research library resource systems define new populations as new types of special libraries to whom they can offer services, and as existing systems of the other two types in their region to whom they can offer regional services. Public library system directors define new populations as groups of the public who have not traditionally used library services. These new groups can be delineated by geographic location (thus requiring the expansion of existing services to under-served locations) or by population characteristic (thus requiring the expansion of services to include programs and services targeted to specific client needs, such as English as a Second Language instruction, job finding and/or career changing resource development, etc.). School library system directors identified new populations as special needs clients, meaning students who are mainstreamed from Special Education programs.
Open-ended responses on the Statewide Survey indicated that some of the library system directors believe that outreach to new populations can be enhanced through networking and collaboration with other community agencies.
Statewide Services Supporting Responsive Programming in Libraries
The New York State Library has a key role in the planning and provision of library programming that is responsive to the needs of all New Yorkers. This role has developed in the period covered by the present LSTA Five Year Plan, and has been clarified through the activities of the evaluation reported here. The State Library has three functions in this area of the development of a system which is easily accessed by all New Yorkers. First, the New York State Library provides leadership and support to address the changing context of library services in the State. Supporting this is a second role whereby the Division of Library Development provides information to library advocates and ensures that the library community is aware of legislation pertinent to library services in the state. Third, specific services to enhance access by special populations, for example for the visually impaired, are provided by the New York State Library through State funds. The New York State Library uses LSTA funds to support the first two of these three functions.
The following discussion is divided into a discussion of specific issue areas within the changing context of library services in New York and a discussion of advocacy and policy information dissemination as a form of change management and support. The broader of the two topic areas, the changing context of library services in New York, contains three related issue areas: collaboration within the library community; collaboration outside the library community; and, changes in professional practice among librarians in the State.
The Changing Context of Library Services in New York State
The New York State Library managers of the LSTA program in the State have been concerned with external outcomes for the program. These impacts are such things as changes in the structure of information resources in the State, including changes in the nature and characteristics of library services. However, the innovation and change provided in part by LSTA support has also produced internal outcomes, some of which could be considered as more important than the external impacts. These internal impacts operate on the personnel within library services to change professional practice and the organizations through which those services are delivered to change organizational culture, climate and context. They are important to include when reporting on the progress of and planning for the future of library system development in the State.
When asked to discuss this topic, focus forum participants referred to libraries being responsive to the needs of their community. Because of the differences in the roles which the three different system types operating in New York State see for themselves in the context of service, each defines their community differently. School library system directors and members at the forums noted that their role is in the support of curriculum delivery. They see their community as the school community and not the general population. The public library system members represented at the forums seemed to see their community as the community in which each library is situated. The public library systems considered their member libraries as their community. The reference and research resource libraries have a more expansive definition of their community. They see their member academic and special libraries as one community to which they provide services for a specific purpose. In addition, the reference and research library resource system requirements mean that these systems also work together with other system types in their region. Therefore, a second community through which this type of library system functions is the general library service delivery community in their region. Finally, both the public library system directors and the reference and research library resource directors who attended the focus forums indicated that they see themselves as a community of system directors, thus forming a secondary structure below the State Library focused on the management and improvement of library services in the state.
Focus forum participants saw collaboration within and across their service delivery communities as important to their role and mission. In addition, they indicated that they see collaboration as a powerful tool in bringing libraries to the table at all levels of policymaking because it functions as a mechanism for situating libraries in the human service delivery system. According to the statewide surveys, all three types of system directors also see collaboration as a way to deliver better quality services at cost efficient rates to their member libraries.
Collaboration Within the Library Community
Library systems have been encouraged to enter into coalitions within the library community using LSTA Grant support. Respondents to the Statewide Survey indicated their systems levels of participation in collaborations within the library community. As can be seen by the following table, library systems report high levels of collaboration both in and outside of their region. Collaboration reported as both indicates collaboration with the indicated system type both in and outside of their region. The table is read from the left column across. Same library system types have been shaded for ease of reading.
Table 11 indicates that there is a great deal of collaboration in New York State both within library system types and across library system types. In addition, collaboration is not confined to geographic regions. All of the respondents indicated that they collaborate with same type library systems both in their own region and outside of their region, that is, nine of the nine reference and research library resource system directors, sixteen of the sixteen public Library system directors and nineteen of the nineteen school library system directors. In addition, all of the reference and research resource library system directors report working with public library systems and school library systems both in and outside of their regions. Fourteen of the sixteen (88%) of the public library system directors report collaborating with school library systems, seven in their region only and seven both in and outside of their region. Only two of the school library system directors did not indicate that they collaborate with reference and research resource library systems, and only one indicated that they do not collaborate with any public library systems.
Libraries and Community Coalitions
Findings from this evaluation indicate that libraries have become members of community coalitions in order to play an important role in the provision of seamless service delivery at the community level in New York State. Increasingly, library system directors told the evaluators, this participation in the development and provision of integrated human services in their communities has meant that libraries and library systems have modified their own role in the community. Libraries have discovered that they are effective partners in this system-level integration of services for a number of reasons, not least of which because their role is not colored by an agenda which includes traditional agency level territorial issues. Because of the differences in the roles which the three different system types see for themselves in the context of services, the defining characteristics of collaboration differs from system type to system type.
Directors of all three types of library system directors do agree that collaboration is a powerful tool in bringing libraries to the table at all levels of policymaking. Some of the directors of each system type reported that they participate in collaborations outside of the library community. Five of the nine reference and research library resource system directors (56%), twelve of the sixteen public library system directors (75%) and seven of the nineteen school library system directors (37%) completing the Statewide Survey indicated that they participate in collaborations with community agencies outside of the library community. The majority of these respondents said that they participate in these collaborations because networking is important and that it offers an opportunity to further the librarys mission.
In addition, system directors were asked if they believed that it was important to link libraries in their system to educational, social or information services outside of the library community. Their responses also indicated the extent to which these links have already been constructed. As can be seen from the following table, a majority of both the reference and research library resource system directors and the public library system directors indicated that this is an important thing to do. A lower proportion of school library system directors answered that this is important for them to attend to. Those school library system directors that did indicate that it is important to link their system to other service delivery agency systems noted the career planning and job search area and higher education information systems as of primary concern.
Libraries have the potential to be influential change agents in our society. There are a number of reasons for this. First, libraries already exist in most American communities and contexts so they have a presence. Second, although experienced in the management of information storage and retrieval, librarians have a history of value neutral association with the collections that form the core of their professional practice. Finally, libraries are institutions to which entrance and entitlement are not controlled by measures of ability or achievement. Public libraries are open to the public, and the work on a statewide network exemplifies this fundamental belief in the free distribution of information regardless of its geographic or temporal situation.
There are also a number of attributes that might prevent libraries from functioning as influential change agents. One is that many graduate schools of library and information science focus heavily today on technology and information management, and do not include the management of public services and community outreach in the pre-service curriculum. A second is that becoming proactively involved with services that have a change in participants values and beliefs (as in Family Literacy programs) as part of their purpose presents a crisis of conscience among many librarians due to the long history of value neutrality inherent in their professional practice. And finally, libraries traditionally serve individuals rather than targeted populations or groups within the community. Focus forum participants addressed these concerns when they noted that they understand that change has to happen, but that they need help in finding a comfort zone around these changes so that they can continue to deliver high quality service through their professional practice.
Changes to Professional Practice
Both in the pre-survey interviews and the focus forums carried out as part of this evaluation, library system directors and the directors of member libraries indicated that there had been a significant change in services for users over the five year period covered by the present LSTA Five Year Plan. These services are delivered more efficiently in many instances, as one librarian in a university told the evaluators during a pre-survey interview: "My clients can do a lot of their own searching now, which frees me up to spend more time doing the more complex tasks of my work. The outcome is that my clients have shorter waiting periods for the work that I have to do for them."
The professional practice of librarians has changed considerably over the past decade. Participants in the focus forums carried out as part of this evaluation indicated that these changes in practice are not always comfortable, and that they " need to gain a comfort level with the new changes". Some work has already been done in the area of providing the support in the development of this comfort level with the new changes. This has been done using professional development targeted specifically at the issues of changes in library practice and what that means to professional librarians.
The Library Services and Technology Act grants program in New York has been used extensively in the area of professional development during the last five years. Reports on the use of professional development to increase technology based skills among librarians in the state were provided elsewhere in this report. One question on the Statewide Survey addressed the use of professional development specifically to address issues of changes in library practice. The answers to that question established a high magnitude of need for this type of professional development (reported below as % some need) and whether directors felt that the call for this type of professional development is high, moderate or low.
The Statewide Survey also collected some open-ended information which indicated an increased awareness among library system directors regarding the need for the development of an infrastructure which will support the changes which are now taking place in professional practices and the role of libraries. As part of that infrastructure, public library system directors indicated a need for professional development in change management and organizational development. School library system directors listed developing the infrastructure for library change. These open-ended comments led to the inclusion of questions regarding these issues at the focus forums. Participants there noted that, while the field would like to be part of the policy level discussions and debate that they know are important " we (library systems in the State) don't have the infrastructure to add a diverse voice to policy." The infrastructure they referred to is the structure for systematically eliciting information across stakeholder groups in the state and the process for synthesizing that input. Development of that infrastructure was cited by some focus forum participants as a role for the New York State Library.
Advocacy and Policy Information as Support
Focus forum participants discussed the issues surrounding the general support of library services offered at the local and State levels within the discussion of the LSTA funding. As one participant noted, "How can we translate what we do into a way for people to understand it that's important." It is important because, as participants at all of the forums noted, "we (libraries) need to make a case for library funding".
Respondents to the Statewide Survey indicated support by the New York State Library in a number of areas. The following percent of respondents reported that they contact the New York State Library Division for Library Development for information for planning and advocacy.
| Research and reference resource library system directors | 67% |
| Public library system directors | 63% |
| School library system directors | 95% |
In addition, the same system directors indicated that they believe the following services by the New York State Library aids in their system level planning processes.
The New York State Library is seen as the source of policy level information for the library service delivery system in the State. In addition, the need for information regarding public policy issues and proposed legislation both to managers at the three types of library system in the State and to advocates of the library services system was indicated as growing both through survey responses and in focus forum discussions. Open-ended responses on the survey by both public library system directors and school library system directors indicated a future need for support in the areas of library advocacy, and the development of sources of financial support for library services.
Section Three -- Findings Regarding Management and Operations
Management of the LSTA Grants Program in New York
The LSTA Grants Program in New York State is managed by staff in the Division of Library Development at the New York State Library. The grant funding process begins with a Request for Proposals distributed to the field in the Spring of each calendar year. The form of the Request for Proposals has been modified over the past five years to make the application process more efficient. Grants are distributed most often on an annual basis, although the option to apply for multiple years of funding for any grant project is available. In the case of multi-year applications the funding is approved annually, and at reduced rates for each year subsequent to the first.
Grant project directors were asked if they believe that the LSTA grant support is worth the effort which has to be expended to apply for and report on the funding. Ninety-eight percent (fifty-six of the fifty-seven completing the survey) of them agreed that the grant support is worth the effort.
The New York State Library and the LSTA Advisory Council requested that the evaluators ask for some specific information about grant management in the following areas:
- Barriers to their participation in the LSTA Grant program
- Identification of best practices
- The use of LSTA funds to leverage further funding for continuation of innovations
- The state of communications regarding LSTA funded programs within the library community in the State
- Institutionalization of grant projects
- Issues of quality within the LSTA funded activities in the State
Based on a series of one-on-one and group interviews carried out during the survey development phase of the evaluation, a checklist of possible barriers to grant participation, questions about funding and institutionalization of LSTA grant funded projects were developed by the evaluators and included on the statewide surveys. In addition, questions for clarification of the survey data were included in each of the six focus forums held in September and October 2001.
Limited Categories as Barriers to Participation
In 1997 the New York State Library in consultation with the LSTA Advisory Council made the decision to limit the number of grant categories fundable under LSTA local grants. This decision was made in order to maximize the funding for fewer categories so that the available funding could be more effectively targeted. Findings from this evaluation would indicate that early implementers have found the restrictions on the grants categories to have somewhat reduced their ability to further their established pattern of innovative programming.
Specific information regarding the use of the grant funds in the State was collected both on the statewide surveys and through focus forum discussions. When discussing the quality of the programs funded through the LSTA Grant program in New York, some issues surrounding restrictions on what the grant funds can cover were raised.
Discussion at the focus forums added to the evaluators understanding of the survey responses reported in Table 15. The responses do not mean that the system directors do not approve of the restriction of grant categories. Indeed, focus forum participants were in favor of some stricter restrictions being placed on LSTA grant categories and requiring proof that a proposed grant funded project is, indeed, innovative. What these survey responses were referring to is the five-year span of the same set of restrictions.
System directors and the directors of system member libraries expressed the opinion that the root cause of any issue is that the plan is set and typically not formally revisited during its five years of operation. Focus forum participants noted that in a rapidly changing environment such as they find themselves in at present, the five year plan should be revisited at least every two years, in order to allow for proactive programming by libraries and library systems that is change sensitive. In addition, members of the focus forum discussions noted that better communication of any changes to the relative importance of any grant category should be communicated to the field by the New York State Library, in order to allow them to adjust their planning to reflect the availability of this funding.
Identification of Best Practices and Replication
The evaluation also addressed the management issue of identifying the best practices which are funded by the program. Ways to identify best practices from LSTA grant funded programs were addressed through the focus forums. The point was often made at the forums that there has to be a clear and comprehensive vision for the State in order for the field to be able to align their grant funded programs with the States outcomes. The alignment of any practice with these outcomes is necessary for that practice to be considered to be best. The vision is also related to the use of best practices information. "There needs to be a very clear vision, or else we need to get information out about good programs and how to replicate them," as one focus forum participant noted.
The feeling among some of the focus forum participants was that there is not enough of a replication thrust. They expressed the belief that it would be advantageous for replication to be encouraged, and to do that "the field needs more information and research on good quality programming". Indeed, one forums group felt that it might be advantageous to have grantees submit a replication handbook as part of the product of their grant funded project. These materials should be made available electronically via a web-portal maintained by the State Library or a contracted service provider.
Participants also noted that they could use better information about the quality and results of past grant funded programs. They expressed the belief that if the State is to move forward in the innovative environment the program developers at library system and member library level have to receive clearer information regarding what innovations have been tried and tested and with what result. Indeed, forum participants at all six of the focus forums held as part of this evaluation, expressed the opinion that this basic information would allow them to make informed choices regarding the programs which they proposed under LSTA. In addition, they saw this as a means of developing further understanding of the scope and achievements of the program.
Need for Improved Communications
Communication within a system as complex as the library services delivery system in New York is important to the success of innovation and change at the system level. Communication in general was discussed at the focus forums because of open-ended responses on the Statewide Surveys that addressed issues of communication. In particular, focus forum participants discussed the quality of the communication of grant requirements and funded program information from the New York State Library to them, and then broadened their discussion to include the within system communication networks that do and do not exist in our State.
As can be seen from the responses reported by Grant Project Directors on Table 16, the actual management logistics of the grant program are considered to be clear and easy to follow by a majority of the respondents. Seventy-seven percent agree that the application rules and procedures are clear, and eighty-eight percent agree that the management procedures for the grants once awarded are clearly communicated by the New York State Library.
Issues arose around the communication of the nature of the type of proposals which would be funded, and of changes to State level priorities from year to year during the five years of the LSTA Five Year Plan. Focus forum participants noted that the state sends out abstracts about programs that are funded at the time that the grants are awarded, for example, a practice which they find both useful and frustrating. It is useful because they can see the pattern of successful applications and deduce the emphasis which the State Library is placing within the innovation framework for that year. One drawback to this system is that in order to ascertain specific information about a particular program design, success in reaching the target population, and other evaluative details, one has to contact the project recipient directly. They also noted that the evaluation information from each of the programs is not shared, although this information would be helpful. As one participant noted, "you can learn as much or more from ideas that did not work as you can from descriptions of what did work". The field does send huge narrative reports to the State, participants told the evaluators, perhaps that information could be synthesized and presented to the field in general to inform their planning and development of innovative programs.
When asked to consider these points raised in previous focus forums, participants at the Rockville Centre forum (the last one held) proposed the following six items as the most important things to report in an evaluation:
- Things that did not work, and the problems and issues encountered by the program.
- Impacts and benefits on the program participant target population, and other stakeholders.
- Things that the project would do differently if they had it to do over again.
- Things that would stay the same and how they would be improved for the future.
- Feedback from participants on the activities, outcomes, processes of the program.
- A measure of how successful the program was based on whether or not it was institutionalized after the funding ended. This poses the follow-up issue of a better definition of institutionalization for general measurement of the same thing.
There is an understanding on the part of both the informants to this evaluation and the staff at the New York State Library that much of the information listed above (numbers 1-3 and 5) is already collected. The point being made was that the information is collected and then not reformulated into a useable format for the field to use in their work within the LSTA program. This, again, highlights the issue of sharing of information through effective communication strategies. Focus forum participants discussed the potential use of technology to help to get information from grant funded programs and initiatives to the rest of the field in a timely manner. Some way of requesting and presenting information about the grant funded programs electronically, they said, would be ideal.
The focus forum discussion then turned to general communication within the library services system in the State. They noted that the use of a ListServ to disseminate information to them is not efficient. The ListServ has been replaced in many private sector environments with other means of electronic communications (threaded discussions for asynchronous communications, for example, and sametime conferencing software for synchronous communications). Focus forum participants believe that the State Library wants to communicate more clearly with them. Library systems also noted that the systems should practice what they preach by using more cutting edge technology to reformulate their communications strategies. It might be that the more efficient strategy for the reformulation of communication within the library services system in the State would be the design of communications at each level of the system with processes for sharing information across system levels where warranted.
Focus forum discussion was expanded into conversations regarding the emerging need for change management to further the development of the library and technology infrastructure now necessary in New York in order to provide a more detailed dataset based on information collected through the Statewide Surveys. Focus forum participants noted that five years ago any new idea was worth supporting, and many high quality and innovative practices were tested and institutionalized in our State. Times have changed, however, so that the next phase of change in the State will have to be more focused, more closely managed and therefore implemented with more open, clear and regular communication among the concerned parties. The feeling among most focus forum informants was that there has to be a more organized way to coordinate this change process in the State, and that clearer communication strategies are probably the place to start.
New York is a large State with expensive internal airline charges (one forum participant noted that it is often cheaper to fly from Albany to London, England, than from Albany to Buffalo, NY) and long overland journey time from Long Island to Buffalo takes in the region of eight hours by car. Participants expressed the desire to be proactive participants in the management of the change process in the State, but noted that neither they, nor the staff at the State Library, can afford the expense in funds and the opportunity cost in staff time to travel the vast distances that face-to-face meetings would require. Again, the exploration of new technologies to allow for both asynchronous and synchronous communications was recommended.
Barriers to Institutionalization of Grant Funded Innovations
The question of institutionalizing innovations supported by the LSTA Grants programs was a critical question addressed by this evaluation. Some interesting information emerged from this line of questions. For example, in general, participants at the focus forums felt that the time restrictions on the grant projects caused a great many problems, the most significant of which is its negative impact on the quality of the programs that are delivered. As one participant at Batavia noted, "one year of funding prevents high quality programming, especially if you want to work cooperatively with agencies." Forum participants spoke of the time it takes to get a program up and running, the unforeseen issues that arise when an innovative program or practice is introduced into their system and the time it can take to work these things out. In addition, the one time only policy for LSTA funded programs means that a good innovative program introduced into the existing system has no follow-up to ensure its institutionalization. One year is considered just too little time to ensure this.
One participant summed this up thus:
We cant get any impacts in this timeframe. The question is not really what other data should we collect, it is more about what timeframes we should establish to ensure that programs are of high quality, properly and fully implemented and being given a chance to become part of the institutional landscape.
The other restriction which is seen by many of the system directors as a barrier to institutionalization of LSTA funded innovations is the inability to use the grant funds to pay existing staff to work on grant funded activities, discussed in detail below.
Responses on the Statewide Surveys indicated that there was an issue among system directors and grant project directors around the restriction that disallows the use of LSTA grant funds to pay any existing staff to work on grant funded projects, including administrative staff. The information presented below is taken from those Statewide Surveys.
The inability of library systems to pay for existing staff to work on LSTA grant funded projects was discussed at all six focus forums. This restriction was seen by forum participants to negatively impact on the quality of the work being done and on the ability of the systems to institutionalize the work of many of the grants. In addition, the forum participants raised the inability of the systems to pay for administrative costs out of the LSTA grants as a challenge to the efficiency of the grant administration. The findings are reported as part of the evaluation, but no recommendations are made in reference to them.
The data collected on the Statewide Surveys, however, presented a problem when the evaluators came to interpret their meaning. In the light of the previously stated concern with the institutionalization of LSTA funded grant initiated innovations, the report by 96% of the Grant Project Directors that work started with LSTA funds tended to be continued in their context seemed a contradiction. One interpretation was that the only grant project directors that completed the survey were those for whom the institutionalization of their grant project had been successful. It was decided to ask the focus forum participants what this response meant. The results of these and other questions regarding funding of innovative practices are reported in the next section of this report.
The Nature of Innovation Funding and Continuation of Support
Library system directors in New York State understand the need for funding of innovative approaches to the delivery of library services. They also indicated a strong belief in the use of innovative funding to support only innovation, and expressed their commitment to encourage a restriction of the use of funding such as that provided by LSTA to support ongoing programming. Indeed, none of the reference and research library resource system directors, and between twenty and twenty-five percent of the other system-type directors, consider required matching funds as a barrier to the use of LSTA grant funds for innovation.
However, when asked on the Statewide Survey if they had used LSTA Grant funds to leverage funding from another source, very few of the system directors indicated that they had. Three of the fifteen (20%) school library system directors, four of the twelve (33%) public library system directors and four of the nine (44%) reference and research library resource system directors who have had LSTA Grant funds used them to leverage funding from another source. The evaluators included questions about support and funding in the focus forums in order to understand this reported leveraging of funding.
What focus forum participants told the evaluators was that LSTA funds are often a small part of the overall funding for one of these projects. If one considers the fact that existing staff cannot be paid out of the LSTA grant, including administrative staff, the opportunity cost of running one of these programs can be quite high. The point was made at more than one forum that the funds used to implement the programs supported by LSTA grants is from local budgets. This means that funds are leveraged prior to the implementation of the program being funded, and will continue to be expended to support the core program components as long as the library systems believe the objectives of the program are important. The commitment of the systems to the programs that they propose is evidenced by this willingness to support the programs with their own funds, as one participant put it: "carved painfully from our resources describes it all".
This helped the evaluators to make sense of the moderate to low levels of support for various innovative programs and services reported on the Statewide Surveys and reported here as Tables 21 and 22.
Forum participants were clear that communication of information regarding what had been funded, with what overall purpose in mind, and to what end should be implemented for a number of reasons. One of those reasons is that the system directors would be able to make informed decisions regarding their application for and use of LSTA funds in the future. A second reason is that the system directors expressed the belief that they need a clearer big picture in order to effectively support the work of the New York State Library in this area. The point made at one forum, but repeated at all of them, that "if federal funds are being used we (system directors) should have a say in how it is spent" was part of this more general discussion about LSTA grant management. This would be relatively easy to do following a redesign of the communication processes now operating in the LSTA Program in New York.
Conversations in the focus forums around how to evaluate these programs effectively yielded a set of questions about the use of seed money to test innovative practices. Central to this was the question of what the research says about the effectiveness of this type of funding on innovative practices. The continuation of support for LSTA funded programs merged with the forum conversations regarding the use of results-based planning to manage system development around library services in New York.
A summary statement regarding this relationship was made at the Albany forum, where one participant stated that the critical question to be answered is: "Can we (library system directors) as a group align our resources?" This summarizes the dominant theme of the data collected as part of this evaluation. There was no dissent regarding the need to change library services in the state, the need to expand those services to include traditionally underserved and not served populations in the state, and the need for libraries to readdress their role in community development. These things were accepted as generally correct assumptions by all those interviewed, surveyed and included in focus forum discussions. What emerged as the questions now challenging the library services delivery system in New York State are issues regarding managed change, synthesizing of resources and the development of a coordinated and consolidated service delivery infrastructure. This study has produced the following set of conclusions about the LSTA Program in New York State and recommendations regarding next steps to be considered in maximizing the contribution of LSTA to planning and implementing development of library services across the state.
Section 4 -- Conclusions and Recommendations
One funding source, and the programs and services supported by it, will not create the sole impact on the development of a system for library service delivery as complex as that now being developed in New York. However, findings of this evaluation would indicate that the use of the LSTA funds has made a positive contribution to this development at the local, regional and State levels. Of course, there is still a great deal of work to be done.
Conclusions and recommendations expressed are based on the evaluation that is reported in this document. They should be combined with evaluative information from other sources and not considered to be pertinent to any but the LSTA programs and services included in this study. The conclusions and recommendations are presented in answer to each of the three areas of evaluation question which this evaluation sought to answer: evaluability questions; implementation evaluation questions; and, impact evaluation questions.
Evaluability Questions
This evaluation sought to answer two evaluability questions.
- Is the information presently collected for the LSTA program sufficient for the measurement of the impact of the program on libraries, library systems and the quality of services to the people of New York?
- What are the performance indicators most appropriate for measuring interim, short-term and long-term outcomes of the LSTA programs and services.
Conclusions Evaluability Questions
Based on the data collected in this evaluation, the information presently collected for the LSTA program is not sufficient for the measurement of the impact of the program on the library services system of the State. The data elements now collected are more relevant to measuring the activities of programs and scope of services than they are to measuring the changes in the systems behaviors which they effect.
At present the information collected formally is done well but is limited to compliance reporting requirements that were in place when the present LSTA legislation first went into effect. While the quality of the information collected is not questioned by the results of this evaluation, the fact that it is limited does have an effect on evaluation utilization. LSTA managers are aware that small amounts of evaluative information might be misinterpreted without situating it in the larger context. Therefore, the limited quantity of formal evaluation information which is available is not shared with all levels of the system. Managers report seeking feedback and other information which is noted and referred to, but that information is not collected, organized, analyzed and interpreted in any demonstrably formal manner, thus at times making decisions using it appear to other levels of the system to be based on opinion rather than fact. In addition, the Government Performance and Results Act, and the impacts of that legislation on program management and measurement mean that new regulations will probably require significant changes to the purpose of data collection within this program. For these reasons, the evaluators conclude that the program would profit greatly from a continuous rather than a periodic evaluation effort.
Recommendations Evaluability Questions
The Library Services and Technology Act Program in New York State is a complex, multifaceted program that requires an ongoing evaluation strategy. Ongoing evaluation is a formal type of evaluation that elicits information from the data sources that contain or provide information about performance indicators. Performance indicators are defined as quantifiable expressions of those program variables that are measured. Typically, performance indicators operate in clusters or three to five and are measured as a single entity. The LSTA Program managers will have to identify the performance indicators for the program, and then organize ongoing data collection, analysis and interpretation processes which will measure and monitor the program through those indicators. In order to do this it is recommended that they use the logic model developed by the evaluators for the program and identify performance indicators for the logical strands originating with each of the three core activity types within that model.
The development of a performance indicator system which addresses the LSTA program, and not any other program, would be a mistake. The interim effects of each of the LSTA funded grant programs and services have been reported in this evaluation. Changes in professional attitude, organizational context, service delivery systems and scope of services would be worthwhile performance indicator categories for short-term outcomes. A cluster of indicators to measure improved quality of services to all New Yorkers in each of the communities outlined by the system directors of the three system types in the State would constitute a serviceable set of performance indicators on the long-term outcomes for this program. This would ensure that the LSTA would be measured as it fits into the other initiatives for change and development of library services in the State.
The evaluation procedures recommended here are referent to the use of a mixed method approach to data collection, analysis and interpretation. Data collection procedures would include interviews, questionnaires, observation or inspection of practices, and review of records, files and already existent data. The data collection process is the compiling of data on indicators to show performance during a specified time period. Many ongoing evaluation systems particularly those dealing with program effectiveness also contain a set of program standards. These performance standards would set a desired level of achievement for each cluster of performance indicators. The evaluators strongly recommend that the LSTA Grants Program managers consider establishing a set of program performance standards for both the grants program and the statewide services funded under LSTA.
LSTA Grant Program:
Under the present five-year plan, the LSTA Grants Program has supported grants in specific categories. Evaluation strategies for each grant category should be tailored to the focus of those categories. However, there are a number of general design characteristics that all evaluation of grant funded activity should have. First, all funding proposals should contain baseline data to support their application for support. This data should be prescribed by the New York State Library, be standard across grant types, and constitute a measure of identified program performance indicators. This baseline data should not be used as a criteria for funding, and functions in an ongoing evaluation only as baseline or anchor data for the calibration of change in the system over all grant funded activities. Second, a performance agreement, based on the baseline data and clear performance standards should be set with all funded programs. Third, all programs funded should be required to provide output data at the end of the funded activities, and periodically for an appropriate time following the end of the funded activities. This is often done quarterly for professional development grants, for example, and semi-annually for process change grants, such as those given for retroactive conversion. Output data is data which focuses on change in practice and/or attitude for human resource development and change in efficiency and/or effectiveness in process change. The program performance indicators again should be used to develop these measures.
The purpose of the measurement of the grant funded activities is not to establish that the activities took place, but to ascertain what changes to participants and/or processes resulted from the grant funded activities. Collecting information about the processes undertaken within each grant funded program design will inform the system about the most effective strategies to support and facilitate change. This product of the evaluative process replaces the previous activity of identifying best practices.
Some system directors and grant project directors attending the focus forums indicated that they would like some professional development in evaluation, including new and emerging evaluation practices. The evaluators recommend that the New York State Library ascertain the magnitude of this need.
Statewide Services:
The New York State Library should put in place a system to formally measure both the level of activity for each Statewide Service area and the change which these activities facilitate and support. Measurement of the impact of LSTA supported Statewide Services on the library experience of residents of New York is complex. Because the library services delivery system is structured as three library systems delimited by type, impact measurement has to include an intermediate step. The LSTA Grants Program supports innovation and change at each of the system levels, so that measurement of its impact is more transparent. The innovation implemented is designed by the grant recipients to have an effect on their member libraries, as change in professional knowledge and skills or change in the audience for their services, for example. Measurement of the change facilitated and supported by Statewide Services is more complex.
Development of EmpireLink/NOVEL
The development of NOVEL (the New York On-line Virtual Electronic Library) is different from the grant funded activities, but similar in its ability to be measured. The impact of the innovation on the ability of New Yorkers to access electronic resources is direct and, because of its linear nature, it was more obvious to the informants of this evaluation. One key evaluation finding is that library system directors can see and therefore support the development and wide dissemination of access to NOVEL, and that many of those same directors take the more support and facilitation of change oriented Statewide Services for granted. The evaluation has established that both roles for the Statewide Services are highly valued and judged to be important.
The effectiveness of NOVEL as the structure within which the network of systems are linked is actually difficult to measure. Systems in the state reported to this evaluation that those that have a choice to use NOVEL (EmpireLink) or to buy its equivalent, use NOVEL (EmpireLink) in order to free-up resources for other uses; and those that do not have a choice use it because they can not afford any other system. The development of a set of performance indicators for this activity is critical to allowing the New York State Library to measure the effects of this strategy.
Evaluation strategies for the Statewide Services follow the recommendation that the variables measured be part of a performance indicator system developed by the New York State Library. Development of NOVEL should be measured as both the structural framework of the network of systems that will constitute the system of seamless operability that the state is developing and as a quality improvement strategy for client services in the state. For the support and facilitation focused Statewide Services, measurement of activity is recommended to be cycled into a monitoring of perceived effect.
The New York State Library should consider monitoring activity in the support and facilitation services they provide. This can be achieved through the use of staff activity logs, for example, completed by all staff on a random sample of days each month. Electronic logs would facilitate the input of this data. In addition to the measurement of levels and nature of activity, a formal mechanism for collecting New York State Library staff perceptions as professional observations of the operation of the library services delivery system in the state would be a strong component of any ongoing evaluation.
Using the information about staff activity, periodic feedback from the field regarding these services and their effect on the library systems and their member libraries should be collected. Short surveys, online if possible, and/or threaded discussion in a managed environment where respondents answer specific questions can also take place online. It might be helpful to hold semi-annual focus forums either in person or live online to elicit feedback on gaps in services, issues of communication or other emerging concerns. This data collection strategy would allow the New York State Library to elicit information from key stakeholders who are noted as not participating in other data collection activities as well as those who are.
Implementation Evaluation Questions
This evaluation sought to answer six implementation evaluation questions in reference to the LSTA program in New York.
|
1. |
How closely do each of the program activities align with the intended implementation objectives of the LSTA Program? |
|
2. |
Are the various objectives of the grant programs aligned with the intended outcomes of the LSTA Program? |
Conclusions Implementation Evaluation Questions 1 & 2
Review by the evaluators of a set of spreadsheets prepared by the New York State Library Division for Library Development indicated that the grant programs which were funded were closely aligned with the scope and objectives articulated in the New York State Five Year Plan. However, some issues around this adherence to the five year plan scope arose during the course of the evaluation. The use of a plan fixed for a five year period might be detrimental to the development of library services in the State in this innovation driven funded environment.
Recommendations Implementation Evaluation Questions 1 & 2
The evaluators recommend that the LSTA Five Year Plan be revisited bi-annually in an environment which is open and interactive with system directors and other key informant groups in the library services delivery system in the State. In addition, any changes in emphasis or objective should be communicated with the field in a reasonable timeframe and with a clear communication method.
|
3. |
Is the expectation clear from the State level that grantees should seek ongoing support of LSTA supported projects where appropriate? |
|
4. |
How do grant recipients provide for ongoing support of the outcomes of successful projects, e.g., reallocation of funds, establishment of partnerships, and other activities? |
Conclusions Implementation Evaluation Questions 3 & 4
This question is part of the larger question regarding the institutionalization of LSTA funded programs. All indications from the data collected within this evaluation are that this is the case. Indeed, the sequence of the introduction of local funds into the support of LSTA supported projects is one of the more surprising findings of this evaluation. It seems that in many cases the New York State Library can consider the application for LSTA grant support for innovative programming as a tentative assertion that local support for the idea has already been procured.
Recommendations Implementation Evaluation Questions 3 & 4
There are some things that the New York State Library should consider that will facilitate further institutionalization of the LSTA funded programs. Informants to this evaluation indicated that the short timeline of LSTA grant funded projects restricts their ability to seek institutional support for the supported innovations. This was reported as the case because they could not collect impact data, for example, and because of the budget cycles in their organizations they could not be up to twelve months out of cycle for ongoing funding applications. Collection of follow-up data from grantees six and twelve months following the grant funded program, specifically reporting on the institutionalization of components of that grant funded program (including funding) would be helpful to the grant program managers and to possible replication sites.
|
5. |
What are the characteristics of effective partnerships and collaborations that libraries and library systems enter into in order to improve efficiency and effectiveness of services? Are there readiness criteria which can help to guide libraries and library systems as they enter into partnerships and collaborations, both for the libraries and for the agencies with whom they will collaborate? |
Conclusions Implementation Evaluation Question 5
A review of the research on change literature indicated that the movement into collaborations, both within their same institutional type and with other community support agencies, will strengthen any human service delivery agency. The changing role of library services means that the contours within which those services are delivered will also change. The evaluation concentrated more on whether there are collaborations and partnerships both within the library community and with agencies other than libraries. What we found is that most library systems collaborate with the other types of systems in their regions, and many collaborate with library systems outside of their region as well. Because this evaluation is the first formal collection of data on these issues in New York, it is difficult to tell if this is an increase over past behavior, although focus forum participants and State Library staff all reported that it is. Work with other agencies is sketchy in the State, with most system directors reporting some collaboration, mostly for joint funding ventures and to show their face in the interagency collaboration environment.
The findings of the evaluation also provide some tentative information about the readiness criteria in this evaluation question. Many library system directors see themselves as important participants in the interagency service delivery environment. This role is linked to the changing context of library services in the state as well as to the changes in professional practice now taking place for librarians. The evaluation found that indicators of readiness criteria are emerging, both for library systems and other agencies which enter into these collaborations. However, there is not sufficient data now being collected in the state to establish what these readiness criteria are.
Recommendations Implementation Evaluation Question 5
The evaluators recommend that a formal monitoring of the characteristics of and impacts resulting from collaboration be developed and implemented by the New York State Library as part of the performance indicator system. Measurement can be done in a number of ways. The most promising might be to consider using a composite case methodology that would identify and measure in depth on the characteristics of within library and outside of library networking, cooperation and collaboration on a small sample of identified successful examples of collaboration. The product of this activity would be a description of the pre-collaboration characteristics of all operators in the collaborative, perceived benefits, opportunity cost, and perceived barriers.
Impact Evaluation Questions
This evaluation sought to answer eight evaluation questions and eight sub-questions concerning impact of the LSTA funded program.
1.Do all New Yorkers have electronic access to information resources?
- How do LSTA Technology Grants contribute to this access?
- How do LSTA Technology Statewide Services contribute to this access?
Conclusions Impact Evaluation Question 1
All New Yorkers do not have electronic access to information resources. However, the number of New Yorkers with access has increased over the five year period of the present LSTA Five Year Plan, as reported to the evaluators by system directors. LSTA grants have helped these system directors to increase and improve access for their member libraries and the clients of those libraries. However, the rapidly changing nature of the technology applications which are the substance of this provision means that professional development and other grant provided services will have to continue into the next five year plans period. The contribution by the Statewide Services was more difficult to establish. The evaluation findings clearly indicate that the development of NOVEL is thought to be an important and valuable contribution to the delivery of services in the state. The impacts of the Statewide Services focused on facilitation and support of change and innovation were more difficult to measure. This is due in part to the lack of a clear articulation by the New York State Library of how all of the various components of this system development fit together. The contribution of leadership in results based planning was finally identified as the most important Statewide Service to this outcome.
Recommendations Impact Evaluation Question 1
The New York State Library should continue to provide and if possible expand the human resource development support that has been provided through this grant program. In addition, consider the technology infrastructure issues which were raised during this evaluation (telecommunications issues, purchase of commercial databases).
The New York State Library should formulate and develop a communication strategy that disseminates a version of their articulation of the components to systemic change in the State that identifies the relationship of those components to the three library system types in New York. This articulation does exist, but it is typically focused in terms of the ultimate users of the library services system. This system-type focused articulation should situate all of the components contributing to this reformulation of library services in the State in such a way as to provide the three types of system in the state with a strong sense of where each of them fits into a plan for managed change.
2. Do libraries and library systems use LSTA funding to deliver programs that meet and anticipate the dynamic needs for library services?
- How do LSTA Technology and Special Services grants support this outcome?
- How do LSTA Technology and Special Services Statewide Services support this outcome?
Conclusions Impact Evaluation Question 2
The data collected within this evaluation would indicate that the restrictions on use of LSTA funds and on the type of library which can use certain types of program funds, seriously restricts the systems ability to use LSTA funding to anticipate dynamic needs for library services. In addition, more success in this outcome area is evidenced in the Technology Grants than is evident in the Special Services grants. In most cases, the Special Services grants were reported as delivering needed services to special populations, but not as innovations per se. Innovation in the Special Services programs were more often in the mode of program delivery (using computer based instruction in an adult basic education program, for example, or helping job-seekers to search help wanted databases) rather than in the content of the programs.
The findings regarding the Statewide Services were more promising. These services seem to provide necessary information to the systems in the State for planning, advocacy and merging of support. This therefore allows for the systems and libraries to meet and anticipate dynamic needs at the local and regional levels.
Recommendations Impact Evaluation Question 2
The difficulty in addressing this outcome could well be the outcome and not an indication of any failing in the systems being evaluated. Measurement of operational variables[3] (such as those associated with professional development) is straightforward. However, establishing that professional development has changed librarian practice, for example, does not establish that the change in practice improved client services. The true level of this knock-on effect is difficult to ascertain without more comprehensive in-depth questioning of key informants in the systems and a sample study of representative sites. We would recommend further data collection, using the focus forum technique would be effective, to seek this information from system directors and member library directors around the State and to identify possible sample case sites.
[3 Operational, sometimes called mediating, variables are variables that effect the independent or predictor variable but which cannot effect the dependent or outcome variable directly.]
3. Has public policy support for libraries strengthened through activities of libraries, library systems, other library organizations and the State Library?
Conclusions Impact Evaluation Question 3
The perception of the system directors and library directors who attended the focus forums is certainly that public policy support for libraries has not strengthened. The defeat of various library support options in the New York State budget over the years has led to an overall feeling among the members of the field that their work is not valued, their position in the human service delivery system is not understood, and the need to fund libraries is not seen as an important issue to the State.
Recommendations Impact Evaluation Question 3
More work needs to be done to articulate the role of the services which libraries and librarians can deliver. The appropriate group to do this is the system directors, a role in which they expressed an interest which solidified during the focus forums.
- How do LSTA Special Services Grants support this outcome?
- How do LSTA Special Services Statewide Services support this outcome?
4. Do all New Yorkers have access to library resources and services that advance and enhance their lives as workers, citizens, family members and lifelong learners?
Conclusions Impact Evaluation Question 4
All New Yorkers do not have access to library resources and services that advance and enhance their lives. However, more have access than had access five years ago, and the quality of that access has improved along with the magnitude of the access provided. Informants to this evaluation reported that they have increased the amount of access in their areas, by using EmpireLink, for example, and they have increased the quality of the services they provide, for example, by collaborating with other libraries and/or service delivery agencies in their regions.
Increased literacy in the adult population, through both adult basic education and English as a second or foreign language instruction, can be measured as the removal of a barrier to employment. And, for programs that are family oriented, such as family literacy programs, the program content can include elements such as information regarding human services available to New Yorkers, family wellness and parenting support. In addition, increased literacy means increased use of libraries by adults and potentially by their children. What is new and increasing is that, especially in urban areas, greater responsibility for the provision of the adult and family literacy programming is falling to the libraries. Interviewees and focus forum participants in this evaluation indicated that there is an increased need for training of library staff in the management of this type of programming, for example, not in how to deliver this type of programming. This change has been subtle, moving the library (typically the public library) from the site of service to the source of service, but it brings with it increased stress on the library service delivery system in the state.
Recommendations Impact Evaluation Question 4
This is a sensitive area in human service delivery programming, and one which libraries should approach with some care. Taking over adult and family literacy responsibilities in a librarys community, whatever community type they serve, can present complex and expensive problems. This is a role which may be emerging for libraries in the new interagency human service delivery environment, so that it warrants close monitoring. LSTA should not be used to provide ongoing services, however this is the one area where that has the highest risk of happening.
- Are new populations being served?
- Has technology changed practice? If so, how?
5. How have services to users been transformed by this program?
Conclusions Impact Evaluation Question 5
The findings of this evaluation are that many new populations are being served in New Yorks library service delivery system, with new services and innovative approaches to delivering those services. The new populations include residents new to the United States, traditionally underserved populations using the library, and professional support services being provided to new audiences. The nature of the librarys delivery of information has changed. Librarians are finding they have more time for more complex tasks as their clients can do more routine information seeking on their own, etc. As noted elsewhere in this report, the scope of library responsibility has increased in recent years, which focuses the need to address changes in scope of services as they impact the library services delivery system as a whole.
Recommendations Impact Evaluation Question 5
Continue to monitor the changes in service delivery through the ongoing evaluation recommended here. Change is often good, but all change is not necessarily good.
- Is information distributed through the State Library website effective?
6.Are the communication mechanisms now operating in the LSTA funded system adequate to provide dissemination of the programs work?
Conclusions Impact Evaluation Question 6
The communication within this system needs serious redevelopment. The sense from the data is that there is a great deal of effort yielding very little efficiency of communication. For example, most informants to this evaluation find the State Library web-site confusing and difficult to use.
Using communication to develop an ability by all levels of the library service delivery system in the State to recognize the interconnectedness of the activities, services and initiatives implemented to effect change and development of the system is an important strategy for the State Library to attend to in the future. In combination with other recommendations made by this evaluation, the development of a simple and effective communication strategy would strengthen the change process in the state.
Recommendations Impact Evaluation Question 6
Once the other components of the management of change recommended by the evaluation are in place, communication within the library services delivery system should be more easily focused. The use of existing communication mechanisms that are good as the basis of any new system of communication is strongly recommended. Informants to the evaluation expressed their wish to continue one on one communication with the New York State Library staff, for example, and expressed a wish to have the opportunity to meet with one another in person at least annually. While this is provided by some of the professional organizations to which professional librarians already belong, the key stakeholders to this evaluation noted that those meetings are already full of information and material. What was requested are meetings attended by key stakeholders where key informants would discuss issues specific to their library system type or to their region. The evaluators recommend that every effort be made to accommodate this request. Some online conferencing should be considered, along with a combination of asynchronous and synchronous applications for targeted discussion on critical issues. New York has a system for video conferencing which could be utilized, as well as the personal computer based video conferencing applications now available.
7.What is the best way to identify best practices from among those supported through LSTA funding?
Conclusions Impact Evaluation Question 7
A better system of communicating the program components of any programs supported would be the place to start here. The general feeling is one of uncategorized activity leading to hidden outcomes for unidentified populations. Once these are organized, the best practices should emerge clearly, and keyed to needs analysis identified causal characteristics.
Recommendations Impact Evaluation Question 7
The newer approaches to program evaluation and dissemination of promising practice might make the concept of best practices difficult to continue. If the ongoing evaluation proposed in this report is implemented, then the strategies that are most effective for change will be able to be identified. The sequence of events here are important. A performance indicator system has to be developed, measures based on that system have to be designed, and the baseline data for the ongoing evaluation have to be completed.
Informants to this evaluation also expressed an interest in the development of a strategy whereby innovative programs funded under the grants program would be piloted in one funding cycle, replicated in a controlled design in the following funding cycle, and that the replication site would produce as a final product a handbook on the program. Finally, these handbooks and other pertinent information on innovative programs would be available in an electronic format for consideration for use by other sites. While this may not be appropriate for all types of grant funded programs, the New York State Library should consider adopting this approach where appropriate.
8. Is there important information to collect from LSTA grant recipients that was not included in their final reports to the State?
Conclusions Impact Evaluation Question 8
Informants to this evaluation told us there is not. This is mainly due to the fact that with the short timeframe allotted to the grant programs, very little impact or outcome data can be collected. However, the evaluators believe that other findings call this into question. LSTA Grant project directors indicated that they believe the program theory of change, if not all of the program components, is successfully incorporated into their system in a majority of the time.
Recommendations Impact Evaluation Question 8
The measurement of the percent effect attributable to LSTA support becomes important. Use of approaches such as those developed by Mohr would be appropriate. In order to do this the New York Sate Library would need the services of an external evaluator, at least initially. Once this was developed it could be incorporated into the ongoing evaluation recommended in this report.
Appendix I -- Evaluation Design for New York State Library Services and Technology Act (LSTA) Program
The evaluation of the LSTA Program is an evaluability study incorporating an impact evaluation of the program and all of its components. The design in detail here incorporates the following fourteen evaluative components:
- Conduct a data audit which compares data presently collected with the data needs of a system of outcome indicators for all aspects of the LSTA program, both local and statewide.
- Identify and analyze program user outcomes across library types, both local and statewide.
- Identify further data necessary to measure program user outcomes across library types.
- Identify instances of leveraging of LSTA funds to obtain other sources of financial support and to enhance services begun with LSTA funding.
- Demonstrate that programs funded by LSTA are ongoing beyond the terms of the initial LSTA support.
- Establish the effectiveness of LSTA funded services using data collected by grantees after the final project report.
- Describe the nature of partnerships/collaborations developed by service agencies aimed at increasing efficiency and effectiveness of services.
- Document the ways in which services to users have been transformed as a result of LSTA programs.
- Document the effect of programs on secondary stakeholders, such as users of library services.
- Document the ways in which results of grant funded programs were communicated.
- Establish that new populations were served.
- Overall, measure how the New York State Library's statewide services have affected users.
- Design and make recommendations for the implementation of a process for collecting data from program service delivery professionals as users.
- Design and make recommendations for the implementation of a process for collecting data from library program users.
Purpose of the Evaluation
This evaluation has five specific purposes. They are:
- To measure the impacts of the LSTA supported activities at all appropriate levels of the library infrastructure in New York State.
- To improve program effectiveness of both Grants and Statewide Services.
- To evaluate the quality of the information which routine evaluation activities have yielded during the past three years and to make recommendation on how to improve the quality (including the timeliness and applicability) of that information.
- To inform planning activities that will be undertaken by the New York State Library as they develop their strategic and operational plan(s) and next LSTA Five-Year Plan. This will include consultation with the evaluator on the application of results-based planning to the strategic planning process.
- To provide information that will strengthen advocacy for reauthorization of LSTA.
Questions answered by the evaluation
It is often helpful to think of the work of an evaluation as being the collection of information which will allow the program managers to answer specific questions posed about the program. The questions which this evaluation will answer are based on the fourteen evaluative components and include questions of evaluability, program implementation and program impacts. Evaluability is the measurement of whether a program is collecting enough of the correct types of information about itself to enable an implementation and/or impact evaluation to be done. Implementation evaluation is the measure of fidelity of the program as implemented with the design of the program in the first place. This programs implementation will be measured at the State and system levels. Impacts (also called outcomes or results) will be measured relative to short and long-term program outcomes which have been identified.
In addition, numbers thirteen and fourteen in the list are evaluation system design requirements under this contract and do not answer specific evaluation questions. They have been included here as the points under "Recommendations Data Collected in This Evaluation Will Support".
Evaluability Questions:
Is the information presently collected for the LSTA program sufficient for the measurement of the impact of the program on libraries, library systems and the quality of services to the people of New York?
What are the performance indicators most appropriate for measuring the Interim, Short-term and Long-term Outcomes of the LSTA programs and services?
Implementation Evaluation Questions
How closely do each of the program activities align with the intended implementation objectives of the LSTA Program?
Are the various objectives of the grant programs aligned with the intended outcomes of the LSTA Program?
Is the expectation clear from the State level that grantees should seek ongoing support of LSTA-supported projects where appropriate?
Are there issues of narrowness of funding parameters? How do these grants and services support dynamic needs with static funds?
What are the characteristics of effective partnerships and collaborations that libraries and library systems enter into in order to improve efficiency and effectiveness of services? Are there "readiness" criteria which can help to guide libraries and library systems as they enter into partnerships and collaborations, both for the libraries and for the agencies with whom they will collaborate?
Impact Evaluation Questions
- How do LSTA Technology and Special Services grants support this outcome?
- How do LSTA Technology and Special Services Statewide Services support this outcome?
- How do LSTA Special Services Grants support this outcome?
- How do LSTA Special Services Statewide Services support this outcome?
Do all New Yorkers have electronic access to information resources?
How do LSTA Technology Grants contribute to this access?
How do LSTA Technology Statewide Services contribute?
Do libraries and library systems use LSTA funding to deliver programs that meet and anticipate the dynamic needs for library services?
Has public policy support for libraries strengthened through activities of libraries, library systems, other library organizations and the State Library?
Do all New Yorkers have access to library resources and services that advance and enhance their lives as workers, citizens, family members and lifelong members?
How have services to users been transformed by this program? For example:
Are new populations being served?
Has technology changed practice? If so, how?
Is information distributed through the website effective?
Are the communication mechanisms now operating in the LSTA funded system adequate to provide dissemination of the programs work?
What is the best way to identify best practices from among those supported through LSTA funding?
Is there important information to collect from LSTA grant recipients that was not included in their final reports to the State?
How do grant recipients provide for ongoing support of the outcomes of successful projects, e.g., reallocation of funds, establishment of partnerships, and other activities?
Recommendations Data Collected in This Evaluation Will Support
Proposed design for collecting evaluative data from program delivery professionals as users. Initial implementation of this design.
Proposed design of a process for collecting data from library program users, including the collection of information regarding the effect of programs on secondary stakeholders. Initial implementation of this design.
Evaluation Methodology
This evaluation will use a mixed method design, which uses a combination of quantitative and qualitative data collection techniques which are analyzed and incorporated into the ongoing study. The timeline is important in this type of evaluation work because each wave of data collection depends to some extent on the results of the preceding data collection, analysis and interpretation activities. The following plan presents the activities which will be undertaken to answer the evaluation questions and the months during which the activities will take place.
Evaluation Timeline, Activities and Product Delivery Dates
October 1, 2000 January 31, 2001
Review materials provided by State Library staff.
Review summary of grantee data for the past three years prepared by State Library staff.
Initial contract called for CDA Corp. to perform secondary analysis of data compiled by State Library staff which has been reported in grant reports. Following analysis of the type of data reported in grant reports, the evaluator will not perform a secondary analysis. Instead, a report on the quality of the data reported will be prepared and submitted to the State Library staff.
Product: Report on quality of data supplied through grant reports, January 31.
Use a logic model to identify the relationship between LSTA supported activities and their short and long-term outcomes. The model will be presented in both graphic and narrative format.
Product: Program Logic Model in graphic and narrative formats, January 31.
Develop an evaluation design including data collection timeline for the program.
Product: Written Evaluation Design including data collection timeline, January 31.
February 1, 2001 April 30, 2001
Review Evaluation Design and Program Logic Model in graphic and narrative formats with Evaluation Committee.
In consultation with State Library staff and evaluation consultant, identify appropriate key informants for interview(s) described below.
Conduct group and one-on-one interviews with identified sample of grant recipient system and program directors to ascertain information regarding: ongoing funding; enhancement of services; description of program components and fidelity of implementation; and communication of program information.
Timeframe dependent on availability of informants. Collection complete by March 31, 2001. Analysis of interview data complete by April 15, 2001.
Ongoing analysis of interview data. Information from these interviews will be used to develop two surveys. One survey will be developed for grant recipients and one survey will be developed for those who have not applied for grants.
Conduct interviews with State Library staff regarding the nature of their services and the identity of their target population.
Interviews completed by February 28, 2001.
Ongoing analysis of interview data. Information from these interviews will be used to develop the logs for use by State Library staff and in the collection of impact evaluation data.
Develop and distribute random dated logs for State Library Staff to complete regarding their day-to-day activities in support of the outcomes of this program.
Logs in place by March 15, 2001 and activity completed by April 30, 2001.
Develop surveys tailored to clusters of grant program types based on the information gathered through the tasks described above and clarified through the interviews.
Survey instrument(s) complete and piloted by April 30, 2001.
Develop surveys for library and library systems staff as the users of statewide services such as, Empirelink and Inter-library Loan, based on the information collected through the tasks described above.
Survey instrument(s) complete and piloted by April 30, 2001.
Product: formal written Interim Report delivered April 30, 2001.
May 1, 2001 June 30, 2001
Analyze State Library Staff log data regarding day-to-day activities.
Mail-out surveys. Input returned surveys, analyze and interpret data.
Conduct four regional focus groups and/or key informant group interviews to help in the interpretation of the interview and survey data. In this part of the evaluation CDA Corp. will collect recommendations for future data collection, identification of appropriate outcome measures, discussion of a rubric for measurement of collaboration in libraries and library based programs and statewide service, and other information necessitated by prior evaluative activities.
Product: Rubric for measurement of collaboration in libraries and library based programs and statewide service.
Product: Program Logic Model formalized with input from the field regarding Outcomes, Indicators, Implementation Objectives and Supporting Activities and Resources.
In early June, begin the formal process of developing ongoing evaluation strategy for use in this program in the future.
July 1, 2001 October 31, 2001
Continue development of ongoing evaluation strategy for use in this program.
Product: Recommendations of evaluation methodologies that can be used routinely to collect follow-up information about LSTA funded programs; a menu of qualitative and quantitative choices for each project category identified in this evaluation study; recommendations for improving the LSTA program structure to enhance evaluation and program results.
Use telephone interviews, short written surveys, inclusion of questions in focus groups and/or key informant discussions, site visits to collect information from library clients, and targeted review of existing data, the evaluation will begin to construct impact evaluation data using a composite case design delineated by type of grant program identified in the program categorization portion of the study.
Compile representative cases by grant program type using the information collected through the analysis of grantee reports and this collection of data.
Final analysis of evaluative information collected in this evaluation will be presented in summary form in a formal written evaluation report. There will be a presentation meeting where data will be presented in table, chart and figures with discussion.
Product: Draft Final Report delivered for comment August 31, 2001. Comments returned September 30, 2001 and Final Report delivered October 31, 2001. The Final Report will be delivered in both camera-ready form and fifteen bound copies.
Appendix II: Evaluation Instruments
Questions for Initial Interviews; LSTA Evaluation 2001
The evaluation of the LSTA in New York State includes the measurement of present practices to establish the impacts that those practices have had during the past three years and a description of how the grant's management might look in the future. In order to complete both of these the evaluators will do four things:
- Conduct initial interviews with a representative sample of library and library system staff who are identified by the New York State Library staff as key informants.
- Collect information during interviews with Reference Library and Library Development staff at the State Level.
- Collect information from a broad base of library and library system staff through a survey developed based on (2) and (3) above.
- Present the data from the survey to a second set of key informants to elicit their expert response to the findings of the evaluators.
The evaluators will prepare a final report on the LSTA in New York for submission to Washington later this year. In addition, the evaluators will provide planning information to the State Library for use in the development of their next Three Year Plan for inclusion in their application for LSTA funds in the next funding cycle.
Areas of Questions
Based on meetings with State Library staff, review by a State Library consultant and review by the LSTA Advisory Board Evaluation Committee, the initial interviews of key stakeholders in the LSTA grant management and operation in New York are purposely broad. There are five areas of questions which the evaluator will address. They are:
- The role of the LSTA in your system's work. A brief overview of the use of LSTA funds in your system.
- The LSTA and innovation and change in your context. Sometimes we cover this under (1). LSTA is supposed to provide seed money for innovation with the purpose of initiating systemic change. We are interested to know how well you think the program actually achieves that purpose.
- The cross-system type relationships in your context. The LSTA now provides funds to all library types in the State. Are there any existing cross-type collaborations in your location? If so, do they work?
- The role of the State Library in reference to the LSTA funds and their use in your system. Technical assistance is always an important part of systemic change. Other roles that the State Library has, such as organization of advocacy; information dissemination; and communication across library types are also important. What should the role be and what outcomes of that role would you expect?
- Things it might be an idea to change. The new plan and the re-application for LSTA funds that will follow it are opportunities to change some existing policies and practices. Any thoughts on the future and how the LSTA can be managed to support that future vision and mission would be very helpful to us.
As is always the case in evaluation work, all information is confidential. Reports to the State Library and to other stakeholders will be presented without the identification of the informants and with patterns of response rather than individual responses provided. Any questions, please telephone Dr. Kate Toms at 518-238-0968.
Questions for the School Library Directors
DDIP Evaluation Questions
Further to our meeting on January 26, I have just gone over my notes and the original questions and propose the following sequence of questions for the School Library Director focus forums in March.
1. We are interested in the context in which this type of funding support yields good return on investment for all concerned. LSTA invests money and you and your systems invest both time and money to achieve the goals of this program. I would like to begin by asking you to give me some background on why this funding was attractive to you.
Probes
- Did your schools think it was important to automate?
- Those that did automate, were the converted records added to the union catalog, regional catalog or both?
- Did they think it was important to automate their own record keeping with, say, circulation workstations or the purchase of management-type software?
2. If you had to pick the one thing about this program that stands out in your mind as positive, as a strength or as something that went really well, what would it be?
Probes
- Probe if the positives were contextual or constructed.
- Probe about technical assistance, form and function of it and person(s) providing it.
3. Every innovation has its barriers, can you talk for a minute about the barriers that you experienced when implementing the activities under this program? When you answer, can you also refer to strategies to remove, overcome or circumvent any barriers.
Probes
4. The LSTA DDIP is an incentive program. Once started, will this work continue?
- Have primary clients (district/building administrators) seen the value of this?
- Are they prepared to support this work from their own funds?
- Has there been any long range effect on the library program: staffing, budget, other technology added, usage statistics, PR impact on school and the community, communication and reporting in building and district? Define "long range effect".
Survey of Library System Directors in New York State
Section I: Experience with New York State Library Statewide Services
This section of the survey asks for your opinion as the director of your system of the effect of the Statewide Services provided by the New York State Library in areas supported by the Library Services and Technology Act since 1998.
- In which of the following areas does your system contact the New York State Research Library for help? Please
check
all that
apply.
Area
Level of Satisfaction with Research Library Services
Information on special services for the visually impaired. 
Very Somewhat Not at all satisfied.Reference information.
Very Somewhat Not at all satisfied.Access to special interest collections.
Very Somewhat Not at all satisfied.Electronic Inter-library loan queries.
Very Somewhat Not at all satisfied.Other (please specify) ________________
Very Somewhat Not at all satisfied. - In which of the following areas does your system contact the New York State Librarys Division of Library
Development (DLD) for help? Please check all that apply.
Area
Level of Satisfaction with DLD Services.
Technical assistance with new funding sources, e.g., Gates Foundation, E-Rate, etc.
Very Somewhat Not at all satisfied.Technical assistance with competitive grants.
Very Somewhat Not at all satisfied.Statewide reading programs.
Very Somewhat Not at all satisfied.Technical assistance with training services.
Very Somewhat Not at all satisfied.Technical assistance with State aid.
Very Somewhat Not at all satisfied.Technical assistance regarding charters.
Very Somewhat Not at all satisfied.Technical assistance with construction.
Very Somewhat Not at all satisfied.Consulting on State initiatives.
Very Somewhat Not at all satisfied.Obtaining cost-free access to full-text electronic databases
Very Somewhat Not at all satisfied.Technical assistance with system member library concerns.
Very Somewhat Not at all satisfied.Information about New Yorks libraries and library services.
Very Somewhat Not at all satisfied.Information for planning and advocacy.
Very Somewhat Not at all satisfied.The EmpireLink Help Desk.
Very Somewhat Not at all satisfied.Other (please specify) _______________
Very Somewhat Not at all satisfied. - Looking back over your answers to Questions 1 and 2, please complete this statement by choosing from the
following three choices. Check all that apply.
The New York Statewide Services provided by the New York State Library since 1998 have:
Helped the libraries in our system to
accommodate the changing and shifting needs for library services Helped to support system-wide strategies
which will stimulate change and/or transform services in our system The New York Statewide Services have
not really been helpful to us in either of these areas. - Has your system applied for LSTA Grant support in the past five years?
No. Please tell us why you have not
applied for an LSTA grant.
___________________________________________________________________
___________________________________________________________________
Thank you. Please go to Section III, page 5.
Use this space for any additional comments on Statewide Services.
Section II: Impact of Library Services and Technology Act (LSTA) Grant Support on Your System
This section of the survey asks for your opinion as the director of your system of the effect of LSTA Grant funded activities in your system in areas supported by the Library Services and Technology Act.
- One of the purposes of LSTA Grants is to support demonstration projects around the State that others might want to consider replicating in their systems. We are interested in knowing how you hear about LSTA supported work in other library systems and if you believe that you are given enough information about these activities to judge if they might be worth replicating or modifying for use in your context.
In the grid below we list a number of communication means through which you might receive this information, and the quality of the information received through this means. Please check all that apply.
I hear about successful New York State LSTA funded projects: The information I get from this source is usually:
______________________________________
______________________________________
______________________________________2. Has your system used LSTA grant funds to support strategies which will stimulate change in professional practice among librarians in your system?
Professional development area:
Of the support needed, LSTA Grant funds provided a:
If you have used LSTA grant support in strategies that target changes in librarian professional practice other than those listed above, please tell us how.
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
3.Has your system used LSTA grant funds to support strategies which will change and/or transform services in areas other than professional practice?
Program Area
Of the support needed, LSTA Grant funds provided a:
Training in Internet use, digitization, and new and diverse technologies. If you have used LSTA grant funds to support strategies which will change and/or transform services in areas other than professional practice of the librarians in your system which are not listed above, please tell us how.
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
4. Please complete this statement by choosing from the following three choices. Check
Grants from the Library Services and Technology Act (LSTA) since 1998 have:
5. Please choose any of the following which present a barrier to the use of LSTA grant funds by your system.
6. Has your system used LSTA support to leverage funding from another source?
Use this area for additional comments on LSTA Grants
Section III: System Planning and Collaboration
This section of the survey asks for information about any relationship between LSTA supported activities and system planning activities. In addition, we ask about collaborations which your system is part of, both with library system types other than your own and with agencies outside of the library community. The questions in this section are meant to give the evaluators some understanding of the context in which the LSTA funded activities are taking place.
- Has the planning process to develop your new Plan for Service affected your use of LSTA support in your
system? No. Please go to question 2.
Yes. Please tell us how.
_____________________________________________________________________________
_____________________________________________________________________________
_____________________________________________________________________________
- In which of the following have New York Statewide Services helped in general planning in your system? Please
check all that
apply.
- By coordinating strategic planning
at a statewide level, e.g., Regents Commission on Library Services, NOVEL Planning Team, Third Statewide Automation
Plan.
By communicating a vision for services
in New York.
By communicating information about proposed
legislation of interest to libraries and library systems.
By organizing a response to public policy
issues.
By providing information for advocates
for library and library system related issues.
Other (please specify) _____________________________________________________________
- As part of your planning process, do you do a needs assessment to determine library related issues and to
plan for addressing those issues? No. Please
go to question 4.
Yes.
Is this assessment done in collaboration with agencies or organizations other than libraries?
No. Please go to question 4. Yes. Please tell us who they are. ___________________________________________________________________________________________________________________
- Please complete the following sentences:
In addition to the libraries that are members of your system:
- Our system collaborates with school library systems in our region and/or outside our
region.
- Our system collaborates with public library systems in our region and/or outside our
region.
- Our system collaborates with reference and research library systems in our region and/or
outside our region.
- Our system collaborates with school library systems
- Is your system part of any partnerships or collaborations outside the library community? No. Go to Section IV.
Networking is important. Shared funding applications.
Economy of scale to address local needs.
We have/had a particular issue to address.
It is an opportunity to further the
librarys mission.
Other (please specify) _______________________________________________________
Please list your partners and collaborators:
____________________________________________________________________________
____________________________________________________________________________
Use this space for additional comments on Planning and Collaboration.
Section IV: A First Look Forward
This section of the survey asks for your opinion regarding some issues that are presently circulating in the library community in New York which could influence the LSTA Five-Year Plan (2002 -2007) which the New York State Library will have to prepare for continued funding. The important thing for this first survey to address is a measure of the magnitude of the issue, not the value of specific points within any debates.
- In your context, is the resolution of telecommunications development issues important to future technology
development?
Yes, very important. Yes, somewhat important. No, not very important.
- Do you think it will be important to develop methods for tracking public use of electronic library resources
in the future?
Yes, and we have already developed them.
Yes, and we are starting to develop them. Yes, and we need help in developing
them. No, existing systems for tracking use
are sufficient.
- In the past twelve months, has your system sought expert advice in any of the following areas?
Computer system development. Computer software development. Telecommunications. Published computer software support.
Other related technology area: _________________________
If you selected any of these areas, where did the experts come from?
A consulting company. From one of the libraries in your system. In-house Other (please specify) _________________________________________________ Other (please specify) _________________________________________________ - What portion of the libraries in your system have had as much of their bibliographic records as necessary
converted?
_________ out of ________ libraries have been converted.
Once libraries are converted, how are they linked? Please check
all that apply. To a state system. To a regional catalog. To the system catalog.
To each other. - Please indicate your level of agreement with the following statements by choosing from the scale in the
right hand column below.
In order to deliver high quality electronic information access to all New Yorkers, our system will need access to: Strongly Agree
Agree
Disagree
Strongly Disagree
a. Unique and historic documents resident in academic, research and special libraries.
b. Unique and historic documents resident in public libraries and library systems.
c. Unique and historic documents resident in school libraries.
d. Academic, research and special libraries bibliographic records.
e. Public libraries bibliographic records.
f. School library systems bibliographic records.
- Is it important to link the libraries in your system to educational, social or information services outside
of the library community? No. Please go to
question 7.
Yes. Are any of the libraries in your
system linked to educational, social, or information services outside of the library community? No. Please go to question 6. Yes, they all are. Yes, some are. Yes, but very few are.
Please list the services that they are linked to: ___________________________________________
__________________________________________________________________________________
- Please list your systems level of present need in the following areas:
Professional Development on: High Need
Moderate Need
Low Need
No
Needa. How to access information on the Internet.
b. Helping the public to access information on the Internet.
c. How to use office-based computer software.
d. How to use collection management software.
e. Targeting changes in library practice.
f. Area not listed (please specify) _________________________
- Things have changed in New Yorks libraries during the past five years. Where would you say support will be needed most during the next five years? Please review the following list of areas for library development and indicate which four of these areas will need the most support during the next five years. If we have not chosen the areas that you believe are most important, please provide them in the Other spaces provided.
The four most important areas that will need support in the next five years are:
- __________________________________________
- __________________________________________
- __________________________________________
- __________________________________________
Access to commercial databases.
Shared electronic catalog development.
Digitizing of full text resources.
Improvement of telecommunications access.
Adult literacy services.
Family literacy services.
Services to business/entrepreneurs.
Services to job seekers and career changers.
Services to new populations.
Please specify which new populations you have in mind:
Other (please specify) _________________________________________
Other (please specify) _________________________________________
Other (please specify) _________________________________________
Other (please specify) _________________________________________
Use this space for additional comments on looking forward.
Please use this space for additional information.
Thank you for taking the time to complete this survey.
Complete and return to: Dr. Kathleen Toms, CDA Corp., 21 Page Avenue, Third Floor, Cohoes, NY 12047
LSTA Focus Forum Question to Start
Are you a library system or a system member library?
Library System System member library Other _______________
1. Looking back over the past five years, what are the three biggest changes you see in the libraries in your system?
- ______________________________
- ______________________________
- ______________________________
2. What are the two most important things that you do with LSTA support?
- ______________________________
- ______________________________
3. If you were given complete freedom to choose how you would report on the impacts of LSTA supported projects, what would you consider the most important things to report?
- ______________________________
- ______________________________
- ______________________________
4. Are they are particular barriers you have to confront when considering the use of LSTA funds in your professional context?
5. Please use this space (and the back of this page if you need it) for any information you particularly want to make sure we include in the data from today.
THANK YOU
Survey of LSTA Grant Program Directors in New York State
Section I: Your Library Services and Technology Act (LSTA) Grant
This section of the survey asks you about your experience with the LSTA grant which you have managed between 1998 and the present.
- 1. In which of the following areas have you managed LSTA grants for your present system?
Services for Individuals:
Economic Opportunity
Services for Business Electronic Content Adult Literacy Family Literacy Technology Training SLS Database Development- 2. Choose the response to the questions below that best characterizes your experience with the programs and activities funded through the LSTA grants which you have managed using the scale in the right hand column.
Strongly Agree
Agree
Disagree
Strongly Disagree
The work that we start with LSTA grant support tends to be continued after the funding ends.
Our system has learned a great deal through the LSTA supported opportunity to innovate.
I would say that LSTA grant support is worth the effort .
LSTA grant application rules and procedures are clear and easy to follow.
LSTA grant management procedures (paperwork, submitting reports, etc.) are clearly communicated by the New York State Library.
3. Please choose any of the following which present a barrier to the use of LSTA grant funds by your system. Inability to use funds to purchase commercially
available databases. Inability to use funds to pay current
staff to do grant supported work. Required matches for grant funds. Reporting forms not appropriate to system
type. Limited grant categories. Other (please specify) _________________________________________________________________- 4. There is an evaluation requirement for all LSTA Grants. However, the reporting date for that evaluative information can be too early for the director to judge if the activities funded under the grant have had an impact, or if the grant supported activities will be merged with other initiatives in their system. Did you feel that it would have been better to report on the success of your grant supported program and/or activities three to six months following the end of the grant period?	r No.
- 2. Choose the response to the questions below that best characterizes your experience with the programs and activities funded through the LSTA grants which you have managed using the scale in the right hand column.
_____________________________________________________________
If possible, please attach any examples of further information that you now have available.
Section II: Impact of Library Services and Technology Act (LSTA) Grant Support on Your System
This section of the survey asks for your opinion as the director of an LSTA grant of the effect of LSTA Grant funded activities in your system in areas supported by the Library Services and Technology Act.
- 5. Has your system used LSTA grant funds to support strategies which will stimulate change in professional
practice among librarians in your system?
No, please go to question 6.
Yes. If any of these strategies included
professional development, please indicate which of the following professional development areas LSTA grant(s) helped
your system to address and the level of support LSTA funds provided to the professional development area overall.
Please check all
that apply.
Professional development area: Of the support needed, LSTA Grant funds provided a: Professional development on how to access
information on the Internet. Significant Moderate Very small amount. Professional development on how to help
the public to access information on the Internet. Significant Moderate Very small amount. Professional development on how to use
office-based software. Significant Moderate Very small amount. Professional development on how to use
electronic means to track electronic usage. Significant Moderate Very small amount.If you have used LSTA grant support in strategies that target changes in librarian professional practice other than those listed above, please tell us how.
___________________________________________________________________________
___________________________________________________________________________
___________________________________________________________________________
- 6. Has your system used LSTA grant funds to support strategies which will change and/or transform services in areas other than professional practice?
No, please go to question 7.
Yes. Please indicate in which of the
following program areas your system has used LSTA grant support to enhance or expand programs by indicating the
level of support provided by LSTA grant funds to the program area overall. Please check all that apply.
If you have used LSTA grant funds to support strategies which will change and/or transform services in areas other than professional practice of the librarians in your system which are not listed above, please tell us how.Program Area Of the support needed, LSTA Grant funds provided a: Services to job seekers and career changers. Significant Moderate Very small amount. Services to entrepreneurs. Significant Moderate Very small amount. Adult literacy programs. Significant Moderate Very small amount. Family literacy programs. Significant Moderate Very small amount. Digitizing local history documents. Significant Moderate Very small amount. Development of new technologies for distance
learning. Significant Moderate Very small amount. Training in Internet use, digitization,
and new and diverse technologies. Significant Moderate Very small amount._________________________________________________________________________
_________________________________________________________________________
_________________________________________________________________________
- 7. Please complete this statement by choosing from the following three choices. Check
all that apply.
Grants from the Library Services and Technology Act (LSTA) since 1998 have:
Helped the libraries in our system to
accommodate the changing and shifting needs for library services Helped to support system-wide strategies
which will stimulate change and/or transform services in our system LSTA grants have not really been helpful
to us in either of these areas.- 8. In which of the following areas does your system contact the New York State Librarys Division of Library Development (DLD) for help? Please check
all that apply.
9. Looking back over your answers to Question 8, please complete this statement by choosing from the following three choices. Check
all that apply.The New York Statewide Services provided by the New York State Library since 1998 have:
Helped the libraries in our system to
accommodate the changing and shifting needs for library services Helped to support system-wide strategies
which will stimulate change and/or transform services in our system The New York Statewide Services have
not really been helpful to us in either of these areas.10.Has your system used LSTA support to leverage funding from another source?
No. Go to question 11. Yes. Please list those sources. _______________________________________________ - 6. Has your system used LSTA grant funds to support strategies which will change and/or transform services in areas other than professional practice?
11. One of the purposes of LSTA Grants is to support demonstration projects around the State that others might want to consider replicating in their systems. We are interested in knowing how you hear about LSTA supported work in other library systems and if you believe that you are given enough information about these activities to judge if they might be worth replicating or modifying for use in your context.
In the grid below we list a number of communication means through which you might receive this information, and the quality of the information received through this means. Please check all that apply.
I hear about successful New York State LSTA funded projects: The information I get from this source is usually:
_________________________
Use this area for additional comments on New York Statewide Services and/or LSTA Grants
Section III: System Planning and Collaboration
This section of the survey asks for information about any relationship between LSTA supported activities and system planning activities. In addition, we ask about collaborations which your system is part of, both with library system types other than your own and with agencies outside of the library community. The questions in this section are meant to give the evaluators some understanding of the context in which the LSTA funded activities are taking place.
- As part of your planning process, do you do a needs assessment to determine library related issues and to
plan for addressing those issues? No. Please
go to question 2.
Yes.
Is this assessment done in collaboration with agencies or organizations other than libraries?
No. Please go to question 2. Yes. Please tell us who they are. __________________________________________ - Please complete the following sentences:
In addition to the libraries that are members of your system:
- Our system collaborates with school library systems r in our region and/or outside our region.
- Our system collaborates with public library systems in our region and/or outside our
region.
- Our system collaborates with reference and research library systems in our region and/or
outside our region.
- Our system collaborates with school library systems r in our region and/or
- Is your system part of any partnerships or collaborations outside the library community? No. Go to Section III.
Networking is important. Shared funding applications.
Economy of scale to address local needs.
We have/had a particular issue to address.
It is an opportunity to further the
librarys mission.
Other (please specify) _______________________________________________________
Please list your partners and collaborators:
____________________________________________________________________________
____________________________________________________________________________
Use this space for additional comments on Planning and Collaboration.
Section IV: A First Look Forward
This section of the survey asks for your opinion regarding some issues that are presently circulating in the library community in New York which could influence the new LSTA Five-Year Plan (2002-2007) which the New York State Library will have to prepare for continued funding. The important thing for this first survey to address is a measure of the magnitude of the issue, not the value of specific points within any debates.
- Please list your systems level of present need in the following areas:
Professional Development on: High Need
Moderate Need
Low Need
No Need
a. How to access information on the Internet.
b. Helping the public to access information on the Internet.
c. How to use office-based computer software.
d. How to use collection management software.
e. Targeting changes in library practice.
f. Area not listed (please specify) _____________
- Things have changed in New Yorks libraries during the past five years. Where would you say support will be needed most during the next five years? Please review the following list of areas for library development and indicate which four of these areas will need the most support during the next five years. If we have not chosen the areas that you believe are most important, please provide them in the Other spaces provided.
The four most important areas that will need support in the next five years are:
- __________________________________________
- __________________________________________
- __________________________________________
- __________________________________________
Access to commercial databases. Shared electronic catalog development.
Digitizing of full text resources. Improvement of telecommunications access.
Adult literacy services. Family literacy services.
Services to business/entrepreneurs. Services to job seekers and career changers.
Services to new populations.
Please specify which new populations you have in mind:
Other (please specify) _________________________________________
Other (please specify) _________________________________________
Other (please specify) _________________________________________
Other (please specify) _________________________________________
Thank you for taking the time to complete this survey.
Complete and return to: Dr. Kathleen Toms, CDA Corp., 21 Page Avenue, Third Floor, Cohoes, NY 12047
Appendix III
Background to Statewide Automation and Electronic Doorway Libraries
The statewide automation plan for libraries in New York began with the publication of a two part plan. In 1987 Libraries & Technology: A Strategic Plan for the Use of Advanced Technologies for Library Resource Sharing in New York State was published. In 1989 Technology & Access: The Electronic Doorway Library was issued as the operational part of that plan. The second edition of the Statewide Automation Plan for Libraries, The Electronic Doorway Library: Meeting the Information Needs of the People of New York State, was issued in 1993, with a second printing in 1994. As stated in the Introduction to that second edition:
Statewide library automation is substantially advanced through the Regional Bibliographic Data Bases (RBDBs) and Interlibrary Resources Sharing Program and the Library Services and Construction Act. Funding from these programs has facilitated the evolution of automation in New York State libraries from being limited and disparate to being more widespread and part of a coordinated statewide effort to make electronic services routinely availableThe original plan introduced the concept of the electronic doorway library as a way to explain this change in the delivery of library services. (p. 1)
The third in the series of statewide library technology plans issued during the eleven years between 1987 and 1998 was Doorways to Information in the 21st Century: Every New York Library an Electronic Doorway Library. This final edition of the statewide library technology plans provided New Yorkers with a statewide plan for technology-based library services for the years 1998-2000.
The definition of an electronic doorway library was stated in the third plan as:
- Create, assemble, evaluate and use information;
- Extend access to library services from homes, schools, work places and other locations;
- Facilitate access by people with disabilities and other special needs; and
- Go beyond the librarys walls to obtain information and resources.
An integral part of the statewide electronic learning community, which uses computers and telecommunications technology, a full range of library resources, and the services of skilled library personnel to:
Electronic doorway libraries meet the ongoing library and information needs of education, government, business, and all people regardless of age, background and location.
Go to Library Development home page