LSTA: Evaluation of NOVEL, Appendix B

Evaluation of New York's Library Services and Technology Act

Evaluation of the New York Online Virtual Electronic Library (NOVEL)

Himmel & Wilson, Library Consultants
February 18, 2007

This document also available in .PDF


Executive Summary

The Evaluation


The Evaluation Process


  1. Focus Group Sites
  2. Web Survey Respondents

Summary of Focus Groups -- Librarian and End-User Sessions

Summary of Interviews with Library Professionals

Summary of Interviews with End-Users

Summary of Web Survey Results


Maps 3-12: NOVEL Usage 2005

  1. Entire State
  2. Western New York
  3. Finger Lakes
  4. Rochester, Syracuse, Watertown Area
  5. Adirondack Region
  6. Capital Region
  7. Catskill Region
  8. Greater New York City area
  9. New York City
  10. Long Island

Responses to Specific Questions Raised in the Request for Proposal for the NOVEL Evaluation


Appendices -- NOVEL Evaluation --

  1. Focus Group Report -- Librarian Sessions
  2. Focus Group Report -- End-User Sessions
  3. Interviews with Library Professionals
  4. Web Survey Report

Appendix B: NOVEL Evaluation - Focus Group Report - End-User Sessions

Nine focus group sessions with users were scheduled for July 10 to July 18th.  A total of 29 people took part in the sessions, which were held in Batavia, Canastota, Farmingdale, Fredonia, Highland, Ithaca, New York City, Saratoga Springs and Watertown.

Participants included homemakers, retirees, library trustees, writers, library volunteers, a church pastor, an independent living advocate, college students, foreign-born immigrants to the U.S., a psychologist, and people who said they were making career changes and were using the electronic resources to upgrade their information and skills.

Participants in each of the sessions discussed the same general questions although the discussions varied a great deal depending upon their interests and experiences.  Generally they discussed which databases they used or what information they sought during their last use of databases, what was good about databases, what training they had had in using databases, what sorts of difficulties they had, and how the NOVEL databases could be marketed to increase use.  They were also asked why some people might choose not to use the databases and what other/additional subjects or databases should be included in the program.  Several also shared personal stories about using the information from databases in their personal lives.


  • EBSCO, Gale, NOVEL, and “databases,” are all terms that don’t have much meaning to people using the resources.
  • People are looking for a seamless way to get into the databases; they want to search quality resources more “like Google.”
  • Participants described their difficulties in searching.  Difficulties seemed to be related to differences in the screens from different access points and different searching strategies required.  They don’t like having to key in their ID number each time they move to a different database.
  • Databases they’d like to have available include Heritage Quest, a good basic encyclopedia, specific periodicals, and test preparation materials.
  • Users often said they got their training in using databases and software at the library; however, many seemed to be largely self taught as well; they “just keep trying different things, trial and error, until they find what they’re looking for.”
  • What’s great about the databases is that they’re “free!”
  • The participants (who probably do not reflect all users) did know that databases were different from the Internet; databases are “juried” or selected and therefore more trustworthy.  They’re “better quality” than the Internet.
  • Their recommendations were to make searching more user friendly; do training by topic/subject; and make the databases more accessible to people with disabilities.
  • They think librarians are missing a great PR opportunity with the databases.  A PR campaign needs to go beyond the libraries to bring new users in.  One lady said, “Publicity about it (NOVEL) is severely lacking!”

Compilations of the responses follow.

Which databases did you use the last time you searched a database? What subjects/topics were you looking for?  (Note a number in parenthesis indicates the number of times the specific answer was given.) 
MasterFILE Select  (3)
Newspaper databases  (2)
Health databases  (2)
New York Times
Periodicals that I don’t have time to read at the library.

Magazines and health articles
ProQuest and EBSCO
My kids use Searchasaurus, primarily for schoolwork
Literature and arts, mostly school related
HeritageQuest and the newspaper indexes more for personal interests 

I wanted to see what journals would be there in Psychology, my college major.  There were some in the medical databases and in MasterFILE.
I use MasterFILE Premier most of the time.  It has thousands of periodicals like New Yorker, Better Homes, and Gardens, etc.
Letter writing, children’s books for South America
Article on high school drop out rates
Consumer Reports

Steamship on the Hudson, the Mary Powell
Glacial advance in the Mid-Hudson area
Long Island Sound
Ethnic diversity
Literacy data

What’s good about the databases?
It’s kind of like subscribing to all these newspapers and magazines, but they’re free.
For business, small businesses, it’s a tremendous resource.
I like the images that my kids can get.  There’s nothing wrong with Google, but NOVEL has great historic images.
One of the other things is that all of the information you find is only as good as the publisher.  We don’t have to doubt what’s on the database.
The databases are pretty neat compared to the Internet.

I’ve been wanting to share how useful and great the databases are; they should be promoted with workshops and hands on things.
A lot of the Internet is just people’s opinions.  Databases don’t have as much of the opinion stuff.  Sometimes you put in your credit card number when you could have found the same thing for free.
They’ve opened me to the existence of many periodicals I didn’t know about; I’ve discovered environmental publications I hadn’t known about, things on climate change, for example.  I used to know about the books of a particular writer, but now I can also keep track of how he’s developing his ideas and thoughts via periodicals.
I find them easy to navigate; the interface is very user friendly.  Remote access is relatively easy as well.  They should be promoted as “things you can do in your jammies…”  I’ve found almost everything I’ve been looking for.  My husband is a journalist and he uses them as well.

I had been looking for a specific article and had only some names from a magazine cover; librarians pointed me in the right direction; I’d looked on Internet, card catalog, microfilm, and then they pointed me to the databases.  They had a printed copy of it within 3 minutes—full text of the article!
There’s too much information available; people get lost, so the databases are good because they structure what’s available and help you get information.
Health is one of hardest (electronic information sources) to choose. It’s important that the selections are made by some reputable organization.

What training have you had in learning how to use the databases?
Classes at the library  (4)
Librarian showed me  (4)
Self-taught (2)
They do have some written things as well.
I often end up at the desk asking questions.

I just keep testing and trying, looking for the right words and the most effective search strategies.
We all do it ourselves; trustees do get a tour of libraries as a part of their orientation and that includes exposure to databases.  I couldn’t believe I could get the New York Times free!
I got a little training, but the use of Boolean logic makes it tedious; Google’s easier.
But none of the Google things brings me last week’s Times’ article!

Did you encounter any difficulties in using the databases?
EBSCO doesn’t mean anything to anybody.  People learn, but you have to decide which one you want to use, Gale or EBSCO.  Maybe someone in the advertising world could help.
People have to have some working knowledge before they use databases.
People say being able to use a search engine that looks at multiple databases would be very helpful; but the transition from being a newbie to experienced is difficult.
They should have software that provides some choices and alternatives to guide people.
The first time I used it, it said that my driver’s license was invalid although my license is OK.  I got in the second time, but I had to enter it a third time to get what I wanted.

Occasionally I try to go back to something that I’ve been to before and I can’t go back.
When I’m calling up magazines and journals at home, if I click on the one on the left it wants a user name and password on the right.
I like to read science journals.  It would be nice to be able to find the whole issue.
I think people are looking for a more seamless way to get into it.  The databases look different and feel different.  But people don’t understand that.
I shared the site with a consumer who is blind; it would be good if it was friendlier for JAWS.  They can’t read as well because of the graphics.

When I took a class, it seemed that there were a lot of other things that you could do that were hidden when I tried to use the database on my own.
Make them more user friendly, more intuitive.  Even if they gave you a sample search to show you how you should go about doing it; that would be helpful.
I like going to specific databases, but at the library students like being able to do federated searches.
I think you could sell the meta engine too because it would increase the use of databases.

The problem in going from one database to another is that the vendors own them.  Their concerns are economic.  There’s the issue too of who gets credit for using a database.  Right here there are multiple turfs conflicting.  So there’s a lack of motivation for creating the tool to get into things more easily.

How would you recommend NOVEL be marketed to increase use?
If people could search it more like Google, with one place where you typed in what you were looking for, that would be great.  And probably more people would use it.
Make it more user-friendly.  There are still lots of steps to take.   Make it more Google like in the first place. 
Along the same lines, you get to it lots of different ways.  It’s different getting into NOVEL in different places.   If you’re trying to tell somebody how to use it, it might be better if getting in was the same everywhere.
Do lots of publicity in lots of different ways.  
Should do lots of hands-on workshops for people, teachers especially
If a tutorial was set up, people could take it on their own.  Do TV ads and online tutorials.
Tell the students that it will help them with their term papers.

Make it clear what NOVEL is versus other things that are available in libraries.   There’s confusion about why you wouldn’t get everything in the library in NOVEL.
NOVEL has a lot going for it in terms of having lots of topics.  You could do a lot of topical training.  Car repair, vehicle repair – that would be a good one.   Chilton’s manuals…
The libraries in general are missing a great PR opportunity.
I wish that more public knew more about it.
I want to go back to the user friendly idea, user friendly and marketing.  Those are the big points that need to be made.

Libraries should partner more on the training aspect.   Do training by topic.
The idea of publicizing is important. It’s one-stop shopping.  An adult can use it; a child can use it.  There’s a variety of resources in the databases.
Any marketing efforts need to extend beyond libraries if we want to bring in new users.   People who use their libraries may be introduced to the databases, but lots of others are being missed.
The first question that I had was what’s NOVEL?  Publicity about it is severely lacking.  
I work with lots of people who can’t get to the library physically and this could be their access to information.
I’d list the magazines and stress the popular aspects of them.

Call NOVEL something else.
The name thing for sure needs attention.  I love the database NOVELIST, but it’s confusing.  
I sent out a memo to my Friends’ group about NOVEL.  Several of them thought it was a database of novels.  If people really understood what it was, they’d use it more. 
Some people have confused NOVEL with NOVELL.
Or Novelist
I hadn’t even recognized NOVEL meant something.
Have an advertising agency to come up with something that identifies it more with the databases.

Initially the word database is no help, unless you know it stands for something.
I think it leads people in the wrong direction.
Needs to be called something else; that’s too cumbersome.
People aren’t as used to databases as to magazines; they think databases means groups of numbers, etc.
Name is just one issue; the complexity of them is also an issue.  In your mind and thinking, any search at the beginning is pretty diverse.
NOVEL isn’t a useful label either.

Please distribute information through the Independent Learning Centers. 
Libraries should target students, especially when they need peer-reviewed journal articles.  From what I was searching, I would like to see more specific sub-categories, especially if you were able to align (subjects) with college majors.
Tie into more community groups.  The schools are a captive audience.  The general public is harder to reach.  Could there be grants or money that could go around to the public libraries for marketing?
Do mailings.

You need professionally produced commercials.
You do have to saturate the market.  It can’t be a one time event or ad.  Has to be a campaign.
People want to know about the newest technologies.  Promote the databases at conferences and conventions.
Emphasize marketing and publicizing the databases.  I think that there are a lot of people out there who would benefit if we could just get them trained.  Reach them through businesses and community groups.
I think that they need to market to specific targets.  Did the school libraries say the kids are using the databases?
Maybe webcasts would be more effective.

I wonder if down the road NOVEL could be expanded into a New York State Google, a one-stop shopping site?  Not only would it be useful, but it would lead people to the databases.
Right now NOVEL could pay to be a sponsored link
Publicize…and put a button on the NOVEL webpage soliciting feedback.
Everybody talks about PR; libraries should do workshops.
You have to promote the databases in a way that people can hear.  Too often it doesn’t register.

Why do you think some people might choose not to use the databases?
The fact that the screen looks different – first thing the librarian says is “the screen is going to look different for you at home.”  There should be a similar front end wherever you are.
Possibly the name “database” may turn off some people.
How do you make me aware of it when I think that I don’t have time?
You have to know what you’re looking for.
I would reiterate the ease of use thought. I would think that if you want to bring in more people, you’d want to make it easier to use.

Maybe if searching the databases was more like Google or Yahoo….  Some type of broader search would help people feel more confident about searching? People are familiar with using Google or Yahoo. 
They might think it’s too hard; they haven’t been taught how to search or how to put together the best search strategy.  They might think it’s too technical.
We don’t do a good job in schools in teaching people how to find information; people don’t know the logic of searching.
Databases need to be searchable like Google; there’s another level with databases and it’s very frustrating.
It’s easier to use Google.  People don’t understand that they can get higher quality information through databases.

They should have everything on one page, just one place to get in and not have to go back to the top to enter your card number every time.  You should be able to put in your number and then click on the icon and go to what you want.  It should be straightforward instead of having to put in your number again.
That page isn’t the same all over the state.
Sometimes some jargon comes up that I don’t get.  Who’s the target user with all this?
Unless you’re a regular user, you don’t know; how much promotion has been done?  They haven’t done publicity that speaks to the average person.
Schools aren’t as excited about using Google as they once were; they’ve learned that it’s not reliable.  If it’s in a database, an editor has reviewed it, “juried” it.
Too many people are satisfied with Google; they don’t understand you get what you pay for!  They need education!
People don’t use databases because they already have too much information.
It has to be as easy as Google, but more reliable information than Google.

To maximize the use of library resources, you have to push (educate on their value) and pull (make access easier)
I’d like to see the average person using them, so you have to eliminate the jargon.
People don’t use the databases because they lack publicity; labels like database and NOVEL aren’t helpful; you have to sign in (as opposed to Google); searching is different—you have to start broad with databases, but with Google it’s specific, so people have to think differently.
Maybe there is too much available; maybe simplify until people get used to using the basic databases.
I don’t like our own library’s web page; it’s not current enough. A lot of library web pages are pretty obscure to get into.   Many of them use a lot of obscure icons.

What other resources/topics should be available in this way?
Old newspapers
Test preparation - old Regents exams
Local history collections are big things in many libraries.  People are always searching for local history.
I would like to use be able to use Heritage Quest from home.
I don’t subscribe to Christian Century; I would like to have that.

College degrees – careers as topics
Testing software
You have to look at your target audience(s)    Are they trying to attract the general public?  If you’re trying to cover everyone, that’s hard.
A front end that had more general offerings would be helpful, audio and video offerings.
There are a lot of resources that are public domain.  Could those be available to the public?  Could they integrate that kind of thing?

More critical analysis rather than plot summaries
More of the newspapers and the ability to get more articles
Scholarly American History; archaeology journals; biology; you could just go through the disciplines!

I wish they had more publications about writing and books.  Things like Bookmarks (book reviews), Pages (for the average reader), Poets and Writers, Writer’s Digest.
MasterFILE Premium is general information; I’d like access to more scholarly and scientific things.  Wish we could have Academic Search Premier here.
Many people have access to scholarly things through their academic institutions, but independent writers and researchers don’t have those institutions.
I’d like to see more databases in foreign languages…Chinese, French, etc.
A few more databases that are more academic would be good; NOVEL still lacks those.

Other comments?
Many people aren’t aware of the databases.  When you explain they’re impressed and excited.
The general public doesn’t understand what a database is; they just Google.  The example I use with second graders: use World Book and say to people “this encyclopedia was put into the computer and you know it’s a good source.”
Sixteen year olds are very savvy about electronic resources; more so than the 20 year olds.
Marketing should be more general (include more) than NOVEL.  The school nurse went bonkers when she found out about the health resources available.

Perhaps public libraries could partner with the schools to use their labs for training.
We need an explanation of what a database is composed of on the webpage; also, if there’s a change, do some kind of transition for ‘old’ users to the new database or approach.
Each library tends to have its own way to present information.
NOVEL is a federally funded program, but most people don’t know the NOVEL name.

There’s so much information in databases, but maybe the State should allocate some money to help people learn to navigate –and how to do it.  How would I know about the different databases?  How many people are like me and are enthusiastic about it? 

Stories about how participants used the information they found in the NOVEL databases:
I was having a problem with an over the counter product, calcium citrate.  I switched products based on information I found in the database and it worked.

I recently wrote an article on playing poker and I didn’t have enough knowledge of magazines to submit the article.   I looked at list of databases, sent an email to a librarian online and within a few days they gave me a link to Ulrich’s.  It didn’t hit me immediately; I don’t quite understand how to use a database. You have to calm yourself down and remind yourself that you’re new.

For last four years I’ve tried to help my family in Poland and Russia who are writing—I can find out information and names of family—incredible that my family can’t get that information where they are.  There’s a federal program in US—work travel program—brings students to US to work; they work for three months and travel a month.  I was able to contact a lot of people who were coming and told them they can do the research they need to do their school papers, masters’ theses.

Back to: Contents | LSTA Evaluation Documents and Reports Page | LSTA home page | Library Development home page
Last Updated: July 24, 2019