Website Evaluation: Evaluation of librarycareersny.org
Sam Cook
August 6, 2007
|
|
REAP Change Consultants |
This report also available in .PDF format 
.
Overview
From its beginning as a minor side note in the Making It REAL! grant application, the librarycareersny website has developed into a fully operational website, designed to assist New Yorkers in starting and building a library career. This website has the potential to develop into a great resource for anybody interested in the field of librarianship. This report includes recommendations on how to improve the librarycareersny website so that it can reach its full potential, although it is already a highly useful and welcome addition to the resources currently available.
This report analyzes the website in four major areas: content, usability, accessibility, and user statistics. The content section focuses primarily on the services that were requested in the Request for Proposal (RFP #05-015) and whether the website meets those requirements. The usability section includes information collected through an online usability survey, in addition to an online usage survey that was available through the website itself. The accessibility section is based on a report written by David Comeaux, one of the Making It REAL! scholarship recipients. The user statistics have been automatically collected throughout the life of the website, and will be analyzed here for useful information. The report concludes with a brief summary of the future of the librarycareersny website.
Content
As previously stated, the Making It REAL! grant application provided very little information regarding what the planned website would include. According to Mary Linda Todd, the Project Director, the original intent was to create a website similar to the American Library Association JobLIST, except with a focus on New York State. Over time, however, the website plan evolved into one that would focus more on providing information about the librarianship career in New York. The final plan for the website was documented in Request for Proposal #05-015, which presented prospective website developers with a description of what information was to be included on the website. This section of the evaluation will examine whether the librarycareersny website contains the content requested in RFP #05-015.
Diversity
One of the first requirements of the website was that it "assist current and future librarians from diverse backgrounds in exploring, starting and building a library career in New York State." The website has mixed results in this area. As diversity is such a major aspect of the Making It REAL! project, the website should reflect that importance. In the area of the diversity of employment in the library profession, the website is highly successful. It presents information on many different types of positions available to those with a Library and Information Science degree, both inside and outside of libraries. The website also provides links to external sites that provide further information on each type of employment.
In the area of individual diversity, however, the website is not as successful. The Making It REAL! program used the American Library Association Staff Diversity and Inclusion Action Plan definition of diversity, which states:
"The American Library Association recognizes that in addition to race, creed, color, religion, gender, disability and national origin, there are a multitude of differences (language origin, regional and geographic background, economic class, education, learning and communication styles, sexual orientation and personal lifestyle) that individuals bring to the workplace."
The website does succeed in demonstrating the diversity of librarians, with some of the abovementioned categories included, via librarian profiles, articles that attempt to break the librarian stereotype, and a video about working in Queens. However, there is little information that would actually assist people with diverse backgrounds in "exploring, starting and building a library career." The Scholarships & Funding page does provide information on scholarships for racially diverse people, but there is little else.
As part of the usability survey, users were asked if the website provides information for people of diverse backgrounds. Out of the 39 people who answered this question, only 11 felt that there was extensive information on this topic. Comments on this topic generally agreed with the above evaluation. One user wrote, "The people in the pictures are very diverse - but I don't specifically see information for people of diverse backgrounds." In order for diversity to be a major aspect of this website, the information needs to be more in depth and more prevalent.
The RFP also requested that the website have portions in Spanish. The only portion of the website in Spanish is a two-page pamphlet available in the Medical Libraries section. This is not enough for the website to be considered to have portions in Spanish. It would be far more valuable, although time consuming and possibly expensive, to create Spanish versions of at least the six top-level pages.
Employment Information
As a website that focuses on library careers in New York, one of the primary reasons people will use this site is to find information about library employment. The three requirements in the RFP regarding information in this area were that the website have information on the number of positions across the State, that it identify shortage areas, and that it host a Job Bank.
The librarycareersny website does include information on the number of positions across the state on the "Get the Facts" page, within the Careers section. While this page provides just basic information, it includes links to the source material from which this information was gathered, which often include more detailed information. This page does not include information on shortage areas, however. If this information is present within the pages used as sources, this needs to be made more obvious, as these are off-site pages.
While there is a form of Job Bank available on the website, found in the Job Listings page within the Jobs section, this page needs to be much more prominent to be as useful as possible to users seeking library jobs in New York. On the online usability survey, users were asked to take one minute to find the Consolidated List of New York State Job Listings. 7 out of 43, or approximately 16%, of users were not able to find this page. For a website aimed at assisting people in "exploring, starting and building a library career in New York State," this number is too high. The Job Listings page should be given a more prominent place on the website, possibly with a direct link on the home page, rather than just being hidden among several other links on the Jobs page. Even some people who were able to find the Consolidated List of New York State Job Listings felt that the job information should be more readily available, one user writing, "The job listings could be a bit more pronounced or in their own section. It is likely one of the key things people will look for." Another user wrote, "I might consider adding a link in a more prominent place, since lots of people will go straight to that."
Education
The two library education related requirements in the RFP were for the website to include scholarship information and links to the websites of all of the library schools in New York. The librarycareersny website includes these as well as further information on library education. In addition to listing all of the New York library schools, the website also lists schools that offer online degree options, including examples of library students’ distance education experiences.
Library Organizations
A requirement listed in the RFP was that the website was to include information on library organizations. This requirement is met in the Associations & Organizations page. This page, found in the career growth section, links to major library organizations and associations, at both the national and state level. One problem with this page is that the link for it is labeled "Get Involved: Professional Organizations." This label should match the actual name of the page, which describes the page’s intent more clearly.
Usability
To determine the usability of the librarycareersny website, a survey was put online to evaluate four areas of usability: navigation, appearance, website intent, and content. 43 people completed the survey. Although this was not a completely random sample, as everybody who completed the survey was specifically requested to do so via email, the sample included males and females, people ranging in age from mid-20s to over 60, and both those affiliated with libraries and those with no affiliation. Although this method was not ideal compared to one with a random and larger sample, or a live one-on-one test, both of which would have taken more time and money, it is still highly useful, as it provides specific information from website users on the usability of the site and how to improve it. According to Steve Krug, in his book Don’t Make Me Think, "Testing always works. Even the worst test with the wrong user will show you things you can do that will improve your site" (142).
Navigation
The librarycareersny site ranked very high on ease of navigation. Of the 43 people surveyed, no one found the navigation difficult, and 37 found it either easy or very easy. Users tended to like the large, side button navigation. Users also found that it was always clear what section they were in, and the content in each section was that which was expected.
Although the website is already fairly easy to navigate, it could be improved by raising the navigation menu and the content higher up on the page. On a computer with a lower resolution monitor, the user may be required to scroll down to see the side-bar navigation and the content. Navigation would be simpler if at least the navigation menu were visible immediately, without any scrolling.
One problem with the website navigation is that there is no distinction between on-site and off-site links. It is possible that a user who selected an off-site link would remain off-site and would not see the rest of the librarycareersny site. Creating a distinction between the two types of links would increase the probability of a user examining the entire site before selecting off-site links.
Appearance
The appearance of the librarycareersny site was highly ranked, with 34 out of 43 users finding it either appealing or very appealing, and only 3 finding it either unappealing or very unappealing. Words used to describe the website included light, clean, well lit, colorful, and professional. Some users, while finding the appearance appealing, felt that it was not interesting enough, one user writing, "It could be made more visually sophisticated. Right now, it’s very simple and plain. I think people associate visually appealing design with quality content…"
One user also found that the blurring on the sides of the header picture was not appealing. Although the limited blurring on the home page image is successful in drawing the user’s focus toward the image, the excessive blurring on the other pages takes up space without contributing to the website. Removing the blurred section of this picture should allow more space at the top of the page for the navigation, which could solve the scrolling problem discussed in the Navigation section of this report.
Website Intent
According to Steve Krug, "Right off the bat, the Home page has to tell me what site this is and what it’s for…" (97). Part of the online survey asked if users felt that the website intent was made clear on the Home page. Out of the 43 people who completed the survey, 37 felt that the intent was very clear, and only one felt that the intent was not clear. The Home page is very clear in it’s intent, as it lists the four major phases of a library career (considering a career, pursuing an education, getting a job, and career growth). This accurately indicates that the librarycareersny is intended to assist potential and current librarians at all stages of their careers.
The only significant complaint about the Home page of the website, as noted by one person who completed the usability survey, is that the regional focus is unclear. Although this website has value for future and current librarians from any location, the focus is on New York State, and this should be reflected in the Home page. As it is presented, New York is only mentioned on the Home page in two links and in the acknowledgment at the bottom of the page. Although "NY" is in the title of the site, some users may not be aware that this is referring to New York, or may just not notice it. The Home page content should mention New York State at least once.
Content
Whereas the previous Content section was analyzing if the required content was available on the librarycareersny site, this section is analyzing if the current content is useful. Of the 43 people who completed the usability survey, 36 found that the content either helpful or very helpful if they were pursuing a library career. More valuable data, however, can be taken from a usage survey that was placed on the librarycareersny website, collecting information from users who were already visiting the website, rather than from users who were asked to visit the website. In addition to asking users how useful the website was, that survey also asked why the user was visiting the website, a question that could not be asked in the usability survey.
The survey that was placed on the website showed that the website was more useful for users who were using the site to find information on the field than for those who were looking for jobs in the field. Of the twelve users who visited the site looking for information on jobs, only one found it very useful. Seven of these users found it useful, so the problem is not that it is not useful at all, as also indicated in the usability survey, but that it could be improved. One user wrote, "I came for the job leads, but all you have are the same as other library job sites." Another problem could be the difficulty some users have had in finding the job listings.
In the usability survey, many suggestions were made as to how to improve the usefulness of the content. These include "information about getting and maintaining NYS School Library Certification," further information on music librarianship, including all of the New York job listings in one database, further information on archives as a career possibility, further information on civil service positions for public librarians, and listing jobs by public and school system, not just by 3Rs system.
Accessibility
One of the goals of the Making It REAL! project, as stated in the grant application, was to recruit and educate "the next generation of librarians to enhance diversity in the profession and serve diverse populations." This includes serving people with disabilities. One aspect of achieving this goal is to provide websites that are accessible to people with various disabilities. As a website funded through the Making It REAL! program, and one that strives to bring greater diversity to the field of librarianship, it is essential that librarycareersny meets certain accessibility standards.
Visual and motor disabilities are the two primary disabilities that must be considered when developing a website. Based on both automated and manual tests, librarycareersny.org has no major accessibility problems and is accessible to people with both visual and motor disabilities. Although the website has a few minor accessibility errors, based on the World Wide Web Consortium’s Web Content Accessibility Guidelines, these errors should present no major obstacles to accessibility by people with disabilities. These errors should be fixed eventually, but it is not a major priority.
The following accessibility analysis of the librarycareersny website was prepared by David Comeaux, one of the scholarship recipients of the Making It REAL! program. Mr. Comeaux studied at the University at Buffalo, SUNY, where he had a focus on accessibility issues on the web.
Tests Run
Automated Tests
The WatchFire (formerly Bobby) automated test was used to analyze each page. Each page was tested against both the Web Content Accessibility Guidelines (WCAG) and Section 508 Standards. For detailed results of the automated tests, see Appendix A: Results of Automated Tests for Accessibility.
Discussion of Automated Test Results
The WCAG testing found no Priority 1 errors on this web site. Two Priority 2 errors, "Use relative sizing and positioning, rather than absolute," and "Do not use the same link phrase more than once when the links point to different URLs" were found. Two Priority 3 errors, "Provide a summary for tables", and "Separate adjacent links with more than whitespace" were found.
The Section 508 testing found only one reported error, "Make sure there is a link to download accessible plug-ins." This error is usually found when a page has active content, such as Flash or Video content, without providing a link to the plug-in, (e.g., the Flash player) needed to display the content. In this case, a manual check indicates that there is no active content, so this error is difficult to explain.
Overall, the automated testing indicates that no major accessibility errors are present.
Manual Testing
In addition to running automated tests, some manual tests were performed.
Using the site with Images disabled
The most common accessibility error found in web pages is the use of images without providing descriptive alt text for the image. This practice can render a site useless for a visually disabled user. A quick test for this is to disable images and ensure that the site can still be read and navigated.
After disabling images, it was found that the site remained completely usable. All images include descriptive alt tags.
Using the site with keyboard only
Another key test is to determine if a site can be navigated using the keyboard only. This is important because many people have motor disabilities which make it difficult or impossible to use a mouse. An important element to consider is the use of a "Skip Navigation" or "Jump to Content" link. This feature allows for the user to skip repetitive navigation links and jump directly to the page content.
It was found that this site can be navigated easily without the use of a mouse. The focus jumps from tab to tab logically. In addition, a "Jump to Content" link is provided which allows users to skip directly to page content.
View site with various Font Sizes
Often, users with sight disabilities will view web sites at the largest possible font size. Sites should be designed so that the layout remains intact no matter how large or small the font size is set.
It was found that this site layout remains intact for whatever text size is selected in the browser.
Statistics
Since the librarycareersny website went online in May 2006, statistics for the site have been collected by SmarterStats. A major flaw with this statistics collector is that, while spiders/bots (automated programs that access websites to provide search engines with information) are acknowledged and accounted for, they cannot always be filtered out of the data. For example, California is listed high on the statistics of where users were located, but as it is the home of both the Google and Yahoo corporations, this is possibly due to the number of spiders/bots coming from these companies. Due to this problem, not all usage statistics available through SmarterStats use accurate statistics. For this report, only usage statistics that mostly remain unaffected by this problem will be included.
Daily Website Usage
Although exact daily usage numbers are unavailable due to the problem mentioned above, general trends can be seem by viewing the overall usage change during the life of the website. Chart 1 shows the number of daily visitors from May 2006, when the website was put online, to July 2007.
The first rise in website usage occurs in the beginning of November 2006, after the site was advertised at the 2006 NYLA Conference. The sudden spike in website usage circled on the graph is in mid-April 2007. This is when official publicity for the website began. This graph indicates that the publicity was successful in immediately drawing in a large number of visitors to the website, as well as increasing the regular flow of visitors thereafter.
Most Viewed Pages
The distribution of pages visited by spiders should be fairly even, so an analysis of the most viewed pages should determine what pages are most popular among the website users. Chart 2 shows the top ten most viewed web pages since May 2006.
Chart 2: Most Viewed Web Pages
The most viewed page, as would be expected, is the Home page. This is shown in Chart 2 in entries one and five, both of which connect the user to the Home page. Of the remaining eight most viewed pages, four are within the Jobs section. This implies that the primary reason users are on the website is to locate job information. Because of this, it is highly important for the Jobs section to be improved and kept up-to-date.
Search Phrases
An analysis of search phrases determines what users are looking for when they encounter the website. Chart 3 shows the top ten search phrases used to find the librarycareersny website.
The entries that include the website name (librarycareersny.org, www.librarycareersny.org, and librarycareersny) can be ignored, as anyone searching with these phrases is already aware of the site and could just as easily have put the site address into the address bar and reached the site. That leaves the search phrases "library careers", "new york public library jobs", "library careers ny", and the names of several scholarship recipients who are featured in the Profile section of the website.
Chart 3: Top 10 Search Phrases
It is interesting to note that entry 3, "new york public library jobs," is the only entry to include the word "job." It is possible, however, that many of those searches were done by people searching for jobs within the New York Public Library, rather than those searching for public library jobs in New York. As the majority of people using the site are searching for jobs, as evidenced by the most viewed pages, it is important that people searching for "New York library jobs" easily find the librarycareersny site, which is currently not the case with either Google or Yahoo. One simple method to improve this would be to add a "jobs" or "library jobs" keyword into the home page’s keywords meta tag.
Browsers
As spiders do not operate via web browsers, data on what browsers are used to access the website are accurate. Chart 4 shows the ten browsers that are most used to access librarycareersny.
The most common browser used to access the website is Internet Explorer 6. It is important to note that this is not the most recent version of Internet Explorer. The second most common browser is Mozilla Firefox. Although the website should be compatible with all web browsers, special care should be made to keep it the most compatible with these two browsers, if changes are made to the site in the future.
Future of librarycareersny
As the Making It REAL! project comes to completion at the end of August, 2007, so does the contract with the website developer. RFP #05-015 required the website developer to "develop and implement a transition plan… so the New York State Library is able to sustain the site after the contract ends." As an alternative to this, the State Library is contracting with the website developer to continue hosting the site, at least for one more year. Eventually, the State Library plans to move the website onto the NYSL server, but this may not come for several years.
It would be beneficial if the website were continually improved to turn it into an even more useful resource for future and current librarians in New York. If funds are available, this should be made a consideration when contracting for the next year with the website developer. The website is useful as is, but it has the potential to become the leading library career website in New York.
The most valuable improvement that could be made to the librarycareersny website would be to broaden the scope of the job listings to make it a true Job Bank. The original intent was to make the website similar to the ALA Joblist, but with a focus on New York. While there were not enough funds to achieve this goal originally, it would be ideal as a future goal. If employers saw this site as the most effective method to reach potential employees in New York, it would quickly become an invaluable tool for both employers and prospective employees.
In order to become a greater resource, the value of librarycareersny needs to be recognized by libraries, library systems, and library schools throughout New York. These organizations need to find methods, or be provided with methods, to advertise the site, as this would benefit the field of librarianship throughout the state. Some organizations have already begun to do so, such as Queens Library, which is using the website to do outreach in its Job Information Centers. Mid-York Library System used the site for a career day at a high school. New York Public Library provides a link to the site on its intranet as a resource for clerical and technical support staff members who are interested in pursuing a career in librarianship. Others, such as Metropolitan New York Library Council, NYLA, Mid-Hudson Library System, and University at Albany have included links to the librarycareersny site on their own websites. This is a good start, but further collaboration is needed for this website to be used to its full capacity.
Appendix A: Usability Survey Results
WCAG Results
URL |
Results Summary |
||||||||||||||||
http://librarycareersny.org/ |
|
||||||||||||||||
http://librarycareersny.org/ |
|
||||||||||||||||
http://librarycareersny.org/ |
|
||||||||||||||||
http://librarycareersny.org/ |
|
||||||||||||||||
http://librarycareersny.org/ |
|
||||||||||||||||
http://librarycareersny.org/ |
|
||||||||||||||||
Section 508 Results
Page |
URL |
Results Summary |
Home |
http://librarycareersny.org/ |
Complies with all automatic Checkpoints |
Careers |
http://librarycareersny.org/ |
1 error reported, "Make sure there is a link to download accessible plug-ins" |
Education |
http://librarycareersny.org/ |
Complies with all automatic Checkpoints |
Jobs |
http://librarycareersny.org/ |
Complies with all automatic Checkpoints |
Career Growth |
http://librarycareersny.org/ |
Complies with all automatic Checkpoints |
Profiles |
http://librarycareersny.org/ |
Complies with all automatic Checkpoints |
Appendix B: Works Cited
American Library Association. (n.d.). Staff Diversity and Inclusion Action Plan. Retrieved August 6, 2007, from http://www.ala.org/ala/diversity/diversityactionb/ALA_Diversity_Action_and_Inclusion_Plan.pdf
American Library Association. (2007). JobLIST. Retrieved August 6, 2007, from http://joblist.ala.org/
Krug, S. (2000). Don’t make me think! A common sense approach to web usability. Indianapolis, IN: New Riders Publishing.
Back to the IMLS page | Back to the Library Development page