• Register
• Key Dates
• Schedule
• Local Info

April 11-14, 2007
San Francisco, California

Do You Know Who Your Users Are? The Role Of Research In Redesigning sfmoma.org

Overview

Museum Web sites, like the cultural institutions they represent, must serve the needs of a wide array of users, from elementary-school students to scholars and everyone in between. How do we know who these users are and what they want? How can we make our site relevant and, dare we say, compelling, to these diverse audiences?

At the San Francisco Museum of Modern Art (SFMOMA), we have taken a user-centered approach to finding an answer. In spring 2006 we partnered with Hot Studio, a leading user-centered design firm, to conduct an intensive research and strategy project as part of the development of a new SFMOMA Web site scheduled to launch in early 2008.

In this paper we share some of the unexpected things we learned (for example, our visitors include more business and finance people than designers and educators!) and cover the background of how this project came to be. We give an overview of the research tools and methods we used, share our findings, and give concrete examples of how we applied these findings to our new Web site design. And, finally, we offer tips for conducting your own user research on a budget.

Why a Redesign?

The SFMOMA Web site redesign initiative began in 2005, when the museum decided that a complete front- and back-end revamp of sfmoma.org was long overdue. The site, which was launched in 1995, receives in excess of two million visits annually – more than triple the number of visits to the SFMOMA building. Yet almost ten years have passed since the site was last redesigned (in 1998), twice the lifespan of a typical Web site. Although the visual design of our site has aged relatively well and our traffic is continuing to rise steadily, the underlying code is obsolete, and the information architecture is inflexible; even the most minor updates are major undertakings. Furthermore, several of our peers have redesigned their sites within the past few years, offering fresh designs, enhanced educational resources, opportunities for on-line giving, easy membership registration, and improved e-commerce experiences. If SFMOMA is to retain its current audiences – and build new ones – its Web site must advance along these lines and offer the wide array of on-line services and features that today’s sophisticated users have come to expect.

Why User-Centered Design?

Once we decided to undertake a complete visual, functional, and technical redesign, we wanted to do it right. The last time the SFMOMA site was redesigned, there was no user research or testing to speak of, and the primary focus was on the visual aspect. The field of user-centered Web design was still emerging at that time, and rigorous research was not the norm for non-profit and museum sites. For most museums, SFMOMA included, it was enough to have an attractive and up-to-date site. This time around, however, understanding our target audience was our number-one priority. We felt the best way to achieve this was to enlist the expertise of outside professionals who live and breathe user-centered design.

User-centered (also called human-centered) design is exactly what it sounds like: design based on the needs of users, as opposed to the needs of the people who are executing the design. Much has been written about the field – some of those resources are listed at the end of this paper – and many organizations have used it to great effect. Still, however, only 25 percent of American businesses today actually conduct customer research (Terry, 2004). Too often, organizations avoid doing research because they think it is too expensive. But we knew that spending money up front on research would generate income in the long run by making our users happy (and therefore more willing to spend time – and money – on our site). We also believed that this approach would result in a site that was more efficient to manage and maintain.

Qualitative Benefits

We knew there was a lot to gain from the user-centered approach. In our case, we expected the following qualitative benefits:

• Increased visitor satisfaction
  If our site is a pleasure to use and relevant information is easy to find, users will experience an overall higher satisfaction with the site and with SFMOMA in general. While this is certainly a qualitative benefit, it is also something that can be quantified through before-and-after surveys.
• Increased awareness of SFMOMA’s offerings
  A better site experience will make our visitors more aware of what SFMOMA has to offer and more likely to visit the brick-and-mortar museum.
• Increased operational efficiency
  If the site improves the front- and back-end of key tasks such as buying or renewing memberships or purchasing products from the store, we gain some efficiencies in how we fulfill these transactions – and, we hope, increase the number of transactions as well.

Quantitative Benefits

Studies have proven that user-centered design has measurable results. For example, Nielsen has found that Web sites demonstrate an increase across several usability metrics, including site traffic and sales/conversion rates, by an average of 135 percent after a user-centered redesign (Nielsen, 2003).

For our project, we identified a handful of very specific tasks that we wanted to impact through the redesign. This gave us some specific metrics that we could use both to benchmark our progress over time and to identify areas for ongoing refinements. These metrics included:

• Memberships purchased or renewed on-line
• On-line sales from the museum store
• Donations collected on-line

Project Funding and Goals

To pay for this undertaking, we applied to the Haas Jr. Foundation to fund a $30,000 five-month Web site audience research project. We applied the grant funds toward hiring an outside firm. We chose to seek help outside the museum for several reasons. Our in-house staff did not have the bandwidth to take on such a project. We also felt that we would gain valuable knowledge and expertise from a firm that conducts user research with many different industries. If we worked on this project with experts, our in-house staff might gain valuable experience that could be applied to future projects. And, finally, we knew that determining what needed to change on our current site would be better received internally if it came from a neutral third party. The departmental politics of projects like these should not be underestimated.

Before contacting potential outside firms, we identified what was important to us in a partner: physical proximity (to allow for face-to-face collaboration), a range of expertise of the individual team members, and the company’s overall commitment to user-centered design. After soliciting proposals, we chose the San Francisco-based Hot Studio for their reputation in the field, their exceptionally strong team, and their extensive experience with both commercial and nonprofit organizations.

Our goal was to develop a nuanced picture of our users and their needs, to set institutional priorities for the new site, and to build internal consensus along the way. These were just a few of the many questions we hoped to answer in this ambitious undertaking:

• Who really uses our site, and what do they like and dislike about it?
• Should we think of our Web site as an extension of the museum building, or as a destination unto itself?
• How can our Web site encourage people to become visitors to the physical building, members of the museum, or even donors?
• How knowledgeable are our current site users about art, and what is our role in educating them?
• How much detail do people need about our collection and exhibitions (and do they even understand the difference between the two)?

To answer these questions, Hot Studio applied several best-practice methodologies, described in the next section.

What We Did

We had to be selective about what types of research activities to undertake. This was not an academic exercise for us; our research had to answer the questions above, and it had to be actionable. We concentrated our efforts primarily on interviews and surveys with end users and SFMOMA staff, while leveraging knowledge and expertise from museum experts. The project took approximately four months to complete.

Our research activities included the following:

• Best practices and heuristic evaluation
  We reviewed numerous other museum sites, including the Best of the Web winners from past Museums and the Web conferences, and identified positive features relevant to our project’s goals. Because we brought in an outside firm to lead this effort, we had the benefit of fresh eyes on our own site as well as on those of our peers. Together with Hot Studio we also conducted a heuristic evaluation against the SFMOMA site. A heuristic evaluation involves analyzing and critiquing a Web site to assess its compliance with some basic usability principles, or heuristics, such as consistency and user control (Nielsen, 1994). More details on how to conduct a heuristic evaluation can be found at the end of this paper.
• On-line survey
  We launched a one-week survey from the SFMOMA home page. More than six hundred site visitors participated. We offered participants a $5 Amazon.com credit or SFMOMA museum store coupon in return for completing the survey. The results gave us great insights into our visitors’ demographics as well as the content and features they want and need.
• Interviews with new and returning Web visitors
  We interviewed twelve target users over the phone about their site-related needs and expectations. We recruited them through a pop-up window that included some screening questions (to ensure a mix of members and non-members from various geographic locations). We found these interviews tremendously grounding in terms of our perception of who our on-line visitors are, what they need from us, and their relationships with art and with the museum.
• Other activities
  Other activities that we conducted as part of our research included a museum think tank session, a review of existing research and data, and a technical assessment. The start date of our project dovetailed nicely with the 2006 Museums and the Web conference, so we invited nine conference attendees to get together in Albuquerque, New Mexico, for a think tank discussion about various Web-oriented activities and emerging trends. The discussion, facilitated by Hot Studio, helped us understand how other institutions address challenges similar to ours. Back at SFMOMA, we reviewed an extensive collection of research materials that had been collected by various groups within the museum, including Web traffic reports, internal documents, and several years of in-building visitor surveys. And finally, we met with our IT department to discuss institutional needs and technical requirements for the redesign.

What We Learned

Our research yielded a wealth of insights that helped us develop a clearer understanding of our users and how we could better serve them. Below are some highlights from our findings:

Finding 1. “I may not be an art expert, but that doesn’t mean I’m not interested in art.”

One of our assumptions at the beginning of the project was that our site visitors were predominantly scholars, educators, researchers, or individuals with some type of professional art background. We couldn’t have been more wrong. Our surveys and interviews showed that site visitors included everyone from human resource managers to engineers. (One even identified himself as a farmer!) This had a big impact on how we saw the role of our site in educating and serving the public. Just over half of the survey respondents told us their occupations, which we grouped into the following general categories:

• 18.56%: Business/Finance/Technology professionals
• 16.47%: Students
• 12.28%: Architects/Designers
• 8.68%: Teachers/Educators
• 8.38%: Other
• 7.49%: Artists
• 6.89%: Healthcare professionals
• 6.29%: Professors/Art Historians/Researchers
• 3.89%: Museum/Gallery professionals
• 3.29%: Retired
• 2.99%: Publishing professionals
• 2.69%: Scientists/Engineers
• 2.10%: Lawyers

Finding 2. “Help me understand the art.”

Although the majority of our users aren’t art professionals, they do have a strong interest in learning about art. Our research showed that the information we are currently providing does not go far enough to expand our users’ understanding of our artworks and exhibitions. Users want a narrative experience that is clearly presented in accessible language. They also want the human story behind the art: who is this artist, in other words, and why should I care? They want someone or something to connect with as an entry point rather than a more technical or art-historical presentation.

Finding 3. “How are you different from any other museum?”

The on-line survey and interviews made it clear that visitors are not aware of SFMOMA’s distinctive character and the cutting-edge, contemporary aspects of the institution. Our Web site needs to convey more clearly who we are, why we are here, and what that means to the public. It needs to help users understand what kinds of art and what sort of overall experience they can expect at the physical museum.

Finding 4. “SFMOMA is just a place to see art.”

Beyond the art, users have no sense of the things going on at SFMOMA and the breadth of resources and content we offer on-line. They think of the physical institution simply as a place to view artworks, and our Web site as no more than a place to get information about the art. The site needs to provide intuitive access to our various on-line resources and better promote SFMOMA’s many activities and offerings.

Finding 5. “Help me find what I need.”

For people looking to accomplish specific tasks or find particular information, our current site can be very difficult or even impossible to navigate. We need a more intuitive navigational framework to aid users’ overall comprehension of the site, and it needs to offer more information.

The following excerpts from our survey present a sense of what people are looking for on our site, or hope to find there in the future. (Note that only the top three responses are listed below, and that users could make multiple selections).

“Why are you visiting the site today?”

• 65%: To get information on current exhibitions
• 43%: To get information on museum hours, directions, parking, etc.
• 14%: To view the collection on-line

“What things would you like to be able to do on the SFMOMA site in the future?”

• 61%: Find out if a specific artwork is currently on view
• 47%: Receive event reminders
• 40%: Watch videos of artists, lectures and interviews

Finding 6. “Help me plan my trip.”

At the beginning of the project, we struggled with whether we should think of our site as an extension of the physical museum or as a destination unto itself. Our research made the answer clear: the vast majority of users are coming to the site to learn what is on view in the building and to facilitate their in-person visits. Therefore, the content and tools we provide on-line should ultimately lead to, and enhance, in-person interactions with the art.

What We’re Doing About It

Our research helped us envision a new site that will better convey the breadth and magnitude of SFMOMA’s offerings. It will encourage our visitors to interact with the artwork, and make their eventual visit to the physical museum more meaningful.

In the following examples, we explain how our findings translated into tangible site design ideas. In each case, we start with the original finding, then the insight we gained from that finding, and finally the design application. Some of the examples include screenshots from our schematics. These schematics, or “wireframes,” do not represent the final visual design of the new site, but rather are intended to convey content organization and page elements that will appear there.

Example 1. Collections vs. Exhibitions

The finding: The vast majority of visitors want to know what is on view at SFMOMA regardless of whether the artwork is part of our collection or part of a temporary exhibition.

The insight: Most visitors do not intuitively understand the difference between “collections” and “exhibitions” and are confused by those categories as they are presented on the current Web site. While this distinction may be important in the museum world, the majority of our visitors don’t know or care about it.

The design: We decided to meld the two sections into a new category called “On View.” The main page of this section mixes images of works from temporary exhibitions with images of collection objects that are currently in the galleries. This allows users to discover what they can currently see at SFMOMA, or to perform searches across what is on view if they are looking for specific works.

Example 2. Breadth And Depth Of On-Line And On-Site Content And Programs

The finding: Our audience is not aware of what we have to offer.

The insight: On-line, we offer everything from information on lectures and public programs to streaming media footage of artists at work, but users are not aware of most of this.

The design: We need to find ways to highlight the variety of public programs happening at the museum, and to expose the rich educational materials available on-line.

The new home page (Fig. 1, Screenshot of Homepage schematic) will promote SFMOMA’s programs, upcoming events, and announcements in a much more aggressive manner than the current home page (Fig. 2, SFMOMA Homepage, http://www.sfmoma.org). Right now, for example, we dedicate a very small portion of the page to one-line, text-based news items, whereas the new home page will devote much more screen real estate to enticing visitors to explore our on-line and off-line offerings.

Fig 1: Screenshot of Homepage schematic

Fig 2: SFMOMA Homepage http://www.sfmoma.org

We also need to extend the promotion of events and content to second-level pages. On new screens devoted to individual artworks, we will highlight relevant public programs and on-line content related to that artist or object. For example, on the screen for Frida Kahlo’s painting Frieda and Diego Rivera, there could be prominent links to related events and programs (such as a docent tour that includes works by Kahlo and Rivera or a lecture on post-revolutionary Mexico), streaming video of a contemporary artist discussing Kahlo’s work, and pictures of Kahlo and Rivera at their home in Mexico.

Example 3. Web 2.0

The finding: When we talked with our users about potential Web 2.0 features we could offer on our site (blogs, wikis, etc.), they showed surprisingly little interest in them. The users we interviewed were fairly passive about the types of interactive things they would like to do on our site. Instead of asking an artist a question, they would rather read what other people asked. Instead of giving feedback about an exhibition, they would rather read what other people wrote.

The insight: We realized that if we were going to add any of these new types of Web 2.0 features, we should not invest in designing things that our visitors would not use. And if we were to incorporate any of these features in the future, they should extend the interpretation dimension and make the artwork more accessible.

The design: In addition to providing an authoritative museum perspective on an artwork, we must include features that incorporate perspectives from a variety of users, from front-line staff to visitors. On the “On View” main page, for example, we plan to include a feature called something along the lines of “Guest Take” that will present rotating works from SFMOMA’s collection selected by prominent local community members, artists, writers, museum members, etc. These guests will write about what the works mean to them and share their personal reactions, thoughts, and musings. Another feature, called something like “In Focus,” will allow museum staff members at different levels throughout the organization to select works from the collection and share their personal thoughts and reactions. This informal, multi-vocal approach will bring Web 2.0 values to the site and complement what we are already doing with SFMOMA Artcasts, our podcast audio-zine. SFMOMA Artcasts feature “Guest Take” commissions of music, poetry, and prose in response to works on view as well as “Vox Pop” pieces that capture live reflections from visitors in the galleries. We see these as methods of engaging the community in a dialogue of art and ideas; they are excellent ways to bring Web 2.0 values to the interpretative dimension of the museum experience.

Example 4. Layering information

The finding: All kinds of people are coming to our site, from physical therapists to curators. How can we serve this wide range of users most effectively?

The insight: General site visitors are looking for very different information than scholars and academics are. While academic content should be available, it should not get in the way of the experiences of general users.

The design: (Fig. 3, Detail of Artwork Detail Page schematic) The “Artwork Detail” page shows our solution to this problem. General information about an artwork, relevant to the majority of site visitors, is immediately accessible. More scholarly info is situated on a “closed” section of the page that can be “opened.” Once opened, it reveals a tabbed interface with detailed information pulled from SFMOMA’s collections management system.

Fig 3: Detail of Artwork Detail Page schematic

You Can Do It, Too

We were in the fortunate position of having a research grant that allowed us to hire an outside firm and invest the necessary time and effort in this project. However, there are many ways to incorporate these research techniques in-house on a shoestring budget, using tools and resources you may already have. Below are just a few examples.

On-line surveys

On-line surveys are an affordable and easy way to get demographic information about the people coming to your site and what they are looking for. You can also use surveys to ask questions about new features or content you are considering for your site. Using a hosted service such as Survey Monkey or Zoomerang will range in price from nothing (for the most basic service) to $350 per year for more advanced features. We used Zoomerang’s professional subscription. Not only can you easily create your own custom survey, but you can also access the results in an on-line report and then cross-tabulate the data. We offered users an incentive to take our survey (a $5 coupon redeemable at Amazon.com or at our museum store), but 30 percent of them opted to “donate back” the $5 to SFMOMA.

Interviews with end users

One-on-one interviews with real users can offer invaluable insights into how they think. While you can spend a lot of money hiring outside interviewers and a recruiting firm to find interviewees, you don’t have to. To save money on recruiting costs, start with your own network. Send an e-mail explaining the type of people you want to interview to all your contacts, and ask them to send it to every suitable person who fits that description. You can also recruit by posting notices at local universities or on-line through Craigslist and other community-based sites. We recruited through our own site with a pop-up form that asked users for some basic information (age range, geographic location, and whether they were members) and then followed up over the phone with the respondents who matched our criteria. We offered an incentive to participants (a $25 Amazon.com coupon), which they all accepted.

For the interviews, develop a script with open-ended questions that do not lead people to particular answers. Try to limit your script to a dozen or so questions to prevent “interview fatigue,” and focus on the areas that will have the biggest impact on your work. For example, if you are trying to decide what types of content to provide on your site, ask people what they do on the site now (or if they use it at all) and what they would like to find, and then follow up with questions about some specific features that you have considered adding. And don’t be afraid to stray from your script if you sense someone has valuable information about a topic that you didn’t anticipate. The script should be a guide. The more people you interview, the more patterns you will see emerging from their responses.

Interviews with internal stakeholders

The above technique applies to internal staff as well. People working in technology, communications, outreach, and other areas often have a lot to share about their needs for communicating with the public. They are bound to have insights into your target audiences. To motivate our internal stakeholders to communicate with us, we provided snacks at every meeting and found that free food works well!

Usability testing

This is another technique that can get quite expensive but doesn’t have to. If you can forgo a professional lab with a one-way mirror, video cameras, and a moderator (who will also write up the reports), you can conduct your own test quite affordably. All you need is a quiet room and paper printouts of your new designs. Formulate a predetermined set of questions to ask your users about their impressions of the pages they are viewing and the tasks they might need to complete. Take good notes, and keep your ears open for unexpected information.

Heuristic evaluations

There are some basic improvements to your site that you can probably identify without doing a lot of user research. The goal of a heuristic evaluation is to identify these “no-brainer” types of improvements by reviewing some general Web conventions and then analyzing how your site measures up. This can be a good exercise for you to do internally because it forces you to document necessary improvements that you might otherwise overlook. To perform your own heuristic evaluation, start with a list of principles for good Web design and click around your site to determine if you are meeting these basic standards. You may want to start with a set of known heuristics, such as Jacob Nielsen’s ten general principles for user interface design, and adapt them to fit your project (Nielsen 1994).

Conclusion

Conducting research to learn who our Web site users are has informed every aspect of SFMOMA’s site redesign. By gathering qualitative and quantitative data, we have been able to make informed decisions about desirable new features and interfaces and justify those decisions to organizational stakeholders. From on-line surveys to telephone and in-person interviews, the experience of conducting this research has put us in touch with our users and given us the confidence to move on to the next phase of the project: visual design strategy. During this phase we will refine the schematics we have developed, do paper prototype testing with users, and then develop the complete visual design for the new site.

References

Kuniavsky, M. (2003). Observing the User Experience: A Practitioner’s Guide to User Research. San Francisco: Morgan Kaufmann Publishers.

Morville, P., and L. Rosenfeld (2002). Information Architecture for the World Wide Web, 2^nd Edition. Sebastopol: O’Reilly & Associates, Inc.

Nielsen, J. (1994). “How to Conduct a Heuristic Evaluation.” Useit.com: http://www.useit.com/papers/heuristic/heuristic_evaluation.html. Consulted January 6, 2007.

Nielsen, J. (2003). “Return on Investment for Usability. ” Useit.com: http://www.useit.com/alertbox/20030107.html. Consulted January 6, 2007.

Nielsen, J. Useit.com. http://www.useit.com

Snyder, C. (2003). Paper Prototyping: The Fast and Easy Way to Design and Refine User Interfaces. San Francisco: Morgan Kaufmann Publishers.

Terry, P. (2004). “Survival of the Fittest.” The DMA Insider, 30–34. http://www.creativegood.com/doc/creativegood-survival-fittest.pdf. Consulted January 6, 2006.

Wodtke, C. (2002). Information Architecture: Blueprints for the Web. Berkeley: New Riders.

Cite as:

Mitroff, D., and K. Alcorn, Do You Know Who Your Users Are? The Role Of Research
In Redesigning sfmoma.org, in J. Trant and D. Bearman (eds.). Museums and the Web 2007: Proceedings, Toronto: Archives & Museum Informatics, published March 1, 2007 Consulted http://www.archimuse.com/mw2007/papers/mitroff/mitroff.html

Editorial Note