|
|
To Use or Not to Use? Evaluating
Usability of Museum Web Sites
HOC-Hypermedia Open Center
Department of Electronics and Information, Politecnico di Milano
P.zza Leonardo da Vinci, 32 - 20133 - Milano, Italy
E-mail: {garzotto, matera, paolini}@elet.polimi.it
Phone: +39-2-23993623; Fax: +39-2-23993411
Contents
1. Introduction
2.
The SUE (Systematic Usability Evaluation) Approach
2.1 What is SUE?
2.2
The SUE Preparatory Phase
2.3 The SUE Execution Phase: Inspection
and User Testing
3.
The SUE Preparatory Phase for On-line and Off-line Hypermedia
3.1 The HDM Model
3.2 Usability
Attributes
3.3 Abstract tasks
4. Examples of
Inspection Findings
4.1 Musei Vaticani
4.2 National Gallery of Art in
Washington
4.3 Further Examples
(Virtual Uffizi, and Kimbell Museum)
5. Conclusions
6. Bibliography
1. Introduction
The advent of the World Wide Web has introduced a new communication
channel, through which people can easily access museum content,
events, activities, and schedules. Since the use of museum web sites
is spreading all over the world, and these sites are attracting
an increasing number of "virtual" visitors, it has become crucial
to improve their usability, i.e. the visitor's ability to
use these sites and to access their content in the most effective
way. As a consequence, it has become compelling to provide both
quality criteria that WWW sites must satisfy in order to be usable,
and systematic methods for evaluating such criteria.
This paper describes a technique for evaluating the usability of
hypermedia applications, either off-line (i.e., multimedia CD-ROMs
or Information Points) and on-line (i.e., Web sites). The approach
presented here is based upon a general methodology named SUE - Systematic
Usability Evaluation. Four attributes can be used
to characterize SUE: heuristic, empirical, systematic,
and model-based. It is heuristic according to the
definition proposed by Nielsen [Nie93],
in that it identifies a set of usability principles, or heuristics,
that evaluation experts must check when the application is inspected
for usability. Heuristic evaluation falls into the broader range
of usability engineering methods called "inspection", which do not
involve end users, but only "expert" evaluators. SUE is also
empirical, in the sense that user testing is used to validate
and refine the result of the inspection. The combination of inspection
and empirical testing ensures the most accurate evaluation results,
coupled with cost-effectiveness. SUE is systematic in
the way usability inspection and empirical testing are performed.
The inspection is carried on by executing a set of predefined evaluation
activities, called abstract tasks. The use of abstract
tasks makes the heuristic evaluation better organized and more effective,
and also encourages standardization across different evaluators
and evaluation processes. The empirical testing is carried on by
requiring some end-users to perform concrete tasks, i.e.
specific activities, based upon the result of inspection. The use
of concrete tasks makes the empirical testing more organized and
cost-effective. SUE, finally, is model-based since models
are used to precisely shape the evaluation activities. In this paper,
as we are examining hypermedia, we have used HDM (Hypermedia Design
Model), developed at Politecnico di Milano.
The rest of this paper is organized as follows. Section 2 briefly
presents the overall approach of the SUE methodology. In section
3, SUE is specialized for hypermedia and we will discuss the model,
the usability criteria, and (a sample of) the abstract tasks, that
represent the cornerstones of our method. Section 4 illustrates
our approach by presenting some (fragments of) evaluations of museums
sites. Section 5 draws some conclusions.
2. The SUE
(Systematic Usability Evaluation) Approach
2.1 What is SUE?
SUE is a general methodology for usability evaluation of interactive
systems, developed at Politecnico di Milano, in co-operation with
the industrial partner CORINTO (COnsorzio RIcerca
Nazionale Tecnologia ad Oggetti - an Italian
consortium involving IBM-SEMEA, APPLE-Italy, and SELFIN), and the
Universities of Lecce and Bari.
The aim of SUE is twofold:
- to support a broad, multi-dimension analysis of usability, and
- to make usability evaluation systematic and cost effective.
Most existing approaches to usability evaluation address only general
interface issues, e.g. the lay-out design, the choice of icons and
fonts, the interaction style, etc.. The main assumption of SUE is
that usability analysis, instead, should consider the specific nature
of the application to be evaluated.
SUE suggests analyzing a system from different perspectives:
- general perspective, where the features common to all
interactive applications are considered;
- application-category perspective, that considers
only the peculiar features of a specific functional class of applications
(e.g., "hypermedia", "word-processor", "transaction processing
system");
- application-destination perspective, that focuses on
the specific domain of an application within a given category
(e.g., "tourism" hypermedia, or "education" hypermedia).
For each different perspective considered during the evaluation,
SUE requires two operational phases of evaluation: i.e. the preparatory
phase, and the execution phase, briefly described in
the rest of this section.
2.2 The SUE Preparatory Phase
The purpose of the preparatory phase is to create the conceptual
framework that allows the evaluation to be carried on in a systematic
and effective way. The preparatory phase (for each perspective)
consists of three different activities:
- the choice of a model;
- the definition of a set of usability attributes; and
- the specification of a set of abstract tasks.
The model identifies unambiguously the constituents of an interactive
system that are the "entities of interest" [Fen91]
for evaluation. The term "model" is used here in a very broad and
loose sense, intending a set of primitives capable of describing
an application. SUE does not prescribe which model should be used;
the only requirement is that a model is used, and that the chosen
model is sufficiently expressive to describe, in a non-ambiguous
way, all the "entities of interest".
The usability attributes are quality factors (closely related to
the specific evaluation perspective being considered), under which
usability is decomposed, in order to be more specifically analyzed
and measured.
Abstract tasks are "generic" operational activities that evaluators
must perform during inspection. We call them "abstract" since they
are formulated independently from a specific application, and they
do not refer to the specific objects and functionality of a specific
system, but, rather, to classes of objects and functionality that
are relevant for the chosen perspective.
2.3 The SUE Execution Phase:
Inspection and User Testing
The execution phase occurs each time a specific product is evaluated,
and systematically combines inspection and empirical testing.
Inspection is a generic term for a range of usability engineering
methods that have seen an increasing widespread use. As we mentioned
in the introduction, inspection methods do not involve end users,
but "expert" evaluators only; since they "save users" [Nie94], and require neither special equipment, nor
lab facilities, they are cheaper than other evaluation techniques.
Another defining characteristic of inspection methods is that they
are more informal than other techniques, since evaluation is "based
on the considered judgment of the inspectors" [Nie94]. If compared with other inspection methods,
the novelty of SUE-based inspection is that evaluators systematically
exploit the results of the preparatory phase, in order to be more
organized, standardized and cost-effective.
Inspection is carried out first, and detects a preliminary set
of potential usability problems. It is then followed by empirical
testing, which consists of selecting a sample from the real user
community, assigning them a set of tasks to be performed by using
the application, and recording their actions. The novelty of SUE
is that empirical testing is precisely driven by the output of inspection.
3.
The SUE Preparatory Phase for On-line and Off-line Hypermedia
Since the focus of this paper is not SUE, in general, but its application
to hypermedia, this section presents the conceptual framework tailored
for hypermedia: the chosen model, HDM, the set of usability attributes,
and a sample set of abstract tasks.
3.1 The HDM Model
HDM defines a number of dimensions along which a hypermedia
application can be modeled: structure, navigation,
behavior, user control, and presentation.
Structure describes how the application content is organized. Navigation
concerns the definition of the browsing paradigms, i.e., the
actual links available to the user for exploring the application
structures.
Behavior concerns two aspects:
- the dynamics of time-based media such as video, sound, animation,
i.e., how their state evolves with time, by effect of user interaction
or in dependency of the state of other media;
- the dynamics of links, i.e., the effects of link traversing
on the state of time-based media in the source and destination
nodes.
User control concerns the operations available to the user
to interact with the application. Presentation concerns
how all the previous features are "shown" to readers, and is related
to the lay-out of objects (anchors, buttons, windows, menus, etc.),
and their visual properties.
In its current version, HDM focuses mainly on structure, navigation,
behavior, and user control. In this section we introduce a very
short synthesis of the model, mainly addressing structural and navigational
primitives, which are the most relevant for modeling WWW hypermedia.
The reader is referred to [Gar93, Gar95, Gar98a] for a complete description of the HDM model.
HDM distinguishes between hyperbase structures, which constitute
the so called hyperbase layer (hyperbase for short) of the
application, and access structures, which constitute the
so called access layer. Hyperbase structures are used to
represent domain information, while access structures provide entry
points to the hyperbase.
The hyperbase consists of typed entities and typed
semantic connections. Semantic connections represent domain-dependent
relationships among (parts of) entities. Entities denote conceptual
or physical objects of the application domain and are composite
representation structures; their logical constituents are called
components, and are organized according to some topological
patterns (e.g., sequences, trees, lattices). Components in turn
are made of a set of typed nodes. Different nodes
within a component describe different "perspectives" along which
the component subject can be represented. Nodes are the containers
of content; they correspond to "pages" (or, sometimes, page sections),
and aggregate a number of content elements called slots. A
slot can be complex, i.e., it may include a number of different
media, but it is always an atomic unit from the user interaction
point of view. A slot can be static or dynamic,
depending whether it stores time-independent media (such as formatted
data, text strings, images and graphics) or time-based media (as
video, sound, or animation). The type of a slot is defined by the
conceptual and physical nature of its media.
The access layer consists of collections. A collection groups
a number of "members", in order to make them accessible. Members
of a collection are hyperbase elements (entities, components, or
nodes) or other collections. Collections including other collections
among their members are called nested. A collection may have
a center node, (although it is not mandatory) which is the
starting point of the navigation within the collection and usually
contains some information about the collection itself. Members of
collections are collected according to some semantic criteria (e.g.,
in a museum application, "all paintings of a specific painting School"),
or according to an expected user's goal (e.g., "the top ten paintings"
for a quick visit of the museum pieces). Guided tours [Tri88]
or tables of contents, commonly found in hypermedia applications,
are modeled by collections in HDM.
Navigation defines how to explore entities and collections. It
is specified by means of navigational links (links for short)
that connect nodes within the various structures. Navigational links
are classified as structural, applicative, or collection links.
Structural links connect nodes within an entity according
to its topology; applicative links connect nodes of different
entities related by some semantic connections; collection links
connect the constituents of a collection. Links in the various
categories are typed. A collection is called guided tour
if it has only links connecting the center to the first member,
and the each member to another. A collection is called index
if it has only links connecting the center to each member, and vice
versa. A guided tour index is a collection which includes
both sets of links.
3.2 Usability Attributes
Usability attributes are empirical qualities that we expect from
a well designed and a well implemented application. To define our
usability attributes, we have first considered two general and
universally acknowledged principles - learnability and
efficiency, and have decomposed them into a number of hypermedia
specific sub-principles that represent our usability attributes.
Learnability concerns the features of an interactive system that
allow novice users to use it initially, and then to attain a maximum
level of performance [Dix93], and is decomposed into two attributes -
consistency and predictability. Efficiency concerns
the features that support the successful achievement of the user's
goals, with a high level of productivity, and can be described in
terms of accessibility and orientation.
The HDM model helps focus our analysis of general usability principles,
to identify the various classes of features which should be considered
for usability, and to organize our attributes. Since HDM does not
support presentation modeling, our approach does not consider presentation
features. However, with the relevant exception of anchors, most
presentation features of a hypermedia are largely independent of
the specific nature of this class of systems, and concern evaluation
from the general perspective. Furthermore, we will not consider
in this paper usability features related to the dynamics of a hypermedia
application, i.e., behavior of (and user control of) links and active
media. For example, we will not discuss the usability of mechanisms
that allow users to manipulate the state of video, sound, animation,
etc. (by means of commands, such as "start", "stop", "pause", "restart",
"forward", "backward" etc.), or the effects of link traversing.
These features are relevant for hypermedia which include a significant
amount of active media, which is not the case of most WWW applications.
The reader is referred to [Gar97, Gar98b, Gar98c] for a discussion of all these aspects.
Consistency
Consistency means that conceptually similar elements are treated
in a similar fashion, while conceptually different elements are
treated differently. From a structural perspective, consistency
prescribes that entities representing domain objects of the same
class aggregate components of similar types, and with a similar
topology, components of the same type aggregate nodes of the same
type. Nodes representing similar components, or similar perspectives
of a component, contain similar sets of slots. Analogous considerations
can be made for the access layer. From a navigational perspective,
consistency prescribes similar navigation patterns (i.e., similar
sets of navigation links) within entities of the same type, within
similar collections, and across entities related by semantic connections
of the same nature.
Predictability
Predictability focuses on the user's ability to identify the meaning
of a structure or foresee the results of an interaction, having
seen a similar structure or operation in analogous situations. Predictability
measures the user's ability to form a predictive model of how the
content is organized and how the system behaves, and also to "learn"
complex situations after a significant amount of time spent with
the system. The main factors that contribute to predictability are
regularity, collection ordering coherency, and user's
knowledge conformance.
Regularity governs the degree of similarity in structures (and
behaviors and user control, not discussed here), not only in similar
situations as it is prescribed for consistency, but also across
the overall application. It can be measured in terms of use of recurrent
topological and navigational patterns, both for entities and collections.
Collection Ordering Coherency requires that the navigation
order among the members of a collection corresponds to the order
in which such members are described in the collection center (e.g.,
by means of a list of descriptors such as titles, icons, miniaturized
pictures, or similar). If the center shows the descriptors of two
members, say X and Y, one after the other, according to some visual
order, navigating forward from X the user expects to find Y, and
not something else.
User's Knowledge Conformance measures how well the elements
available in the application match the mental model of the intended
users, i.e., their set of assumptions, information, or preconceptions
about the application domain, about hypermedia in general, and about
the actual system. Clearly, different users have different mental
models, and measuring user's knowledge conformance of a system is
a very subjective process. The evaluator should verify if structures,
navigation, behavior and interaction metaphors available in the
application are appropriate for representing domain concepts, semantic
relationships, styles of use that can be considered somehow standard,
or at least intuitive, in a given cultural context and for the intended
class of users.
Accessibility
Accessibility measures how easy is for the users to locate
the piece(s) of information they are looking for. It promotes usability
particularly for users who have a specific information goal or must
perform a specific task. Accessibility means that, from a given
starting point, users can quickly locate the items that are needed
for their task, without navigating through non-relevant material;
or, alternatively, they can quickly discover that those items are
not in the application. A number of factors contribute to accessibility.
Some of them concern the completeness and efficacy of access structures.
Completeness refers to how the organization of access structures
and the content of their centers can be mapped to the set of hyperbase
entity types and to their instances. When users enter a WWW site,
for example, the home page, i.e., the center of the top level index
of the application, should immediately give a global sense of the
application content, and the perception of the hyperbase entity
types. If an entity type or its instances are not mentioned by any
collections, neither at the top level, nor at some level of nesting,
users might not discover them, unless they traverse some applicative
link (if any) in the hyperbase. The efficacy of access structure
organization is mainly related to the trade-off "breadth versus
depth" in nested collections (i.e., the number of collection members,
at a given level, versus the number of levels of sub-nesting). For
efficiency, it is often more appropriate to minimize the levels
of nesting. Another important factor for efficiency is the navigational
richness, i.e., the abundance of links connecting the various
information elements. Which links are more appropriate in which
situation is largely dependent on the nature and complexity of content.
In a short and simple-content guided tour, for example, the link
"next" - from a member to the following one, and its inverse ("previous"),
might be enough. Still, in a complex guided tour, made of several
steps and storing complex content, users may also need to jump to
the end or to the beginning of the tour, or to rapidly jump to any
member starting from the current one.
Orientation
Orientation is related to the "getting lost in the hyperspace" problem,
a common usability issue for large hypermedia. Orientation measures
the users' ability to understand their current location and their
own movements, to grasp their current navigation context, and to
return to previously visited items. A number of aspects can be considered
to support orientation, such as backtracking soundness, context
observability, and reuse soundness.
Backtracking is the mechanisms by which the user can return
back to previously visited nodes, and is one of the most common
orientation facilities in hypermedia, especially on the WWW (where
it is provided directly from the browser). Unfortunately, applications
very often provide the backtracking function in place of explicit
navigation links (e.g., as a surrogate for inverse links), thus
overburdening backtracking commands with a number of meanings.
Context Observability concerns the possibility for users
to evaluate the actual status of their navigation session, and to
understand the positioning of the current node within larger structures
or within the entire application. Many applications use active maps
and overview diagrams, with indications of the user's current location
(and of previous steps), both to summarize the application content
and to support orientation. Observability can also be supported
by means of some perceivable visual cues on the nodes. For example,
the relative position in a guided tour, or in a linear sequence
of nodes, can be presented by a descriptor, such as "3rd
of 10" (which also helps users to perceive how long is the way to
the last element).
Reuse Soundness analyses if reuse is well designed. Reuse
means that structures, links, and operations, already used somewhere,
are used again in a different context and for different purposes.
Reuse has a number of advantages but it may also have some critical
drawbacks. To avoid the increase of cognitive complexity and disorientation,
reused elements must not include any context-dependent feature,
or they must be adapted to the new context [Gar96].
For example, if a piece of content in a node depends on a given
context, it should be removed when reusing the node in another context
(and may be replaced with information needed by the new situation).
Alternatively, if there is important information that is left implicit
in a node and can be induced from a given context, it should be
made explicit when the node is reused in a new context.
3.3 Abstract Tasks
Abstract tasks describe the activities that, during the inspection,
the evaluators must perform in order to detect potential usability
problems. We use the term "abstract", since they are formulated
independently from a particular application, and they refer to categories
of application constituents.
An abstract task is described by five elements: the identification
code; the title; the focus of action, i.e.,
the constituents which are the focus of the activity; the activity
description, which describes what evaluators have to do, the
intent, i.e., a short statement explaining what is the rationale
of the abstract task. Optionally a comment, clarifying some
concepts or definitions mentioned in the previous elements, is provided.
The terminology used for the formulation of abstract task is based
on the HDM model.
The reader is deferred to [Gar98a] for a complete list of thirty abstract
tasks, which cover the complete set of features of an hypermedia
application. In the rest of this section we shortly introduce a
few examples, just to provide the reader with the "flavor" of what
abstract tasks are. The abstract tasks described here are mainly
concern with the primitives that in HDM are called access structures,
i.e., the collection of objects upon which indexes and guided tours
are built, that are commonly used in almost every WWW application.
In particular the first three abstract tasks (from AS-O1 to AS-O3)
refer to the organization of the access structures, while
the last four (from AS-N1 to AS-N2) refer to the navigation
within the access structures.
AS-O1
Title: "Extent of access structures"
Focus of Action: an entity type, and the set of collections.
Activity Description: consider an entity type (e.g.
"painting"):
1. verify if there are collections (indexes or guided tours) which
allow the direct access to its instances;
2. verify if there is at least a collection which allows the access
to all its instances.
Intent: to check if access methods efficiently support
access to the hyperbase entities.
Comment: a typical usability problem occurs when some
entity types, i.e., classes of application objects, are not directly
accessible by any collection (index or guided tour), and the only
way to access them is through links from other entities. Another
type of problem arises when the collections do not cover the complete
extension of the entity type, in the sense that some of the instances
are left out. Both types of problems are, surprisingly, quite common.
AS-O2
Title: "Information quality in collection centers"
Focus of Action: a collection center (i.e., the front
page of an index or a guided tour).
Activity Description: verify if the information content
in the collection center accurately describes the content of the
collection itself. For example, verify:
1. the completeness, i.e., if all the members of the collection
are clearly identifiable;
2. the correctness, i.e., if the description of each member
precisely describes the member itself;
3. the ordering, i.e., if the order in which members are
listed precisely corresponds to the navigation order in which members
are traversed in the collection.
Intent: to verify how well the center of a collection
supports user's understanding of what is and what is not in the
collection.
Comment: centers of collections, i.e. the page through
which indexes or guided tours are presented, are crucial for the
effective usability of application. Imprecise, incomplete, or low
quality information in a center, incurs fruitless user navigation
(in the sense that they access items not really relevant to them)
or can result in potentially interesting pieces of information being
overlooked.
AS-O3
Title: "Nesting levels of access structures"
Focus of Action: the set of collections (i.e. indexes
and guided tours).
Activity Description: starting from the home page,
reach the hyperbase objects (entities), following different paths
across indexes and guided tours.
Intent: to identify the access structure depth, and
verify if there are hierarchies of collections with a high degree
of nesting.
Comment: it is always a difficult trade-off to structure
a large application in the sense of "breadth" or "depth". Sometimes
nesting is a necessity, but it should be kept in mind that an high
level of nesting almost inevitably creates usability problems (in
the sense that the reader may never reach the bottom of the nesting,
or find his/her way up from the bottom).
AS-N1
Title: "Complexity of collection navigation patterns"
Focus of Action: a collection, and the set of collection
links.
Activity Description: in a collection:
1. verify the complexity of the collection navigation pattern. For
example:
- from the collection center, access a generic collection member;
- from a generic member of the collection, access:
- the previous and the next member in the collection
order;
- the first collection member and the last one;
- another arbitrary member;
- the collection center (if any);
- from the first (respectively, the last) member, try to go "previous"
(respectively "next");
2. verify the appropriateness of such a pattern, in accordance
with the collection content and structure.
Intent: to verify if the navigation pattern of a collection
is adequate for the content and the purpose of the collection itself.
Comment: again it is a difficult trade-off between
rich, complex navigation patterns, which can be difficult to handle,
and simplified, easily understandable navigation patterns, that
may lack the necessary power. This is an area where, despite the
difficulty of finding a proper design choice, technical mistakes
are easily found.
AS-N2
Title: "Up-navigation in nested collections"
Focus of Action: nested collections (indexes and/or
guided tours).
Activity Description: verify the existence and the
appropriateness of mechanisms for navigating "upwards" within nested
collections. For example, starting from the home, go down in the
hierarchy of collections, and then:
1. go up through the same indexes;
2. go up through different indexes (if any);
3. traverse applicative or structural links, then try to go up to
the higher level indexes, up to the home.
Intent: to assess if adequate navigation patterns
are provided for nested collections. The relevance of this task
clearly depends on the use (or abuse) of the nesting within the
application.
Comment: a typical problem that generally arises is
that poor mechanisms are provided for navigating upwards through
nested collections. Therefore, once users go down through the collection
hierarchy, they do not have any way to go up to the centers of higher
level indexes, and in particular to the home page.
AS-N3
Title: "Visibility of navigation status in collection
navigation"
Focus of Action: a collection (index or guided tour).
Activity Description: in a collection, access an arbitrary
member, and identify its position in the collection structure, uniquely
from what you see (forgetting the path that has been followed).
Intent: to verify if members are provided with clear
indications about their location in the collection.
Comment: identifying the position of a member within
a collection is a crucial aspect of user orientation. It becomes
even more important (and technically more difficult) if the same
member belongs to different collections or appears in different
positions within the same collection.
AS-N4
Title: "Backtracking versus collection navigation"
Focus of Action: a collection.
Activity Description: in a collection, verify if and
how backtracking is used within collection navigation. For example:
1. go to a generic intermediate member, without passing through
the previous members in the collection order; then access the previous
member;
2. go to a generic member, without passing through the collection
center; then go up to the center.
Intent: to assess the effect of using backtracking
commands as a surrogate for navigation commands.
Comment: backtracking, which returns the users back
to the previously visited node, is one of the most important and
common navigation facilities. As discussed by Nielsen [Nie95],
"almost all hypertext systems provide some form of backtracking,
but not always very consistent". Many hypermedia (especially on
the Web!) make an intense use of the backtracking function (oftentimes
called "back" or, ambiguously, "previous") to overcome the lack
of explicit navigation links. Although in some cases it may work,
in most cases it does not, and it can result in user disorientation.
4. Examples of Inspection Findings
In this paper we only limit ourselves to the inspection activity.
We have inspected a number of museum web sites, using our abstract
tasks for hypermedia, a sample of which has been described in the
previous section. The overall structure of the sites is quite simple
(in the worst cases, just a tree structure), in comparison, for
example, to structures that could be found in multimedia CD-ROM.
Simple structure can be easy to manage, but ineffective, from a
usability point of view. When more articulated and organized structures
are attempted, some usability problems seem to arise.
|
