158 Lee Avenue
M4E 2P3 Canada
ph: +1 416-691-2516
fx: +1 416-352-6025
info @ archimuse.com
published: March 2004
analytic scripts updated:
October 28, 2010
User Evaluation: Sharing Expertise To Build Shared Values
David Dawson, ReSource; Alice Grant, Alice Grant Consulting; Paul Miller, JISC, UK; John Perkins, Mus-Info, Canada
In January 2003, cultural organizations internationally
were invited to submit user evaluations of digital resources for review and
analysis by the Cultural Content Forum (CCF). Established in 2002, the CCF
exists to allow organizations with a strategic and/or national involvement
in the creation of cultural digital resources to share knowledge and experience
in the field. Over 80 responses were received as a result of the call for
submissions; these were initially the subject of a broad-based review as they
were catalogued and descriptions of the evaluation projects made accessible
on-line to the wider cultural community. Although the scope of the submissions
received was not as wide as it might have been, many reports received were
of a high standard, meaning that further, detailed analysis was possible.
In October 2003, a second stage of research was completed; this explored the
potential for proposing common methodologies and user profiles for use in
This paper describes the research in more detail, identifying
key issues arising from the research and proposing a way forward for establishing
a shared understanding of best practice for evaluation projects within the
digital cultural sector.
Keywords: user evaluation, user studies, metrics, comparative analysis
In 2002, the Cultural Content Forum (http://www.culturalcontentforum.org)
agreed to undertake research to identify and analyse material relating to
the evaluation of digital cultural resources. The project was planned in two
stages: the aims were:
- To research and publish a catalogue of evaluation undertaken
relating to digital cultural information resources.
- To undertake and publish an analysis of available
evaluation material. The aims of this analysis would be to:
- identify common indicators and trends relating to the development
and use of cultural information resources;
- identify common issues relating to the provision of digital
- identify gaps in available research and propose an evaluation
research agenda for the future
In January 2003, a call was issued across the international
library, digital library, museum and archive domains using professional e-mail
lists. Significant interest in the project was stimulated; many of the contributors
of the 86 documents received during the initial stage requested access to
A dataset documenting the material received was prepared
using the Dublin Core Metadata Element Set as a basis for the record structure.
Headings were adapted for use to reflect the particular nature of the research,
in particular to enable the recording of aspects of the evaluation methodologies
used by different projects.
As a result of this process, the CCF agreed in their meeting
in Pistoia in March 2003 to pursue the analysis of the material submitted.
Following the production of an initial report outlining the scope and nature
of the submissions received, a second general call for evaluation material
was made, and key organisations were approached individually. This resulted
in a limited amount of additional material to be added to the body of documents
already submitted in July and September 2003 for analysis. The CCF commissioned
this analysis to address two particular aspects of the evaluation material
- an assessment of the extent to which common user profiles were
used in different evaluation projects and whether these could form the basis
of a standardised approach;
- an assessment of any metrics used in evaluation projects which
could form the basis for a benchmark for assessing digital resources.
For the purposes of this report, a user profile is a means of categorising
and defining a user (or potential user) of a digital resource, using one or
more different attributes to do so. The term user profile is frequently
used to describe the process by which individuals’ access and use of specific
applications is configured, managed and monitored, particularly in technical
terms. This use of the expression does not fall within the remit of this analysis,
Of the 94 documents analysed, 27 (22%) either did not use user profiles,
or had no discernible intention to target a specific, named audience group.
Many (but not all) of this number were theoretical papers or quantitative
analyses of statistics. The remaining 78% of the sample either used a profile
or, more often, stated the intended audience for their resource.
User profiles were used at different stages of the projects described in
the material. The following were stages in the process where user profiles
were defined prior to the use or evaluation of a resource, and where users
were sought to fit the specified profiles:
- definition of the scope of a project or resource during the
project development process;
- describing and grouping users involved in the formative evaluation
- describing and grouping users evaluating a resource after development.
User profiles were also used to categorise users reactively, in the following
- grouping users and non-users consulted in market research;
- categorising users who used or provided feedback on resources.
The user profiles identified can be divided into three general types as follows:
- General descriptions of users
- Descriptions of users within the digital and/or cultural sector
- Descriptions of users within individual organisations
User profile types
- Origin, ethnic or cultural categories
- descriptions of users relating to their cultural background,
ethnicity or country of origin.
- Visit categories
- Description of whether the user intends to visit or has visited
a museum, library or archive, and the use of the resource in relation to that
visit (or non- visit).
- Formal learners
- Users in some kind of formal education: schools, tertiary education
(16-18), further education and higher education.
- Informal learners
- Users wishing to develop their personal knowledge either for
reasons of leisure or to pursue self-directed learning outside the scope of
any formal education organisation.
- Socio-economic categories
- Categorisation of users by standard demographics, including
age and gender
- Work-related categories
- Users whose occupation reflected specific information requirements
and modes of use.
- Users with special needs
- Users with sensory, physical or learning difficulties which
may affect the way in which they are able to use a resource.
- Access to ICT
- The way in which users access on-line resources, including technical
considerations, availability of hardware and connectivity.
A number of problematic issues were identified regarding any common approach
to user profiles within the cultural sector. These included the following
- Granularity of user profiles
- Identifying and handling multiple profiles
- Uneven coverage
Proposing a user profile matrix
The analysis of the use of user profiles clearly demonstrates the need to
record multiple attributes in order to build a full user profile. In order
to accommodate this need, it is proposed that a matrix approach should be
adopted, whereby a provider might build a detailed user profile by selecting
characteristics from more than one of the User profile types
A number of issues need to be addressed before there will be agreement on
a standard which the wider cultural community feels able to implement. They
include the following issues which contributors should be invited to discuss
during the consultation outlined above:
- terminology used to describe user profiles, including the definition
of a user profile (as opposed to the different user attributes which might
comprise a full profile)
- the definitions of the more general descriptions, as well as
those which appear to be overlapping
- the scope and evenness of coverage of the existing material:
unevenness remains an issue, with little input from some areas of the cultural
What are metrics?
In the context of the current analysis, metrics refers to ways in
which standard measurements can be applied within the methodologies and results
of digital resource evaluation. The aim of this stage of the analysis was
to identify common approaches to the use of metrics within evaluation exercises
and to explore ways in which these approaches could be implemented across
the digital cultural sector. The potential end result of this process would
be one or more indices which could be applied to evaluations in order to gain
an understanding of their comparative performance in relation to other services.
Methodologies and metrics used within the evaluation reports
A wide range of methodologies was used within the evaluation reports.
Type of evaluation
Desk-based assessment of a resource
against established standards.
Paper concerned with exploring
or developing evaluation methodology rather than evaluating a specific
Evaluation in the conceptual
and development stages of a project aimed at identifying user needs
and ensuring that the planned resource will meet these needs.
Evaluation investigating the
extent to which a resource has affected a user community, its activities
or the way it behaves.
Exploration of the potential
audience for a resource which aims to scope the structure of the market
and identify the primary user groups.
Consultation with user groups
to assess their responses to a potential new resource or service.
Consultation of organisations
using or providing digital resources.
Analysis of retrieval queries
submitted to a database.
Assessment of projects undertaken
as a research exercise as opposed to with users.
Evaluation undertaken to assess
the use of a completed resource in order to determine the extent of
its success, to establish how it is being used and what lessons might
be carried forward to future projects.
Evaluation of a prototype or
initial completed version of a resource to identify development errors
and/or the extent to which users are able to use the resource effectively
and accurately. May cover programming and design/navigation issues.
Evaluation to study the ways
in which users use a particular resource, aimed at developing a greater
understanding of users rather than evaluating a specific resource, although
the results may inform the way in which a resource is developed.
Consultation to establish user
needs at the outset of the development of a project.
Web site analysis
Assessment of existing Web sites
and their use for comparative, research and survey purposes.
Web site usage statistics
showing numbers, domains etc. of Web site users.
in the definition and implementation of metrics within evaluation studies
Overall, the submissions to the project did not enable firm recommendations
to be made on how metrics might be developed for implementation across the
sector. The reasons for this are as follows:
- many submissions did not cite any methodological information
- of those submissions which contained methodological information,
few provided a sufficiently detailed account of the methodologies used in
evaluation to permit its use as the basis for a proposal;
- those which did provide detailed accounts of methodologies,
or which proposed methodologies in detail, did so in a highly specific and/or
theoretical manner which was inappropriate for consideration as a scheme suitable
for t implementation in other organisations.
Areas of Strength
- Market research
The market research material submitted
was specifically focused – it addressed users’ responses to digital resources;
this was not simply one aspect of broader research exercises.
- Task analysis
A number of projects included task
analysis as elements within evaluation projects. These projects were potentially
informative in terms of the specification of the activities users were asked
to carry out on applications being evaluated. In one instance, highly detailed
information was provided about the observation process.
- Observation and/or interview
It was the number of projects (over
40 projects used one or both of these techniques) which demonstrated the extent
to which face-to-face and/or one-to-one contact is regarded as a key element
in successful evaluation.
Areas of Weakness
The following aspects of the evaluation process were under-represented in
the evaluation material submitted:
- Non-user surveys
Very few evaluation projects explored
non-users of digital cultural resources. For some providers (e.g. higher education),
it could be argued that their core user base is a strong one which is highly
motivated to use digital resources. However, for most domains within the cultural
sector there would be clear benefits to be gained from developing a greater
understanding of the motivations, priorities and interests of its non-users.
- Impact evaluations
Only one report dealt with the issue
of how cultural organisations affected the subsequent actions and activities
of users. No report explored the impact specifically of cultural digital resources.
The significance of this type of research is that it provides insights into
user behaviour and behaviour changes brought about by resources, and this
should inform future development.
- Understanding the use of resources within users’ activities
Developing a greater understanding
of the use of resources within user activities is a specific aspect of impact
evaluation which deserves greater attention in evaluation, and which was not
explored in any of the reports submitted. There is evidently a paucity of
understanding of how users might, for example, incorporate the use of digital
resources into a research project, a school homework project or a plan for
a ‘real’ visit to a museum, archive or library. Knowledge about how users use information from digital resources may affect the modes of access
to, and presentation of, digital resources.
The analysis demonstrated that there exists an opportunity to fill a gap
by developing guidelines for best practice, since the amount and quality of
evaluation, both formative and summative, applied to most projects is inadequate
- given the investment in digital resources,
insufficient investment is made in the evaluation of individual projects;
- where evaluation is undertaken, the scope and nature of the
evaluation methods deployed often results in less information being gleaned
from the process than should be possible.
The following are recommendations
for addressing evaluation issues which will need to be considered by the digital-cultural
community as it prepares to improve and extend the way in which it evaluates
its on-line resources.
Recommendation 1: Promote skills and training
Cultural organisations should promote
the development of appropriate skills within existing staff, providing training
Recommendation 2: Promote organizational change
Cultural organizations should take
formal account of the need for evaluation when planning and funding digital
Recommendation 3: Raise awareness of evaluation
The CCF should work to raise awareness
of the need for evaluation and promote good evaluation practice
Recommendation 4: Increase openness and sharing of evaluation results
Cultural organizations should be
more willing to share their approaches to, and results of, evaluation projects.
The review has found strong evidence that there is an emerging
consensus for the use of common user profiles across the sector, but that
the development of any standard profiles should reflect the need for multiple
attributes to be assigned to groups of users.
The analysis of evaluation material found, however, that
there is little if any evidence of a common approach to metrics in place at
the present time, and that this is unlikely to become a possibility until
the sector has reached consensus on the methodologies to be used in evaluation
projects and the integration of evaluation results into the project development
At its next meeting, the Cultural Content Forum will consider how to address
these recommendations, and identify key priorities for future research.