User Evaluation: Sharing Expertise To Build Shared Values

David Dawson, ReSource;  Alice Grant, Alice Grant Consulting;  Paul Miller, JISC, UK; John Perkins, Mus-Info, Canada

Abstract

In January 2003, cultural organizations internationally were invited to submit user evaluations of digital resources for review and analysis by the Cultural Content Forum (CCF).   Established in 2002, the CCF exists to allow organizations with a strategic and/or national involvement in the creation of cultural digital resources to share knowledge and experience in the field.   Over 80 responses were received as a result of the call for submissions; these were initially the subject of a broad-based review as they were catalogued and descriptions of the evaluation projects made accessible on-line to the wider cultural community. Although the scope of the submissions received was not as wide as it might have been, many reports received were of a high standard, meaning that further, detailed analysis was possible. In October 2003, a second stage of research was completed; this explored the potential for proposing common methodologies and user profiles for use in evaluation projects.  

This paper describes the research in more detail, identifying key issues arising from the research and proposing a way forward for establishing a shared understanding of best practice for evaluation projects within the digital cultural sector. 

Keywords: user evaluation, user studies, metrics, comparative analysis

Introduction

In 2002, the Cultural Content Forum (http://www.culturalcontentforum.org) agreed to undertake research to identify and analyse material relating to the evaluation of digital cultural resources. The project was planned in two stages: the aims were:

• To research and publish a catalogue of evaluation undertaken relating to digital cultural information resources.
• To undertake and publish an analysis of available evaluation material. The aims of this analysis would be to:
  • identify common indicators and trends relating to the development and use of cultural information resources;
  • identify common issues relating to the provision of digital cultural resources;
  • identify gaps in available research and propose an evaluation research agenda for the future

In January 2003, a call was issued across the international library, digital library, museum and archive domains using professional e-mail lists. Significant interest in the project was stimulated; many of the contributors of the 86 documents received during the initial stage requested access to the results.

A dataset documenting the material received was prepared using the Dublin Core Metadata Element Set as a basis for the record structure. Headings were adapted for use to reflect the particular nature of the research, in particular to enable the recording of aspects of the evaluation methodologies used by different projects.

As a result of this process, the CCF agreed in their meeting in Pistoia in March 2003 to pursue the analysis of the material submitted. Following the production of an initial report outlining the scope and nature of the submissions received, a second general call for evaluation material was made, and key organisations were approached individually. This resulted in a limited amount of additional material to be added to the body of documents already submitted in July and September 2003 for analysis. The CCF commissioned this analysis to address two particular aspects of the evaluation material as follows:

• an assessment of the extent to which common user profiles were used in different evaluation projects and whether these could form the basis of a standardised approach;
• an assessment of any metrics used in evaluation projects which could form the basis for a benchmark for assessing digital resources.

User Profiles

For the purposes of this report, a user profile is a means of categorising and defining a user (or potential user) of a digital resource, using one or more different attributes to do so. The term user profile is frequently used to describe the process by which individuals’ access and use of specific applications is configured, managed and monitored, particularly in technical terms. This use of the expression does not fall within the remit of this analysis, however.

Of the 94 documents analysed, 27 (22%) either did not use user profiles, or had no discernible intention to target a specific, named audience group. Many (but not all) of this number were theoretical papers or quantitative analyses of statistics. The remaining 78% of the sample either used a profile or, more often, stated the intended audience for their resource.

User profiles were used at different stages of the projects described in the material. The following were stages in the process where user profiles were defined prior to the use or evaluation of a resource, and where users were sought to fit the specified profiles:

• definition of the scope of a project or resource during the project development process;
• describing and grouping users involved in the formative evaluation process;
• describing and grouping users evaluating a resource after development.

User profiles were also used to categorise users reactively, in the following ways:

• grouping users and non-users consulted in market research;
• categorising users who used or provided feedback on resources.

The user profiles identified can be divided into three general types as follows:

• General descriptions of users
• Descriptions of users within the digital and/or cultural sector
• Descriptions of users within individual organisations

User profile types

• Origin, ethnic or cultural categories
  • descriptions of users relating to their cultural background, ethnicity or country of origin.
• Visit categories
  • Description of whether the user intends to visit or has visited a museum, library or archive, and the use of the resource in relation to that visit (or non- visit).
• Formal learners
  • Users in some kind of formal education: schools, tertiary education (16-18), further education and higher education.
• Informal learners
  • Users wishing to develop their personal knowledge either for reasons of leisure or to pursue self-directed learning outside the scope of any formal education organisation.
• Socio-economic categories
  • Categorisation of users by standard demographics, including age and gender
• Work-related categories
  • Users whose occupation reflected specific information requirements and modes of use.
• Users with special needs
  • Users with sensory, physical or learning difficulties which may affect the way in which they are able to use a resource.
• Access to ICT
  • The way in which users access on-line resources, including technical considerations, availability of hardware and connectivity.

A number of problematic issues were identified regarding any common approach to user profiles within the cultural sector. These included the following significant issues:

• Granularity of user profiles
• Identifying and handling multiple profiles
• Uneven coverage

Proposing a user profile matrix

The analysis of the use of user profiles clearly demonstrates the need to record multiple attributes in order to build a full user profile. In order to accommodate this need, it is proposed that a matrix approach should be adopted, whereby a provider might build a detailed user profile by selecting characteristics from more than one of the User profile types

Further research

A number of issues need to be addressed before there will be agreement on a standard which the wider cultural community feels able to implement. They include the following issues which contributors should be invited to discuss during the consultation outlined above:

• terminology used to describe user profiles, including the definition of a user profile (as opposed to the different user attributes which might comprise a full profile)
• the definitions of the more general descriptions, as well as those which appear to be overlapping
• the scope and evenness of coverage of the existing material: unevenness  remains an issue, with little input from some areas of the cultural sector

Metrics

What are metrics?

In the context of the current analysis, metrics refers to ways in which standard measurements can be applied within the methodologies and results of digital resource evaluation. The aim of this stage of the analysis was to identify common approaches to the use of metrics within evaluation exercises and to explore ways in which these approaches could be implemented across the digital cultural sector. The potential end result of this process would be one or more indices which could be applied to evaluations in order to gain an understanding of their comparative performance in relation to other services.

Methodologies and metrics used within the evaluation reports

A wide range of methodologies was used within the evaluation reports.

Issues in the definition and implementation of metrics within evaluation studies

Overall, the submissions to the project did not enable firm recommendations to be made on how metrics might be developed for implementation across the sector. The reasons for this are as follows:

• many submissions did not cite any methodological information at all;
• of those submissions which contained methodological information, few provided a sufficiently detailed account of the methodologies used in evaluation  to permit its use as the basis for a proposal;
• those which did provide detailed accounts of methodologies, or which proposed methodologies in detail, did so in a highly specific and/or theoretical manner which was inappropriate for consideration as a scheme suitable for t implementation in other organisations.

Areas of Strength

• Market research
  The market research material submitted was specifically focused – it addressed users’ responses to digital resources; this was not simply one aspect of broader research exercises.
• Task analysis
  A number of projects included task analysis as elements within evaluation projects. These projects were potentially informative in terms of the specification of the activities users were asked to carry out on applications being evaluated. In one instance, highly detailed information was provided about the observation process.
• Observation and/or interview
  It was the number of projects (over 40 projects used one or both of these techniques) which demonstrated the extent to which face-to-face and/or one-to-one contact is regarded as a key element in successful evaluation.

Areas of Weakness

The following aspects of the evaluation process were under-represented in the evaluation material submitted:

• Non-user surveys
  Very few evaluation projects explored non-users of digital cultural resources. For some providers (e.g. higher education), it could be argued that their core user base is a strong one which is highly motivated to use digital resources. However, for most domains within the cultural sector there would be clear benefits to be gained from developing a greater understanding of the motivations, priorities and interests of its non-users.
• Impact evaluations
  Only one report dealt with the issue of how cultural organisations affected the subsequent actions and activities of users. No report explored the impact specifically of cultural digital resources. The significance of this type of research is that it provides insights into user behaviour and behaviour changes brought about by resources, and this should inform future development.
• Understanding the use of resources within users’ activities
  Developing a greater understanding of the use of resources within user activities is a specific aspect of impact evaluation which deserves greater attention in evaluation, and which was not explored in any of the reports submitted. There is evidently a paucity of understanding of how users might, for example, incorporate the use of digital resources into a research project, a school homework project or a plan for a ‘real’ visit to a museum, archive or library. Knowledge about how users use information from digital resources may affect the modes of access to, and presentation of, digital resources.

The analysis demonstrated that there exists an opportunity to fill a gap by developing guidelines for best practice, since the amount and quality of evaluation, both formative and summative, applied to most projects is inadequate as follows:

• given the investment in digital resources, insufficient investment is made in the evaluation of individual projects;
• where evaluation is undertaken, the scope and nature of the evaluation methods deployed often results in less information being gleaned from the process than should be possible.

Recommendations

The following are recommendations for addressing evaluation issues which will need to be considered by the digital-cultural community as it prepares to improve and extend the way in which it evaluates its on-line resources.

Recommendation 1: Promote skills and training

Cultural organisations should promote the development of appropriate skills within existing staff, providing training where necessary.

Recommendation 2: Promote organizational change

Cultural organizations should take formal account of the need for evaluation when planning and funding digital projects

Recommendation 3: Raise awareness of evaluation

The CCF should work to raise awareness of the need for evaluation and promote good evaluation practice

Recommendation 4: Increase openness and sharing of evaluation results

Cultural organizations should be more willing to share their approaches to, and results of, evaluation projects.

The review has found strong evidence that there is an emerging consensus for the use of common user profiles across the sector, but that the development of any standard profiles should reflect the need for multiple attributes to be assigned to groups of users.

The analysis of evaluation material found, however, that there is little if any evidence of a common approach to metrics in place at the present time, and that this is unlikely to become a possibility until the sector has reached consensus on the methodologies to be used in evaluation projects and the integration of evaluation results into the project development cycle.

At its next meeting, the Cultural Content Forum will consider how to address these recommendations, and identify key priorities for future research.