Accessing Museums Through the Web: A Model for Evaluating the Impact of Museum and School Partnerships

Dianna L. Newman, The Evaluation Consortium, University at Albany, State University of New York and Patricia Barbanell and John Falco, Schenectady City Schools, USA

http://www.projectview.org

Abstract

The purpose of this paper is to present an overview of the process used to evaluate the design and implementation of museum-supported videoconferencing and demonstrate selected videoconference uses and outcomes for both museum educators and classroom teachers. Project VIEW, a federally funded Technology Innovation Challenge Grant, located at Schenectady City Schools, has developed an innovative program that has as its objective the sharing of museum resources by bringing the museum to the classroom via interactive, point-to-point videoconferencing. Under this model, core groups of museum educators and teachers collaboratively develop curriculum units that integrate museum staff and supporting artifacts into classroom instruction through the use of videoconferencing via the web.

Utilizing a stakeholder based participatory model of evaluation, the project has been able to provide program staff, local educators, and museum teams formative and summative information on the process of designing and implementing videoconferences as well as information on the direct outcomes and systemic organizational changes that have occurred as a result of participation. As part of this process, the evaluation effort also documents best practices in the provision of professional development for museum educators and classroom teachers. The focus of this effort is to document model methods of training museum educators and classroom teachers on the development and the use of point-to-point videoconferencing. These formative data are utilized by museum administrators and project staff to improve the preparation of curriculum and the presentation of the videoconferences; summative data are used to document long-term impact of the training and the subsequent use of collaborative videoconferencing the museum and the school.

Key Words: Videoconferencing, Evaluation, Partnerships, K-12 Educators, Museum Educators

Introduction

This paper presents an overview of the process used to evaluate the design and implementation of museum-supported videoconferencing and demonstrate selected videoconference used and outcomes for both museum educators and classroom teachers. Over the past several years, severe budget cuts and growing safety concerns have impacted the typical classroom's ability to physically access the rich resources available in museums, environmental centers, and active learning centers. Paradoxically, the mission of many museums, as reflected by their mission statements and their subsequent allocation of staff and resources, have changed to reflect a growing emphasis on service to educational settings. Project VIEW, a federally funded Technology Innovation Challenge Grant, (United States Department of Education Award Number R3030A000002) located at Schenectady City Schools, has developed an innovative program that has as its objective the sharing of museum resources by bringing the museum to the classroom via interactive, point-point videoconferencing. Under this model, core groups of museum educators and teachers collaboratively develop curriculum units that integrate museum staff and supporting artifacts into classroom instruction through the use of videoconferencing via the web. In addition to designing a method of developing curriculum, the project also has designed a professional development program that trains classroom teachers and museum educators in point-to-point videoconferencing and has established a method of evaluating the efforts that support the program and the resulting outcomes to teachers and students. (Geehan, & Meuwissen, 2001). Utilizing a stakeholder based participatory model of evaluation, the project has been able to provide program staff, local educators, and museum teams formative and summative information on the process of designing and implementing videoconferences as well as information on the direct outcomes and systemic organizational changes that have occurred as a result of participation. (Newman, Spaulding, Catapano, 2002).

Background of the project and related literature

Computer and information technologies play increasingly important roles in education at all grade levels and within all subject areas. From Internet-based research to telecommunications exchanges, learners and instructors have, at their disposal, an abundant and expanding base of resources that bring information into classrooms, libraries, and homes quickly and efficiently. Videoconferencing, for example, is a medium through which great potential exists for interaction among students, educators, and the subject matter. Unlike distance learning, which features a teacher at one location and students at another, videoconferencing maintains direct interaction between educators and students in a classroom setting while allowing content experts or other educators to become part of the classroom through audio and video communication. Moreover, videoconferences foster an environment in which ideas can be exchanged through a variety of means including: 1) point-to-point communications, in which one site establishes direct audio and video interactions with another site; and 2) multi-presence communications, in which one site acts as a hub for communications that take place among multiple locations. (Berg, 1999, Hildebrand, 1995; Klemm, 1998).

Prior research has shown that computer networks are model channels for fostering collaborative learning and the expansion of educational resources (Hardwick, 2000). As a medium that utilizes computer networking for instant communication, videoconferences are means of collaborative education through which valuable resources can be brought into classrooms. Nonetheless, studies have concluded that interactive, computer-based environments are sometimes prone to inefficient delivery of information (Furst-Bowe, 1997), with that, when compared with student and teacher anxiety, and a lack of motivation to master the subject matter and manipulate the learning environment, they frequently lead to a lack of participation (Bures, 2000). Moreover, some argue that the benefits of educational technologies are hindered by a lack of collaboration among practitioners, researchers, decision makers, and subject matter experts (Clark & Estes, 1999) and overgeneralizations with regard to perceived advantages, such as improved student learning, multicultural awareness, skill mastery, and future economic possibilities (Fabos & Young, 1999). In the past, it was assumed that the mere presence of computers and other information technology within classrooms was a testimony to their effectiveness as instructional aides. Today, however, educators must be able to justify the acquisition of new technologies by demonstrating that it overtly facilitates student attainment, performance, participation, and retention.

The role of videoconferencing as one of these technologies is increasingly important. Prior to the development of infrastructure that is capable of transmitting both video and audio data over analog and digital telephone and cable lines, text-based electronic conferences were used for educational purposes. This medium has been shown to be an effective means of education, provided the instructor: 1) ensures that students are comfortable with and able to use technology; and 2) solidifies the connection between learning course material and effectively utilizing the medium (Bures, 2000). With the ability to transmit video and audio, the "virtual classroom" has become a reality, where teaching and learning take place within a computer-mediated communication system (Hiltz, 1994). This learning environment, compared to the environment that is solely text-based, provides greater opportunity to learn through exploration and inquiry and to communicate and cooperate with others. (Edelson, 1999; Mioduser, 2000).

Specifically, the utilization of the videoconference as an educational medium is reported to have a variety of benefits. Included among these benefits are: 1) the provision of equity of access to subject matter experts and other instructional resources (Omatseye, 1999); 2) the potential for in-depth collaboration of participants from various backgrounds (Wilson, 2000); 3) the ability for students to adopt a more collaborative, contextual environment in comparison to other forms of distance education (Peplow, 1996); 4) increased student interest in and understanding of the perspectives brought to the videoconference by other participants (Peat, 2000); and 5) the potential to reproduce a multimedia learning environment, incorporating video, discussion, activities, texts, and graphics, by videotape (Paul et al., 2000).

Nonetheless, critics have proposed that videoconferences pose unique difficulties that traditional learning environments do not. For example, exchanges can lead to confusion or non-participation due to cultural or communication barriers, anxiety, and technological incompatibility (Telg, 1996). Moreover, it was reported that the time and energy required to prepare and implement effective videoconference-based instruction greatly exceeds those elements in comparison to a more traditional classroom setting, regardless of the style of instruction (Boettcher, 1997). In addition, no conclusive evidence exists that connects videoconferencing with improvements in student achievement, and there is little understanding of or research about the most effective methods of utilizing conference-based education technology (Ess, 2000). Thus, educational videoconferencing, particularly in relation to student performance and longitudinal impact, is an area that would benefit from further research.

Methodology

A multiphase, mixed methodology plan, which included the collection of both formative and summative data from multiple stakeholder groups, was developed and employed to support the evaluation of this program. Both qualitative and quantitative methods were employed as means of meeting the needs of project decision makers. Data from stakeholders, consisting of educators, content provider personnel, students, and project staff, were solicited through paper-pencil surveys, interviews, and observations. Formative data were used to identify key strengths and concerns and recommend programmatic improvements based on stakeholder needs and program objectives. Summative data served as a means of: 1) documenting the number of products created, the number of videoconferences implemented, and the outcomes of those processes; and 2) establishing a foundation for studying longitudinal systemic change across the lifetime of the program.

The following is a summary of the data sources and collection methods used, as well as target stakeholders from whom information was gathered, for the evaluation:

Building Capacity for Videoconferencing:

Archival data were obtained from project staff to document the number of schools and content providers supported through the project and the extent to which resources were provided to each. These data assisted the evaluation team in documenting the establishment of foundations for change in schools' and museums' awareness and utilization of videoconference technology and their assessments of its value.

Interviews were conducted with project staff to verify the approaches utilized to build capacity for videoconferencing and the impacts of those activities. These data were utilized to document the channels by which resources in schools and content providers were changed to accommodate the use of videoconferencing technology in classroom settings.

Interviews were conducted with content provider and district-level support personnel to document the resources that were currently being instated or utilized that had not been available before the implementation of Project VIEW capacity building efforts. These data were employed to document the impact of videoconferencing technology on participating districts and informal education providers.

Creating Videoconferences and Related Curricula

Archival data were obtained from project staff to document the number of teachers and museum educators trained in videoconference development through the project and the number of videoconferences and related curriculum units produced. These data provided evaluators with the information necessary to conduct the summative evaluation of videoconference and curriculum development efforts.

Interviews were conducted with Project VIEW trainers to document the goals and objectives of videoconference and curriculum generation efforts and perceptions surrounding the professional development activities that were planned and implemented by project coordinators. These data were utilized to assist in mapping professional development needs and compare the intended training outcomes to the actual results.

Pre-training and mid-training group interviews were conducted with teacher teams and museum personnel to document the process of videoconference and curriculum development and participant perceptions of that process and its interim results. These data were pivotal in defining teacher and museum personnel perceptions of Project VIEW and the congruency of their goals and objectives, perceived strengths and barriers surrounding videoconferences and Virtual Field Trips, and content provider resources that were utilized in the development process.

Post-training interviews were conducted with teacher developers and museum personnel to document the outcomes of the videoconference and curriculum development process and plans for utilization. Data served to provide feedback for formative evaluation purposes, specifically with regard to the utility of the training and resultant products, the resources and materials provided for videoconference and curriculum development assistance, and the need for additional resources and training.

Observations of videoconference and curriculum development sessions were conducted at different points in the training process. Data were used to document the implementation of the professional development model and identify strengths and concerns surrounding the approaches to videoconference and curriculum development.

Utilization of Videoconferences and Related Curricula

Archival data were obtained from project staff to document the number of teachers trained in videoconference use through the project and the number of videoconferences implemented in participating schools. These data provided evaluators with the means to summatively evaluate videoconference integration in classroom instruction and training designed to improve such integration efforts.

Interviews were conducted with Project VIEW trainers to determine the goals and objectives of the teacher training and anticipated outcomes. Combined with observations of teacher training workshop sessions, these data were utilized to document evidence of training effectiveness and compare anticipated outcomes to actual results.

Pre-training and post-training paper-pencil surveys were distributed to workshop participants to document outcomes of the videoconference utilization training, changes in perceptions and competencies related to videoconferencing, and noted barriers and needs concerning videoconference implementation. These data were utilized to provide project coordinators with formative feedback concerning training design and implementation and assess the effectiveness of the workshops.

Interviews and paper-pencil surveys were conducted with teachers who utilized videoconferences within their classrooms to assess perceptions of the broadcasts and related curricula, the integration of informal education resources, and technology utilization. These data were used to assess the strengths and concerns associated with videoconference implementation; the utility of videoconferences in instruction, and additional needs concerning videoconference integration.

Group interviews and paper pencil-surveys were utilized with students who participated in project-sponsored point-to-point videoconferences to measure their perceptions and recollections of the programs and document their activities during the broadcasts. These data were used to assess the impact of videoconference technology on student learning and evaluate the facilitators and barriers to using interactive video as an instructional tool.

In-class observations were conducted at project-sponsored videoconferences to document utilization of the interactive media and curriculum resources. These data were used to assess the instructional approaches that were used in conjunction with the video broadcast tools, compare the intended outcomes of videoconference utilization from the perspectives of both user and developer, and verify strengths and concerns associated with broadcast implementation.

Summary of Findings

Data presented in this section represent only one component of the project. For complete findings please contact Project View at www.projectview.org.

Finding: Overall, multiple school districts, and content providers were afforded the means to engage in videoconferences. These organizations represent a diversity of content providers as well as schools.

Table 1: Capacity Building Resources Provided to School Districts and Content Providers By Project VIEW

Project VIEW program staff worked to build capacity by providing equipment, funding, and technical support to urban, suburban & rural school districts, and a diversity of content providers. Specifically, these institutions were allocated videoconferencing tools, including computers, cameras, monitors, and software, assistance in installing, piloting, and troubleshooting equipment, and the funding and connectivity necessary to communicate via ISDN and IP networks.

At the current time, a majority of schools and content providers that were provided with financial, personnel, and equipment resources have full capacity to videoconference in at least one location. Prior to the provision of these resources, these institutions lacked the necessary hardware, software, and knowledge required to engage the videoconference medium effectively.

Finding: In addition to specific assistance to schools and content providers, Project VIEW coordinators initiated long-term efforts to support comprehensive systemic change in technological capacity among program participants.

Table 2: Initiation of Systemic Change to Develop Technology Infrastructures

Overall, project staff initiated systemic change within a number of schools, museums, and other informal education providers. The initiation phase of the systems change process incorporated the development and strengthening of a shared vision and common goals among Project VIEW stakeholders.

Finding: Overall, educators from multiple school districts have participated in the process of developing videoconferences and supporting curricula through Project VIEW. The educators who received professional development in the creation of interdisciplinary videoconferences and curriculum products represent multiple grade levels and content areas.

Finding: Respondents generally noted a high level of collaboration between teacher development teams and museum personnel, increasingly reporting a common vision and a defined set of goals and objectives for educational videoconferencing.

Finding: Teachers who were trained through Project VIEW in the creation of videoconference and curriculum products have generated multiple replicable, interdisciplinary videoconference programs and related curriculum units; all completed modules are currently being used within school settings.

Table 3: Characteristics of Sample Products Created Through Project VIEW

Conclusion

Throughout the first three years of program implementation, Project VIEW has effectively linked a consortium of demographically varied school districts with multiple diverse, content-rich informal education providers to establish the groundwork for educational videoconferencing that brings unique resources to students who may not otherwise have access to them. The multifaceted nature of the project, which seeks to both increase access to videoconferencing as an educational tool and improve the use of that tool, has served as a foundation for initiating comprehensive change in educational technology integration. In general, stakeholders perceived that educational videoconferencing served to benefit schools by increasing their access to authentic and exceptional resources, from historic and scientifically unique locales, while informal education providers were endowed with the means to publicize their unique resources and provide greater access to the public. Moreover, Project VIEW has begun to build collaborative partnerships among institutions with varied interests, promoting a common purpose for the utilization of informal education providers through educational videoconferencing.

References

Anderson, T. & Garrison, D. R. (1995) Critical thinking in distance education: developing critical communities in an audio teleconference context. Higher Education, 29, 183-199.

Bellamy, R. (1999) Design experiments with media-rich messaging. Journal of Interactive Learning Research, 10, 149-177.

Berg, C. (1999) Software to facilitate collaborative inquiry and on-line communities of learners. A paper presented at the 10^th annual society for information technology and teacher education international conference, San Antonio, TX: February 28-March 4, 1999.

Boettcher, J. V. & Cartwright, G. P. (1997). Designing and supporting courses on the web. Change, 29, 10-16.

Bures, E. M. (2000) Student motivation to learn via computer conferencing. Research in Higher Education, 41, 593-622.

Chen, A. Y. (1999) Teaching, learning, and inquiry strategies using computer technology. Journal of Computer Assisted Learning, 15, 162-172.

Clark, R. E. & Estes, F. (1999) The development of authentic educational technologies. Educational Technology, 39, 5-16.

Edelson, D. C. (1999) Addressing the challenges of inquiry-based learning through technology and curriculum design. Journal of the Learning Sciences, 8, 391-450.

Ess, C. (2000) Wag the dog? Online conferencing and teaching. Computers and the Humanities. 34, 297-310.

Fabos, B. & Young, M. (1999) Telecommunications in the classroom: rhetoric versus reality. Review of Educational Research, 69, 217-259.

Freeman, M. (1998) Video conferencing: a solution to the multi-campus large classes problem? British Journal of Educational Technology, 29, 197-210.

Furst-Bowe, J. (1997) Comparison of student reactions in traditional and videoconferencing courses in training and development. International Journal of Instructional Media, 24, 197-205.

Hardwick, S. W. (2000) Humanising the technology landscape through a collaborative pedagogy. Journal of Geography in Higher Education, 24, 123-130.

Hildebrand, J. (1995) Videoconferencing in the business curriculum. Journal of Business and Technical Comunication, 9, 228-241.

Hiltz, S. R. & Turoff, M. (1994). Virtual classroom plus video: technology for educational excellence. A paper presented at the 1994 Educational Multimedia and Hypermedia World Conference, Vancouver, BC. June 25-30, 1994.

Hurley, J. M. (1999) Collaborative inquiry at a distance: using the internet in geography education. Journal of Geography, 98, 128-140.

Jacobs, G. (1997) Remote teaching with digital video: a trans-national experience. British Journal of Educational Technology, 28, 292-304.

Klemm, W. R. (1998) Eight ways to get students more engaged in online conferences. T H E Journal, 26, 62-66.

McKnight, M. (2000) Distance education: expressing emotions in video-based classes. A paper presented at the 51^st annual meeting of the conference on college composition and communication, Minneapolis, MN: April 12-15, 2000.

Meuwissen, K.W. & Geehan, M. M. (2001) Creating a Model for Educational Videoconferencing: Three Approaches to Developing Videoconference Products and Related Curricula. A paper at the 15^th Annual Edward F. Kelly Evaluation Conference, Toronto, Canada, 2001.

Mioduser, D. (2000) Web-based learning environments: current pedagogical state. Journal of Research on Computing in Education, 33, 55-76.

Newman, D.L., Smith, J., & Geehan, M.M. (2000). Systems change in the human service domain: A model for documenting and evaluating change. A paper presented at the 2000 American Evaluation Association Annual Meeting, Honolulu, HI.

Newman, D.L., Spaulding, D.T. & Catapano, N., (2002). The Virtual Informal Education Web Project: Formative Evaluation of the Schenectady City School District Technology Innovation Challenge Grant.

Omatseye, J. N. (1999) Teaching through teleconferencing: some curriculum challenges. College Student Journal, 33, 346-364.

Paul, L., Johnson, A. O., & Cranston, G. M. (2000) A successful videoconference satellite program: providing nutritional information on dementia to rural caregivers. Educational Gerontology, 26, 415-426.

Peat, M. (2000) The role of information technology in biology education: an Australian perspective. Journal of Biological Education, 34, 69-73.

Peplow, P. (1996) Videoconferencing for small-group discussion sessions of a case-based learning programme in anatomy. Medical Teacher. 18, 309-318.

Petracchi, H. E. (2000) Social work students and their learning environment: a comparison of interactive television, face-to-face instruction, and the traditional classroom. Journal of Social Work Education, 36, 335-346.

Telg, R. W. (1996) Instructional design considerations for teaching international audiences via satellite. International Journal of Instructional Media, 23, 209-219.

Wilson, K. (2000). Virtual seminars in European studies: A model for collaborative learning. Computer and the Humanities, 34, 345-347.