Methodology

From HorizonMuseum

In this section of the wiki, the process behind the Horizon Project is described, and the research plan for the Horizon Report for Museums is detailed.

The Horizon Project Process

In order understand the process for creating the Horizon Project and Report for Museums, it is useful to be aware of how the higher-education-focused Horizon Report is developed. The Horizon Report is produced each fall using a carefully constructed process that is informed by both primary and secondary research. Nearly a hundred technologies, as well as dozens of meaningful trends and challenges are examined for possible inclusion in the report each year; an internationally renowned Advisory Board examines each topic in progressively more detail, reducing the set until the final listing of technologies, trends, and challenges is selected. The entire process takes place online and the process for 2008 and the past several years is fully documented at horizon.nmc.org/wiki.

The process of selection, a modified Delphi process now refined over five years of producing Horizon Reports, begins each summer as the Advisory Board is assembled. About half of the thirty to forty members are newly chosen each year, and the board as a whole is intended to represent a wide range of backgrounds, nationalities, and interests. To date, more than 175 internationally recognized practitioners and experts have participated. Once the Advisory Board is constituted, their work begins with a systematic review of the literature—press clippings, reports, essays, and other materials that pertain to emerging technology. Advisory Board members are provided with an extensive set of background materials when the project begins, and are then asked to comment on them, identify those which seem especially worthwhile, and add to the set. The group discusses existing applications and brainstorms new ones. A key criterion is the potential relevance of the topics to teaching, learning, and creative expression. A carefully selected set of RSS feeds from a dozen leading publications ensures that these resources stay current as the project progresses, and they are used to inform the thinking of the participants throughout the process.

Following the review of the literature, the Advisory Board engages in the process of addressing the five research questions that are at the core of the Horizon Project. These questions are the same each year, and are designed to elicit a comprehensive listing of interesting technologies, challenges, and trends from the Advisory Board:

1. What would you list among the established technologies that learning-focused institutions should all be using broadly today to support or enhance teaching, learning, or creative expression?
2. What technologies that have a solid user base in consumer, entertainment, or other industries should learning-focused institutions be actively looking for ways to apply?
3. What are the key emerging technologies you see developing to the point that learning-focused institutions should begin to take notice during the next 3 to 5 years? What organizations or companies are the leaders in these technologies?
4. What do you see as the key challenges related to teaching, learning, or creative expression that learning-focused institutions will face during the next 5 years?
5. What trends do you expect to have a significant impact on the ways in which learning-focused institutions approach our core missions of teaching, research, and service?

One of the Advisory Board’s most important tasks is to answer these five questions as systematically and broadly as possible, so as to generate a large number of potential topics to consider. As the last step in this process, past Horizon Reports are revisited and the Advisory Board is asked to comment on the current state of technologies, challenges, and trends identified in previous years, and to look for metatrends that that may be evident only across the results of multiple years.

Once this foundational work is completed, the Advisory Board moves to a unique consensus-building process based on an iterative Delphi-based methodology. In the first step, the responses to the research questions are systematically ranked and placed into adoption horizons by each Advisory Board member in a multi-vote system that allows members to weight their selections. Each member is asked to also identify the timeframe during which they feel the technology would enter mainstream use—defined for the purpose of the project as about 20% of institutions adopting it within the period discussed. (This figure is based on the research of Geoffrey A. Moore and refers to the critical mass of adoptions needed for a technology to have a chance of entering broad use.)

These rankings are compiled into a collective set of responses. From the more than 80 technologies originally considered, the twelve that emerge at the top of the initial ranking process—four per adoption horizon—are further researched. Once this "short list" is identified, the potential applications of these important technologies are further explored by higher education practitioners who were either knowledgeable about them, or interested in thinking about how they might be used. A significant amount of time is spent researching applications or potential applications for each of the areas that would be of interest to practitioners.

Each of these twelve "short list" items is written up in the format of the Horizon Report. With the benefit of the full picture of how the topic would look in the report, the “short list” is ranked yet again, this time with a reverse ranking approach. The six technologies and applications that emerge at the top of the rankings—two per adoption horizon—are detailed in the annual report.

The Horizon.Museum Process

The Horizon Report for Museums starts with the findings of the Horizon Report. For the museum edition, this sum of the work done by the higher-education Advisory Board seeds the process of selecting the topics for Horizon.Museum. The museum Advisory Board will use the 2008 Short List as the starting point for their process, and build a report specially crafted for the museum community on that foundation. A collection of published reports, papers, chapters, blog posts, and other resources, gathered collaboratively, will provide the Advisory Board a solid basis in the literature. Readers and Advisory Board members are encouraged to add to these resources using a special del.icio.us tag, hzmuseum08.

Using an online Delphi process similar to that of the primary report, the first round of decisions will focus on the appropriateness of the items in the higher-education Short List, and their applications for museum practice.

The Advisory Board will be initially asked the following questions. Research Notes will ensure consistency of practice by Board Members in considering and answering the questions.

1. How do the technologies that museums should be using today or during the next five years compare with the 2008 Short List of twelve technologies identified by the 2008 Horizon Report for higher education? That is to say, which are likely to enter mainstream museum use within the targeted timeframe? Which will be important, but on a different timeframe? Which of these technologies would fall off the list when considered through the lens of potential applications for museums and museum practice?

2. What technologies that are not on the list developed so far need to be considered in creating a "short list" expressly for museums? These might include technologies that have a solid user base in consumer, entertainment, telecommunications, imaging, or other industries.

3. What do you see as the key challenges related to the adoption of emerging technology in museums during the next five years?

4. What technology trends do you expect will have a significant impact on the ways in which museums approach their missions during the next five years?

From the responses of the Advisory Board to these questions, a new "Short List," specially targeted for museums, will be created. The twelve top-ranked technologies (four per adoption horizon) and the top-ranked trends and challenges will be written up in the style of the Horizon Report and distributed to the Advisory Board for their consideration.

In the final round of selections, with the benefit of the full picture of how the topic might look in the report, the “short list” will be sorted using a reverse ranking approach. The six technologies and applications that emerge at the top of the rankings — two per adoption horizon — will be those detailed in the Horizon Report for Museums.

The Call to Scholarship

As with its higher education companion, the Horizon.Museum is intended to be the first step in building a research agenda rather than the final result of one; the NMC membership uses the Horizon Report each spring to generate an annual Call to Scholarship (see the Call to Scholarship for details) based on the input of hundreds of faculty and staff working in campus-based groups. The Call details recommendations for research, demonstration projects, policy formulation, tools, and technology support systems related to each topic. These recommendations are a starting place for continued dialogue and reflection around the six topics in the Horizon Report, and are acknowledgements that while these technologies offer considerable promise and potential, much work remains to be done before many of them are really ready for mainstream use.

The Call to Scholarship is also a call to action, and it is our hope that the Museum Edition will also generate a cascade of activities in museums around the world. The NMC is deeply interested in such activities and hopes to see new demonstration projects, papers, and presentations at conferences focused around the ideas in each new edition of the Horizon Report for Musems. Simultaneous with the release of the 2008 edition of the Horizon Report for Museums, the NMC will launch the process for creating its related Call to Scholarship, which will be released in the early 2009.

Another ongoing component of the project involves a special set of del.icio.us links that have been established to help extend the findings of the project and allow new information to be shared within the community. These del.icio.us tagged resources will be collected in the “Further Reading” section for each of the six topic areas, and readers will be invited to consult not only the resources that will be published in the report, but many others that are collected as part of the research as well. Readers are also encouraged to add their own examples and readings to these dynamic lists by tagging them for inclusion in each category.