Methodology

From HorizonMuseum

In this section of the wiki, the process behind the Horizon Project is described, and the research plan for the Horizon Report for Museums is detailed.

The Horizon Project Process

In order understand the process for creating the Horizon Project and Report for Museums, it is useful to be aware of how the higher-education-focused Horizon Report is developed. The Horizon Report is produced each fall using a carefully constructed process that is informed by both primary and secondary research. Nearly a hundred technologies, as well as dozens of meaningful trends and challenges are examined for possible inclusion in the report each year; an internationally renowned Advisory Board examines each topic in progressively more detail, reducing the set until the final listing of technologies, trends, and challenges is selected. The entire process takes place online and the process for 2008 and the past several years is fully documented at horizon.nmc.org/wiki.

The process of selection, a modified Delphi process now refined over five years of producing Horizon Reports, begins each summer as the Advisory Board is assembled. About half of the thirty to forty members are newly chosen each year, and the board as a whole is intended to represent a wide range of backgrounds, nationalities, and interests. To date, more than 175 internationally recognized practitioners and experts have participated. Once the Advisory Board is constituted, their work begins with a systematic review of the literature—press clippings, reports, essays, and other materials that pertain to emerging technology. Advisory Board members are provided with an extensive set of background materials when the project begins, and are then asked to comment on them, identify those which seem especially worthwhile, and add to the set. The group discusses existing applications and brainstorms new ones. A key criterion is the potential relevance of the topics to teaching, learning, and creative expression. A carefully selected set of RSS feeds from a dozen leading publications ensures that these resources stay current as the project progresses, and they are used to inform the thinking of the participants throughout the process.

Following the review of the literature, the Advisory Board engages in the process of addressing the five research questions that are at the core of the Horizon Project. These questions are the same each year, and are designed to elicit a comprehensive listing of interesting technologies, challenges, and trends from the Advisory Board:

1. What would you list among the established technologies that learning-focused institutions should all be using broadly today to support or enhance teaching, learning, or creative expression?
2. What technologies that have a solid user base in consumer, entertainment, or other industries should learning-focused institutions be actively looking for ways to apply?
3. What are the key emerging technologies you see developing to the point that learning-focused institutions should begin to take notice during the next 3 to 5 years? What organizations or companies are the leaders in these technologies?
4. What do you see as the key challenges related to teaching, learning, or creative expression that learning-focused institutions will face during the next 5 years?
5. What trends do you expect to have a significant impact on the ways in which learning-focused institutions approach our core missions of teaching, research, and service?

One of the Advisory Board’s most important tasks is to answer these five questions as systematically and broadly as possible, so as to generate a large number of potential topics to consider. As the last step in this process, past Horizon Reports are revisited and the Advisory Board is asked to comment on the current state of technologies, challenges, and trends identified in previous years, and to look for metatrends that that may be evident only across the results of multiple years.

Once this foundational work is completed, the Advisory Board moves to a unique consensus-building process based on an iterative Delphi-based methodology. In the first step, the responses to the research questions are systematically ranked and placed into adoption horizons by each Advisory Board member in a multi-vote system that allows members to weight their selections. Each member is asked to also identify the timeframe during which they feel the technology would enter mainstream use—defined for the purpose of the project as about 20% of institutions adopting it within the period discussed. (This figure is based on the research of Geoffrey A. Moore and refers to the critical mass of adoptions needed for a technology to have a chance of entering broad use.)

These rankings are compiled into a collective set of responses. From the more than 80 technologies originally considered, the twelve that emerge at the top of the initial ranking process—four per adoption horizon—are further researched. Once this "short list" is identified, the potential applications of these important technologies are further explored by higher education practitioners who were either knowledgeable about them, or interested in thinking about how they might be used. A significant amount of time is spent researching applications or potential applications for each of the areas that would be of interest to practitioners.

Each of these twelve "short list" items is written up in the format of the Horizon Report. With the benefit of the full picture of how the topic would look in the report, the “short list” is ranked yet again, this time with a reverse ranking approach. The six technologies and applications that emerge at the top of the rankings—two per adoption horizon—are detailed in the annual report.

The Horizon.Museum Process

The work on the 2008 Horizon Report, Museum Edition started with the findings of the 2008 Horizon Report. For the museum edition, the work done by the higher-education Advisory Board seeded the process of selecting the topics for Horizon.Museum. The museum Advisory Board used the 2008 Short List as the starting point for their process, and provided the data for a report specially crafted for the museum community building on that foundation. A collection of published reports, papers, chapters, blog posts, and other resources, gathered collaboratively, provided the Advisory Board a solid basis in the literature. Readers and Advisory Board members were encouraged to add to these resources using a special del.icio.us tag, hzmuseum08.

Using an online Delphi process similar to that of the primary report, the advisory board was asked to first answer a set of research questions on the wiki, and then to distribute 40 votes among the responses to the research questions for the technologies they felt were most likely to be adopted within the timeframes of the report, and for the challenges and trends they felt would be most important over the same period, ten per question. To allow for weighting, multiple votes were allowed. Additionally the Advisory Board members were asked to place those adoptions on one of three planning horizons -- within 12 months, within 2 to 3 years, and within 4-5 years. The results of these rankings determined the selections that would appear in the final report.

The Advisory Board was asked the following research questions, guided by a set of Research Notes intended to ensure consistency of practice by Board Members in considering and answering the questions.

1. How do the technologies that museums should be using today or during the next five years compare with the 2008 Short List of twelve technologies identified by the 2008 Horizon Report for higher education? That is to say, which are likely to enter mainstream museum use within the targeted timeframe? Which will be important, but on a different timeframe? Which of these technologies would fall off the list when considered through the lens of potential applications for museums and museum practice?
2. What technologies that are not on the list developed so far need to be considered in creating a "short list" expressly for museums? These might include technologies that have a solid user base in consumer, entertainment, telecommunications, imaging, or other industries.
3. What do you see as the key challenges related to the adoption of emerging technology in museums during the next five years?
4. What technology trends do you expect will have a significant impact on the ways in which museums approach their missions during the next five years?

After the responses of the Advisory Board to these questions were tallied, a decision was planned as to whether additional rounds of voting would be needed, based on the clarity of the results. If an additional round was determined to be needed, the twelve top-ranked technologies (four per adoption horizon) and the top-ranked trends and challenges would be written up in the style of the Horizon Report and distributed to the Advisory Board for a final round of selections, using a reverse ranking approach. In either case, the six technologies and applications that emerged at the top of the rankings — two per adoption horizon — would be those detailed in the Horizon Report, Museum Edition. In 2008, only a single round of rankings was needed as the final six were clearly identified in the initial round of voting.

The Call to Scholarship

As with its higher education companion, the Horizon.Museum is intended to be the first step in building a research agenda rather than the final result of one; the NMC membership uses the Horizon Report each spring to generate an annual Call to Scholarship (see the Call to Scholarship for details) based on the input of hundreds of faculty and staff working in campus-based groups. The Call details recommendations for research, demonstration projects, policy formulation, tools, and technology support systems related to each topic. These recommendations are a starting place for continued dialogue and reflection around the six topics in the Horizon Report, and are acknowledgements that while these technologies offer considerable promise and potential, much work remains to be done before many of them are really ready for mainstream use.

The Call to Scholarship is also a call to action, and it is our hope that the Museum Edition will also generate a cascade of activities in museums around the world. The NMC is deeply interested in such activities and hopes to see new demonstration projects, papers, and presentations at conferences focused around the ideas in each new edition of the Horizon Report for Musems. Simultaneous with the release of the 2008 edition of the Horizon Report for Museums, the NMC will launch the process for creating its related Call to Scholarship, which will be released in the early 2009.

Another ongoing component of the project involves a special set of del.icio.us links that have been established to help extend the findings of the project and allow new information to be shared within the community. These del.icio.us tagged resources will be collected in the “Further Reading” section for each of the six topic areas, and readers will be invited to consult not only the resources that will be published in the report, but many others that are collected as part of the research as well. Readers are also encouraged to add their own examples and readings to these dynamic lists by tagging them for inclusion in each category.