IA Summit 2007, March 22-26 at the Flamingo Las Vegas, Las Vegas, Nevada, USA

IA Summit 2007 Review Process

Many people ask the IA Summit committee members just how we go about selecting presentations for the conference. Here's how we did it for 2007.

First some statistics…

We had:

  • 139 presentation submissions
  • 13 research submissions
  • 12 panel submissions
  • 29 pre-conference workshop submissions
  • 25 poster submissions

We accepted:

  • 40 presentations (29%)
  • 6 research papers (46%)
  • 5 panels (42%)
  • 19 pre-conference workshops (66%)
  • 20 posters (80%)

We did it with:

  • 62 reviewers
  • 805 reviews

Research papers averaged:

  • 13 pages
  • 4000 words

Scheduling

We set up a voting process, and asked people to vote for their most popular sessions. We used that information in the schedule (to make sure popular sessions didn't clash), asked some presenters to present twice and also used the information to prioritise the sessions we audio-recorded.

But before we started…

The presentations at the IA Summit are absolutely critical to the conference's success (almost as much as the social program) so we take the process very seriously and endeavour to select a diverse, challenging and interesting program.

Before we started the review process, we outlined what we expected in the call for proposals.

We prepared criteria for our reviewers and a substantial set of reviewing guidelines.

The scoring criteria for presentations was a 1-5 score for each of the following criteria (with 1 being low and 5 being high):

  1. Contribution to IA practice, theory and community
  2. Value and appeal to conference attendees
  3. Original, inspirational or thought-provoking content
  4. Quality of proposal
  5. Would you recommend this presentation be included in the program? (No, Maybe, Yes)

Panels and pre-conference workshops were reviewed using slightly different criteria. For panels, we looked at whether the topic would make a good panel discussion and that the panellists had a range of opinions. For pre-conference workshops, we wanted to make sure they were practical, valuable and taught by people with a lot of experience in their topic.

And reviewers provided comments on why they scored as they did, plus feedback for the proposer.

The review process

Reviewers were randomly assigned a block of proposals to review, but were not able to review their own proposal or that of obvious colleagues. If they had the time and energy, they could review proposals other than their assignments.

All presentations were blind reviewed - reviewers could not see who the presentation was from. This was done for the first time for the 2007 conference in order to ensure the most interesting topics were chosen, not just the most prominent faces.

The review process for research submissions

Research submissions underwent a double-blind review process commonly used for scholarly work. For each paper, reviewers were asked to read the abstract and indicate if they were interested in reviewing the paper, had sufficient expertise to review it, or were unable to review it due to either lack of expertise or a potential conflict of interest. This information was used to randomly assign papers to reviewers. Each paper received at least three reviews.

The review criteria included both quantitative and qualitative elements. The quantitative criteria were used to create an initial ranking. The final selection involved both quantitative and qualitative aspects, including additional discussion amongst the review committee.

Reviewers in the research track included university faculty members, doctoral students, and practitioners.

The selection process

Creating a conference program is not as simple as picking the highest scoring presentations. A small team (experienced IAs, previous IA Summit attendees and people active in the IA Community) examined all the reviews. After much discussion, they made a selection of what they thought would be the best submissions for the conference. Their overall aim was to select a diverse set of proposals that would suit the wide range of experiences and expectations of our attendees.

Just like a regular design process, this is where some creativity and judgement came in and we can't absolutely define why we chose one proposal over another.

But when do we find out who got in?

Now the proposals have been selected, we have to prepare the website, registration forms etc. The program will be available in mid-January. Keep your eye on mailing lists and this website for the announcement.

IA Summit 2007