Reviews

Where do I configure review questions?

Review questions are configured in the Review Designer.

  1. Navigate to your event, and then Call for Papers
  2. Click on the name of the relevant Call for Papers
  3. In the left-hand menu, click on “Review Designer”
  4. Scroll down to the section called “Review Questions”

What type of review questions can I ask my reviewers?

Lineup Ninja has 3 special types of review questions that are used in particular ways:

  1. Score questions: you can use as many score questions as you like, and weight them differently if you wish.
  2. Recommendation questions: by default the recommendations are “accept”, “waitlist” or “decline”, but you can edit these to mean something else.
  3. Feedback questions that are shown to submitters. These questions allow you to capture feedback from reviewers and pass it to the submitter. Technically, this is an option that can be set on any review questions. So if, for example, you wanted submitters to see reviewers’ scores, you set this option on the score question. The reviewers’ names are annonymised in the feedback.

Apart from these special question types, like the CFP form designer, you can add your own questions of various types:

  1. Short and long text questions
  2. Numerical questions
  3. Multiple choice
  4. File upload
  5. Email and web addresses etc.

How do I preview what reviewers will see when I assign a review to them?

The best approach would be to assign a test submission to yourself to review: then you get to preview the whole reviewer experience process - email invitation, login flow etc.

How do I set the review deadline?

The review deadline is set when you assign the reviews (each review could, theoretically, have a unique deadline, rather than there being one single review deadline for all reviews.)

When you assign the new reviews, the dialogue that pops up will ask you what deadline you want to use for them: Review Deadlines

How do I run a blind review? / How can I anonymise submissions? / How can I hide some questions’ answers from reviewers?

Review questions are configured in the Review Designer.

  1. Navigate to your event, and then Call for Papers
  2. Click on the name of the relevant Call for Papers
  3. In the left-hand menu, click on “Review Designer”
  4. Scroll down to the section called “Question Visibility”.
  5. For each question that appears in your CFP, you can use the toggle switch to show or hide the answers from your reviewers.

How do I assign submissions to a reviewer?

You can assign reviews like this:

  1. Go to the list of submissions
  2. Select the submission you want to assign to the reviewer(s)
  3. Scroll down to the bottom of the page
  4. Select ‘assign reviewers’ from the drop-down menu
  5. In the dialogue box that pops up, select the reviewer you want to assign the review(s) to
  6. You can optionally set a review due date here
  7. Click on the ‘assign reviews’ button
  8. This will trigger an automated email to the rewviewer notifying them that they have been assigned a review.

See also: How to assign to individual reviewers or a group of reviewers

How do I assign a group of submissions to all the reviewers in a reviewer group?

You can bulk assign reviews to a group of reviewers like this:

  1. Go to the list of submissions
  2. Select the submissions you want to assign to the group reviewers
  3. Scroll down to the bottom of the page
  4. Select ‘assign reviewers’ from the drop-down menu.
  5. In the dialogue box that pops up, apply a filter on the “Reviewer Groups” column.
  6. Check the check-box next to the “Reviewer Name” column heading. This will select all of the reviewers in the currently filtered list
  7. You can optionally set a review due date here
  8. Click on the ‘assign reviews’ button
  9. This will trigger automated emails to the reviewers notifying them that they have been assigned a review.

See also: How to assign to individual reviewers or a group of reviewers

How do I set automatic reminders for reviewers?

When you assign submissions to reviewers, the following dialogue box pops up. If you set a Review due date (near the bottom), it will automatically send the reviewers an email 72 & 24 hours before the due date: Review Deadlines

How do I track the review progress of a submission?

Go to the submissions list and search for the submission. There are several ways to find it:

  1. When submissions have reviewers assigned to them, they move into the “Reviews in Progress” tab. Once all the reviews have been completed, they move into “Reviews Complete”. In Both cases, you can see reviewers’ scores and recommendations.
  2. You can search for the submission by its title, submitter name, submitter email by using the search tool in these columns’ headings.
  3. You can use the advanced search tool to build a complex query to find the submission you’re interested in.
  4. You can filter and sort various columns in the submissions list.

How do I track a single reviewer’s activity or progress?

You can see how many reviews are assigned, in-progress and completed for each reviewer in the reviewer list in the Call for Papers list view.

How do I nudge reviewers to finish their reviews?

A “nudge reviewers to finish their reviews” feature is currently in the pipeline. In the meantime, the easiest way would be to unassign the reviews and reassign them, setting the deadline again. This will trigger an email that they’ve been assigned, and configure the reminder emails to be sent at 72 and 24 hours prior to the due date. And it retains any work the reviewer has already done.

The deadline for the abstract reviewers has now passed but I need to reassign a few submissions to get reviewed ASAP. How do I reset the deadline to tomorrow?

The review deadline is set when you assign the reviews (each review could, theoretically, have a unique deadline, rather than there being one single review deadline for all reviews).

When you assign the new reviews, the dialogue that pops up will ask you what deadline you want to use for them.

We want to assign 3-4 submissions to each reviewer. We have trialed this but once we have assigned a submission to a reviewer it sends them an email straight away before we can add the other submissions to the reviewers name. Is there a way around this?

Yes, you can bulk assign reviews to a group of reviewers like this:

  1. Go to the list of submissions
  2. Select the submissions you want to assign to the group reviewers
  3. Scroll down to the bottom of the page
  4. Select ‘assign reviewers’ from the drop-down menu.
  5. In the dialogue box that pops up, apply a filter on the “Reviewer Groups” column.
  6. Check the check-box next to the “Reviewer Name” column heading. This will select all of the reviewers in the currently filtered list
  7. You can optionally set a review due date here
  8. Click on the ‘assign reviews’ button
  9. This will trigger automated emails to the reviewers notifying them that they have been assigned a review.

Can I show reviewers some questions based on their answers to previous questions?

Yes, you can use ‘Display rules’:

  1. Edit the question that you want to show (or hide) based on a previous answer
  2. Scroll down to the Display Rules section
  3. CLick on the blue button that says “Display Rule +” on it.
  4. Choose the question whose answer you want to determine whether this question will be shown
  5. Configure the rule that will determine whether this question will be shown or not

We’re now at a point where we’ve had enough reviews and now want to look at a short list, without emailing submitters anything. Is there somewhere we can drop that shortlist?

There are a couple of ways you could do this:

  1. Use the waitlist instead of a shortlist (assuming you don’t already have a load of submissions on the waitlist). When you move the submissions from ‘reviews complete’ to ‘waitlist’ it will pop up the waitlist email template and ask if you want to send it - you can skip sending this email, so submitter won’t be any the wiser.
  2. Create a label called ‘shortlist’ and assign it to the submissions you want to shortlist, then use that as a way of filtering out the shortlisted one from the list etc.

How do I automatically assign submissions to reviewers?

This feature is in the pipeline. In the meantime, we suggest that you assign submissions to reviews in bulk.