Call for papers 2023 insights

2023 is a big year for us, as we are planning our first two day event since the pandemic started. While last year was great, it was a single day conference, with most of the speakers re-invited after the cancellation of the 2020 event. But for 2023, to fill in the 20 slots, we brought back the call for papers.

In this article, I want to share some insights and learnings from our cfp selection process.

The plan

Our content team owns the speaker selection and meets up months ahead of the event to decide on:

  • a broad theme for the event
  • speakers we want to invite for the event
  • number of seats we keep for the call for papers
  • call for papers process and selection criteria

We always had and always will have a single track event. So the number of speakers we have every year is limited by this factor.

We also want to balance the number of speakers we invite with the ones we select from the CFP. This year, we ended up with 9 invitations and 11 selections.

The call

To make the CFP process more efficient, we used sessionize to collect the proposals and to assess them.

Sessionize takes care of a lot of operational details and has good filtering mechanisms on the proposals. It also helps you with some standard fields when building the form.

We created a compact proposal form because:

  • we didn’t want speakers to spend too much time filling in unnecessary data at this stage (eg: travel details, food allergies)
  • we wanted to look at just a few fields (eg: title, abstract, description) for each proposal

We started collecting data at the beginning of November in 2022 and we closed the form on December 31st, giving it two full months.

Then the hard part started.

The evaluation

One cool feature sessionize has is to turn on anonymous mode for all the proposals. You see all the talk details, but you don’t see any details about the speaker.

We used this anonymous mode for the first phase. We called this the screening evaluation.

Each of the five reviewers had a few weeks to go through all the proposals and rate them from 1 to 5.

While doing this, we also categorized the proposals, based on the kind of talks we wanted to have at the event. For example, we had a category called: “Architecture and Patterns”, another one for “Performance”, one for “Security” and so on.

Some interesting figures from the CFP:

  • number of proposals: 367
  • number of speakers: 186
  • total time spent rating: 22 hours and 9 minutes
  • average time spent per session per reviewer: 49.9 seconds

Another nice feature in sessionize is that you can write a short message when you rate a talk. These messages were super useful when reviewing the proposals later. But they also helped us give feedback points to people that asked us over email upon receiving their rejection.

The second phase started by eliminating all talks under a certain threshold. So everything rated below 3.0 was rejected. This left us with 62 shortlisted talks.

At this point, we de-anonymized the results and started comparing speakers and proposals head to head. The earlier mentioned categories helped us here, as we were able to compare 3-4 proposals on a similar topic.

Even so, this is the most complicated part, because there’s no recipe for doing it right. You have to consider many things:

  • the quality of the proposals
  • the ability of the speakers to deliver the proposals
  • the background of the speakers and what makes them subject matter experts
  • the topics that are of most relevance and interest to the community
  • the diversity of the speaker group

I must mention here that the overall quality of the shortlisted talks was amazing. We could have filled 2 tracks for 2 days with the content we had in there. But ultimately we had to pick 11 speakers and reject the other 51 proposals.

The learnings

These are some of our learnings from this year and from previous editions. We hope these learnings are useful for other event organizers:

  • If you expect more than 100 proposals, use a tool to manage everything, it is worth every penny.
  • Keep the form as short as possible, but make sure you have a non-public field where the speaker can explain what makes the proposal stand out
  • Limit the number of proposals per speaker. Almost all the speakers that were selected had a single proposal.
  • Ask speakers to select a category when submitting, but don’t limit them to your predefined categories
  • Wait until everyone you accepted confirms, before sending the rejection emails to the others
  • Don’t publish any info about accepted speakers until you have sent those rejections

Acknowledgements

Before I wrap this up, I want to give a big shout out to our content team: Andrei Antal, Ioana Chiorean, Jeremias Menichelli and Sara Vieira. You can add to those 22 hours many more spent in meetings and async conversations about: evaluation criteria, categories and comparisons. Thank you all for your effort! JSHeroes could not be done without your contribution.