How we measure forecasting ability

Brier score
Accuracy score

Accuracy score = Brier score - Median brier score

How we aggregate individual predictions to produce a better forecast

Some of the methods we apply to create a better forecast are:

  • Decay: Selecting a specific subset of the predictions based on when the predictions where made.
  • Ability: Weighting a persons prediction based on his accuracy score relative to other forecasters.
  • Frequency: Weighting a persons prediction based on the frequency of which he updates his predictions relative to the other forecasters.
  • Extremizing: Transforming the probability estimate based on the statistical distribution and forecasters experience.

Factors such as the number of forecasters per question and the number of previous forecasts per participant will affect the degree to which algorithms can be applied to improve the forecast.

Our aggregation algorithms will be continuously developed and updated as we learn from the data over time.