Critical Assessment of Fully Automated Structure Prediction

CAFASP4 Third Announcement

After extensive consultation with a large number of predictors, and to maximize the value of CAFASP, we have decided to make two changes in the way that CAFASP-4 will operate. These changes affect only the submission schedule of targets and the deadline schedule for the collection of predictions. All other previously announced rules remain the same.

Change in target submission schedule: approximately 2 weeks before the CASP deadline. The period between the CASP target release and the deadline to file predictions in CASP usually spans a number of weeks. Due to the fast growth in the sequence and structure databases, server predictions for some targets obtained close to the deadline date may be more accurate than those obtained at the target release time. Thus, to attempt to collect the most up-to-date information, in CAFASP-4 the target sequences will be submitted to the CAFASP servers close to the CASP deadline and not at the CASP release date, as in previous CAFASP experiments. To allow predictors enough time to make use of the CAFASP servers' predictions, CAFASP will attempt to submit the targets no later than 2 weeks before the CASP deadline.

Change in the prediction collection schedule: 1 week after the CAFASP metaserver submission. Because a growing number of servers require extensive computation, the time allowed to submit a prediction to CAFASP will be 1 week after the metaserver target submission, and not two working days, as in previous CAFASP experiments.

Notice that for some targets it will be necessary to slightly modify the above schedule, e.g. if a CASP target is released with less than a 2 week deadline, or if many targets are released at the end of the prediction session, etc.

The above changes have a number of implications. First, there will be a reduction of the information duplication between CASP and CAFASP, because the CAFASP predictions will be more up-to-date than those collected weeks before by CASP. Second, there will be a reduction of the load and pressure on servers, because it is likely that many of the CAFASP submissions will be carried out after the release of most targets. Another implication is that servers will have significantly more time to submit their predictions. This is especially relevant for ab-initio servers.

Another important implication of the above changes is the following. It is clear that it will now be harder to distinguish which predictions were obtained through a fully-automated process from those obtained with some human intervention by the developers. CAFASP will not attempt to make this distinction, although it is fairly simple to learn which CAFASP participants correspond to fully-automated servers open to the community, and which are not. Thus, all CAFASP participants should be regarded as "service providers", regardless of the exact process that takes place at the servers' end. Anybody wishing to provide the service of filing his/her predictions to CAFASP and wishing to have his/her predictions evaluated by the CAFASP automatic evaluation, is welcome to participate in CAFASP as a "service provider".