Critical Assessment of Fully Automated Structure Prediction

CAFASP4 Second Announcement

The prediction session is about to begin as soon as the first targets are released by CASP. At this point, over 50 servers have registered in CAFASP and have already been coupled to the meta-server. Please note that future registrations may require a few days until the coupling is complete, which may result in missing the targets already released. So, those of you planning to register new servers, please do so asap.

All data at CAFASP will be publicly available immediately upon collection through the meta-server. As stated before, everybody can make use of these data for any purpose, and participation in CAFASP implies acceptance of these terms.

This time we have initiated two new categories in CAFASP: MQAP and DP. A number of servers have already registered. Please notice that the results of these servers will also be publicly available at all times. Because MQAP's are meta-selectors, the MQAP programs will be evaluated at CAFASP along with the other servers. Developers (or anybody else) may wish to use the MQAP results to also file predictions to CASP.

In CAFASP4, like in LiveBench, we attempt to reduce the load on developers, and we accept the data from the servers in various formats, with various models per prediction, without a number of compulsory fields, and without requiring to reply to CAFASP queries differently than to any user. The CAFASP meta-server, and transparently to the server developer, translates the raw output of the servers into a PDB-like format, which is the one that will be used for the CAFASP automatic evaluation after the structures of the targets become available.

Because of differences in the formats of the collected data at CAFASP and the format required at CASP, an automatic transfer of the data to the CASP Prediction Center can be error-prone. Thus, in CAFASP4, there will be NO automatic transfer of the data to the CASP Prediction Center. However, as stated above, all the data will be available at all times from the CAFASP web-site.

Unfortunately, we were unable to convince CASP to make their format more flexible like the one we use in CAFASP. We also tried to convince them that it is best that they get the servers' data from the CAFASP site and that they do any necessary formatting for all the servers, rather than requesting from all servers at this point to adjust their servers to comply with the CASP format. Our efforts were unsuccessful. Because at CAFASP we didn't want to bother developers with this formatting, and because CASP is interested in evaluating servers too, they decided to get the data from the servers directly. This means that CASP now will run their own CAFASP experiment with those servers that will be able to submit predictions in the required format. This will not reflect the state-of-the-art because many of the best servers will not be included. This is unfortunate, because of the confusion that this may create, the duplication of efforts, the extra burden for developers and the extra work for the servers.

This also means that those CAFASP developers wishing to have their servers' predictions entered in CASP in addition to CAFASP, will have to work it out independently of CAFASP with the CASP Prediction Center. Hopefully, this will not be a heavy burden for those developers with servers already generating predictions in casp-like format. As of today, this does not apply to the new MQAP and DP categories, which will be run only by CAFASP as previously described.

In summary, instead of getting the collected data from CAFASP and running a few scripts to format, CASP decided to override CAFASP and run a separate, limited experiment at the expense of the community.

Despite these unfortunate events, and because of the service that CAFASP provides to human and automated meta-predictors, CAFASP will run independently and as planned. Because CAFASP is an independently ran experiment, participation in CAFASP is independent of participation in CASP or any other experiment, and vice-versa. The CAFASP predictions will be evaluated by the CAFASP automated methods, and when the CASP predictions are made available, a comparison of humans vs servers will be reported. With our policy of announcing the evaluation rules before the experiment starts, and of making the data available at all times, we wish to continue to provide the valuable service we have been providing with our various fully-automated experiments.

Leszek and Daniel