Critical Assessment of Fully Automated Structure Prediction


CAFASP3 RULES AND INFORMATION

ANNOUNCEMENT NO. 1.

GOAL

The goal of CAFASP is to evaluate the performance of fully automatic structure prediction servers available to the community. In contrast to the normal CASP procedure, CAFASP answers the question of how well servers do without any intervention of experts, i.e. how well ANY user of automated methods only, can predict protein structure. As in previous CAFASPs, CAFASP3 assesses the performance of methods without the user intervention allowed in CASP.

CAFASP3 will held by mid-2002. All developers of automatic prediction servers are invited to participate in CAFASP3. Interested parties are invited to send an e-mail to dfischer@cs.bgu.ac.il so they will be included in the CAFASP3 mailing list. Predictors wishing to include their servers in CAFASP3 should register as soon as possible at the CAFASP3 meta-server, and/or email Leszek Rychlewski (leszek@bioinfo.pl).

TERMINOLOGY

A "server" is the automated structure prediction server that participates in CAFASP. A server will have an identification name and its corresponding url. Servers for any aspect of structure prediction can participate in CAFASP.

A "server's developer" is the person in charge of the maintainance and function of a server.

A "raw server output" is the output produced by a server.

REGISTRATION TO CAFASP3

A server's developer willing to include his/her server in CAFASP3 needs to register at the CAFASP3 site , and recieve an explicit acknowledgment that the server has been included in the list of CAFASP participants. Participation in CAFASP3 is independent of CASP5. Servers that run different methods should register each of the methods separately.

CAFASP3 SUBMISSION PROCEDURE

As each target sequence is released, the CAFASP meta-server will submit it to all the prediction servers. These will send their raw-output to the meta-server, and all the results from all servers will be stored at the CAFASP website.

The raw server output MUST be in CAFASP3 valid format. CAFASP3 accepts two types of formats:

  • CASP format, or
  • a format acceptable by the meta-server. Each server developer should check with the corresponding coordinator (see below) to see if his/her server's output is acceptable.

Notice that there will be NO manual intervention after the raw output has been collected. The raw output is the only, unique entry in CAFASP3 (in CAFASP2 we required to also submit a formatted submission to the Prediction Center; in CAFASP3 there will not be any parallel CASP submission; the only valid submission to CAFASP is the servers' raw output).

It is the server's developer responsibility to make sure that the meta-server understands his/her server's raw output. Unparsable, or invalid CAFASP3 formats will result in a non-evaluation.

As the raw servers' output of the registered servers is collected, it will be made available to all, in the CAFASP3 web-site. This will allow anybody to use the predictions from the automated servers for any purpose. Participation in CAFASP explicitly implies that the developer agrees that his/her server's results will be published and used by others.

SUBMISSION DEADLINES

The CAFASP deadline for receiving the automated "raw-output" from the servers will be 2 working days after the target was sent. There will be no "late" entries.

In CAFASP3 we expect to have a number of meta-predictors, i.e. servers that use the results of other servers. The deadline for meta-predictors will be 2 additional working days.

UN-REGISTRATION

Any cafasp participant can ask to withdraw his/her participation at any time, which will mean that his/her server's url will be removed from cafasp's meta-server.

CATEGORIES FOR CAFASP3

Any automated server can participate in CAFASP3, including those for Homology Modeling, Threading, Ab-Initio, Secondary Structure Prediction and Contacts Prediction.

ASSESSMENT OF PREDICTIONS

Predictions submitted to CAFASP will undergo an independent evaluation by the CAFASP automatic evaluation programs. The evaluation procedures will be determined by each coordinator and will be announced as soon as possible. If the CASP5 data will be made available, a comparison of the performance of CAFASP3 versus the regular CASP5 submission will also be carried out.

We encourage all server developers to understand the evaluation methods BEFORE they register, and if they find them inadequate, they may choose not to participate in CAFASP.

CAFASP COMMITTEES

Currently the people involved in the CAFASP organization are Leszek Rychlewski, (leszek@bioinfo.pl), Arne Elofsson (arne@sbc.su.se), and Daniel Fischer (dfischer@cs.bgu.ac.il).

The coordinators of the different prediction categories are:

  • Homology modeling: Roland L. Dunbrack (RL_Dunbrack@fccc.edu)
  • Fold Recognition or threading. Leszek Rychlewski (leszek@bioinfo.pl), Arne Elofsson (arne@sbc.su.se), and Daniel Fischer (dfischer@cs.bgu.ac.il).
  • ab-initio: Angel Ortiz (angel.ortiz@inka.mssm.edu)
  • Secondary structure: Burkhard Rost (rost@columbia.edu).
  • Contacts prediction: Alfonso Valencia (valencia@cnb.uam.es).

Each coordinator will be in charge of the procedures that will be used in the evaluation and will be responsible for the automated evaluation of the servers in the corresponding category.

We invite interested parties to be involved in the various sub-committees.

CAFASP3 url: http://www.cs.bgu.ac.il/~dfischer/CAFASP3