Critical Assessment of Fully Automated Structure Prediction


CAFASP4 RULES AND INFORMATION

ANNOUNCEMENT NO. 1.

GOAL

The goal of CAFASP is to evaluate the performance of fully automatic structure prediction servers available to the community. In contrast to the normal CASP procedure, CAFASP answers the question of how well servers do without any intervention of experts, i.e. how well ANY user of automated methods only, can predict protein structure. As in previous CAFASPs, CAFASP4 assesses the performance of methods without the user intervention allowed in CASP.

CAFASP4 will be held by mid-2004. All developers of automatic prediction servers are invited to participate in CAFASP4. Interested parties are invited to send an e-mail to dfischer@cs.bgu.ac.il so they will be included in the CAFASP4 mailing list. Predictors wishing to include their servers in CAFASP4 should register as soon as possible at the CAFASP4 meta-server, and/or email Leszek Rychlewski (leszek@bioinfo.pl).

TERMINOLOGY

A "server" is the automated structure prediction server that participates in CAFASP. A server will have an identification name and its corresponding url.

A "server's developer" is the person in charge of the maintainance and function of a server.

A "raw server output" is the output produced by a server.

REGISTRATION TO CAFASP4

A server's developer willing to include his/her server in CAFASP4 needs to register at the CAFASP4 site , and recieve an explicit acknowledgment that the server has been included in the list of CAFASP participants. Participation in CAFASP4 is independent of CASP6. Servers that run different methods should register each of the methods separately.

CAFASP4 SUBMISSION PROCEDURE

As each target sequence is released, the CAFASP meta-server will submit it to all the prediction servers. These will send their raw-output to the meta-server, and all the results from all servers will be stored at the CAFASP website.

The raw server output MUST be in CAFASP4 valid format. CAFASP4 accepts two types of formats:

  • CASP format, which includes "pdb" format and "casp"-like format (AL) or
  • a format acceptable by the CAFASP meta-server. Each server developer should check with us to see if his/her server's output is acceptable.

We strongly encourage developers to have their servers produce raw output in pdb format, as this requires no further, error-prone processing. All evaluations will be carried out using the pdb format. Sequence numbers should start from 1 and end at n, where n is the length of the submitted sequence.

Notice that there will be NO manual intervention after the raw output has been collected. The raw output is the only, unique entry in CAFASP4 (no parallel or duplicate submission to CASP is required).

It is the server's developer responsibility to make sure that the meta-server understands his/her server's raw output. Unparsable, or invalid CAFASP4 formats will result in a non-evaluation.

As the raw servers' output of the registered servers is collected, it will be made available to all, in the CAFASP4 web-site. This will allow anybody to use the predictions from the automated servers for any purpose. Participation in CAFASP explicitly implies that the developer agrees that his/her server's results will be published and used by others.

SUBMISSION DEADLINES

The CAFASP deadline for receiving the automated "raw-output" from the servers will be 2 working days after the target was sent. This time there will be no extensions for meta-metas nor for ab initio servers. After the 2 days deadline, a list of compiled models will be published, and no further predictions will be accepted for the particular target. This limitation may not be suitable for servers that require more time, or for higher order metas. We encourage those servers to submit their predictions to CASP rather than to CAFASP.

UN-REGISTRATION

Any cafasp participant can ask to withdraw his/her participation at any time, which will mean that his/her server's url will be removed from cafasp's meta-server.

CATEGORIES FOR CAFASP4

Automated servers in CAFASP4 will include those for Homology Modeling, Threading and Ab-Initio.

In CAFASP4, two new assessment categories are introduced: MQAP, Model Quality Assessment Programs and DP, domain prediction.

ASSESSMENT OF PREDICTIONS

Predictions submitted to CAFASP will undergo an independent evaluation by the CAFASP automatic evaluation programs. The evaluation procedures will be announced as soon as possible, and are likely to be very similar to those used in CAFASP3. When the CASP6 data will be made available, a comparison of the performance of CAFASP4 servers versus the regular CASP6 groups will also be carried out.

We encourage all server developers to understand the evaluation methods BEFORE they register, and if they find them inadequate, they may choose not to participate in CAFASP. Submission of predictions to CAFASP entails an implicit acceptance of the evaluation procedures.

CAFASP4 url: http://www.cs.bgu.ac.il/~dfischer/CAFASP4