CAFASP3 RULES AND INFORMATION
ANNOUNCEMENT NO. 1.
The goal of CAFASP is to evaluate the performance of fully automatic structure prediction servers available to the community. In contrast to the normal CASP procedure, CAFASP answers the question of how well servers do without any intervention of experts, i.e. how well ANY user of automated methods only, can predict protein structure. As in previous CAFASPs, CAFASP3 assesses the performance of methods without the user intervention allowed in CASP.
CAFASP3 will held by mid-2002. All developers of automatic prediction servers are invited to participate in CAFASP3. Interested parties are invited to send an e-mail to firstname.lastname@example.org so they will be included in the CAFASP3 mailing list. Predictors wishing to include their servers in CAFASP3 should register as soon as possible at the CAFASP3 meta-server, and/or email Leszek Rychlewski (email@example.com).
A "server" is the automated structure prediction server that participates in CAFASP. A server will have an identification name and its corresponding url. Servers for any aspect of structure prediction can participate in CAFASP.
A "server's developer" is the person in charge of the maintainance and function of a server.
A "raw server output" is the output produced by a server.
REGISTRATION TO CAFASP3
A server's developer willing to include his/her server in CAFASP3 needs to register at the CAFASP3 site , and recieve an explicit acknowledgment that the server has been included in the list of CAFASP participants. Participation in CAFASP3 is independent of CASP5. Servers that run different methods should register each of the methods separately.
CAFASP3 SUBMISSION PROCEDURE
As each target sequence is released, the CAFASP meta-server will submit it to all the prediction servers. These will send their raw-output to the meta-server, and all the results from all servers will be stored at the CAFASP website.
The raw server output MUST be in CAFASP3 valid format. CAFASP3 accepts two types of formats:
Notice that there will be NO manual intervention after the raw output has been collected. The raw output is the only, unique entry in CAFASP3 (in CAFASP2 we required to also submit a formatted submission to the Prediction Center; in CAFASP3 there will not be any parallel CASP submission; the only valid submission to CAFASP is the servers' raw output).
It is the server's developer responsibility to make sure that the meta-server understands his/her server's raw output. Unparsable, or invalid CAFASP3 formats will result in a non-evaluation.
As the raw servers' output of the registered servers is collected, it will be made available to all, in the CAFASP3 web-site. This will allow anybody to use the predictions from the automated servers for any purpose. Participation in CAFASP explicitly implies that the developer agrees that his/her server's results will be published and used by others.
The CAFASP deadline for receiving the automated "raw-output" from the servers will be 2 working days after the target was sent. There will be no "late" entries.
In CAFASP3 we expect to have a number of meta-predictors, i.e. servers that use the results of other servers. The deadline for meta-predictors will be 2 additional working days.
Any cafasp participant can ask to withdraw his/her participation at any time, which will mean that his/her server's url will be removed from cafasp's meta-server.
CATEGORIES FOR CAFASP3
Any automated server can participate in CAFASP3, including those for Homology Modeling, Threading, Ab-Initio, Secondary Structure Prediction and Contacts Prediction.
ASSESSMENT OF PREDICTIONS
Predictions submitted to CAFASP will undergo an independent evaluation by the CAFASP automatic evaluation programs. The evaluation procedures will be determined by each coordinator and will be announced as soon as possible. If the CASP5 data will be made available, a comparison of the performance of CAFASP3 versus the regular CASP5 submission will also be carried out.
We encourage all server developers to understand the evaluation methods BEFORE they register, and if they find them inadequate, they may choose not to participate in CAFASP.
Currently the people involved in the CAFASP organization are Leszek Rychlewski, (firstname.lastname@example.org), Arne Elofsson (email@example.com), and Daniel Fischer (firstname.lastname@example.org).
The coordinators of the different prediction categories are:
Each coordinator will be in charge of the procedures that will be used in the evaluation and will be responsible for the automated evaluation of the servers in the corresponding category.
We invite interested parties to be involved in the various sub-committees.
CAFASP3 url: http://www.cs.bgu.ac.il/~dfischer/CAFASP3