What follows includes a general description of CAFASP, the rules of CAFASP1, the detailed results per group and target and other data.

One of the most important events in the protein structure prediction community is the Critical Assessment of Structure Prediction Meeting, which was held on December 1998.

As a complement to the enormous value of this event, here I propose to discuss the differences between the Critical Assessment of COMPUTER AIDED Structure Prediction (CACASP) and the Critical Assessment of FULLY AUTOMATED Structure Prediction (CAFASP).

In the current recipee of CASP, predictors submit their predictions using whatever techniques they choose; some report the exact results of fully automated methods without any changes; some "edit" the results from the programs and create a "computer aided" + human prediction; and some use mostly their brains with little aid from computers. Thus, the current recipee is assessing the performance of humans and does not allow for the strict assessment of the methods themselves. The latter is what ultimately non-expert users are interested in. Predictions in which human intervention has played a role can not always be objectively assessed, and in most cases, they are hard, if not impossible to reproduce by others. The value of the current critical assessment is thus not fully exploited.

What I propose is to take advantage of the invaluable framework provided by CASP3 and run in parallel, and independently a CAFASP1 evaluation. To this end, for each target in the fold recognition track, results from fully automated programs and servers will be compiled and made available thru the web to all, along with their assessment.

Deadline for submissions:    1.8.99
                                               Should have been as in CASP3, but we missed it, so it
                                               should be immediate, but we cant, so it
                                               should be next week, but it is christmas, so it
                                               should be the following week, but it is new year, so
                                               we will settle for 1.8.99 until 11:59 PM (GMT).

                Any server not ready by that date will be able to participate in CAFASP2.

 SERVERNAME:   xxxx
 TARGET:               T00xx
 PARAMETERS:     DEFAULT/ others
 URL:                      local url where (i) an exact copy of this submission and (ii) the exact server's
                               output will be kept
 SERVER'S URL:   url where submissions to this server are accepted

followed by 10 to 15 lines each containing 2 mandatory columns and 3 optional ones:

RANK  FOLD  SCORE  LENGTH_ALIGNMENT   SEQ_ID%

after this list, the keyword END must appear. After the END, the predictor can include the sequence-structure alignments in whatever
format. See example .

If parameter setting is required, the parameters must be described. However, the same
setting must be used for all targets!

If the SCORE column is provided, a "documentation" line explaining the reliability of the
scores can be included such as: "scores in the range 3-3.5 are 70% of the time correct".

To help in the assessment process and to avoid future misunderstandings, each submission can be acompanied by an "auto-assessment" which will say at which ranks the predictor believes he got the correct fold. If the predictor doesn't know, then he/she can mention this.Obviously, the assessment procedure will not use this annotation; this may only be of help in defining our list of correct hits, which for some cases it is not easy to determine.

To compile information on alignments, we encourage predictors to include in their submissions the alignments for each top hit in the exact casp3 format. We may eventually use this data to assess alignment quality. However, we will not enforce this at this time.

Notice that for those targets for which the structural match comprises only parts of the sequence, and for which we can define this unambiguously, it is required that the server be run using the partial sequence. For now we are requiring to submit in addition the full target sequence. We will see later how to account for these targets.

It may be the case that for some of the targets below we won't be able to define an unambiguous structural match. At this point it is encouraged to submit all of them, but it might happen that we will remove from the assessment a few of them.

Validation of submissions: Each participant will be assigned two "policemen" which will be responsible
   to validate that the submission exactly corresponds to the output of the server. Inconsistencies will be
   reported to all participants, and a vote will be carried out to decide whether the submission should be
   disquailified. The predictor will have a chance to correct his submission once. A disqualified target will receive a penalty of 3 points. Police will be in effect until the proofs of the    paper are returned.

The best servers will in addition have the right to write one paragraph of 100-150 words describing
how their automated results differ from the CASP3 submissions, what was done differently for CASP3,
both for good or for bad, etc.

Any participant may request to be removed from this experiment with or without explicit mention.

Cafasp1 is a fully democratic event. All rules and decisions will be voted for. Exact rules will be known to all before the deadlines. Exact dates will be published with time. If delays appear, we will inform all. All programs developed for cafasp1 will be available here to all. All submission and assessment data will be available to all at all times.

Targets for which a non-ambiguous scop fold can be assigned will be considered. A correct hit is one that
has the same scop classification (first 2 numbers). ( see D. Jones alternative suggestion ). The "correct" hits were decided using some data D. Jones compiled at: ftp://ribosome.bio.warwick.ac.uk/pub/CAFASP1/comparisons. For a correct hit at rank i, 1/i points will be credited.
As more than one correct hit can appear in the list, only the hit at the highest rank will be considered.

A server may execute more than one single method, but no more than 7. A method is defined as a single
set of parameters and programs, the same for all targets.

A table with the performance is the result of this experiment. Its columns will be the targets, its rows the
methods used. The results are divided into 5 categories: 1) targets with members at the family level (easy, homology modeling targets), 2) targets with members at the superfamily level and with an unambiguously identifiable fold, 3) targets with members at the fold level and with an unambiguously identifiable fold, 4) targets with no unambiguosly identifiable fold or with similarities beyond current capabilities of structural comparison and/or fold recognition methods and 5) targets with yet unsolved structure (still blind predictions).
 
 

TARGETS WITH MEMBERS AT THE FAMILY LEVEL

	T0055	T0057	T0068	T0070	T0062	TOTAL
frsvr_SDP	1	1	1	1	1	5
frsvr_SDPMA	1	1	1	1	1	5
Karplus1	1	1	1	1	1	5
Karplus2	1	1	1	1	1	5
Karplus3	1	1	1	1	1	5
Topits	1	1	0	2	1	3.5
GenThreader	1	1	1	1	1	5
3D-PSSM	1	1	2	1	1	4.5
(1D+3D)-PSSM	1	1	1	1	1	5.0
pscan	1	1	0	1	1	4
BASIC	1	1	1	1	1	5
frsvr_SDPMA2	1	1	1	1	1	5
PSINCBI	1	1	1		1	5
PSIBork	1	1	1	1	1	5
total	14	14	10.5	12.5	12
OTHER
LATE	ARRIVALS	SHOWN	BUT ONLY	PARTIALLY	EVALUATED:
H3P2	1	2	0	0	7	1.64

In each entry the number of points are shown followed by a normalized number which is computed as
points squared divided by total number of points in target.

TARGETS WITH MEMBERS AT THE SUPERFAMILY LEVEL (AND WITH AN UNAMBIGUOUSLY IDENTIFIABLE FOLD):

TARGETS WITH MEMBERS AT THE FOLD LEVEL:

TARGETS WITH DOMAIN DEFINITIONS IN THE ABOVE TABLES, FOR WHICH WE SUBMIT THE INDIVIDUAL DOMAINS

TARGETS WITH UNSOLVED STRUCTURE:

TARGETS WITH NEW FOLDS

	T0052	T0056
frsvr_SDP	HIGHEST SCORE: 4.52	HIGHEST SCORE: 3.70	MAX SCORE: 4.52
frsvr_SDPMA	HIGHEST SCORE: 4.30	HIGHEST SCORE: 3.05	MAX SCORE: 4.30
PSIBork
Karplus1	HIGHEST SCORE: -4.06	HIGHEST SCORE: -3.85	MAX SCORE: -4.06
Karplus2	HIGHEST SCORE: -162.05	HIGHEST SCORE: -5.64	MAX SCORE: -162.05
Karplus3	HIGHEST SCORE: -7.18	HIGHEST SCORE: -6.53	MAX SCORE: -7.18
topits	HIGHEST SCORE: 3.09	HIGHEST SCORE: 2.46	MAX SCORE: 3.09
GenTHREADER	HIGHEST SCORE: 0.420	HIGHEST SCORE: 0.365	MAX SCORE: 0.420
PSINCBI
3D-PSSM	HIGHEST SCORE: 2.99	HIGHEST SCORE: 6.09	MAX SCORE: 2.99
(1D+3D)-PSSM	HIGHEST SCORE: 6.36	HIGHEST SCORE: 6.44	MAX SCORE: 6.36
pscan
BASIC	HIGHEST SCORE: 25.6	HIGHEST SCORE: 4.6	MAX SCORE: 25.6
frsvr_SDPMA2	HIGHEST SCORE: 3.57	HIGHEST SCORE: 3.20	MAX SCORE: 3.57

TARGETS WITH MEMBERS AT THE FOLD LEVEL BUT IMPOSSIBLE TO FIND OR EVALUATE (OR NO HITS BY VAST):

OTHER TARGETS (NOT EVALUATED)?

	T0084	T0064	T0065	T0079.1.1	T0079.2.1
frsvr_SDP
frsvr_SDPMA
GenTHREADER	1/1/0.??	1/1/0.??	1/?/0.??
(1D+3D)-PSSM				1/3/0.??	1/?/0.??
BASIC	1/2/0.??

TARGETS WITH MEMBERS AT THE FAMILY LEVEL (<30% Seq. Id.)

TARGETS WITH MEMBERS AT THE SUPERFAMILY LEVEL:

TARGETS WITH MEMBERS AT THE FOLD LEVEL AND/OR WITH PARTIAL TOPOLOGY: (TARGETS FOR WHICH AN UNAMBIGUOUS SCOP ASSIGNMENT CAN NOT BE MADE WILL NOT BE ASSESSED AT THIS POINT!)

TARGETS WITH DOMAIN DEFINITIONS IN THE ABOVE TABLES, FOR WHICH WE SUBMIT ALSO THE DOMAIN(S)

TARGETS THAT ARE IMPOSSIBLE TO FIND (NO VAST HIT OR SIMPLY NOT SUITABLE FOR FOLD RECOGNITION):

TARGETS THAT STILL ARE BLIND PREDICTIONS:

TARGETS WITH NEW FOLDS

T0052

T0056

 -  Average server's time response.
 -  Availability of the server.
 -  Quality of the interface, ease of use, ease of understanding the results, etc.

Please send your comments to dfischer@cs.bgu.ac.il. Valuable suggestions will be incorporated until 12.31.98. Other suggestions will be compiled here for future use in CAFASP2.