<!--#set var="TITLE" value="Obtaining CinC Challenge 2001 Scores"-->

<!--#include virtual="/head.shtml"-->

<h2>About the events</h2>

<p>

The PAF Prediction Challenge Database consists of 50 pairs of half-hour ECG

recordings. Each pair of recordings is obtained from a single 24-hour ECG.

Subjects in group A experienced PAF; for these subjects, one recording

ends just before the onset of PAF, and the other recording is distant in

time from any PAF (there is no PAF within 45 minutes before or after the

excerpt). Subjects in group N do not have PAF; in these, the times of the

recordings have been chosen at random.

<p>

The number of subjects in group N is <em>n</em>

(where 20 &lt; <em>n</em> &lt 30).  The number of subjects in group A is

thus 50-<em>n</em>, a number also between 20 and 30.

<p>

<b>Event 1 (Screening) is intended to determine if subjects in group A can

be distinguished from those in group N</b>. (In other words, can individuals at

risk of PAF be identified within a larger population, based on their ECGs?)

Since each subject is represented by a pair of consecutively numbered records

in the database, entrants in event 1 of the challenge should provide one

classification (`A' or `N') for the 50 odd-numbered records only

(<tt>t01</tt>, <tt>t03</tt>, <tt>t05</tt>, ... <tt>t99</tt>).  The number of

correct classifications (0 to 50) is the event 1 score.

<p>

<b>Event 2 (Prediction) is intended to determine if subjects in

group A have distinctive and detectable changes in their ECGs

immediately before PAF</b>.  (In other words, is the imminent onset of

PAF predictable in an individual known to be at risk of PAF?)  A

successful method for doing so should be able to determine which

record of each pair of group A records immediately precedes PAF.  If

the identities of the group A records were known, it would be

sufficient to classify these records only; since the goal of event 1

is to identify group A, however, we have not provided this

information!  Entrants in event 2 of the challenge must therefore

classify exactly one of each pair of records in the test set as `A'

(defined as ``<em>immediately preceding PAF, if the patient belongs to

group A</em>''), and the other as `N' (defined as ``<em>not

immediately preceding PAF</em>'').  One point is awarded for each

correctly classified record pair, so that the event 2 scores range

from <em>n</em> to 50 (the lower bound is <em>n</em> because the group

N subjects are always considered correctly classified).

<h2>Preparing your entry</h2>

<p>

Your entry needs to be prepared in a special text format, which depends on

the event you are entering:

<ul>

<li> For event 1 (Screening), download this <a

href="template-test-1">template</a>,

and replace the `?' characters with your classifications (`A' or `N')

for each of the 50 record pairs in the test set.

<li> For event 2 (Prediction), download this <a

href="template-test-2">template</a>,

and replace the `?' characters with your classifications (`A' or `N')

for each of the 100 records in the test set.

</ul>

<h2>Submitting your entry for scoring</h2>

<p>

Please double-check your entry before submitting it to be sure it is in

<strong>plain text</strong> format, exactly like the template files.

<em>HTML-formatted entries will confuse the automated scorer and will not be

scored.</em>

<p>

To submit your entry successfully, you will need:

<ul> 

<li> A web browser that supports HTTP uploads, such as Netscape 2.0 or

 later, or MS Internet Explorer 4.0 or later.

<li> A valid e-mail address that is known to us.  Please

 <a href="/register.shtml">sign in</a> before submitting your entry.

 (If you signed in on a previous visit, you do not need to sign in again.)

 All entries are logged by date, time, and the e-mail address of the

 entrant.

 <p>

 <b>Important:</b>  Your score will be sent by e-mail to the address you

 specify.  Please be sure that your e-mail address is entered correctly,

 or you won't be able to view your score.

<li> A password.  If you don't yet have a password, or if you

 have forgotten your password, go to the entry form (see below), fill

 in only your e-mail address, click on the "Get password" button, and a

 password will be sent to your e-mail address immediately.

</ul>

<p>

After filling in your e-mail address and password on the 

<a href="http://www.physionet.org/cgi-bin/cinc-2001-score">entry form</a>.

enter the name of the file that contains your correctly-formatted entry

(you may use the <b>Browse</b> button to find it), indicate which

event you wish to enter, and click on <b>Upload entry</b>.

<p>

If you wish to enter both events, you may do so, but you will need to

submit separate entries, one in each format, in this case.

<h2>Notes</h2>

<p>

Please be careful not to submit more than one copy of the same results, since

each submission will be counted against your quota of submissions.  If this

happens inadvertently, please

<a href="mailto:webmaster@physionet.org subject=duplicate entry">send a

note</a> quoting the date and time of the duplicate submission;  your

submission counter will be adjusted manually if the error can be confirmed.

<p>

If you are tempted to try submitting many entries in order to learn about the

correct classifications, why not play <a

href="http://www.lizardpoint.com/fun/mastermind/mastmind.html"

target="other">Mastermind</a> instead?  We will reject obvious attempts to

circumvent the spirit of the challenge in this way, including entries that do

not have the correct number of classifications (50 for event 1, 100 for event

2), entries in event 1 with fewer than 21 or more than 29 `N' classifications,

entries in event 2 that do not have exactly one `A' classification for each of

the 50 record pairs, and large numbers of entries from a single entrant.

Delivery of your score may be delayed if you submit more than one entry per

event per day.

<p>

UNIX versions of Netscape, and possibly other browsers, will allow the

user to specify a directory name in the <b>File to upload</b> field of the

entry form, but the browser will not transmit anything in this case and will

hang until the user interrupts it by clicking on <b>Stop</b>.  This behavior

is a bug in the browser and cannot be detected or avoided by the PhysioNet web

server.  Other browsers will detect and disallow attempts to upload

directories.

<a name="faq"><h2>Frequently asked questions</h2></a>

<p>

If your question is not answered below, please consult the

<a href="/faq.shtml">PhysioNet FAQ</a>.

<p>

<b>Group B subjects are not explicitly defined in the text. I assume these

are Group N subjects?</b>

<p>

This is correct.  Group B was renamed Group N.  There should no longer be any

references to Group B.  Sorry for the confusion!

<p>

<b>We have to put exactly one A under one of the subjects of each even-odd

pair in event 2.  Does this mean that exactly 50 As and 50 Ns are expected in

a submission for event 2 scoring?</b>

<p>

This is also correct.

<p>

<b>I presume this scoring scheme is produced in order to allow submission for

only one of the events, and/or to eliminate the possibility of contradictory

submissions for event 1 and event 2, from the same author?</b>

<p>

You are welcome to submit results for both events, but they must be separate

submissions.  Because of the nature of the events, it would be a poor strategy

to submit the same classifications in both cases, even if the autoscorer

permitted this.  In event 1, the goal is to minimize the number of

misclassified subjects in both groups, so it would be a mistake to mark one

record in each pair as `A' (unless you really believe that all 50 subjects

belong to group A; as stated above, however, there are between 20 and 30

subjects in group A).

<p>

In event 2, the goal is to minimize the number of misclassified records in

group A only.  In this event, it would be a mistake to omit classifying one

record in any pair as `A', because if that pair turned out to belong to group

A, you would forfeit the opportunity to score a point for that pair.

<p>

The submission formats for the two events were deliberately designed to be

different.  This makes it easy for the autoscorer to recognize a result

submitted to the wrong event in error, so that such a case doesn't get counted

against your quota of submissions.

<p>

<b>Still, one can have a high score on one event and a poor score on the

other. How do you treat this?</b>

<p>

The two events are really addressing rather different questions, using the same

data set.  (This, in a nutshell, is one of the principal <em>raisons

d'Ítre</em> for PhysioNet: to make it possible for data to be used to address

more than one question!)  We will award US$500 to the winner of event 1, and

US$500 to the winner of event 2.  If you win both events, you will receive

US$1000.

<p>

<b>Who is eligible to win an award?</b>

<p>

To be eligible for an award, you must enter the competition

officially (by submitting at least one set of results for scoring, and an

abstract describing your work to Computers in Cardiology 2001, no later than

the abstract deadline of 1 May 2001).  You may submit revised results at

any time until noon GMT on 21 September 2001.  At that time, the winners

will be determined based on the highest scores received by official entrants.

The date and time of the submission will be used as a tiebreaker if necessary.

<p>

Submissions from members and affiliates of our research groups at MIT, Boston

University, Harvard Medical School, Beth Israel Deaconess Medical Center, and

McGill University are not eligible for awards, although all are welcome to

participate.

<p>

If you have missed the deadline for submitting an abstract to Computers in

Cardiology 2001, we still encourage you to participate unofficially.  If you

receive a high score, your achievement will be recognized on this web site.

If you are able to attend Computers in Cardiology, you will have an opportunity

to discuss your work informally with other participants during a workshop

session.

<p>

<b>Would it be possible to submit my classification result and ask for

scoring during the weekend?  How long will it take for scoring?</b>

<p>

The scoring is performed automatically -- just follow the instructions

above for submitting your results.  Your score is sent to you by return email

immediately.

<p>

If your results are not properly formatted, you will receive an email reply

that will indicate this; if this happens, please review the instructions

carefully and try again.  Your friendly challenge organizer reads his

<a href="mailto:webmaster@physionet.org?subject=CinC Challenge 2001 Help!">

email</a> frequently (on the weekends, too), especially as the critical

deadlines approach, and promises to answer your questions quickly if you get

stuck!

<p>

<b>If I would like to submit more than one result for an event, should I

use the same file name for each submission of the text file (the default

names are template-test-1.txt for event 1 and template-test-2.txt for event

2)?</b>

<p>

This doesn't matter (your submission is archived using a file name

constructed from the date, time, and your email address).

<p>

<b>About the result template for event 1 (PAF risk assessment), there are

only odd-numbered records in the template.  What should I do if the

classification results for the two records in a pair are different?

Should I just write down the results for the odd-numbered records, or can I

choose the more reliable one from each pair of records?</b>

<p>

Definitely, you should choose whatever classification you consider to be more

reliable, provided that you are using objective criteria for making the choice.

Each pair of records comes from one subject, so for purposes of event 1 the

correct classification is by definition the same for both records in the pair:

either the subject has PAF (`A') or not (`N').  Please keep in mind that the

classifications must be determined by an algorithm, not manually, so if your

algorithm produces a confidence level, for example, and you choose the

classification with the higher confidence level each time, this is acceptable

(your decision is algorithmically determined); but it would not be acceptable

for you to decide on subjective grounds which classification to use in each

case.

<p>

<b>According to the instructions for abstract submission, I should give the

result of my algorithm.  Does that mean the scoring result?  So I have to

get the scores first before I can submit the abstract, right?</b>

<p>

This is correct: your abstract should include your score.  You should consider

this score to represent preliminary results; your abstract will be judged on

the basis of the ideas you discuss and not on the score itself (although a good

score might help a marginal abstract, a poor score will not rule out a good

abstract).  You may submit revised results for scoring before or after the

abstract deadline, at any time until noon GMT on 21 September 2001.  Your best

results determine your standing in the competition.  In your finished

manuscript (which is not due until 25 September 2001) you should include your

best results and describe what changes, if any, you made in order to improve

your initial results.

<hr>

<!--#include virtual="/footer.shtml" -->

</body>

</html>