Directions for participating in the WASSA-2017 Emotion Intensity Shared task via CodaLab

Steps:

1. Create an account in CodaLab (https://competitions.codalab.org/). Sign in.
   
   
2. Edit your profile appropriately. Make sure to add a team name, and enter names of team members. (Go to "Settings", and look under "Competition settings".)
   
   
3. Proceed to task webpage on CodaLab. A direct link to the Emotion Intensity CodaLab competition is here. Read information on all the pages.
   
   
4. Download data: training, development, and test (when released)
   
   
5. Run your system on the data and generate a submission file. A valid submission file for CodaLab is a zip compressed file containing the following files:
   • anger-pred.txt
   • fear-pred.txt
   • joy-pred.txt
   • sadness-pred.txt
     

   Each of these files must follow the official submission format. CodaLab does not place any restrictions on the name of the zip file name. Note: Do not submit a zip compressed folder or directory, that is, do not submit a zip of a folder that includes the four files.
   

6. Make submissions on the development set (Phase 1).
   • Wait a few moments for the submission to execute.
   • Click on the ‘Refresh Status’ button to check status.
   • Check to make sure submission is successful:
     • System will show status as “Finished”
     • Click on ‘Download evaluation output from scoring step’ to examine the result.
     • If you choose to, you can upload the result on the leaderboard.
   • If unsuccessful, check error log, fix format issues (if any), resubmit updated zip.
   • Number of submissions allowed is restricted to 50.
     
     
7. Once the evaluation period begins, you can make submissions for the test set (Phase 2). The procedure is similar to that on the dev set. These differences apply:
   • The leader board will be disabled until the end of the evaluation period.
   • You cannot see the results of your submission. They will be posted on a later date after the evaluation period ends.
   • You can still see if your submission was successful or resulted in some error.
   • In case of error, you can view the error log.
   • Number of submissions allowed is restricted to 10. However, only your final valid submission will be your official submission to the competition.
     

FAQ

Q. Why restrict the number of official submissions to one?

A. Since this is a competition, we do not want teams to submit a large number of submissions using different parameters and systems without being confident which will work best.

Even though the number of official submissions is restricted to one, the gold data will be released soon after the evaluation period. Thus you can use it to determine results from many different system variants. You are strongly  encouraged to report these additional results in the system-description paper in addition to the official submission results.

Q. Can I be involved in more than one team? (And thus make more than one official submission on the test set?

A. No. This can be used to abuse the one-official submission rule, which can be unfair to other participants.

Q. How do I include more than one score on the leaderboard?

A. CodaLab allows only one score on the leaderboard per user.