Test Harness

From ICPC-Contest Control Standard

A Test Harness is an automated testing framework.

This article contains information about the input, output and tests for systems implementing the CCS

Comment This needs to be split into two articles, keep requirements in this one, and move other information to another article Douglas Lane 17:47, 17 April 2011 (CEST)

Input to the Test Harness

File name
contest.yaml file
problem.yaml and all associated files
group.tsv file
teams.tsv file
userdata.tsv file

Testing Configuration Changes

Description Input Compare to Notes
Test contest information Load contest.yaml file Event Feed missing default-clars
Test problems configuration Load problemset.yaml and Problem format files Event Feed missing everything except:
letter, name, balloon color, balloon color RGB
Test language configuration Load languages.yaml file Event Feed missing compiler, compiler-args,
runner, runner-args
Test testcase configuration Load Problem format file Event Feed TBD
Test group configuration Load group.tsv file Event Feed missing Group ID
Test team configuration Load teams.tsv file Event Feed missing Reservation ID, Group ID, Country
Test validator Load Problem format file TBD TBD
Test judgement names contest.yaml Event Feed TBD
Test notification Judgement Event Feed TBD

Testing Run Submissions

Description Input Compare to
Test run submission Scriptable Submissions runs Event Feed
Test run judgement Scriptable Submissions runs Event Feed

Testing Clarifications

Description Input Compare to
Test clar submission Scriptable Submissions clars Event Feed
Test clar answered Scriptable Submissions clars Event Feed
Comment: There is currently no CCS-defined way to automatically answer a clarification Douglas Lane 03:57, 20 February 2011 (CET)

Testing Standings

Description Input Compare to
Test standings change Scriptable Submissions runs scoreboard.tsv

Testing Notifications

Description Input Compare to
Test notification sent Scriptable Submissions runs Not defined

Testing Prohibited Operations

In order to test each of the Prohibited Operation a set of standardized messages needs to be defined.

Operation Expected Message
Using libraries except those explicitly allowed Team's program used unauthorized library/function
Executing other programs Team's program executed program "/bin/perl"
Creating new processes Team's program tried to start new process
Creating new threads Team's program attempted to create a thread
Reading any files Team's program tried to read file "c.in"
Creating files Team's program tried to create file "c.in"
Sending signals to other programs Team's program tried to use a signal N
Side-stepping time or memory limits Team's program attempted to exceed limit
Team's program attempted to exceed memory limit
Sending or receiving network traffic Team's program tried to write network output
Team's program tried to read network input

Testing Validators

Per the Output validator article test validators to report proper results, esp. file format.

Comment TODO complete writing this section Douglas Lane 23:31, 28 February 2011 (CET)
Reporting a judgment TBD
Reporting Additional Feedback TBD