The VOC challenge encourages two types of participation: (i) methods which are trained using only the provided "trainval" (training + validation) data; (ii) methods built or trained using any data except the provided test data, for example commercial systems. In both cases the test data must be used strictly for reporting of results alone - it must not be used in any way to train or tune systems, for example by runing multiple parameter choices and reporting the best results obtained.
If using the training data we provide as part of the challenge development kit, all development, e.g. feature selection and parameter tuning, must use the "trainval" (training + validation) set alone. One way is to divide the set into training and validation sets (as suggested in the development kit). Other schemes e.g. n-fold cross-validation are equally valid. The tuned algorithms should then be run only once on the test data.
In VOC2007 we made all annotations available (i.e. for training, validation and test data) but since then we have not made the test annotations available. Instead, results on the test data are submitted to an evaluation server.
Since algorithms should only be run once on the test data we strongly discourage multiple submissions to the server (and indeed the number of submissions for the same algorithm is strictly controlled), as the evaluation server should not be used for parameter tuning.
We encourage you to publish test results always on the latest release of the challenge, using the output of the evaluation server. If you wish to compare methods or design choices e.g. subsets of features, then there are two options: (i) use the entire VOC2007 data, where all annotations are available; (ii) report cross-validation results using the latest "trainval" set alone.Policy on email address requirements when registering for the evaluation server
In line with the Best Practice procedures (above) we restrict the number of times that the test data can be processed by the evaluation server. To prevent any abuses of this restriction an institutional email address is required when registering for the evaluation server. This aims to prevent one user registering multiple times under different emails. Institutional emails include academic ones, such as firstname.lastname@example.org, and corporate ones, but not personal ones, such as email@example.com or firstname.lastname@example.org.
The main challenges have run each year since 2005. For more background on VOC, the following journal paper discusses some of the choices we made and our experience in running the challenge, and gives a more in depth discussion of the 2007 methods and results:
The PASCAL Visual Object Classes (VOC) Challenge
Everingham, M., Van Gool, L., Williams, C. K. I., Winn, J. and Zisserman, A.
International Journal of Computer Vision, 88(2), 303-338, 2010
Bibtex source | Abstract | PDF
The table below gives a brief summary of the main stages of the VOC development.
|2005||Only 4 classes: bicycles, cars, motorbikes, people. Train/validation/test: 1578 images containing 2209 annotated objects.||Two competitions: classification and detection||Images were largely taken from exising public datasets, and were not as challenging as the flickr images subsequently used. This dataset is obsolete.|
|2006||10 classes: bicycle, bus, car, cat, cow, dog, horse, motorbike, person, sheep. Train/validation/test: 2618 images containing 4754 annotated objects.||Images from flickr and from Microsoft Research Cambridge (MSRC) dataset||The MSRC images were easier than flickr as the photos often concentrated on the object of interest. This dataset is obsolete.|
||This year established the 20 classes, and these have been fixed since then. This was the final year that annotation was released for the testing data.|
|2008||20 classes. The data is split (as usual) around 50% train/val and 50% test. The train/val data has 4,340 images containing 10,363 annotated objects.||
|2009||20 classes. The train/val data has 7,054 images containing 17,218 ROI annotated objects and 3,211 segmentations.||
|2010||20 classes. The train/val data has 10,103 images containing 23,374 ROI annotated objects and 4,203 segmentations.||
|2011||20 classes. The train/val data has 11,530 images containing 27,450 ROI annotated objects and 5,034 segmentations.||
|2012||20 classes. The train/val data has 11,530 images containing 27,450 ROI annotated objects and 6,929 segmentations.||
The preparation and running of this challenge is supported by the EU-funded PASCAL2 Network of Excellence on Pattern Analysis, Statistical Modelling and Computational Learning.