Segmentation Contest

The Segmentation Contest

Keywords: MRI, heart, myocardial infarction, normal case, delayed-enhancement, no-reflow, segmentation

The goal of the Segmentation contest is to compare the performance of automatic methods on the segmentation of the myocardium for all the DE-MRI exams (including normal and pathologic cases), and the myocardial infarction and no-reflow areas on exams classified as pathologic ones.

The cohort consists of data extracted from 150 MRI exams (all from different patients) divided into 50 cases with normal MRI after the injection of a contrast agent and 100 cases with myocardial infarction (and then with a hyperenhanced area on DE-MRI), whatever their inclusion in the cardiac emergency department. The cases were randomly selected from our database. The inclusion criteria are patients received in the cardiac emergency department with acute disease (with symptoms of heart attack) and that undergo cardiac MRI. The exclusion criteria are patients with contraindications to the MRI and cardiac chronic diseases. There is an unbalanced distribution between normal and pathological cases, corresponding roughly to real life in managed exams in a MRI department.

The overall dataset was created from real clinical exams acquired from the MRI department at the University Hospital of Dijon (France). Each group was clearly defined according to physiological parameters and the presence or absence of a disease area on DE-MRI. The data are DE-MRI in short axis orientation, and a series of images covering the left ventricle.

Training and test case characteristics

The dataset provides 100 cases for the training and 50 cases for the testing.

Every training and test case represents a DE-MRI exam of the left ventricle. An exam (i.e. a case) consists of a series of 5 to 10 short-axis slices covering the left ventricle from the base to the apex. The ground-truths (contours of the relevant areas) will be provided with the training dataset.

The training set with full ground-truth will comprise 100 cases (67 pathological cases, 33 normal cases) randomly selected among the 150 subjects.

The testing set is made of data from 50 subjects (33 pathological cases, 17 normal cases), all different from those in the training set.

For the training and the testing set, the unbalanced distribution between normal (1/3) and pathological (2/3) cases corresponds roughly to real life in managed exams in a MRI department.

Annotation characteristics and data pre processing

The contours are manually drawn for all the cases. The gold standard for the contours is obtained through a manual segmentation carried out by two experts.

First, the left ventricular endocardial and epicardial borders, as well as the infarcted area and the no reflow areas, if present, were first outlined by the first expert, an experienced user (a cardiologist with 10 years of experience in cardiology and MRI).

Then, the second expert (a well-trained biophysicist with 20 years of experience) went through every outline and made some changes when necessary. These contours are then transcribed in label field image. From the contours, specific labels are assigned to each voxel depending on their location: in the background, in the myocardium, in the cavity, in the myocardial infarction area and in the no-reflow area, respectively.

To prevent the drawback of the displacement of the heart location between slices due to different breath-holds, the slices are realigned according to the gravity center of the area defined by the epicardial contour. The same process is applied for the training and test cases.

Assessment method - Metrics

The clinical metrics are those that are the most widely used in cardiac clinical practice and the geometrical metrics are the classic ones used in the segmentation evaluation.

The clinical metrics include the average errors for the volume of the myocardium of the left ventricle (in mm3), the volume (in mm3) and the percentage of MI and no-reflow. For the calculation of the area of MI (and the percentage of MI), the no-reflow area is included in the MI

The geometrical metrics are the average DICE index for the different areas and Hausdorff distance (in 3D) for the myocardium. The Dice index gives an overall information about the quality of the segmentation, the Hausdorff distance highlights the outliers.

For each geometrical and clinical metric, a ranking will be done, and the final ranking consists of the sum of the ranking for each metric.

Submission method

The participants are invited to submit their results for all the cases (corresponding to the segmentation of the different areas, the algorithms are not be requested) by loading files in Nifti format (then 50 files are requested) with all the areas of interest to this email address : submission@emidec.com . The name of each Nifti file with the contours is the same as the name of the file with the images (e.g. case_101.nii).

During the training phase, the code for the metric calculation will be available to the participants and thus they can assess themselves their method.

During the testing phase, each participant can submit only three times their results. The best one will be retained. The retained metrics will be provided for each team after the challenge (end of September 2020). The global ranking will be provided after the challenge, during the conference.

Moreover, the participants will be requested to submit an article of four pages, following the MICCAI format, describing the methodology, with the opportunity to publish a long version of the article along with LNCS Challenge Proceedings. Articles will be reviewed by the organizing board and published on line if meeting the expected quality.

Important dates


Mid-April : Release of the training cases
September, 1 and for 2 weeks: Release of the testing dataset

Registration and challenge

Mid-April : Start of the registration process
September, 15: End of registration and End of the challenge (deadline for the submission of the papers).
October, 4: Results of the challenge during the STACOM workshop.