Repository navigation
Evaluation metrics - Object detection #1132
Description
Is your feature request related to a problem? Please describe.
Need for the following metrics when dealing with object detection problems
Intersection over Union (IoU) measurements can either be made at the level of the bounding boxes of the objects or of the segmentation themselves.
At the element level
- Dice per element
- Centroid distance
At the level of the image considering independent elements:
- F1 score with IoU thresholds ranging from 0.5 to 0.95
- Average precision with IoU thresholds ranging from 0.5 to 0.95 https://cocodataset.org/#detection-eval
- Average recall with IoU thresholds ranging from 0.5 to 0.95 https://cocodataset.org/#detection-eval
- Correlation of detected volumes
- Number difference
- F1 with minimum overlap of 1 voxel
- Outline error https://link.springer.com/article/10.1186/1471-2342-12-17
- Detection error https://link.springer.com/article/10.1186/1471-2342-12-17
Additional context
This list of metrics is the output of the initial brainstorming of the metrics task force
Activity
mibaumgartner commentedon Nov 9, 2020
Amazing list of detection metrics, thanks for all the effort of the metrics task force to collect them!
I am curious about the
F1 with minimum overlap of 1 voxelmetric, could someone add some papers which use that metric? (I would like to read them ^^)Also a small addition:
The Free-Response Receiver Operating Characteristic (FROC) might also be a metric which should be added (used by LUNA , DeepLesion and RibFrac)
iliathesmirnov commentedon Nov 12, 2020
Hello everyone,
I am new here and would like to contribute to implementing some (or all) of the evaluation metrics on @csudre's list. Did anyone start working on them? If not, should I create a pull request (labelled [WIP]) and get going?
Thank you,
Ilia
wyli commentedon Nov 12, 2020
welcome and please go ahead! (I don't think anyone is currently working on them)
wyli commentedon Jan 5, 2021
today we discussed the requirement of FROC curves for lesion based evaluation
Can-Zhao commentedon Mar 30, 2023
Please check https://github.com/Project-MONAI/MONAI/blob/dev/monai/apps/detection/metrics/coco.py
TheHuntsman4 commentedon May 3, 2026
Hey everyone,
If no one is working on this, I would love to take this up. I do have a few questions regarding this; coco.py seems to be working with boxes, while the element-level and outline/detection metrics are more fit to be worked with masks. Is this assumption fine?
I believe the Dice per element metric was recently added here #8774 I would be happy to make a PR for the Centroid distance and then eventually work through the other 6 until all are implemented.
Thaks,
Aniketh
csudre commentedon May 5, 2026
Dear Aniketh
I think most of the requirements for allocation and localisation using centroid distance or box overlap... is available on the MetricsReloaded project linked to MONAI - the best way would probably be to make a wrapper around what is available there - in the association_localization file
Could that be an option?
Best wishes
Carole
TheHuntsman4 commentedon May 27, 2026
Thanks Carole, that makes sense.
I had a look and MetricsReloaded.utility.assignment_localization.AssignmentMapping seems like a good fit here, especially since it already supports the association/localization step for boxes, masks, and centroid-based matching.
My current thinking is:
That seems more consistent with the current MONAI approach as well, since we already have optional MetricsReloaded wrappers under monai.metrics.
If that sounds right, I can start with a small PR for centroid distance using this wrapper pattern, and then follow up with the remaining metrics in separate PRs.
Thanks,
Aniketh
csudre commentedon May 27, 2026