Automatic Evaluation of Labeling Quality in Crowdsourced Object Detection and Segmentation

Overview

Large-scale object detection and segmentation datasets (COCO, Open Images, Waymo) depend on crowdsourced annotations from multiple annotators, inevitably introducing labeling inconsistencies. These inconsistencies—missing annotations, bounding box misalignments, segmentation boundary errors, and class confusion—inject noise that degrades downstream model performance. While manual review could identify such errors, it scales poorly across millions of annotations. Current datasets lack systematic methods to automatically evaluate and detect labeling quality issues, and standard model training can unknowingly absorb and perpetuate label noise rather than flag problematic annotations for correction.

Research engagement

The student will engage in literature review (surveying label-quality and annotation-consistency methods), lab-based computational work (implementing and testing detection/segmentation models and quality-assessment algorithms), data analysis (annotation comparison, statistical evaluation), and technical writing (documenting methods and findings in a final report).

Research activities

The student will conduct a literature review on label quality assessment methods, analyze annotation inconsistencies in existing crowdsourced datasets (e.g., COCO, Open Images), implement and benchmark automatic quality-detection methods (e.g., agreement metrics, model ensembles, statistical outlier detection), and validate findings through empirical experiments comparing detected errors against ground truth; the student will work closely with their supervisor through regular meetings, with opportunities for collaboration with other research students in the SAIVT lab working on related computer vision projects.

Research skills

Students will gain hands-on experience in computer vision and dataset quality assessment, including practical skills in annotation analysis, statistical evaluation metrics (e.g., IoU, inter-annotator agreement), and implementing detection/segmentation models using PyTorch or TensorFlow; they will also develop research skills in critically evaluating academic literature, designing empirical experiments, and communicating technical findings through written reports."

Outcomes

The expected outcome is a validated, practical method for automatically detecting inconsistent or low-quality labels in crowdsourced object detection and segmentation data, improving dataset reliability and downstream model performance."

Skills and experience

Candidates should have basic Python programming skills and coursework or project experience in computer vision or machine learning; no prior experience with annotation quality assessment is required.

Start date

2 November, 2026

End date

19 February, 2027

Location

GP-S825

Keywords

• data labelling
• object detection and segmentation
• computer vision
• automatic quality checking

Contact

osman trusun

0416920626

osman.tursun@qut.edu.au