Beyond Agreement: Standardizing Crowdsourced Synapse Annotations through Proofreading in EM Connectomics
Reliable synapse identification in volumetric EM is hampered by subtle, 3D cues that yield variable human judgments. We present a standardized proofreading protocol that pairs explicit, operational criteria with machine-learning candidate generation and a two-stage calibration of annotators. In two larval Drosophila melanogaster volumes imaged at 8x8x8 nm, five raters (expert + 4 calibrated annotators) reviewed model-proposed candidates using efficient node-based labels. Multi-rater judgments were aggregated with a probabilistic Dawid-Skene (DS) model to produce consensus labels with calibrated uncertainty. Post-calibration, individual annotator accuracy versus the expert improved (McNemar p<0.05 for all raters), DS-expert agreement increased, and DS posterior entropy decreased for true positives/negatives, indicating more decisive consensus; gains were modest and dataset-dependent in chance-corrected agreement (Krippendorff's ). By making uncertainty explicit, this protocol converts noisy judgments into auditable supervision suitable for training and evaluation, while honestly communicating residual ambiguity essential for reliable and robust connectomics at scale.