When a held-out split is not reproducible: an Alchemyst-Crucible audit

Kvasir-SEG is a genuine contribution to medical-imaging research: 1,000 colonoscopy images with segmentation masks, released by @Simula Research Laboratory through SimulaMet and the Cancer Registry of Norway. The accompanying 2020 paper from Simula researchers reports a ResUNet baseline on an 80/10/10 train/validation/test split.

Source paper: *Kvasir-SEG: A Segmented Polyp Dataset* (Jha et al., MMM 2020). An open-access author manuscript (PDF) is included with this Story 2 package.

That is a useful benchmark. It is also the kind of result that can look fully reproducible while leaving one important question unanswered: were images from the same underlying source kept on one side of the split?

The reproducibility gap

This is not a claim that the authors deliberately leaked data, nor can it recover the leakage rate of their historical run. It identifies what cannot be reproduced from the paper alone:

• The 80/10/10 proportions are reported, but not the exact random seed or the

image-to-split assignment.

• No grouping policy is stated for images from the same procedure, lesion,

patient, or near-duplicate video-frame cluster.

• With image-level random splitting, related images can fall on different sides

of a train/test boundary even though the test image is formally held out.

Without the original assignment and grouping record, a later reader cannot reconstruct the particular experiment, test whether candidate same-source frames crossed the boundary, or determine whether the published score would change under a grouped split. The last question needs a separate controlled retraining study.

The failure that does not look like a failure

Reproducibility failures are often imagined as missing code, unavailable data, or a result that refuses to run. There is a quieter kind. The code executes. A metric appears. The test set is formally held out. But if two near-identical frames of one lesion land on opposite sides of an image-level random split, the test image is not an independent clinical case. A model can appear to generalise partly because it saw an almost identical view during training.

Nothing has to be malicious or broken for that to happen. It is an evaluation and reproducibility weakness, and it is hard to see from a methods paragraph or a green “run succeeded” badge.

That is the question the Alchemyst-Crucible reproducibility QC auditor raised for this pipeline. The agentic layer surfaces the check; the result below is a recorded computation, not an agent’s opinion.

Double-clicking the dataset node keeps the underlying image collection visible to the researcher: the live preview shows twelve representative Kvasir images and records the 1,000-image collection being audited. It is the starting point for checking whether image-level splitting is a meaningful proxy for an unseen clinical case.

What the permanent audit did

The platform ran the released 1,000 images through a pretrained ResNet50 and recorded its 2,048-dimensional embeddings. It compared each image with its nearest neighbour, defined a candidate near-duplicate with a cosine similarity of at least 0.92, and then repeated random 80/10/10 image partitions for 25 fixed seeds. A test image counted as exposed when its candidate cluster was also represented in training.

The permanent production record is run_CT7bpz89SipR, completed on 25 August 2026. Two preceding recorded confirmation tests, run_GCyS32v7CDNa and run_Hjl7wXapbBVW, produced the same values.

The audit found 12 candidate images in six pairs: 1.2% of the 1,000-image dataset. The closest inspected pair is a near-identical view of the same lesion. The remaining pairs are deliberately still labelled candidates: an embedding threshold is an audit trigger, not a substitute for human inspection.

The dashed red line is the stated audit threshold. The high-similarity tail is small, which is the honest result: the audit does not turn the dataset into a scandal. It identifies a concrete source-overlap risk that should be controlled before interpreting a held-out score as performance on unseen cases.

What 25 splits show—and what they do not

Across the 25 recorded random 80/10/10 partitions, the mean fraction of test images belonging to a candidate cluster also present in training was 0.56%, with a standard deviation of 0.70%. Several seeds had no candidate overlap; others had one or two exposed test images.

This is not the leakage rate in the authors’ original experiment. The historical split and seed were not published, so that quantity cannot be recovered from the paper. It also cannot tell us how much the published Dice score would change under a grouped split. Answering that requires a controlled retraining study, which is a separate next experiment.

It does establish something narrower and useful: the documented image-level 80/10/10 convention can place candidate same-source frames on opposite sides of the split. That risk is invisible unless somebody performs the audit.

The recommendation is modest because the evidence is modest

The right response is not to erase a benchmark result or assign intentions to its authors. It is to inspect the saved pairs, confirm which ones genuinely share a source, and keep every confirmed cluster wholly in train, validation, or test. The split rule and seed should then travel with the run, together with the pair table, plot and environment that produced the result.

That is what research reproducibility QC should do. It should surface checks a researcher may not know to ask for, keep the evidence behind the warning, and leave the final methodological judgement visible and human-owned.

The canvas keeps the logic visible

This is the entire audit logic in one view: the image dataset feeds a pretrained feature extractor, which feeds the near-duplicate audit. In a conventional notebook, resolving files, loading images, producing embeddings, comparing pairs, recording thresholds, repeating splits and writing artifacts can spread across hundreds of lines of implementation. Here, those steps are composed visibly and their settings are attached to the nodes and connections.

That is not an invitation to hand judgement to software. The researcher chooses the backbone, threshold, split count and the question worth asking; they can change the canvas, inspect or edit the code of an individual node, test that change before saving it, and decide whether a candidate pair is truly same-source. Alchemyst-Crucible helps stitch the implementation together. The scientific logic and the decision to stand behind a result remain human-owned.

Evidence files

• Live production snapshot: ../../story2/2026-08-25_kvasir_production_recommendation.png
• Nearest-similarity plot: ../../story2/2026-08-25_kvasir_nearest_similarity.png
• Six candidate pairs: ../../story2/2026-08-25_kvasir_near_duplicate_pairs.csv
• Twenty-five recorded split outcomes:

../../story2/2026-08-25_kvasir_random_split_sensitivity.csv

• Live canvas with the two editing panels collapsed:

../../story2/2026-08-25_kvasir_pipeline_canvas_panels-collapsed.png

• Live dataset-node preview with twelve Kvasir image samples:

../../story2/2026-08-26_kvasir_dataset_node_image_samples.png

• Original Kvasir-SEG author manuscript (open-access preprint):

../../story2/Kvasir-SEG_A_Segmented_Polyp_Dataset_Jha_et_al_2019_preprint.pdf

Reference

Jha D, Smedsrud PH, Riegler MA, Halvorsen P, de Lange T, Johansen D, Johansen HD. *Kvasir-SEG: A Segmented Polyp Dataset.* MMM 2020. Dataset: https://datasets.simula.no/kvasir-seg/.