What it takes to check a paper, when the authors have already done the hard part

A 2024 pharmacology paper from Tromsø, rebuilt on AlkhemYst in an afternoon.

---

The cells

Liver sinusoidal endothelial cells line the smallest blood vessels in the liver. They are strange cells. Their surfaces are perforated with open holes called fenestrae, they are among the most efficient scavengers in the body, and they begin losing their character within hours of being taken out of a liver. That last property is what makes them hard to work with: whatever you measure, you are partly measuring the cell's reaction to no longer being in a liver.

Which makes the control question sharp. If you treat these cells with a drug and something changes, how much of the change was the drug?

The paper

In 2024 the Vascular Biology Research Group at UiT The Arctic University of Norway asked what dexamethasone does to them.

Dexamethasone is a glucocorticoid receptor agonist and one of the most prescribed drugs in medicine; it is also routinely dropped into cell culture media, which is part of why the question matters. The group measured the entire detectable proteome at one hour, ten hours and forty-eight hours, with the drug and without it, three biological replicates, tandem mass tags read on an Orbitrap Fusion Lumos. Six thousand and twenty-eight proteins.

The paper is Bhandari and colleagues, Frontiers in Pharmacology 15:1377136. It passed peer review at a pharmacology journal and it is open access.

What they did next

They deposited the study on DataverseNO under CC0, which puts it in the public domain outright.

The amount matters, so it is worth being exact about what is in that deposit. Not a results spreadsheet. The tandem mass tag reporter intensities for every protein in each of the fifteen samples; a README running to fourteen thousand words covering the liver perfusion, the CD146 bead purification, the labelling chemistry, the high-pH fractionation and the MaxQuant search parameters; and their own edgeR output for all thirteen comparisons they report, carrying fold change, p value and false discovery rate for every protein.

They published the answers alongside the questions.

Most deposited data is not like this. The common shape is a supplementary spreadsheet of results with the inputs left behind, which lets a reader read the conclusion a second time without being able to test it. Nobody is doing anything wrong there; it is the default, and the default does not support checking. This group deposited the inputs, the design, the method and their answer.

What we did

We rebuilt their analysis on AlkhemYst. Seven steps on a canvas, no code written, nothing custom: every step is a node from the standard library.

Those seven steps became 445 lines of R. Nobody typed them and nobody has to read them, but they are kept with the run and they are what actually executed, so anyone who wants to check the reproduction against the paper can read the code rather than take our word for the canvas.

Their intensities and their sample table go into edgeR with the normalisation and the statistical test their methods section names, trimmed mean of M values and the exact test. That choice was deliberate and slightly against instinct. We were not trying to analyse their data well; we were trying to run their analysis, which is a different job, and the temptation to reach for something more modern is exactly the temptation that would have made the comparison meaningless.

Out of edgeR comes a volcano plot, a heatmap of the most strongly changed proteins, and an over-representation analysis against GO biological process.

Then the step that turns a demonstration into a check. Their published fold changes load from the same deposit, join to ours on the gene symbol, and go into a correlation. One number, which says whether the rebuild agrees.

The canvas runs in R. That was forced rather than chosen: edgeR is an R node and the platform has no Python equivalent. It turned out to be the right constraint, since R with edgeR is the instrument the authors used, so this is their analysis run a second time rather than a near neighbour of it.

The result

The rebuild agrees.

Across the 5,792 proteins that cleared edgeR's expression filter and joined to their published table, our log2 fold changes and theirs correlate at r = 0.9999. Plotted against each other they are not a cloud around a trend. They are a line.

The volcano puts the proteins a pharmacologist would expect a glucocorticoid to move at the top of the plot: Fkbp5, the canonical glucocorticoid-responsive gene, alongside Ndrg1, Shroom2 and Bcam.

We did not find anything. That is the point. A successful reproduction produces no news, which is why so few of them get done and why they matter.

The part that was actually hard

Almost none of it was statistics.

The work was reading their README closely enough to set the design up correctly. Their column names carry the treatment and the timepoint inside the string, Dex_48h_TMT_Exp2 and so on, and the honest way to handle that is to write the design out as a table with one row per sample saying which sample is which condition. Parsing it at run time would have been faster. It would also have put the study design inside a regular expression, where nobody can inspect it, which is a strange thing to do in the middle of a reproduction.

There were two false starts worth admitting, because both are ordinary and both would have been invisible in a tidier account.

The first agreement check we ran, by hand, gave r = 0.77, and for about ten minutes that looked like a story about the paper. It was not; it was a story about us. The quick check had skipped normalisation altogether. Once the trimmed mean of M values went in where the authors had put it, the correlation moved to 0.99, and the platform's own run took it to 0.9999.

The second was the canvas. We first built it with three separate data steps, one per table, which is the obvious shape and the wrong one on an R pipeline: two of the three never reached the program at all. The run executed, reported every step as done, and quietly produced the wrong contrasts, because edgeR had no design table and invented its own. The fix was to carry all three tables in one collection. We found two platform faults doing it, both now closed: R runs were having their progress markers misread, so a finished run was recorded as zero steps executed; and figures were drawn in a fixed frame whatever they contained, so an enrichment plot with twenty long labels printed them through each other.

Neither false start is interesting in itself. They are here because the version of this article where everything worked first time would be a worse guide to what reproducing a paper actually costs.

Why this story

There is a version of this article that treats a successful reproduction as a surprise and uses it to say something about how much published work does not hold up. We are not writing that one. This paper was reviewed at a pharmacology journal, the review did its job, the result stands, and there is nothing here to correct.

The story is the deposit. Somebody in Tromsø spent a day, probably several, preparing data and results for people they will never meet, with no citation guaranteed. That day is the reason our afternoon was possible. Without it there is no check to run; not because anyone was careless, but because a result you cannot re-derive is one you have to take on trust.

If you have a dataset sitting behind a paper you have already published, that is the entire ask. Not a new methodology, not a change to how you work. A day, once.

And if the reason you have not is that preparing an analysis so a stranger can follow it is itself a week of work, that part is what we build.

---

Evidence

Paper. Bhandari S, Kyrrestad I, Simón-Santamaría J, Li R, Szafranska KJ, Dumitriu G, Sánchez Romano J, Smedsrød B, Sørensen KK (2024). Mouse liver sinusoidal endothelial cell responses to the glucocorticoid receptor agonist dexamethasone. Frontiers in Pharmacology 15:1377136. doi 10.3389/fphar.2024.1377136. Open access, CC BY.

Data. doi 10.18710/T77K01, DataverseNO, CC0 1.0. Deposited by Sabin Bhandari; contact Karen Kristine Sørensen, Vascular Biology Research Group, Department of Medical Biology, UiT The Arctic University of Norway. 6,028 proteins, 15 samples, README updated 24 January 2024.

Pipeline. pipe_pWbBoLKMgJDC, seven library nodes, language R. One data step carrying all three tables into edgeR (TMM, exact test, reference NT_48h), then Volcano Plot, Expression Heatmap and Enrichment Analysis; edgeR into Merge (inner, on gene, validated one-to-one) against the published table, then Correlation Test.

Run. run_AHnbsGWbFb4O, 7 of 7 steps, 445 lines of generated R, six figures. Every number and figure in this article comes from it.

Comparison reproduced. Dexamethasone 48 h against untreated 48 h, three biological replicates per arm.

Agreement. Pearson r = 0.9999, p = 0, across the 5,792 proteins that cleared filterByExpr at the canvas's minimum count of 10 and matched a gene symbol in the published table. 578 proteins move at FDR < 0.05.

Credit. The work reproduced here is the Vascular Biology Research Group's, at the Department of Medical Biology, UiT The Arctic University of Norway. The deposit is theirs, the analysis is theirs, and the result is theirs. This article describes a second run of it and claims nothing further.