A standard curve with R² = 0.97 looks reassuring on a report. In practice, that same curve can mis-quantify low-abundance samples by 30 % or more — and nothing in the R² number tells you it happened. For ELISA, where most clinically and ecologically interesting samples sit in the lower third of the dynamic range, accepting a curve on R² alone is one of the most common silent failure modes in the workflow.

This post explains what R² actually measures on a sigmoidal ELISA standard curve, why it systematically over-reports quality on heteroscedastic data, and how a three-tier acceptance gate — the same one Labreador enforces — turns a soft number into a defensible accept/warn/reject decision.

What R² actually measures (and doesn't)

R² is the fraction of variance in the response explained by the model. On a sigmoidal ELISA curve, almost all of that variance lives in the steep middle of the dose–response transition. A model can therefore nail the inflection region, drift on both asymptotes, and still report R² in the high 0.9s.

R² does not measure:

whether residuals are symmetric around the curve,
whether the low-end (where samples often are) is fit better than the upper plateau,
whether the chosen model is the right shape (4PL vs 5PL vs log-log),
whether the asymptotes are stable enough to define LLOQ/ULOQ.

In other words, R² confirms a model fits the data on average. It does not confirm the model is fit-for-purpose for sample interpolation.

Why ELISA data is heteroscedastic

ELISA responses scale with signal: replicate variance at OD 2.0 is typically an order of magnitude larger than at OD 0.1. Unweighted least squares treats both as equally informative, so the fit gets pulled toward the noisy upper plateau and quietly abandons the low end.

Two practical consequences follow:

Global R² is dominated by the high-signal points. A curve can post R² ≈ 0.98 while the three lowest standards are off by 20–40 %.
Residuals become asymmetric. Standard parametric confidence intervals built on a symmetry assumption underestimate true uncertainty near LLOQ.

The fix at the fitting stage is variance-stabilising weighting (typically 1/Y or 1/Y²). The fix at the acceptance stage is to stop trusting a single R² value.

The 3-tier acceptance gate

Labreador grades every standard curve into one of three tiers and ties downstream behaviour to that tier:

Tier	Weighted R²	Decision	What the app does
1 — Accept	≥ 0.98	Curve is fit-for-purpose	Interpolate freely within LLOQ–ULOQ
2 — Warn	0.95 ≤ R² < 0.98	Borderline	Interpolate, flag wells, surface a smart suggestion
3 — Reject	< 0.95	Curve is not interpretable	Nonlinear interpolation is blocked

The thresholds are not arbitrary. 0.98 is the value above which, on typical heteroscedastic immunoassay data, the dominant residual pattern is random noise rather than systematic curvature. Between 0.95 and 0.98 you are usually looking at one or two questionable standards — recoverable, but the user needs to know. Below 0.95 the model is misaligned with the data shape; no amount of confidence-interval reporting will rescue a sample value interpolated from it.

Tier 3 is a hard gate, not a warning. Labreador will not return interpolated sample concentrations from a curve below 0.95 — the user is asked to re-fit with a different model, exclude an outlier standard, or re-run the plate.

Why 0.95 is the fail-safe (and not 0.90)

In heteroscedastic, asymmetric-residual regimes, a globally low R² is not a small fit problem — it is a signal that the chosen model shape is wrong for the data. Once that happens, nonlinear interpolation produces concentrations with confidence intervals that look reasonable on paper and are wrong in reality. The 0.95 cutoff is set deliberately conservatively because the cost of a false-accept (a published number derived from a misaligned curve) is much higher than the cost of a false-reject (a re-run plate).

What R² doesn't catch — and what Labreador adds on top

The quality gate is only one layer. Around it, the engine enforces several checks R² is blind to:

F-test for 4PL → 5PL. The choice between 4PL and 5PL is not made on R². Labreador runs an F-test on the residual sum of squares and only accepts 5PL when the extra asymmetry parameter is statistically justified. This prevents the overfitting that an R²-only comparison would always favour.
Weighted residuals (1/Y, 1/Y²). Variance-stabilising weights are applied during fitting so the low end is not sacrificed to the upper plateau.
Replicate CV gating. Replicates with CV > 20 % are flagged before they ever influence the curve.
LLOQ / ULOQ enforcement. Sample values outside the quantifiable range are not silently extrapolated; when extrapolation is permitted (polynomial / log-log only), values are explicitly tagged < LLOQ (ext) or > ULOQ (ext).
Confidence intervals via the covariance matrix. EC₅₀ uncertainty is propagated through the delta method in log space, not approximated from R².

A practical checklist

Regardless of which software you use, the following five checks will catch most of what R² alone misses:

Inspect the residuals plot, not just R². Look for systematic curvature, especially at the low end.
Compare 4PL and 5PL with an F-test, not by picking the higher R².
Apply 1/Y or 1/Y² weighting for any immunoassay with > 1 decade of dynamic range.
Set a hard R² floor for sample interpolation — 0.95 is a defensible default.
Report LLOQ and ULOQ explicitly, and refuse to interpolate outside them (or flag clearly when you do).

Cite Labreador

If the 3-tier gate or any of the QC logic above supported your analysis, please cite Labreador in your publications — the citation is available from the Cite Labreador button on the home page.

R² Is Not Enough: A 3-Tier Quality Gate for ELISA Curves