solomonR

solomonR is an R package for analyzing, teaching, and studying the Solomon four-group design.

The package brings together three perspectives that have often been treated separately:

• the historical Solomon analysis workflow, preserved for teaching and methodological replication;
• modern model-based and randomization-based analyses for applied research; and
• SEM approaches, including latent-variable extensions when outcomes are measured with multiple indicators.

The goal is not to replace the history of the Solomon design with a single new procedure. Instead, solomonR makes the historical methods transparent while providing modern alternatives in one reproducible workflow.

solomonR is written for graduate students and applied researchers who need to analyze, interpret, and report a Solomon study. Each function’s help page cites the methodological sources it implements. New users should start with the article Getting Started: Analyzing a Solomon Four-Group Study.

Release status: v0.8.2 is the current stable release. It fixes a bug in perm_solomon() that made permutation p-values too small when permutations tie with the observed statistic, as they do with binary outcomes, counts, and ratings: rerun any such analysis. v0.8.1 fixed wrong contrasts in fit_solomon_sem() and fit_solomon_sem_latent() when the data did not begin with a pretested, treated participant. v0.8.0 brought together the work planned for v0.5.0 through v0.8.0:

• binary and count outcomes, designs with several treatments, and clustered designs with cluster-level randomization inference;
• nonrandomized designs, designs with several posttest occasions, and sensitivity analyses for missing posttests;
• APA-style reports, reanalysis from summary statistics, and analysis plans for new studies;
• Solomon’s (1949) original analysis, the published versions of the historical test sequence, teaching data, and published data sets with worked examples.

Former function and argument names keep working, with a warning, through v1.x, when the arguments that follow a former argument name are also named. Functions marked experimental may still change before version 1.0; each help page shows the function’s lifecycle stage. Development version 0.8.2.9000 is working toward v1.0.0.

---

The Solomon four-group design

The Solomon design (Solomon, 1949), Design 5 in Campbell and Stanley (1963/1966, p. 24), combines a randomized treatment comparison with an experimental manipulation of whether participants receive a pretest.

Group	Pretest	Treatment	Posttest
1	Yes	Yes	Yes
2	Yes	No	Yes
3	No	Yes	Yes
4	No	No	Yes

This allows researchers to ask not only:

Does the treatment work?

but also:

Does receiving the pretest change the treatment effect?

That second question is the pretest-by-treatment interaction, often described as pretest sensitization.

---

Installation

The latest stable release is available from R-universe:

install.packages(
  "solomonR",
  repos = c(
    JUhalt = "https://juhalt.r-universe.dev",
    CRAN = "https://cloud.r-project.org"
  )
)

The development version can be installed from GitHub:

install.packages("pak")
pak::pak("JUhalt/solomonR")

Then load the package:

library(solomonR)

solomonR is distributed through R-universe; CRAN submission is planned for version 1.0.0.

---

A 60-second analysis

The package includes a simulated example study whose true effects are known (see ?solomon_example):

data(solomon_example)

head(solomon_example)

The principal modern observed-variable analysis fits one unified model, the model Newman et al. (1990) proposed for the design, and estimates four Solomon-specific contrasts:

fit <- fit_solomon_glm(
  y_post, treat, pretested, y_pre,
  robust = "HC3",
  data = solomon_example
)

fit

The key estimands are:

• ATE (avg over pretest) — the treatment effect averaged equally across the pretested and unpretested conditions;
• Pretest x Treatment — whether the treatment effect differs depending on pretesting;
• Treatment | pretested — the treatment effect among participants who received the pretest;
• Treatment | unpretested — the treatment effect among participants who did not receive the pretest.

It also reports the pretest (testing) effect, pretested minus unpretested participants, among controls, among treated participants, and on average.

The model handles the structural absence of pretest scores in Groups 3 and 4 without discarding those groups.

To see all four contrasts and their confidence intervals at once:

plot_solomon_effects(fit)

and to see the Pretest x Treatment interaction itself, as model-adjusted group means:

plot_sensitization(fit)

Every contrast is reported with a confidence interval (conf.low, conf.high; set the level with conf_level). HC3 covariance is the default (Long & Ervin, 2000), and Gaussian models use t tests with residual degrees of freedom.

For Gaussian models, the output also reports a Wald-based partial R-squared for each one-degree-of-freedom contrast. With conventional covariance this is the usual partial R-squared, with a noncentral F confidence interval (Steiger, 2004). With robust covariance it is a descriptive Wald-based approximation without an interval.

---

Historical Solomon analysis

The historical analysis is retained because it is important to understand how the Solomon design developed and how it has traditionally been taught.

classic <- with(
  solomon_example,
  fit_solomon_classic(
    y_post,
    treat,
    pretested,
    y_pre
  )
)

classic

fit_solomon_classic() calculates the historical Tests A-I of Walton Braver and Braver (1988), whose Tests E and H follow Huck and Sandler (1973), and shows the decision path that would have been reached under the traditional conditional workflow.

The package distinguishes between:

• tests that were calculated, and
• tests that were actually reached along the historical decision path.

Test I, the Walton Braver & Braver (1988) Stouffer combination, is included for teaching and replication, but it is not the default modern inferential recommendation. Later simulation work raised concerns about Type I error in conditional versions of this procedure (Sawilowsky et al., 1994).

---

Randomization-based inference

When treatment was genuinely randomized within the Solomon pretesting conditions, perm_solomon() provides a treatment-label permutation test.

perm <- perm_solomon(
  fit,
  contrast = "ATE (avg over pretest)",
  reps = 5000,
  seed = 123,
  return_dist = TRUE
)

perm

Treatment labels are permuted within the pretested and unpretested strata, preserving the Solomon design.

The test uses an HC3-studentized statistic and a finite Monte Carlo correction for the permutation p-value.

The permutation distribution can also be visualized:

plot_perm(perm)

Randomization inference should reflect the design that actually generated the treatment assignments. For fits with a cluster variable, perm_solomon() permutes whole clusters. It uses covariate-adjusted cluster-level summaries, with exact enumeration when the design is small (Gail et al., 1996; Hayes & Moulton, 2017).

---

Full-information maximum likelihood

fit_solomon_ml() implements a full-information likelihood approach inspired by van Engelenburg’s (1999) treatment of the Solomon design.

Unlike analyses that simply discard structurally missing pretests, the likelihood recognizes that Groups 3 and 4 were never intended to have pretest observations.

ml <- with(
  solomon_example,
  fit_solomon_ml(
    y_post,
    treat,
    pretested,
    y_pre
  )
)

ml

The ML model estimates the same central Solomon quantities:

• average treatment effect;
• pretest-by-treatment interaction;
• treatment effect among pretested participants; and
• treatment effect among unpretested participants.

By default it uses t tests, with Welch-Satterthwaite degrees of freedom for the contrasts that combine the pretested and unpretested groups (Satterthwaite, 1946; Welch, 1947); in the package’s simulation validation these were calibrated at every group size studied. inference = "wald" gives van Engelenburg’s (1999) large-sample Wald inference, whose intervals were too narrow with fewer than 40 participants per group.

---

Structural equation models

solomonR also supports SEM formulations of the design through lavaan.

Observed-variable SEM

sem_fit <- with(
  solomon_example,
  fit_solomon_sem(
    y_post,
    treat,
    pretested
  )
)

sem_fit

The four-group observed mean-structure model is saturated. Consequently, global indices such as CFI and RMSEA are not diagnostic of model fit for that model.

An ANCOVA-style SEM can also be fit within the two pretested groups:

sem_ancova <- with(
  solomon_example,
  fit_solomon_sem(
    y_post,
    treat,
    pretested,
    y_pre = y_pre,
    ancova = TRUE
  )
)

sem_ancova

Latent outcomes

For multi-item outcomes, fit_solomon_sem_latent() estimates Solomon contrasts at the latent-variable level.

Latent mean comparisons require scalar measurement invariance across the four Solomon groups, or partial scalar invariance with the freed parameters chosen before the analysis (partial_post, partial_pre). solomonR imposes these constraints rather than silently interpreting latent means from configural or metric-only models, and checks them first with invariance_solomon(). For identification, the latent mean of the unpretested control group is fixed at 0; the Solomon contrasts do not depend on this choice.

See:

?fit_solomon_sem_latent

for details.

---

Check the design first

Before fitting a model, validate_solomon() confirms that all four cells are present, the design indicators are coded 0/1, and each cell has enough observed outcomes. It reports every problem at once instead of stopping at the first.

with(
  solomon_example,
  validate_solomon(y_post, treat, pretested, y_pre)
)

check_solomon_missing() separates the pretests that are absent by design in Groups 3 and 4, which must never be imputed, from incidental missing values, and explains the supported response to each with its sources.

Diagnostics

check_solomon_assumptions() provides diagnostics relevant to common Solomon analyses, including:

• Brown-Forsythe variance checks;
• cell-level normality summaries; and
• ANCOVA slope-homogeneity assessment.

These diagnostics are descriptive, not a sequence of gatekeeping tests: choosing an analysis because a preliminary test was significant can distort Type I error rates (Zimmerman, 2004). HC3 standard errors are a reasonable default for the unified GLM regardless of the results.

checks <- with(
  solomon_example,
  check_solomon_assumptions(
    y_post,
    treat,
    pretested,
    y_pre
  )
)

checks

---

Which analysis should I use?

A useful starting point is:

Goal	Suggested solomonR approach
Plan a study and pre-register its analysis	plan_solomon(), analysis_plan_solomon()
Modern primary analysis	fit_solomon_glm()
Randomization-based inference	perm_solomon()
Test whether sensitization is negligible	equivalence_solomon()
Compare analyses and their estimands	compare_solomon_methods()
Full-information likelihood	fit_solomon_ml()
Teach or reproduce historical methods	fit_solomon_classic(), fit_solomon_1949()
Teach with data whose effects are known	simulate_solomon()
Observed-variable SEM	fit_solomon_sem()
Multi-item / latent outcome	fit_solomon_sem_latent()
Check design coding and missingness	validate_solomon(), check_solomon_missing()
Sensitivity analysis for missing posttests	fit_solomon_mi(), tipping_point_solomon()
Several posttest occasions, with dropout	fit_solomon_mmrm()
Several treatments (six- and eight-group designs)	fit_solomon_glm(control = )
Model diagnostics	check_solomon_assumptions()

For many ordinary randomized Solomon experiments with continuous outcomes, the unified GLM with clearly defined contrasts is a useful primary analysis.

For the origin, assumptions, limitations, and sources of every method, see the article Solomon Methods: History, Recommendations, and Extensions. It labels each analysis as a historical procedure, a contemporary recommendation, a published Solomon proposal, or a solomonR extension.

The historical Tests A-I remain valuable for understanding the development of the design, but they should not automatically be treated as the preferred contemporary analysis.

---

Important statistical notes

A nonsignificant interaction is not evidence of no sensitization

Failure to reject the pretest-by-treatment interaction does not establish that sensitization is absent. To test whether sensitization is negligible, use an equivalence test with bounds set in advance at the smallest effect size of interest (Lakens, 2017):

# Bounds must be justified and fixed before examining the data.
equivalence_solomon(fit, bounds = 2)

The result reports both one-sided tests, the 90% confidence interval, and one of four outcomes: equivalent, different from zero but trivially small, different, or inconclusive.

Reference distributions and intervals

HC3 is the default covariance estimator, following Long and Ervin (2000) and Hayes and Cai (2007). Tests and confidence intervals use the t distribution with residual degrees of freedom for Gaussian models, the normal distribution for binomial and Poisson models, and Satterthwaite degrees of freedom for CR2. Maximum-likelihood and SEM results use large-sample normal intervals. Under robust covariance, the Wald R-squared is a descriptive quantity rather than an exact decomposition of model variance.

CR2

For clustered data, robust = "CR2" combines the bias-reduced cluster-robust covariance estimator with Satterthwaite degrees of freedom (Pustejovsky & Tipton, 2018). With few clusters, report the degrees of freedom alongside each test.

Latent means

Latent Solomon mean contrasts require scalar, or partial scalar, measurement invariance. fit_solomon_sem_latent() checks it with invariance_solomon() and warns, rather than refusing, when a criterion does not support it: no criterion studied held its false-rejection rate in Solomon-sized groups (issue #55).

Power

power_solomon() reports the rejection rate of each Solomon test with its Monte Carlo standard error, and a pre-specified simulation study validated it against normal-theory benchmarks. With 20 or fewer participants per cell its GLM-based figures are conservative, because the package’s default HC3 standard errors are conservative there. plan_solomon() finds the smallest design, at a chosen allocation across the four cells, that reaches a target power for each Solomon estimand. The sensitization contrast always has four times the sampling variance of the average treatment effect, so a study powered only for the average effect is usually underpowered for sensitization.

---

Package philosophy

solomonR is organized around three layers.

1. History

Preserve and reproduce the classical Solomon literature accurately.

2. Modern analysis

Provide unified regression, robust inference, randomization inference, and full-information likelihood methods.

3. Extensions

Develop SEM, latent-variable, longitudinal, generalized-outcome, design planning, visualization, and other modern Solomon methods while clearly distinguishing established methodology from package-specific extensions.

The aim is to make the Solomon four-group design easier to teach, understand, analyze, and extend without erasing the methodological history that produced it.

---

Roadmap

Development plans are maintained in ROADMAP.md. Each planned item is a GitHub issue on a milestone. The milestones v0.5.0 through v0.8.0 were released together as v0.8.0. Two remain:

• v0.9.0: the release candidate, with CRAN pre-submission checks;
• v1.0.0: the remaining sources named by Steyn (2009), the archived release, the companion manuscript, and the CRAN submission, after the maintainer’s review.

Version 1.0.0 is the release in which solomonR covers the published Solomon four-group methodology to date. Every source the package draws on is listed on the References page, and the article Coverage of the Published Methodology maps each work on the design to what the package does with it.

---

Citation

A formal package/methods manuscript is in development.

For the installed package citation, use:

citation("solomonR")

---

References

Campbell, D. T., & Stanley, J. C. (1966). Experimental and quasi-experimental designs for research. Rand McNally. (Original work published 1963)

Gail, M. H., Mark, S. D., Carroll, R. J., Green, S. B., & Pee, D. (1996). On design considerations and randomization-based inference for community intervention trials. Statistics in Medicine, 15(11), 1069–1092. https://doi.org/10.1002/(SICI)1097-0258(19960615)15:11%3C1069::AID-SIM220%3E3.0.CO;2-Q

Hayes, A. F., & Cai, L. (2007). Using heteroskedasticity-consistent standard error estimators in OLS regression: An introduction and software implementation. Behavior Research Methods, 39(4), 709–722. https://doi.org/10.3758/BF03192961

Hayes, R. J., & Moulton, L. H. (2017). Cluster randomised trials (2nd ed.). Chapman and Hall/CRC. https://doi.org/10.4324/9781315370286

Huck, S. W., & Sandler, H. M. (1973). A note on the Solomon 4-group design: Appropriate statistical analyses. The Journal of Experimental Education, 42(2), 54–55. https://doi.org/10.1080/00220973.1973.11011460

Lakens, D. (2017). Equivalence tests: A practical primer for t tests, correlations, and meta-analyses. Social Psychological and Personality Science, 8(4), 355–362. https://doi.org/10.1177/1948550617697177

Long, J. S., & Ervin, L. H. (2000). Using heteroscedasticity consistent standard errors in the linear regression model. The American Statistician, 54(3), 217–224. https://doi.org/10.1080/00031305.2000.10474549

Newman, I., Benz, C., & Williams, J. D. (1990). Alternatives in analyzing the Solomon four group design. Multiple Linear Regression Viewpoints, 17(2), 91–103. https://ojs.lib.ua.edu/glmj/article/view/125

Pustejovsky, J. E., & Tipton, E. (2018). Small-sample methods for cluster-robust variance estimation and hypothesis testing in fixed effects models. Journal of Business & Economic Statistics, 36(4), 672–683. https://doi.org/10.1080/07350015.2016.1247004

Satterthwaite, F. E. (1946). An approximate distribution of estimates of variance components. Biometrics Bulletin, 2(6), 110–114. https://doi.org/10.2307/3002019

Sawilowsky, S. S., Kelley, D. L., Blair, R. C., & Markman, B. S. (1994). Meta-analysis and the Solomon four-group design. The Journal of Experimental Education, 62(4), 361–376. https://doi.org/10.1080/00220973.1994.9944140

Solomon, R. L. (1949). An extension of control group design. Psychological Bulletin, 46(2), 137–150. https://doi.org/10.1037/h0062958

Steiger, J. H. (2004). Beyond the F test: Effect size confidence intervals and tests of close fit in the analysis of variance and contrast analysis. Psychological Methods, 9(2), 164–182. https://doi.org/10.1037/1082-989X.9.2.164

Steyn, R. (2009). Re-designing the Solomon four-group: Can we improve on this exemplary model? Design Principles and Practices: An International Journal—Annual Review, 3(1), 383–394. https://doi.org/10.18848/1833-1874/CGP/v03i01/37588

van Engelenburg, G. (1999). Statistical analysis for the Solomon four-group design (Research Report 99-06). University of Twente. ERIC. https://eric.ed.gov/?id=ED435692

Walton Braver, M. C., & Braver, S. L. (1988). Statistical treatment of the Solomon four-group design: A meta-analytic approach. Psychological Bulletin, 104(1), 150–154. https://doi.org/10.1037/0033-2909.104.1.150

Welch, B. L. (1947). The generalization of “Student’s” problem when several different population variances are involved. Biometrika, 34(1–2), 28–35. https://doi.org/10.1093/biomet/34.1-2.28

Zimmerman, D. W. (2004). A note on preliminary tests of equality of variances. British Journal of Mathematical and Statistical Psychology, 57(1), 173–181. https://doi.org/10.1348/000711004849222

---

License

solomonR v0.3.0 and later is licensed under the GNU General Public License, version 3 only (SPDX: GPL-3.0-only). See LICENSE.md for the full terms and inst/NOTICE for copyright and retained historical notices.

The published v0.2.0 release remains under its original MIT license. Releasing later versions under GPL-3.0-only does not relicense previously published versions.