contentvalidR

Current stable release: 0.10.1 (on R-universe; CRAN has 0.4.0, see Installation).

Development plans: Roadmap · Future research · Open issues. Documentation: Package website · Changelog.

contentvalidR analyzes the content-validity evidence that judges and experts provide before a scale reaches respondents: whether each item represents the construct it was written for, whether a panel agrees on it, whether the conclusions depend on who sat on the panel, and whether the items cover the domain at all. Every analysis prints a header naming what it holds, then the facts of the design, then its verdict; it reports its numbers in APA style and explains each index it shows, so the output can be read without first consulting the source papers.

Quantitative content-validity statistics are one part of a broader validity argument. They complement, rather than replace, construct definition, domain coverage, qualitative expert feedback, cognitive interviewing, and other evidence about relevance, comprehensiveness, and comprehensibility.

Installation

Install from CRAN:

install.packages("contentvalidR")

CRAN is updated less often than GitHub, because it asks packages to update no more than every one to two months, so its version can trail the newest release. To install the newest stable release, use the JUhalt R-universe:

install.packages(
  "contentvalidR",
  repos = c(
    "https://juhalt.r-universe.dev",
    "https://cloud.r-project.org"
  )
)

Install the current development version directly from GitHub:

# install.packages("remotes")
remotes::install_github("JUhalt/contentvalidR")

A first analysis

Twenty judges sorted three items, each written for construct A, into the construct each one best represents:

library(contentvalidR)

sort_dat <- data.frame(
  item = rep(c("Clear 1", "Clear 2", "Needs review"), each = 20),
  rater = rep(1:20, 3),
  target_construct = "A",
  assigned_construct = c(
    rep("A", 18), rep("B", 2),
    rep("A", 16), rep("B", 4),
    rep("A", 12), rep("B", 8)
  )
)

fit <- sort_validity(sort_dat)
fit
#> <contentvalid_sort> Item-sort analysis
#> Items: 3 | Judges: 20 | Target constructs: 1
#> Test: Howard-Melloy exact target-count test (p0 = .50, alpha = .05)
#> Judges: naive, meaning drawn from the kind of people who will answer the
#> items.
#> 
#> 2 of 3 items meet the exact target-assignment criterion.
#> Flagged for review: Needs review
#> 
#> Item-level evidence
#>   Item         Target Decision Judges Psa     95% CI Csv Competitor      p
#>   Clear 1      A      Retain    18/20 .90 [.70, .97] .80 B          < .001
#>   Clear 2      A      Retain    16/20 .80 [.58, .92] .60 B            .006
#>   Needs review A      Review    12/20 .60 [.39, .78] .20 B            .252
#> 
#>   Judges: assignments to the target construct, out of the judges who sorted
#>   the item.
#>   95% intervals for proportions: Wilson score (the default). Newcombe (1998)
#>   compared seven methods and recommends score intervals over the Wald
#>   interval. An interval reflects how few ratings an item received, not whether
#>   the right people rated it.
#> 
#> Scale-level Colquitt benchmarks
#>   Target  Items  Mean Psa  Psa level  Mean Csv  Csv level
#>   A           3       .77  Moderate        .53  Moderate
#>   Benchmark set: Overall (not correlation-normed)
#>   These bands come from tasks with three definitions (one focal, two
#>   orbiting); judges here used 2 (set `n_constructs` if more were offered), so
#>   the comparison is approximate. This is a contentvalidR caution: Colquitt et
#>   al. do not discuss other numbers.
#> 
#>   Colquitt labels are empirical percentile norms derived from scale-level
#>   averages, not universal cutoffs or automatic scale-retention rules. They
#>   place a scale against published scales. Psa and Csv sit on different scales,
#>   so their values cannot be compared with each other; their labels can,
#>   because each is a percentile position among published scales.
#> 
#> What these columns mean
#>   Psa -- Proportion of Substantive Agreement. Share of judges who put the item
#>       in the construct it was written for (0 to 1; higher is stronger).
#>   95% CI -- Interval for Psa. Wider when fewer judges sorted the item; the
#>       method is named above.
#>   Csv -- Coefficient of Substantive Validity. How much more often judges chose
#>       the intended construct than its closest rival (-1 to 1; 0 is a tie).
#>   Competitor -- Strongest competing construct. The construct other than the
#>       intended one that judges chose most often.
#>   p -- Howard-Melloy exact test. Probability of at least this many target
#>       assignments if each judge picked the target at rate p0; compare with
#>       alpha.
#> 
#> What the decisions mean
#>   Retain -- met the exact target-assignment criterion.
#>   Review -- did not meet the exact target-assignment criterion; the competitor
#>       column shows where judges put it instead.
#> 
#>   Full definitions: contentvalid_glossary(). To hide this key:
#>   options(contentvalidR.show_key = FALSE).
#> 
#> A flag for review is not an automatic deletion decision. Use theory,
#> construct-domain coverage, item wording, and qualitative judge feedback
#> alongside these statistics.
#> 
#> See summary(x) for the flagged items and content_report(x) for an APA table.

The first line names the object, <contentvalid_sort>, and what it holds; then, after the facts of the design, the verdict: two items meet the exact test of Howard and Melloy (2016), and one is flagged for review. Review is not a decision to delete; it marks an item to look at again, together with what the judges wrote about it.

The scale-level bands place the mean Psa and Csv against published scales (Colquitt et al., 2019). Those norms come from sorts with three definitions, and these judges used two, so the printout says the comparison is approximate.

For a manuscript, content_report() gives the same evidence as an APA table, with a general note that defines its abbreviations and columns and states the criterion behind each decision:

content_report(fit)
#> <contentvalid_report> Results table in APA style
#> 
#>   Item         Target Judges Competitor Psa     95% CI Csv      p Decision
#>   Clear 1      A       18/20 B          .90 [.70, .97] .80 < .001 Retain
#>   Clear 2      A       16/20 B          .80 [.58, .92] .60   .006 Retain
#>   Needs review A       12/20 B          .60 [.39, .78] .20   .252 Review
#> 
#> Note. Psa = proportion of substantive agreement; CI = confidence interval;
#> Csv = coefficient of substantive validity. Judges = target assignments, out of
#> the judges who sorted the item. 95% CI = Wilson score confidence interval.
#> Retain = at least the number of target assignments the exact one-sided
#> binomial test needs at alpha = .05 with p0 = .50 (Howard & Melloy, 2016).
#> 
#> See content_report(fit, format = "markdown") for the table as Markdown, ready
#> for a manuscript.

format = "markdown" writes the same table for Quarto or R Markdown; print it from a chunk with the option results = "asis", so that it renders as a table.

plot() draws each item’s share of judges with its interval, against the share the exact test needs for that item:

plot(fit)

summary(fit) condenses the result to the items that need attention, and content_handoff(fit) carries the retained items into the analysis of response data. There is deliberately no helper returning “the items that passed”: filtering on status is a substantive decision that belongs in your own visible code.

Which workflow answers your question

Question put to judges or experts	Workflow	Built on	Guide
Which construct does each item belong to?	sort_validity()	Anderson & Gerbing (1991); Howard & Melloy (2016); Colquitt et al. (2019)	Item sorts
How well does each item match each construct definition?	rating_validity()	Hinkin & Tracey (1999); MacKenzie et al. (2011); Colquitt et al. (2019)	Construct ratings
Is each item relevant, essential, or matched to its objective?	expert_validity()	Aiken (1980); Lawshe (1975); Lynn (1986); Polit et al. (2007); Rovinelli & Hambleton (1977)	Expert panels
Has a Delphi panel reached consensus, and stopped changing?	delphi_validity()	Holey et al. (2007)	Delphi rounds
Do the conclusions depend on the particular judges?	judge_validity()	Crocker et al. (1988); Linacre (1989); Engelhard (1994)	Reading the output
Do the items cover the blueprint?	domain_validity()	A blueprint tally; Sireci & Geisinger (1992, 1995) for the optional structure analysis	Reading the output

The first four ask whether each item behaves as intended or, in a Delphi study, whether the panel has settled on it. The last two ask questions no item-level index can reach: whether your conclusions depend on the particular judges you recruited, and whether your item set covers the domain you set out to measure. An item can only be rated if someone wrote it, so a perfect relevance index says nothing about the facet you forgot.

How many judges or experts to recruit is answered from the design rather than by a rule of thumb: exactly by sort_power() and expert_power(), and as an estimate from a panel’s ratings by gtheory_content(); see Design and reporting. When the items move on to response data, content_handoff() carries them and their evidence; see the handoff guide and one item set through both stages. New users can start with Getting started.

Reading the output

Every print opens with a header naming the object, then the facts of the design and the verdict, and reports numbers as the APA Publication Manual (7th ed.) prescribes: no leading zero on values that cannot exceed 1, p to three decimals, and intervals as [LL, UL]. A short key under each result explains only the columns and decisions it shows. contentvalid_glossary() has the full definitions, and options(contentvalidR.show_key = FALSE) hides the key once the terms are familiar. See Reading the output.

What every workflow returns

All six workflows return an object with the same parts:

• results: the evidence at the workflow’s unit of analysis, one row per item for sort_validity(), rating_validity(), expert_validity(), and delphi_validity(), per judge for judge_validity(), and per blueprint cell for domain_validity();
• scale_summary: target-scale or panel-level evidence;
• settings: the analysis choices, which a report should state;
• design: sample size, missingness, and design metadata; and
• details: method-specific supporting results.

Every results table has a status column with the shared categories Supported, Review, Insufficient data, and Descriptive only, beside the method’s own recommendation word, such as Retain in an item sort.

What you can rely on

The six workflows, the object contract they share, and the handoff to empirical validation are stable: breaking changes wait for a major version and never arrive without a deprecation cycle. Component indices and planning helpers are supported to the same standard and may gain arguments. Auxiliary and compatibility helpers may be deprecated with one minor release of warning. A changed default is treated as a breaking change, because it can silently change published numbers. See ?contentvalidR for the full policy.

Examples, reporting, and citation

Five deterministic example data sets, covering the item-sort, construct-rating, relevance, essentiality, and congruence input shapes, are installed as CSV files, for example system.file("extdata", "sort_example.csv", package = "contentvalidR"). Four more files hold the one item set the walkthrough follows through every stage: its items, the sort, the expert ratings, and the responses. vignette("reporting-examples") gives methods and results scaffolds and a minimum reproducibility statement.

Cite the package with citation("contentvalidR"). The reference list below is also installed in BibTeX form: system.file("REFERENCES.bib", package = "contentvalidR").

The handoff to nomologR

Content review decides which items go on to be tested with response data. content_handoff() packages that decision so the empirical stage can pick it up without retyping anything, and nomologR is the partner package that reads it. A handoff carries:

• the items carried forward, and the items held back, each with its status, its recommendation, and the rule that made the decision;
• the scales, where the design maps items to constructs;
• each item’s statistics, with their intervals and criteria, and the panel’s agreement where one was computed;
• two facts about the instrument that only you know: which items are reverse-worded (reverse_keyed, or character(0) if you checked and none is) and the response scale respondents will answer on (response_scale).

h <- content_handoff(fit, reverse_keyed = character(0), response_scale = c(1, 5))
h$items
#> [1] "Clear 1" "Clear 2"

Once responses are collected, pass the handoff itself to nomologR, not h$items, so the keying and the reasons for anything held back travel with the items:

library(nomologR)
nomo_screen(responses, items = h)  # screens the carried items
nomo_run(responses, scales = h)    # runs the empirical stage on the handoff's scales

Reading a handoff. Take each item’s decision from carried and status; never re-derive it by comparing a statistic with its criterion, or by branching on the version that produced it. recommendation states the same decision in the workflow’s own words and, like rule, is prose. A keying of NA means nobody said, never that an item is forward-worded.

Versions. The handoff is schema version 1, frozen since 0.7.0: its fields and columns keep their names, positions, and types, and new optional ones may only be added at the end. A change that broke this would be schema version 2, produced beside version 1 for at least a release cycle. The reader in nomologR is tested against handoffs from contentvalidR 0.6.0 through 0.10.1, and neither package depends on the other. See ?content_handoff for the full contract, vignette("handoff-to-empirical-validation") for a guide, and vignette("one-item-set-both-stages") for the joint walkthrough.

References

Works cited in this README, the help pages, and the vignettes.

• Agresti, A., & Coull, B. A. (1998). Approximate is better than “exact” for interval estimation of binomial proportions. The American Statistician, 52(2), 119–126. https://doi.org/10.1080/00031305.1998.10480550
• Aiken, L. R. (1980). Content validity and reliability of single items or questionnaires. Educational and Psychological Measurement, 40(4), 955–959. https://doi.org/10.1177/001316448004000419
• Anderson, J. C., & Gerbing, D. W. (1991). Predicting the performance of measures in a confirmatory factor analysis with a pretest assessment of their substantive validities. Journal of Applied Psychology, 76(5), 732–740. https://doi.org/10.1037/0021-9010.76.5.732
• Ayre, C., & Scally, A. J. (2014). Critical values for Lawshe’s content validity ratio: Revisiting the original methods of calculation. Measurement and Evaluation in Counseling and Development, 47(1), 79–86. https://doi.org/10.1177/0748175613513808
• Brennan, R. L. (2001). Generalizability theory. Springer. https://doi.org/10.1007/978-1-4757-3456-0
• Chaffin, W. W., & Talley, W. K. (1980). Individual stability in Delphi studies. Technological Forecasting and Social Change, 16(1), 67–73. https://doi.org/10.1016/0040-1625(80)90074-8
• Clopper, C. J., & Pearson, E. S. (1934). The use of confidence or fiducial limits illustrated in the case of the binomial. Biometrika, 26(4), 404–413. https://doi.org/10.1093/biomet/26.4.404
• Cohen, J. (1968). Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. Psychological Bulletin, 70(4), 213–220. https://doi.org/10.1037/h0026256
• Colquitt, J. A., Sabey, T. B., Rodell, J. B., & Hill, E. T. (2019). Content validation guidelines: Evaluation criteria for definitional correspondence and definitional distinctiveness. (10), 1243–1265.

License

contentvalidR is licensed under the GNU General Public License, version 3 only (SPDX: GPL-3.0-only; R metadata: GPL-3). See the full license. Copyright (c) 2025-2026 Joshua Uhalt.

Releases from v0.2.0 onward carry the GPLv3-only license. The earlier v0.1.0 release remains available under its original MIT license, and historical attribution is preserved in inst/NOTICE.