icr provides functions to compute and plot Krippendorff's inter-coder reliability coefficient α and bootstrapped uncertainty estimates. The bootstrap routines are set up to make use of parallel threads via OpenMP.
## ---- install icr ----
# Install the released version from CRAN:
install.packages("icr")
# Or install the development version from GitHub:
# install.packages(devtools)
devtools::install_github(staudtlex/icr)
Load the library and Krippendorff's example data:
## ---- load icr ----
library(icr)
## ---- data ----
data(codings)
codings
[,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8] [,9] [,10] [,11] [,12]
[1,] 1 2 3 3 2 1 4 1 2 NA NA NA
[2,] 1 2 3 3 2 2 4 1 2 5 NA NA
[3,] NA 3 3 3 2 3 4 2 2 5 1 3
[4,] 1 2 3 3 2 4 4 1 2 5 1 NA
Compute the reliability coefficient α for nominal-level data.
## ---- compute alpha ----
# use defaults
krippalpha(codings, metric = nominal)
Krippendorff's alpha
Alpha coders units level
0.743 4 12 nominal
Bootstrapped alpha
Alpha Std. Error 2.5 % 97.5 % Boot. technique Bootstraps
NA NA NA NA Krippendorff NA
NA NA NA NA nonparametric NA
P(alpha
alpha_min):
alpha_min krippendorff nonparametric
0.90 NA NA
0.80 NA NA
0.70 NA NA
0.67 NA NA
0.60 NA NA
0.50 NA NA
To check how uncertain α may be, or whether it actually differs from various minimal reliability thresholds, bootstrap α. For reproducibility, do not forget to set the seed (defaults to seed = c(12345, 12345, 12345, 12345, 12345, 12345)).
Given that bootstrapping may take quite some time for large amounts of reliability data, increase the number of cores across which krippalpha may distribute the computations (note that if your version does not support the use of multiple cores, krippalpha will reset cores to 1).
# bootstrap uncertainty
alpha <- krippalpha(codings, metric = nominal,
bootstrap = TRUE, bootnp = TRUE, cores = 2)
print(alpha)
Krippendorff's alpha
Alpha coders units level
0.743 4 12 nominal
Bootstrapped alpha
Alpha Std. Error 2.5 % 97.5 % Boot. technique Bootstraps
0.72 0.075 0.562 0.85 Krippendorff 20000
0.73 0.146 0.417 1.00 nonparametric 1000
P(alpha
alpha_min):
alpha_min krippendorff nonparametric
0.90 0.003 0.126
0.80 0.137 0.350
0.70 0.640 0.607
0.67 0.739 0.683
0.60 0.940 0.822
0.50 0.997 0.932
Compare the distributions of bootstrapped α. The distributions resulting from Krippendorff's algorithm and the non-parametric bootstrap (resampling the coding units) look quite different. For a quick look, just plot().
## ---- plot distribution of alpha ----
# generic plot function
plot(alpha)
Alternatively, use ggplot2 to plot the distributions of α:
# use ggplot
df <- plot(alpha, return_data = TRUE)
library(ggplot2)
ggplot() +
geom_line(data = df[df$ci_limit == FALSE, ],
aes(x, y, color = type)) +
geom_area(data = df[df$ci == TRUE, ],
aes(x, y, fill = type), alpha = 0.4) +
theme_minimal() +
theme(plot.title = element_text(hjust = 0.5)) +
theme(legend.position = bottom, legend.title = element_blank()) +
ggtitle(expression(paste(Bootstrapped , alpha))) +
xlab(value) + ylab(density) +
guides(fill = FALSE)
The vectors of bootstrapped α may also be accessed and plotted directly:
hist(alpha$bootstraps) # Krippendorff-bootstrap
hist(alpha$bootstrapsNP) # nonparametric bootstrap
