I would like to calculate a squared Cohen’s kappa to compare observations between 2 raters for a variable with three ordered levels: Shape1, Intermediate, Shape2.

Here is my working example:

> # Create the dataframe
> InterRaterExample <- data.frame(
+   Rater1 = c("Shape1", "Shape1", "Shape1", "Shape1", "Intermediate", 
+              "Intermediate", "Intermediate", "Intermediate", "Shape2", "Shape2"),
+   Rater2 = c("Shape1", "Intermediate", "Shape1","Shape1", "Intermediate", 
+              "Shape1", "Shape2", "Intermediate", "Shape2", "Intermediate")
+ )
> # Define the ordered levels
> ordered_levels <- c("Shape1", "Intermediate", "Shape2")
> # Convert columns to ordered factors
> InterRaterExample$Rater1 <- factor(InterRaterExample$Rater1, ordered = TRUE, levels = ordered_levels)
> InterRaterExample$Rater2 <- factor(InterRaterExample$Rater2, ordered = TRUE, levels = ordered_levels)
> kappa2(InterRaterExample[, c("Rater1", "Rater2")], weight="squared")
 Cohen's Kappa for 2 Raters (Weights: squared)

 Subjects = 10 
   Raters = 2 
    Kappa = 0.107 

        z = 0.339 
  p-value = 0.735 
> # Convert ordered factor values to numeric
> InterRaterExample$Rater1_numeric <- as.numeric(InterRaterExample$Rater1)
> InterRaterExample$Rater2_numeric <- as.numeric(InterRaterExample$Rater2)
> kappa2(InterRaterExample[, c("Rater1_numeric", "Rater2_numeric")], weight="squared")
 Cohen's Kappa for 2 Raters (Weights: squared)

 Subjects = 10 
   Raters = 2 
    Kappa = 0.643 

        z = 2.03 

I’m trying to figure out why I’m getting two different kappa results, when the underlying data is the same.