Using level_of_measurement='interval' with arbitrary values is extremely memory intensive and slow

[agrizzli]

I am running Krippendorff's alpha with level_of_measurement=interval for arbitrary data and without restricted domain values. This means, there are:

• gold standard values [-1.0, 0.0, 1.0]. See attached targets.csv file
• predicted values as real numbers within the mathematical interval -1.25 to 1.25. See attached preds.csv file

preds = np.loadtxt('preds.csv', delimiter=',')
targets = np.loadtxt('targets.csv', delimiter=',')

krippendorff.alpha(reliability_data=[preds, targets], level_of_measurement='interval')

When running computation, the process uses extremely high amounts of memory and is much slower compared to the computation with restricted domain values [-1.0, -0.5, 0.0, 0.5, 1.0]. The more items are included in the data, the slower and memory eager it becomes. I tested it on two systems for about 2,400 data points in total:

1. MacOS with python=3.9.6, numpy=1.26.4, and krippendorff=0.6.0

=> The computation needs more than 60 secs time and consumes 32GB of memory, but finishes.

2. Ubuntu Linux with python=3.10.12, numpy=1.26.4, and krippendorff=0.6.0.

=> The computation takes too long (I had to kill the process due to memory issues) and consumes some hundreds of GB within seconds. (Unfortunately, in one unsupervised case it even led to a complete depletion of RAM and OS killed important processes...)

The process appears to go crazy at the line 117

unnormalized_coincidences = value_counts[..., np.newaxis] * value_counts[:, np.newaxis, :] - diagonals

preds.csv
targets.csv

I am not sure whether this problem is related to this numpy bug report:
numpy/numpy#26395

[bryant1410]

Hey. This is a known limitation of this library. It scales with the number of different values (in compute and memory, didn't check the math but probably O(V^2) or O(V^3)). It's what makes it fast, as most use cases I have run into employ a relatively small domain.

Some ideas of what you can do:

• You can quantize the data, as you proposed. You can use many more interval values (e.g., .05 intervals), and I bet you'll get a good approximation. I guess you can get an edge with the latest version of everything (Python, NumPy, and this library).
• You can try using a different implementation. Maybe this one?
• You can send a PR with a fix. I guess a different implementation could be used if the domain values (provided or computed) exceed a certain size.