An Aligned Rank Transform Procedure for Multifactor Contrast Tests

An Aligned Rank Transform Procedure for Multifactor Contrast Tests Lisa A. Elkin, Matthew Kay, James J Higgins, Jacob O. Wobbrock UIST'21: ACM Symposium on User Interface Software and Technology Session: Illustration, Games & Accessibility Abstract Data from multifactor HCI experiments often violates the assumptions of parametric tests (i.e., nonconforming data). The Aligned Rank Transform (ART) has become a popular nonparametric analysis in HCI that can find main and interaction effects in nonconforming data, but leads to incorrect results when used to conduct post hoc contrast tests. We created a new algorithm called ART-C for conducting contrast tests within the ART paradigm and validated it on 72,000 synthetic data sets. Our results indicate that ART-C does not inflate Type I error rates, unlike contrasts based on ART, and that ART-C has more statistical power than a t-test, Mann-Whitney U test, Wilcoxon signed-rank test, and ART. We also extended an open-source tool called ARTool with our ART-C algorithm for both Windows and R. Our validation had some limitations (e.g., only six distribution types, no mixed factorial designs, no random slopes), and data drawn from Cauchy distributions should not be analyzed with ART-C. DOI:: https://doi.org/10.1145/3472749.3474784 WEB:: https://uist.acm.org/uist2021/ Talk Recording of the UIST 2021 Papers Program