Seeing the trade-offs: Evaluating visualizations for multi-criteria comparison of set alternatives-test

Comparisons are central in multi-criteria decision-making, where alternatives must be evaluated across multiple, often conflicting goals. This process becomes especially complex when alternatives are grouped into sets. Based on formative research with domain experts, we first characterize set comparisons in trade-off analysis. In two design workshops, we identify potential visualization solutions to support them. Finally, we conduct a user study with lay participants and collect expert feedback to evaluate the two most promising visualizations that encode trade-offs, alongside a baseline that conveys this information numerically. Our findings show that visual encodings are preferred. The decoupled visual encoding, which separates different trade-off metrics, is favored by the domain expert. For non-experts, this decoupled visualization leads to higher perceived mental load and lower confidence although it may support more varied decision strategies. In contrast, the coupled visualization that integrates these metrics, results in higher confidence but may promote strategies that min-max only the top priorities. Supplementary material are available at https://osf.io/xnbm2/overview?view_only=4b5b940dce0f4ced8b933d9eaceaf3f5.