Beyond NDCG: behavioral testing of recommender systems with RecList


As with most Machine Learning systems, recommender systems are typically evaluated through performance metrics computed over held-out data points. However, real-world behavior is undoubtedly nuanced: ad hoc error analysis and deployment-specific tests must be employed to ensure the desired quality in actual deployments. In this paper, we propose RecList, a behavioral-based testing methodology. RecList organizes recommender systems by use case and introduces a general plug-and-play procedure to scale up behavioral testing. We demonstrate its capabilities by analyzing known algorithms and black-box commercial systems, and we release RecList as an open source, extensible package for the community.

In Proceedings of WWW
Federico Bianchi
Federico Bianchi
Postdoctoral Researcher at Stanford University

My research interests include meaning in natural language and programming languages.