Membership Inference Attacks as Privacy Tools: Reliability, Disparity and Ensemble
Wang, Zhiqi, Zhang, Chengyu, Chen, Yuetian, Baracaldo, Nathalie, Kadhe, Swanand, Yu, Lei
–arXiv.org Artificial Intelligence
Membership inference attacks (MIAs) pose a significant threat to the privacy of machine learning models and are widely used as tools for privacy assessment, auditing, and machine unlearning. While prior MIA research has primarily focused on performance metrics such as AUC, accuracy, and TPR@low FPR - either by developing new methods to enhance these metrics or using them to evaluate privacy solutions - we found that it overlooks the disparities among different attacks. These disparities, both between distinct attack methods and between multiple instantiations of the same method, have crucial implications for the reliability and completeness of MIAs as privacy evaluation tools. In this paper, we systematically investigate these disparities through a novel framework based on coverage and stability analysis. Extensive experiments reveal significant disparities in MIAs, their potential causes, and their broader implications for privacy evaluation. To address these challenges, we propose an ensemble framework with three distinct strategies to harness the strengths of state-of-the-art MIAs while accounting for their disparities. This framework not only enables the construction of more powerful attacks but also provides a more robust and comprehensive methodology for privacy evaluation.
arXiv.org Artificial Intelligence
Jul-4-2025
- Country:
- North America
- United States
- Texas (0.04)
- Utah > Salt Lake County
- Salt Lake City (0.04)
- Pennsylvania > Philadelphia County
- Philadelphia (0.04)
- New York
- New York County > New York City (0.04)
- Rensselaer County > Troy (0.04)
- Colorado > Denver County
- Denver (0.04)
- California
- Los Angeles County > Los Angeles (0.14)
- Santa Clara County > San Jose (0.04)
- Canada > Ontario
- Toronto (0.04)
- United States
- Asia > Taiwan
- Taiwan Province > Taipei (0.05)
- North America
- Genre:
- Research Report > New Finding (0.93)
- Industry:
- Information Technology > Security & Privacy (1.00)
- Technology:
- Information Technology
- Security & Privacy (1.00)
- Data Science > Data Mining (0.94)
- Artificial Intelligence > Machine Learning
- Performance Analysis > Accuracy (0.96)
- Neural Networks (0.67)
- Statistical Learning (0.67)
- Information Technology