In the realm of image classification, annotations often describe a single category. However, images might contain multiple objects including spurious ones with respect to the annotation. In few-shot image classification, where data is scarce, the ambiguity of these labels can severely impact classification performance. This paper addresses this issue by localizing objects in test images before classification and providing a disambiguated image embedding. We first show that using ground truth localization information can significantly improve performance. Second, we propose a method that leverages unsupervised object segmentation to detect and segment objects in images, in a trainingfree manner. Through extensive experiments and evaluations, we illustrate the efficacy of our method, highlighting its capacity to improve state-of-the-art classifiers in few-shot classification.