Ideal summarization models should generalize to novel summary-worthy content without remembering reference training summaries by rote. However, a single average performance score on the entire test set is inadequate in determining such model competencies. We propose a fine-grained evaluation protocol by partitioning a test set based on the lexical similarity of reference test summaries with training summaries. We observe up to a 5x (1.2x) difference in ROUGE-2 (entity recall) scores between the subsets with the lowest and highest similarity. Next, we show that such training repetitions also make a model vulnerable to rote learning, reproducing data artifacts such as factual errors, especially when reference test summaries are lexically close to training summaries. Consequently, we propose to limit lexical repetitions in training summaries during both supervised fine-tuning and likelihood calibration stages to improve the performance on novel test cases while retaining average performance. Our automatic and human evaluations on novel test subsets and recent news articles show that limiting lexical repetitions in training summaries can prevent rote learning and improve generalization.

A Poisson Hidden Markov Model is a mixture of two regression models: A Poisson regression model which is visible and a Markov model which is ‘hidden’.

The code below is a boilerplate of image classification models seen elsewhere and has been retooled specifically for this dataset. For the dataset, I have built a web scraper using Beautiful Soup to download the images of top 50 dog breeds as reported in American Kennel Club. In total, there were 5000 images, 100 images per breed, allowing us to maintain the same distrubtion of training and validation dataset between classes. I have chosen ResNet34 over ResNet101, ResNet50 and ResNet18 as the model architecture here because of its optimal performance metrics (speed and accuracy). To faciltate model generalization, default data augmentation was applied to the training dataset using a batch size of 8. I have used a batch size of 8 here because of an'out of memory' error when 32 or 64 were used in AWS SageMaker notebook instance.