Multimodal ML: Quantifying the Improvement of Calorie Estimation Through Image-Text Pairs

Various developed countries have experienced a continuous rise in obesity [1]. This can be burdensome on healthcare and has thus prompted governments to introduce laws and regulations to restaurants and food chains in an attempt to promote healthier eating [1]. One patent rule is the mandatory display of calories in menus, which has not come without an added cost to businesses. For example, Section 4205 of the Affordable Care Act (ACA) in the USA required certain food chains to display caloric information on menu items which was estimated to cost $315 million to comply [2]. These costs were primarily due to nutritional analysis - if one could drastically diminish this overhead, it would greatly lower business spending [2].