Solve a mystery box like a data scientist

What happens when a data scientist gets a riddle in form of a box? Of course he will (try) approach it as a data problem. In this article I will describe the whole process, and to be honest, it was not as easy as I thought. As with many problems, you can get completely lost and only by talking to a couple of friends, I got back on track again. As a data scientist, I like to approach this problem in a data manner. I realize that this method is far from the most obvious solution. But it was a very fun endeavor. Collecting too much data, train a transformer model to extract values from a video, and eventually use a minimizer to find the solution. This article is a summary of this (mostly) fun journey! I have divided this article in a couple of (for me) logical steps. All images in this article have been taken or are generated by me unless stated otherwise in the separate captions (which is none in this article).