Loss Functions are one of the most important parts of Neural Network design. A loss function helps us to interact with the model and tell the model what we want -- the reason why it is related to an "objective function". Let us look at the precise definition of a loss function. In mathematical optimization and decision theory, a loss function or cost function is a function that maps an event or values of one or more variables onto a real number intuitively representing some "cost" associated with the event. An optimization problem seeks to minimize a loss function.

I am currently struggling with a NLP task, for which I am unable to find any real relevant literature or other source of information. A chat consists of N messages where each message is sent by either person1 or a person2. In this chat questions from person1 are answered by person2. My goal is to label the individual messages as either a question, answer, or neither. Additionally, I am also trying to find which messages that are labelled as question constitute 1 question and messages that are labelled as answer constitute 1 answer and which questions and answers belong together.

We keep the BERT encoder unfrozen so that all weights are updated with every iteration. Given the number of trainable parameters it's useful to train the model on multiple GPUs in parallel. I used 4 Tesla K80's for about 4500 training samples. Just remember that to access any model attribute, you can access it using modelName.module.attribute I used Stochastic Gradient Descent with momentum as the optimizer and found that cycling both the learning rates and momentum really helped to get the training and validation losses down.