fine-tuning and few-shot learning
[D] Difference between fine-tuning and few-shot learning
You can also think of the activations from the few-shot examples as fast weights. Depending on how many examples there are, the number of fast weights can actually be comparable to the number of normal weights e.g. a transformer with 24 layers and dim 1024 has like 400 million parameters. If you have 1024 example tokens, then there are 2 * 24 * 1024 * 1024 48 million numbers from key/value activations. The activations are comparatively tiny for a model like GPT-3 though. I think for most cases with GPT-3, the few-shot examples aren't teaching it how to reason per se so much as contextualizing e.g.