Our models, called the Gemstones Scaling laws are typically fit using a family of because they are loosely based on scaled-down variants models with a narrow range of frozen hyperparameter of the Gemma architecture, vary in their parameter count, choices. In this work we study scaling width/depth ratio, training tokens, learning rates, and laws using a wide range of architecture and hyperparameter cooldown schedules. By fitting scaling laws to these choices, and highlight their impact on checkpoints, we confirm that scaling law parameters and resulting prescriptions. As a primary artifact of interpretations indeed depend strongly on the selection of our research, we release the Gemstones: the most models and fitting procedure used, and we quantify the comprehensive open-source scaling law dataset degree to which these decisions impact predictions. By to date, consisting of over 4000 checkpoints from exploiting the variation among our model checkpoints, we transformers with up to 2 billion parameters; these also fit a number of unique scaling laws and analyze their models have been trained with different learning predictions to discern whether they are consistent with rates, cooldown schedules, and architectural design choices we see in industry models.

If you've ever battled crowds to get a coveted toy you know that early October is the perfect time for parents to start holiday shopping. Amazon knows this, which is why the company is holding a second Prime Day sale event--which ends tonight--featuring some great Prime Day toy deals. You can find all the best Prime Day deals here. But if your kids are like my kids, they're already working on their wish lists. If you say you haven't already started budgeting, you are either lying, financially irresponsible, or your children are much less demanding than mine are (I know, it's my fault).