Can Language Models Discover Scaling Laws?

Overview

We frame scaling law discovery as a benchmarked "scientific agent" task by curating thousands of experiments across diverse settings. We introduce SLDAgent, an evolution-based agent that jointly searches scaling-law functional forms and fits parameters. Across tasks, the discovered laws extrapolate more accurately than established human-derived laws, validating usefulness in both pretraining and finetuning scenarios.

Benchmark Tasks

Parallel Scaling Law

Predicts language-modeling loss as a function of model size N and inference parallelism P.

Vocabulary Scaling Law

Models unigram-normalized loss as a function of non-vocabulary model size N, vocabulary size V, and dataset size D.

SFT Scaling Law

Models finetuning loss based on the supervised fine-tuning dataset size D.

Domain Mixture Scaling Law

Models each domain's pre-training loss based on that domain's proportion in the training mixture.

MoE Scaling Law

Models loss in relation to network size N and number of experts E.

Data-Constrained Scaling Law

Models pre-training loss as a function of network size N, dataset size D, and number of unique tokens U.

LR & Batch Size Scaling Law

Models pre-training loss based on learning rate l, batch size b, dataset size D, and network size N.

U-shaped Scaling Law

Models performance (Brier score) as a non-monotonic function of compute measured in FLOPs C.

Citation

@article{lin2025sld,
  title={Can Language Models Discover Scaling Laws?},
  author={Lin, Haowei and Ye, Haotian and Feng, Wenzheng and Huang, Quzhe and Li, Yujun and Lim, Hubert and Li, Zhengrui and Wang, Xiangyu and Ma, Jianzhu and Liang, Yitao and Zou, James},
  journal={arXiv preprint arXiv:2507.21184},
  year={2025}
}