What is Learning Rate Scheduling?

Success metrics include system uptime, model performance stability, deployment velocity, and operational cost efficiency.

Start with cosine annealing with warm restarts for most deep learning tasks. It gradually reduces the learning rate following a cosine curve, then resets, allowing the model to escape local optima. For fine-tuning pre-trained models, use linear warmup followed by linear or cosine decay, starting with a lower rate to avoid destroying pre-trained representations. Step decay, which reduces the rate by a factor at specific epochs, is the simplest option and works adequately for many problems. Avoid constant learning rates since they almost always underperform scheduled approaches.

Use the learning rate range test also known as the Smith method to find the optimal range. Gradually increase the learning rate from very small to very large over one epoch while recording loss. The best initial rate is typically one order of magnitude below where loss starts increasing rapidly. For Adam optimizer, start with 1e-3 and adjust. For SGD, start with 0.01-0.1. For fine-tuning, use 10-100x smaller rates than training from scratch. Run the range test once per model architecture; it transfers across datasets of similar scale.

When increasing batch size by factor k, scale the learning rate by approximately sqrt(k) or k depending on the optimizer and model. This linear scaling rule works well for SGD but requires warmup periods for large batch sizes. Adam is less sensitive to batch size changes but still benefits from rate adjustment. When using gradient accumulation to simulate larger batches, the effective learning rate already accounts for the larger batch so no additional adjustment is needed. Always validate that training dynamics remain stable after scaling.

Start with cosine annealing with warm restarts for most deep learning tasks. It gradually reduces the learning rate following a cosine curve, then resets, allowing the model to escape local optima. For fine-tuning pre-trained models, use linear warmup followed by linear or cosine decay, starting with a lower rate to avoid destroying pre-trained representations. Step decay, which reduces the rate by a factor at specific epochs, is the simplest option and works adequately for many problems. Avoid constant learning rates since they almost always underperform scheduled approaches.

Use the learning rate range test also known as the Smith method to find the optimal range. Gradually increase the learning rate from very small to very large over one epoch while recording loss. The best initial rate is typically one order of magnitude below where loss starts increasing rapidly. For Adam optimizer, start with 1e-3 and adjust. For SGD, start with 0.01-0.1. For fine-tuning, use 10-100x smaller rates than training from scratch. Run the range test once per model architecture; it transfers across datasets of similar scale.

When increasing batch size by factor k, scale the learning rate by approximately sqrt(k) or k depending on the optimizer and model. This linear scaling rule works well for SGD but requires warmup periods for large batch sizes. Adam is less sensitive to batch size changes but still benefits from rate adjustment. When using gradient accumulation to simulate larger batches, the effective learning rate already accounts for the larger batch so no additional adjustment is needed. Always validate that training dynamics remain stable after scaling.

Start with cosine annealing with warm restarts for most deep learning tasks. It gradually reduces the learning rate following a cosine curve, then resets, allowing the model to escape local optima. For fine-tuning pre-trained models, use linear warmup followed by linear or cosine decay, starting with a lower rate to avoid destroying pre-trained representations. Step decay, which reduces the rate by a factor at specific epochs, is the simplest option and works adequately for many problems. Avoid constant learning rates since they almost always underperform scheduled approaches.

Use the learning rate range test also known as the Smith method to find the optimal range. Gradually increase the learning rate from very small to very large over one epoch while recording loss. The best initial rate is typically one order of magnitude below where loss starts increasing rapidly. For Adam optimizer, start with 1e-3 and adjust. For SGD, start with 0.01-0.1. For fine-tuning, use 10-100x smaller rates than training from scratch. Run the range test once per model architecture; it transfers across datasets of similar scale.

When increasing batch size by factor k, scale the learning rate by approximately sqrt(k) or k depending on the optimizer and model. This linear scaling rule works well for SGD but requires warmup periods for large batch sizes. Adam is less sensitive to batch size changes but still benefits from rate adjustment. When using gradient accumulation to simulate larger batches, the effective learning rate already accounts for the larger batch so no additional adjustment is needed. Always validate that training dynamics remain stable after scaling.