In selecting a loss function for a regression model, which characteristic is most crucial for ensuring that outliers do not disproportionately influence the model's performance?

The most important feature is robustness, meaning the loss grows slowly or is capped for large errors so that a single outlier cannot dominate the sum. A loss with bounded influence, such as the Huber or quantile loss, keeps the contribution of a huge residual from blowing up the objective. In a typical linear regression, using squared error lets a single extreme point produce an enormous gradient, but a robust loss limits that effect. For example, if one data point has a residual of 1000, a squared‑error loss adds 1,000,000 to the objective, whereas a Huber loss adds only about 1,000, making the model less sensitive to that outlier. Thus, choosing a loss with bounded or sub‑quadratic growth is key to preventing outliers from skewing the model.