In some contexts, human learning greatly exceeds what the sparsity of the available data seems to allow, while in others, it can fall short, despite vast amounts of data. This apparent contradiction has led to separate explanations of humans being equipped either with background knowledge that enhances their learning or with suboptimal mechanisms that hinder it. Here, we reconcile these findings by recognising learners can be uncertain about two structural properties of environments: 1) is there only one generative model or are there multiple ones switching across time; 2) how stochastic are the generative models. We show that optimal learning under these conditions of uncertainty results in learning trade-offs: e.g., a prior for determinism fosters fast initial learning but renders learners susceptible to low asymptotic performance, when faced with high model-stochasticity. Our results reveal the existence of optimal-paths-to-not-learning and reconcile within a coherent framework, phenomena previously considered disparate.