Subjectivity in statistics
Andrew Gelman on subjectivity in statistics:
Bayesian methods are often characterized as "subjective" because the user must choose a prior distribution, that is, a mathematical expression of prior information. The prior distribution requires information and user input, that's for sure, but I don't see this as being any more "subjective" than other aspects of a statistical procedure, such as the choice of model for the data (for example, logistic regression) or the choice of which variables to include in a prediction, the choice of which coefficients should vary over time or across situations, the choice of statistical test, and so forth. Indeed, Bayesian methods can in many ways be more "objective" than conventional approaches in that Bayesian inference, with its smoothing and partial pooling, is well adapted to including diverse sources of information and thus can reduce the number of data coding or data exclusion choice points in an analysis.
People worry about prior distributions, not because they're subjective, but because they're explicitly subjective. There are many other subjective factors, common to Bayesian and Frequentist statistics, but these are implicitly subjective.
In practice, prior distributions often don't make much difference. For example, you might show that an optimistic prior and a pessimistic prior lead to the same conclusion.
If you have so little data that the choice of prior does make a substantial difference, being able to specify the prior is a benefit. Suppose you only have a little data but have to make a decision anyway. A frequentist might say there's too little data for a statistical analysis to be meaningful. So what do you do? Make a decision entirely subjectively! But with a Bayesian approach, you capture what is known outside of the data at hand in the form of a prior distribution, and update the prior with the little data you have. In this scenario, a Bayesian analysis is less subjective and more informed by the data than a frequentist approach.