In this post, “theory” refers to a set of explanations and principles that accounts or aims to account for empirical phenomena more parsimoniously than competing explanations. Here it should not be treated as a synonym for conjecture or guesswork.
While induction is often the basis of scientific investigation even today, and the aim of induction is still very often to arrive at knowledge of some factual state of affairs in the physical world, science does not always consist of using induction to try to arrive at the truth of a theory, law, or principle. As Karl Popper articulated, science seems to function most effectively when data or observations are gathered for the purpose of contradicting rather than for the purpose of bolstering current theory. To claim that all science is carried out on the basis of falsificationist principles is simplistic and even naïve, but falsifiability is certainly an ideal demarcation of genuine science from pseudoscience. If this demarcation was not made, and no equally rigorous demarcation was established, then any explanation that seemed to accurately account for an experience or set of experiences could arguably be regarded as scientific, no matter how great the possibility of error in that explanation. This may sound like an unlikely result, but many of the most popular pseudoscientific ideas out there today claim to be scientific because they implicitly reject falsificationism and because they do not rigorously attempt to account for alternative explanations.
Paradigm shifts (that is, shifts from one grand explanatory theory to another) occur when mounting data and observations require that the prevailing scientific consensus theory be modified, specifically when the old theory cannot adequately account for new data and observations. At times, modifications in the prevailing theory are minor and do not constitute fundamental theoretical shifts (as when genetics became part of the basic framework of the theory of evolution under the influence of Mendel and others). In these situations, the basic framework stays intact, but a significant new element is added.
In contrast, paradigm shifts are major changes in prevailing theory, such as occurred when Einstein’s theories of relativity and multiple physicists’ development of quantum mechanics provided a more complete picture of the physical universe than classical mechanics (though classical mechanics remains valid as a quite accurate approximation of the physical laws that prevail in circumstances other than those involving very strong gravitational fields, very fast speeds, and very small objects). With relativity and quantum mechanics, our fundamental understanding of the basic laws of the physical universe was altered. Of course, people had known about the existence of gravity and electromagnetic radiation prior to the development of relativity and quantum mechanics, but the new theories provided a much more complete understanding of these things. Quantum mechanics also eventually led to awareness of two other fundamental forces, the strong and weak nuclear forces.
Despite its rigor and the truly dazzling technological breakthroughs it has directly engendered in the modern era, the scientific method or set of methods is not the only means of arriving at apparently justified beliefs based on experience. This method or set of methods is certainly the most proven and reliable means for forming beliefs on the basis of experience, but that does not make it the only valid means.
Many experiences cannot by their very nature lead to proposed laws and principles that are publicly testable or testable in a controlled and systematic way. Virtually all wholly private experiences fall into this category. Such experiences are not suited for data collection/observations by multiple individuals operating under a common set of principles and assumptions, and thus cannot lead to any scientific theory. This does not, however, mean that such experiences do not reflect some aspect of the world that is consistent from person to person and epoch to epoch.
Even if it would be difficult to establish that such experiences do in fact reflect “the way things are,” this problem alone does not mean that these experiences categorically do not reflect “the way things are.” Similarly, it would be difficult to formulate general principles that can be tested in a controlled and systematic way on the basis of many everyday experiences, even those experiences that are not wholly private.