Are there any parameters imposed upon the interpretation of Torah, or is it a free-for-all?
Many find it strange that passages in the Torah that Chazal once took literally are being read by moderns in less and less literal ways. Yet, these very same moderns are unwilling to question the authority of the Rabbis when it comes to halachic matters. On the surface this appears to be hypocritical on their part - either the viewpoints of the Rabbis should be treated as sacrosanct, or we should be able to challenge them across the board!
Furthermore, the very basis of our acceptance of the Torah is our reliance upon tradition. Once we begin to undermine the infallibility of Chazal with respect to the explanation of the Torah's text, isn't the tradition as a whole placed on shaky foundations?
In order to address these difficulties, we must first distinguish between matters of halacha and non-normative matters. The field of halacha is an autonomous, rigorous discipline that has its own methodology and is grounded in unique, indigenous legal principles that were received at Sinai. Halachic "science" was intended to be a living tradition of research and analysis. Rabbis formulate theories, test hypotheses, and seek to develop the most elegant understanding of the structure of each mitsvah and its relationship to the halachic system as a whole.
The process of in-depth Torah study is, then, not unlike scientific investigation, in the sense that it demands the use of abstract reasoning combined with careful attention to the nuances of "empirical" data. Where it parts ways with the physical sciences is in the source of the material it works with. The data that scientists analyze is primarily derived from sense perception and observation. By contrast, the "data" of the halachic system are received through tradition. Aside from this important difference, though, the standards employed in halachic research are quite similar to those enshrined in fields of scientific inquiry. Just as the validity of a theoretical construct in the sciences is dependent upon its compatability with all of the empirical evidence available, so too an halachic formulation's validity is contingent upon its compatability with all of the facts provided by the Oral Torah.
The primary function of the Baale Hamesora (Masters of the Tradition), then, is to interpret and transmit the factual information of the Oral Torah faithfully. In the process, the Rabbis develop a comprehensive vision of how the various laws and principles interrelate harmoniously, and apply the law to new cases according to this paradigm.
Sometimes the theoretical perspectives of some Rabbis may diverge from that of their colleagues or from the positions held in generations past. On the surface, this appears problematic. Doesn't difference of opinion undermine the sanctity and reliability of tradition? The reality is, though, that the Rabbis never act as mavericks, casting aside precedent and injecting their own innovative ideas into the realm of halachic discourse. Like scientific researchers who are accountable to the empirical data before them, the Rabbis remain beholden to the parameters of the halachic system and must exercise their creativity within that framework. As long as they abide by this basic rule, their theoretical formulations, and halachic decisions, retain their legitimacy and holiness.
When it comes to non-halachic aspects of Torah, on the other hand, a Rabbi is entitled to exercise almost unbridled creativity. He need not feel constrained by the teachings of tradition, because there is no formal tradition with regard to the interpretation of such passages. Since the explanation of the narrative portions of Scripture is not formulated as a rigorous, internally consistent "science" like halacha, there is much more leeway to be found in this area of study. We see evidence of this fact in the Midrashic literature, wherein the divergence of Rabbinic opinion is far more pronounced than in the Talmud.
Outside of the realm of halachic analysis, then, a scholar is permitted - and even encouraged - to develop an independent understanding of the Torah that appeals to his or her intellect. This allows our appreciation of the philosophical wisdom and subtlety of the Torah to grow continuously, generation after generation. Rather than compromising the tradition, this built-in flexibility guarantees the Torah's eternal relevance to our lives.