To avoid unnecessary suspense, here is the quick version of at least some of the major factors contributing to the (relatively) recent advancement of ecology (and other areas of science):
1. The proper application of the scientific method.So how have these factors shaped ecology? Lets have a look...
2. Technological advances and advances in other natural and physical sciences.
3. Various other factors (some helpful, some not) arising from our growing population. Foremost among this last category are some of the big questions regarding things like climate change, responsible (sustainable) use of natural resources, public health issues, and so on. On to the not-so-quick version!
Ecology is a relatively new science - born of numerous biological disciplines, only arriving as its own field in the late 1800s to early 1900s. It is broad and overarching in scope, and is rooted in many of the other sciences - after all, that bit about the environment in the given definition of ecology frequently requires ecologists to dabble in other areas of the physical and natural sciences in order to answer ecological questions. Because of this, progress in other scientific fields affect progress in ecology (e.g. imagine doing ecological research without chemistry or genetics!).
So what about that "using the scientific method" bit? Just to give things some context (and yes, this is a bit of a tangent), consider the question "How long have people been using critical thought and (even crude implementations of) the scientific method as a way to understand the physical and natural world?" If you need a refresher on the history of life and the geologic timescale, you might check out my previous post on the subject, or the geologic time scale page on Wikipedia.
The punchline here is that modern humans are thought to have appeared around 200,000+ years ago with the first known attempts to try and learn about the world through reason and careful observation of natural phenomena occurring a little over 3,000 years ago. So as far as human existence goes, we're pretty new to the game of doing science!
Why this little diversion back to the pleistocene and the formation of the earth? First, because a lot went on before we humans showed up to the party, and that immense history has shaped the world we live in. It took us a while to get even a crude understanding of the big picture (e.g. we recently thought the earth was flat!) and the more that history of life on earth is pieced together (thanks to the efforts of scientists in areas like physics, chemistry, geology, biology, and paleontology), the better we understand today's world and how it works.
Secondly, the human population size and ability to create and share knowledge has changed dramatically in recent centuries, and this has had a resounding impact on the world of science. As you may know, the human population has experienced near exponential population growth over the past few thousand years. It has more than doubled in the past 50 years, and has increased more than 20 fold in the last 1000 years. The increase has raised new problems and questions to address (although, human population growth isn't just a 20th century concern), and it has also lead to increased means of communication, transportation, the accumulation and availability of knowledge (e.g. the internet), and of course the simple increase in worker-hours available for doing scientific research. In short - demographic changes have had (and will likely continue to have) a bit impact on the progression and direction of scientific advancement.
With that, let's finish with a more focused look at ecology (and its ancestors like natural history, biogeography, botany, zoology, etc.), by comparing it to what are commonly considered "hard sciences" like physics and chemistry.
Reaching to my nearest chemistry text (Physical Chemistry, by P. Atkins) and opening it to page 1, the book begins with an introductory chapter laying out what physical chemistry is: "the branch of chemistry that establishes and develops the principles of the subject. Its concepts are used to explain and interpret observations on the physical and chemical properties of matter." The first section of this chapter isn't about chemistry, or physics for that matter - but instead something more basic and fundamental to the topic of physical chemistry: the section is titled "The structure of science," and gives an overview of terms like law, hypothesis, and theory as applied to the subject at hand. This is how (again, in my opinion) every science text book should begin: lay out the foundations of using the scientific method for the subject at hand, then build up from there.
This is in large part what makes a "hard science" - emphasizing how to do good science, and properly applying it to understand natural phenomena. Admittedly, it also helps that atoms and molecules are more predictable in their behavior when it comes to chemistry and physics (versus the behavior of organisms), and in many ways easier to measure for purposes of data collection.
Field ecology, for example has its roots in natural history - which I'll (perhaps unfairly) use here as an example of a field that was slow to move from making observations to making testable hypotheses and conducting experiments to see which ideas about how things work would hold up to empirical evidence. In addition to the significant practical difficulties of studying living organisms, this relatively slow acceptance to use the scientific method to understand ecological phenomena seems attributable to: (1) the fact that life is amazingly diverse, and broad generalizations about that diversity are hard to make. Plus, for some those generalizations can spoil the beauty and mystique of nature, leading to less focus on general and easy to understand phenomena, and more focus on things that are unique, complex and harder to understand; and (2) insightful observations of natural (undisturbed) phenomena were long deemed valuable enough - which can elevate the process of observation above the importance of doing experiments (that is, testing hypotheses) - distracting from the development of general theories by the scientific method, while perhaps over-focusing on describing observed phenomena.
As technology and our understanding from other areas in science have progressed, so to have our observational capabilities (and thus what sorts of things we can test experimentally). In ecology and other areas of biology, these advancements have opened up entire new worlds to observe and questions to be answered.
Early on, physics and chemistry had had a few things going in their favor in this regard. First, they are by their very nature easier to think about and study in a quantitative and general sense (perhaps early physics more so than early chemistry). This imparts to them three important qualities as disciplines in science. First, theories apply broadly to natural phenomena (e.g. nearly all objects fall just the same when dropped from a moderate distance - contrast with understanding the basic aspects of respiration, which pose greater practical challenges to study and don't generalize as easily to broad categories of organisms). Second, the quantitative nature of important phenomena allows the use of powerful mathematical and statistical tools, leading to well defined predictions about measurable quantities and a greater ease in testing and ruling out bad hypotheses. Science is all about ruling out bad hypotheses, and doing this efficiently means efficient progress towards well supported theories. Finally, there are fewer ethical conflicts in studying things that aren't alive - people don't respond to seeing someone smash a rock, drop a marble, or melt down metals in the same way they respond to seeing someone dissect a live dog. All biologists know about animal rights but to a geologist, mineral rights are rarely a problem in the lab!
Finally, thanks to the suggestion by Nick Sly, I took a look at the 1964 Science article "Strong Inference: Certain systematic methods of scientific thinking may produce much more rapid progress than others". I highly recommend reading it over, as well as T.C. Chamberlin's 1890 paper "The Method of Multiple Working Hypotheses" which can be found here (from the Wikipedia entry on Chamberlin), plus the text here (ed. 1999) with typographical errors corrected, and subheadings added, and lastly this modern version re-written by L. Bruce Railsback in case you find "Chamberlin's paper is too long, too high-blown, and too sexist for modern students."
In Platt 1964, he describes how some areas of science fail to progress by leaving behind the method of testing multiple hypotheses and "strong inferences" - complete with a rather entertaining list of pseudoscientific approaches I'll leave you to consider:
I think, there are other areas of science today that are sick by comparison, because they have forgotten the necessity for alternative hypotheses and disproof. Each man has only one branch-or none-on the logical tree, and it twists at random without ever coming to the need for a crucial decision at any point. We can see from the external symptoms that there is something scientifically wrong. The Frozen Method. The Eternal Surveyor. The Never Finished. The Great Man With a Single Hypothesis. The Little Club of Dependents. The Vendetta. The All-Encompassing Theory Which Can Never Be Falsified.If you've made it this far, I hope you enjoyed the read and found it at least somewhat thought provoking. Feel free to share any comments or questions by posting below :)