This is the crux of what's dubbed "Fermi's Paradox." There's an amusing response that intelligent species become addicted to computer games before they can propagate too much, and that only Luddite species can eventually colonize the galaxy.
While amusing, the version of the paradox that interests me more is that involving time travel. If we assume that time travel is possible, we should expect to see all sorts of time tourists all the time, and yet we haven't had a single (credible) report of one. Does this mean that time travel is impossible? And besides, how do we resolve chronological/continuity/causality paradoxes, like when you kill your grandfather or knock your younger self up? The possibilities are:
- Time travel is impossible. We haven't seen time tourists because there aren't any.
- Time travel is possible, but the timeline protects itself - that is, prevents paradoxes somehow. This solves the chronological paradox issue, but not Fermi's, and therefore is incomplete.
- Time travel is possible, and chronological paradoxes (or any time a choice is made) spawn parallel universes. (See Larry Niven's All the Myriad Ways short story - and if you have a link to the full text online for free, I'd love to see it!) This could possibly explain why we don't see time travelers because they get shunted into alternate universes. Not entirely sure why the one universe we observe happens to be the one without the time travelers, the chance of it happening randomly is vanishingly small, so therefore this hypothesis is not satisfying either.
- Time travel is possible, and paradoxes all play out in the same universe. This is my personal favorite. Each time someone does something, the changes ripple down through future history and things somehow work themselves out. That last bit's a bit fuzzy, but oh well. Eventually something will be changed in the past that prevents time travel from ever being invented. At this point, there can be no more time travelers to alter the timeline, and therefore things are static. So in short, if time travel can happen unrestricted, it won't. *glee!*