You do have a point here, one that I have heard from very reputable astro-physicists as well (e.g. Prof. Harald Lesch of "Centauri" fame).Given a billion year head start, then an intelligence at our level would have spread quite readily through the galaxy. We would see signs of it. We do not. Therefore, it doesn't exist.
His point and it is indeed a point worth discussing is: Where are they?
It would be very unlikely that all other intelligent species in the Galaxy are on the same developmental level as well are.
So if intelligent life was common, then there should alos be life forms that are millions of years ahead of us in there development. If these existed, they would most likely have conquerd the stars by now. So where are they?
I honestly do not have a satisfying answer to this question.
The evidence is very contradictive. On one hand, we find many systems that could have habitable planets and etrapolating from that there could be hundreds of millions of planets in the goldy locks zone in the miky way. Yet there is no hard evidence that other intelligent life forms exist, only a statistical likelyhood.
I do very much hope that we will find habitable planets one day and in that case, I also hope that at least a few of them are not inhabited by intelligent life. So we can seize them and collonize them.