That may be true, but the vast distances they would need to travel to get to earth would mean they would need to:200 billion trillion stars estimated in the universe - each with its own system. It would be unbelievable to think that the balance of things required to achieve life, is just limited to our solar system. We maybe the only life, but the odds of that are ridiculously high.
A) Be able to live for hundreds if not thousands of years just to make the journey to earth.
B) be able to withstand massive amounts of G-force if they can fly at abnormal speeds
C) have the ability to navigate through wormholes or portholes to get here quicker (which I think is a lot of marvel movie bull).