Last Friday, Google announced that they started to use site speed as one of the 200 signals that influence the position of a website in the search results:
"As part of that effort, today we're including a new signal in our search ranking algorithms: site speed. Site speed reflects how quickly a website responds to web requests. [...]
We've decided to take site speed into account in our search rankings. We use a variety of sources to determine the speed of a site relative to other sites."