Jonathan Band (http://www.policybandwidth.com/summary.html) authoritatively illustrates below my previous post.
…
The Big Picture
Stepping back from the technicalities of the four fair use factors, it becomes clear that the Print Library Project is similar to the everyday activities of Internet search engines. A search engine firm sends out software “spiders” that crawl publicly accessible websites and copy vast quantities of data into the search engine’s database. As a practical matter, each of the major search engine companies copies a large (and increasing) percentage of the entire World Wide Web every few weeks to keep the database current and comprehensive. When a user issues a query, the search engine searches the websites stored in its database for relevant information. The response provided to the user typically contains links both to the original site as well as to the “cache” copy of the website stored in the search engine’s database. Significantly, the search engines conduct this vast amount of copying without the express permission of the website authors. Rather, the search engine firms believe that the fair use doctrine permits their activities. In other words, the billions of dollars of market capital represented by the search engine companies are based primarily on the fair use doctrine.
In addition to fair use, search engine firms rely on the concept of implied license. Search engine firms assume that if information is posted on a website, the website operator wanted the information to be found by users, and search engines are the most efficient means for users to find the information. Thus, search engine firms assume that most website operators want their sites copied into the search engine database so that users will be able to find the site. If an operator does not want his site crawled and copied, he can use an exclusion header, a software “Do Not Enter” sign, which most
search engine firms respect. But if a website operator does not use an exclusion header, a search engine will assume that the operator wants the site included in the search database.
…
here the whole document