Question

Some Web pages are "invisible."  That is, no search engine will return them in a query.  Why do these pages exist?

a)no  other  Web page links to it
b)it is synthetic
c)it was not written in HTML
d)all of the above

Answer 1

I think its C can anyone confirm ?

Answer 2

as I understand it, web crawlers for search engines index links to other pages.

you can still find =.txt, .php, and other misc files using search engines

Answer 3

E. None of the above. Well mannered spiders ( the programs that crawl the web indexing content ), will first request /robots.txt, which is a text file written by the webmaster. It has to follow a strict format, but the webmaster can decide what parts of his/her site should be indexed. Well mannered spiders will respect what robots.txt allows them to index.

I have a few sites that are business to business. I don't need the genral public knowing about these sites, so my robots.txt files ask that nothing be indexed.

Answer 4

If no other page links to your page, the spider won't have a starting point and your page will be invisible. This is why, when you create a brand new site, you usually have to submit it manually to search engines, this way they know there is a new starting point. The other way is to have other sites linking to your new site.
`robots.txt` is used to tell the spider which resources it should index and which ones it should not, but to use it the spider has to know there is something here.