Does Not Find All the Pages
There are a few possible reasons why SiteSurfer Builder's robot would
would not find all the pages on a site:
- Pages are Excluded by Robots Directives. If
robots support is enabled in SiteSurfer Builder,
and either the web site has a robots.txt file or pages with
ROBOTS meta tags, then some pages of the site will either (1) not be indexed
or (2) not crawled to find links to other pages or both (1 and 2).
- Pages are Excluded by Name Patterns. If a page's address matches
a pattern in the Patterns options page, and that
pattern is enabled and set to Exclude, then that page will be
neither indexed nor crawled for links to other pages.
- Invalid or Missing Links. If the starting page
from which you indexed the site did not have links that extend to all
parts of the site--either directly or through child pages--or if some of the
links are malformed, SiteSurfer may not be able to find every page. If you
use the local mirror method of indexing a web site
off of your hard drive, then this problem will not affect finding pages, since
every file under the directory containing the root page--and every
subdirectory--will be located.