Importing and Exporting additional start URLs |
![]() ![]() ![]() |
In spider mode, you can import additional spider URLs from a text file by clicking on the Import button and specifying its filename. The following is an example of a text file that can be imported by this feature. http://news.mysite.com/ http://www.myalternativesite.com/ http://www.mysite.com/spider_sitemap.html http://www.friends_site.com/
These URLs will be imported into the list and given the default option of “index and follow”. It can also import from a list of URLs with comma-separated values to specify the indexing option of that URL. The format for this advanced import list is as follows: <URL>, <URL option>, [<base URL> [, LIMIT=<limit>] [, WEIGHT=<weight>] ]
The following is an example list in the above specified format: http://news.mysite.com/, INDEX_AND_FOLLOW http://www.myalternativesite.com/, INDEX_AND_FOLLOW http://www.othersite.com/, INDEX_AND_FOLLOW, http://pages.othersite.com/ http://www.mysite.com/spider_sitemap.html, FOLLOW_ONLY http://www.myschool.edu/bob/index.html, INDEX_AND_FOLLOW, http://www.myschool.edu/, LIMIT=50, WEIGHT=-2 http://www.friends_site.com/, INDEX_ONLY
The URL options available correspond to the “Advanced Spider URL options” described in section 2.1.4 as follows:
See "Start spider URL" for more details on each of these options. The LIMIT= flag must be followed by a value specifying a restriction on the maximum number of files to index for a particular start point. This is the same setting as the one which you can specify on the "Advanced Spider URL options" window. The WEIGHT= flag must be followed by a value which specifies the boosting or indexing weighting for the pages under this start point. This should be a value between +5 and -5 similar to the PAGEBOOST meta value. It is the equivalent of specifying PAGEBOOST meta tags for each of the individual pages that are indexed under the start point in question. See "Weightings" for more information. You can also export your current list of URLs to a CSV text file by way of the Export button, which will export in the latter layout as given in the above example. |