Search engines like Google and Yahoo will constantly scour the Internet looking for new and updated content.To assist these efforts, the website should have a file known as a 'sitemap' which is kept as up-to-date as possible.

A current sitemap will help the search engines to locate the data on the website. Additionally, there are ways to limit the access of the searchbots to the website so that pages that should not be indexed are left alone.

The files which provide the sitemap are often search engine-specific, meaning that one file format will be used for Google while a different file file will be used for Yahoo or MSN. This is due to incompatibility in technologies and should change in the near future. The file which limits the access of the searchbots is common to all search engines and needs to be updated regularly as well.