Month: September 2017

You can use a special HTML  tag to tell robots not to index the content of a page, and/or not scan it for links to follow. This tag controls the behavior of the spider/robot.

<meta name=”robots” content=”selection”>

For more Detail on Robot

Map of A Website To Help Crawler : In simple terms, a Sitemap is an XML file that is full of your all webpage’s URLs. It’s like a list of every webpage in your website. This file should be easily discoverable in your site in order for search engine crawlers to trip over it.

A Sitemap is usually used to make the search engine crawlers follow all the links of your individual webpages so that it does not miss out anything on your webpage.

As the name implies, it is just a map of your site – i.e. on a single page you show the structure of your site, its sections, the links between them, etc. Sitemaps make navigating your site easier and having an updated sitemap on your site is good both for your users and for search engines. Sitemaps are an important way of communication with search engines.