P priya.du Member Jan 8, 2024 #2 A robots.txt file tells the search engine crawlers which url should be crawl into the website. Robots tag helps to search engine crawlers about the particular webpage to crawl or not.
A robots.txt file tells the search engine crawlers which url should be crawl into the website. Robots tag helps to search engine crawlers about the particular webpage to crawl or not.
