A amar495 New member Aug 12, 2019 #3 By default robots.txt does not block any bots from crawling your page.
R RH-Calvin New member Aug 12, 2019 #4 You can just list the URLs in allow section in robots.txt file to allow crawling of all webpages in search engines.
You can just list the URLs in allow section in robots.txt file to allow crawling of all webpages in search engines.
D davidweb09 Active member Aug 15, 2019 #5 You have to allow complete website for indexing in Robots.txt. https://phantomvapesupply.com/