A robots.txt file lets you limit the entry of online search engine robots that crawl the world wide web and it could protect against these robots from accessing specific directories and internet pages. Furthermore, it specifies wherever the XML sitemap file is located. If it does not redirect, you'll want to do an htaccess 301 redirect to verify