Editing the robots.txt is an essential part of an onsite SEO strategy, particularly with Magento.
The robots.txt is a file that allows you to tell Google and other search engines which parts of the website they should not access. By default, all Magento stores have a robots.txt that can be found at: yourdomain.com/robots.txt
You’d probably expect to locate and edit the robots.txt under Marketing then SEO & Search, however, Magento being Magento this is not the case (hence this post)
How do you edit and configure the robots.txt in Magento 2?