Webmasters use this device to block bots from all web pages on a site, directing them only to a few limited pages or giving no access at all. Sometimes our bots get blocked from pages Google bots can access. This makes our link pointing less accurate, less detailed and weakens our crawl prioritization due to limited next jumps we can go to. The younger bots, like DotBot are able to operate more similarly to Google, while bots with a long history like Magestic seem to be blocked most. Bots are often singly excluded, so tweaking may be required to access certain websites by using certain bots.
Read more: Backlink Blindspots: The State of Robots.txt