Check noindex, nofollow, and meta robots directives on a page.
Checks a page's meta robots tag, X-Robots-Tag header, and canonical for anything that would keep it out of search results.