.Google's John Mueller responded to a concern concerning why Google.com indexes web pages that are actually prohibited coming from creeping by robots.txt and also why the it is actually safe to disregard the similar Search Console documents concerning those creeps.Crawler Visitor Traffic To Query Criterion URLs.The individual asking the inquiry recorded that bots were making web links to non-existent concern guideline URLs (? q= xyz) to webpages along with noindex meta tags that are actually also shut out in robots.txt. What cued the question is actually that Google is actually creeping the links to those web pages, getting blocked by robots.txt (without watching a noindex robots meta tag) after that getting turned up in Google Explore Console as "Indexed, though blocked out by robots.txt.".The individual talked to the adhering to question:." Yet right here is actually the significant inquiry: why would Google.com mark web pages when they can not also see the information? What is actually the perk during that?".Google.com's John Mueller affirmed that if they can't creep the web page they can't find the noindex meta tag. He also creates an intriguing reference of the website: hunt driver, urging to overlook the results since the "average" individuals won't view those outcomes.He created:." Yes, you're appropriate: if our experts can't creep the webpage, our team can't see the noindex. That pointed out, if our team can't creep the web pages, after that there's not a lot for our company to index. Therefore while you might observe a few of those web pages along with a targeted internet site:- inquiry, the ordinary user won't find them, so I wouldn't bother it. Noindex is actually also fine (without robots.txt disallow), it just implies the Links will end up being actually crawled (and find yourself in the Browse Console file for crawled/not recorded-- neither of these conditions induce issues to the remainder of the site). The integral part is that you do not create all of them crawlable + indexable.".Takeaways:.1. Mueller's solution validates the restrictions in operation the Site: search evolved search driver for diagnostic causes. One of those causes is because it is actually certainly not attached to the frequent search index, it is actually a different trait altogether.Google's John Mueller discussed the site hunt driver in 2021:." The short solution is that a site: inquiry is actually certainly not suggested to be comprehensive, nor utilized for diagnostics functions.A site query is a particular type of hunt that confines the outcomes to a specific internet site. It's basically merely words site, a digestive tract, and after that the web site's domain.This query confines the outcomes to a certain site. It's certainly not meant to be a detailed collection of all the webpages coming from that web site.".2. Noindex tag without using a robots.txt is fine for these type of conditions where a robot is connecting to non-existent webpages that are actually receiving found out through Googlebot.3. Links along with the noindex tag will generate a "crawled/not catalogued" item in Browse Console and that those won't have an unfavorable impact on the remainder of the web site.Check out the concern and also answer on LinkedIn:.Why will Google.com index pages when they can't also observe the material?Included Graphic by Shutterstock/Krakenimages. com.