.Google's John Mueller addressed a question concerning why Google indexes webpages that are actually refused from crawling by robots.txt as well as why the it's secure to ignore the similar Browse Console files concerning those crawls.Bot Web Traffic To Query Guideline URLs.The individual asking the inquiry recorded that bots were actually developing links to non-existent question parameter URLs (? q= xyz) to web pages with noindex meta tags that are actually likewise blocked out in robots.txt. What prompted the inquiry is actually that Google.com is actually crawling the web links to those webpages, obtaining shut out through robots.txt (without seeing a noindex robotics meta tag) after that receiving reported in Google.com Look Console as "Indexed, though blocked through robots.txt.".The individual asked the following question:." But listed here is actually the major question: why would certainly Google index webpages when they can not even see the web content? What's the advantage in that?".Google's John Mueller validated that if they can not crawl the page they can't view the noindex meta tag. He additionally helps make an appealing mention of the site: search driver, advising to neglect the outcomes because the "average" customers won't view those outcomes.He created:." Yes, you are actually appropriate: if we can not crawl the web page, our team can't see the noindex. That mentioned, if we can't crawl the web pages, at that point there is actually certainly not a whole lot for our team to index. Thus while you might find a number of those pages with a targeted site:- query, the normal individual will not find them, so I definitely would not fuss over it. Noindex is actually additionally great (without robots.txt disallow), it simply means the URLs will definitely wind up being crawled (as well as end up in the Look Console report for crawled/not indexed-- neither of these conditions create problems to the remainder of the internet site). The fundamental part is that you do not make all of them crawlable + indexable.".Takeaways:.1. Mueller's answer affirms the restrictions in operation the Internet site: search advanced search operator for analysis main reasons. One of those reasons is given that it is actually not connected to the normal hunt index, it's a distinct point entirely.Google.com's John Mueller commented on the web site hunt operator in 2021:." The short solution is that a website: question is certainly not indicated to be comprehensive, neither utilized for diagnostics purposes.A site query is a specific sort of hunt that restricts the results to a particular internet site. It's basically merely words internet site, a bowel, and after that the web site's domain name.This query restricts the outcomes to a certain site. It's not implied to become a comprehensive assortment of all the web pages from that website.".2. Noindex tag without making use of a robots.txt is actually fine for these kinds of scenarios where a crawler is actually connecting to non-existent webpages that are actually acquiring found out by Googlebot.3. Links with the noindex tag are going to produce a "crawled/not catalogued" item in Explore Console and that those won't have a negative impact on the rest of the site.Read the concern as well as answer on LinkedIn:.Why would Google mark web pages when they can not even observe the web content?Included Photo by Shutterstock/Krakenimages. com.