for more than a week’s time has passed, there is still a lot of forbidden love Shanghai directory found in search results. See the November 21st diary, found that other search engines are no longer grab prohibited directory, while Shanghai still has a lot of love to grab the catalogue of prohibited.
November 13th the search spiders crawl to the robots.txt file
This is my
November 21st the search spiders of the robots.txt file.
some time ago the Internet industry in 360 do not follow the robots.txt file for the group, love Shanghai also stressed that he is to comply with the robots.txt agreement. But from the analysis of a website recently, love Shanghai to comply with the robots.txt file is not complete. In November 15th, I wrote an article on the search spiders robots.txt file change reaction, because around November 13th, I put the robots.txt file in my work on the site, explicitly prohibit the search engine grab some directory. The robots.txt file is placed after the search engines have to respond, including the love of Shanghai.
November 14th the search spiders crawl to the robots.txt file, love Shanghai grabbed 2 times robots.txt
love Shanghai grab robots.txt catalogue of prohibited
, in the robots.txt file directory to prohibit certain 7 days later, other search engines to the catalogue of prohibited capture, and love Shanghai of these directories crawl is still in progress. It is not just that Shanghai is not comply with the robots.txt agreement. The answer is not temporary. To see why Shanghai didn’t keep love. From the beginning of the 13 set of robots.txt files, November 13th, November 14th, November 15th log, love only in Shanghai No. 14 visited robots.txt file two times (here to declare, I this site is a collection of 30 thousand, update everyday snapshots of
obviously see, only love Shanghai in the catalogue of prohibited
November 15th the search spiders crawl to the robots.txt file
captureThe fact that