Robots.txt Doesn’t Support Regular Expression
by matt @ 12:13 pm. Blogged under Web Technology, Search Optimisation
Having recently discovered a client using supposed ‘regular expression’ (wildcard matching) in their robots.txt file to meet my SEO recommendations on duplicate file exclusion, I simply had to put something down in writing about this. It seems Google is advising webmasters via Google tools to use wildcard matching with asterisks to exclude files by URL variable e.g Disallow: *ref=68*. Whilst a very nice method for excluding files easily from Google (and MSN it seems) this is still not an accepted method under official robots.txt protocol and will still leave a webmaster with problems on other search engines. (more…)
Loading ...