Help:Controlling search engine indexing

From TechHelp
Jump to: navigation, search

Template:PD Help Page

There are a variety of ways in which MediaWiki allows users to attempt to control search engine indexing, commonly termed "noindexing", as the default position is that pages are indexed. Most of the methods rely on using the noindex HTML meta tag, which tells search engines not to index certain pages. Respecting the tag, or the robots.txt file, especially in terms of removing already indexed content, is up to the individual search engine, and in theory they may be ignored entirely.

The control methods are:

  1. Controlling an entire namespace, via MediaWiki software settings
  2. Controlling classes of pages, via MediaWiki:Robots.txt
  3. Controlling individual pages by adding the __NOINDEX__ magic word into them, either directly or using a template made to contain it.
  4. Controlling pages by adding the __NOINDEX__ magic word into standard templates used in certain situations.

Software settings and robots.txt

Software settings

Entire namespaces can be noindexed via $wgNamespaceRobotPolicies.

At the same time, __NOINDEX__ can be disabled in certain namespaces. This is controlled by the MediaWiki software setting $wgExemptFromUserRobotsControl, which defaults to $wgContentNamespaces.

$wgArticleRobotPolicies can be used to specify different policies for individual pages.

Robots.txt noindexing

Robots.txt can be used to noindex pages. Unlike the software settings, it can be edited via MediaWiki, at MediaWiki:Robots.txt.

NOINDEX magic word

Individual pages

Individual pages can be noindexed by adding the __NOINDEX__ magic word into that page. This can be done either directly or using a template designed to contain the magic word (eg Template:Tl); this can make use of the keyword easier to track.

Pages with the keyword are tracked by MediaWiki, and automatically categorised in a tracking category. The category name is determined by MediaWiki:Noindex-category.

Standard template noindexing

Standard templates can include the __NOINDEX__ keyword, thereby noindexing pages to which the templates are applied. A template used to flag a particularly sensitive type of page can then automatically noindex it.

INDEX magic word

Individual pages

Individual pages can override namespace noindexing by adding the __INDEX__ magic word into that page. Pages with the keyword are tracked by MediaWiki, and automatically categorised in a tracking category. The category name is determined by MediaWiki:Index-category. However, INDEX does not override noindexing via MediaWiki:Robots.txt. It does override $wgArticleRobotPolicies.

Current issues

  • Template:Bug – "Create an __NOINDEX__ equivalent to prevent indexing by internal search engine"


Template:Languages

Personal tools
Namespaces
Variants
Actions
Navigation
Technology Services
Toolbox
Wiki Help