I have recently received few emails asking me if I use robots.txt file to stop search engine spiders from indexing pages in my forum. The answer is yes, I do use robots.txt file to ensure that search engine spiders do not spider pages which serve no purpose or might be different type of links to the same content i.e. duplicate content.

I am posting the robots.txt file that I currently use in my forum for your reference, in case any one has any questions regarding it; feel free to post a comment and I will try my best to answer it.

User-agent: *
Disallow: /forum/ajax.php
Disallow: /forum/attachment.php
Disallow: /forum/cron.php
Disallow: /forum/editpost.php
Disallow: /forum/global.php
Disallow: /forum/image.php
Disallow: /forum/inlinemod.php
Disallow: /forum/joinrequests.php
Disallow: /forum/login.php
Disallow: /forum/misc.php
Disallow: /forum/moderator.php
Disallow: /forum/newattachment.php
Disallow: /forum/newreply.php
Disallow: /forum/newthread.php
Disallow: /forum/online.php
Disallow: /forum/postings.php
Disallow: /forum/printthread.php
Disallow: /forum/private.php
Disallow: /forum/profile.php
Disallow: /forum/register.php
Disallow: /forum/report.php
Disallow: /forum/reputation.php
Disallow: /forum/search.php
Disallow: /forum/sendmessage.php
Disallow: /forum/showgroups.php
Disallow: /forum/subscription.php
Disallow: /forum/threadrate.php
Disallow: /forum/usercp.php
Disallow: /forum/usernote.php

*replace forum for folder where your forum resides.