What is robots.txt file?
Robots.txt is a text file that must be place into the root directory of your website so that search engine spiders may easily find it. In a very short answer for what is robots.txt file?
Robots.txt files inform search engine spiders that how to interact with indexing your content.
robots.txt files for WordPress with Better SEO
Search engines are very rapacious for the content. They crawl the web with the certain crawl rate to find, and index the high quality content automatically. Suppose, you didn’t submit a particular page to the search engine but still your page is being indexed on the search results that causes automatically indexing your page. Why it is happening? There are so many factors for it, but the main reason is that your page link is placed somewhere on the web, which is already index in the search engine so when the search engine spiders crawl that page, they visit all the inbound & outbound pages too, which causes to index your page in the search engines.
If you wish to no-index any part of your blog, such as pages, posts, feeds, directory, sub-directory and anything you can make a rule for the search engine spiders via robots.txt file. For example,
Allow indexing of everything
User-agent: *
Disallow:
Disallow indexing of everything
User-agent: *
Disallow: /
I am using User-agent: * in the above example. User-agent means the search engine bot, and * means that these rules are for every search engine bots. Let we make a rule only for Google search result, then we will use User-agent: Googlebot that will be valid for Google bot.
Disallow Googlebot from indexing a folder, and allow entire website.
User-agent: Googlebot
Disallow: /wp-content/
The above example shows that Googlebot will index your entire WordPress blog, except the content on the folder wp-content and its sub-folder (i.e., /wp-content/uploads).
Disallow Googlebot from indexing a folder, except for allowing the indexing of a sub-folder in that folder.
User-agent: Googlebot
Disallow: /wp-content/
Allow: /wp-conent/uploads/
If you are familiar with WordPress, then you can easily go to your WordPress files and folders through your FTP account and look on the wp-content folder, which contains at least four sub-folders as plugins, themes, upgrade and uploads. It is not necessary to index the plugins, themes and upgrade folders into search engine results. The uploads folder has all the images of your WordPress blog posts so you may index it. The above example is perfect for this task for Google search engine. If you wish to apply this rule to all search engine spiders, then you can replace Googlebot with *.
Disallow indexing a folder, except allowing for indexing a file in that folder.
User-agent: *
Disallow: /wp-content/uploads/
Allow: /wp-conent/uploads/activate-wordpress-plugin.png
Advantage of robots.txt File in WordPress SEO Optimization
If you know something about SEO, then you would be familiar with Google panda & pengium update. Pengium targeted mostly the sites that have the duplicate contents. It may be for internal or external pages. Let’s explain that what is the duplicate content on internal pages.
Your blog has 25 posts in a month, and you don’t allow anyone to write on your blog so only you are the author of that 25 posts. The author archive and the monthly archive page has the same content which occurs duplicate content issue.
You may include the URLs to not be crawled by the bots via WordPress robots.txt files. Here is how to add robots.txt file to WordPress for better SEO.
robots.txt file Code to WordPress With SEO Optimization
We have generated a general robots.txt file with the consideration of SEO. You may copy all the code and paste to your robots.txt file. The code is,
Sitemap: http://www.yoursite.com/yourindex.xml Sitemap: http://www.yoursite.com/yourindex-video.xml User-agent: * Disallow: /cgi-bin/ Disallow: /wp-admin/ Disallow: /wp-content/ Disallow: /wp-includes/ Disallow: /go/ Disallow: /comments/feed/ Disallow: /feed/ Disallow: /trackback/ Disallow: /index.php Disallow: /xmlrpc.php Disallow: ?comments=* Disallow: /search? Disallow: /?p=* User-agent: Mediapartners-Google* Allow: / User-agent: Googlebot-Image Allow: /wp-content/uploads/ User-agent: Adsbot-Google Allow: / User-agent: Googlebot-Mobile Allow: /
Make sure to change yoursite, yourindex and yourindex-video with your own data. If you are using Go URL for your WordPress blog, then you may block these URLs by adding “Disallow: /go/” into your robots.txt file but if you are not using Go URL, then you can delete it from the above code.
Now, below is the method to edit robots.txt file, if you are using SEO by Yoast plugin.
Edit robots.txt file via SEO by Yoast Plugin
- Go to your WordPress dashboard admin panel.
- Click on SEO in the left menu, and then click on Edit files as shown in the image below,
- Once you clicked, you will see the robots.txt file editor like the below preview,paste your copied code here, and finally click on Save changes to Robots.txt.
Now, you may check your robots.txt file by going to http://www.yoursitename.com/robots.txt. Your robots.txt code will have updated. Below is another method to add robots.txt file via FileZilla FTP client.
How To Add robots.txt Files to WordPress by FileZilla FTP Client
Copy the above code and open the notepad in your desktop, and paste the above code inside your notepad, and rename this file to “robots” without quotes. Now you have to host your robots.txt file into the root directory of your WordPress blog.
For hosting your robots.txt file you need,
Once you open FileZilla, go to your root directory of your WordPress blog, and upload the robots.txt file from your desktop to your WordPress directory, Now, you may check your robots.txt file by going to http://www.yoursitename.com/robots.txt. Your robots.txt code will have updated.
[VIDEO] How to Add/Edit robots.txt files to WordPress
We have created a live video to add your SEO friendly robots.txt file to WordPress blog, and I hope it may help you.
Thank you So much I gonna try this file Robot.txt. It’s really interesting to aware search engine about your site’s content.
Yeah, it will remove the duplicate content issue from your blog.
HI
Robots.txt is an important part of SEO, you have explained it detail. very useful article
thanks for sharing
Bryan
When I click on the “Edit Files” button on Yoast SEO, it takes me to a page to edit my .htaccess file. How do I get to the robots.txt file?
Oops!!
Simply go to your root directory and create a text file with the name “robots.txt” (You may search on this blog for how to access your WordPress files via FTP clients), then continue to click on “Edit Files”. Reply back with the result you got.
Best Regards!
Some people especially new bloggers most of the time forgets the need to have a robots.txt to control the robots, spiders, or crawlers in the Internet. With this article you are giving all of us the information and reason why there is a need to put such file within our websites.
Arsie Organo Jr recently posted..T-Mobile Samsung Galaxy Tab 7.0 Plus Ice Cream Sandwich Update
Thanks you so much man for this awesome info.I was little confuse in this term.And now,i have successfully edited my Robot.txt file according to my preferences.Thanks again
Govind Choudhary recently posted..How To Set Up Namecheap Domain In Blogger
Your welcome Govind.
Thanks for a very exellent explanation, i have one question though, and that is:
I have a webshop with a robot.txt in the root but i also have a map in the root where i placed the wordpress blog.
I now would like to know if i shall add the “wordpress Disallows” (with correct path´s) in the existing robot.txt (at the root) or should i place a new (second) robot.txt in my wordpress map?
Sorry for my stupid question *lol*
You don’t need second robots.txt for your WordPress blog in any case. If you want to create any rule as allow or disallow property, you must edit the existing robots.txt file.
very nice post. I am 100% satisfied after reading this blog.
Thanks a lot for the share, It was helpful as i had problems with my robot.txt files. Now solved.
Cheers
Hi Gilbert,
Happy to see that it helped you.
You said after connected to FileZilla, go to the root directory of your WordPress blog and upload the robots.txt file. Would you mind tell me where or how do I go the root directory of my blog? Sorry I’m IT-challenged.
Hi Leslie,
You may check out the video, or after connected to FileZilla go to your WordPress folder where wp-content, wp-admin exist. Just paste the robots.text file to this directory. If you still have any problem then please don’t hesitate to speak with us.
Yes, I got it. It was my own mistake as my FTP wasn’t working properly initially. I got it up and running. Your video is a big help indeed.
A big thank you.
Happy to see it helped you
Great post. Thnk you so much. 1 question. I read on the internet that by disallowing the folder wp-content you block all your pages from being indexed. Or not?
br Gijs
Indexed.
Hi
I just downloaded SEO plugin, but I can’t seem to find a way to edit robots.txt, only htaccess file is available to edit. Any suggestions?
Hi Lucas,
go to your theme root directory, and create a text file named “robots.txt”, then you will get your robots.txt file here in SEO by Yoast setting.
Hi
Well, not sure where to put this file. Is it going into wp-admin, wp-content or wp-includes?
why, anti right click and block
, i can’t copy paste the code 
because the script 
…
Dear, hover your mouse on the source code area, you would see a “view source” or “copy to clipboard” option on the top right corner. Use it to copy the code.
thnx for sharing in brief, but problem while copying the code. .
For copying the code, hover mouse on the code a tool will pop up at the top right corner that will do the task.
I added this code to yoast. now it’s appear on mysite/robots.xml, but when i submitted sitemap to index in google webmaster it says 52 warnings Sitemap contains urls which are blocked by robots.txt. what should i do to get index properly?
Hello Pam,
It is normal to see the files blocked by robots.txt file. These are the files which we don’t want to index in Google. If you want to index any file you see blocked by robots.txt, go to robots.txt file and delete the corresponding code.