Comments on: How To Add/Edit robots.txt File to WordPress for Better SEO

By: WordPressians

WordPressians — Fri, 02 Aug 2013 19:52:09 +0000

These are simply a path of your sitemap, and video sitemap URL. Simply remove both URL in the code, and paste your own URLs.
Regards!

By: Dan

Dan — Tue, 02 Jul 2013 01:08:07 +0000

Thanks a bunch for this. This was the first tutorial I came across that was written in plain English. It made a lot of sense when I was looking at the sample.

Just put it into use on my site, looking forward to seeing some results soon.

By: Kenjie S. Suarez

Kenjie S. Suarez — Fri, 14 Jun 2013 04:27:43 +0000

One of the best tutorials around!… I’ve been searching articles about robots.txt since google webmaster tools cannot access my robots.txt. I will certainly implement the code on my new site..one question though, you made mention about make sure to change yourindex and yourindex-video with your own data. Can you please illustrate this? I am a newbie, i’m sorry…Thank you so much.

By: WordPressians

WordPressians — Tue, 11 Jun 2013 12:08:31 +0000

Hi Praveen,
Thank you for dropping a comment here. We have disallowed the feed directory simply because do not crawl the feed content which is same as the real content, and search engines never find the copied content of real content via our own RSS feed.

By: WordPressians

WordPressians — Tue, 11 Jun 2013 11:54:08 +0000

Hi Pankaj,
Thank you for dropping a comment here. Keep in mind that the change in robots.txt file is not suddenly impact in search engines. It will take some time to re-crawl your pages (including robots.txt) by the search bots for a limited time. If you have made changes in your robots.txt file it will read by the bots, and follow the rule you have changes into this file.

By: Pankaj

Pankaj — Sat, 01 Jun 2013 19:43:01 +0000

Thanks ,actually robots.txt was missing in my site when i acessed through fileztlla .Now with help of ur video ,i was able to edit robots.txt and upload in server and can also acess.But the question here WHEN AND HOW CAN THE SEE THE CHANGE IN THE WEBSITE..what are the parameters to look in to really see that this change has really made some impact..Highy appreciate if could answer this .Thanks ,Pankaj

By: Praveen

Praveen — Sat, 11 May 2013 10:57:04 +0000

Thanks for the great Article published here, this is very informative and helpful. i am little confuse, why to Disallow: /Feed/

By: WordPressians

WordPressians — Sat, 23 Mar 2013 05:58:36 +0000

Hello Pam,
It is normal to see the files blocked by robots.txt file. These are the files which we don’t want to index in Google. If you want to index any file you see blocked by robots.txt, go to robots.txt file and delete the corresponding code.

By: Pam

Pam — Wed, 20 Mar 2013 16:32:23 +0000

I added this code to yoast. now it’s appear on mysite/robots.xml, but when i submitted sitemap to index in google webmaster it says 52 warnings Sitemap contains urls which are blocked by robots.txt. what should i do to get index properly?

By: WordPressians

WordPressians — Fri, 22 Feb 2013 09:27:43 +0000

For copying the code, hover mouse on the code a tool will pop up at the top right corner that will do the task.