It has been a half month since ThemeLib.com was created, not a long time, huh ?. Today, I decided to test how is my site in two popular Search Engines : Google and Yahoo.
Here is the result :
Google: get 48 results returned with keywords
Yahoo: get 35 results returned with keywords
All of [...]
ArticleManiac.com Search Submit And Find Free Articles For Your Web sites Newsletters and Ezines Articles Robot.txt file is used to give instructions to the robot what they crawl and what should not be crawled. This gives you some level of control over the robots. You can also issue some indexing instructions to search engines through this source. - Bali based Web Design, Development and SEO Servi
Having a effective robots.txt file will help your website or blog rank higher in Search Engines, receive higher paying relevant Ads, and can increase your blog traffic. Tell bots what to index and what they should not by creating a Robots.txt file and placing it to your root folder. Here's how...
What is a robots.txt file?
The robots.txt file is a simple text file that must be placed in your root directory (http://www.example.com/robots.txt). It tells the search engine spider which web pages on your website should be indexed and which web pages should be ignored.You can use a simple text editor to create a robots.txt file. The content of a robots.txt file consists of so-called “records”.A record contains the information for a special search engine. Each record consists of two fields: the user agent line and one or more Disallow lines.
Here’s an example:
User-agent: googlebot
Disallow: /cgi-bin/
This robots.txt file would allow the “googlebot”, which is the search engine spider of Google, to retrieve every page from your site except for files from the “cgi-bin” directory. All files in the “cgi-bin” directory will be ignored by googlebot.
How to check your robots.txt file
Open your web browser and enter www.yourdomain.com/rob
I briefly covered the importance of a robots.txt file in my recent article: 5 Tips for Proxy Webmasters. There has been alot of discussion about this over at the proxy.org forums and most people have come to the conclusion it is an essential part of building a quality proxy website.
A correctly configured robots.txt tells search engine spiders where they can go on your website, if you dont stop spiders from accessing your proxy script then you could damage your website search engine rankings and trust. Search engines can be a vital source of organic traffic to proxy websites, if you dont block spiders from visiting proxied pages your site could get penalized (more…)
I just read aaron wall’s blog about robots.txt that help help him to make 10K from just one page. That owesome!. Actually i’ve already wrote about robots.txt before. It’s about robots.txt to the rescue. I wish i can make money like him..Ganbate to me!
Aaron Wall is one of SEO expert that provide a lot free [...]
Robots.txt files tell Search Engines what should and should not be crawled NOTE – This is very different from the Robots Meta Tag. The crawler will see this file before it tries to call the page, so this file will override the Robots Meta tags on the pages. Robots.txt files should be stored in the root directory Remember, the point of the robots.txt file is to exclude pages from being crawled. So if a page or directory is banned, it will never even get to see what code is on those page(s). Accordingly, no code of those pages could change the bots behavior to re-index the page. So, robots.txt will overwrite the meta and robots tags on the page. More information regarding robots.txt can be found at Sample robots.txt file to allow all pages to be crawled: User-agent: *Disallow: With o
Not many web master take the time to use a robots.txt file for their website. For search engine spiders that use the robots.txt to see what directories to search through, the robots.txt file can be very helpful in keeping the spiders indexing your actual pages and not other information, such as looking through your stats!
The [...]
If you are thinking you have developed a truly great keyword-rich-unique-content fully optimized website for the search engines and an attracting site for the visitors - that’s fine, but do you know you are missing something? A robots.txt file. Did you include it? By the way do you know what’s the importance of a robots.txt [...]
If you have a website you really need to have a robots.txt file. It gives search engine spiders specific commands and it is easy to use and easy to maintain. Here is an easy guide to a robots.txt file in five minutes.
There are times when you don’t want a search engine to index a page [...]
I’ve been trying to find out how to block some dynamic urls from the Googlebot. The search bots for Yahoo! Slurp and MSNBot use the same or very similar syntax to block dynamic urls. As an example I have this one line in my htaccess file which allows me to use static pages instead of [...]
