As mentioned above, the robots.txt file is a simple text file. Open a simple text editor to create it. The content of a robots.txt file consists of so-called "records".
A record contains the information for a special search engine. Each record consists of two fields: the user agent line and one or more Disallow lines. Here's an example:
User-agent: googlebot
Disallow: /cgi-bin/
This robots.txt file would allow the "googlebot", which is the search engine spider of Google, to retrieve every page from your site except for files from the "cgi-bin" directory. All files in the "cgi-bin" directory will be ignored by googlebot.
The Disallow command works like a wildcard. If you enter
User-agent: googlebot
Disallow: /support
both "/support-desk/index.html" and "/support/index.html" as well as all other files in the "support" directory would not be indexed by search engines.
If you leave the Disallow line blank, you're telling the search engine that all files may be indexed. In any case, you must enter a Disallow line for every User-agent record.
If you want to give all search engine spiders the same rights, use the following robots.txt content:
User-agent: *
Disallow: /cgi-bin/
Hi, everyone, I'm recommending a great, truth professional hacker that assisted me when I needed funds desperately. I was introduced to him by a friend he had already helped. I hired his service for pay pal transfer and bank transfer to my wells Fargo bank. Contact him for bank transfer, pay pal transfer, bit-coin transfer, social network hack, social network verification (blue badge) etc. He is the most trusted hacker out there. Contact: cybergoldenhacker at gmail dot com
ReplyDeleteIf you ever want to change or up your university grades contact cybergolden hacker he'll get it done and show a proof of work done before payment. He's efficient, reliable and affordable. He can also perform all sorts of hacks including text, whatsapp, password decrypt,hack any mobile phone, Escape Bancruptcy, Delete Criminal Records and the rest
ReplyDeleteEmail: cybergoldenhacker at gmail dot com
ReplyDeleteAre you willing to know who your spouse really is, if your spouse is cheating just contact cybergoldenhacker he is good at hacking into cell phones,changing school grades and many more this great hacker has also worked for me and i got results of spouse whats-app messages,call logs, text messages, viber,kik, Facebook, emails. deleted text messages and many more this hacker is very fast cheap and affordable he has never disappointed me for once contact him if you have any form of hacking problem am sure he will help you THANK YOU.
contact: cybergoldenhacker at gmail dot com