Introduction to advanced website files
Setting up the HTML, CSS and other basic files for your website are of first importance in getting your site online and operating, but there are some additional types of files that, when implemented properly, can help your site become more serach engine friendly and provide additional functionality. Even though these files aren't required to make your site operate correctly, their benefits are worth considering and implementing into your website.
As we mention in other areas, robots are automated programs that search engines use to crawl the internet, looking for new information. In some cases, there may be files and/or directories that you don't want robots to find, crawl and index. By using a robots.txt file, you can instruct search robots where you don't want them to go to find information on your site. This files can also tell them which search engines are welcome to crawl the site and which ones aren't, as well as how often you want your site crawled.
Sitemaps are basically pages that act like a website directory that list your websites pages, showing both visitors and search engines the pages you want seen. There are 2 basic types of sitemaps, one for visitors and one for search engines. The visitors sitemap usually looks just like the rest of your web pages, and uses links to let your visitors go to any page within your site that are listed in the sitemap. For search engines, an XML sitemap is often used, and these can easily be created by online sitemap generators or programs you add into your website directory. These XML files can be submitted directly to the search engines so that they find all of the pages you want indexed in their systems.
htaccess files are extremely versatile, and most websites can benefit from using them. htaccess can be used to redirect from old or nonexistant pages to new pages, stop bandwidth loss from hotlinked images on your site, deny certain IP addresses and countries from accessing your site, password protect directories, and implement URL rewriting for nicer looking, and SEO friendly, url addresses among other things. The smallest mistake or incorrect character in your htaccess file can bring your site to a grinding halt, but when written correctly can bring huge benefits to your website.