Author: Kaysha Yonehara

10 Ways to Improve your Crawl Rate

Most people have heard of “crawl rate”, but what is it? Crawl rate is the frequency and speed at which Google bots – and other search engines – “crawl” your website looking for new content to index. Even though most people know being indexed is a good thing, many may not realize that crawl more about speed than frequency. A high crawl rate makes your website means it has a better chance of being properly indexed and discovered. Here are some pointers on how to improve your crawl rate. 

SEO vs. Crawlers

Although both of SEO and crawlers are related to your website and will affect traffic, what is the difference? To put it simply, SEO is optimization for users, and Crawl Rate Optimization (CRO) is optimization for the Google spider.

1. Why Spiders are Awesome

Crawl rate is assuming greater significance in the SEO process, as businesses develop their websites and provide the Google spider with more information daily. One myth about crawl rate is that it refers to the frequency of crawling. The fact is, crawl rate refers to the speed of crawling. The question is how do you get Google to crawl faster? Here are some tips and tricks to ensure the Google spiders visit you regularly and find information quickly. Crawling, according to the SEO periodic table, helps is one of the more effective actions (given a +3) for SEO.

2. Feed the Bot

hamburger graphic — Photo by Metsi

Feed the bot, improve your crawl rate  — Image by Metsi

Content is the fodder on which the bot feeds. Have a blog on your website that not only provides information to your visitors (human and bot), but also invites them to take a look through the site right then and there. There are those who update daily, inviting Google for a look, but if you can’t do it daily – and it’s overkill anyway – once to three times a week is a good bet.

3. Make Sure You Have 99% Uptime

website uptime and content — Photo by justyre

Bots (and people) need to be able to access your site — Image by justyre

Your website will be down some of the time for maintenance. But if your website is down even half the time, Google can’t pay you a visit – not to mention humans. Getting a good hosting plan that will ensure your site stays up 99% of the time is as important as having a well-developed website. This article suggests some great hosting plans. Having a high uptime not only ensures that Google spider can crawl and do its job, but it also ensures that you are available when visitors come knocking –  remember the internet never sleeps! Check out this article here that suggests some great tools to check your uptime.

4. Pay Attention to the Title

ipad click content — Photo by justyre

So much content out there — Image by justyre

Make your page title tells the story of the page. Having a common <title> tag for all your pages may be a good idea programming wise. But it’s not so good when you consider CRO.  When you write about women’s wear, having “Women’s Wear” in your title will only tell Google that the page has content about women’s clothing. Having separate titles like “Women’s Summer Wear” and “Accessories for Black Outfits” will tell the crawler exactly what is on the page. By having more specific, even unique titles and meta descriptions, it will help your visitors and Google understands the content a lot better. Google Webmaster even encourages this.

Google webmaster HTML improvements screenshot

Duplicate information looks a bit spammy

5. Make use of robot.txt

seo search — Photo by janjf93

Make it easy for people to look through your website! — Image by janjf93

There is nothing more frustrating than traveling a complicated route without a map. Make navigating your website easy. Give the spider a site map that tells it what to look for and where it can be found. Websites today have a lot of images and other media. While this is necessary, all that flash can confuse the bot that is looking for words. A site map can guide the bot to the relevant pages where it can get the information it seeks to index your site. It will also help the many users visiting your site. There are many different ways to design your sitemap but here are a few suggestions.

When Google pays you a visit, it uses robot.txt to understand which pages are available for crawling. However, use the robot.txt list to tell the crawler what not to crawl. Use this to keep Google out of routine, mundane, or unchanged pages. This will speed up the crawling and encourage the bot to return sooner for another visit. One difference between humans and the Google spider is that humans need repetition while if Google gets too much repetition, it may reject your site on its next round. So make sure it stays out of repetitive pages, media-heavy pages, and other irrelevant pages. Save those for your customers.

6. Reduce Loading Time

hourglass waiting — Photo by asi24

Nobody likes waiting! — Image by asi24

Let’s get back to your hosting plan for a moment. If your site uploads slowly for reasons such as heavy media load, the bot will not wait around. A site that loads quickly, can be crawled and indexed quickly by the Google spider. One may argue that loading time is a necessary evil because media is required. However, a media heavy web page not only takes time to load, it is also likely to confuse spider and human visitors! So save the media for the converts. Give the spider words to understand your product. By all means throw in an image or two so that people get a feel for the product, but keep the resolution low, and scale the file down or compress if you have to. Try to think of image sizes as less than 1 MB. Give the Google spider more information about your business than pictures of it. That will give you the advantage of Google Index as well as improve your conversion rate. If you’d like to quickly check your load time try out with the PageSpeed tool from Google.

7. Use Google Tools Effectively

google browser — Photo by JuralMin

Google has many tools to offer — Image by JuralMin

Over the years, Google has offered up a number of tools like Google Ad-words and Webmaster to help you optimize your website for search engine crawling. Use these tools effectively to tell the Google Spider how to crawl your site and what to look for. Google ad-words is a bit more about SEO wording but can still be very effective for your site. For the Webmaster tool, it is very helpful to tell Google how to crawl your site, for instance, if your site is in multiple languages, the spider will have trouble crawling your website so you need to provide that information to make the spider’s job easier. In its own article about Google Bot, Google says, “Google spider shouldn’t access your site more than once every few seconds on average.”

8. Interlink Your Content


Easier and faster to crawl for humans and bots — Image by ivke32

Back to maps once again! If your web pages are well linked, the spider can crawl faster since one page leads it to the next, so interlink your content. Interlinking creates a veritable map route for the bot to follow. A linked site is easier and faster to crawl than a site that has many independent pages. Again, Google’s Webmaster is a handy tool to check whether your interlinking is optimal. It also lets Google know you are updating.

9. Use Tags and Alt Text

programming graphic — Photo by Tumisu

Don’t forget to make your images readable — Image by Tumisu

No website is complete without images, so following the less is more rule, caption the ones you have effectively. The images you do have should be accompanied by alt text, captions, and tags that can help Google spider understand what the images are about. Make good use of all attributes of the <img> tag. There are many tools on the web to check your alt text, like this one.  Again while alt-text should be brief, give as much information as you can so that a clear picture can be formed.


As the internet gets flooded with more and more websites, Google is forced to decide which sites NOT to visit. Improving your crawl rate is not just about doing the right thing, but also about what not to do. In fact, it would not be wrong to say that the next step after SEO is CRO (Crawl Rate Optimization). They happen together and are built into the fabric of your site and content creation. So here’s to better building!

14 Apps to Help You Work Smarter, Not Harder

Work isn’t always easy. Everyone faces a road bump from time to time and we lose steam with our projects. Luckily many desktop and mobile apps are available to make our lives more productive and easier. The phrase “work smarter, not harder” is even backed by science!

Continue reading

7 Great Ways to Learn Code for Womens Entrepreneurship Day

It's never too late to get involved with coding! — Photo by ivke32

A coding plate-form  — Photo by ivke32

The tech industry is one of the fastest growing industries, but only 28% of proprietary software jobs are held by women. The industry is slowly changing, and diversity initiatives are supporting women to get jobs and rise in company ranks. At Oursky, we have women across all our teams and leadership whom have helped shape our inclusive company policies. For Women Entrepreneurship Day (Twitter #choosewomen), we wanted to share the awesome organizations — founded by women! — that help women and girls take off as awesome developers.

kiki's delivery service

Image courtesy of Girl Getting Her Geek On

Continue reading

© 2017 Oursky Blog

Theme by Anders NorenUp ↑