How to get search engine bots frequency

At first you can get list of all search engine bots which access you website. Prepare one concrete spider you can explore. Try to imagine how you can identify it in you log files (typically with User agent string or part).

Explore daily activity

We explore number of requests made from one particular search engine bot each day. Log file is for one month (prepare your own interval).

grep " Googlebot/" log.txt | cut -d\" -f1 \
| cut -d' ' -f4 | cut -d':' -f1 | sort | uniq -c

And we get something like this:

1       01/Jan/2007
2       03/Jan/2007
7       07/Jan/2007
4       11/Jan/2007
1       23/Jan/2007

Dates without robot access are be missing. Now we have all requests from Googlebot for entire January 2007.

Explore monthly activity

If you have access log for one year, you can explore activity of spiders for each month. We have to make small modification to our program code.

grep " Googlebot/" log.txt | cut -d\" -f1 \
| cut -d' ' -f4 | cut -d':' -f1 | cut -d'/' -f2,3 | sort | uniq -c

And we have all months and years activity of Google's spider.