Wettone.com

Reducing SlimStat's database size

stephen@wettone.com (Stephen Wettone) — Thu, 20 Apr 2006 09:08:10 +0100

One of the most common themes in the feedback I get is that the database becomes quite large over time. That was the main reason behind the Admin plug-in. I've been tinkering with my database this evening and I thought I'd recount what I've done so other people could try it and see how their performance is affected.

The main difference between the databases in ShortStat and SlimStat is that I added lots of indexes. These are the main reason SlimStat is faster than ShortStat, but they are also the main reason that the database grows so quickly.

I added indexes on multiple fields, for maximum speed when selecting. The downside is that their cardinality can be huge. For some of them there is one entry in the index for each row in the table. So an obvious strategy is to reduce these to single fields and therefore reduce their cardinality and size.

To remove the multiple-field indexes, run these queries:

ALTER TABLE `slimstat` DROP INDEX `resource_total_uniques`;
ALTER TABLE `slimstat` DROP INDEX `searchterms_total_uniques`;
ALTER TABLE `slimstat` DROP INDEX `domain_total_uniques`;
ALTER TABLE `slimstat` DROP INDEX `platform_total_uniques`;
ALTER TABLE `slimstat` DROP INDEX `browser_version_total_uniques`;
ALTER TABLE `slimstat` DROP INDEX `country_total_uniques`;
ALTER TABLE `slimstat` DROP INDEX `language_total_uniques`;
ALTER TABLE `slimstat` DROP INDEX `dt_total_uniques`;
ALTER TABLE `slimstat` DROP INDEX `visit_total_uniques`;

Then add new single-field indexes:

ALTER TABLE `slimstat` ADD INDEX (`resource`);
ALTER TABLE `slimstat` ADD INDEX (`searchterms`);
ALTER TABLE `slimstat` ADD INDEX (`domain`);
ALTER TABLE `slimstat` ADD INDEX (`resource`);
ALTER TABLE `slimstat` ADD INDEX (`platform`);
ALTER TABLE `slimstat` ADD INDEX `browser_version` (`browser`,`version`);
ALTER TABLE `slimstat` ADD INDEX (`country`);
ALTER TABLE `slimstat` ADD INDEX (`language`);
ALTER TABLE `slimstat` ADD INDEX (`visit`);

This will all take quite a while, so I suggest pasting it all in at once and going to put the kettle on. If removing any index breaks, that probably indicates a bug in the setup script, so please let me know. :)

There's still one multiple-field index in there, on browser and version, but it's a useful index to have because one of the modules uses both those fields. I decided not to create an index on dt because it would still have large cardinality, as it's based on a timestamp to the nearest second. If there were separate fields for year, month, day, hour, minute and second, indexing them would probably be worthwhile. That's something I have been considering changing.

Almost forgot: before you do any of this, you may want to remove the HTML comments from line 190 of index.php, so you can see the execution time on screen more easily. It just shows the number of seconds that it took to display the current page. Having several before-and-after times would be useful.

If you do try this, please send feedback with as much detail as you like. I've noticed a bit of slowdown, but nothing drastic. My database is certainly significantly smaller, so I think I'm going to alter the setup script unless I get vastly negative feedback.

Comments

SlimStat via JavaScript

stephen@wettone.com (Stephen Wettone) — Tue, 14 Mar 2006 21:43:33 +0000

Help wanted!

Probably one of the most commonly-requested features for SlimStat is to be able to call it via JavaScript.

If you would like to help test this new feature, please download and extract slimstat-js.zip. It should all be fairly self-explanatory.

The simplest thing to do is to unzip it to the /slimstat-js/ directory on your site, and then include slimstat.js in all pages you wish to track. Note that you should stop including the PHP version of SlimStat, otherwise each hit will be recorded twice.

Code like this should do the trick:

<script type="text/javascript" src="/slimstat-js/slimstat.js"></script>

It uses Simon Willison's addLoadEvent function, so if that's already included in another JavaScript file on the same page, you should remove it from this one.

I'm looking for as much feedback as possible, from a wide variety of web servers and browsers, so please feel free to either comment here or on the SlimStat Google Group.

SlimStat 0.9

stephen@wettone.com (Stephen Wettone) — Tue, 24 Jan 2006 23:22:59 +0000

Version 0.9 of SlimStat, my simple but powerful web stats analyser, is now available.

Here's a brief summary of what's new in this version.

Ability to remove older data to reduce database size

By popular demand, you can now remove older data from SlimStat to help stop your database growing too large. Edit the max_data_age_days setting in _config.php and then visit the new Admin plugin (which you should see in the menu) to remove older data. SlimStat keeps a summary count of hits, visits and IPs in a separate table for posterity.

It was suggested that this should all happen automatically, but I found it to be too slow. If you'd like to fiddle around in the code, it should be reasonably simple to make the relevant changes for this to happen. I would really appreciate feedback on this function.

Ability to specify the first day of the week

Weeks start on Sunday by default, but you can now edit the week_start_day setting in _config.php to make them start on any day you wish.

New wider layout for some modules to show more text

Some modules had too much text to show in SlimStat's layout, so I've widened a few of them to make more text visible.

Translations

Many thanks to various contributors who have translated SlimStat into these languages: German (de-de), Dutch (nl-nl), and Traditional Chinese (zh-tw). To use any of these translations, change the i18n setting in _config.php to your desired language code.

If you've translated SlimStat into a different language, please send me the file and I'll include it in future releases. I've created a mailing list for discussion of i18n at Google Groups.

Windows Media Components for QuickTime

stephen@wettone.com (Stephen Wettone) — Thu, 12 Jan 2006 18:11:48 +0000

Free download from Microsoft for playing Windows Media in QuickTime; much nicer than their own player.

Best blonde joke ever

stephen@wettone.com (Stephen Wettone) — Thu, 12 Jan 2006 17:52:23 +0000

Words fail me. This is absolutely brilliant.

Cryptoquote

stephen@wettone.com (Stephen Wettone) — Wed, 11 Jan 2006 13:32:18 +0000

Crack the code to reveal the hidden quotation. Horribly addictive.

Google Safe Browsing

stephen@wettone.com (Stephen Wettone) — Thu, 15 Dec 2005 09:53:27 +0000

A new plugin for Firefox 1.5 that alerts you if the web site you're visiting is asking for your information under false pretences. Nice one.

The Elements of Typographic Style Applied to the Web

stephen@wettone.com (Stephen Wettone) — Wed, 14 Dec 2005 15:39:07 +0000

Stepping through the principles from the book The Elements of Typographic Style and applying them to the web.

Google Homepage API

stephen@wettone.com (Stephen Wettone) — Wed, 14 Dec 2005 12:37:22 +0000

Wow. It's now possible for anyone to write modules that can appear on your personalised Google homepage.

Samorost 2

stephen@wettone.com (Stephen Wettone) — Tue, 13 Dec 2005 11:54:13 +0000

Sequel to the brilliant Flash game. Chapter 1 is free, but if you get addicted you'll have to pay $9.90 for Chapter 2.

SlimStat 0.8

stephen@wettone.com (Stephen Wettone) — Tue, 13 Dec 2005 11:45:06 +0000

Version 0.8 of SlimStat, my simple but powerful web stats analyser, is now available.

Here's a brief summary of what's new in this version.

Huge internal rewrite for better extensibility

This version doesn't have many changes visible on the front end, but SlimStat's internals have been rewritten and code has moved around quite a bit. This is really just another step towards making a useful API for which plugins can be written in future.

New 'Referrers' module which appears when filtering by domain

Select a domain to filter by, and you'll see a new module showing all referrers from that domain. So if your site is linked to from various pages on a site, you can see a breakdown of which specific URLs are generating the most hits. This is also handy for looking at what strings people enter in search engines when they find you.

Links to whois services when filtering by visitor or domain

Now when you filter by visitor or domain, you'll see a question mark appear next to the filter's label. This links to a whois server where you can find out more about your visitors.

i18n

It's now possible to internationalise SlimStat. All the text is now in a separate file, which can be rewritten in the language of your choice.

To translate SlimStat, add a new file at i18n/foo/index.php, by making a copy of i18n/en-gb/index.php, where foo is the language code. To use the new language, change the language setting in _config.php from its default of en-gb.

If you've translated SlimStat into a different language, please send me the file and I'll include it in future releases.

Update: I've created a mailing list for discussion of i18n at Google Groups.

Ten reasons to learn and use web standards

stephen@wettone.com (Stephen Wettone) — Wed, 07 Dec 2005 17:13:32 +0000

Show this to anyone who doubts that web standards are a good thing.

Snowstorm

stephen@wettone.com (Stephen Wettone) — Wed, 07 Dec 2005 17:03:25 +0000

Good game for winter: clear away all the snow.

Ricky Gervais podcasts for The Guardian

stephen@wettone.com (Stephen Wettone) — Sat, 03 Dec 2005 11:01:48 +0000

Starting on Monday, he'll be making a series of 12 podcasts, tackling the big questions such as, “Jellyfish - do we need them?” Should be interesting.

Best of 2005

stephen@wettone.com (Stephen Wettone) — Sat, 03 Dec 2005 10:57:43 +0000

Like last year, here's a page that's keeping track of all the 'best of 2005' lists.

Streetcar

stephen@wettone.com (Stephen Wettone) — Wed, 30 Nov 2005 23:29:50 +0000

A self-service pay-as-you-go car rental service.

Time travel

stephen@wettone.com (Stephen Wettone) — Wed, 30 Nov 2005 23:29:15 +0000

Design of a time-based Tube map.

Conversion University

stephen@wettone.com (Stephen Wettone) — Thu, 24 Nov 2005 12:53:39 +0000

An excellent series of articles from Google on how to drive traffic to your site and how to convert visitors into buyers once they're there.

35mm Lego camera

stephen@wettone.com (Stephen Wettone) — Wed, 23 Nov 2005 14:04:13 +0000

Very cool. Who needs a digital camera when you could make yourself one of these?

5 men in a limo

stephen@wettone.com (Stephen Wettone) — Tue, 22 Nov 2005 09:53:01 +0000

You may not recognise their faces, but their voices are oddly familiar.