HowTo: Make your own Kickstarter charts

Back to the Kickstarter Progress

Boring Ramble
I am totally dumbfounded on why Kickstarter doesn't do this sort of charting themselves for their hosted projects. Perhaps they offer this sort of thing to the project owners, just not the backers and the general public? [Yup, they have project charts for owners only] Although, if that were the case, you would think that copies of the charts would have 'leaked' by now [and a week later, someone leaks their charts, see Auditorium 2 (thanks blazerfrost for letting me know!)] . I realized how useful even simple versions of these charts would be after the Order of the Stick Kickstarter where Rich Burlew included a hand drawn chart with each update. The loss of these sort of charts was really noticed during the DoubleFine Kickstarter. So, for the next kickstarter I cared about (Wasteland 2), I figured I could do 'something.' And here it is!

Tools Used

Part 1: Getting and Saving the Data
(please try not to cry when you see how horrid the screen scraping 'code' is, it was really only meant for a one off)

This is handled by a file called getdata.pl.

I'm just grabbing the URL with the numbers with curl and grepping out data with the 'num' class. The first two happen to be Backers and Pledges. I remove the surrounding HTML and there's the data. 

#!/usr/bin/perl
use DBI;

@data = `/usr/bin/curl http://www.kickstarter.com/projects/inxile/wasteland-2 2> /dev/null | grep class=\\\"num `;

$backers = $data[0];
$backers =~ s/\<div class=\"num\"\>|\<\/div\>|\n|\,|\$//g;
$money = $data[1];
$money =~ s/\<div class=\"num\"\>|\<\/div\>|\n|\,|\$//g;

Update Sep 14, 2012: Kickstarter changed their page layout at some point, here is something that works for the new look: 
#!/usr/bin/perl
use DBI;

@data = `/usr/bin/curl http://www.kickstarter.com/projects/obsidian/project-eternity?ref=live 2> /dev/null | grep data-value `;

foreach $line (@data) {
  if($line =~ /data-backers-count\=\"(\d+)\"/) {
    $backers = $1;
  }
  elsif($line =~ /shorter_money/) {
    $line =~ /data-pledged\=\"(\d+\.\d+)\"/;
    $money = $1;
  }
}
Now that I have the data, I stick it into a database (although, for datasets this small, even a flat file would be ok) 
$dbh = DBI->connect('DBI:mysql:**DB_NAME**', '**USERID**', '**PASSWORD**'
	           ) || die "Could not connect to database: $DBI::errstr";

$sql = "INSERT INTO wastelands (backers, money) VALUES ($backers, $money)";
$sth = $dbh->prepare($sql);
$sth->execute();
Finally, I create static versions of the charts I want. The chart code can be run directly from the web, but I've found that people tend to refresh charts a lot, even when they only get new data once an hour so it's easier on the server if it doesn't have to think too much when that happens. 
$dir = "/home/ruinedkingdoms/public_html/wasteland2";
system("/usr/bin/php $dir/makechart.php 2>/dev/null > $dir/current.png");
system("/usr/bin/php $dir/public_chart.php hours 8 total > $dir/last8.png");
system("/usr/bin/php $dir/public_chart.php days 8 total> $dir/last8days.png");
system("/usr/bin/php $dir/public_chart.php hours 8 diff > $dir/last8diff.png");
system("/usr/bin/php $dir/public_chart.php days 8 diff> $dir/last8daysdiff.png");
Here is the whole file.

I added a line to the crontab to run this script every hour: 

00 * * * * /home/ruinedkingdoms/public_html/wasteland2/getdata.pl

Part 2: What the Database Table Looks Like
The table used is really simple: 

mysql> desc wastelands;
+---------+-----------+------+-----+-------------------+-------+
| Field   | Type      | Null | Key | Default           | Extra |
+---------+-----------+------+-----+-------------------+-------+
| backers | int(11)   | YES  |     | NULL              |       |
| money   | int(11)   | YES  |     | NULL              |       |
| ts      | timestamp | NO   | PRI | CURRENT_TIMESTAMP |       |
+---------+-----------+------+-----+-------------------+-------+
3 rows in set (0.02 sec)
If this gets a lot of use, it would make sense to have another field as part of the primary key for which project is being tracked. that way one table could handle multiple projects (like I said earliler, this was meant to only be a one-off).

Part 3: The "Small" Charts
All four of the "small" charts are handled by one bit of code, public_chart.php. It can either be run from the command line (like in the perl script) to generate static files, or directly from the web for 'live' charts allowing for real-time changes to the parameters (like how many hours/days to show). The three paramaters that this code takes now are (1) unit - days or hours (2) period - how many units to show and (3) measure - total or diff (show accumulating values or by unit values). Goals and chart titles are hard coded. If this were a real 'project' I would have a database table for each kickstarter project's title and goals so the charts could use that directly. That would require passing in the project's identifier to the chart code as well.

As an example, here is a 12 hour version of the hourly delta chart (running 'live' with some help from JavaScript):


Here is the chart code. 268 lines of commented PHP I wrote in an hour or two. That means there might be bugs!

Part 4: The "Big" Charts
I'm actually re-working this right now, so I'll update when that's done.
Update Aug 16, 2012: Doesn't look too good on me re-working them, so here they are... warts and all. I was going to merge the two "Big" Charts into one codebase because they are really the same chart. The problem I was having was I ran out of kickstarters that I cared about so I lacked motivation. Anyway, here are makechart.php (the accumulating total "top" chart) and testchart.php (the hourly difference "bottom" chart).

Conclusion
Hopefully that will give anyone else wanting to do these sort of charts in the future a kick start (yup, I went there :-P ).

Good Luck,
Adam