How InfiniteInsight Can Make You A Great Data Scientist

Discussions abound in blogs and forums about the performance of KXEN’s InfiniteInsight™ predictive analytics solution. I think it is time to state KXEN’s public position on this.

This earlier post explained why KXEN developed its own algorithms. As I mentioned there, a majority of KXEN’s customers bought its InfiniteInsight™ technology after benchmarking it against the first-generation tools, like SAS Enterprise Miner and SPSS Clementine, and finding InfiniteInsight™ to be superior.

At KXEN, we strive to automate the tedious, error-prone tasks inherent to predictive analytics — tasks like missing value and outlier processing, variable encoding, variable selection, post-hoc analysis, and so on. Certainly, this does not diminish the job of the data scientist; rather, InfiniteInsight allows the data scientist to focus on more important tasks, like gaining insight from the data.

We believe the role of a data scientist is to solve business problems by using data mining functions to turn raw data into strategic decisions. When a data scientist is using KXEN’s InfiniteInsight as part of his toolbox, he will reach a solution 10 times faster and more consistently than when using first-generation tools.

Frequently, tools like InfiniteInsight are used to apply advanced analytics to processes which previously had no analytics at all. Here we generally see great leaps in predictive accuracy over the methods used before. On some occasions, however, we are called upon to optimize problems to which analytics have already been applied; sometimes an extra 1% improvement in predictive accuracy can be worth millions of dollars!  In these rare cases, this last 1% requires a lot of advanced knowledge and some very sophisticated techniques. Here, too, data scientists can use KXEN’s InfiniteInsight as part of their toolbox to build ensemble models, for example, to get every last bit of predictive goodness out of their data.

As I myself am a data scientist, I wouldn’t be worth my salt if I didn’t have data to back up my claims! So I created a demonstration video (I blogged about it earlier) using the data from the KddCup 98 challenge:

The task is basic enough: to predict donations to a charity for veterans. The data is available to everyone (even the test data is available without the need to register on a site), and there is a nice presentation of the leaderboard, so we can compare the KXEN results with the set of data scientists that submitted their results. The training data set contains ~95,000 lines and 481 columns (which we will heretofore refer to as “variables”).  What is also nice is that the results are not based on an abstract mathematical measure, such as AUC, but on a simple, concrete measurement: dollars donated!  Models are compared with the profit associated with the persons contacted by a model (a person is considered “contacted” by a model if the estimated donation amount is higher than 68 cents); the profit is the donation minus the 68 cents (68 cents being the cost of the contact).

The process with InfiniteInsight is simple: 1) you select the training data set, 2) you describe your 480 variables, 3) you select the continuous target and… you run!. The entire training process takes ~5 minutes and the apply into a database (to be able to use InfiniteInsight Explorer to calculate some statistics) is made in a few seconds.

To represent the fact that a random process is always a difficult beast to forecast, I have used a trick to pretend as if, instead of using KXEN’s InfiniteInsight on 12 different problems, I ran it in 12 parallel universes.  As part of building a model, InfiniteInsight randomly separates the input data into an Estimation set (for training the model) and a Validation set (for automatically testing the model).  One can mimic the variability of InfiniteInsight’s results on a given problem by playing on the random seed used internally by KXEN to separate the training data set into these two parts.

The results are presented below. Table 1 shows the results of the best 21 data scientist teams that participated in this challenge (out of an overall 37 teams who submitted results), and Table 2 presents 12 runs of InfiniteInsight™ V6.0 using different seeds.

KDDCup Participant Results:

KXEN’s InfiniteInsight Results:

There are several findings that can be extracted from these tables.

A given InfiniteInsight run (using a random seed) may not be the best, but it is not far!  Even our worst run is ranked 3rd.  That is OK: we have never claimed to be the best always (that would just be vanity), but to be among the best almost every time, and without tuning.  Bear in mind that while the teams of data scientists in Table 1 probably each spent weeks on their results, each InfiniteInsight run took just 5 minutes!

All the winners used advanced techniques like ensemble models, two-stage models and so on.  However, we have simply used InfiniteInsight Modeler’s regression module out of the box, and that is it!  On average across the 12 runs, a company using KXEN’s InfiniteInsight would have made almost $2,000 more on this problem than the best 12 best data scientist teams that participated to this challenge.

Some data scientists did worse than random (the 13th entry).  Some data scientists even lost money on this challenge (the 21st entry).  InfiniteInsight came up ahead with every run.

I know that is just a single data point, and that one point is not enough to draw a generic rule, but I will ask you to trust me (and the hundreds of KXEN customers) on this:  the performance we see here is very similar to that which we find in real life problems!

The conclusion is this: there are good data scientists, and there are less good data scientists.  I’m not saying that the teams that did worse than random in this challenge are not good, they may have just been unlucky. But if there is a dramatic shortage of data scientists, as is generally acknowledged, you can imagine that good data scientists are even harder to find!

KXEN’s InfiniteInsight automates a lot of the work that a good data scientist does, and so the results you get from it are generally equal to or better than that of the best data scientists.  InfiniteInsight can turn just about anyone into a good data scientist, and it can make a good data scientist into a great data scientist.