Creating my first Tableau web data connector : part 3

At last, the final part of the trials and tribulations of creating my first Tableau Web Data Connector… Part 1 went through the pre-requisites, and building what I might generously term my “user interface”. Part 2 was a struggle against the forces of web security. And in this part, we battle against the data itself, until the error-message dragon is truly slayed.

So, the in the last part, we ended up with a connector that could ask for the username of a BoardGameGeek user, connect to a CORS-enabled version of the API, and complain that whatever objects it found “are not valid as a React child”.

That makes sense, as I didn’t yet tell it what sort of data objects to expect.

As I’m following the Tableau web data connector tutorial, which accesses data from URLs like this one, I figured I’d look at what data that URL returns precisely, compare it to the web data connector tutorial code, and then, when I understand the relationship between the code and the data from the Yahoo URL, I might be able to adapt the code to fit the format my chosen data arrives in.

Here’s the data one gets if you point your web browser to the URL that the tutorial is teaching you how to build a WDC from:

Capture

OK, stock-pickers will immediately recognise that it’s stock quote data, with dates, highs, lowers, symbols and all the other lovely gubbins that comes along with basic stock charts. It’s in JSON format, which is human-readable, but only in small doses.

So I need to understand the relationship between the above data, and this piece of code:

if (data.query.results) {
              var quotes = data.query.results.quote;
              var ii;
              for (ii = 0; ii < quotes.length; ++ii) {
                  var entry = {'Ticker': quotes[ii].Symbol,
                               'Day': quotes[ii].Date,
                               'Close': quotes[ii].Close};
                  dataToReturn.push(entry);
              }

I can kind of see it by eye – there’s obviously JSON entries for Symbol, Date, Close, query and so on. But can’t we make it easier?

Yes we can, because there’s such a thing as a web JSON viewer. Past all that stock-related text into something like http://www.jsoneditoronline.org/ and you get a nice hierarchical visualisation of the JSON structure, like this:

Capture

So if we assume that “data” is referring to the overall set of data itself, and use dot notation to traverse the hierarchy, we can see a line in the code that says:

var quotes = data.query.results.quote;

That would seem to fit into the JSON structure above, where below the “object” level you have a level called query, a sublevel called results, and a sublevel called quote.

Capture

The variables “quotes” that the code creates therefore is basically referring to everything at/below the “quote” level of the JSON hierarchy.

Then you get a bunch of records which are the data of interest itself. These are numbered 0 for the first datapoint, 1 for the next, and so on.

If you know a bit of basic programming, you might note that the WDC code has “for loop” with a counter variable called “ii” that is set to 0 at first and is incrementing by one each time it runs. That seems to fit nicely in with the idea of it iterating through datapoint 0, 1, …n until it gets to the end (which is effectively the length of the quotes dataset, i.e. quotes.length).

Each one of these JSON records then has a few attributes – and they include the ones mentioned in the code as being part of whichever “quotes” subrecord we’re on; “Symbol”, “Date”, “Close”.

   var entry = {'Ticker': quotes[ii].Symbol,
                               'Day': quotes[ii].Date,
                               'Close': quotes[ii].Close};

Capture

Awesome. So where does “Ticker”, “Day” and “Close” on the left hand side of those code lines come from?

Well, that’s up to us. Remember the “add code for defining columns” part of the tutorial?

myConnector.getColumnHeaders = function() {
    var fieldNames = ['Ticker', 'Day', 'Close'];
    var fieldTypes = ['string', 'date', 'float'];
    tableau.headersCallback(fieldNames, fieldTypes);
}

There they’ve defined the fieldnames they want to have Tableau display (which can be totally different from the JSON names we saw above in the datasource) and the field types.

Field types can be any from this list:

  • bool
  • date
  • datetime
  • float
  • int
  • string

So, let’s look at what’s returned when I make the call to my Boardgames datasource.

Capture.PNG

OK, I can see some fields in there, but it’s kind of messy. So I pasted it back into the online JSON editor.

Here’s what the relevant part looks like when formatted nicely:

Capture.PNG

So, the interesting information there is probably the boardgame’s name, year published, URLs to its image and thumbnail, and the various statuses (own, want, etc.) and the number of times its been played (numplays).

I therefore edited the “getColumnHeaders” code snippet to reflect the fields I wanted, and the type of data that would feature in them.

myConnector.getColumnHeaders = function() {
		var fieldNames = ['BoardgameName','YearPublished','ImageURL','ImageThumbnailURL', 'Plays','Owned','PreviouslyOwned','ForTrade','Want','WantToPlay','WantToBuy','Wishlist','Preordered','LastModified'];
		var fieldTypes = ['string','int','string','string','int','int','int','int','int','int','int','int','int','datetime'];
		tableau.headersCallback(fieldNames, fieldTypes);
	}

Now I’ve defined the fields, I can go back to the retrieving results section of code (if (data.query.results)…) and, now knowing the structure of the JSON generated by my API, parse out and assign to the above variables the data I want.

I decided to call the collection of data I was building “games” rather than “quotes”, because that’s what it is. I next noted that each individual “game” within the JSON is listed in a hierarchy below the “item” entry which itself is below “items”.

Capture.PNG

(“Amyitis” is the name of a boardgame, rather than an allergy to people called Amy, believe it or not).

So, I assigned all the items.item data to “games”

if (data.items.item) {
var games = data.items.item;

And copied the tutorial to loop through all the “items.item” entries, each of which is a boardgame, until we’d reached the end i.e. when the number of times we looped is the same as the length of the data table..

var ii;
for (ii = 0; ii < games.length; ++ii) {

Finally, it’s time to assign the relevant bit of data returned from the API to the variables I’d defined above.

Capture

At first I got a little confused, because the JSON output had a bunch of _ and $ entries that didn’t seem similar to what was returned in the tutorial dataset. But it turns out that’s nothing to worry about. Just treat them as though they were any other text.

var entry = {'BoardgameName': games[ii].name[0]._,
'YearPublished': games[ii].yearpublished[0],
'ImageURL': 'https:' + games[ii].image[0],
'ImageThumbnailURL': 'https:' + games[ii].thumbnail[0],
'Plays': games[ii].numplays[0],
'Owned': games[ii].status[0].$.own,
'PreviouslyOwned': games[ii].status[0].$.prevowned,
'ForTrade': games[ii].status[0].$.fortrade,
'Want': games[ii].status[0].$.want,
'WantToPlay': games[ii].status[0].$.wanttoplay,
'WantToBuy': games[ii].status[0].$.wanttobuy,
'Wishlist': games[ii].status[0].$.wishlist,
'Preordered': games[ii].status[0].$.preordered,
'LastModified': games[ii].status[0].$.lastmodified
};
dataToReturn.push(entry);

In the above, you can think about games[ii] as being the individual boardgame record. The “for loop” we defined above substitutes each record into the ‘ii’ variable, so it’s accessing games[0], games[1] etc. which translates into data.items.item[0] and so on, if you remember how we defined the games variable above.

Then to find the boardgame’s name we need to traverse into the “name” chid of the boardgame itself, then look for the first entry below that (always the first one here, so we can refer to that as entry [0]), and look for the field that is shown as an underscore, _.

Capture.PNG

 

Hence:

'BoardgameName': games[ii].name[0]._,

Rinse and repeat this for each element of interest, and you’ve collected your dataset ready for Tableau to use!

Of course, the world is not perfect, and, in reality, I did not manage to do the above without making the odd mistake here and there. I got blank fields sometimes, when I knew they shouldn’t be, or fields with the wrong data in. As I was just using Notepad and the online simulator, I wasn’t really getting many useful error messages.

Lesson learned: you can use the code

tableau.log("YOUR MESSAGE HERE");

to display the message you write in the Google Chrome developer console. You might remember from part 2 that you can bring that up by pressing Ctrl Shift I in Google Chrome, and selecting the “Console” tab.

Why is that useful? Well, one, if it displays your message then it means that piece of code ran, so you can check you’re not skipping over any important section. And secondly, you can append text stored in variables to it. So for instead I could write:

tableau.log("the name of the game is " + games[ii].name[0]._);

And, if my code is correct, it should print out the name of the boardgame as it runs.

If I messed up, perhaps using the wrong index, or forgetting the underscore, it will output something different, and perhaps also an error message, guiding me towards the source of the problem.

That, as they say, is basically that! I ran it through the hosted Tableau Web Data Connector simulator again,  this ticking the box so it will “automatically continue to data gather phase”. And, lo and behold, below you can see that it has interpreted the information about that famous Amyitis game into a nice tabular format.

Capture.PNG

Once you’ve got to that stage, you’ve already won. Just load up Tableau Desktop for real, select to connect to a web data connector and up should pop your interface, and later, your data.

 

Creating my first Tableau web data connector – part 2

Now for the next part of the creating-a-web-data-connector journey: previously, I had got all the software I needed to create a web data connector installed (well, to be fair Notepad comes with Windows so that wasn’t hard) and designed my user interface.

Now it was time to move on to Tableau tutorial section 5 – using the information retrieved from the user of my UI in order to have Tableau connect to the BoardGameGeek API and retrieve the relevant data for analysis.

I read through the Tableau WDC tutorial, and, using some previous programming-other-stuff knowledge, realised that this tutorial was a mix of explaining the fundamentals of connecting to a generic web data connector, and explaining some very specific details of the example Yahoo finance web data connector the example showed.

The first thing to do then was to determine which bits were essential for my simple attempt and which were only there to facilitate the specific Yahoo API.

  • The “code for defining columns” section is obviously important. I need columns to put my data in.
  • The Yahoo Query YQL less so – my API is nothing to do with Yahoo.
  • The helper functions (buildUri, getFormattedDate,makeTwoDigits), no, these are just functions to help construct the YQL query needed to access their example API. It’s great to know that it’s possible to use helper functions like this if needed, but chances are, if I did need some, they’d not be the same unless my API was very similar to their Yahoo example.
  • The “code to get stock data”, well, I don’t want stock data, but I do want some sort of data so obviously some of that’s going to be relevant, so I pasted that into my file as-is, noting from the code that some of it was clearly going to need to be replaced given I don’t want boring finance data when exciting boardgames data is available instead.

It seemed obvious that providing the weblink to my API of choice -http://boardgamegeek.com/xmlapi2/collection?username=[whatever] – was going to be key.  So I replaced their connectionURI  code with a simple text variable of that, using the user-supplied BGG username I had got from the previous part.

connectionUri = 'http://boardgamegeek.com/xmlapi2/collection?username=' + bggUsername

 

(a URI is a Uniform Resource Identifier)

Of course this wasn’t going to be enough to make it work – I have not yet told Tableau what data I want it to receive and how to format it – but figured I’d run it through the simulator and see what happened. Nothing especially good did. But I did learn about “CORS”.

Now, again, no particular error messages tend to pop up by default even when things go fatally wrong in the simulator . It might look like it hung, it might go blank or similar, but no friendly message.

Lesson learned: If you are working with the super-basic tools I had at hand, what you need then is the built in developer tools Chrome has which you can bring up by pressing Ctrl Shift + I together. Switch it to show the “Console” tab, and you might see an error message there to help you out, if you’re lucky.

Here’s what I saw:

Untitled picture

Now, it’s been a long time since I did any serious web developing, and I had never suffered an Access-Control-Allow-Origin problem before. A swift bit of Googling led me to this nice explanation where I learnt that it’s a security feature “implemented in browsers to restrict interaction between documents (or scripts) that have different origins”. That makes sense; I don’t own either the BoardGameGeek site nor Github, I’m sad to say.

Of the 3 solutions that Jvaneyck suggests, it seemed like if I could find a JSONP source for my API instead of the XML one then that would be simplest, given I have no control over either server nor intention of building any infrastructure. So, can I get the BoardGameGeek API in JSONP format? Well, another quick Google ™ let me to a post on the BoardGameGeek forum, where a kind user named strathmeyer had offered up “Ajax-ready BGG API for any web programmers out there”.

The best thing is that I’ve enabled CORS headers, so you can use the API straight from the web browser

Yes, here indeed that was the best thing! Access-Control-Allow-Origin issue solved, and all thanks to the work of someone else 🙂

Lesson learned: I later found that Tableau 1) has a whole page dedicated to this exact problem, and 2) there’s an example in their SDK of various methods to get around this issue when connecting to XML. So next time, I should probably RT original M before going Google crazy.

But for now, all I needed to do was change the URI I pointed to from my web data connector to:

connectionUri = 'http://bgg-api.herokuapp.com/api/v1/collection?username=' + bggUsername

I ran that through the simulator once more, of course not expecting it to work before I had defined what data I was actually after – but it did eliminate the previous error message so I knew things were moving on nicely.

So when it ran through, Ctrl Shift + I back into Chrome to bring up the inspector and saw this message.

Untitled picture

My variable is not defined?! Where did it go? Oops, my fault, I’d embedded a previous function in this one. I needed to be more careful with my bracket-pasting finger. But it was simplicity itself to move the code back to where it should be, and re-simulate.

 

Untitled picture

OK, “uncaught invariant violation” is perhaps a little cryptic, but it’s sounds like it’s talking about having found something it didn’t expect. That could be my data (spoiler: it was). So now I needed to tell it what to expect and what to do with it.

Tune back in, same place, different time, for the exciting conclusion!

 

Creating my first Tableau web data connector – the highs and the lows: part 1

After having successfully (enough) completed my introduction to creating a real live Tableau web data connector, I wanted to follow the lead of one of the inspirations for this otherwise unnecessary effort – Chris Love’s Persiscope on the subject of sharing Tableau-related failure as well as unbridled success –  and document something about the less smooth aspects of the journey it took. If nothing else, they’ll be useful notes if I have to do this again for real in the future.

First of all, I gave myself an unnecessarily harsh setup 🙂 . I was working on my company computer (shhhhh) as it has Tableau Desktop installed, and I figured that’d be important.  The significance of that is that I don’t have the admin rights needed to install any new software on it. And I don’t do web development as part of the day job, so I was restricted to using only very generic tools. In practice, this meant Windows Notepad for the most part, and Google Chrome for testing.

I had downloaded the Tableau Data Connector software development kit  I was a little worried at claims that I’d need to install a web server to use it, which I’m not sure our work IT security overlords would appreciate – but in the end, I learned that you can successfully create a Tableau web data connector without having:

  • the web data connector SDK
  • a web server
  • admin rights
  • …Tableau (yes, seriously).

What I did need:

  • a web browser (I used Chrome)
  • a text editor (I used Window notepad, because, well, what more could you want?)

I had also decided to try and work it out without asking anyone anything. The Tableau user community is pretty amazing in terms of generously giving up answers, and I’m sure someone would have solved any problem I encountered so fast all the fun would have gone.

My approach then was to follow the Tableau Web Data Connector tutorial. In this tutorial, they build a web data connector to a financial website to retrieve stock data. I figured for a first try I’d just follow along and just put the details of the BoardGameGeek API I wanted to use in instead of the Yahoo API.

I’m not going to go through the tutorial line by line, as you might as well just read the original. Instead I’ll just intend to highlight some points where I had problems, and what I had to do to overcome them.

The first thing I’d note in hindsight is that the tutorial is not actually “how to build the simplest possible web data connector” as I had imagined it would be from its description as “basic”. It surely is a relatively simple one (no authentication needed etc.) but it does contain at least 3 functions that are only necessary because of the specific API they were connecting to. These are:

  • buildUri
  • getFormattedDate
  • makeTwoDigits

Pretty obvious when you see what they do – but it would be a mistake to imagine you should just copy the function structure of the basic connector and replace it with your own code. You won’t need the above 3 functions for the most part. I felt the tutorial could make that clearer, especially for someone even more novice than I, who had never touched web programming (I’ve done some in the distant past).

But anyway, begin at the beginning. I raced happily through parts 1 and 2 of the tutorial, copying and pasting their base code. Nothing important to change there unless you want to give your version a more appropriate title.

Part 3 instructed me how to create a form for the user to enter information about their data on. Again, I can imagine some data connectors won’t need to prompt the analyst using it for any information, in which case you could also disregard this section.  However I did actually need that feature, as I wanted it to let me enter a BoardGameGeek username in and only retrieve board games that person had in their collection.

In the mean time, I had discovered – joyfully – that there’s a hosted Tableau Web Data Connector simulator on Github, thank you so much! The reason it made me happy is that it meant that I didn’t need to install a web server on my own computer, or do anything with the SDK that might have involved needing to beg for admin rights over my laptop.

Once I developed a connector in Notepad, I could just upload it somewhere (I chose Github Pages myself) and use the hosted simulator to see if it works – which is the reason you don’t actually need a copy of Tableau in the first place.  I’m sure it would have been far quicker to iterate if I had installed it locally, but it works just fine in a hosted version – as long as you beware the cache.

Lesson learned: It seems like there’s a lot of caching going on somewhere between or within the hosted Tableau Web Data Connector simulator and my web host (Github). Oftentimes I’d make changes to the code, reupload, but it didn’t seem to change the result as shown in the simulator. Sometimes closing browsers, deleting cookies etc. helped – but not always. I’m sure there is some science to this that one could work out, but given I didn’t need to make a lot of edits in developing a relatively simple data connector, I basically settled for renaming my file “version 1.html”, “version 2.html” etc. whenever I made a modification, which guaranteed a lack of annoying caching action.

But anyway, defining the user interface per the tutorial went smoothly. I entered the web address where I’d uploaded my connector: “amedcalf.github.io/BGGConnector.html”, unticked the “automatically continue to data phase” box (as the data phase is something I defined later, in line with the tutorial), and ran the “interactive phase”.

Up popped my bare-bones form asking for a BoardGameGeek username, I filled it in and pressed OK, and it took me back to the simulator and showed me a form that indicated that it had indeed recognised the information I gave it. Success!

Capture

No data was returned just yet – but it clearly recognised what I had entered into the form just fine.

Lesson learned: Although this wasn’t a problem for me in this stage, I later noticed that if there’s anything wrong in your code then you do not always get an error message per se. Sometimes the simulator just doesn’t close your UI, or it does but doesn’t return you any information – i.e. acts like a faulty web page. In my experience that tends to be when I’d made a minor mistake in my code – a missing semicolon or bracket or similar. Once that was fixed, the simulator worked nicely again (if you want to go wild and use something other than notepad that does syntax highlighting, you’ll probably not suffer this much.) So you should assume that it’s your code, not the simulator, that is broken in such instances 🙂

It became clear the user interface is defined in normal HTML – so if you are a nifty web designer/coder, or want to learn to be one, you can make it as beautiful as you like. That wasn’t the point of the exercise for me though, so I just made a plain default font and default box to type information into for my effort.

So, that was the user interface “designed” and tested, which takes us up to part 4 of the Tableau tutorial.

Next up: time to actually access the data I wanted. This proved to be a little more difficult first time through though.

Journey complete: a BoardGameGeek collection Tableau web data connector

Several months (!) after setting myself the challenge of becoming familiar with how to create a Tableau Web Data Connector, the good news is…it’s done! Well, done enough that I now understand how to create web data connectors should I ever need one in reality anyway.

As planned, its a connection via the BoardGameGeek API to your very own board game collection, if you have catalogued one at that site. For the uninitiated, BoardGameGeek is an amazing site if you’re the sort of person that likes boardgames, and, yes, cataloguing.

So, for starters, if you have indeed got a collection catalogued, or want to analyse someone else’s collection, then you can now point your Tableau Desktop software to a web connector at http://amedcalf.github.io/BGGConnector.html

 

Important / annoying note (explained below): BEFORE you do this, go and visit the below link in your web browser to make sure your BGG collection is not going to be stuck in an API queue.

https://boardgamegeek.com/xmlapi2/collection?username=Adam121

replacing the “Adam121” with the username you’re interested in. If you get a message about queues, caches or other errors, wait a few seconds and try again, until you end up with some unpleasant looking XML code that looks something like a whole string of this.

Capture

 Yes, this is not user friendly and would not be acceptable in a “production” environment, but this is just for fun; I’ll explain more below.

Once you’ve done that, go back into Tableau.

Once you have entered http://amedcalf.github.io/BGGConnector.html into the web data connection datasource part of Tableau (see Tableau’s instructions here), you should get a basic prompt to enter your BoardGameGeek username. You can test it with “Adam121” if you like, without the quotes. Do that, and up should pop the relevant collection ready for usage in Tableau.

Here is a description of the fields you’ll be presented with:

  • Boardgame Name: Textual name of your board game
  • Image Thumbnail URL: text URL of a small image relating to the boardgame, usually its cover.
  • Image URL: same as above, but a big version.
  • Last Modified: date the record was last changed.
  • Year Published: the year the game was published.
  • Plays: how many times you logged that you played this game.

Then there’s a series of flags that are set to either 0 (false) or 1 (true). These are:

  • For Trade
  • Owned
  • Preordered
  • Previously Owned
  • Want
  • Want To Buy
  • Want To Play
  • Wishlist

and are all attributes you can input yourself per game with the BGG collections feature.

Whilst it is based on the BoardGameGeek API, I should also to give maximum thanks to the BoardGameGeek user “Strathmeyer”, whose past repurposing of the BGG API into CORS-enabled JSON is what I ended up using as my source, rather than the API directly. When I come to document my journey, I’ll go through why.

One thing to note: if you try it on a username and it doesn’t work (perhaps giving a “typerror”) then this is probably down to the caching of the API on the server. BoardGameGeek queues API calls for collections that haven’t been requested in a while, and the JSON source seems to cache the “in a queue” response meaning that no data appears.

You can see if this is why you’re having problems if you go to this link, putting the BoardGameGeek user ID you’re after after the last equals sign of the link:

http://bgg-api.herokuapp.com/api/v1/collection?username=YOURUSERNAME

If it returns the message “Your request for this collection has been accepted and will be processed. Please try again later for access” then you’re in the queue. And it seems to cache the fact you’re in the queue, which is why you should go visit https://boardgamegeek.com/xmlapi2/collection?username=YOURUSERNAME before you do anything else, even visit the herokuapp link.

I haven’t taken the time to come up with a nice solution for this, but this is what worked for me. Obviously it’s a very unsuitable requirement should this be the mainstay of someone’s production system, but I’m afraid I was just playing about to see what works 🙂

So what can you do with this, except poke around for fun?

Well, ever since the great-yet-terrible event that was the downsizing of my infeasibly oversized board games collection, I don’t really have more than would fit on a single screen of BGG to be honest, and I can’t usually find many people to play them with me anyway (insert sympathy sound effect). But if you had a bigger collection, you might find it useful to answer such questions as “How many board games do I own?”, “Which board game have I played the most?”, “How many board games do I want to buy?” and so on. If so, please enjoy!

Just to prove to myself that it works, here’s an example very simple, quite pointless, dashboard that displays my collection and allows you to click on the name of a game in it, and see what the box looks like. Feel free to click through and play with it.

Capture

What’s next? Well, the point of the exercise was not really to find a way to list the remnants of my board games collection – but rather to learn something new, and document something of the journey I went through doing so, warts and all. With that in mind, I’ll be writing a few notes as to the trials and tribulations I went through in creating this, for future reference if nothing else.

 

The journey towards creating a Tableau Web Data Connector

I recently watched a video by Tableau/Alteryx/general guru Chris Love in which he emphasised the importance for people to not only retrospectively present their mindblowing successes at cajoling Tableau into running the universe – but also the journey leading up to that point, including any outright failures. After all, there is no shame in failing to do what one sets out to achieve; instead, what would actually be detrimental would be to refuse to acknowledge and learn from failures.

With that in mind, I considered what features of Tableau I was not at all familiar with, might be of future benefit to me, and are likely complex enough to require a “journey” rather than a double-click.

I settled on web data connectors. This a feature I know of, but never encountered either as an author or a user. I don’t have a desperate need for them right now in my day-to-day activities, but I can imagine many useful scenarios and have at least some intellectual curiosity about how they work. So here goes with post #1 on “how do I learn how to make one?”.

What do I think a Tableau web data connector is?

I was lucky enough to go to the Tableau Conference in 2014, where (I think) the concept was announced. From that, my current belief is that a web data connector is a feature of Tableau whereby you can have it directly access structured data from other websites that have a usable API.

Fitness is a common example usecase. The likes of Fitbit.com record and analyse your step count for instance, and show you a few nice charts – but they also offer an API through which you can export your data out of their website and into your analytical software to do your own custom analysis, perhaps using different techniques or adding more external data to get further insight from all the juicy information they store on ones’ activity (or lack of).

A web data connector for a site then would let you access whatever data it makes available to you in a way suitable for immediate consumption in Tableau, avoiding the need to manually export and rebuild it into a local database and so on before analysis. Hopefully it will also allow your Tableau analysis to update when new info is in the datasource.

What do I think I probably need to create a Tableau web data connector?

Web data connectors were a feature of Tableau 9.1, which I already have installed.

From my memory of the Tableau conference, I believe that web data connectors are written in Javascript, so a good working knowledge of that language is probably useful. I can’t say I have this…many years ago I did dabble in a little Javascript as it happens, as I wanted to make a matched betting calculator for a website I had to enable people to make lots of free money from bookies (a topic for another day, but yes, there is a genuine opportunity to make lots of $$$ sitting in front of the internet every day, although that sentence couldn’t sound any scammier I know!).

However that was a long time ago and I can’t really remember anything about it. Except that it’s basically a case of writing code in a text file, and I already have Notepad++ installed which I know supports syntax highlighting of Javascript – so that’s probably an acceptable tool to help me write it in.

I also know Tableau has famously good help pages on their website, so am pretty confident I can look there to learn how to do it.

I also know that many people in the Tableau community, most notably probably the Information Lab, have kindly released examples of web data connectors that one can perhaps dissect to see what it’s all about.

What API shall I try and connect to?

My criteria in picking an API that may or may not work with this were that it should:

  • support XML or JSON, which I guess are the formats that Tableau web connectors most likely use.
  • be free to access; I’m feeling no urge to invest financially for a first attempt.
  • not require logging into with a username/password – as a) I suspect logging into an API needing authentication adds complexity, and b) should I develop something that vaguely works, I don’t really want to be responsible for anything “security-wise” for my first attempt.
  • be of a subject matter I find interesting, as I have no immediate need in my career or life to do this, so it’ll have to be vaguely mentally stimulating in order for me to bother having a go at it for any length of time.

So, the winner, for now anyway, is the BoardGamesGeek API.

For the non-initiated, BoardGameGeek is a website visited by, well, the clue is in the name, and I’m afraid I am rather the target audience – although I do my level best these days to keep my habit under control. Gone are the days when the box collection was so big that it had to be stored in a cardboard mountain on the living room floor as no shelving storage system could realistically hold it. I remain, however, extraordinarily interested in the topic, even having got rid of most of them (at least as long as you don’t poke around my phone too much…believe it or not you can have a passable game of Agricola et al on an iPhone).

Back to the subject at hand: one feature of BoardGameGeek is that users can record a list of which boardgames they own. This is publicly accessible info on the normal web; see my now-minuscule collection now here.

They also offer this information via an API, which is documented here. Here for instance is the API call needed to return the info in the above boardgamegeek collection in XML: http://boardgamegeek.com/xmlapi2/collection?username=Adam121

Sometimes it takes a couple of tries to return actual data, but when it does, you get something starting like this:

Capture

Quite clearly there’s some potentially interesting data available here, but as it stands, it’s not going to slot into Tableau nicely for analysis at all without some preprocessing by other tools first. But who wants to have to load TWO analytics programs when one will do?

Therefore, I would like to build a web data connector to allow Tableau to connect to it directly nice and live.

Next steps I have in mind

  1. Try using someone else’s web data connector so I can see what they look like in practice.
  2. Go on Tableau’s site and see what instructions are available about building them.
  3. ????
  4. PROFIT! (yes, just like in South Park).