Hands-on : navigating graphs with midi control surfaces

Using a mouse to get around a graph like HiveMindMap can be a bit fiddly. The music industry is way ahead of us in using hands-on hardware to control things and here I (ab)use a midi music controller with dials and sliders to zip around the map. I'm using a new feature in HTML5 which allows the browser to respond to MIDI messages coming from the hardware.
While a piece of hardware designed for mixing music is perhaps not the ideal tool,  spinning a dial feels like a pretty effective way of selecting lines (or "edges" in graph lingo) and beats trying to hover over lines with a mouse. I would expect future keyboards to come equipped with fancy dials and sliders that can send MIDI messages for use in web browsers in all sorts of inventive ways.

FutureTech: navigating the HiveMindMap with a midi control surface from mark h on Vimeo.

WebGL - not ready for prime time?

So one of the visualizations I wanted to add to the HiveMindMap website was a 3D map of the globe using the software here.

The good news is the images are quite interesting and respond very quickly to mouse and keyboard events:

The bad news is :

1. A limited set of browsers support WebGL
2. Those that do seem unstable

Using the latest versions of Chrome and Safari on OSX I have found that browser windows will occasionally burst into periods of rapid flashing that look like they could induce epileptic fits. Also the screen will occasionally lock up a machine with the dreaded spinning beach ball for long periods. This happened on both desktop and laptop machines.

For these reasons I have opted to stick with a Google Earth KML file for this sort of content until WebGL support improves.

Mapping language use

There have been several maps published lately that use Twitter data to show what languages are in use where.  Google Earth lends itself nicely to this job for several reasons:

1. Tweet volumes in an area can be shown by 3D towers - this helps give a better sense of scale than a flat picture (and I'm normally not a fan of 3D)
2. Each language can be hidden or shown using the Google Earth checkboxes that control layer visibility. If you want to focus in on just French or remove all English, just check the appropriate boxes.

A happy side-effect of planting several "towers" representing different languages in the same spot is that Google Earth shows the most dominant language most clearly.

Here's a screenshot:

If you want to explore the data yourself the Google Earth file is here 

HTML5 and interactive graphs

The HiveMindMap project  uses HTML5 technology to present a zoomable diagram of hashtag connections. The promise of HTML5 is that it should be possible to write a web app once and have it run on many devices.
However, the reality is that while modern desktop browsers (Firefox, Safari, Chrome, Opera and IE9+) seem to work well the iPad has been particularly problematic. The performance gap between desktop/laptop machines and tablets proves to be quite large. Using pinch-to-zoom or swipe-to-drag on a first generation iPad is painful as there is a delay that is measured in seconds. There are various tweaks that I have tried along the way to try and up the speed:

Swapping SVG for Canvas

The current implementation of the visualisation is using SVG and the other major contender in the HTML5 stable for graphics is to use the Canvas API. Using the KineticJS library I was able to quickly recreate the basic graph drawing capabilities with zoom and pan behaviour but the rendering was not noticeably any better.

Hardware acceleration

Upping the rendering speed should in theory be possible if I can use a GPU to perform much of the heavy lifting.  One approach advocated here is to add CSS3 styles that suggest the use of hardware acceleration:
 -webkit-transform: translateZ(0);
This may have marginally speeded up manipulation on the iPhone but a visible form of tiling occurred where the screen was briefly divided into sections containing pre-pan images and post-pan images before the repaint catches up and all sections then render from the same scene.
Worse, on the desktop Chrome browser a pop-up div that is overlaid over the hardware-accelerated image was visibly blurred for some reason, making the text unreadable. I chose to turn this off.

One other CSS3 technique I trialled was to avoid use of the SVG transform functions to zoom and position the image and instead controlled zooms and pans by setting CSS3 translate styles. Surprisingly this would zoom the image without introducing pixelation - I had assumed that it would treat the rendered SVG vector as a raster in the re-scaling and introduce pixelation artefacts with the zoom. Unfortunately it was not noticeably faster than pure SVG based transforms so I reverted back to using those.

Limiting repaints

The main enhancement to help support low-performance devices was to limit the number of repaints generated by swipe or pinch gestures.  While a desktop browser can happily respond to every mouse event the touch gestures on a tablet or phone generates events at a rate faster than can be repainted.

SVG issues on iOS

Aside from the performance issues of the iPad/iPhone I have also found that the iOS devices differ from all other browsers in the calculation of coordinates (including OSX Safari). The calculations used to centre an SVG element on the screen are miscalculated on iOS and elements are positioned off-screen rather than centred.

Conclusion

Overall, I suspect it is not important to cater for smart phones as the limited display space is not conducive to large-scale visualisations like this mindmap. The iPad however would be a nice platform to cater for and it is frustrating that the performance lags when using HTML5 technologies. Newer versions of the iPad have faster processors and seem better but ultimately a native app may be the way to go.