Which events are significant?

Which events are significant?
Topic
This guide is about compiling (in tabular form) a compendium of activity data to assist in planning what recommenders may be possible in a given institution or group of institutions that share activity data.
The problem
E H Carr famously explored the difference between ‘facts of the past’ and significant ‘historical facts’ (What is History? 1961), comparing cake burnings in general with royalty-related cake burnings. We are faced with a comparable challenge of judgement in determining what to select from the mass of data arising from the use of computer systems. Given an area of interest (such as student success or utilisation of subscribed resources), the problem of selecting significant facts or ‘events’ can be split in to two along the lines suggested by Carr:
  • What exists? What event data are we collecting or what could we collect?
  • What is significant? What event data is worth preserving?
Before we can address those questions, it may help to have a sense of the possibilities. Put simply, event data records any user action (online or in the physical world) that can be logged on a computer, ideally containing a reliable form of user identification. We can usefully think of it in three categories:
  • Access – logs of user access to systems indicating where users have travelled (e.g. log in / log out, passing through routers and other network devices, premises access turnstiles)
  • Attention – navigation of applications indicating where users have been are paying attention (e.g. page impressions, menu choices, searches)
  • Activity – ‘real activity’, records of transactions which indicate strong interest and intent (e.g. purchases, event bookings, lecture attendance, book loans, downloads, ratings)
The solution
Given this definition, you may already be concluding that you do in fact have a lot of data in your systems, not all of which is likely to tell useful stories. The first step is therefore to determine the types of event data that may be of relevance. Perhaps start with a checklist along these lines:
Problem space – Analysis of learning resource usage by undergraduate students
Event Category
Event
Logged by
Logged now?
User ID
Annual Volume
Value for my purposes
Access
Student logs on to VLE
VLE
Yes
Standard University ID
2.2m
Possibly just noise
Access
Student logs on to cash card top up
Cash Card
Yes
Standard University ID
410k
Out of scope
Attention
Student loads module resource page
VLE
Yes
Standard University ID
1.4m
YES
Activity
Student downloads local learning resource
VLE
Not sure
Standard University ID
360k
YES
Activity
Someone downloads a JORUM resource
Jorum
Yes
University IP address
5k
YES
Activity
Student borrows a book
LMS
Yes
LMS Borrower ID
310k
YES but ID needs mapping
Taking it further
It may be useful to undertake a more general exercise with your ‘team’. Use a similar table (e.g. the first six columns). Don't start with the problem to be addressed but rather work together to compile the list of every example of event data that exists, could or should exist. One you have the list, you can add extra ‘value’ columns for different analytical purposes – for example a VLE team might add columns for such as student success, local content utilisation, usage patterns by faculty, VLE performance optimisation. You can then do a second pass assessing the significance of each event for each purpose – perhaps initially in a scale of yes / possibly / no.
Additional resources
The LIDP project considered a range of library ‘events’ - http://library.hud.ac.uk/blogs/projects/lidp/about/ and http://www.slideshare.net/gregynog/bw-dave-pattern-lidp
The EVAD project investigated to range of event logging in the Sakai VLE - http://vledata.blogspot.com/
The JISC Business Intelligence projects have taken a wider sweep of what events might be useful to derive business intelligence - http://www.jisc.ac.uk/whatwedo/programmes/businessintelligence/
Educause reports a survey of the data being used for ‘academic analytics’ - http://www.educause.edu/ers0508