Learning Analytics – an overview Thursday, 7 February 2013

By Ralph Kiel,  2012

This document is also available for download.

1.         Introduction

In the last year or so Learning Analytics has been raised as a topic at CAUL meetings on a number of occasions and has been identified as a topic of interest by the CAUL Learning and Teaching Advisory Committee. Both EDUCAUSE in its Top-Ten IT Issues: 2012 and NMC along with Griffith University in the Technology Outlook: Australian Tertiary Education 2012-2017 have identified Learning Analytics as being one of the major issues to watch, the former at number 6 and the latter number 3.

The aim of this brief report is to give as simple and clear an overview of this complex issue as possible and to try to identify ways in which learning analytics are of use  to Australian University Libraries. For beginners a good place to start is the EDUCAUSE paper: 7 things you should know about analytics which can be found on their website at:http://net.educause.edu/ir/library/pdf/ELI7059.pdf. This paper defines analytics in a higher education context and identifies how it works, why it is significant, the implications to teaching and learning, legal and ethical issues and the future of analytics in higher education.

van Barnegeld et al state that there is confusion in the meaning and understanding of what analytics means especially in higher education - “the analytics field is now at a point where clarification and consensus of terms is merited” and in their paper set out a conceptual framework to establish a common terminology and application in the field of analytics in the academy” (van Barnegeld 2012).

EDUCAUSE has a working definition for analytics: “Analytics is the use of data, statistical analysis, and explanatory and predictive models to gain insights and act on complex issues”. (Bichsel 2012). Bichsel then goes onto say that analytics is a process that is more than just metrics. This process starts with a strategic question, finds and collects the data to answer that question, analyses the data “with an eye towards prediction and insight”, presents the findings and feeds back into the process of addressing the strategic question and creating new ones. She makes it clear that when it comes to analytics, things have changed in recent years as data are more extensive and automated and the process used to get the information is becoming more repeatable. Also, a broader range of programs have been established to apply the data, analyse it and then make decisions and develop plans.

In the next section some of the benefits of learning analytics are laid out. But are the costs in using them worth the effort? The benefits of using analytics can be costed as described in Campbell and Oblinger: “Retention of students saves institutions the cost of recruiting students to replace those who withdraw without completing a degree.” (Campbell and Oblinger, 2007)

2.         Learning / Academic Analytics

Universities in this competitive and accountable age are “ . . adopting practices to ensure organizational success at all levels by addressing questions about retention, admissions, fundraising and operational efficiency.”  (van Barneveld 2012). Learning analytical tools can provide statistical evaluation of institutional data sources to determine patterns that can inform decisions about student recruitment, course offerings etc. In learning and teaching, Learning Management System (LMS) data can be used to identify students at risk of not succeeding academically. Data can also be mined to show student interaction with peers, course content, and teachers, and then measuring and draw conclusions about these interactions.

From this, reports can be developed that reveal patterns, trends and exceptions. Analytics can indicate course strengths and weaknesses, align resources with student need and track student progress. Algorithms which perform calculations to determine a particular outcome can be used in this analysis and can also set off a trigger sending a note to a teacher indicating a student is at risk of failure. In this case the goal would be to improve student performance and to show the accountability of the University.

Legal and ethical issues around privacy, security and ownership involving students and teachers have been raised especially whether such profiling is intrusive. However it could be argued that not identifying such knowledge of student underachieving could be regarded as irresponsible when the teacher or the University has this data at hand. Two other potential shortcomings could be that algorithms may come to false conclusions and send the wrong message about a student, or that a student or teacher could misinterpret what the data is telling them and come to false conclusions. However in both these cases, as Universities get better at this activity these potential problems are likely to be ironed out and learning analytics could become standard practice.

van Barneveld et al set out two types of analytics for Universities within an overall framework based on the Scholarship of Teaching and Learning (SoTL): academic analytics and learning analytics, and state that analytics can “supplement the established theory and practice of the field.” (van barneveld 2012)

The Horizon Report defines learning analytics as follows:
 
“Learning analytics loosely joins a variety of data gathering tools and analytic techniques to study student engagement, performance, and progress in practice, with the goal of using what is learned to revise curricula, teaching, and assessment in real time. Building on the kinds of information generated by Google Analytics and other similar tools, learning analytics aims to mobilize the power of data-mining tools in the service of learning, and embrace the complexity, diversity, and abundance of information that dynamic learning environments can generate.”

The report goes onto say that learning analytics has generated a lot of attention over the last few years and is already enjoying growing interest within higher education but that there is a two to three year horizon over which they expect to see the beginning of widespread adoptions of learning analytics. The benefit of learning analytics is that it will enable teachers and universities to tailor educational opportunities to each student’s level of need.

The Horizon report goes onto say that  
“Data are collected from explicit student actions, such as completing assignments and taking exams, and from tacit actions, including online social interactions, extracurricular activities, posts on discussion forums, and other activities that are not directly assessed as part of the student’s educational progress. …. Learning analytics promises to harness the power of advances in data mining, interpretation, and modeling to improve understandings of teaching and learning, and to tailor education to individual students more effectively.” (Horizon Report 2012)

The bottom line is that academic analytics is all about improving “... one of higher education’s most important challenges: student success” especially measured through student retention and graduation. Obtaining data from sources such as a LMS and student information systems and by using predictive modelling to identify students at risk and then intervening, analytical tools can be used to improve student retention and graduation. (Campbell and Oblinger 2007)

This simple diagram from an article by Siemens depicts the process of learning analytics; however a more complex diagram is also presented in this article. (Siemens, 2010)
 


Learning analytics is related to similar developments in other areas such as Big Data, e-science, web analytics, and educational data mining, with the main, common element being they rely on large data collections. In all of the above a tool can be used to help make sense of the data called a dashboard or in this case a ‘learner dashboard’. “For learners and teachers alike, it can be extremely useful to have a visual overview of their activities and how they relate to those of their peers or other actors in the learning experience.” (Duval)
 

3. Analytics and Australian Universities

Social Networks Adapting Pedagogical Practices (SNAPP) tool developed by the University of Wollongong which is highlighted in the 2012 Horizon Report uses visual analysis to show how students interact with discussion forums within commercial and open source Learning Management Systems. As a diagnostic it allows teachers to evaluate student patterns of behaviour against learning objectives. Interaction data that is stored within the discussion forum is interpreted by SNAPP by inferring relationships from post reply data and develops a social network visualisation.
 
“The social network diagrams can be used to identify:
 
1.      isolated students
2.      facilitator-centric network patterns where a tutor or academic is central to the network with little interaction occurring between student participants
3.      group malfunction
4.      users that bridge smaller clustered networks and serve as information brokers”
(SNAPP 2012)

At Victoria University the use of data in this way is at an embryonic stage. The annual course monitoring process uses a limited range of data for trend analysis and faculties are able to use this data for their own local analysis. However the use of this data is only partly automated. A project called ‘Flag and Follow’ manually tapped into enrolment data to determine which students were at risk before classes commenced in 2010. The aim is to develop a university wide inter-departmental model that is evidence based to inform retention initiatives. (Brown 2011?) A number of ‘at risk’ factors such as ATAR score, socio-economic status and distance to campus were used to ‘flag’ students at potential risk and then these students were ‘followed’ and interventions developed to support their learning. At risk students were also identified once classes had started by criteria such as absences from class and failure to hand in the first assignment. Preliminary findings to date suggest the flagging criteria were successful and that interventions especially in person as opposed to SMS or telephone were successful.
 

4.         Analytics and University Libraries

Libraries produce a lot of data through Library Management systems. However results of a survey conducted by EDUCAUSE, showed that although Libraries are regarded as one of the top 17 areas in which data is being collected, they are one of the most laggard areas in using this data to make predictions or take action. In the same survey it also showed that Libraries were regarded as having one of the lower levels of potential in using the data and being able to make predictions and take action. (Bichsel 2012)

By having the Library system talk to the University systems, the University of Wollongong Library with its Library Cube project has used analytics to demonstrate the contribution the Library is making to student learning and teaching. A correlation between Library loans gathered from data in the Library Management system and student achievement in grades from the University systems was shown. That is, the more a student borrows library books the more likely the student is to achieve at a high level in academic scores. (Jantti 2011)

This is groundbreaking research and it was cited in an ECAL publication as being a distinctive example of the use of analytics in an area outside of student recruitment and retention. (Bichsel 2012) The case could be put that in Australia at least where University Libraries operate in consistent way and where the measures of library performance are relatively close, that this outcome would hold for all CAUL Libraries. This research is important for CAUL Libraries because it can be used to provide accountability, can be used in marketing of library services to the rest of the University and can be compared overtime as a quality measure.

References

7 things you should know about analytics, EDUCAUSE, 2012 downloaded 12 April 2012 http://net.educause.edu/ir/library/pdf/ELI7059.pdf

van Barneveld, Angela, Arnold, Kimberly E and Campbell, John P. Analytics in Higher Education: Establishing a Common Language, EDUCAUSE, 2012, downloaded 13 April 2012http://www.educause.edu/Resources/AnalyticsinHigherEducationEsta/245405

Bichsel, Jacqueline. Analytics in Higher Education: Benefits, Barriers, Progress and Recommendations (Research Report). Louisville, CO: EDUCAUSE Center for Applied Research, August 2012, downloaded 29 August 2012
http://www.educause.edu/library/resources/2012-ecar-study-analytics-higher-education

Cox, Brian and Jantti,Margie. Capturing Business Intelligence Required for Targeted Marketing, Demonstrating Value, and Driving Process Improvement, 2012, downloaded 13 April 2012
http://ro.uow.edu.au/cgi/viewcontent.cgi?article=1248&context=asdpapers&sei-redir=1&referer=http%3A%2F%2Fwww.google.com.au%2Fsearch%3Fsourceid%3Dnavclient%26ie%3DUTF-8%26rlz%3D1T4SKPB_enAU383AU384%26q%3Dlibrary%2Bcube%2Bwollongong#search=%22library%20cube%20wollongong%22

Jantti, Margie H and Cox,Brian. Measuring the value of library resources and student academic performance through relational datasets, 2011, downloaded from the Internet 13 April 2012
http://ro.uow.edu.au/cgi/viewcontent.cgi?article=1120&context=asdpapers&sei-redir=1&referer=http%3A%2F%2Fwww.google.com.au%2Fsearch%3Fsourceid%3Dnavclient%26ie%3DUTF-8%26rlz%3D1T4SKPB_enAU383AU384%26q%3Dlibrary%2Bcube%2Bwollongong#search=%22library%20cube%20wollongong%22

Brown, Claire and Evagelistis, Yolanda. Flag and Follow: a whole of institution approach to early identification and intervention for students potentially at risk, 2011?, downloaded from the Internet 26 April 2012
http://www.fyhe.com.au/past_papers/papers11/FYHE-2011/content/pdf/7A.pdf

Campbell, John P.  and Oblinger, Diana G. Academic Analytics, EDUCAUSE, October 2007, downloaded from the Internet 9 May 2012 http://net.educause.edu/ir/library/pdf/PUB6101.pdf

Duval, Erik. Attention Please! Learning Analytics for Visualization and Recommendation, LAK’11:
Proceedings of the 1st International Conference on Learning Analytics and Knowledge, 2011

Horizon Report: 2012 Higher Education Edition, NMC, 2012. downloaded 15 May 2012
http://net.educause.edu/ir/library/pdf/HR2012.pdf
Grajek, Susan, the 2011–2012 EDUCAUSE IT Issues Panel, and Judith A. Pirani. Top-Ten IT Issues, 2012, EDUCAUSE.

http://www.educause.edu/EDUCAUSE+Review/EDUCAUSEReviewMagazineVolume47/TopTenITIssues2012/250724

The New Media Consortium (NMC) and Griffith University, 2012, Technology Outlook: Australian Tertiary Education 2012-2017
http://www.nmc.org/publications/2012-technology-outlook-au

SNAPP: Social Networks Adapting Pedagogical Practice, downloaded 15 May 2012
http://www.snappvis.org/

Siemens, George, What are Learning Analytics? eLearnspace, 25 August 2010. downloaded 15 May 2012
http://www.elearnspace.org/blog/2010/08/25/what-are-learning-analytics/