Assessment and credentialisation are central to the future of MOOCs and, more broadly, of open learning. The Open Education Studio raises a global conversation on a subject that is crucial to achieving the objective of opening up education to all. Many students - especially in the developing world - are committed to learning and looking for qualifications that will boost their professional careers and change their lives. What credentials can they obtain from open learning, and based on what kind of assessment? Will it be compatible with the principles that guide open learning?
Our conversation starts with an adapted version of an article(1) by Gabi Witthaus, Mark Childs, Bernard Nkuyubwatsi, Grainne Conole, Andreia Inamorato dos Santos and Yves Punie that shares the lessons learned during the OpenCred study of the Institute for Prospective Technological Studies. The OpenCred final report is available here.
Open Educational Resources (OER) and Massive Open Online Courses (MOOCs) have emerged in recent years and are triggering a mind-set change in institutions. Hundreds of MOOCs are now being offered by European institutions, yet insufficient research has been conducted regarding the degree to which formal credentialisation of learning is important to learners. While badging may prove motivating for some learners, formal certification may be the main goal for others. Certificates for open learning achievements vary in terms of their level of formality of recognition, depending largely on how they are linked to assessment. The OpenCred study looks into emerging practices around the issuing of certificates for open learning in Europe, and their relationship to assessment.
2. MOOCs and the issuing of certificates
Credentialisation of open learning is not just an add-on to the established recognition procedures of prior learning(2) ; it requires a substantial shift in mind-set, particularly from educational institutions, where traditionally the roles of teaching, content provision, assessment and accreditation have all been bundled together(3) .
The Trends Report by the Open Education Special Interest Group in the Netherlands (4) looks at the aspects that would need to be considered in order to recognise learning achievements in open, non-formal education:
1. Is the certificate merely proof of attendance or does it provide evidence of learning? If the latter, how robust was the assessment? (Multiple-choice questions with automated marking at one end of the range; the completion of an examination under supervision at the other).
2. To what degree is the student’s identity validated, and how much supervision is provided?
The report identifies four levels for these two intertwined elements:
a) No validation of identity – the MOOC relies on the honour of the student;
b) Online validation by facial recognition or keystroke tracking;
c) Online monitoring, which requires a moderator/ proctor to have a 360-degree view of the student’s room transmitted via a webcam. The Trends report notes that some institutions would not accept online proctoring as a qualifying examination environment, regarding it as being prone to fraud, although it is increasingly being seen as legitimate;
d) Attendance at a physical examination site.
Similarly, a report published by the Norwegian Ministry of Education(5) , proposes that for the awarding of formal academic credit, proof of learning will need to be demonstrated via examination, and the importance of validation of the identity of the examinee is stressed. The report lists the following situations in which validation of identity is required:
• A student wants to transfer credits obtained in a MOOC conducted by a foreign provider to a degree at a local institution;
• A student wants their achievements in a MOOC studied with a foreign provider to be validated as part of the admission process to higher education in a local institution;
• A local institution offers a MOOC and awards credits for successful completion of assessment tasks;
• An employee wants to include their participation in a MOOC in their documentation of competence when applying for a job.
These reports underline the importance of linking the means of assessment with the nature of recognition awarded to learners.
3. The OpenCred Study
This study was based on publicly available information from open education websites, working groups, projects and studies across Europe. The aim was to identify any general principles regarding recognition of open learning that could inform discussions in the field and support developers and learners, by clarifying the range of options and models for credentialisation and recognition of open learning that existed and might be replicated.
During the study, it became apparent that the concept of ‘recognition’ was rather blurred, and that it can mean different things to different people. In some contexts, the term ‘recognition’ is used to refer to credentialisation by the MOOC provider, and in other contexts ‘recognition’ is the process that takes place when a different institution (or an employer) formally recognises/ validates the learning done in the MOOC. In the OpenCred report, we used the term 'credentialisation' to refer to certification, badging or awarding of credits to learners upon successful completion of MOOC assessment, and 'recognition' to refer to the later process of validation of those credentials. This paper focuses on the concept of credentialisation and its relationship to assessment.
The initiatives identified ranged considerably in the degree to which open learning was formally credentialised, with some offering no credentials at all, or simply a completion certificate or badge, and others offering exemption from examinations or courses, or ECTS credits. They also varied enormously in relation to the robustness of assessment, both in terms of the nature of the assessment, and in the degree to which the learner’s identity was verified.
A proposed hierarchy of descriptors for the formality of credentialisation of open learning in European open education initiatives is shown in Table 1. A value has been provided for each descriptor, and these values will be used to organise the information about institutional open education initiatives in the following section.
Table 1: Formality of credentialisation
|0||No credentials awarded|
|1||Unauthenticated completion certificate/statement of accomplishment or badge showing proof of participation or completion(6)|
|2||Authenticated certificate or badge which either (a) contains limited/no information on the nature of the course, the nature of the learner’s achievement and the nature of the assessment process used, or (b) indicates that the learner’s identity was verified online but there was no supervision during assessment (as is typical in Coursera MOOCs with Signature Track)|
Certificate providing exemption from a specified entrance exam
Certificate conferring between 1 and 4 ECTS credits
Certificate conferring a minimum of 5 ECTS credits
Certificate providing exemption from a specified module/course or part of qualification at the issuing institution
Certificate from an accredited institution which ‘(a) formally and clearly states on whose authority it was issued, provides information on the content, level and study load, states that the holder has achieved the desired learning objectives, provides information on the testing methods employed and lists the credits obtained, according to a standard international system or in some other acceptable format, (b) is demonstrably and clearly based on authentication [i.e. student’s identity is verified] and (c) states that the examinations have been administered under supervision and specifies the nature of this supervision.’
Continuing professional development (CPD) credits
Table 2: Robustness of assessment
Record of completion of activities
Assessment with automated checking, e.g. multiple-choice questions (MCQs), submission of programming code, or acceptance of a submission of text on the basis of word count (No verification of identity)
Peer assessment (No verification of identity)
|2||Online examination with verified identity and no real-time supervision, e.g. Coursera’s Signature Track or Accredible’s ‘self-proctoring’ (in which a recording is made of the student’s screen and face while examination is in progress, and is compressed into a 2-minute time-lapse video, embedded in certificate)|
Submission of coursework and/or performance of practical tasks where the student is personally known to the examiner. (The context may be either face-to-face or online. The assumption is that inconsistencies in performance style will be picked up and this minimises the likelihood of cheating. This is common practice in traditional online courses, e.g. online MBA programmes)
Online examination with identity verification and real-time proctoring (e.g. ProctorU , Proctor2Me or Remote Proctor, which has a panel of proctors check individual examination recordings)
On-site examination (including on-site challenge exams)
Recognition of prior learning (RPL) conducted by recognised expert(s) (e.g. based on portfolio submission and/or interview – requires a relatively low candidate-to-assessor ratio and hence generally not scalable to open initiatives)
The open education initiatives discussed below were selected from those included in the European-wide desk research undertaken for the OpenCred study. The initiatives are organised according to formality of credentialisation, as per Table 1, starting with Level 1.
Examples of Level 1 credentialisation initiatives (unauthenticated certificates or badges)
The openHPI(7) MOOC platform of Hasso Plattner Institute focuses primarily on fundamentals of computer science and digital technologies. A ‘graded record of achievement’ is offered to candidates upon ‘successful completion’ of openHPI courses. ‘Successful participation means that you earn at least 50% of the sum of maximum possible points for homework and final exam.’ In this case, the assessment is at Level 1 (see Table 2), illustrating the typical case where low robustness of assessment is related to a low formal credentialisation.
Examples of Level 2 credentialisation initiatives (authenticated certificates; no credits)
Since Coursera courses offer ‘verified certificates’ to students who complete a course on the Signature Track, the Coursera MOOCs by European providers that offer this option fall under Level 2 in the formality of credentials hierarchy. This correlates directly with Level 2 in the hierarchy of robustness of assessment.
Examples of Level 3 credentialisation initiatives (authenticated certificates; fewer than 5 ECTS credits; exemption from entrance exam)
Many of the European MOOC-providing institutions that promote their offerings under the umbrella of the OpenUpEd portal award formal certificates, which they describe on their website as ‘official credits that can count towards obtaining a degree (i.e., ECTS).’ The Università Telematica Internazionale (UNINETTUNO) in Italy provides the vast majority of the courses listed on the OpenupEd portal, covering a very wide range of subjects. Learners who want to get ECTS credits for these MOOCs need to enrol in the corresponding course offered by this university. Then, a tutor is assigned to the enrolled student, whose learning activities are also recorded. A final exam is administered to the MOOC participants, and those enrolled students who pass the exam are awarded ECTS credits. In this case, the assessment is at the most robust end of the scale (Level 4 in Table 2), while the credentials received (5 or fewer ECTS credits) is pitched at Level 3 for formality of credentials. Several other examples were found of this pattern in the MOOC initiatives reviewed.
In Finland, the University of Helsinki’s Department of Computer Science runs courses in which students are required to produce programming code that is automatically assessed using the institution’s TestMyCode (TMC) testing system(8) . This made the tasks easily adaptable to a MOOC format. This is an atypical example, in that the relatively low robustness of assessment (Level 2 in Table 2) appears to be incommensurate with the formality of the credential awarded. A very small number of such outliers were found in the OpenCred study, and in all such cases an explanation was found within the particular context of the examples. (In the Finnish example, the developers of this course had developed an algorithm which could effectively evaluate students’ code. This was only possible because of the nature of the subject matter, and the skills of the academics who designed the course.)
Examples of Level 4 credentialisation initiatives
Many of the initiatives that award credentials at the highest level tend to offer a range of credentialisation options for students. In some cases, providing institutions award credentials only to students who are enrolled in their fee- bearing programmes, and in others, open learners are given the option to pay a fee for assessment and credits. Typically in these cases, learners are assessed by the most robust forms of assessment, again reflecting the close correlation between robustness of assessment and formality of credentials.
As a rule, robustness of assessment and credentialisation of learning are very closely linked for the majority of open learning initiatives. This raises a contradiction in the argument that MOOCs represent an opportunity for more accessible and inclusive educational provision. Credentialisation requires robust assessment, and robust assessment requires tutors to review performance and students to have their identities validated. This all requires financing. To the extent that these costs have to be passed on to learners, or learners have to be enrolled in one of the providing institution’s mainstream programmes to receive recognition, MOOCs become that much less open and less inclusive. The challenge for institutions is to overcome this low cost / high value incompatibility in the most cost-effective way.
(1) This article is an abridged version of an article published by the author in the eLearning papers and available here.
(2) Eurotech Universities(2014, July 7). EuroTech Universities session animates debate on MOOCs and future of education at ESOF2014. Retrieved from http://eurotech-universities.eu/eurotech-universities-session-animates-debate-on-moocs-and-future-of-education-at-esof2014/
(3) Camilleri, A.F. and Tannhäuser, A.C. (2013). Chapter 4: Assessment and Recognition of Open Learning. In L. Squires and A. Meiszner (eds) Openness and Education. Bingley: Emerald Group Publishing Limited, p.28
(4) Verstelle, M., Schreuder, M. & Jelgerhuis, H. (2014). Recognition of MOOCs in the Education Sector. 2014 Open Education Trend Report, p.25. Retrieved from http://www.surf.nl/binaries/content/assets/surf/en/2014/trendrapport-open-education-2014-eng.pdf
(5) Kjeldstad, B., Alvestrand, H., Elvestad, O.E., Ingebretsen, T., Melve, I., Bongo, M. et al. (2014). MOOCs for Norway: New digital learning methods in higher education. Retrieved from http://www.regjeringen.no/pages/38804804/PDFS/NOU201420140005000EN_PDFS.pdf
(7) openHPI (2012-2014). FAQ. Retrieved from https://openhpi.de/pages/faq.
(8) Vairimaa, R. (2013). Joustava MOOC: Avoimet verkkokurssit käyvät vaikka pääsykokeen korvikkeeksi. Yliopistolainen 2, s.3-6. Retrieved from http://palvelut.unigrafia.fi/yliopistolainen_2_2013/.pp.3-6