Policy & Practice - A Development Education Review

 

 

Development Education on a Massive Scale: Evaluations and Reflections on a Massive Open Online Course on Sustainable Development

issue26
Development Education in Politically Interesting Times
Spring 2018

Silvia Gallagher

Abstract: Online platforms have been increasingly used to disseminate development education to a broad range of learners around the globe.  This case study examines the development, implementation, and facilitation of a multidisciplinary Massive Open Online Course (MOOC) on Achieving Sustainable Development by Trinity College Dublin.  It explores how MOOCs can be a method for sharing educational content to an international audience, and as a means for understanding public perception of the Sustainable Development Goals (SDGs).

          A mixed-method approach was used for the course evaluation. Quantitative pre-and post-course survey data was analysed to explore learner demographics, motivations, and content perceptions.  Qualitative analysis of in-course learner comments illustrated common themes emerging from the course.  A total of 3,958 learners from 159 countries registered for the four-week free online course.  Learners engaged successfully with the content via 6,116 comments posted and 35,967 completed content steps.  Satisfaction with the interdisciplinary and collaborative nature of the course was expressed by learners.

          Common themes that emerged in learner comments included changing personal perceptions of sustainability, satisfaction with case studies, and improved understanding of the interrelatedness of the SDGs.  This research contributes to the body of knowledge of development education, and positions MOOCs as a positive means of disseminating knowledge to an online global audience.  Development organisations seeking to engage with the wider public should consider MOOCs as a method for educational practice, or employ pedagogical strategies used in MOOCs in face-to-face learning environments.  In addition, MOOCs can also provide timely information about public perceptions towards sustainable development, and offer a means to connect with learners on a global scale.

Keywords: Massive Open Online Courses; Social learning; Sustainable Development Goals; Online learning.

Introduction
Since their introduction in the early part of this century, Massive Open Online Courses (MOOCs) have grown in popularity for presenting free educational materials to large scale cohorts of online learners (Sinclair et al., 2015; Baturay, 2015).  Individuals use MOOCs to learn new topics or increase current knowledge (Hew and Cheung, 2014).  Universities and institutions develop MOOCs to extend their reach and access, increase innovation in teaching and learning (Hollands and Tirthali, 2014), and improve reputation (Davis et al., 2014).  MOOCs are now attracting more than 58 million learners worldwide, with over 700 universities developing online courses on a massive scale (Shah, 2016).

          These types of online courses can address some of the challenges facing development education.  Open online content enables access by any individual with an internet connection and computer.  Massive numbers of learners form a global learning community who share experiences, diverse opinions and critical thought.  Sustainable development research and education applied by universities can be shared with the public, aligning with strategic policy objectives.  In addition, the lack of prior learner educational requirements supports life-long learning for all global citizens.

          This case study outlines the design and development of a MOOC in sustainable development by Trinity College Dublin.  It examines how MOOCs can be a successful means for disseminating sustainable development educational content to a large scale international learner cohort.  Furthermore, it aims to evaluate whether MOOCs can be a means for understanding public perception of sustainable development issues.

The structure and pedagogy of MOOCs
At their simplest definition, MOOCs bring together learners and teachers in a free, online platform where open access videos, text based articles, discussion questions, links, references, interactive materials and assessments are structured around learning objectives to deliver an educational course (Liyanagunawardena et al., 2013).  MOOC platforms, such as EdX, Coursera, Udacity, Udemy and FutureLearn, facilitate large scale access, with some MOOCs attracting over 380,000 learners concurrently (Parr, 2015).  Most MOOCs do not require any prior experience or knowledge, and in general, provide no formal qualifications.  However, a move towards accreditation for some MOOCs has been developing in recent years, and some institutions are now using MOOCs for credit bearing qualifications.  At present, though, most MOOCs are non-credit bearing.

          Historically, MOOCs were divided into ‘cMOOCs’, focusing on connectivism between learners, and ‘xMOOCs’ with an emphasis on information transmission.  Connectivist cMOOCs use a networked and decentralised approach to learning whereby knowledge is transferred via the contributions and interactions of learners (Margaryan et al., 2015). Conversely, xMOOCs follow a more traditional information transmission pedagogy.  Learners are presented with online materials which they complete in their own time and are encouraged to interact with one another through discussion boards within the MOOC platform.  Although there are now many diverse MOOC models such as SPOCs (Small Private Open Online Courses), BOOCs (Big Open Online Courses), LOOCs (Local Open Online Courses), and HMOOCs (Hybrid Massive Open Online Courses) (Pérez-Sanagustín et al., 2016; Chauhan, 2014), variants of the xMOOC structure are most commonly used today. 

The rise of MOOCs for sustainable development education
Although MOOC development has seen rapid growth in recent times, the first MOOC on sustainable development education was implemented relatively recently in 2012.  Since then, Zhan et al’s (2015) content analysis highlighted 51 MOOCs within the lens of sustainable development education.  These MOOCs are related to themes such as energy, sustainable development, natural resources, ethics, and climate change.  More recent activities such as the SDG Academy (2017) and the SDG Initiative (2017) offer MOOCs with a focus on the Sustainable Development Goals (SDGs).

Within the context of development education:

“MOOCs (...) can offer learning resources and opportunities for people to cultivate their awareness of global environmental protection, of a sense of sustainability, and also to learn about the ways in which universities teach sustainability-related knowledge in an open online environment” (Zhan et al., 2015: 2279).

          In addition, cost savings in their delivery, provision of open materials, and global access to educational resources have earmarked MOOCs as being an opportunity to address SDG 4 on ‘Quality Education’ (McGreal, 2017).  However, challenges remain, with concerns over low completion rates, and issues with quality assurance, accreditation, cultural biases, and inclusivity for individuals lacking digital skills (Yuan and Powell, 2013; Laurillard and Kennedy, 2017).

          MOOCs can be valuable for sharing educational content on sustainable development issues, however, the pedagogy behind MOOCs serves other disciplinary-specific benefits.  The facilitation of social learning and learner interaction within some MOOC platforms, such as FutureLearn, supports intercultural dialogue, interdisciplinary communication and collaboration, and knowledge generation (Barth and Burandt, 2013).  Open learning environments and content support inclusive and lifelong learning opportunities, aligning with SDG 4 (UNESCO, 2017).  Learners communicating on a massive scale with others from different countries and cultures encourage critical engagement and awareness of key sustainability issues.  Furthermore, institutions delivering MOOCs on sustainable development topics address global sustainability strategic and policy objectives (Cotton et al., 2007).

          In effect, MOOCs can respond to one of the key challenges of sustainable development education, to:

“…focus on sharing knowledge, skills, values and perspectives throughout a lifetime of learning in such a way that it encourages sustainable livelihoods and supports citizens to live sustainable lives” (UNESCO, 2005).

          This case study seeks to describe and evaluate a MOOC on sustainable development delivered by Trinity College Dublin in 2017.  It also explores the potential for MOOCs to enhance public understanding of the SDGs. 

Case study: Achieving Sustainable Development MOOC
The Achieving Sustainable Development MOOC (FutureLearn, 2017) ran from the 11 September to 15 October 2017.  The course was developed through a partnership between Trinity Online Services Limited (TOSL), the Trinity International Development Initiative (TIDI), and hosted on the FutureLearn platform.  An interdisciplinary approach to content development was used, with contributions from 17 Trinity College academics connected in some way to the Trinity International Development Initiative.  These included those at the Department of Economics, the Department of Geography, the Department of Sociology, the School of Ecumenics, the School of Engineering, the School of Medicine and the Trinity Impact Evaluation Unit.  One of the key challenges of development education is facilitating multidisciplinary skills and knowledge (Sharma et al., 2017), and the compilation of this MOOC sought to address this challenge.

MOOC educational content
Structured into four weeks, this MOOC contained 73 learning units known as ‘steps’.  Steps included short videos (between 3 and 8 minutes), text based articles with images and references, discussion questions, multiple choice questions for formative assessment, and multimedia exercise steps (e.g. interactive timelines).  Access to the MOOC was free, however, learners could upgrade for a small fee to download a certificate of completion.  Each week was balanced using a combination of step types to encourage course completion and learner commentary.

Table 1: Steps (learning units) in the Achieving Sustainable Development MOOC

Step Type

Frequency used in the MOOC

Video

18

Text article

39

Discussion question

8

Interactive element

3

Multiple choice question

40

 

The four weeks were structured around five SDGs; SDG 16 ‘Peace, Justice and Strong Institutions’, SDG 3 ‘Good Health and Wellbeing’, SDG 6 ‘Clean Water and Sanitation’, SDG 11 ‘Sustainable Cities and Communities’, and SDG 5 ‘Gender Equality’.  The fourth week bookended the course by focusing on questions of measuring sustainability, with SDG crossover and interconnectedness addressed.  Overarching this learning content, were the core learning objectives, namely ‘to reflect on the challenges to achieving sustainable development’ and ‘to identify and analyse some of the root causes of underdevelopment from a multidisciplinary perspective’.  At the onset of course development, it was imperative that each step addressed the learning objectives of the overall course and encouraged interaction between learners.

          The content used a narrative ‘storytelling’ approach, rather than academic language to facilitate understanding from learners who may not have studied the concept of sustainability before, nor had English as a first language.  Case studies were used to illustrate key concepts, and reflection questions were posed on each step to encourage participation and learner interaction.

Social learning
To ensure quality learning outcomes, interaction between learners and critical discussion on a massive scale, social learning, one of the underpinning philosophies of MOOCs, was integrated through the course (Brinton et al., 2014).  Social learning in the context of sustainable development has been identified as helping ‘facilitate knowledge sharing, joint learning and knowledge co-creation between diverse stakeholders around a shared purpose’ (Kristjanson et al., 2013).  Given the importance of social learning from a development education perspective, developing a MOOC on sustainable development where social learning is heavily encouraged in the design of the course, creates an interesting nexus.

          In practice, course design integrated social learning within each step by posting a discussion question, related to learning content and objectives, at the bottom of each step.  For example, in the first week, learners were provided text and a graphic on Galtung’s three types of violence.  After reviewing this content, they were asked ‘Which form of violence do you think creates the most challenges to sustainable development?  Why?’.  Having read the text and viewed the image, learners posted comments (n=158) supporting their reflection.  The discussion thread included both single author comments and multi-author comment ‘threads’ where learners would discuss with one another their responses.  Learners could also ‘like’ comments that they agreed or emphasised with (n=150).  Interactions were supported by comments from academics and a student moderator, who responded to recurring comment themes, or most ‘liked’ comments.  In addition, learners could also ‘follow’ other learners and academics to keep track of comments from individuals they were interested in. Table 2 provides details of five of the most commented on discussion questions used.

Table 2: Most commented on discussion questions in the MOOC

Question

Number of comments

Describe what you think is the most challenging SDG to achieve? What do you think is the greatest challenge to achieving all of the goals?

349

Pick one criticism or one positive statement about SDGs out of the lists above. Do you agree or disagree with this statement? Why / why not?

259

Thinking about Johan Galtung’s conception of violence:

Which form of violence do you think creates the most challenges to sustainable development? Why?

159

Select one country and comment below on what development challenges or strengths you think could have affected its life expectancy.

148

Thinking about your experiences with water treatment:

How is the water in your area treated? What chemicals are added? Is there an ongoing requirement for energy to treat the water? How much do you pay for the supply of water – directly or indirectly – through taxation?

112

 

Multidisciplinary bridging
A fundamental challenge in developing the course, was how to integrate each of the separate SDGs in a cohesive course narrative.  Given that the MOOC addressed six different SDGs from multiple disciplines, it was key to the course narrative that each of these were addressed both in isolation, and in relation to one another.  To safeguard this course narrative, a linking step between each week was designed.  This ensured that learners understood the nature and importance of connections between the SDGs, and created a ‘flow’ between the different weeks.

          For example, the connection between Week 1 SDG 16 ‘Peace, Justice and Strong Institutions’ and Week 2 SDG 3 ‘Good Health’ was illustrated using a video step at the beginning of Week 2.  This video described how peace was a determinant of health, and explored issues such as the ‘Weaponisation of Healthcare’.  The content was supplemented with a discussion question for learners to reflect on the connection between these two SDGs.

          Creating a bridge between each week was essential in encouraging learners to move forward in the course and reduce drop off (Ferguson and Clow, 2015).  However, it was also contextually important to ensure that each SDG was not only understood in isolation, but that the key learning objectives ‘multi-disciplinarity’ and ‘interconnectedness’, were also addressed. 

Evaluation methodology
Pre-and post-course optional surveys were used to evaluate learner profiles, registration motivation, course perception, and satisfaction with learning resources.  Survey items used a five-point satisfaction Likert scale, and open-ended questions.  Pre-course surveys were disseminated during registration, and in the first step of the course.  Post course surveys were disseminated in the final step of the course, and in the final weekly email.  Data provided by FutureLearn illustrated total student registrations, social engagement statistics (i.e. number of comments, step completion rates) and geographical location of learners. This data was downloaded and analysed using Microsoft Excel and SPSS (Statistical Package for the Social Sciences).

          Comments throughout the course were reviewed during its implementation.  Post-implementation, comments were imported into NVivo to ascertain whether key themes emerged from learner comments.  This thematic analysis aimed to explore whether learner comments could provide information about public perceptions towards sustainability issues.

Results
Learner demographics and registration
A total of 3,958 learners from 159 countries registered for the MOOC, with 2,181 learners accessing at least one step of the course.  Most registered learners who provided details of their location were from the UK (24 per cent), Ireland (18 per cent), US (4 per cent), Nigeria (4 per cent), India (3 per cent), Australia (2 per cent), Germany (2 per cent), and Spain (2 per cent).  Although there was a relatively even proportion of learners from different age ranges, most learners were between 26 and 35 years (23 per cent).

          The pre-course survey (n=144) explored learner experience and rationale for taking the course and had a response rate of 6.6 per cent.  Many learners had previous experience with the topic of sustainable development (77.1 per cent), with a proportionally high percentage of learners working in a related field (41.73 per cent) or interested in the topic as a hobby (43.31 per cent).  Generally, learners were confident in their knowledge of the topic, with 81 per cent stating they were either a little, moderately or extremely confident in their current knowledge of the topic.  These results suggest that the majority of learners were from the sustainable development sector, or had previous experience or knowledge of the field.

          This was reinforced by results outlining learner rationale for taking the course.  High proportions of surveyed learners were taking the course to keep up to date with new developments in the subject (72.66 per cent) and to contribute to their continuing professional development (64.93 per cent).  However, learners were also motivated to join the course to learn from others’ experiences or perspectives (70.5 per cent).

Engagement: Social and course content
Data from FutureLearn described how learners engaged with the course content in the form of comments and step activity.  The MOOC facilitated learner engagement through posed discussion questions.  Learners responded to these questions using comments, resulting in the development of a community of learners.  A total of 6,116 individual comments were posted within the MOOC platform by 535 unique learners.  On average, a social learner (i.e. posts at least one comment) posted 10.5 comments.  Aggregate data on step access and video views (Table 2) demonstrate the relatively high engagement with learning content from learners.

Table 3: Summary of MOOC engagement statistics

Metric

Frequency

Total number of comments posted

6,116

Total number of likes

6,626

Average number of comments posted by a social learner

10.5

Total number of steps accessed

42,073

Total number of video views

10,439

 

In-course comment analysis
Throughout the course, learner comments were read and responded to by academics and moderators.  By reflecting on these comments, and using an auto-coding function on NVivo, an understanding of key themes from learners emerged.  Comments included personal reflections, descriptions of culturally specific events, experiences and case studies, interaction between learners, and critical thought.  The thematic analysis illustrated that many of the comments provided insights into public understanding of the SDGs.  For example, misunderstanding around the concept of sustainability, whereby some learners perceived it as being solely to do with environmental or ‘green’ issues.  Learners also commented that they had not previously considered the interconnectedness of the SDGs, nor understood the impact of them in a wider context (i.e. how the presence or absence of peace can have an impact on healthcare, sanitation, and gender equality). 

Post course evaluation
The post-course survey asked learners about their perceptions of FutureLearn and of the course.  A total of 62 responses (response rate = 3.2 per cent) were collected in the post-course survey, which was a relatively low response rate, but gave some indication of the general satisfaction with the course.  All learners surveyed rated their course experience as excellent or good, and were either satisfied or very satisfied with videos, written course content, and subtitles.  Satisfaction with written content (e.g. articles) was marginally higher than video content.  Key positive themes that emerged from the open-ended questions were the use of case studies, satisfaction with the structure and videos, and the discussion sections.  Open ended questions illustrated that learners enjoyed interactions between learners and academics, and felt that they were beneficial and conducive to learning.

          As the response rate was low for the post course survey, comments from the final step of the course were also analysed (n=59).  These provided additional insight into the perceptions of learners who may not have completed the post course survey, but had reached the end of the course.  Learners in these comments reported satisfaction with case studies, references, additional materials, and practical challenges posed in the discussion questions.  They also commented on ease of platform use, and satisfaction with materials covering new knowledge and information they had not been previously aware of.  In addition, some learners commented how they learned from the comments posted by others, supporting the benefits of social learning methodologies used in the course.  

Discussion
This case study has addressed how MOOCs can be a successful means of sharing educational content on sustainable development issues to large scale learner cohorts worldwide.  They can enhance communication between individuals, and further their understanding of sustainable development issues.  Encouraging multidisciplinary development of learning content, supporting critical thinking, and enabling social learning around sustainable development themes can address some of the challenges of development education.  Practitioners should consider the value of MOOCs for sharing educational content.  With the availability of open source MOOC platforms, this is potentially achievable at low cost.

          Nevertheless, implementing a MOOC may not be feasible for many organisations due to resource, time and technology limitations.  However, the approach used to develop the course could be mirrored in other online or face-to-face courses.  Creating bridging materials between disciplines, for example, was successfully delivered in the MOOC.  Educators can collaborate to produce materials which deliver a strong interdisciplinary narrative.  Identifying and bringing together common themes in multiple disciplines, while also addressing key learning objectives, can facilitate this approach.

          The value of social learning as an underlying pedagogy of this MOOC has multiple benefits, both to learners and to educators or institutions implementing the MOOC.  Incorporating social learning into the instructional design process encourages two-way commentary between learners and academics.  Not only does this support critical thought and learning motivation for learners, but it allows educators to understand public perception of sustainability topics.  For educators and institutions, comments illustrate public opinion towards the SDGs and development issues, outlines where the public may have difficulty in understanding concepts, and highlights which issues are of most interest to learners.  MOOC comments could potentially drive the development of future MOOC content and research agendas, and give a qualitative measurement of SDG understanding.  Learners can read comments that may offer them a different perspective, and interact with others from around the world who may be facing similar or different development challenges than themselves.

          However, social learning may not be appropriate in all learning environments due to curriculum design, and time and resource limitations.  Nevertheless, the use of reflection questions in face-to-face or smaller online course settings can encourage critical thought and engagement with learning materials.  They can also provide insight into learner perceptions, knowledge and viewpoints of a topic which can support academic instruction.  In face-to-face courses, questions can be pre-designed, posed, and a ‘think, pair, share’ approach used to encourage discussion.  In smaller online courses, reflection questions can boost learner discussion, but educators may need to spend more time facilitating the discussion for robust conversations to emerge.      

          Although MOOCs have shown some benefits within the development education space, some challenges remain.  The high proportion of MOOC learners with previous experience of development education highlights some of the overarching challenges with MOOCs.  Learners will gravitate to subjects that they are interested in, or have some prior knowledge.  Broadening development education MOOCs to the wider public who may not have prior interest in the topic, may require additional spending on marketing or engagement with organisations outside the sustainable development space.  Although MOOCs can address many challenges to development education, if they are not visible to those who have lower prior engagement with or interest in the subject, they may fail to be as massively diverse as they aim to be.

          Aligned with this challenge is the issue of exclusivity and accessibility.  This MOOC was accessed by learners from many different countries, however, many learners were from English speaking countries with western perspectives and traditions.  To register for the course, learners needed to have access to a computer and internet.  To adequately address a more globally balanced cohort of learners, English language MOOCs could provide subtitles in different languages, or offer automatic translation for comments in a non-English language.  In addition, MOOCs could be promoted within programmes that offer computer training to sectors of society that may be facing a digital divide (e.g. older learners, disadvantaged, and people with refugee status).  On a broader level, higher education institutions providing MOOCs should be cognisant of the diverse needs of learner cohorts when developing, implementing, and promoting MOOCs. 

Conclusion
MOOCs offer many benefits to organisations seeking to disseminate learning content, within the context of development education.  This case study has described how the successful design of a multidisciplinary MOOC can generate learner satisfaction, interaction between learners and disseminate education to large numbers of learners.  In addition, the use of social learning tools to encourage learner interaction and commentary demonstrates the benefits of MOOCs to sustainable development professionals.  In effect, learners not only learn from the content being provided, but educators and institutions can learn from learners themselves.  In these ways, MOOCs are highly effective means of disseminating sustainable development educational content to a large scale international learner cohort.  However, challenges do also remain in attracting those with little connection to the sector, those in non-English speaking countries (a strong focus of this MOOCs content) and those with poor access to digital resources. In these ways, equality and equity of access is a challenge to be addressed.

          Finally, this article, in aiming to evaluate whether MOOCs can be a means for understanding public perception of sustainable development issues, found that within the MOOC, comments illustrated public opinion towards the SDGs and development issues, outlined where the public may have difficulty in understanding concepts, and highlighted which issues were of most interest to learners.  This case study thus demonstrated that MOOCS on sustainable development can generate a rich level of understanding for educators and institutions, on the public perception of their sector.

Acknowledgements
The author would like to acknowledge the support of FutureLearn and members of the Trinity International Development Initiative, in particular Dr Mairead Finn, during the development of the course and review of this research.

References
Barth, M and Burandt, S (2013) ‘Adding the “e-” to learning for sustainable development: Challenges and innovation’, Sustainability, Vol. 5, pp. 2609-2622.

Baturay, M H (2015) ‘An Overview of the World of MOOCs’, Procedia - Social and Behavioral Sciences, Vol. 174, pp. 427-433.

Brinton, C G, Chiang, M, Jain, S, Lam, H, Liu, Z and Wong, F M F (2014) ‘Learning about social learning in MOOCs: From statistical analysis to generative model’, IEEE transactions on Learning Technologies, Vol. 7, pp. 346-359.

Chauhan, A (2014) ‘Massive open online courses (MOOCS): Emerging trends in assessment and accreditation’, Digital Education Review, Vol. 25, pp. 7-17.

Cotton, D, Warren, M, Maiboroda, O and Bailey, I (2007) ‘Sustainable development, higher education and pedagogy: a study of lecturers’ beliefs and attitudes’, Environmental Education Research, Vol. 13, pp. 579-597.

Davis, H C, Dickens, K, Leon Urrutia, M, Vera, S, Del Mar, M and White, S (2014) ‘MOOCs for Universities and Learners an analysis of motivating factors’, 6th International Conference on Computer Supported Education, Barcelona, Spain.

Ferguson, R and Clow, D (2015) ‘Examining engagement: analysing learner subpopulations in massive open online courses (MOOCs)’,  Proceedings of the Fifth International Conference on Learning Analytics And Knowledge, 2015, Association for Computing Machinery, pp. 51-58.

FutureLearn (2017) ‘Achieving Sustainable Development’ (online course), available: https://www.futurelearn.com/courses/achieving-sustainable-development/1 (accessed 12 December 2017).

Hew, K F and Cheung, W S (2014) ‘Students’ and instructors’ use of massive open online courses (MOOCs): Motivations and challenges’, Educational Research Review, Vol. 12, pp. 45-58.

Hollands, F M and Tirthali, D (2014) ‘Why do Institutions Offer MOOCs?’, Online Learning,Vol. 18, pp. 1-19.

Kristjanson, P, Harvey, B, Van Epp, M and Thornton, P K (2013) ‘Social learning and sustainable development’, Nature Climate Change, Vol. 4, No. 1, pp. 5-7.

Laurillard, D and Kennedy, E (2017) ‘The potential of MOOCs for learning at scale in the Global South’, Centre for Global Higher Education Working Paper Series, Vol 31.

Liyanagunawardena, T R, Adams, A A and Williams, S A (2013) ‘MOOCs: A systematic study of the published literature 2008-2012’, International Review of Research in Open and Distributed Learning, Vol. 14, No. 3, pp. 202-227.

Margaryan, A, Bianco, M and Littlejohn, A (2015) ‘Instructional quality of Massive Open Online Courses (MOOCs)’,  Computers & Education, Vol. 80, pp. 77-83.

McGreal, R (2017) ‘Special Report on the Role of Open Educational Resources in Supporting the Sustainable Development Goal 4: Quality Education Challenges and Opportunities’, International Review of Research in Open and Distributed Learning, Vol. 18.

Parr, C (2015) ‘“Biggest-ever Mooc” starts on FutureLearn’, Times Higher Education, available: https://www.timeshighereducation.com/news/biggest-ever-mooc-starts-on-futurelearn/2020257.article (accessed 5 December 2017).

Pérez-Sanagustín, M, Hilliger, I, Alario-Hoyos, C, Delgado Kloos, C and Rayyan, S (2016) ‘Describing MOOC-based hybrid initiatives: The H-MOOC framework’, Proceedings of the European MOOC Stakeholder Summit, Graz, Austria.

SDG Academy (2017) What we do: The SDG Academy, available: http://unsdsn.org/what-we-do/education-initiatives/ (accessed 4 December 2017).

SDG Initiative (2017) About us, available: http://www.sdginitiative.org/about-us.html (accessed 4 December 2017).

Shah, D (2016) ‘Monetization over massiveness: A review of MOOC stats and trends in 2016’, available: https://www.class-central.com/report/moocs-stats-and-trends-2016/ (accessed 29 December 2017).

Sharma, B, Steward, B, Ong, S K and Miguez, F E (2017) ‘Evaluation of teaching approach and student learning in a multidisciplinary sustainable engineering course’, Journal of Cleaner Production, Vol. 142, pp. 4032-4040.

Sinclair, J, Boyatt, R, Rocks, C and Joy, M (2015) ‘Massive open online courses: a review of usage and evaluation’,  International Journal of Learning Technology, Vol. 10, pp. 71-93.

UNESCO (2005) UN Decade of Education for Sustainable Development, Paris: UNESCO.

UNESCO (2017) ‘Unpacking Sustainable Development Goal 4 Education 2030’, Paris: UNESCO.

Yuan, L and Powell, S (2013) MOOCs and Open Education: Implications for Higher Education, CETIS: The University of Bolton.

Zhan, Z, Fong, P S, Mei, H, Chang, X, Liang, T and Ma, Z (2015) ‘Sustainability education in massive open online courses: A content analysis approach’,  Sustainability, Vol. 7, pp. 2274-2300.

Silvia Gallagher is the Open Education Project Manager in Trinity College Dublin. She holds a Ph.D. in Computer Science and a M.Sc. in Applied Social Research from Trinity College. Her research interests include qualitative analysis, online learner behaviour, and the development, implementation and evaluation of Massive Open Online Courses. Email: gallags6@tcd.ie.

Citation: 
Gallagher, S (2018) 'Development Education on a Massive Scale: Evaluations and Reflections on a Massive Open Online Course on Sustainable Development', Policy and Practice: A Development Education Review, Vol. 26, Spring, pp. 122-140