3. Research Design and Methodology

MSc Report - Discoverability of Open Educational Resources - Kelly TerrellThe activity around open educational resources (OERs) is still a relatively new space for higher education, yet often the impetus of funded initiatives has been upon achieving or being seen to achieve a critical mass of released resources. When coupled with rapidly changing and increasingly diverse open education practices this can make it difficult to determine the overall state and progress made in the field so far, a necessary process for identifying areas of research. In order to respond effectively to the challenges around discoverability there is firstly a need to understand what has been done in the field so far in terms of the methods chosen for release as well as any overt discoverability strategies which have been considered and applied by providers. By correlating these approaches to available technologies and to our understanding of how users actually undertake the task of retrieving such content online will help to identify any missed opportunities and whether approaches are aligned to those theories

This study applied a methodology known as a scoping study, a broad comprehensive literature review intended to summarise and disseminate a wide range of evidence to those who might otherwise lack time and resources to do so (Arksey and O’Malley 2005). Through the synthesis and analysis of evidence on OER discoverability including published articles, project reports and other forms of information available online, this wide-ranging base of knowledge will be brought into a single realm conveying the breadth and depth of the field providing greater clarity on this broad topic. Through a process commonly referred to as mapping, the evidence uncovered by the review will be used to identify gaps in the knowledge (Levac et al 2010).

3.1 Design

This study applied the Arksey and O’Malley six stage methodological framework as outlined in Table 4 which supports the rigorous identification and synthesis of narratives and by taking an iterative approach balances the need for comprehensive coverage of the broad area within the practical limitations of what can be achieved.

Stage 1 Identify the initial research question: defining of a clear but broad research question intended to guide the subsequent search strategies, with parameters which are intentionally wide at the outset to generate breadth of coverage.
Stage 2 Identify the relevant studies: comprehensively answer the research questions by developing a plan of which search terms to use and what sources will be looked at, such as electronic journals, reference lists, conferences. Potential limiting factors upon the search, such as time and language, will need to be considered upfront
Stage 3 Study selection: based upon criteria drawn from the research question terminology and increasing understanding of the materials determine post-hoc inclusion and exclusion of evidence.
Stage 4 Charting the data: use a data charting form to extrapolate key data from the selected studies, while a ‘narrative-view’ or ‘descriptive-analytical’ methods are used to extract contextual or process related information respectively.
Stage 5 Collate, summarise and report the results: using an analytical or thematic construction an overview of all materials is made. A numerical analysis of the results is presented using appropriate charts and graphs, and a thematic analysis is then clearly articulated.
Stage 6 Consultation: providing opportunities for consumer or stakeholder involvement to identify additional references and provide insights beyond what is covered in the literature
Table 4: Summary of the scoping study framework by Arksey and O’Malley (2005)

3.1.1 Design limitations
Limitations were imposed upon the scope of the evidence gathering which will be described in more detail later in this section. Limitations also applied to the methodology itself which must be noted due to their potential influence upon the outcomes. Although not a requirement, Arksey and O’Malley make reference to multiple reviewers being involved in the tasks of searching for and selecting studies for inclusion. This is further strengthened further by Levac et al (2010) who suggests that a team of at least two people should be involved in all the key stages of identification, review and analysis of evidence to ensure a more rigorous approach can be demonstrated. For this study only one researcher was available throughout, however it was the intention to recruit assistance in Stage 3, study selection, and Stage 4, charting the data to increase the reliability and validity of the resulting evidence base. By being given a small sample of the total JISC/HEA projects this person was to apply the documented criteria to ensure consistency in the selection and for Stage 4 using the data charting form (see Appendix 1) was to review the relevant evidence for a number of projects to verify the data charted by the main researcher. Unfortunately, due to unforeseen circumstances this person was not able to fulfil this role. Therefore it must be noted as a limitation on this study, though with the criteria and charting form available it may still be verified at a later date to add to this study.

Stage 6 of the framework describes consultation with consumers and stakeholders in order to gain insights not covered in the literature. This is an optional stage and was not conducted within the remit of this study. Such insights from consumers of OER however are likely to provide highly valuable insights into the approaches taken to discovering resources that are unlikely to exist in the literature to date and should therefore be regarded as a limitation on the results of this study, and an aspect that should be included in any future work to extend this study.

3.2 Research question

The aim of this review is to determine strategies used within the UK higher education community to release OERs to address the challenge of discoverability by education users.

The specific objectives are to:

  • Identify the range of methods chosen to release and share OERs
  • Explore the processes offered by these methods to address discoverability
  • Identify and explore the evidence of our understanding of online information discovery
  • Identify any gaps between our understanding of information discovery and the discoverability strategies being used
  • Determine if chosen strategies may hinder discoverability by the suggested beneficiaries of OERs

The reference to ‘education users’ is an ambiguous term which requires further definition in order to shape the search strategy. In its definition of OERs the OECD refers to users as ‘educators, students and self-learners’ however these terms themselves may still be considered too vague for the purpose of refining a search strategy. Arksey and O’Malley recommend maintaining a wide definition at the outset to reduce the likelihood of missing relevant articles however for this study a sector parameter is declared within the research question as we are looking within the UK higher education community only. This therefore enables us to further refine our education users without losing the dimension of breadth. The educations users at the outset are therefore regarded as being educators within higher education, students within higher education or self-learners of higher education material and content.

3.3 Identifying relevant studies

With the major JISC/HEA OER programme funded by HEFCE running from 2009 – 2012 to support higher education institutions in creating and releasing OERs, hereafter referred to as UKOER, 79 projects ran over a three phase programme providing a major source of evidence for this study to utilise with full access to project reports, websites and the outcomes of projects themselves through the JISC website providing a case study of OER contributors to review.

Searches for relevant literature were conducted across a number of sources using a number of individual search terms as shown in Table 5 below.

Search terms included: open education, discovery, discoverability, online information retrieval, open education resources, oer

As well as individual search terms, these were then strengthened through the use of Boolean operators
Sources included: University of Edinburgh library searcher service searching across the library catalogue database, ejournals and full-text records
• ALT Journal
• Reference lists
• Search engine (including Google Scholar)
• Citations
• Grey literature
Table 5: Search strategy terms and sources

It had also been the intention at the outset to search across mailing lists and the previous UK based ALT and OER conference schedules, but there was insufficient time available to pursue these sources.

In line with Arksey and O’Malley’s framework it is considered sensible to apply a practical view to the time span and language at the outset for searches. For this study searches focused on the evidence produced from 2001 onwards in recognition of this year being considered the first major milestone for the identification and release of OERs. However as the topic of information discovery is much broader and not confined to OERs the decision was taken to extend the date to 1990 in line with the emergence of the world-wide-web. Foreign language material was excluded from the search due to lack of translation skills of the researcher.

3.4 Data selection

In order to refine the initial results of relevant evidence and literature searches a set of inclusion and exclusion criteria were defined. In the case of the UKOER projects it was clear that not all projects had been designed to create and release OERs with a number developing tools or exploring issues around policy and culture change. On this basis an exclusion condition was defined at the outset to disregard projects that did not include the release of OERs. As part of an iterative process of review, the project list was refined further which would not have been possible at the outset (Levac et al 2010) and exclusion on the basis of there being no evidence of the continued existence of the primary output of a project was created. For example unreachable web addresses or redirections to generic institutional pages due to platforms being closed.

In total, 79 UKOER projects from the JISC/HEA OER programme were initially identified for inclusion in this study. Of this total, 44 were excluded (see Table 6) with the primary reasons for exclusion as a result of reports not being accessible for analysis and insufficient evidence of projects releasing OERs for use. Many such cases were projects which conducted evaluations or investigated the culture and policy issues around the implementation of OERs. A full list of the reasons justifying exclusion of UKOER projects and the outcomes for each can be found in Appendix 2.

Total JISC/HEA OER Projects 79
Exclusion Criteria
1. Report could not be retrieved  11
2. Site(s) hosting OER no longer active  4
3. Insufficient evidence of OERs released  10
4. Multiple partners with distinct approaches  4
5. Technical Rapid Innovation projects  14
Projects suitable for inclusion  35
Table 6: Total UKOER projects excluded against criteria

A small number of projects were also excluded as their primary aim was to cascade their knowledge, experience and lessons learned to guide multiple other institutions in their own initial release of OERs. It was found in these cases that those institutions operated so independently from one another in their chosen approaches that they could not be accurately reflected under a single parent project for analysis purposes. With more time these cases should be analysed as separate entities to add to the body of evidence presented here.

The body of evidence on user behaviour towards information discovery and technologies supporting discoverability were reviewed for their suitability to answer key objectives of the research questions, with many already being identified from the earlier background literature review process. With time limited on this study a deadline was imposed after which no further studies were included.

3.5 Data analysis

In line with Stage 4 of the framework, an Excel spreadsheet form was devised to capture the key data extracted from the selected UKOER projects. The template form (see Appendix 1) included fields to capture:

  • Programme phase and year
  • Lead institution(s)
  • Project category (institutions, subject or individual)
  • Aim of project
  • Sharing platform(s) including identification of a primary platform and underlying software
  • Tools and practices utilised to raise visibility

It quickly became apparent in the early stages of analysis that multiple platforms had been employed by projects to release and raise the visibility of their resources, yet a primary platform was evident in almost all cases either explicitly declared or implied through the narratives in project reports. It was necessary to take a flexible and adaptive approach to the data form and adjust the design accordingly to reflect the increasing familiarity with the topic being uncovered (Levac et al 2010).

Through a descriptive-analytical approach, the final report for each UKOER project and its associated sharing platforms were reviewed to extract the key information on the project, platforms and methods utilised to aid discoverability. Reports varied significantly in length, structure and substance due to the variation in project size, partners, complexity of aims and the level of detail on the approaches taken making this a time consuming process. Following the framework, the objective of this process was to be able to provide a numerical summary of the strategies in use across the programme and provide an initial indication of popular approaches and obvious gaps.

Secondly, the projects were then ordered thematically, according to the target audience of the OERs they released. Due to available time the decision was taken at this point to focus on those projects who had explicitly identified the public or learners outside of their own institutions as a target audience for the OERs they released. This then became the primary area for the final literature review.

3.6 Scope limitations and constraints

The notion of open education is of course a global one however this study intentionally focuses on the UK higher education sector and will therefore miss out on potentially valuable insights and approaches being taken around the world which a future study may wish to address. With a single researcher conducting this study, even with a geographical and sector focus, it quickly became apparent a number of practical limitations would need to be applied to the scope of this study to keep it manageable. Initially the desire for a broad representation of the sector led to a search for evidence from any institution or project releasing OERs. It quickly became apparent this was an unrealistic goal for a single researcher with limited time. The UKOER projects presented itself as well defined case study and though it must be acknowledged that other valuable contributors of OER from UK higher education institutions may have been missed, there would be little value in this study simply providing a superficial summary of key features of many contributors. By focusing on this case study and exploring links with different elements of the topic of discoverability and associated theories the usefulness of the findings can go beyond a simple summary and presentation of raw figures (Davis et al 2009).

4. Discussion and Findings >>