Assessing a Methodological Proposal for the Design, Creation and Evaluation of Learning Objects Oriented to Educators with Diverse Educational and Technological Competencies

Keywords: Learning Objects, Design Methodologies, Kolb Cycle, Digital Educational Materials
Keywords in Spanish: Objectos de Aprendizaje, Metodologías de Diseño, el Ciclo de Kolb, Materiales Digitales Educativos
Received: 2015-10-30 Revised: 2016-02-14 Accepted: 2016-03-01
DOI: http://dx.doi.org/10.19153/cleiej.19.1.3

1 Introduction

Digital Educational Materials (DEM) have made their way to the traditional classroom - audiovisual media, such as an electronic presentation or a video, have gradually replaced printed texts and become content mediators in the learning process. This has enriched educational processes with a variety of materials and resources available for students. Conducted research [1] concluded that computer-based instruction is a better motivator, and students who use computers learn more than those who learn using traditional materials. This would require educators to be prepared to configure new learning scenarios for students, and they would also need to expand their educational material collections with new, digital materials that are good enough as additional learning materials, beyond the classroom. To achieve this, educators make use of various strategies that are directly dependent on their knowledge (pedagogy, didactics, technology) [2] and whose purpose is obtaining specific learning results. However, the strategies that have been adopted by educators (in some cases, digital immigrants) to produce DEM in contexts where students are digital natives, have been insufficient [1]. Digital immigrants are the result of a digital migration process that involves moving towards a highly technological environment created by Information and Communications Technologies (ICTs) - while students (digital natives) have grown up with technology and are therefore innately prepared for the digital jargon and environment, this is not the case of their educators, the digital immigrants [1]. There is therefore a need for strategies and guidelines that help educators design and create DEMs.

The Learning Objects (LOs) paradigm offers a strategy to achieve this. A LO can be understood as ”an independent digital didactic unit designed to achieve a specific learning objective and to be reused in different Teaching and Learning Virtual Environments (TLVEs) as well as different learning contexts. It must also include metadata that allow locating and contextualizing it.´´ [3]. The creation of DEMs through the LO paradigm requires the application of several disciplines for their design, development, production and storage, such as: a) Instructional Design, b) Computer Science, c) Library Science, and d) Human-Computer Interaction. Instructional Design guides educators in the design of learning strategies that will help students achieve the learning objectives proposed. Computer Science provides mechanisms that will allow adapting and converting the contents to be taught into digital materials supported by the use of digital formats; it also provides communication channels with users. Library Science allows cataloging digital materials by using labels that include a description of the corresponding digital material to facilitate their storage, localization, and retrieval. Human-Computer Interaction is aimed at improving the interaction between LO users and LOs themselves, with the purpose of improving information exchange, strengthening the stimulus received by students to engage and motivate them.

However, the concept of LO is not a unified one [4, 5, 6, 7, 8, 9, 10, 11, 12] which has resulted in a wide variety of interpretations that propose a number of LO design methodologies [13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 2, 24, 25, 26, 27] that have mostly been conceived to cover specific educational material production needs within each university space, from various perspectives, in specific environments that are adapted to the needs of the case at hand, and which require educators with mid-to-advanced computational knowledge to focus on the use of a given software tool or, alternatively, requiring educators to master specific instructional methods to propose meaningful activities to facilitate learning.

In this paper, we will introduce and describe a methodological proposal for the design, creation and evaluation of LOs; this proposal is the result of a through compilation and analysis of LO design methodologies in Ibero-America. Our proposal, which we have called DICREVOA, has 5 phases - analysis, design, implementation, evaluation and publication. This methodological proposal is conceived to be used by educators with any level of educational or technological knowledge. Therefore, it focuses on the design of LO activities around Kolb Cycle (educational) and the use of the ExeLearning tool (technological) to implement the LO structures. Then we present a case study in which the methodology was assessed.

This article contributes to the research on Learning Objects and methodologies in two ways. First, it extends the literature on Learning Objects by providing a solid concept about what is and what isn´t a LO. Second, it provides a practical methodology intended to those teachers who are willing to drive the entire process related to the design, creation and evaluation of LO through educational and technological sides. In this way the evaluation is intended in two ways, one is under the eye of the teacher as producer of DEM and the other one under the eye of the students as a consumer.

The article is organized as follows: section 2 describes the state of the art in relation to methodologies and methodology proposals for the design of learning objects; section 3 discusses the phases of the DICREVOA proposal; section 4 present a case study in which the methodology was assessed by students and educators; section 5 discusses the results obtained; and section 6 presents the conclusions. ]]>

2 Related Work

In this section, thirteen methodologies and methodology proposals that are used for designing and creating LOs are analyzed. For the analysis, a tool that provides an analysis framework for selecting LO design methodologies, called MASMDOA [28], is used. This tool establishes three dimensions that allow identifying the most relevant aspects, both educational and technological, of each methodology being considered. The results obtained are the basis for developing the methodology proposal presented in this paper.

2.1 Analysis of Learning Objects Design Methodologies

MASMDOA allows analyzing methodologies, proposed methodologies, and heuristics used to implement LOs. In this frame, MASMDOA proposes certain criteria, obtained by systematically reviewing bibliography, on agreements on the conceptual definition of a LO by the most relevant authors in the last 17 years (1998-2015). These criteria are closely related to the characteristics a LO should have, but they also include technological and pedagogical aspects. Criteria are grouped under three dimensions: a technological dimension, an educational dimension, and a general dimension. This allows analyzing them from various perspectives, and knowing the orientation of each methodology without actually implementing it; for instance, knowing if it is aimed at dealing with aspects that are more on the technological side than the educational or vice versa, or if there is a balance between both aspects.

From the technological perspective, the criteria used are: 1) Locatable, 2) Guidelines/Techniques, 3) Reusable, 4) Publication, 5) Interoperable; from the educational perspective, the criteria used are: 6) Educational Design, 7) Assembly Possible, 8) Learning Objects Components; from the general perspective, the criteria used are: 9) Definition Included, 10) Methodology User, 11) Author Licenses. For more detail on each of the dimensions and their criteria, see [28]. Table 1 compares the technological and educational aspects covered by LO Design Methodologies.

The methodologies described in Table 1, have been conceived at different points in time and for specific contexts, and were aimed at meeting existing needs in the production of LOs. Some of these methodologies, such as MEDEOVA, MIDOA, ISDMELO, UBOA, and UAT, suggest that a multidisciplinary team would be required to adopt them (coordinator, programmers, content reviewers, content experts, etc.) which, for an educational institution, would be expensive given the number of professionals required. On the other hand, the Techno-Pedagogical and LOCOME methodologies are oriented to educators with knowledge of Computer Science, that is, educators with technical training in Computer Science, while the Standard Technologies, Pattern-Based, UPV, and ISDOA proposals are oriented to educators with basic Computer Science knowledge. Unlike the others, CROA and AODDEI propose their own instructional design model and are oriented to educators who do not have experience in the development of LOs; however, they do not explain how the LOs are assembled.

Figure 1 summarizes the number of criteria in each of the three perspectives defined in MASMDOA that are covered by each of the methodologies. Eight of these, namely CROA, ISDMELO, ISDOA, Techno-Pedagogical, LOCOME, Standard Technologies, AODDEI and Pattern-Based, cover most of the criteria linked to the technology dimension. However, only CROA, ISDMELO, UAT and UPV encompass all of the criteria considered for the educational dimension.

]]> Fig. 1: LO Design Methodologies Compared

The Polytechnic University of Valencia [13] in its proposal for the creation of LOs, uses, in its content model to produce the Learning Module, something that is very similar to CISCO’s content model, which groups between 5 and 9 Reusable Information Objects (RIOs) to produce a Reusable Learning Object (RLO), using an introduction or overview, a summary and an evaluation. However, in order to create LOs using this proposal, educators must be familiar with, or at least have basic knowledge of, the computer tools required.

The Pattern-Based proposal [14] considers LO production issues from a strategic and cognitive view that consists in using predefined LO patterns. When building LOs, patterns are used to identify and select the processes to develop, such as learning dimensions or various types of cognitive activities; these, together with computer supplies (readings, images, audio, video) and the adaptation of instructions, will make up a LO.

The MEDEOVA methodology [15] was conceived with the purpose of providing guidelines for the design, development and publication of Virtual Learning Objects (VLOs) that can be used to support the teaching and learning process at the various courses offered by the Saint Thomas Aquinas University at Tunja, Colombia.

The MIDOA methodology proposal [16] presents a conceptual model for the creation of LOs that is based on the use of a Software Engineering development methodology and the creation of production rules under an instructional design. It proposes modeling the process based on the use of either of the following Software Engineering methodologies: 1) Evolutionary Prototyping, 2) Extreme Programming.

The ISDMELO methodology [17] is focused on the design and development of educational content and is based on the instructional design model. ISDMELO considers that, given their expertise, instructional designers are the ones responsible for assembling LOs. This methodology is strongly supported by learning theories and widely encompasses the LO-supported instruction design process. ISDMELO recommends analyzing student learning styles; however, it does not relate LO design aspects to the analysis of these styles, meaning that there is no link between the profile of the students from the standpoint of their learning style and the design of the LOs from a pedagogical point of view. It is oriented to multidisciplinary teams.

The LOCOME methodology [18] comes from the need for a robust methodology for building LOs. Design is based on necessary standards and mechanisms that can guide the process of building software objects using the RUP (Rational Unified Process) methodology. This methodology is fully iterative, which means that, when inconsistencies, failures or weaknesses are found in any of the lifecycle phases, several iterations can be run on that phase, or even go back to previous phases.

The UBoA methodology [19] establishes its theoretical and pedagogical foundation in accordance to the virtual pedagogical model of the University of Boyaca. The pedagogical model is used to build the basis for defining requirement levels to be taken into account when building LOs. These include conceptualization, design, production and distribution phases designed to answer the questions What does it teach?, How does it teach it?, and What is assessed, and how?. The UBOA methodology is organized in five phases, each with their respective activities and results specification.

The Techno-Pedagogical proposal [20] combines knowledge from the areas of Education, Software Engineering, and Human-Computer Interaction. Human-Computer Interaction deals with all aspects pertaining to interface development (it should motivate students), Education is used to describe the teaching and learning process to be carried out, emphasizing learning scenarios, and Software Engineering dictates how to implement the learning process on the computer.

The ISDOA methodology proposal [21] is a problem-based LO design proposal with well-defined characteristics in relation to the design of the problem, design of the graphic interface, software architecture pattern, self-evaluation process, and metadata structure that must be in agreement with the needs and learning styles of the target audience and deployment platforms defined in non-functional requirements. The lifecycle for the development of LOs proposed by ISDOA by means of Software Engineering is supported on two fundamental pillars - test planning and quality assessment. This proposal considers that both pillars must be run in parallel for all LO creation phases. ]]> The CROA methodology [22] resorts to a number of questions in each of its phases; these questions guide both the analysis and the design of LOs. Upon completion of each phase, certain output deliverables are obtained; these deliverables become the documentation that supports the LO. This methodology is oriented to any type of educators.

The UAT methodology [23] establishes a methodological link between pedagogy and its theories (constructivist, cognitive, interaction and communication) and the technological development of the LOs. It takes aspects from Software Engineering, Dick and Carey’s instructional design. The LOs that can be obtained are of one of two types - oriented to the educator or oriented to multidisciplinary teams.

The LO design proposal that is based on Standard Technologies [2] proposes the design and implementation of LOs under parameters that facilitate interoperability, reusability and maintenance, taking into account the characteristics of learners and using standard architectures that support e-learning, such as Learning Technology Systems Architecture (LTSA), IMS Content Packaging specification and LO metadata (IMS-METADATA).

The AODDEI methodology [24] is based on L’Allier’s definition [4] and it was proposed by the Autonomous University of Aguascalientes. This methodology is aimed at solving some of the issues that face educators with no experience in LO development.

The AODDIE methodology is based on the ADDIE instructional model [29] , which guides the pedagogical aspects of the methodology and provides educators with a series of templates that they can use to collect information.

---

TABLE 1: Comparison of Related Design Methodologies

---

]]>

2.2 Comparison of Considered Methodologies

Some of the methodologies and proposals analyzed here have been conceived to be used in contexts with specialized teams available for designing and creating LOs (designers, programmers, style correctors, educators, teachers, etc.), while others require educators themselves to tackle the entire LO design and creation process, which would imply that educators are required to have all the technological and educational skills needed for implementation. In this sense, it should be noted that educators coming from non-specialized technological contexts (e.g., Educational Sciences, Philosophy) will have difficulty when attempting to use the tools that support LO creation. On the other hand, those educators that come from specialized technological contexts (e.g., Engineering, Architecture) might have difficulties when tackling the educational and instructional design aspects of the LOs. As regards instructional design, it should be noted that the educational material that is designed may be suitable for students with a certain profile or learning style but not for other students. It would be interesting then to consider how to propose LOs that can motivate students with different student profiles or different learning styles.

In this context, DICREVOA is intended to be used by teachers who do not have a multidisciplinary team in their institutions of higher education, and it is they who can carry out the whole process of design, development and evaluation of LOs. This methodological approach emphasizes the proposed instructional design, introducing the cycle Kolb as its centerpiece of the activities. It also recommends the use of eXeLearning tool for designing the main structure of LO and at the same time it facilitates the incorporation of metadata into the LO, whether LOM or Core Dublin used. In addition to this it proposed evaluating LOs from the perspective of students and teachers.

Thus, the following questions gain relevance in relation to LO design and creation: RQ1: How should LO requirements be analyzed? RQ2: How can LO design be tackled from both an educational and a technological perspective for educators from different contexts? RQ3: What types of activities are appropriate for the different student learning styles and which of these can be designed for inclusion in LOs? RQ4: What are the tools that facilitate LO implementation and annotating LO metadata? RQ5: How can LO quality be assessed from the perspective of the educator as producer and from the perspective of the student as consumer? The proposal presented in this article is aimed at answering each of these research questions.

3 PHASES OF THE METHODOLOGICAL PROPOSAL FOR THE DESIGN, CREATION AND EVALUATION OF LEARNING OBJECTS

In this paper, we present a 5-phase methodological proposal called DICREVOA, which is the result of reviewing 29 proposals, heuristics and methodologies [30] used in Ibero-America for designing and creating LOs that are stored in Learning Object Repositories (LORs).

The phases in DICREVOA are as follows: A) Analysis, B) Design, C) Implementation, D) Evaluation, and E) Publication. The purpose of these phases is guiding educators in the design of LOs; to this end, a number of guidelines are provided to allow educators to make significant decisions in relation to LO-facilitated learning planning. This methodological proposal is aimed at educators who do not have access to a multidisciplinary team to provide the necessary support for the creation of LOs, which means that the educators themselves are responsible for the entire process of designing, creating and evaluating LOs. See Figure 2. ]]>

---

Fig. 2: Phases in DICREVOA

---

3.1 Analysis

In this phase, the requirements behind the need to develop a LO are surveyed, as well as its relation to the target audience profile. The following questions are answered: What is it that the target audience needs to learn? What is the topic to be developed? Who is the target audience of the material? To this end, the template shown in Table 2, can be used. It includes a description of each of the issues that must be considered.

---

]]> TABLE 2: Learning Objects Needs Analysis

---

As regards students’ profiles, it is important to know what the prevailing learning styles are among students. This could be key for the creation of digital educational materials, and its purpose is to adapt to students’ preferences to help them learn.

Learning styles are the cognitive, affective and physiological features that act as relatively stable indicators of how students perceive, interact with, and respond to their learning environments [31]. The cognitive features are linked to preferences students have for perceiving and processing the information they learn; affective features are related to student motivations and expectations for learning, and physiological features are connected to biotype and biorhythm aspects. For Kolb [32], learning styles are the preferred abilities to learn, which are determined by hereditary factors, previous experiences, and the demands of the current environment in which the individual acts. Kolb’s perspective is a constructivist one, hence his emphasis on the experiential nature of learning, understanding it as a knowledge-building process with a creative tension among the four modes mentioned.

Alonso and Honey, the same as Kolb, defend that the best learning occurs when students go through four cyclical phases, and they defined the following learning styles based on such phases: Active, Reflective, Theoretical, and Pragmatic [31].

To survey student profiles, the CHAEA questionnaire [33] is recommended. It allows taking a snapshot of the learning style of a single student or a group of students. This is significant information for future decisions that have to be made during the design phase, when the designer of the LO will decide if the predominant learning styles are to be favored, or if activities will be designed to take into account all four learning styles. Figure 3 shows the learning style profile model of a group of students.

3.2 Design

During this phase, the LO is designed both from a pedagogical perspective as well as a technological perspective. Design aspects that affect the educational and technological components of the LO are developed as shown in Table 3.

---

TABLE 3: Template for the Learning Objects Design Stage

---

To be able to better understand the criteria used in the design of the latter template and for the sake of its use, two main components have been taken into account, 1) Instructional Design and 2) Multimedial Design.

3.2.1 Instructional Design

Instructional design involves defining aspects pertaining to the internal structure of the LO. In accordance with the definition adopted in this article, an LO is mainly made up of four elements, a learning objective, contents, activities, and a self-evaluation. ]]> Learning Objective: a LO should have a single learning objective. Taxonomies that allow identifying progressive complexity levels of cognitive operations are used to define LOs, using verbs that are associated to the cognitive category that is to be described. One of the recommended taxonomies to do this is the one proposed by Anderson and Krathwohl, which is an adaptation of Bloom’s taxonomy levels [34], and which defines the following 6 taxonomic levels:

1. Remember: recognize, list, describe, identify, recover, locate.
2. Understand: interpret, summarize, infer, classify, compare, explain, and translate.
3. Apply: execute, implement, perform, use
4. Analyze: differentiate, distinguish, select, integrate, structure, outline.
5. Evaluate: review, critique, experience, judge, test, detect, monitor.
6. Create: design, build, produce, conceive, trace, elaborate, generate, invent.

To formulate the learning objective, one of the verbs listed is used in combination with the specific contents that will be covered by the LO, and then the conditions or circumstances in which the student is expected to perform are added. Figure 4 shows an example.

1. Content selection: it allows defining what is going to be included in the LO and what is not. Not all knowledge related to the topic covered by the LO is relevant for inclusion; the learning objective defined at the beginning of the instructional design, and its degree of specificity in particular, determines the answer to the question ”What should be included?”.
2. Organization: it allows identifying the appropriate way of organizing the various types of contents (conceptual, procedural, conditional). In the case of conceptual contents, it is recommended to organize them hierarchically, starting from the more general concepts in the topic, then moving to more specific ones, and finally introducing specific examples, always following a deductive logic. See Figure 5.

   ---

   
   

   Fig. 5: Example of Deductive Logic

   ---

   In the case of procedural contents (processes, techniques, skills, etc.), it is recommended to first introduce the task or end result of the procedure so as to provide context for the information, and then go into the details for each of the steps, in the corresponding order. See Figure 6.

   ---

   
   ]]> Fig. 6: Example of Procedural Contents

   ---

   Conditional contents are the criteria used by experts to make decisions - what, when and how to use certain pieces of knowledge (conceptual/procedural) to solve a case, situation or problem. When emphasis is on solving problems, it is recommended that the problem or situation to be solved be used as structural platform around which contents are organized, in order to favor content transfer and application. See Figure 7.

   ---

   
   

   Fig. 7: Example of Conditional Contents

   ---

3. Content presentation: How contents are presented is also key, especially when dealing with digital materials. Mayer’s SOI (Select, Organize, Integrate) model [35] is proposed, which was designed to emphasize these cognitive processes; see Figure 8.

   ---

   
   ]]> Fig. 8: Contents Presentation

   ---

Activities: Activities should be selected based on: learning objective, type of contents, target audience, purpose of the activity, and learning processes it stimulates. There are four different types of activities [22]:

1. Diagnostic: They are aimed at identifying previous knowledge, learning about what students know and what they do not know, and activating what they do know before the learning process begins.
2. Motivational: They are aimed at generating reasons to learn.
3. Comprehension, application, transfer: They are aimed at processing, organizing, storing, and using the information.
4. Integration: They are aimed at promoting the interrelation, integration, and summarization of the various topics studied.

When adjusting the design of the LO to the specific needs of the target students and using Kolb-Honey and Mumford proposal in relation to learning styles, focus is on the activities that must be included to accommodate the different learning styles. The learning styles model created by Kolb assumes that, to learn something, the information received must be used or processed. To Kolb, the optimal learning is the result of working with the information in four phases: acting, reflecting, theorizing, and experimenting. See Figure 9.

These are possible starting points:

• A direct and concrete experience: active student.
• An abstract experience, which is the kind of experience obtained when reading about something or hearing someone talk about it: theoretical student.

All experiences, both concrete and abstract, become knowledge when they are elaborated upon in either of these ways:

• By reflecting upon and thinking about them: reflective student.
• By actively experimenting with the information received: pragmatic student.

It is recommended to introduce a brief initial activity as a learning trigger that favors the active style, then a reflective activity that allows developing a deeper learning, followed by a third activity that provides students the tools needed to theorize over the topic, and finally a fourth activity that encourages the application of the new knowledge in a contextualized situation or problem.

Self-evaluation: Self-evaluation allows establishing if the learning objective has been achieved, generating a reflective instance over what has and what has not been learnt. For the self-evaluation stage, the following criteria should be taken into account:

1. Analyzing the difficulty of the items proposed taking into account the level of knowledge and understanding.
2. Designing random open or closed questions and tasks; these can be configured considering number of attempts, time available to provide an answer, additional time bonus, etc.
3. Preparing instructions, questions and options for students to have an advantage that is not linked to knowledge.

As regards student feedback, when they provide a correct answer, additional information should be provided and they should be redirected to other sources. If they provide an incorrect answer, hints or clues should be provided to guide new attempts, the reasons why the answer is wrong should be explained, and the students should be invited to review certain topics or they should be redirected to other sources.

3.2.2 Multimedial Design

Interface Design: the design of the interface for the LO must be specified; templates that apply metaphors in relation to the topic can be used. This will be dependent on the tool that is later on used to implement the design of the LO. It is recommended to use Cascading Style Sheets (CSS) or templates developed to support Web pages (HTML).

Layout Structure: LO layout structure must be specified; one of the four designs mentioned in this work must be selected, such as:

1. Navigation block on the left
2. Navigation block on top
3. Navigation block to the right
4. Combined block

Considering this, interface design can be based on a structure that shows the navigation block on the left. See Figure 10.

Path: content organization should follow a sequence going from known to unknown, from immediate to medium-term, from concrete to abstract, and from easy to difficult. For instance, a simple structure can be used that allows organizing target knowledge and skills into a hierarchy; or a branched linear structure where users can follow different routes based on their interests or previous knowledge.

3.3 Implementation

The purpose of this phase is implementing the LO that was designed in the previous stage. Therefore, it is important to take into account the following references to implement the ”core´´ of the LO and to label it.

Computer tools: it is recommended to use authoring tools that allow integrating each of the elements considered in the design. In this proposal, the eXelearning tool allows implementing the designed LO. Table 4 shows the review of 6 tools that allow creating ”learning activities´´ that can be used and integrated into eXelearning.

---

TABLE 4: Computer Tools Comparative Chart

---

Metadata: after implementing the LO, its metadata card must be filled in; this information will be used to describe and then locate the LO in a Learning Objects Repository (LOR). To fill in the metadata, the eXelearning 2.0 tool allows generating them using the LOM (Learning Object Metadata) standard or the DCMI (Dublin Core Metadata Initiative) standard. Figure 11 shows a metadata input card using the DCMI standard. ]]>

3.4 Evaluation

This phase is focused on evaluating the LO considering two perspectives:

1. From the standpoint of the user as consumer of the LO (student): users participate to determine if the LO really helped them learn. The CUSEOA (LO Student Satisfaction Questionnaire) questionnaire was used to gather this information; this questionnaire asks about the global reaction to the LO and then identifies its collaboration from a pedagogical and technical perspectives [36].
2. From the standpoint of the producer of the LO (educator): a digital educational material evaluation signature called CODA is used to gather the minimum set of criteria from LO quality models from a didactic and technological perspective [37]. It should be noted that, at this point, an evaluator can be an academic peer who is not necessarily familiar with the LO target topic.

3.5 Publication

The purpose of this phase is to package the LO using a standard that allows publishing it on different TLVEs. To achieve this, it is recommended to use a formal standard and to consider that the LO can be stored in a LOR so that it can be searched for and retrieved from. Deploying the LO in a TLVE and Publishing the LO in a LOR will be now explained in further detail: ]]> Deploying the LO in a TLVE: to deploy a LO in a TLVE, the following steps must be carried out: a) Package the LO using a standard that can be recognized by a Web environment (SCORM, IMS). Reload Editor is recommended to package LOs. b) Deploying the LO in a TLVE that is supported by the standard used to package it. See Figure 12.

Publishing the LO in a LOR: the final step after producing and evaluating the LO is publishing it and making it available to the students through a LOR.

4 Case Study

To assess DICREVOA, a case study was used, since it is an empirical method that allows analyzing phenomena in their own context. This method is used when there is no marked boundary between phenomenon and context, or when there is a lack of experimental control and information is gathered from a few entities [38]. This case study was carried out with 3 groups that were trained on the design and creation of LOs using the DICREVOA methodology. Each group took 16 hours of training - Group 1 included educators from a university and several high schools from the same city; Group 2 and Group 3 included several educators from high schools and universities from various cities, in the context of two post-grade courses offered by 2 universities. Table 5 shows the characteristics of each group.

]]>

---

TABLE 5: Groups

---

Group 1 members were secondary- and tertiary-level educators; they had little theoretical knowledge about LOs and almost no experience in the design and creation of LOs. Group 2 members were secondary- and tertiary-level educators; they had little theoretical knowledge about LOs and no experience in the design and creation of LOs. None of the participants had professional degrees related to the area of Computer Science; however, they did in the area of Educational Sciences. Group 3 members were secondary- and tertiary-level educators and they neither had theoretical knowledge about LOs nor experience in their design and creation. All of the members in this group had a professional degree - either a bachelor degree or a degree in computer engineering.

The case study was used to determine the advantages of using DICREVOA in contexts where users have different types of profile (technological and educational). It would also allow analyzing the feasibility and possibilities offered by the methodology developed to help educators design and create LOs and incorporate them to the teaching and learning process. To carry out the case study, a protocol containing the activities was developed:

1. Case definition
2. Case design and planning
3. Preparation for data collection
4. Data collection
5. Analysis and interpretation of the data collected
6. Results report

For the data collection stage, two questionnaires were administered at two different points in time (before training and after it) in each of the groups that took the training course (Group 1, Group 2 and Group 3) and passed it. The questions included in the questionnaire were reviewed and validated by academic peers that are content experts. Answers were collected using Google’s online survey system. The questionnaires used in this research can be found at:

Pre-course questionnaire:

• http://ow.ly/YfABj

• http://ow.ly/YfB9a

As regards the questions asked, the following results were obtained:

• Is the definition of LO provided by the DICREVOA methodology appropriate?
  • For Group 1: 86.6% of the participants considered that the methodology provides an appropriate definition for LOs.
  • For Group 2: 92.5% of the participants considered that the methodology provides an appropriate definition for LOs.
  • For Group 3: 100% of the participants considered that the methodology provides an appropriate definition for LOs.
• About the concept of LOs.
  • For Group 1: in the diagnostic questionnaire, 90% of the educators indicated that they did not know what LOs were; however, after taking the course, 90% of the participants were able to define them.
  • For Group 2: in the diagnostic questionnaire, 100% of the educators indicated that they did not know what LOs were; however, after taking the course, 85.71% of the participants were able to define them.
  • For Group 3: in the diagnostic questionnaire, 100% of the participants indicated that they did not know what LOs were; however, after taking the course, 93.3% of the participants were able to define them.

  It can be inferred then, that before taking the course, almost no one knew what LOs were but this paradigm significantly shifted after they took the course. See Figure 13.

  ---

  
  

  Fig. 13: Results about familiarity with the concept of LOs before and after the course

  ]]>

  ---

• About the instructional design of the LO
  • For Group 1: 80% can define it, 16.6% can define it to a certain extent.
  • For Group 2: 64.3% can define it and 35.7% can define it to a certain extent.
  • For Group 3: 80% can define it and 16.7% can define it to a certain extent.
• To what extent do you consider that this is a recommendable methodology for designing and creating LOs?
  • For Group 1: 33.3% highly recommendable and 66.7% recommendable.
  • For Group 2: 57.1% highly recommendable and 42.9% recommendable.
  • For Group 3: 83.3% highly recommendable and 16.7% recommendable.

  It can be inferred that the methodology had a positive impact on participants, since no one was indifferent to it or found it to be little recommendable or not recommendable at all. See Figure 14.

  ---

  
  

  Fig. 14: Results about potentially recommending the DICREVOA methodology

  ---

• Which is your global score for the methodology used?
  • For Group 1: 60% considered it to be useful and 40% considered it to be very useful.
  • For Group 2: half of the participants (50%) considered it was useful and the other half considered it was very useful.
  • For Group 3: 16.7% considered it to be useful and 83.3% considered it to be very useful.
• How do you assess the balance between the theoretical and the practical components developed in this course? ]]> The purpose of this question was to avoid the influence that the trainers might have when teaching the methodology; the ends of the scale were ”very theoretical” and ”very practical”.
  • For Group 1: results showed that the training course was very practical for 33.3% of the participants, not practical enough for 20% of the participants, and balanced for the remaining 46.7% of the participants.
  • For Group 2: results showed that the training course was very practical for 28.6% of the participants, not practical enough for 35.7% of the participants, and balanced for the remaining 35.7% of the participants.
  • For Group 3: results showed that the training course was very practical for 80% of the participants, not practical enough for 3.3% of the participants, and balanced for the remaining 16.7% of the participants.

5 Discussion

Based on the gathered results from the case study, it is possible to state that the definition adopted about what an LO means, allows us to understand its nature, and most importantly, it allows us to make the creation of digital educational material, based on this paradigm, feasible, through DICREVOA.

In Figure 14, it can be clearly seen that the percentages obtained from the three groups are uneven, in relation to what extent DICREVOA is recommended to design and create LOs. Group 1 states that DICREVOA is strongly recommended, as seen in its 33.30%. On the other hand, groups 2 and 3 show an increase in their percentages. They strongly recommend it in a 57.10% and in an 83.30%, respectively. The variability of the percentages can be explained by the context predominantly (educational or technological) of the coming course participants of each group and the orientation (techno-pedagogical) DICREVOA has.

Regarding the results of how practical the training that group 3 (83.3% - more technological context) as well group 2 (28.6% - more educational context) had, it could be thought that a strong feature of DICREVOA is the instructional proposal presented when designing the LO. It is noteworthy that DICREVOA, is the result of the analysis and incorporation of the best practices that have taken a number of methodologies to create LO during the last 15 years.

The paradigm of LO is still a matter of discussion and despite of not reaching agreement within the scientific community in its definition, several proposals for its creation have been generated, which try to be de facto. A problem that is evident in these proposals and that has been addressed in this work, is the orientation they usually have, since depending on the context they come from, they focus on exploiting either more technological implementation of LO as software artifact leaving out consistent educational design which the LO should have to fulfill the purpose for which it was created. In other cases, the focus is on instructional design detail so that it is not possible its implementation through software tools. In addition, the size a LO should have is another opinion that varies among different proposals, where larger LOs are less reusable in different educational contexts, and smaller Los tend to be more reusable in different educational contexts and this feature also allows them to be combined with others to form larger units.

However, despite the amount of work done in relation to this paradigm, there are still some questions that should be addressed in the future. One has to do with the way LOs should be assembled to generate larger units and the second question addresses the fact that the proposed instructional design of this methodology can be scaled to design e-learning courses, such as MOOCs.

6 Conclusions

The production of digital educational material under the Learning Object paradigm is still a challenge on tertiary education both for educators as well as production teams, since creating LOs involves design logics for technical and educational reutilization. When facing the design of a LO, it is important to have a practical definition as starting point that allows understanding the characteristics of LOs. From a technological point of view, the LO must be considered as a software artifact. It must also be updatable, both in relation to software as well as contents. The LO must be stored in a LOR (published) and it must be described by means of metadata (locatable). It must be possible to deploy it on different TLVEs (technological reutilization) using packaging standards (interoperable). ]]> From an educational point of view, the LO must be considered as a teaching and learning aide, with an educational intent (educational design to favor learning) aimed at facilitating comprehension, the representation of a concept, a theory, a phenomenon, etc., and promoting the development of skills, abilities and competencies in individuals. The LO must also have an internal structure or components (objective, content, activity, evaluation) (granularity) to favor the possibility of integrating the LO into larger collections, such as lessons, courses, etc. (generativity).

The DICREVOA methodological proposal presents 5 phases aimed at guiding educators through the design and creation of LOs. To achieve this, the analysis phase allows surveying the needs that will be covered by the LO; the design phase is aimed at providing the necessary guidelines to consider both the educational and technological perspectives; the implementation phase allows creating the LO using the eXeLearning authoring tool that facilitates both the creation of the LO and the provision of its metadata; the evaluation phase allows evaluating the LO from the point of view of the educator as producer and the student as consumer; and the publication phase allows deploying the LO on a LOR or a TLVE.

The case study presented allowed testing the feasibility and usability of the DICREVOA methodology, which provides an accessible mechanism to help educators make appropriate decisions when creating LOs that are of good quality but, above all, that are suitable for their needs.

Finally, this study makes evident that the creation of LOs from a ”de facto´´ definition provides teachers understanding of this paradigm from a techno-pedagogical position to create digital educational material.

Through the methodology presented in this paper, DICREVOA, teachers will have in their hands a tool for analyzing initial requirements that cause the need to create a LO, seeking to balance the design of LO in both the educational and technological scope, trying to meet various learning styles through the cycle of Kolb and bringing it into practice through the use of the eXelearning tool for structuring the conceptual basis of a LO, easing the loading of metadata. The given templates CUSEOA and CODA allow teachers a first approach to the evaluation of this type of MED in order to be evaluated by teachers and students.

As future work, we expect to demonstrate the efficacy of DICREVOA through new experiments. To achieve this, there will be groups of students that will use the LO and will assess if it helped them achieve the learning objective proposed or not. We are also planning to promote DICREVOA to be used by educators with various competencies, be these educational and/or technological.

7 Acknowledgments

This study has been funded by the Research Direction of the University of Cuenca (DIUC) through research project ”Assessing the impact of a methodology to be developed for the creation of quality digital educational material when applied to educators at the School of Engineering of the University of Cuenca´´ and project ”Assessing a flipped learning model based on Learning Objects (LOs) to reduce university student failure,” for which we are truly thankful. We also thank all the individuals and universities that participated in the training course given as part of this methodological proposal and who answered our surveys.

References

]]>

[1]   F. García, J. Portillo, and M. Benito, “Nativos digitales y modelos de aprendizaje.” SPDECE, 2007.

[2]   D. Álvarez, J. Mendoza, and J. Kelsy, “Diseño e implementación de objetos de aprendizaje basado en tecnologías estándares,” ., . [Online]. Available: http://aves.edu.co/ovaunicor/webroot/recursos/1/index_DIOTE.pdf

[3]   G. Astudillo, C. Sanz, and P. Willging, ”Análisis del estado del arte de los objetos de aprendizaje. Revisión de su definición y sus posibilidades”. Universidad Nacional de La Plata, La Plata, Buenos Aires., 2011. [Online]. Available: http://sedici.unlp.edu.ar/ARG-UNLP-TPG-0000002954/12061.pdf

[4]   J. L’Allier, “NETg’s precision skilling: The linking of occupational skills descriptors to training interventions,” netg, 1998. [Online]. Available: http://www.netg.com/research/pskillpaper.htm

[5]   C. Barritt, D. Lewis, and W. Wieseler, “Cisco systems reusable information object strategy. definition, creation overview, and guidelines,” Cisco Systems, Inc., 1999.

[6]   IEEELTSC, “Draft standard for learning object metadata,” New York: Institute of Electrical and Electronics Engineers, Inc., 2002.

[7]   D. Wiley, “Connecting learning objects to instructional design theory: A definition, a metaphor, and a taxonomy,” The Instructional Use of Learning Objects: Online Version, 2000. [Online]. Available: http://reusability.org/read/chapters/wiley.doc

[8]   D. Wiley, S. Waters, D. Dawson, B. Lambert, M. Barclay, D. Wade, and L. Nelson, “Overcoming the limitations of learning objects,” Journal of Educational Multimedia and Hypermedia, vol. 13, no. 4, pp. 507–521, 2004.

[9]   L. García Aretio, “Objetos de aprendizaje. características y repositorios,” Bol. Electrónico Not. Educ. Distancia BENED, 2005. [Online]. Available: http://www.uned.es/catedraunesco-ead/p7-04-05.htm

[11]   A. Chiappe, Y. Segovia, and Y. Rincon, “Toward an instructional design model based on learning objects,” Educational Technology Research and Development, vol. 55, no. 6, pp. 671–681, 2007.

[12]   J. Silva Sprock, C. Ponce Gallegos, and Y. Hernández, “Estado del arte de las metodologías para el desarrollo de objetos de aprendizaje,” Conferencias LACLO, vol. 4, no. 1, 2013.

[13]   S. Martínez Naharro, P. Bonet, P. Cáceres, F. Fargueta, and E. García, “Los objetos de aprendizaje como recurso de calidad para la docencia: criterios de validación de objetos en la universidad politécnica de valencia,” in IV Simposio Pluridisciplinar sobre Diseño y Evaluación de Contenidos Educativos Reutilizables (SPDECE 2007), 2007.

[14]   J. Delgado, J. Valdivia, R. Morales, S. González, and M. Chan Núñez, “Desarrollo de objetos de aprendizaje a basado en patrones,” Presentado en Virtual Educa Brasil, 2007. [Online]. Available: http://e-spacio.uned.es/fez/eserv/bibliuned:19220/n03delgado07.pdf

[15]   J. Monsalve Pulido and F. Aponte Novoa, “MEDEOVAS-metodología de desarrollo de objetos virtuales de aprendizaje,” Conferencias LACLO, vol. 3, no. 1, 2012.

[16]   A. Barajas Saavedra, J. Muñoz Arteaga, and F. Álvarez Rodríguez, “Modelo instruccional para el diseño de objetos de aprendizaje: Modelo MIDOA,” Presentado en Virtual Educa, Brasil, p. 11, 2007.

[17]   L. Blondet Baruque and R. Nascimento Melo, “Learning theory and instruction design using learning objects,” Journal of Educational Multimedia and Hypermedia, vol. 13, no. 4, pp. 343–370, 2004. [Online]. Available: http://apan.net/meetings/busan03/materials/ws/education/articles/Baruque.pdf

[18]   M. Medina Balda and M. López López, LOCoME: Metodología de Construcción de Objetos de Aprendizaje. Universitat de Oviedo y REDAOPA. Actas del III Simposio Pluridisciplinar sobre. Diseño, Evaluación y Descripción de Contenidos Educativos Reutilizables (SPDECE), 2006. [Online]. Available: http://spi03.sct.uniovi.es/moodle_cv/mod/resource/view.php?id=233

[19]   R. Correal Cuervo, C. Montañez Torres, and L. Bernal Zamora, “UBoa - metodología para la creación de objetos de aprendizaje de la universidad de boyacá,” Universidad de Boyacá, 2009.

[21]   C. Castro, M. Serna, and R. Taborda Botero, “Ingeniería de software para desarrollar objetos de aprendizaje - ISDOA,” in EATIS ’12: Proceedings of the 6th Euro American Conference on Telematics and Information Systems. New York, NY, USA: ACM, 2012, pp. 347–353.

[22]   C. Sanz, F. Barranquero, and L. Moralejo, “Curso de doctorado metodología CROA,” Universidad Nacional de la Plata, 2014.

[23]   G. Padilla and M. Hernández, “Metodología -UAT: Una metodología para el diseño de oa, en avances en objetos de aprendizaje. experiencias de redes de colaboración en méxico, editado por f.j. Álvarez y jaime muñoz.” Eduweb, vol. 5, no. 1, pp. 47–70, 2011.

[24]   J. Muñoz, B. Osorio, F. Álvarez, and P. Cardona, “Metodología para elaborar objetos de aprendizaje e integrarlos a un sistema de gestión de aprendizaje,” Revista Apertura del Sistema de Universidad Virtual, Universidad de Guadalajara, México. ISSN 1665 - 6180, 2006. [Online]. Available: http://ingsw.ccbas.uaa.mx/sitio/images/investigaciones/13TEMunozArticulo.pdf

[25]   V. Štuikys and R. Damaševičius, “Towards knowledge-based generative learning objects.” Information technology and control, vol. 36, no. 2, 2015.

[26]   J. Maldonado, J. P. Carvallo, and J. Siguencia, “Metodologías y propuestas metodológicas para el diseño de objetos de aprendizaje: Un estado del arte en iberoamérica,” Conferencias LACLO, vol. 6, no. 1, 2015.

[27]   Y. Collaguazo, L. Chamba-Eras, and A. Padilla, “Propuesta de un modelo genérico para el diseño y valoración de objetos de aprendizaje basado en estándares e-learning,” Conferencias LACLO, vol. 6, no. 1, 2015.

[28]   J. Maldonado, A. Fernández-Pampillón, and C. Sanz, “Analysis framework for tailored selection of learning objects methodologies.” The International Workshop on Collaboration and e-Learning, CTS 2015, Atlanta, Georgia, Estados Unidos. (In Press), 2015.

[29]   P. L. Smith and T. J. Ragan, “Instructional design,” Wiley New York, p. 3, 1999.

[31]   C. Alonso, D. Gallego, and P. Honey, “Los estilos de aprendizaje. procedimientos de diagnóstico y mejora,” Ediciones Mensajero. Universidad de Deusto. Bilbao, 1994.

[32]   A. Y. Kolb and D. A. Kolb, “The kolb learning style inventory,” www.learningfromexperience.com, Recuperado el 4 de Julio de 2014, 2005.

[33]   P. Honey and C. Alonso, “Baremo cuestionario chaea de estilos de aprendizaje.” [Online]. Available: http://www.estilosdeaprendizaje.es/menuprinc2.htm

[34]   L. Anderson and D. Krathwohl, “Revision of bloom’s taxonomy of educational objectives. en a taxonomy for learning, teaching and assessing.” En A Taxonomy for Learning, Teaching and Assessing. Longman, New York., 2001.

[35]   R. E. Mayer, “Diseño educativo para un aprendizaje constructivista,” in Diseño de la instrucción: teorías y modelos: un nuevo paradigma de la teoría de la instrucción. Santillana, 2000, pp. 153–172.

[36]   S. Massa, “Objetos de aprendizaje: Metodología de desarrollo y evaluación de la calidad,” Ph.D. dissertation, Facultad de Informática, 2013.

[37]   A. M. Fernández-Pampillón, E. Domínguez Romero, and I. D. Armas Ranero, “Herramienta de evaluación de la calidad de objetos de aprendizaje universitarios COdA: Guía para la producción y evaluación de materiales didácticos digitales. v. 1.1,” ., 2011.

[38]   F. Benbasat, D. Goldstein, and M. Mead, “The case study research strategy in studies of information systems.” MIS Quaterly 11(3), pp. 369–386, 1987.