An Experimental Study of the Use of Design Thinking as a Requirements Elicitation Approach for Mobile Learning Environments
Cynara Lira de Carvalho Souza
Faculdade de Ciências Aplicadas e Sociais de Petrolina (FACAPE), Petrolina, PE, Brazil, 56328-903
Centro de Informática, Universidade Federal de Pernambuco (CIn-UFPE), Recife, PE, Brazil, 50740-560
and
Carla Silva
Centro de Informática, Universidade Federal de Pernambuco (CIn-UFPE), Recife, PE, Brazil, 50740-560
]]> ctlls@cin.ufpe.br
Abstract
Mobile learning (m-learning) is a research field that aims to analyze how mobile devices can contribute to learning. The development of software for mobile devices to support learning is essential for an effective implementation of m-learning or mobile learning environments (MLE). Requirements Engineering processes need to include activities that provoke creativity in the stakeholders to conceive MLEs that actually modify and improve the teaching and learning process. In this context, this paper presents a process for requirements elicitation and documentation of mobile learning environments. This process is based on the concepts of the Design Thinking process that provides a methodology to elicit customer needs, producing simple prototypes that eventually converge to innovative solutions. An experiment was conducted to evaluate if the proposed process contributes to create MLEs that present distinctive and interesting characteristics when compared to existing solutions for a specific problem.
Portuguese Abstract
Mobile Learning (m-learning) é o campo de pesquisa que busca analisar como os dispositivos móveis podem colaborar para a aprendizagem. O desenvolvimento de software para dispositivos móveis é essencial para a efetiva implantação do m-learning ou ambiente virtual de aprendizagem móvel (AVAM). Processos de Engenharia de Requisitos precisam incluir atividades que provoquem a criatividade nos stakeholders para criar AVAMs que de fato modifiquem e melhorem o processo de ensino e aprendizagem. Nesse contexto, o presente artigo apresenta um processo de elicitação e documentação de requisitos para ambientes virtuais de aprendizagem móvel. Este processo se baseia nos conceitos do processo do Design Thinking que fornece uma metodologia para elicitar as necessidades dos clientes, produzindo protótipos simples que eventualmente convergem para soluções inovadoras. Um experimento foi realizado para avaliar se o processo proposto contrinui para criar AVAMs que apresentam características diferenciadas e interessantes quando comparadas a soluções existentes para um problema específico.
Keywords: Mobile Learning, Mobile Learning Environments, Design Thinking, Requirements Engineering.
Portuguese Keywords: Mobile Learning, Ambiente Virtual De Aprendizagem Móvel, Desing Thinking, Engenharia De Requisitos
Received: 2014-07-29 Revised: 2015-02-26 Accepted: 2015-02-26
The mobile computing area has great exploration potential in different segments due to the large number and diversity of mobile devices. This potential can favor the development of educational applications or mobile learning environments (MLE). Glasemann et al. [1] point out that the concept of MLE or m-learning (mobile learning) have attracted interest from researchers by providing flexibility and organizational capacity, awakening the sense of responsibility, supporting and encouraging teaching practices through a pedagogical perspective. However, MLEs also inherit the problems of traditional software requirements elicitation [1].
]]> Also, MLEs actually need to impact the teaching and learning environment. To achieve this purpose, requirements engineering processes for MLE need to include activities that use creativity techniques in order to conceive solutions that really modify and improve the teaching and learning process. Thus, in [2], a requirements elicitation and specification process based on Design Thinking (DT) was proposed to develop MLEs. Also in [2], we report an application of the process in a pilot experiment. In this current paper, we report an experiment conducted in a high school in order to evaluate the applicability of the process in a real environment, and also to verify if the proposed method differs from the traditional process for software requirements elicitation. In addition, a test with users (teachers and students) was conducted to verify which produced software was considered more interesting and if their needs were met. The experiment was conducted in two steps: (i) two MLE solutions were created to a public high school. A team of developers used the proposed process and another team used a traditional process to develop a system. (ii) the prototypes developed by each team was compared. By doing this experiment, we found the same conclusions of the pilot experiment and that it is possible to perform a better requirements elicitation, since with the proposed process we can get more information from the user than using traditional processes. Therefore, there is a greater probability of fully achieving the users needs.The rest of this paper is divided as follows. In the section 2, we present a study of mobile learning environments, regarding significant topics of the process to develop such applications. The third section presents the proposed process based on Design Thinking (DT) for requirements elicitation of MLEs. The fourth section shows the application of the process in the creation of an MLE solution as a pilot experiment. The fifth section presents the experiment design with information regarding the experiment planning and execution. In the sixth section, we report the result of the experiment execution. The last section shows the final considerations and future work to be developed.
Mobile learning has emerged with the advent of distance education, primarily related to the frenetic pace of technological change. That process motivated an evolution of the traditional methods of teaching and therefore, mobile devices came to be used as a tool for the dissemination of knowledge.
Corroborating to that assertion, Carvalho [3] states that the increasing proliferation of wireless networks and mobile devices (cell phones, iPods, tablets, etc.) enabled the development of a new type of distance learning: the m-learning.
Mobile learning refers to exploration of personal communication technologies, focused also on the availability and immediacy of information. So, learning with mobile devices takes advantage of the permanent connectivity provided by wireless networks and everyday user suitability [4].
A study made by the Open University of Brazil (Universidade Aberta do Brasil – UAB) demonstrated that about 94% of students agreed with the creation of courses based on m-learning. Other students suggested the use of mobile phones for more practical learning. These topics include simulation of environments or situations, such as tests and the creation of virtual laboratories and classrooms [5].
M-learning is a very promising area, being described by some authors as a current and modern trend. Mobile learning has the potential to utilize personal and portable technologies for effective education. The term also encompasses technologies that enable learning in an increasingly mobile society, taking into consideration the technological advances in the development of such devices [6].
Maiden et al. [7] said that creativity is a multidisciplinary field of research that has been investigated from the perspective of design, art, psychology, literature and other areas. In recent years, several authors have emphasized the importance of viewing requirements engineering as a creative problem-solving process.
In a systematic review reported in [7], the authors analyzed 46 papers to identify trends and opportunities for research the application of creative approaches to support requirements engineering. This study confirmed that creativity techniques improved the creative thinking in the requirements activities and which strategies of creative thinking should be incorporated in requirements engineering processes, methods and tools [7].
Oliveira et al. [8] proposed to elicit requirements for educational software based on a feature-driven methodology, called feature tree. In this context, a feature is a function with value for a client that should be constructed and included in the system. A feature is expressed by the client in a natural language, which facilitates communication between those involved in the creation of the system. According to the authors, modeling requirements as feature tree stimulates creative thinking and brainstorming. Brainstorming is a creative technique, which is dedicated to combine, extend and encourage unconventional ideas, focusing on the quantity of ideas [8].
]]> Maiden et al. [9] say that there are some exploratory creativity techniques that are available to improve creative thinking related to the requirements process, such as scenarios and use cases. Stakeholders are encouraged to find or create a fictitious story related to the current problem they are facing and this story may reveal the stakeholders’ motivations, desires and needs that other techniques were not able to gather.Therefore, after several authors argue about the use of creativity techniques in requirements engineering, Marçal et al. [10] reports that over 30% of the problems found in software development are related to problems in the requirements phase, caused either by requirements changes or by the lack of user’s involvement. Including creativity techniques in the requirements elicitation process can encourage users to explore creative thinking to find innovative ideas that create new solutions.
In this paper, we propose to include creativity techniques in the requirements elicitation process for developing MLEs. Such techniques will be overviewed in the next section.
Design Thinking is a method designers utilize to think. It can be described as mental processes designers use to create objects, services or systems that result in elegant and useful products [11].
According to Soledad et al. [12], DT is a set of user-centered techniques and tools that supports an iterative process to produce, analytically and creatively, solutions to real challenges. As an example, they report an experience in refining requirements for a learning management system. They involved the users in the process by using DT. As a result, the authors considered adequate to apply DT in the refinement of requirements in this context.
The design thinking process can be divided in three phases. The first phase is called immersion and it is charge of gather, analyze and summarize data. The second phase is called ideation and it is responsible for defining the profile of the users that will collaborate in the creation of the solution based on innovative ideas; and the third phase is called prototyping, able to represent the ideas concretely to promote the validation of the proposed solution [13].
Verterlli et al. [13] propose to use Design Thinking into requirements engineering processes, since DT provides a methodology that provokes identifying the customers’ needs, rather than requirements, and produces fast and simple prototypes that eventually converge to innovative solutions. DT is consistent with early stages of requirements elicitation, quick prototyping and customer engagement, as recommended by the agile development methods.
In this context, the process defined in this paper to develop MLEs is based on Design Thinking in order to include the final user (students and teachers) as an active agent in the creation process of new solutions for mobile learning.
A great challenge to improve the teaching and learning process is to make it more interesting. Technology can assist this process by making it more motivating, enjoyable and pleasurable. Our proposal consists of requirements elicitation process aimed at creating mobile software solutions that really impacts in the teaching and learning process.
Figure 1 shows the proposed process base on Design Thinking for creating mobile learning environments. The process is represented graphically by using BPMN (Business Process Modeling Notation) [14].
]]> The process shown in Figure 1 is divided into three sub processes detailed below. Figure 2 shows the sub process called immersion.
The activities performed in this subprocess intent to gather information provided by users, as well as analyze, identify and organize this information in order to discover the users real needs. The Immersion subprocess is detailed in four steps, as detailed in the following.
1) Gather ideas - This step is carried out with the help of a creative technique called brainstorming [8] and the participants are users that belong to the context of m-learning (students and teachers). This technique was applied and the following question was asked: How can mobile technology collaborate in the learning process? From this question, students and teachers wrote their ideas on insight cards. Insights are real reflections based on solid data from exploratory research. Thus, the observations of the participants are registered into cards that facilitate fast consultation and handling. It usually contains a title, that summarizes the findings, and the original text collected with the question [15]. According to [15], " an idea is solution generated to meet one or more insights and an insight is a finding from the immersion, i.e., the identification of an opportunity".
Figure 2: Immersion Subprocess
2) Analyze ideas - In this step, the information collected in the insight cards are analyzed using the NVivo tool [16]. This tool was chosen due to its easiness and also because it is a tool known in the scientific community. This tool also works with qualitative analysis of texts. Thus, "words" taken from the texts written by students and teachers and they are categorized with the help of NVivo. These "words" are viewed as ideas. The most mentioned and similar ideas represent the real needs of the participants.
The basic structures used in NVivo are "sources", "nodes" and "ratings". These structures are used to insert, organize and classify research material [16]. The sources refer to our research material, and it can be divided into three techniques:
Internal Source: it refers to the central research material imported or created in the software. It includes documents, videos, audios and images;
]]> External Source: it relates to those materials that cannot be imported, but are important to the research. Includes websites, PowerPoint presentations, books, newspaper articles, etc.;Memos: they are summaries and reflections that can be associated with sources or nodes.
The nodes are used to codify the research material, they are "containers that store the encoding, i.e., nodes contain a reference to a portion of the encoded text". Nodes can refer to topics, people, organizations, etc. The encodings are reference indexes added to portions of the text, and it involves analysis of the material thus generating ideas and thoughts related to it, which will form the nodes. A particular document, such as an interview, can be encoded in different nodes [16]. Therefore, in this step, from insight cards previously gathered, information is registered as internal sources, from which nodes are “extracted” from the ideas generated by the participants. These nodes we will refer to the most cited topics that will work as a codification as mentioned above in the text. In this step, a reflection regarding the analyzed material is done to create ideas and thoughts that will origin the nodes. These nodes will work as key points to detect the main need of users.
3) Identify ideas and Organize ideas –The steps of identification and organization of ideas happens at the same time. The ideas were identified and then ideas were organized based on previous analysis, mapping the most creative ideas and separating them from the less creative ones. A Mind Map is created in which on its left side are recorded the less creative ideas and on the right side, the most creative ones. The most creative ideas are those preferred by a higher number of users. The mind map diagram works as a graphic representation of how ideas are organized around a particular focus [15].
The ideation sub process is shown in Figure 3. The purpose of this subprocess is to focus on the most creative ideas gathered from users, for identifying requirements and characteristics to create an educational software with a great potential to facilitate the teaching/learning process with the support of mobile technology.
4) Choose Ideas - this step is performed based on the mind map generated in the previous activity. According to the degree of preference and similarity of users’ needs/desires, an idea is chosen with the assistance of NVivo. The most mentioned ideas considered the most relevant need, thus indicating the problem to be solved.
5) Create Profile – users profiles are created and related to a specific idea. A creative technique called Personas [17] is used. This technique uses fictitious persons to represent users of a service or product. The PERSONAS are embedded in scenarios, stories that represent actual situations. In this activity, personas were created for teachers and students. To collaborate with the identification of personas, exploratory research and interview techniques were used [15].
Figure 3: Ideation Subprocess
Interviews - The interview is a method to seek, in a conversation with the interviewee, relevant information through answers to questions. Interviews are particularly useful to obtain the story behind the life experiences of the interviewee. The researcher meets the interviewee and talk about relevant issues following a predetermined sequence of questions that can be changed depending on the efficacy of the conversation. Thus, it is possible to identify information that will assist in the development of Persona and then providing inputs for generating more ideas for the solution. In our research, the following questions have been defined to assist in this step:
How is student life?
Do students use technology?
Do they share information with colleagues?
Would you like to have an application to collaborate in learning?
How this application could help?
6) Define requirements - in this step, requirements are defined based on the characteristics of PERSONAS and from the information generated in NVivo. All the information related to the most referenced "keyword" will be used to identify the essential requirements for creating the software.
The details of the prototyping sub process are presented in Figure 4. The objective of this phase is to transform requirements and characteristics into a real solution. Also, the sup-process includes a task to check with the users if the solution meets their needs or if it needs to be refined or improved.
]]>
7) Define Prototype – In this step, abstract ideas become concrete. The requirements elicited are represented as a prototype of the software-to-be. It is required to have the support of a prototyping tool.
8) Refine prototype - This step is performed by observing the users (students and teachers) interacting with the prototype. We analyze the common users’ suggestion for improving the prototype using the technique of destructive/constructive brainstorming. The destructive/constructive brainstorming technique helps to detect negative and positive aspects of products, systems, projects, etc., and then propose solutions to those aspects [18].
Therefore, by incorporating the DT methodology in a requirements elicitation process for MLE, we recognize the participation of the users during the development of a solution as an effective strategy. Thus, students and teachers can be active agent in the creation of a solution. In the next section, we present the application of the proposed process to illustrate its use in a real context.
The process was applied to a chemistry course that works with the preparation of students for tests. To participate in the experiment, we selected 120 students whose age range from 16 to 30 years, including both sexes.
Step i (Gather ideas) - We applied the brainstorming technique. We questioned them and explained to the group of students and teachers the need to make a survey. We gave them 120 insights cards containing title, subject and theme to be explored. Initially, before the experiment, the group was receptive, but at time we delivered the card, only 35 students collaborated with our experiment. Some said they didn’t have any idea to share at that time, others asked for more time to think and others ignored the activity justifying the stress of being close to take the college entrance exam. Was also gave it to the teacher to the subject in order to identify their needs. This experiment was made in a class and it endured 45 minutes. Cards were collected and we identified 24 cards written by female students and 11 cards written by male students.
Step ii (Analyze ideas) - After collecting the data in the beginning of the experiment, the data was analyzed with the help of NVivo tool. We followed the following steps:
Registration of information gathered on the cards (see Figure 5) - The first step was registering the cards creating an encoding with the letter C plus a number, which was fixed from C1 to C24 (cards of the female students) and from C25 to C35 the cards of male students. The teacher was coded as "P01" because it was consulted only one teacher;
Creation of nodes - From the Registration cards, we created the nodes (words) related to different contents or any interesting information that could be transformed into a new theme to relate relevant information. We selected 16 themes (nodes) most mentioned by students and one the teacher mentioned to be important in teaching/learning process (see Figure 6).
]]> 
Figure 5: Registration of insights cards
Step iii (Identify ideas and Organize ideas) – The ideas were identified and based on the data analysis conducted with the NVivo tool, and the distribution was organized in a map according to the amount of information cited by the participants related to nodes. Topics related to the subject contents and students’ needs were taken into account. We split the data into two sides (columns); the themes cited by students from 1 to 22 times: on the right, we put topics that were cited above 5 times, and on the left topics cited less than 5 times as shown below on the Figure 7.
Step iv (Choose an idea) – We can observed on Figure 6 that the idea most cited by students was the creation of an application that helps to solve doubts. From that observation, where 22 citations mentioned the proposal to develop something that would meet this need, we understood that the biggest problem in that course/classis that the students don’t have a way to solve doubts. The teacher argued that this problem is aggravated due to lack of time and the student, besides time, claimed shyness and lack of access to the teacher.
Figure 6: List of most cited nodes
Step v (Create Profiles) - Based on the mentioned idea in the last step, we created students profiles and a teacher profile to explore features and requirements for the application.
Based on the data of exploratory field research and semi-structured interviews with students and teachers, we have identified some behaviors regarding perceptions and use of technology and the need to identify main characteristics of the student studying isolated materials for college entrance exam, its context and difficulty of use.
From the collected data we created a persona, a fictional character that has a name, image, and characteristics of real students. The persona has characteristics that assist researchers in understanding real users.
]]> In our experiment, we didn’t find any significant differences between males and females answers. However, as the participant public was divided by 68% of female students and 32% of male students, we decided to create a female persona. Since we had only one teacher in our experiment, we created a persona for the teacher observing his characteristics and behaviors. 
Figure 7: Mind Map that shows the distribution of ideas
The position of each subject, along these axes, allowed the identification of behavior patterns, thereby resulting in the creation of personas below:
Persona 1 can be described as it follows: she is 19. She is a student and she is preparing herself for college entrance exam. She graduated from high school. She likes to hang out with friends, go to movies, go to the mall, but spends most of her time studying. She studies takes different courses and has a busy life, her time to study at home is limited. She has many friends, and, like most teenagers at her age, she likes to keep in touch with all of them, wherever possible, through two social networks: Facebook and WhatsApp. She usually access the Internet daily. It pleases her the idea of being able to share a question with friends or even clarify that exercise directly with the teacher while studying or solving exercises at home. She demonstrated excitement about the ability to do it from anywhere and at any time whenever needed.
Persona 2 can be described as it follows: he is 55 years old. He is a chemistry teacher and he has been teaching for eighteen years. His pace of life it is very intense, since he always needs to review college entrance exams updates in order to teach his students well. He tries to meet with students when he finds an opportunity in his busy schedule to solve exercises and doubts. The persona liked the idea of receiving questions from students during class time, via mobile phone, since some of his classes have 100 students and many of them are shy and some fail to ask questions during class, doing so only when the class ends. Therefore, this method would reduce the number of students and would save him time. In his opinion, one major problem is limited availability of time to speak with students regarding topics and questions.
Step vi (Define requirements) - the requirements were defined based on the characteristics of the users profiles and on this project idea, according to information gathered in the cards. For reasons of space, only some information that was relevant to the elicitation of requirements will be presented. See Table 1 below.
Table 1: Information gathered through users cards and ranked in NVivo
Other information related to the word "doubt" was not presented due to lack of space. The word "doubt" was certainly the most referenced from the information gathered in the cards. It has emerged a category called "doubts" in which all the information related to the idea of the software-to-be was kept. Thus, since students and teachers reported their needs to clarify doubts, an application to ask questions and receive responses was conceived. Table 2 details the functional requirements identified by the RF identifier, referring to the information on Table 1 and the profiles and characteristics found in personas.
]]> Table 2: Requirements obtained
Step vii (Define Prototype) - This step was performed from the requirements definition. The low-fidelity prototype was developed using the Balsamiq Mockups [19] tool (see Figure 8). While prototyping we observed the need for another requirement, the “RF08 - See teacher Schedule”.
Step viii (Refine prototype) - This step consists of the refinement of the prototype. It was performed with the destructive/constructive brainstorming technique, where two groups were selected. The first group was composed by 4 students and the teacher; a demonstration of the prototype was presented to the group in order to get some idea that could improve or eliminate the application. The students showed a good acceptance related to the requirements and they didn’t suggest any new idea. The second group consisted of 6 students; all of them viewed the demonstration and two students suggested sending a picture of the doubt, since the times it is necessary to show the teacher the question’s resolution; another suggestion consisted in making available to colleagues the answers of the questions. Thus emerged two new requirements: "RF09 - Send doubt imaging" and "RF10 - Display replies in case" (see Figure 9).
Figure 8: Prototype
The objective of the proposal was the elicitation of requirements for MLE, enabling the implementation of strategies commonly used in DT. It was possible to describe the experience of developing an application from solutions proposed by users, which differs from the traditional process where usually the administrator or coordinator identifies the problem and the developer search through techniques to define requirements in order to solve the problem. This process achieved the duration of two weeks.
In this process students and teachers participated in the process identifying needs, proposing solutions, generating and refining requirements. The results also indicated that the application of creativity techniques in the process has brought benefits and has been validated by the number of gathered ideas. The teaching-learning process needs to implement the participation of students and teachers during the development of MLE for innovative results.
The pilot experiment was conducted to consolidate the process. In the next section, we will present an experiment conducted in a high school to evaluate the process in its ability to create innovative solutions for MLE.
According to Wainer [20], "within an experiment, the researcher seeks to identify relationships between variables in order to confirm or refute hypotheses leading to the formulation of laws and theories in general." The process was defined from the pilot study mentioned in section 3. The experiment was conducted in a different school where two groups applied each, the proposed method and traditional RE techniques to create MLE solutions.
]]> Experimental design encompasses all the process - from the statement of a problem to interpretation of the results, in order to achieve statistical efficiency in the obtained results [21]. In the next sections, we detail relevant information about the experiment execution. 
Figure 9: Refine prototype
The objective of this experiment was to compare the proposed process with a traditional requirement elicitation process to develop a solution for MLE. The solutions were analyzed by the stakeholders to evaluate what MLE met better their needs.
The experiment was conducted in two phases and involved two types of participants. Two groups of students acting as requirements engineers participated in the 1st phase of the experiment. The experimental group applied the proposed process and the control group used a process based on traditional requirements elicitation techniques (e.g. observation, interviewing and prototyping). Each process elicited needs from the stakeholders (students and teachers from the 2nd series of the 2nd and 3rd grade of high school) and produced a proper solution to meet them. In the 2nd phase, the stakeholders evaluated those two solutions to analyze which of them met their needs in a better way. A total of 13 users (10 students and 3 teachers) were involved in the evaluation of the generated solutions.
In the 1st phase of the experiment, the purpose is to evaluate the proposed process. Thus, the following variables were considered:
Systematicity – a process may consist of activities to be executed in a specific order and it must have a smooth transition between these activities;
Ease of use – activities description must be clear, complete and detailed, to avoid doubts about how each activity should be executed;
Learnability – learning to use the process should not require much training time from the development team.
Innovative product – the solution obtained with the process should be a product that doesn’t replicate ideas of other products and doesn’t be only the automation of manual activities present in the teaching and learning process.
In this experiment, different hypotheses were also defined for both phases. Two hypotheses for each variable must be defined: the null hypothesis, called H0, and the alternative hypothesis, called H1. The null hypothesis is the hypothesis of "no effect", and it is formulated with the goal of being rejected, i.e., it is the denial of the hypothesis we are trying to confirm (H1).
Thus, the following hypotheses were defined for the 1st phase:
H01 – The process isn’t systematic.
H11 – The process is systematic.
H02 – The process isn’t easy to use.
H12 – The process is easy to use.
]]> H03 – The process isn’t easy to learn.H13 - The process is easy to learn.
The purpose of the 2nd phase of the experiment is to evaluate the MLE solutions developed by both processes. Thus the hypotheses considered in this phase are:
H04: The product created with the proposed process does not present innovative features when compared to the product developed with a traditional process.
H14: The product created with the proposed process present innovative features when compared to the product developed with a traditional process.
To conduct the experiment, the procedure was divided into two phases. To perform the 1st phase, two groups composed by three students each were formed. One group was composed by Computer Science students and another group was composed by Bachelor in Informatics students. All students have already attended a software engineering course. These students were invited to collaborate with the experiment.
The experimental group was in charge to use the proposed process and received a 4h/class training. The control group was in charge to use traditional techniques and they didn’t receive training because they already studied the techniques (e.g., interview, observation, oriented survey aiming prototyping) during the software engineering course.
]]> The experiment was conducted in a public high school, so that the groups could create solutions to support teaching and learning process. The stakeholders involved in the experiment were high school students from 2nd and 3rd grades.Threats to internal validity are influences that can affect the dependent variable with respect to causality, without the researcher’s knowledge [22]. As an internal threat, we highlight the possibility of experimental and control groups work with the same stakeholders. This could affect the result obtained from the second process applied so that it could obtain a more complete set of requirements, since the stakeholders have already reasoned about the problem/solution when the first process was applied. To control this threat, each group worked with a different set of stakeholders, but these stakeholders have the same profile and were facing the same problem. Both groups applied the process in the same high school. The control group applied the traditional process to students from the 2nd and 3rd grades, but belonging to A, B and C classes, whilst the experimental group applied the proposed process to student of the same grade, but belonging to D, E and F classes.
A threat to validity detected only after the experiment was the application domain knowledge and experience of the participants in the control group. This influenced the effectiveness of requirements elicitation techniques applied. As the threat was not planned, it could not be avoided.
After applying the process, the experimental group was invited to assess the proposed process by answering a questionnaire to verify if the process is systematic, easy to learn and to use.
To evaluate the product in the second phase, another set of stakeholders was composed of students and teachers of 2nd and 3rdgrades of the high school. Ten students and five teachers were randomly selected and the same group was in charge to evaluate both solutions to achieve a balanced judgment. These stakeholders were invited to interact with the prototype to test its functionality. So, each MLE solution prototype was prepared to allow the users to simulate the use of specific tasks through execution of use cases scenarios. The following instructions were presented to the users to allow their interaction of the prototypes:
The user must interpret and perform a task following a use case scenario;
After starting the evaluation, the user cannot consult the experiment applicator;
The CAMTASIA STUDIO [23] tool will be used in order to capture audio visual information from users aiming to facilitate data collection;
The applicator will press the record button on the CAMTASIA STUDIO and will initiate the observation;
When the user completes the task execution, the applicator will stop recording on CAMTASIA STUDIO.
]]>After interacting with the prototypes (produced by the experimental and control groups), students and teachers (users) are invited to answer a questionnaire to obtain compare both solutions.
In the 1st phase of the experiment, the application of the processes was observed and each group created a requirements specification document. Each member of the experimental group answered a questionnaire in order to assess whether the process is systematic, easy to use and learn.
To perform the 2nd phase when the users were interacting with the prototypes, the screens of the prototypes were captured by the CAMTASIA tool. It is important to highlight characteristics of collecting research data from users of mobile systems. Direct observation is not always feasible, since it requires the observer to track or record the user in transit situations, alternating between public and private places. Three methods of data collection have been used by researchers in the mobile computing field: the first one is called "do it" and it is implemented when the user is asked to write some important data for the research or answer questions asked by the system; the second one is called "use it" and it is applied when the data is collected through the use of the system, such as recording user’s navigation (measurement); and the third one is called "wear it", when the data is collected by sensors and cameras that the user wears [20]. In this research, we used the "use it "method because the user used the prototypes, explored the application, and this interaction was collected by CAMTASIA tool, including detailed description of the steps performed by the user, as well as errors, questions or suggestions that appeared during the experiment.
After interacting with the MLE prototypes, the users (stakeholders) answered a questionnaire to assess their satisfaction with the MLEs experimented and to collect observations about limitations or important features found. Two types of questionnaires were prepared. The first one aims at assessing the users’ satisfaction regarding the functionality of the solutions presented. In this case, the Likert scale was used to measure opinions. This scale is widely used to assess users’ satisfaction regarding products or processes [24]. The second questionnaire consists of subjective questions to gather users’ opinions regarding the characteristics of the solutions presented. The questionnaires were answered by the participants of the experimental and control groups.
According to Wainer [20], content analysis has as its unit of analysis the message, either spoken, textual, gestural, figurative or documentary that should be analyzed qualitatively. Thus, to treat the data collected in the first phase, was analyzed qualitatively.
In the second phase, the content recorded by CAMTASIA STUDIO tool was transcribed to be analyzed qualitatively. Finally, after interacting with the MLE prototypes, students and teachers answered questionnaires to assess the solutions. Answers to the objectives questionnaires were analyzed using the statistical software R² [25] and answers to the subjective questionnaires were analyzed qualitatively [20].
The experiment was conducted in 5 weeks and the experimental group could apply the process with the active participation of users. The control group applied the process in a shorter period-3 weeks. In the first phase of the experiment, both groups accessed 6 classes of high school each -3 classes of 2ndgrade and 3 classes of 3rd grade. The experimental group delivered 215 cards to gather information from users, while the control group began observing and interviewing the users (students and teachers). Interviews were conducted with five students in each class, totaling 30, and 05 teachers. After the interview, a questionnaire was applied to the same students and teachers to gather the requirements to develop the solution.
At the school, the experimental group identified the difficulty of students and teachers to access academic content, due to the lack of a library and books. This led the experimental group to develop a solution called BIBLIVIRTI to provide academic content categorized by types. The control group used traditional elicitation techniques and, based on the activities performed in the classroom, created a solution called "Share More," which maintains information of classes, courses and lessons, that are common information used in a room classroom.
After applying the proposed process, the experimental group answered a questionnaire to evaluate the process.
The control group reported a difficulty when defining requirements through interviews and observation, and many requirements were identified because the team members have knowledge about the application domain, which somehow influenced positively the elicitation process.
To analyze the results obtained from the questionnaire, it was used the Wilcoxon-Mann-Whitney Test [25], which is a nonparametric method for comparison of paired samples [25], since the samples (answers) are independent. It was required to the same group of users to evaluate both solutions, but the answers obtained have in the analysis were placed without a relationship between groups. The same group of users evaluated both solutions, but there isn’t a relationship between the order of participants’ answers. Twenty six questionnaires were evaluated, thirteen for each solution. The analysis was divided into two stages, as described in the following.
Basically, this nonparametric test was chosen because the data do not follow a normal distribution and also they come from independent samples.
The average was calculated using the answers for each question. Solution 1 refers to the experimental group and the solution 2 refers to the control group.
Table 3 shows the average results. The weighted average was used, applying weights from 1 to 5 corresponding a rage from the most negative to the most positive answer.
Table 3: Average calculation of the users answers
After performing the average calculations present on Table 3, it was created a graph comparing the answers obtained for each question regarding the evaluation of both solutions. Graph 1 illustrates the differences and similarities of the evaluation made by users. The horizontal axis represents the questions IDs whilst the vertical axis represents the answers average for each question.
In classical statistics, the p-value or descriptive level is the probability of obtaining a test statistic equal to or more extreme than the one observed in a sample under the null hypothesis. For example, in hypothesis tests, we can reject the null hypothesis if the p-value (significance level of the test) is less than 5%. When analyzing only the overall average of the data, the p-value was <0.05, under some conditions. This indicates the average of the group 01 was higher than the average of the group 2, rejecting the null hypothesis of equality, thereby demonstrating that the solution (group 01) was better evaluated.
]]> We focus on the analysis of the graph at specific points, which may provide interesting results from the point of view of comparing two solutions:The answers of Q3, Q4, Q5, Q7 and Q8 reinforce the existence of significant differences between the two solutions: In the answer to Q3, you can check through the Graph 1 that there was a significant difference that demonstrates that users were interested in the requirements provided by the prototype interface. Answers to Q4 and Q5 maintain the difference and it can be justified due the presence of commands and message in the interface of the solution 1. Finally, answers to Q7 and Q8 refer to an analysis of the characteristics present in other existing software solutions. It demonstrates that users noticed something different and attractive, resulting in a pleasant user experience. All these differences were confirmed through the applied statistical tests;
Analyzing the answers of Q1, Q2, proved that there are no differences between the solutions provided by the two groups. Q1 obtained a p-value equals to 0.419364569 and Q2 obtained a p-value equals to 0.5. It can be observed on Graph 1 that there is no significant difference between the two solutions. But, since the questions are related to the easiness of use and information organization of each prototype, it can be concluded that the two solutions presented these characteristics in a similar way;
The answers of Q6 demonstrate that solution 2 had a better evaluation in relation to the solution 1. In Q6 we obtained a p-value equals to 0.440400698 and it was not possible to reject the null hypothesis. Solution 2 presented functionalities related to ordinary tasks related to the educational environment, improving learnability.
Graph 1: Comparison of the answers average
Answers to the subjective questions (SQ) were qualitatively analyzed. Seven questions were presented to the users:
SQ1: Point out situations in which you found it easy to use the system;
SQ2: Point out situations in which you felt difficulties;
]]> SQ3: Do you think the system reached the purpose for which it was designed, i.e., if it meets the needs of end users? Please, explain.SQ4: Do you think the system has reached the goal for which it was developed with a differential to other systems for the same purpose? If so, please list these differentials.
SQ5: The space below is reserved for you to expose your opinion and suggest improvements to the system, especially regarding the adequacy of the system to support the process of teaching and learning.
SQ6: Please, identify different characteristics between the solutions presented.
SQ7: Among the solutions presented, which one was able to meet your needs related to the teaching and learning process? Why?
Analyzing the subjective questions of the questionnaires, we found the following conclusions:
SQ1:
]]> Solution 1: Analyzing the responses, we conclude that the users felt no difficulty using the application and that it demonstrates an ease of use, as evidenced by the responses below:"Very practical, easy to use and simple to understand." (User 2)
"A system easy to use, the content is quite clear." (User 3)
"It is clear (simple), which facilitated a lot." (User 4)
Solution 2: Regarding Solution 2, users also demonstrated ease of use of its functionalities.
SQ2:
Solutions 1 and 2: In both solutions, it was not observed any difficulty related to the interaction of the functionalities.
SQ3:
]]> Solution 1: Analyzing the answers of all users, it is clear that Solution 1 met the users' needs. Users made it clear the lack of a solution similar to it and demonstrated their desire for it."Yes, the system meets various needs of Brazilian education." (User 4)
"Yes, because we, students, need something influencing the studies through the use of current technology" (User 7).
Solution 2: Analyzing this issue, most users reported that the system achieved the objective for which it was developed, but two users demonstrated doubt or uncertainty.
"Regular. The design of the pages can be improved the design of pages. "(User 7)
"I guess so." (User 9).
SQ4:
Regarding this issue, the Users have reported that, in Solution 1, they identified some improvements and it fulfilled the goal. Regarding Solution 2, they also report that it achieved the goal, but do not expose what improvements it brought and they only mention about the good functionality.
"Yes. It gives a good classification of the student options, with games, clarifying doubts, video classes, etc.. In other hand, other systems do not influence the student to use the system" (Solution 1) (User 7)
]]> "Yes. Regarding easiness and understanding of the program." (Solution 2) (User 9)
SQ5:
Among the 13 users who answered the questions related the two solutions, we emphasize that a user reported an improvement to the Solution 1 and 7 Users requested some kind of improvement to Solution2 as answers listed below:
Solution 1: "Improving the way to search some topics." (User 11)
Solution 2: "By using the system it will be much easier to learn. It can be added some function in order to see grades or activities promoted by the school. "(User 1)
“It can be improved in some aspects such as interacting with the teacher to clarify doubts and to see the grades and activities made by the school.” (User 2)
“It can present more suggestions of activities and allow different access ways.” (User 6)
“User interface design can be improved. It is required to add something more attractive to invite the user to experiment the tool.” (User 7)
“It is required an option to get back to the previous screen.”(User 8)
]]> “It is required a little more animation in the program.” (User 11)“It is required more interaction with the student.” (User 13)
The users choice for the solution 1 is evident and can be justified by a sample of responses given to subjective questions, presented bellow:
SQ6:
Please, identify different characteristics between the solutions presented.
"The solution 1 is more attractive." (User 5)
"The two systems presented are great and well prepared, but the system 1 is more practical." (User 7)
“The Solution1presents an amount of contents and ways to make the student more willing to learn. For the Solution2, it lacks more content.”(User 9)
]]>SQ7:
Among the solutions presented, which one was able to meet your needs related to the teaching and learning process? Why?
"Solution 1 because it present more options to choose." (User 08)
"Solution 1 because the presented solutions differ in some details." (User 10)
It is observed the users always report a difference between both solutions, but they judge solution 1 presented more attractive features, demonstrating that it meets the needs of the users.
All questions and responses were analyzed for each solution. Answers for solution 1 (named BIBLIVIRTI) were compared to answers to solution 2 (called Share More). As demonstrated by the responses presented earlier, the end users preferred solution 1.
The requirements elicitation process proposed in this paper has the purpose of guiding the requirements engineers in the creation of new applications in the context of mobile learning. A requirements process specific for this domain can contribute to make faster the identification of the users’ real needs, helping to promote the creation of innovative applications.
In a pilot experiment, the process was applied to identify the needs of students and professors of a chemistry course that works with the preparation of students for tests and, from the elicited ideas, a MLE solution was proposed.
]]> Different m-learning applications help to explore contents, audio and video resources during the classes, as cited by [10], but they don’t offer any functionality to allow interaction between students and teachers inside the classroom. The MLE solution created with the pilot experiment allows the student to interact with the teacher during classes to clarify his doubts, aiming at promoting an improvement in the teaching and learning process. Since we don’t find in the literature and practice any software with characteristics similar to the one present in the solution created with the DT base process, we believe that our solution is innovative.From the result obtained after performing the pilot experiment, we found relevant conclusions, such as: creativity techniques collaborate in the identification of the right requirements; it is possible to adapt DT, incorporating activities to it, such as text categorization using NVivo, to get a better refinement of the information provided by the users, and; it is possible to involve users during all activities of the process. Moreover, the identification of the real necessity to build the MLE was assertive, since the major necessity reported by the users group was “clarifying doubts”. It promotes the development of a MLE with differentiated characteristics, resulting in a really innovative solution in relation to the existing ones.
It was also verified that DT is consistent with the earlier practices of requirements elicitation activities, such as rapid prototyping and stakeholder involvement, what is also promoted by agile development methods. Moreover, Design Thinking emphasizes the human perspective.
Some limitations were found during the pilot experiment in relation to users’ availability to participate in the process. In fact, it is necessary to take into account a reasonable amount of participants to identify problems that result in interesting solutions. The period in which the pilot experiment was conducted was not suitable because the students were studying to make the university admission exams. This situation has influenced the obtained results. However, it was possible to illustrate the applicability and viability of the process.
The experiment conducted after aimed at evaluating the proposed process. Thus, two solutions were created, one using the RE process for ML e another using traditional RE techniques. As a result, we confirmed the conclusion resulting from the pilot experiment. Moreover, the experimental group also evaluated the process and agreed that the process is systematic, easy to use and learn, confirming the hypotheses H11, H12 e H13.
Also in this experiment performed in a high school, it was identified that the problem faced by the students and professors is related to academic content access, since the school is public and needs a library. Also, it was observed that the both solutions developed during the experiment present similarities, however the manner in which the teams developed their products was different. The experimental group developed a solution called BIBLIVIRTI, that aims to make academic content available and categorizing them. These characteristics conquer the users’ interest, as reported in the experiment results section. The control group, by using traditional techniques, observed what happened in classroom, and based on the needs identified, they created a solution tailored for it. It resulted in a solution called “Share More”, that presents similar functionalities to BIBLIVIRTI, such as choosing disciplines and including academic content, but it elicited more ordinary functionalities, such as register disciplines.
Therefore, we can conclude that the proposed process is able to identify assertively the needs and desires of the users and generate more innovative solutions that using traditional elicitation techniques. Also the DT based process promoted the elicitation of a meaningful number of requirements, what resulted in the conception of an interesting product. The control group has elicited many requirements also, but knew the application domain, what influenced positively the product quality. Thus, based on the evaluation made by the stakeholders, the hypothesis H14 was confirmed.
As future works we intend to: (i) explicit the separation of user profiles, because in a m-learning application the actors (students and teachers) can access different functionalities; (ii) apply the process to another domain and consider the elicitation and operationalization of non-functional requirements; (iii) implement both solutions to perform usability tests, and (iv) deploy each solution in an educational environment to evaluate its impact in the teaching and learning process.
This work was partially supported by FACEPE (Fundação de Amparo à Ciência e Tecnologia do Estado de Pernambuco) research grants.
[1] M. Glasemann, A. Kanstrup, and T. Ryberg. “Design and Exploration of a Mobile Game Scenario in a Diabetic Youth Camp”, In: IADIS International Conference Mobile Learning, Porto, Portugal, 2010.
]]> [2] C. Souza, C. Silva. “Uso do Design Thinking na elicitação de requisitos para ambientes virtuais de aprendizagem móvel”. 17th Workshop de Engenharia de Requisitos – WER. XVII Congresso Ibero Americano de Engenharia de Software. Pucón, Chile, 2014.[3] V. Carvalho. Expectativas dos estudantes adultos do ensino superior a distância sobre a utilização de dispositivos móveis para a aprendizagem. Lisboa: [s.n.], 2012. 153 p. Available in: https://repositorioaberto.uab.pt/handle/10400.2/2598
[4] H. Valentim. Para uma compreensão do Mobile Learning. Reflexão sobre a utilidade das tecnologias móveis na aprendizagem informal e para a construção de ambientes pessoais de aprendizagem. Mestrado em Gestão de Sistemas e-Learning. Universidade Nova de Lisboa, Portugal, 2009.
[5] O. Santana, L. Peixoto. “Student perspectives about mobile learning initiatives at Open University of Brazil: the mobile phone issue”. Revista Acta ScientiarumEducation. Maringá, v. 32, n. 2, p. 219-223, 2010. DOI:10.4025/actascieduc.v32i2.11545
[6] T. Araújo. Autonomia no Estudo: Artefato para Planejamento e Monitoramento em Ambientes Pessoais de Aprendizagem Móveis. Master Thesis (In Portuguese). Universidade Federal de Pernambuco, 2012.
[7] N. Maiden, S. Jones, K. Karlsen, R. Neill. “Requirements Engineering as Creative Problem Solving: A Research Agenda for Idea Finding”. 18th IEEE International Requirements Engineering Conference (RE), 2010.Page(s): 57 -66 ISSN: 1090-705X Print ISBN: 978-1-4244-8022-7 NSW DOI:10.1109/RE.2010.16
[8] C. Oliveira, D. Oliveira, C. Oliveira, R. Cattelan, J. Souza. “Árvore de características de software educativo: Uma proposta de elicitação de requisitos pelo usuário”. Anais do Simpósio Brasileiro de Informática na Educação, 2010. Disponível em:http://www.br-ie.org/pub/index.php/sbie/article/view/1517
[9] J. Lemos, C. Alves, L. Duboc, G. Rodrigues. “A Systematic Mapping Study on Creativity in Requirements Engineering.” Proceedings of the 27th Annual ACM Symposium on Applied Computing, 2012. ISBN: 978-1-4503-0857-1 doi>10.1145/2245276.2231945
]]> [10] E. Marçal, L. Lima, Jr. Melo, W. Viana, R. Andrade, J. Ribeiro. “Da Elicitação de Requisitos ao Desenvolvimento de Aplicações de Mobile Learning em Matemática”. Anais do Simpósio Brasileiro de Informática na Educação. Available in: http://www.brie.org/pub/index.php/sbie/article/view/1441/1206[11] D. Dunne, R. Martin. “Design Thinking and How It Will Change Management Education: An Interview and Discussion”. doi: 10.5465/AMLE.2006.23473212 ACAD MANAG LEARN EDU December 1, 2006 vol. 5 no. 4 512-523
[12] M. Soledade, R. Freitas, S. Peres, M. Fantinato, R. Steinbeck, U. Araújo. “Experimenting with Design Thinking in Requirements Refinement for a Learning Management System”. Anais do Simpósio Brasileiro de Sistemas de Informação. 2013. Disponível em: http://www.lbd.dcc.ufmg.br/colecoes/sbsi/2013/0016.pdf
[13] C. Vetterli, W. Brenner, F. Uebernickel, C. Petrie. “From Palaces to Yurts: Why Requirements Engineering Needs Design Thinking”. IEEE Internet Computing, 2013.
[14] A. Campos. A modelagem de processos com BPMN. Rio de Janeiro: Brasport, 2013. ISBN: 978-85-7452-584-6.
[15] M. Vianna, Y. Filho, I. Adler, B. Lucena, B. Russo. Design Thinking - Inovação e Negócio. (In Portuguese) Rio de Janeiro. MJV Press, 2012. 162p. : il. ; 24 cm. ISBN 978-85-65424-00-4
[16] V. Ames. As possibilidades de uso do software de análise qualitativa NVivo. (In Portuguese) v. 1, n. 2, ago. 2013. Available in: http://www.sociologiasplurais.ufpr.br/v1n2_artigo12.pdf
[17] C. Batista, C. Souza, R. Junqueira. Engenharia de Requisitos para aplicações Móveis. (In Portuguese) Monografia da disciplina de Engenharia de Requisitos da Pós-Graduação da Universidade Federal de Pernambuco, Brasil, 2013.
]]>[18] M. Guimarães. Criatividade na concepção do produto. Dissertação de mestrado em engenharia da Universidade Federal de Santa Catarina, 1995 Available in: http://www.eps.ufsc.br/disserta/marques/index/.date.issued:2012-10-16
[19] Balsamiq Mockup. Available in: http://balsamiq.com/products/mockups/.
[20] J. Wainer. “Experimento em sistemas colaborativos”. Cap 24. (In Portuguese) 2012. Available in: http://www.ic.unicamp.br/~wainer/cursos/2s2012/metodologia/LivroSC.cap24.Experimentos.Jacques.v5.pdf.
[21] D. Filippo, M. Pimentel, J. Wainer. “Metodologia de Pesquisa Científica em Sistemas Colaborativos”. Cap. 23. Livro Sistemas Colaborativo. Available in: http://josemalcher.net/livro-sistemas-colaborativos-material-extra-do-livro/. 2012
[22] C. Wohlin, P. Runeson, M. Höst, M. Ohlsson, B. Regnell, A. Wesslén. Experimentation in Software Engineering: An Introduction. 2000. Kluwer Academic Publishers, Norwell, MA, USA.
[23] Camtasia Studio. Available in: http://camtasia-studio.softonic.com.br/
[24] J. Preece, Y. Rogers, H. Sharp. Design de Interação: além da interação homem-computador. Porto Alegre, RS. Bookman, 2005.
]]>[25] H. Campos. Estatística experimental não-paramétrica. Esalq. Piracicaba-SP. 230p. 1987.
]]>