Participants

From the total sample of 111 participants, 8 professionals participated in the judges’ analysis stage. They had graduated in psychology (n = 5) and pedagogy (n = 3); and were holders of Master’s (n = 4) or Doctoral (n = 4) degrees in Psychology or in an interdisciplinary area. All had experience in neuropsychology (practice and research; min = 3 years; max = 13 years) and teaching in Higher Education (min = 2 years; max = 10 years). Each module of the program (the alpha version had 2 modules) was evaluated by 5 judges, with 2 judges analyzing both modules.

One university professor, who was a Portuguese language teacher and doctoral candidate in Educational Psychology, participated in the pilot study, as well as three classes from a private HEI in the city of São Paulo. The selected activities were applied on different occasions, so that different groups participated (n total = 102) in the application: Application 1 - 30 students of the 2nd semester of the Psychology course, morning; Application 2 - 40 students of the 2nd semester of the Pedagogy course, night; Application 3 - 32 students of the 2nd semester of the Psychology course, night.

Procedures and Instruments

The study was approved by the Research Ethics Committee of Universidade Federal de Santa Catarina. The development of the program was based on 4 stages delimited by ^{Cardoso, Dias, Seabra and Fonseca (2017}); plus a 5th stage, with a pilot application.

Stage 1 - Internal stage: A brainstorming session was carried out focused on reviewing existing activities and investigating: 1) activities used in the literature of the area, based on published studies on interventions in EF and related constructs, with analysis of existing activities and programs; and 2) activities that already exist and are used, for example, in the teaching practice, in which an attempt was made to map activities that were already known and that could have require EF skills.

Stage 2 - Construction: During this stage, the following took place: 1) Selection of activities investigated; 2) Adaptation of activities to the program under development; 3) Creation of new activities; and 4) Systematic description of the activities. Next, the stage was completed with: 5) Allocation into modules based on the general aim of the activities; and 6) Finalizing the initial - alpha - version of the program, here already called the πFex-Academics.

Stage 3 - Judges’ Analysis: The judges’ analysis was carried out using the Program Evaluation Protocol, developed for this study, based on protocols used in previous studies (e.g. ^{Cardoso & Fonseca, 2016}; ^{Dias & Seabra, 2013}), to systematize and make the process of evaluating the activities more objective.

In this protocol, the judges were instructed to evaluate each activity developed, firstly identifying the construct (or constructs) represented in the item/activity. In the second part, they evaluated each activity according to the criteria: a) suitability for the target public; b) plausibility of application in the context; c) clarity of instruction for the application; d) accessibility to materials; e) activity-goal coherence; f) involvement of EF in the activity; and g) existence/previous performance of the activity (familiarity index). These criteria were answered on a four-point Likert-type scale from 1 (No, I completely disagree) to 4 (Yes, I completely agree). The last criterion to be considered in the evaluation of each activity was answered on a Likert-type scale from 1 (very easy) to 4 (very difficult) and refers to the analysis of the difficulty of the application (difficulty index). The protocol also has three open questions: 1) suggestions for changing or re-structuring activities in the modules developed; 2) the judge’s position on the exclusion of any activity; and finally 3) general suggestions and comments. The judges were told that they could also mark their observations in the activity booklet.

Stage 4 - Review and finalize the program: After the judges’ analysis, each activity and its quantitative and qualitative assessment were analyzed by the authors. Therefore, this stage included the review of the entire program considering the judges’ analysis, the reformulation of the initial version and the finalization of the ‘beta’ version of the πFex-Academics.

Stage 5 - Pilot Study: A pilot application was carried out with the revised version of the program (Beta), in order to identify its feasibility and any implementation difficulties. Considering the focus on verifying the applicability of the activities and the fact that the activities of each module share the same structure, it was understood that, for the pilot study, the application of one activity per module would be sufficient. Accordingly, 3 activities were randomly selected, one from each of the three modules of the beta version of the program.

The selected activities were applied in classes of the partner HEI by the professor of the Portuguese Language discipline. One activity was applied to each of the three classes; two activities being used for just one day; while the third activity, due to its character, continued in use for four consecutive weeks. The Portuguese language classes were weekly double classes, each with 50 minutes; totaling 100 minutes/week. The pilot application was carried out from November 7 to 28, 2019, as follows:

-Application 1 (on November 14) - Activity drawn from Module 1; applied in one class (double class; 100 minutes duration) with a group of 30 students from the 2nd semester of the Psychology course, morning;

-Application 2 (on November 22) - Activity drawn from Module 2; applied in one class (double class; 100 minutes duration) with a group of 40 students from the 2nd semester of the Pedagogy course, night;

-Application 3 (on November 7, 14, 21 and 28) - Activity drawn from Module 3; applied over 4 classes (each a double class, with 100 minutes duration), one per week, with a group of 32 students of the 2nd semester of the Psychology course, night.

To collect information about applicability, the professor used the Pilot Study Protocol, developed specifically for this study, based on previous experiences in the application of programs (e.g. ^{Cardoso & Fonseca, 2016}; ^{Dias & Seabra, 2013}).

The pilot study protocol was developed to systematize the assessment of the applicability of the activities. The professor who applied the activities filled it in at the end of each day of application covering each selected activity. The criteria evaluated were: Ease of understanding the activity for the professor; Ease of providing instructions for the activity to the students; Students’ understanding of the activity (from the professor’s perspective); Ease of use of the strategy by the students (from the professor’s perspective); Ease of implementation of the strategy within the students’ content area; and Engagement of the students in the proposed activity. These criteria were answered on a five-point Likert-type scale from 1 (Poor applicability or comprehension) to 5 (Very good applicability or comprehension). Qualitative observations were carried out.

Data analysis

The content analysis of the activities was synthesized from the agreement of the judges regarding the predominant component or components (i.e. those identified with 1 meaning the main or primary demand of the activity). The mean of the judges’ responses to each EF scored for each activity was also provided (means closer to 1 indicate greater agreement among the judges in relation to the greater demand for that skill; means more distant from 1 indicate lower demands for the skill in the activity).

The degree of agreement between the judges in the criteria ‘a’ to ‘f’ of the Program Evaluation Protocol was measured using the Content Validity Index (CVI), with values from .80 being considered satisfactory. The formula used was:

The familiarity (criterion g) and difficulty indices were obtained from the arithmetic mean of the scores assigned by the judges to each item. For the pilot study, the scores assigned by the participant-professor to each item of the Pilot Study Protocol, as well as their qualitative observations, are presented.

Results of Stage 1 - Internal Stage

This step allowed for a better outline of the programs, activities and strategies that could be used or adapted for the πFex-Academics. Although our review was not systematic, we mapped the EF promotion programs available in the national context (Piafex, PENcE, CENA and Glia; computerized programs/cognitive training were not included) which, despite focusing on a younger age group than our program, offered general ideas regarding principles and structure. Proposals were also identified (e.g. ^{Dawson & Guare, 2010}; ^{Meltzer, 2010}) for teaching strategies with reasonable applicability in the university context, with adaptations. Neuropsychological rehabilitation strategies (^{Sohlberg & Mateer, 2009}) that could eventually be considered and adapted to the target context were also consulted.

This stage made it possible to investigate activities already known and used, although without a deliberate intention to promote EF. In this process, the dialogue with university professors who were part of the team was useful, allowing the identification of activities/tasks already carried out and considering their demands and possible use as a tool to promote EF. From the exchanges among the team, the demands that guided the structuring of the modules also emerged: Activity planning, Reading Comprehension and Writing Production, with the initial focus (reported in this article) on the first two.

Results of Stage 2 - Construction

This stage culminated with the alpha version of the πFex-Academics (^{Dias, Costa, Cardoso, Colling & Fonseca, in press}). The activities of the πFex-Academics are focused on stimulating the development of EF within academic activities, encouraging students to use and apply strategies that will help them focus attention, mentally manipulate information, consider different alternatives, and organize and plan for the achievement of goals. All activities are systematically described in order to ensure consistency in their implementation. The program’s activities are structured in modules and all of them are permeated and include demands for 1) Attention/inhibition; 2) Working Memory; 3) Flexibility and 4) Planning/Organization.

In this alpha version, initially, the modules included were: 1) Reading comprehension and study strategies (initially with 5 activities); 2) Planning of work and projects (initially with 2 activities), with the continuing development of Module 3 - Written production. The πFex-Academics is composed of an activity notebook and implemented by the professor in their academic routine throughout the semester (Modules 1 and 2 as described earlier in this topic) or academic year (Modules 1, 2 and 3 described earlier in this topic). A semester application calendar was created as a guide for carrying out the program’s modules already developed. The alpha version of the program was revised and submitted for analysis by the expert judges.

Results of Stage 3 - Judges’ Analysis

Table 1 illustrates the content analysis performed by the judges in relation to each activity of the program. Given the complexity of the tasks and the difficulty of isolating a single executive component, a more comprehensive analysis of the judges was chosen, so that they could identify the various executive demands of the proposed activities, listing them from the highest to the lowest (primary = 1; secondary = 2; and successive demands). The table shows both the identified predominant components (those identified as the primary demand, therefore 1) and the mean of these responses to each skill.

With regard to the predominant components, there was a high agreement among the judges and between these and the previous analysis of the authors regarding the main demand(s) of each activity. For example, in activity 1 of module 1, 5 out of 5 judges identified ‘Planning/Organizing’ as the primary demand. In some cases, more than one main demand was identified, as in activities 2 and 3, also from module 1, in which two or three executive skills were identified as equally predominant by the n of judges illustrated in the table. Complementarily, the mean responses to each skill, by item, also allow for a glimpse of the different degrees and demands of EF involvement. Other demands were also scored by the judges, however, for parsimony, it was decided to consider only those that were reported by at least 3 of the 5 judges. In this case, only metacognition emerged as a coherent demand associated with item 2 of module 2.

Table 2 presents the results of the analysis of activities in relation to criteria ‘a’ to ‘f’ of the Program Evaluation Protocol. All items were satisfactorily evaluated in terms of suitability for the target public, clarity of instruction for the application; accessibility of materials, activity-objective coherence, and involvement of EF in the activity. Six out of seven activities were satisfactorily assessed for plausibility of application in the university context. In this criterion, the exception was the activity ‘Stay focused!’, which in fact was based on tasks used in (re)habilitation contexts, distancing itself slightly from the format of typical classroom activities. The activity was considered, as will be described in the next section.

The last two aspects evaluated were the ‘familiarity index’ and ‘difficulty index’. These are merely informative indices that can give hints about the unusualness and complexity of the tasks for application in the context of the university classroom and, therefore, provide information about the adherence of professors and students to the program. The familiarity indices (scale from 1 to 4) were relatively medium and suggest that the activities, although not common or routine in the context, are not foreign to these professionals. The exception seems to have been the ‘Stay focused!’ activity, for the same reason outlined above. Regarding the difficulty of implementation (scale from 1 to 4), the indices did not suggest activities with such complexity that could indicate their unfeasibility for use in the context.

The judges also made comments and suggested some changes in order to make some activities clearer for the professor-applicator and students. These included: adding information and details to instructions and replacing very technical or neuropsychological terms. Another relevant suggestion was the restructuring of the seven activities in the modules initially proposed. This suggestion led to the division into three modules, rather than two. All suggestions were analyzed and most were incorporated into the new beta version of the program.

Table 1: Analysis of the content demands of the activities of the initial version of the program 

^Notes: * Predominant components were considered those marked with a value of 1 (main demand) by the judges. The complexity of the activities and our protocol allowed more than one skill to be marked as the ‘main demand’. ** The judges were asked to mark, for each activity, which skills were involved in descending order of demand. Scores closer to 1 indicate greater demand for that skill. *** Considered only the skills marked (in agreement) by at least 3 out of the 5 judges. Legend: IC/At = Inhibitory Control and Attention; Flex = Cognitive Flexibility; WM = Working Memory; Plan/Org = Planning and Organization.

Table 2: Global analysis of program activities 

^Notes: * Values range from 1 to 4 - values close to 1 indicate less known/usual activities; values closer to 4 indicate activities already known/performed in the academic context (although with other objectives). ** Values range from 1 to 4 - values close to 1 indicate activities that are easier to apply by the professor; values close to 4 indicate more activities that are difficult or complex for the professor to apply.

Results of Stage 4 - Review and finalization of the program

The integration of the quantitative and qualitative evaluations of the judges gave rise to the current, beta version of the πFex-Academics. One specific activity received further revision, the ‘Stay focused!’ activity of Module 1. It was decided to maintain the activity, however: its objectives were clarified, outlining the purpose of the activity with greater clarity; it was removed from its original module (Reading Comprehension and Study Strategies) and placed in an initial module of ‘Basic Study Skills’. With the creation of this new module, another activity (Be flexible!) that was also identified by the authors as decontextualized within its original module, was relocated.

The final program maintained the seven original activities, revised, however, was restructured into three modules. Table 3 presents the final structure of the program; note that the names of the modules were also revised, and the executive skills worked on in each one were added to them.

Table 3: Final structure - beta version - of the πFex-Academics 

Results of Stage 5 - Pilot Study

For the pilot study, an activity of each module was drawn for application, and the following were obtained: Module 1 - Stay focused!; Module 2 - How do you take notes?; and Module 3 - Project and activity planning.

Immediately after the application, the professor-applicator filled out the pilot study protocol. The results are shown in Table 4. In general, the strategies were positively evaluated by the professor in all criteria. The activity in Module 3 was marked as medium in the criterion ‘Ease of use of the strategy by the students’. In fact, the activity introduces the use of strategies such as organizers and planning sheets that may be unfamiliar to students, hence the greater difficulty in using them. However, it should be noted that this evaluation was based on a single application and that the repeated use of the strategy throughout the semester should consolidate its use. However, a review was carried out in the description of the activity, in order to strengthen the professor’s action as a mediator in the first stages of implementation.

Table 4: Evaluation of the activities - beta version - in pilot application* 

^Note: *Scale ranges from 1 (Poor applicability or comprehension) to 5 (Very good applicability or comprehension).

Some comments made by the professor-applicator:

Module 1 -Stay focused!

•Performed while reading an article in the classroom;

•In total, 10 interruptions/beeps were performed, for the self-assessment of attention, at 5-minute intervals;

•Most of the class completed the activity: 2 students finished reading after 6 beeps; 2 students finished reading after 8 beeps; 4 students finished reading after 9 beeps; most ended with 10 beeps. A few could not finish the full reading of the article in the allotted time; 6 students interrupted the activity (to go to the bathroom, drink water) and then resumed.

•An activity that is easy to understand and apply, one must only consider the variation in the time each student may require for the reading.

Module 2 - How do you take notes?

•Activity performed by introducing the semester content on how to write a dissertation paragraph;

•Activity was productive and important to access students’ prior knowledge.

Module 3 - Project and activity planning

•The activity was carried out to manage the semester content: “How do you conduct an interview?”;

•For this, a project was carried out during the month of November, divided into the following stages, which were organized with the support of this activity in Module 3:

-Reading an article on the topic that each student chose at the beginning of the semester (individual);

-Highlight the main ideas of the article (individual);

-Choose a classmate to send the article to read and underlined (pair);

-Make a conceptual map of the underlined ideas of the colleague’s article (individual);

-From the conceptual map, prepare at least 10 questions to interview the colleague about the topic of the article (individual);

-Interviewing and being interviewed in the classroom by a colleague (pair);

-Organize the interview responses (individual);

-And, finally, post on the virtual platform the questions that were prepared with the answers of the interviewed colleague (individual).

•According to the professor: “Faced with so many individual and group actions, without a proposed systematization, most would not be able to carry out all the stages of the activity”.