Formative E-Assessment: A Qualitative Study Based on Master’s Degrees

Formative assessment is a strategy that optimizes the learning process at any educational level, however its use is not very frequent as literature revision shows. In this paper, we analyse the use of formative assessment in online postgraduate studies (masters) in Spanish universities. Our sample was 31 online master’s degrees coordinators and we analyse the results obtained from a questionnaire with open questions using NVIVO software. Through qualitative analysis of the information supported by cross-queries of codes and attributes, we have considered formative assessment according to fields of knowledge, the type of digital tools used and the main difficulties identified. In this type of online masters, our data show that most of the teachers use this type of formative e-assessment to provide feedback to their students and as part of the final marks of the courses, too. So these results are relevant to understand the E-assessment strategies for master’s degrees. Finally, the main limitation of the study is the fact that it uses a sample limited to the geographical context of Spain. Nevertheless, these data are representative of E-assessment in Spanish master’s degrees and may be of interest for future research, for comparative studies in other contexts and even with face-to-face or blended-learning master’s degrees.


Introduction
Assessment and feedback are fundamental aspects of any teaching-learning process regardless of its modality: virtual, blended or face-to-face (Mimirinis, 2019;Wuisan & Wibawa, 2019). Moreover we can use conventional methods, or electronic, or combined assessment using both of them (Hasani et al., 2021). However, there is a general tendency to link the e-evaluation with the completion of digital final questionnaires to measure the scope of the achievements (Sánchez-Vera et al., 2013).
Generally, any educational context should assess from a multi-perspective, since different teaching methodologies are also often used. In other words, any e-assessment process should contemplate different strategies based on the e-formative assessment, e-peer assessment and e-self assessment (Babo et al., 2021;Lu & Law, 2012;Mosqueda, 2018), not only summative assessment in the end of the courses.
Through this work, we intend to reflect on the importance of other e-assessment strategies -not only the summative-to facilitate the effective learning of any student in a virtual environment. Specifically, we will explore the e-formative assessment, whose main notable element is giving feedback to students. According to Zhu et al. (2021), the feedback can benefit the learning, the engagement, the positive emotions and the disposition toward learning of master's degrees students.
In this qualitative research, we applied a structured interview to coordinators of postgraduate studies from different public and private universities from Spain and whose learning modality is virtual (online masters' degrees). The development of the interviews was online and structured around different predefined questions to extract meaningful information related to our research question and objectives. The analysis of the qualitative data was carried out with NVivo software, which facilitated the coding and creation of nodes. Mainly, by applying the cross searches between codes and attributes, we found interesting results about the used methodologies to implement E-formative assessment at university, digital tools to give feedback, difficulties for teachers and students and other relevant information related to the topic of this paper.

Review of Related Literature
Over time, online and blended learning are becoming two models of great interest when compared to traditional face-to-face teaching, and despite their possibilities, they also highlight different concerns that must be addressed to gain an optimal educational experience (Chandrasekaran et al., 2016;Zwelijongile, 2015). Concerns are generally associated with the assessment process (Cabero, 2017;Sharma & Karforma, 2012), although they may also exist in any other of the substantial elements of an educational process (González-Sanmamed et al., 2018).

About E-Assessment
Focusing on assessment, and specifically on electronic assessment (E-assessment), this emerges as an alternative to overcoming the deficiencies of traditional pencil and paper assessment modes, so that E-assessment is based on the use of digital tools which assess the theoretical and practical knowledge of any student (Azevedo & Azevedo, 2018). Furthermore, in virtual environments, assessment must be one of the most significant elements of educational practice, since students tend to demand greater attention due to the lack of contact with the teacher (González, 2020).
Despite this virtuality, E-assessment not only has to be developed in online teaching models but can also be present in physical contexts and thus enrich the educational experience, since E-assessment should not be based only on online tests (Sánchez-Vera et al., 2013). According to Babo et al. (2021, p. 69), "with individuals' motivation for implementing new tools or improving existing ones, to support unbiased assessments, concepts like usability, user experience, assessment and the relation between them are being studied". Furthermore, students tend to better assess the evaluation modes that are developed through technology (Bahati et al., 2019). Therefore, to guarantee the implementation of E-assessment without the need for technology-equipped rooms, the "Bring Your Own Device" (BYOD) strategy is often recommended (Küppers & Schroeder, 2016). Leaving aside this dimension of digital assessment in physical contexts, we will explore e-assessment in virtual contexts, which has very characteristic benefits: "efficiently managing and streamlining the assessment process; facilitating dialogue and student engagement; enhancing student learning and developing (digital) identity and the community" (Mimirinis, 2019, p. 6). In general, it should be noted that E-assessment provides benefits in lectures, and for students and institutions (Wuisan & Wibawa, 2019).

Types of E-Assessment
The benefits outlined in the previous paragraph do not only apply to digital assessment in general. The distinctive aspects of each type of assessment must be taken into account (Prendes et al., 2021). Four different E-assessment types can be distinguished: summative E-assessment as the addition of the results obtained at the end of lessons, courses or projects (Sewell et al., 2010) through digital tools such as forums, quizzes, etc.; E-self assessment consists of a reflective process which identifies strengths and weaknesses concerning the achievement of objectives using online questionnaires (Mosqueda, 2018) or other alternatives such as E-portfolios, or digital apps like Socrative. E-peer assessment is the digital feedback that is generated among students who define themselves as assessors or assesses (Lu & Law, 2012) and is usually developed through videoconference tools, social networks, and so on. The last type of electronic assessment is formative E-assessment, the type of assessment that is examined in this article.

Formative E-Assessment
Generally, formative assessment is defined as the feedback that teachers give to students to improve their educational experience and to make them more responsible for their learning (Irons, 2008). More specifically, Colmenares (2012) states that formative E-assessment is how the teacher identifies the weaknesses and strengths that each student has which gives them self-awareness through chats, forums, blogs or other digital tools. In this way students can act proactively to meet their academic needs. Colmenares (2012) also demonstrates that formative assessment is an opportunity in physical environments due to the spontaneous interaction that is generated. However, he states that in a virtual environment this could also be maintained as long as there is a self-instructional process. In other words, formative E-assessment should have a self-evaluating aspect, so that the teacher adopts the role of guide and the students of protagonists.
The way to carry out a formative E-assessment depends on the different variables that condition the procedure (discipline, number of students, and available digital tools.). Certain authors (Bahati et al., 2019;Vonderwell & Boboc, 2013) talk about some of the most widely used techniques: role-play, the minute paper, reflection paper, hook questions, things to keep in mind, questions walls, online knowledge survey, and reflective electronic journals, and so on. However, all of them have to adapt to educational reality to avoid inconveniences such as the student not knowing what to do with the comments; not valuing feedback, as they only care about their grades; or using feedback differently than expected (Irons, 2008).
Sometimes it is not possible to take advantage of formative E-assessment, as explained in the previous paragraph. However, from the perspective of a university student, formative E-assessment presents a series of possibilities. It can monitor their progress, motivate them to continue achieving academic objectives, improve understanding and benefit active participation (McCallum & Milner, 2020).
Throughout this examination, formative E-assessment is shown to be a digital version of physical formative assessment, in which teacher-student interaction is essential. However, within a virtual environment there is also a line that links learning analytics with E-assessment and more specifically, with formative E-assessment. We can see that learning analytics allow the monitoring of students through the data collected by using a digital learning environment with interactive resources that facilitate automatic assessment (Barana et al., 2019).

Research Question and Research Objectives
We intend to reflect on the importance of other types of E-assessment -as distinct from the most commonly used summative assessment-to facilitate effective learning by any student in a virtual environment. Specifically, we will explore E-formative assessment, whose most notable element is feedback as we have explained (González, 2020;Zhu et al., 2021). Our research question is: how is E-formative assessment used in online master's degrees in Spain? Our main research objectives are: 1) To analyse the formative strategies used to assess students in Spanish online master's degrees.
2) To identify digital tools used for the development of E-formative assessment strategies in Spanish online master's degrees.
3) To describe the thoughts of different coordinators of online master's degrees about E-formative assessment based on their experience.

Research Design
This research is focused on a qualitative approach. According to McMillan and Schumacher (2005), qualitative methods intend to analyse and describe the research object using the subjective interpretation of the participants. Our research fits into this frame, because we have collected the information from the coordinators of online master's degrees in Spain so we can use their knowledge to describe the use of these assessment strategies. We have used an interview to collect information and the coordinators of the online master's degrees as population.

Phases and Proceedings
The first step was the identification of the participants. We needed to know all the online master's degrees in Spain, so we searched for the list of public and private universities on the page of the Ministry of Education and then we searched on every institutional web page for specific information about online master's degrees (n=141) in universities. Finally, we searched for the names and emails of coordinators.
Once we had this completed list of degrees with their coordinators, we classified the results using three criteria: field of knowledge, area of knowledge and universities. To select the sample, we chose a proportional stratified sampling using these criteria and we sent the information about the research and the interview by email. We found the usual problem of people who did not answer, people who did not want to participate and people who did not want to answer the interview by videoconference. We sent messages by email every three days, trying to complete the desired sample. Finally, we had to use all the contacts in the list so we can say that we wrote to coordinators of all the online master's degrees list. In the field of knowledge of Sciences, we did not receive any answer, although we sent the emails to coordinators twice. To solve the problem of coordinators who did not want to connect by videoconference, we used the email as a communication tool and we sent the written questions.
After two months, we closed the proceeding to collect data and the third phase was the analysis of collected data with NVivo (release 1.3 version).

Sampling Procedures
We have applied a proportional stratified sampling based on these criteria: representativity of every field of knowledge, of all areas of knowledge, of different universities and gender of coordinators. Once the strata were defined, one person from each group was randomly taken until the sample was completed.
The initial population of all online master's degrees was 141, but we excluded three of them because they are not genuine online studies as they were online because of the Covid-19 pandemic. Therefore, we finally included 138 studies. We estimated a minimum of 10% in every group from a total of 14 master's degrees. However, we have 31 ies.ccsenet.
answers (T

Research Instrument
Taking as a starting point the objectives mentioned above, we designed the interview following Alvira (2011) in these phases: a list of questions relative to every topic; question design; structure (order, format); first draft; review by the study team; and finally, validation.
We decided to design a structured interview with open questions. The script consisted of 4 socio-demographic questions plus 13 open questions classified in three types according to McMillan and Schumacher (2005): experience, knowledge and opinion. Moreover, we designed questions relative to two topics: E-assessment strategies and telematic tools used by teachers. It should be noted that at the beginning of the questionnaire a description of formative assessment was added to avoid possible confusion or misinterpretations of the concept since it has been identified that in some cases a formative evaluation has been applied, but has not been identified as such (Aleyaasin, 2018).
Expert judgement was chosen for the validation of the instrument. For this purpose, we selected five experts from the field of Educational Technology with experience both in training and E-assessment. An instruction sheet then was prepared (Delphi method), in which the following aspects were considered for validation, following the proposals of Escobar-Pérez and Cuervo-Martínez (2008): coherence, relevance, clarity and sufficiency of each of the items.
When all the information was collected, a quantitative analysis of the results was carried out, obtaining the average of each of the sections analysed. According to Crocker and Algina (1986), it was decided to eliminate the questions with a score lower than two.

Data Analysis
We have carried out the data analysis with NVivo (release 1.3 version). First, the 31 cases have been classified according to 5 attributes that respond to the sociodemographic questions of the questionnaire. These have been (1) sex, (2) specific training in virtual teaching, (3) experience teaching virtual teaching, (4) field of knowledge of the master and (5) experience coordinating the master.
Then, for each question a free code has been generated, which has different codes associated with it according to the references that have been marked in each of the cases analysed. From this coding process, the results have been analysed both independently and through cross-queries that relate to the results of each question with the attributes mentioned in the previous paragraph.
To analyse the results and extract the most relevant results, different graphs have been used, such as heat, radar, stacked bars and maps. However, the results below show those of greater value according to the objectives set.

Results
Firstly, we found that not all master's degrees see formative evaluation as a possibility since 6 coordinators (19.3%) affirm that their master's degree only takes into account the summative or final assessment. On the other hand, a vast majority represented by 24 cases (77.4%) do confirm that formative assessment is a reality throughout the master's degree. As an isolated case (3.2%), there is a master's degree in the field of Social and Legal Sciences that allows its students to select the type of assessment they prefer to have throughout the course. Next, Figure 2 shows a stacked bar graph that shows the responses according to the coordinator's fields of knowledge. As stated, 3 codes have been generated that group together the previously mentioned information. ies.ccsenet.

Relevant in and Archit degrees in the other h degrees inv
After analy to recalcul master's c different a is the one distributed h the code "diff videoconfere ks that student r data on the u ) confirmed tha se, 17 coordin rs (24%) take nating the mas aph (Figure 3) Vol. 15, No. 2; which improves their learning process (Chanpet et al., 2020;Hansen, 2020). However, according to the data provided by this research, it can be appreciated that this type of assessment is not only used to promote teacher-student interaction but that the results derived from formative E-assessment have an impact on the final grade received by each student.
Spanish online masters mainly use forums, the same student tasks, electronic messages and other digital tools of the official platform of the study program, thus coinciding with the results found by Colmenares (2012), and they show a clear preference for asynchronous tools. The use of synchronous tools could be an aspect to improve, as highlighted by Abate et al. (2021) in their research on the importance of synchrony in online teaching processes. Despite the extensive list of digital tools, the coordinators have not indicated the assessment strategies applied. Even though the questionnaire did explicitly ask about tools and strategies, none of the possibilities that exist was mentioned: role play, the minute paper, hook questions, things to keep in mind, questions walls, etc. (Bahati et al., 2019;Vonderwell & Boboc, 2013).
Another possibility to carry out a formative assessment is to use the analytical data (descriptive statistics and/or learning analytics) produced by the digital platforms used. According to Barana et al. (2019), Karaoglan et al. (2020) and other authors, this is an ideal way to monitor students and can even lead to automatic evaluation. However, most coordinators acknowledge that the teachers in their master's degrees do not use these data or consult them. Only 6 coordinators stand out as taking this type of information into account, but not with the referenced objective, but to know the main problems of the students and thus avoid them dropping out of their studies. Furthermore, it has been identified that the coordinators who do take the learning statistics into account are those who have greater coordination experience. Therefore, it can be affirmed that it is an excellent possibility to complement the formative assessment. However, a certain background as a coordinator is needed.
The benefits associated with formative E-assessment in a university context are usually to monitor progress, promote motivation and improve the learning process, according to McCallum & Milner (2020). This vision fully matches our results. The coordinators who have responded explain that in their opinion, this assessment modality demonstrates that it can improve the teaching and learning process, in addition to allowing better monitoring. It also appears that formative assessment increases satisfaction and motivation in students. The coordinators also point out benefits in terms of improving the autonomy and self-regulation of student learning, as well as facilitating the adaptation of teaching resources and methodologies to the individual needs of the students, thus promoting greater personalization of teaching (Fake & Dabbagh, 2020).
On the one hand, the problems related to formative E-assessment appear to be mostly related to not adequately assessing the comments made by the teacher, focusing only on the grade, design of assessment instruments, or the amount of information generated (Irons, 2008;Webb et al., 2018). On the other hand, our research concludes that the most widespread problem -both for teachers and students-is the time involved in the development of formative assessment. For other authors, this is not the main problem (Hunt & Pellegrino, 2002), so this answer may represent a peculiarity of the context studied (Spanish universities) and may open up interesting lines of research for the future. Mainly, because in the Spanish context temporary dedication to formative assessment has been identified as one of the most widespread difficulties in any educational initiative regardless of level (Aragonés et al., 2020;Cubero & Ponce, 2020;Pineda et al., 2019).
Another difficulty observed by Irons (2008), the interpretation that the students make of the comments made by the teacher, hardly appears in our research, since only one coordinator in the field of Social and Legal Sciences has pointed it out.
Finally, contrary to the codes that list different difficulties for both teachers and students, there have been a small number of cases that think that there are no difficulties associated with formative assessment. Specifically, this idea is related more to the students than to the teachers.
About the limitations shown by the work carried out, the problems in being able to carry out the interview synchronously (by videoconference) should be emphasised. This possibility was rejected by almost all of the coordinators, who opted, as an alternative, to send the information in writing (using email). This change in design implies the absence of interactivity with the data-producing sample, an interaction that would have allowed us to examine the responses more profoundly. However, this alternative has allowed us to expand the participating sample and thus obtain a more complete approximation of the reality studied in terms of the number of cases.
As a future line of research based on this work, we propose to expand learning analytics as a complement to formative assessment, since it has been identified as one of the least used and known elements. This is considered of interest due to the current educational trend towards the automation of evaluation (Skalka & Drlik, 2020;Tan et al., 2020;Wu & Chang, 2020) or smart learning (Maulidiya et al., 2019;Nanning et al., 2020;Yusufu & Nathan, ies.ccsenet.org International Education Studies Vol. 15, No. 2; 2020), thus making these statistics an opportunity to enrich formative assessment and, therefore, to optimize the educational process.
We also believe that this research opens up interesting lines regarding the study of the use of synchronous tools since our data show that the orientation is mostly towards the use of asynchronous tools, thus coinciding with what generally occurs in online teaching (Laato & Murtonen, 2020). And the experience developed in online degrees will be useful for students' assessment in general, especially because of all the changes and challenges in higher education in recent years caused by the pandemic (Ali & Damour, 2021;Fatima et al., 2021;Peytcheva-Forsyth & Aleksieva, 2021).
Finally, we believe that international comparative studies would help us understand why the time factor stands out as the main difficulty, while in other studies not appearing, as we have seen. We do not know if it is something typical of the Spanish context, or the reasons why this data appears so markedly in our research. We are also interested in approaching this study from the students' perspective, as their vision is essential to understand the real magnitude of the impact of this type of assessment. Thus, future research could investigate the impact of formative e-assessment concerning other topics like motivation or engagement to endorse the data found in this work about learning.