Case Studies for Enhancing Student Engagement and Active Learning in Software V & V Education

Two critical problems facing the software (S/W) industry today are the lack of appreciation of the full benefits that can be derived from Software Verification and Validation (V&V) and an associated problem of shortage of adequately trained V&V practitioners. To address this situation, the software V&V course curriculum at the author’s institution is being improved via a National Science Foundation (NSF)-funded project. The basic objectives of this project are to enhance the quality of software education via increased student engagement and by bridging the gap between the basic principles discussed in the classroom and the complexity of real world problems. The teaching method utilized promotes higher levels of student engagement and learning through interactive, hands-on exercises, case studies and discussions. In addition, the instructional materials were purposefully designed not only for university classroom settings, but to also be deployed for on-the-job professional training in S/W industry settings, thereby helping to increase the pool of professionals with contemporary V&V knowledge and skills. The new course curriculum enhancement described in this paper is guided by academic research and industry best practices that focus on four specific V&V focus areas: requirements engineering, reviews, configuration management, and testing. Among many educational tools that are being developed to achieve the project objectives, the work related specifically to the development of one central component, case studies, is described here. Historically, case studies have been educational tools utilized in business, law, and medicine, but are not as prominent in software engineering. The hypothesis is that case studies would be effective educational tools to introduce real-world professional practices into the classroom, which would help the students in both identifying and solving problems, and developing a perspective on applying knowledge. In this paper we describe a set of V&V related case-studies that we have drawn from industry experiences and developed as pedagogical tools. These case-studies cover several important topics in the S/W V&V domain such as software testing, legal issues in software, software consumer protection, and requirements from the customers’ perspectives. We also report on the results of initial implementation of the case studies related to software testing in the classroom and show their effectiveness both in terms of satisfying Accreditation Board for Engineering and Technology (ABET) outcomes assessment and student satisfaction.


Introduction-Case Study Based Education
Engineering education must strike a balance between the knowledge of theoretical concepts and the ability to apply the theory to solve real world problems (Bertha, 2010).It is the practical knowledge that is interesting to the student and immediately useful to the community.However, it has been realized that the practical knowledge cannot be easily taught in a class room setting as it requires both time and experience.The use of case studies is therefore important because it taps into practical knowledge and real world experiences that students are able to relate to and learn from.Among many other educational tools that have been developed to realize the learning objectives in the computing field, tools based on case studies are definitely in short supply (Towhidnejad, Hilburn, & Salamah, 2011).Traditionally, case studies have been widely used as pedagogical tools in business, law, ethics, economics, systems engineering, and medicine but they are not as common in software engineering.Case studies have many unique and distinct advantages in software engineering education including that they allow for: -Application of knowledge or skills in a real-world setting -Identification and clearer definition of the problem -Participative, collaborative, inclusive, and team based approach -Opportunities for creative brain storming, and -Development of solutions to problems.
Therefore, the hypothesis presented in this paper is that the case studies would be effective educational tools to introduce real-world professional practices into the classroom which would help the students not only in honing their problem solving skills but also increasing self-directed learning skills and team skills.As the case studies are grounded in real world situations, they offer opportunities to develop a perspective on knowledge application, project management and project economics in addition to the domain knowledge enhancement in software engineering.
Case studies can be used effectively to contextualize theoretical concepts (Davis & Wilcock, 2003).It has been shown in many studies (Grant, 1997;Raju & Sanker, 1999;Sivan, Wong, Woon, & Kembler, 2001) that benefits of case studies are derived from their interactive learning strategy and the shifting of emphasis from teacher-centered to more student-centered activities.The benefits of the case study method listed by Davis and Wilcock (2003) are that the case studies: -Allow the application of theoretical concepts to be demonstrated, thus bridging the gap between theory and practice -Encourage effective learning -Provide an opportunity for developing key skills such as communication, team work and problem solving -Increase student enjoyment of the topic and hence increase their desire to learn -Allow longer retention of the material.
The sources for case studies can be diverse (Nespoli & Lambert, 2010).For example, case studies may be developed via undergraduate student project work, co-op experiences/summer work, senior year capstone projects, graduate (ME/MS) thesis projects and professional work with industry partners.In the present work, the case studies have been drawn mainly from industrial partners, large scale government projects, and from the direct professional consulting experience of the authors.
Another important and emerging aspect of engineering education is the fully on-line engineering degree program.While there are just a few completely online undergraduate programs available at this time, more and more classes are being offered online to facilitate the education of nontraditional students such as mid-career employees and military personnel (Fisher et al., 2007).In order to ensure that the online engineering education is at least equally effective to (if not better than) traditional face-to-face education in terms academic quality, rigor and outcomes, appropriate teaching tools must be developed to suit the online teaching/learning media.In this regard, we believe case study based education is one of the superior tools to deliver an equivalent and challenging laboratory experience for online students!In this paper the process used for developing case studies is described in Section 2, a fully developed case study in the domain of software testing is presented in Section 3, pedagogy and educational outcomes are discussed in Section 4, an example of the implementation of software testing case studies is given in Section 5, and student evaluations of active learning tools are presented in Section 6.

Case Study Development
The project team has used the three steps process depicted in Figure 1 in developing case studies.Once completed, the case studies are integrated in to the curriculum as in-class exercises or as homework assignments.The case study tasks may be accomplished by the students working individually or as student teams consisting of 2-4 students per team.The case study contains scenarios or real world stories along with the background material such as setting, personalities, sequence of events, relevant data, problems, and conflicts.Students involved in group work related to case studies can develop skills required for success such as group decision making, consensus building, negotiation, and tolerating differences of opinion within a diversified work place (Kauffman, Abdel-Salem, Williamson, & Considine, 2009).Testing is an investigative process in which a software system or component is evaluated against a set of predefined inputs to observe whether or not it gives the expected results.If the results are met then the user requirements are met.Testing results in demonstrating software quality and reliability and helps product development by identifying errors or missing requirements (Acharya & Manohar, 2015).Whether one is writing an individual unit test or designing a product's test plan, it is important to take a step back and think about how effective the tests are at detecting and reporting bugs in the code.

Case Study Template
A template for case study development proposed in this work is shown in Figure 2. The benefits of developing a template for a case study are twofold: (a) template provides a standardized way to document the background information, description and objectives of case studies and (b) facilitates identification of any missing information or gaps of knowledge for the students as they attempt to solve the questions based on the case study.Using a template also allows for systematic improvement of the description contained in the case study as it is delivered using revised iterations in future classes.Figure 3 depicts a domain-specific case study in the area of software testing.For professional software technology companies software testing is a critical aspect of reliability and quality of their business.The basic purpose of testing is to validate the testing coverage of the application being evaluated.Writing effective test cases is a skill that can be achieved by experience with and in-depth study of the application about which test cases are being written.

Focus
For example, Google (Google, 2014) suggests several important qualities such as fidelity (a test that is sensitive to defects in the code), resilience (a test that fails only when the code is defective), and precision (a test that shows the exact location of the defect in the code) that every test should try to maximize.These three qualities are often in competition or opposition with each other.It's easy to write a highly resilient test (the empty test, for example), but writing a test that is both highly resilient and has high fidelity is hard to do.Students and professionals can begin to learn how to do software testing by creating tests scripts.The development of test scripts therefore needs greater attention and a systematic procedure is suggested in Figure 3.

Test Case Script Template
In this task, the students are expected to develop test cases using a given template.A sample test case script template is given in Figure 4.
The developed case studies can be integrated in the curriculum either as homework assignments or as in-class exercises.The authors have utilized the case studies in both ways and it is suggested here that the more complex case studies should be assigned as homework while relatively narrow and focused case studies may be discussed as in-class exercises.It should be noted here that an appropriate theoretical framework needs to be established via lectures to lay the foundation before case studies based on the relevant topics are given to the students.In-class  What is the importance of preconditions in test cases development?

Exercise:
■ Develop ten test cases that are relevant to the application you are testing such as those described in Section 2.3.■ Use the "Test Case Template" given by your instructor to write scripts for the ten test cases that you just developed (example template for writing test cases is provided in Figure 1 in this paper) ■ Identify test cases that can be automated.(Assume appropriate information when necessary.)Instructing Notes: This is to be delivered as a classroom/Homework assignment.Class 1 (50 minutes) 1. Ask the students to individually prepare the test cases 2. Form teams of 4 students in class 3. Ask the teams to discuss the questions given in the case study.4. Discuss as a class (10 minutes) Assessment Procedure: In-class short quiz for this module may be developed and given to the students in addition to their performance in the test cases, test script writing and case review quizzes are then utilized to assess the effectiveness of student learning.Therefore students gain the necessary theoretical framework via lectures and then are able to understand the practical applications of the principles via case studies.
Student performance in case studies can be assessed based on their in-class participation and the accuracy of their response to the case study questions.A grading rubric may be appropriately developed for each case study depending upon whether it is an individual or group assignment and if it is an in-class or homework assignment.

Educational Outcomes Assessment
The study based approach to teaching and learning is broad in terms of its coverage of educational outcomes and it has been suggested that it can be used to deliver all eleven "a" through "k" criteria of ABET accreditation (Towhidnejad, Hilburn, & Salamah, 2011).The flexibility of case studies coupled with the richness of data and information analysis, decision making education and conflict resolution results in strong links with ABET criteria.Kauffman et al. (Kauffman, Abdel-Salem, Williamson, & Considine, 2005) have mapped case study outcomes to the ABET criteria for engineering economy case studies.Such analysis is adopted here for case studies in software engineering as shown in Table 2.  Case studies requires students to learn and apply contemporary engineering tools to solve case problems Pedagogical outcomes that are relevant for software verification and validation have been identified at the author's institution based on ABET Criterion 3 outcomes assessment.The relationships between the specified ABET outcomes for this course and their correspondence with the revised Bloom's taxonomy (Google, 2014) for STEM disciplines are shown in Table 3.It is clear from the information presented in Tables 2 and 3 that it is possible to evaluate student learning outcomes b, c, e, f, g, h and k using the case study based educational tools.

Implementation of the Case Study Method
One of the authors has been delivering a S/W V&V course since 2005 and is required to perform an ABET Criterion 3 outcomes assessment.Figure 5 depicts a graphical display of the class assessment performed in Spring 2013 when the case study approach was not incorporated as a pedagogical approach.The Spring 2013 class had seven software engineering junior level students (all males) and all of them were considered for this study.This chart presents percentages of students scoring 80% or better on a variety of assessment tasks.The student performance in each assessment task was measured and regrouped in terms of ABET outcomes to calculate percentage of students that scored within certain levels of assessment vector as detailed in Table 4 given below.It is seen from Figure 5 that there was a weakness associated with learning outcome "e" (an ability to identify, formulate, and solve engineering problems), where less than 60% of the students scored better than 80% on the assessment tasks, causing ABET outcome "e" to be identified as a "weakness".One of the main reasons for the lower outcome percentage is because the student performance data was obtained through exams, which may not be the best suited tools for assessing outcome "e".It is shown in Table 1 that case study based education can be used to enhance outcome "e".
From the 2013 ABET Outcome Assessment report this instructor realized a need for more applied, higher level learning tools.In the 2015 delivery of the S/W V&V course case study approach was incorporated as a pedagogical approach and relevant outcomes assessment was performed.The Spring 2015 class had twelve software engineering junior level students (all males) and all of them were considered for this study.This time the student performance data for outcome "e" was obtained assessing student performance in case study related tasks.The results of this evaluation are presented in Figure 6.
It can be seen clearly that the student performance related to outcome "e" is now in the excellent range (>= 90%) as compared to being an area of concern (< 60%) in Spring 13.This presents clear evidence that the case study based teaching method is more effective in delivering an ability to identify, formulate, and solve engineering problems to the students.Therefore case study based educational tools will be progressively developed, adopted and delivered in several other aspects of the software V&V area such as legal issues in software, software consumer protection, and requirements from the customers' perspectives.The results of those implementations will be reported later as more data become available.
Figure 6.Student outcomes assessment with respect to the specified ABET criteria in Spring 15 term-case studies were delivered in the class

Student Evaluation of Active Learning Tools
A student survey tool was developed to assess the effectiveness of the active learning tools developed here and receive feedback for future improvement.Appropriate Institutional Review Board (IRB) clearance was obtained for the survey tools employed in this study.Ten software engineering junior level students (all males) participated in this study.The survey was carried out in-class on the last day of class using paper survey.Instructor was not present in the room and the survey was anonymous.The students were asked to rate the value of the in-class activities in enhancing their knowledge and learning process experience (Table 5).The feedback was given on a Likert scale of 0 (don't know), 1 (not useful), 2 (somewhat useful), 3 (moderately useful) and 4 (extremely useful).
The results showed that 90% or more of the students who just completed the class found all of the active learning tools (case studies, exercises, videos) to be moderately or extremely useful.Related specifically to case studies, 100% of the students found the case studies to be moderately or extremely useful.On the other hand, the students found less utility in the written homework assignments and textbook readings.
Another question in the survey asked students about the types of activities they do in the class, including paying attention to lectures, engaging in small group or class discussions, completing real-world applications, thinking critically, reviewing research, or utilizing professional standards to some degree (Table 6).In this class that deployed active learning tools, the majority of the students (> 70%) responded that they completed real-world applications and felt accountable to classmates in full class discussions.Their communication skills were also utilized to a greater extent in these activities, thus increasing the educational value of active learning tools.Finally, the vast majority of the students (>= 90%) reported positive student behavior in this class as compared to other classes in their major in terms of the time spent in learning, subject interest, and understanding (Table 7).

Summary and Future Work
To develop case studies, the needed data, information and scenarios have been drawn from industry partners and from professional consulting work of the authors.The template for developing case studies has been created and presented in this paper.Case notes and instructors' supplementary materials have been prepared.The case studies developed here cover many areas in the software verification and validation knowledge domain including software testing, legal issues in software, software consumer protection, and requirements from the customers' perspectives.
Applicable student learning outcomes for the software V&V course have been determined, and their relationship to ABET criteria and revised Bloom's taxonomy for STEM disciplines has been mapped.The effectiveness of case study based educational tools has been determined based on this evaluation context.Using baseline data from the Spring 2013 semester when case studies were not available, the potential impact on increased student learning and engagement based on the incorporation of the new educational tools described in this paper, has been seen.The newly developed case studies were introduced to the students in Spring 2015 term.The effectiveness of these teaching/learning tools was assessed, and there was evidence that student learning in the areas of problem identification, formulation and solving is significantly improved with the use of case studies.The development of more case studies in different areas of S/W V&V has been undertaken and those case studies will be delivered in the near future.

Figure 1 .
Figure 1.Case study development process

Figure 3 .
Figure 3. Case study for software testing

( b )
An ability to design and conduct experiments as well as to analyze and interpret data Case studies requires students to find or develop the important information and ignore data that is not relevant (c) an ability to design a system, component or a process to meet desired needs Case studies requires students to confront complex issues such as trade off analysis along with time, resource and risk management decisions (d) an ability to function on multi-disciplinary teams Case studies requires students to solve case problems, they must also learn to negotiate and understand different viewpoints prior to their decision making (e) An ability to identify, formulate, and solve engineering problems Case studies requires students to identify important data and ignore irrelevant data, actively look for missing data or make appropriate assumption and use mathematical / computer simulation based tools to solve engineering problems (g) an ability to communicate effectively Case studies requires students to make presentation of case analysis results in both oral and written formats (h) the broad education necessary to understand the impact of engineering solutions in a societal and global context Critical thinking required by case study analysis promotes systems thinking related to larger impact of decision alternatives (k) an ability to use the techniques, skills, and modern engineering tools necessary for engineering practice An ability to communicate effectively III, IV & V h.Broad education necessary to understand the impact of engineering solutions in a global and societal context VI i. Recognition of the need for and an ability to engage in life-long learning.VII j.A knowledge of contemporary issues V & VI k.An ability to use the techniques, skills, and modern engineering tools necessary for engineering practice VI & VII Taxa I-Pre-knowledge Conceptual Experiences: hands-on laboratory experiences via demonstrations, physical models, practical applications to demonstrate, visualize and observe basic concepts.Taxa II-Basic Conceptual Knowledge: learning, understanding, memorizing basic engineering concepts, definitions, terms, symbols, theories, laws and equations.Taxa III-Applied Conceptual Knowledge: solving simple concept-based problems and conducting related laboratory experiments.Taxa IV-Procedural Knowledge: working knowledge of solving multi-concept engineering problems.Taxa V-Advanced Knowledge and Analytical Skills: inter-domain and open-ended problem solving skills.Taxa VI-Project-based Knowledge: creative, conceptual, analytical, design, manufacturing and management skills.Taxa VII-Professional Engineering Knowledge and Practices: life-long learning experiences, skills and practices.

Figure 5 .
Figure 5. Student outcomes assessment with respect to the specified ABET criteria in Spring 13 term-case studies were not available for this class.(E-Excellent, P-Proficient, A-Adequate, C-Concern and W-Weakness) satisfaction of this course compare to your satisfaction in the other undergraduate courses IN YOUR MAJOR in the last 2

Criterion 3 Outcomes: Outcome b:
Before attempting this module you need to have knowledge of: Software Testing, Test Outline and Test Case Development Upon completion of this module you be able to meet the following ABET An ability to design and conduct experiments, as well as to analyze and interpret data.Outcome e: An ability to identify, formulate and solve engineering problems Outcome g: Graduates have an ability to communicate effectively.Outcome k: Graduates have an ability to use the techniques, skills, and modern engineering tools necessary for engineering practice.Image Acquisition Modality The Software Requirements Specification (SRS) for the System has already been completed and accepted by the customer.Your QA Manager has asked your team to come up with test cases for the PRS that will be used to test the PRS features.Everyone on the team should be able to answer the following: What am I going to test?  What's the importance of having a formal template for test cases development?

Table 1 .
Sample Test Case Script Development TemplateUsing the steps shown in Figure1the authors have developed and delivered the test cases studies listed in Table1.List of developed and delivered test case studies //a brief description of the purpose of the test including a reference where appropriate to the requirement that is to be tested (consider providing references to the requirements specification, design specification, user guide, operations guide and/or installation guide), as well as any dependencies from or to other Test CasesTest Environment//a brief description of the environment under which the test is to be conducted (may include a description of the state of the AUT at the start of this test, details regarding the platform or operating system, as well as specific information about data used in this test)

Table 2 .
Case study analysis in software engineering and its relation to ABET criteria

Table 3 .
Expected pedagogical outcomes for software V&V course at author's institution

Table 4 .
Descriptors of ABET outcomes assessment vector

Table 5 .
Descriptive results from the general evaluation of course instructional activities

Table 6 .
Descriptive results from the general evaluation of course instructional activities-Part 1

Table 7 .
Descriptive results from the general evaluation of course instructional activities-Part 2