Data for Flipped Classroom Design : Using Student Feedback to Identify the Best Components from Online and Face-to-Face Classes

Colleges and universities have seen considerable enrollment growth in online courses during the past decade. However, online modalities are not optimal for all subject areas or students. There is growing interest in hybrid, blended, and flipped instruction as a way to incorporate the best of different delivery methods. This study investigates and identifies student preferences for both face-to-face and online learning. Participants were undergraduate students from a mix of freshman, junior, and senior level courses. An open response instrument was used to allow broad insights into students’ responses without biasing or limiting the feedback. Results suggest that the most positive impact with face-to-face learning is interaction through class discussions, group projects and other types of active learning. Females responded more positively than male students to interactivity in face-to-face classes. The data further indicates the most positive impact with online learning experiences is the class structure that supports flexibility, organization, and clear expectations. Nontraditional students reported more positively than traditional students about the benefits of flexible classes with clear course structures. This report should be of interest to educators who wish to take a research-based, student-centric, data-driven approach to the design of flipped or hybrid classrooms.


Introduction
For generations, educators have struggled with the challenge of how to more effectively educate the students they work with.Different theories of education have been proposed, pedagogical approaches developed, and technologies introduced to help improve the learning process (Skinner, 1954;Bates & Poole, 2003;Ross, Morrison, & Lowther, 2010).Changes in education have traditionally evolved slowly, but new developments in information technology are increasing this rate of change.Computers and networking technology are offering educators new and unique ways to connect with their students, inform, collaborate, and assess learning.The Internet, in particular, has been particularly disruptive as it has allowed for education in other than a synchronous, co-located setting (Bates, 2003) During the past decade online education, in its various formats, has grown significantly in terms of student enrollment (Clark & Mayer, 2007;Allen & Seaman, 2013).Many traditional institutions have developed their capacity for online learning and non-traditional entities have emerged and thrive offering exclusively online programs of study.These developments have made learning opportunities available to people who may not have otherwise been able to take advantage of higher education.Students in underserved locations, adult learners, and even traditional-aged college students who live on college campuses are increasingly choosing to take courses in an online format (Bolkan, 2013).
In spite of the growth in online enrollment and popularity of online courses, some student audiences, courses, and programs of study are not well-suited for this type of delivery system.There is growing interest in course-delivery methods that retain a significant element of traditional face-to-face content while augmenting the class with appropriate online elements.Hybrid teaching approaches including blended courses and flipped classes are actively being explored and adopted (Bonk & Graham, 2006;Keengwe, Onchawari, & Oigara, 2014).
The goal of these hybrid delivery models is to combine and leverage the beneficial features that are unique to ground-based with those that are unique to online courses.Like many new pedagogical approaches, instructors involved in hybrid-course design and delivery are still learning what does and doesn't work well for them.A growing body of anecdotal information is available on the web (Flipped Learning Network, 2014), but most early adopters are still largely left to their own experience to guide the process.This report presents the findings of an exploratory study that seeks to better understand which elements of traditional and online education are best suited for hybrid design.In particular, the study seeks to develop a better understanding of "market preference"-what students feel best meets their needs, rather than an individual professor's preferred approach to course development.Results of this study are compared and contrasted with other published guidelines and experiences.The methodology should be a useful starting point for other educators who wish to better understand the wants and needs of their local student market.The findings will also hopefully serve as a catalyst for future research in this area.
The modern university has its roots in the medieval institutions that were established in the 11th century.History suggests that the university evolved from earlier centers of learning (e.g., monastic schools, Madrasahs), and became established as a result of the urbanization of European society.These early universities were set up as independent scholastic guilds that brought together scholars and students.European universities proliferated during the period of 1400-1800 and students traveled from many countries to attend these institutions.The educational product of these universities was a degree, title, or diploma that was awarded after the recipient completed some prescribed course of study.The assumption was that the student would spend 3-4 years in residence at the university to earn a bachelor's degree, six for a master's degree (Colish, 1997;Grendler, 2004).
Modern universities have evolved from those of medieval times, and though the curriculum has broadened significantly, many of the pedagogical approaches that are still in common use have changed little since early times.One of the best known examples of this phenomenon is the use of lectures as a means of instructional delivery.Lectures are defined as oral presentations that are used to convey information or teach a particular topic.In the early days of the university, books were rare and the usual means of instruction was for the professor to read from a manuscript and the students would take notes on the material (Bligh, 2000).The diffusion of knowledge from oral presentation to hand-written notes has been a key element of higher education for centuries.Even today, in spite of criticisms of the method, lecturing is still a widely used form of instruction.Bligh (2000) argues that in spite of its criticisms, lecturing is as effective as most other teaching methods as a tool to transmit information.Even advocates for new approaches to teaching concede that the lecture is not going away."The lecture model-putting dozens, hundreds, or even thousands of students in a room with a professor-endures because it makes economic sense" (Berrett, 2012).
Although lectures remain a mainstay in college education, advances in educational technology have helped refine and improve the delivery of instruction.Educational technology has been defined as tools and techniques that support learning.Technology, in this case, may refer to machines, computers, new processes, systems, and related approaches (Bates, 2005).The evolution of educational technology has followed developments of complementary technologies.
As postal systems matured and delivery times became more consistent, correspondence programs were introduced and enjoyed periods of significant popularity (Pittman, 2003;Lee, 2008).By the 1930s, sound systems for motion pictures were standardized and the equipment needed to produce color movies with sound became widely available.This technology became a key part of the war effort in the 1940s when training films were produced to educate thousands of soldiers and support staff (Orgeron, Orgeron, & Streible, 2012).Broadcast technology gave rise to college courses being offered by radio and later by television (Levering, 1936).Developments in computer and networking technology have led to computer-based instruction, e-learning, and various forms of distance education (Harasim, Hiltz, Teles, & Turoff, 1995).The Internet has enabled on-line education and more recently MOOCs (massive open online courses) (Pappano, 2012).
In the United States, online courses have grown into a major element of higher education.Reports suggest that online enrollment significantly outpaced growth in traditional courses during the past decade.Over half of all college students are taking at least one of their courses online and this is expected to exceed 75% in coming years (Babson, 2013).Online education proponents offer numerous benefits including: improved access to education and learning schedules that compliment full-time work requirements (Bolkon, 2013).A growing number of colleges have expanded their online course inventory in recent years to take advantage of new market opportunities (Babson, 2013).Learning management software (LMS) and related hardware and software innovations are being developed and enhanced in response to this trend (Bates, 2005).However, in spite of the technological developments and improved pedagogy there are still weaknesses inherent in asynchronous delivery.
Recent studies examining large numbers of online students suggest that completion rates, grade performance, and degree completion may be lower than for students in traditional classes (Community College Research Center, 2013).Some of the reasons cited for problems in online courses include: technical difficulties, a feeling of isolation, a relative lack of structure, and a general lack of support (Youngberg, 2012).These findings in no way suggest that online education is not a viable educational option.Rather, that for some students a different approach might be a better fit.One possibility is a blended or hybrid course-a modality that combines face-to-face interaction with web-delivered educational content (Garrison & Kanuka, 2004).A US Department of Education meta-analysis of online learning suggests that students enrolled in blended courses perform better than students in either face-to-face or online classes (Means et al., 2010).A market-research study conducted by the Instructional Technology Council (2009) found that not only do students do well in blended courses, but a growing number of students actually prefer them to those offered in other modalities.
One form of hybrid course design that is receiving a lot of attention is Flipped Teaching (also known as flipped classroom, inverted instruction, backward class, upside-down teaching) (Keengwe, Onchawari, & Oigara, 2014).In a flipped model, students are expected to prepare themselves outside of class using web-delivered materials.During class, the teacher does not spend much time on basic information transfer (as this is to be done outside of class).Instead, the students are expected to apply and expand their knowledge of the subject by solving problems, interacting with classmates, working on projects, or sharing artifacts of their learning (Bergmann & Sams, 2012).
Although there is growing interest in flipped teaching, the approach is still relatively new and few educators have direct experience with it.One potential hindrance to more widespread adoption of a flipped teaching approach may be the simple heuristic commonly used to describe how such classes are designed.The popular explanation of whatever you used to assign as homework-do in class; and whatever you did in class-assign as homework may be suitable for some classes (e.g., a high-school math class), but may not map well to other areas.
As part of the overall effort to better understand and design hybrid courses, an exploratory study was undertaken.The goal of the study was to better understand what elements of face-to-face and online courses work well and should be considered for use in a hybrid class.A related goal is to determine which common elements of face-to-face and online courses are less effective.The objective is to identify the "best of the best" characteristics from the perspective of the students, rather than the preferences of an individual professor.It is hoped that the findings will help guide future research in this area as well as provide educators with useful insights as they experiment with instructional design.

Method
The objective of this exploratory study is to identify the factors influencing students' positive experiences in both face-to-face and online classes.Student data was collected from a mix of 100-level, 300-level and 400-level university courses at a public institution.The university is well established, having offered traditional bachelor's for over a century and online degree programs since the late 1990s.The survey was designed to elicit the salient attitudes of students' experiences in face-to-face and online classes.Students also self-reported their gender (male-female) and generation (traditional student under age 25 or non-traditional student age 25 or over).
Consistent with the exploratory nature of the study, an open-ended questionnaire was used.Open-response surveys allow more insight to students' knowledge and permit a comprehensive collection of student responses without a-priori assumptions limiting the reporting (Nehm & Schonfeld, 2008).However, open response instruments also pose challenges.The first is that students may dislike writing answers, or not know how much to write for each response.To address the first challenge the instrument was kept short, with the following open response questions: Question 1: "Think back on the face-to-face (on-ground) classes you've taken.What were some of the characteristics of the best ones?What was it that made some stand out in your mind?"Question 2: "Think back on the online classes you've taken.What were some of the characteristics of the best ones?What was it that made some stand out in your mind?" A second challenge of an open-response instrument is the scoring element.The approach taken was to identify keywords and group those terms into common themes.This evaluative process was conducted by multiple reviewers and their results were compared and aggregated.A characteristic of open-response questions is that respondents will not all use the exact same phrasing, but similar or analogous terms can be identified and placed in categories.For example, "teacher", "professor", and "instructor" are considered equivalent for the purpose of this study even though a professor and an instructor can be uniquely distinguishing terms in a different context.
Likewise functional descriptions were identified as similar and grouped as well, with "in-class discussions", "classroom interaction", and "class debates and conversations" all sharing a common theme.Through this refinement process five main categories emerged which were labeled as: "Interaction", "Instructor", "Class Structure", "Material" and "LMS (Learning Management System)".Some responses were not included in the results, either because the student left the questions blank, or because the responses could not accurately be associated with any category, such as the following: "I particularly remember efficient and short algorithms-then again that was on an IBM mainframe and computer time meant money-fortunately they had just replaced the punch card readers." The positive experience data was then processed by frequency of student reporting for each of the main categories for face-to-face classes.The face-to-face data was then further analyzed by gender and by age group.The positive experience data for online classes was processed in the same manner, including frequency of student reporting and additional analysis based on gender and age group.

Face-To-Face Classes
Students' responses to the characteristics of their best face-to-face class experiences were revealing.The characteristic most frequently reported by students has having a positive impact on face-to-face learning experiences was "Interaction" which includes class discussions, group projects and other aspects of active learning.T-Test analyses were performed to compare "Interaction" to the other main categories, and the difference was statistically significant in each case (p <= 0.03).
The positive reporting for "Instructor" was significantly below "Interaction" but was above the remaining main categories (p <= 0.05).Table 1 shows the four main categories identified by students as important for face-to-face classes along with some sample student open-response comments for each category."classroom camaraderie and discussion" "more than lecture" "I enjoy the class debates and discussions." "The best part of face to face classes was the instant feedback from the teacher if a question was raised in class, or the ability to expand on or clarify a specific topic." "I love the interaction with the classes." "discussion based; lots of questions asked" "Interaction makes a class in my opinion.Anybody can sit in class and not pay attention, when a teacher gets the students involved I tend to learn more." "classes that you interact in as it's easy to understand in that kind of environment" Instructor (31%) "He made class interesting.He did not talk above us, but like a peer." "the personality of the instructor" "When teachers are passionate about what they teach and they want to help their students not only learn the material but also to become better people." "I appreciate when an instructor can relate the material being covered in class to real life applications." "They all made it relevant with what was going on around us." Class Structure (15%) "concentrate on certain areas of study" "keeping the class info to the point and not making it boring" "Organization is such an important factor in teaching." Material (4%) "interesting material" "interesting assignments" The positive characteristics of face-to-face classes were further sorted and analyzed according to students' Gender and Age Group, with a significant difference found in the Gender results (Figure 1).It is noteworthy that females reported more positive elements for face-to-face classes than their male counterparts, even when adjusted for the larger percentage of female participants in the study (57%).Perhaps females are more tuned to the interactive possibilities of a face-to-face class.Perhaps males are more critical consumers of face-to-face classes.Regardless, in the highest reported category (Interaction), the female/male difference was statistically significant (p <= 0.05).So not only is Interaction the most important characteristic of a positive face-to-face learning experience, it is particularly important to female students.This finding is insightful and raises several interesting questions.
Figure 1.Face-to-face classes by gender

Online Classes
The second question in the survey asked about positive characteristics of online courses, and again the results were informative.The characteristic most frequently reported by students for having a positive impact on online learning experiences was "Class Structure" which includes flexibility, schedule, organization, and clear expectations.T-Test analysis showed Class Structure to be statistically significant compared to each of the other main categories (p <= 0.01).The second most common response for positive impact in online classes was "Interaction", which was reported more frequently than any of the remaining categories (p <= 0.01).
The data suggest that the broad categories of Instructor, Material, and the Learning Management System (LMS) all lag Class Structure and Interaction as factors that positively impact online classes.Table 2 shows the main categories identified by students as important for online classes along with some sample student open-response comments for each category."Unrestricted availability to attend courses regardless of the continent the student is on." "I love working at my own pace.I have the time to make sense of the information and go with other information when I am ready." "I love the video lectures.I also appreciate having a weekly schedule to follow, as opposed to large lumps of work being required at spaced out intervals." "On-line classes are great as long as the teacher is very organized, timely and students know exactly what to expect." "Professor was organized and laid out every due date and assignment on the syllabus." "A well-structured course that can have "beefy" assignments but doesn't require the student to constantly check for random updates and due dates that don't sync well." Interaction (15%) "The class mates were very helpful and very nice to interact with." "Personal interaction with the instructor and classmates through email and discussion boards." "What I really enjoyed about some classes was the feedback teachers gave on my work and the timely grading of the written work." Instructor (3%) "The instructor answers emails and questions in a timely manner." "Professors willing to work with students on assignments." "Instructor discussion videos really help things stick." Interface/GUI (2%) "The online text and discussion boards were very user friendly." "The interface used for taking the quizzes and exams were easy to navigate." The positive online characteristics were sorted and analyzed according to students' Gender and Age Group, with a significant difference found in the Age Group results (Figure 2).Older nontraditional students were particularly impressed by good Class Structure and the difference on this issue between traditional and non-traditional was statistically significant (p <= 0.01).Perhaps this is due to non-traditional students having busier personal schedules, or perhaps they simply have come to appreciate the value of structure more than their younger counterparts.
Figure 2. Generation results for characteristics of best online classes

Discussion
The results of this study provide useful insights that may be helpful for evaluating the many opinions and reported experiences associated with flipping a class.Rather than designing a study to support (or reject) a particular faculty bias, the work reported here allowed student experiences and preferences to guide the results.
The study used open response student data to better understand a flipped classroom design based on the "best of the best" approach of keeping what students report as the strongest characteristics of traditional face-to-face classes combined with what students report as best in the their experiences with online classes.
The results are encouraging, offering a new contribution by providing comparative analysis of student experiences comparing face-to-face and online classes.Furthermore, the findings are consistent with results of other research projects conducted using different instructional designs.For example, the findings show students value active learning and interaction during face-to-face meetings; including peer interaction through small groups and discussion.This is consistent with the pedagogical research which postulates that learning is an active process that is "situated" with both a social and cultural context (Vygotsky, 1978).The flipped classroom provides great opportunities for students to bring outside of class reading and research into a very social interactive classroom discussion.
The findings also suggest that females are more positive than male students in their perception of the value of face-to-face classes.In particular, female respondents were significantly (p <= 0.05) more positive about the role of Interaction in face-to-face classes.Given the national trend of an increasing number of female students earning Bachelor's degrees (Figure 3, DiPrete & Buckmann, 2013), it would be interesting to extend the study to a program, department, or college level (rather than a multi-class level) to see if Interaction is a significant factor in major selection decisions by female students.This could be particularly insightful for programs seeking enrollment growth.

Figure 3 .
Figure 3. Increase in females earning bachelor's degrees

Table 1 .
Best characteristics of face-to-face classes

Table 2 .
Best characteristics of online classes CategorySample Student Comments The flexibility of convenience concerning when you do work for a class.""Beingable to work and go to school at the same time."