Printer-Friendly Version (PDF)

Preliminary Efficacy Study

A preliminary study on the effectiveness of a software program to improve student achievement in math
By Jefferey S. Lagozzino
July 30, 2004

The name of the school was withheld for this web publication.

Background:
M. Elementary School is one of ten elementary schools in a small school district in an eastern suburb of Los Angeles. M. Elementary serves approximately 950 students in grades kindergarten through sixth. Economically, the school’s student population is poor. All students (100%) qualify for the National School Lunch Program (a commonly used measure to express the poverty level of a school’s population). Ethnically, approximately 95% of all students are of Hispanic/Latino origin with the remaining 5% divided between students of Asian descent (3%) and Caucasian (2%). Linguistically, 60 % of the student population is classified as English Language Learners, with the majority of this group coming from Spanish speaking household. Parent educational levels at M. Elementary are predominantly low with 45% of its parents lacking a high school education. Statistics on the higher educational levels of parents are as follows: 34% have completed High School; 14% have taken, or are currently taking some college courses; 5% are college graduates; and 2% have completed or advanced degrees or post-graduate course work.

The preceding demographic information illustrates that students at M. Elementary are faced with numerous educational challenges that are not uncommon to today’s urban schools in Southern California. Not surprisingly, in terms of academic achievement, M. Elementary is in the bottom 20% of schools statewide. In the area of mathematics achievement, 32.9% of the students in grades two through sixth scored at proficient or advanced levels on standardized state tests administered last school year (2002-2003). While this level met the minimum requirement of 16% proficiency established by the state to comply with the No Child Left Behind federal legislation, it is far short of the lofty goal that all students meet proficiency targets by 2014.

In addition to the district’s core mathematics program (Saxon Math), additional instructional resources and programs could be instrumental in helping all students reach higher levels of achievement in mathematics. The purpose of this study is to determine the effectiveness of a new software application for math skill reinforcement and instruction. Get Ahead Math is a series of software programs that present seven grade level specific, core mathematical concepts in grades 3 through 8 through the use of video, step-by-step examples, practice problems, and quizzes. Embedded in the program is a pre-assessment that measures student readiness for instruction on the key concepts. In addition to unit quizzes, the program presents a final test that covers all seven of the key concepts as a measure of the student’s overall mastery of the program’s content.

Methodology/Research Approach:
The purpose of this study was to see if use of the Get Ahead Math software had a measurable impact on the improvement of the mathematical achievement of students. The study was conducted during the 2004 4-week summer school session (18 days) at M. Elementary with students from grades 3 through 6. The research approach was a pre/post test study with a control and experimental group at each grade level. Both groups received mathematics instruction in the district’s core math program (Saxon Math) by the same teacher to control for differences in curriculum and instruction. In addition to the core curriculum, the experimental group at each grade level visited the computer lab with their mathematics teacher 4-5 times each week for 45 minute sessions to utilize the Get Ahead Math program. A pre/post test was designed for each grade level by the researcher. The contents of the tests were grade level standards-based math problems that matched the key concepts presented in the Get Ahead Math program.

At the beginning of the study, the two groups of students at each grade level were compared on the following statistics to ensure that they were evenly matched: CAT-6 math scores, English language proficiency, and parent education levels. Mean scores for these statistics are presented in the Table 1.

Results/Analysis:

A simple comparison of pre- and post-test scores of all students formed the basis of the analysis for this study. A mean pre-test and post-test score was calculated for each group. For all grade levels the pre-test mean scores were nearly identical for both the control and experimental groups. Average post-test scores were higher for all of the experimental groups in comparison with the control groups. Students in the control groups also made progress; however their average post-test scores were lower than that of the experimental groups.

A second level of analysis compared the percentage of total possible correct responses each group achieved in the aggregate on the pre- and post-tests. For example, in Grade 3 the control group had 231 correct responses (sum of all scores) out of a total of 525 possible correct responses (21 students X 25 math problems = 525) on the pre-test, or 44% correct responses. On the post-test, this group correctly answered 56% of the problems for an improvement of 12%. This procedure was repeated for each group at every grade level. In this analysis the experimental group posted greater gains than the control group. A summary of the results appears in Table 2 below.

A multiple regression analysis using SPSS revealed no significant predictive indicators of achievement on the post-test. For the purpose of this analysis the total of correct responses on the post-test was the dependent variable with the group assignment (control or experimental), English language proficiency, and parent education levels as independent variables.

While not part of the formal analysis, some anecdotal observations may indicate that students in the experimental group were more motivated to attend summer school than their counterparts in the control groups. Each third and fourth grade group began the summer session with 30 enrolled students and each fifth and sixth grade group began with 20 enrolled students. Continuous and final enrollment is evident in the number of matched pre- and post- scores that were available for analysis at the end of the study. In all grade levels, more students in the experimental group remained for the duration of the study than students in the control group. Perhaps the individualized nature of the Get Ahead Math program and its presentation in the computer lab accounted for this higher rate of student retention throughout the summer school session. Increased motivation to learn math through the use of technology may be an important outcome of using the Get Ahead Math program and should be explored in future research.

Limitations of Study:

There were two limitations with this study. First of all, the length of the study was only 18 school days over a four week summer school session. In this time, the experimental groups from each grade level were able to complete only three out of the seven Get Ahead Math units. The researcher compensated for this limitation by designing a pre/post test for each level that only addressed the content of the units that would be taught during the short summer school session.

Secondly, the number of student subjects in the study at each grade level was too small to notice any significant patterns in the data. Future research on the effectiveness of the Get Ahead Math software could avoid these limitations by lengthening the time of the study and increasing the number of students participating.

Conclusion:

Considering that this study was conducted over a four week session, the minimal improvement in correct responses on the post-test and the relative greater improvement in correct responses of the experimental group compared to the control group is remarkable. The principal of M. Elementary was encouraged by the results and stated that more often than not she observes no academic growth that can be demonstrated in the short period of time allotted for summer school. She is hopeful that a longer implementation period will produce more dramatic results and looks forward to implementing the Get Ahead Math software as an after-school intervention program during the 2004-2005 school year.

As mentioned above, an analysis of the effectiveness of the Get Ahead Math program would be enhanced by a much larger study conducted over a longer period of time. With greater numbers of participating students at each grade level more sophisticated statistical analyses, such as multiple regression, could be made. Furthermore, the time length of a future study should be sufficient to allow students to progress through all units of the Get Ahead Math program. Considering that seven key concepts are developed at each grade level, the researcher recommends a minimum implementation period of seven weeks of at least four sessions per week.

Finally, in addition to measuring improvement on a test based on the Get Ahead Math curriculum, a comparison of prior year standardized test scores in mathematics with standardized test scores following the Get Ahead Math program implementation is suggested for future research. In this era of heightened national and state accountability for producing greater levels of student achievement, schools are challenged to implement proven, innovative, research-based instructional strategies, academic programs and interventions to enhance instruction and learning of all students. The results of this small study indicate that Get Ahead Math is a promising program worthy of further research.

Return to Research & Results