Worksheet Usage, Reading Achievement, Classes' Lack of

International Journal of Education in Mathematics, Science and Technology

Volume 2, Number 2, April 2014, Page 96-106

ISSN: 2147-611X

Worksheet Usage, Reading Achievement, Classes’ Lack of Readiness, and

Science Achievement: A Cross-Country Comparison

Che-Di Lee

National Taiwan Normal University

Abstract

Instructional written materials play important roles as teachers’ agents in effective teaching practices.

Worksheets are one of the most frequently used materials. In this exploratory study, the relationships between

worksheet usage and science achievement in 32 countries were examined through the use of TIMSS and PIRLS

data and multiple regression analysis. Based on two dimensions, five types of relationships among science

achievement, worksheet usage, and other related variables are identified. The first dimension is whether the

status of significance in the association of worksheets used as a basis and science achievement changes before

and after controlling four teacher and school variables: schools’ emphases on academic success, safety and

orderliness of school, teachers’ confidence in teaching science, and instructional engagement of students. The

second dimension is the interaction of worksheets as a basis and classes’ lack of readiness. The interaction

between worksheets as a basis and reading achievement in science achievement is found to be not significantly

different from zero in all participating countries. Four directions of further investigation are suggested based on

the results.

Key words: Science education, Worksheets, Elementary education, Secondary analysis

Introduction

Worksheets have been used in teaching practices for a long time. In modern time, worksheets have even become

a driving force of curriculum in some countries (Lesley & Labbo, 2003; Martin, Mullis, Foy, & Stanco, 2012;

Reid, 1984). Anderson et al. (1985) reported that in 1985, thousands of elementary students in the United States

completed approximately 1,000 worksheets per person to acquire literacy in a school year. Teachers use

worksheets for the purposes of supporting studying, promoting active learning, raising interest in learning

science, and assessment. Many studies suggest that well-designed worksheets have had positive impacts on

students’ learning achievement (Sasmaz-Oren & Ormanci, 2012). However, researchers observed that there

were many inappropriately designed and misused worksheets that hindered learning (Lesley & Labbo, 2003). In

this exploratory study, the relationships between worksheet usage and science achievement in 32 countries are

examined.

Worksheet and Achievement

Worksheets can be useful in many ways in terms of academic achievement. For example, as supplements to

textbooks, worksheets can be used to add information for particular classes. In addition, blanks in worksheets

are invitations for students to fill in gaps; they are opportunities for knowledge construction. Well-designed

questions in worksheets can draw students’ interest when paired with proper teaching methods. Furthermore,

worksheets play a variety of functions in different contexts. McDowell and Waddling (1985) suggested that

during laboratory investigations, properly designed worksheets can help teachers overcome the problems of time

demanding and enable teachers to enhance students’ acquisition of knowledge and skills. Kisiel (2003) pointed

that in activities while visiting museums, worksheets can function as advance organizers, helping students to

organize their observations and knowledge in a confusing learning environment. Krombab and Harms (2008)

concluded that worksheets are effective in helping students aged 11–15 to acquire knowledge such as

Corresponding Author: Che-Di Lee, ch[email protected]u.tw

Lee

biodiversity in a natural history museum because they can structure the visit, keep students’ attention on certain

objects, and form a basis for follow-up coursework. As an assessment tool, worksheets can be used by teachers

to understand students’ previous knowledge, outcome of learning, and the process of learning; at the same time,

they can be used to enable students to monitor the progress of their own learning.

In order to ensure worksheet’s effectiveness, many studies focused on the design of worksheets (Campbell,

1999; Hoener, Salend, & Kay, 1997; Sasmaz-Oren & Ormanci, 2012). The basis for successfully transferring

message to students is layout. To enhance teachers’ abilities to design worksheets, Rotter (2006) proposed the

layout principle COLA (contrast, orientation, lettering, and artwork). In worksheets, characteristics of questions

are important factors. Calderhead et al. (2006) demonstrated that the arrangement of items with different levels

of cognitive difficulty can affect students’ learning results. Formats of information, or scaffolds, in worksheets

are also of concern to educators. For instance Wolf et al. (2010) devised three formats of critical thinking tools

to promote conceptual understanding in physical geography and Ueckert and Gess-Newsome (2008) advocated

for a “conceptual flow graphic” to make active learning possible.

In addition to the positive impacts of worksheets on academic achievement, there are negative impacts. Lesley

and Labbo (2003) argued that mass-produced worksheets are not helpful for achieving educational goals.

Although the worksheets they observed were focused on literacy learning, their findings can still shed some

light on worksheet usage for science learning. According to their remarks, the aspects of worksheet problems

included the format of texts (e.g., that the print and the spaces allotted for students to write in are too small);

reading demand (e.g., that the language of instruction was too complex and required teacher explanation);

openness of questions, some of which offered only one correct way to respond and could not reward students for

their natural curiosity; the challenge of tasks (e.g., that tasks were boring or designed for practicing skills

repeatedly instead of making students learn new strategies or techniques), and the relationship between students’

interests and tasks. In addition to the quality issues, as students completing worksheets, their cognitive processes

can also make worksheets invalid. Ueckert and Gess-Newsome (2008) noted that students use a word-matching

strategy match words in questions with the corresponding sentences in the textbook, and this keeps them in a

passive learning status.

Worksheet, Reading Achievement, and Science Achievement

Worksheets are a kind of written material, so reading demand may be a barrier to students with low reading

abilities. Researchers suggested that teachers should use easier language to support students (Rix, 2006). For

example, O’Leary (2011) designed a format of worksheets with low average reading difficulty. Questions in the

beginning are carefully matched with low reading ability students and subsequent questions require increasing

levels of literacy. The result showed that this kind of worksheets can improve student engagement and on-task

behavior during independent worksheet activities. There are a number of factors contributing to reading

difficulty, such as organization of materials, syntax, word length, sentence length, word frequency, typeface, and

line spacing (Department of Education and Science, 2007; Meyer, 2003; O’Leary, 2011). If teachers carefully

control these factors and use available readability formulas to reduce the reading demand, or offer oral

explanations to words in worksheets, the association between worksheet usage and science achievements will be

the same regardless of students’ reading achievements.

The debate about the difficulty of language used in worksheets has been raised in literature (O’Leary, 2011).

Some researchers insist that easy language negates worksheets’ abilities to challenge students and offer

opportunities of language acquisition (Hayes, Wolfer, & Wolfe, 1996). In addition, although there are rules for

controlling the reading demand, according to Reid’s (1984) study, only about 30% of worksheet writers checked

readability despite knowing that it is the essential. Under this kind of condition, students with low reading

abilities have problems completing worksheets and the associations between worksheet usage and science

achievement therefore differ among students with different reading levels.

Worksheet, Lack of Readiness, and Science Achievement

According to Reid (1984), teachers tend to use worksheets with low-achievement classes. The reasons for this

tendency may be twofold. One is that textbooks are designed for general students and need to be adapted.

However, worksheets can offer relevant questions and motivate students, both of which are functions that were

ranked as best performed by teachers surveyed in the Reid’s (1984) study. Secondly, as written material,

worksheets are able to act as agents of teachers to lead students’ attentions and give students opportunities to

work independently, so the students can work at their own paces and the teacher can have time to take care of

those students who need more help (McDowell & Waddling, 1985).

IJEMST (International Journal of Education in Mathematics, Science and Technology)

If worksheets are properly designed and used, they can support students’ thought. However, many teachers are

concerned by the “repetitive” nature of worksheets, the risks of student boredom, and a lack of the pedagogical

knowledge and skills required to prevent children from thinking insufficiently before completing worksheets

(Reid, 1984). These factors may cause failure in the application of worksheets in classes lacking readiness.

Research Questions

The purpose of this study is to explore the association between worksheet usage and science achievement in

grade four students across the countries in the fifth cycle of the Trends in International Mathematics and Science

Study (TIMSS 2011) and the third cycle of the Progress in International Reading Literacy Study (PIRLS 2011).

To gain an overview of the association, the first question that arises is: “Is there difference in science

achievement between situations in which worksheets are used as a basis for instruction and those in which they

are used as supplements, and between those in which they are used as supplements and those in which they are

not used?”

To avoid the confounding effect of other teacher and school level variables, the four important variables,

schools’ emphasis on academic success (EAS), safty and orderliness of school (SOS), teachers’ confidence in

teaching science (CTS), and instructional engagement of students (IES), identified by Martin et al. (2012) are

controlled. Accordingly, the second question is “After the four variables are controlled, is there difference in

science achievement (SA) between those situations in which worksheets are used as a basis (WB) and those in

which they are not used as a basis?”

In considering the issues about reading achievement (RA) and classes’ lack of readiness (LR), the third and

fourth questions examine “whether RA contributes to explanations of the variability in the relationship between

WB and SA” and “whether LR contributes to explanations of the variablility in the relationship between

worksheet as a basis (WB) and SA.”

Method

Sample

This study is a secondary analysis research. Data were collected in TIMSS 2011 and PIRLS 2011. Thirty-four

countries participated in both surveys (Martin & Mullis, 2013). The target population is grade four students in

thirty-two countries and grade six students in two countries. In this study, the data from the thirty-two countries

were analyzed.

TIMSS and PIRLS use two-stage stratified cluster sampling design (Martin et al., 2012). In each county, schools

were selected first and one or two classes were selected from those schools. The students in the selected classes

composed the sample. The sample size of each country was ranged from 3,121 to 14,720.

Dataset Preparation

After scaling, the International Association for the Evaluation of Educational Achievement (IEA) released the

TIMSS and PIRLS 2011 datasets in September 2013. In the datasets, there are reading, science, and

mathematics achievement scores with five plausible values; responses of items in achievement tests and

background questionnaires; and derived background variables, such as students’ confidence in learning science,

teachers’ confidence in teaching science, and safety and orderliness of school (Foy, 2013).

In this study, there is one dependent variable, SA, and seven independent variables: WB, RA, LR, EAS, SOS,

CTS, and IES. SA and RA are achievement scores. EAS, SOS, CTS, and IES are derived teacher and school

variables. The above six variables can be directly retrieved from the datasets provided by IEA (Foy, 2013). WB

and LR are defined in this study as the following.

Worksheet usage was one item in the teacher questionnaire (Martin et al., 2012). Teachers were asked when

they taught science to the class how they used “workbooks or worksheets.” There were three options for them to

choose from: “basis for instruction,” “supplement,” and “not used.” Because the percentage of teachers who did

Lee

not use workbooks or worksheets was small or even zero in some countries, the responses were recorded in two

categories, “as a basis” and “not as a basis.”

The variable of LR was constructed with the use of three items in the teacher questionnaire. In the questionnaire,

teachers were asked to what extent “students lacking prerequisite knowledge or skills,” “disruptive students,”

and “uninterested students” limited how they taught their classes. There were four options for them to choose

from: “a lot,” “some,” “not at all,” and “not applicable.” Every item was recoded in two-level variable: “A lot or

some (level 1)” and “not at all or not applicable (level 0).” The three indicators were treated as formative, and

LR was the standardized sum of the three recoded variables.

Analytical Strategy

Multiple regression is used to answer the research questions. The issue of estimation of standard error has to be

addressed first because the sampling method in TIMSS and PIRLS is stratified cluster sampling instead of

simple random sampling. To correctly estimate sampling error, the Jackknife repeated replication methodology

was applied. To correctly estimate measurement error, the plausible-value method was applied. The IDB

Analyzer developed by the IEA Data Processing and Research Center (2013) was used with the statistical

software SPSS to implement the two above methods in regression analysis.

To test the significance of the differences between group means of science achievement, Model 0 was analyzed.

Model 0:

WNOcWABc

WNOWAB



cSA

WAB stands for “using worksheets as a basis” and WNO stands for “not using worksheets.” The reference

group is students taught by teachers “using worksheets as supplements (WSP).” The value of c

WAB

is the

difference in average science achievements for students taught by teachers WAB and WSP. The value of c

WNO

the difference in average science achievements for students taught by teachers WNO and WSP.

To examine the effect of controlling the four teacher and school variables (EAS, SOS, CTS, and IES), Model 1

and Model 2 were compared.

Model 1:

WBc



cSA

WB stands for “using worksheets as a basis.” The value of c

is the difference in average science achievements

for students taught by teachers using worksheets as a basis and not as a basis.

Model 2:

IEScCTScSOScEAScWBc

IESCTSSSEASWB



cSA

To test the significance of the interaction between WB and RA in SA, Model 3 was analyzed. In Model 3, RA is

centered on class mean because the relationship between RA and SA on an individual level is concerned and so

RA should not include the effect of class-level and above-class-level variables (Hoffman & Gavin, 1998).

Model 3:

)()(

cSA

class

RAWB

class

IESCTSSSEASWB

RARAWBcRARAc

IEScCTScSOScEAScWBc







To test the significance of the interaction between WB and LR on SA, Model 4 was analyzed.

Model 4:

LRWBcLRc

RARAWBcRARAc

IEScCTScSOScEAScWBc

LRWRLR

class

RAWB

class

IESCTSSSEASWB









)()(

cSA

IJEMST (International Journal of Education in Mathematics, Science and Technology)

Results

Association between Worksheet Usage and Science Achievement

As shown in Table 1, on average across countries, 42% of students used worksheets as a basis, 56% of students

used worksheets as supplements, and only 2% of students did not use worksheets. The percentage of students

not using worksheets is larger than 5% only in five countries, Australia, Finland, Iran, Malta, and Sweden, and

is zero in five countries, Croatia, Georgia, Norway, Singapore, and Taiwan.

Table 1 shows that the difference in average science achievements for students taught by teachers using

worksheets as a basis and as supplements is significant in eight out of the 32 countries. The average science

achievement of students taught by teachers using worksheets as a basis is higher in only two of the eight

countries, the Czech Republic and Saudi Arabia. The international average difference is significantly less than

zero (-4.2).

Association between Worksheet Usage and Science Achievement after Controlling Teacher and School

Variables

Table 2 provides the information about the difference in average science achievements between students taught

by teachers using worksheets as a basis and not as a basis (as seen in the column of WB in Model 1. i.e., the

coefficient of WB). As shown in the coefficient of WB in Model 2 (Table 2), after controlling the variables of

EAS, SOS, CTS, and IES, the difference in average science achievements for students taught by teachers using

worksheets as a basis and those taught by teachers not using them as a basis is significantly different from zero

in six countries. In five out of the six countries, Italy, Northern Ireland, Qatar, Saudi Arabia, and the United

Arab Emirates, the difference is significant in Model 1, too. The sixth country is Germany, in which the

difference is insignificant in Model 1. After controlling the four teacher and school variables, the coefficient of

WB is insignificant in the Czech Republic, Malta, and Singapore while the coefficient is significant before

controlling the four variables (Model 1) in these three countries.

Interaction between Worksheets as a Basis and Reading Achievement

The coefficient of WB*RA in Model 3 (Table 2) show there is no statistically significant interaction between

worksheets as a basis and reading achievement in the prediction of science achievement in all countries.

Interaction between Worksheets as a Basis and Class’s Lack of Readiness

According to the information provided by Table 2 in the column WB*LR, the interaction between WB and LR

in students’ science achievement is significant in eight out of the 32 countries. In the eight countries, the

coefficient of the interaction term is positive in six countries and negative in two countries.

The coefficient is positive for students in Australia, Finland, Morocco, Norway, Qatar, and the United Arab

Emirates. The positive value of the coefficient of WB*LR means that the effect of WB on science achievement

for students in classes lacking readiness is higher than it for those in classes not lacking readiness. For example,

in Finland, after controlling EAS, SOS, CTS, EIS, and RA, the association of WB and SA for students in classes

not lacking readiness (LR=0) is -1.4 and is not significantly different from zero; the association for students in

classes lacking readiness (LR=1) is 8.6 (=-1.4+10.0), which is significantly different from zero. In other words,

in Finland, worksheets are more effective for students in classes lacking readiness to learn science. In Qatar and

the United Arab Emirates, the situation is slightly different. In both countries in which the association for

students in classes not lacking readiness (LR=0) is negative (-64.3 and -33.2), the association for students in

classes lacking readiness (LR=1) is still negative (-38.1 and -10.5). In spite of this, the effect size becomes

smaller, which means the negative association of worksheets is reduced when they are applied to classes lacking

readiness.

On the contrary, in Italy and Malta, the coefficient is negative. Worksheets in both countries are less effective

for students in classes lacking readiness to learn science. After controlling EAS, SOS, CTS, EIS, and RA, the

association of WB and SA for students in classes not lacking readiness (LR=0) is -11.7 for Italy and 5.1 for

Malta; the association for students in classes lacking readiness (LR=1) is -31.5 for Italy and 1.2 for Malta.

100

Lee

Table 1: Worksheet usage and average science achievement (results from Model 0)

Country

Worksheet As a Basis

Worksheet as Supplements

Worksheet Not Used

WAB

– SA

WSP

WNO

– SA

WSP

Percent of

Students

Average SA

Percent of

Students

Average SA

Percent of

Students

Average SA

Australia

15.5 (3.2)

504.8 (8.7)

75.9 (3.7)

518.5 (4.0)

8.6 (2.4)

513.2 (9.4)

-13.8 (9.5)

-5.3 (10.5)

Austria

33.3 (2.9)

530.2 (3.8)

66.5 (2.9)

526.5 (3.0)

0.2 (0.2)

532.6 (5.8)

3.7 (4.4)

6.1 (6.9)

Azerbaijan

33.9 (3.8)

448.2 (9.0)

65.4 (3.9)

426.6 (7.8)

0.7 (0.5)

410.3 (11.4)

21.6 (11.5)

-16.2(14.2)

Croatia

28.7 (3.4)

511.7 (3.0)

71.3 (3.4)

513.0 (2.2)

0.0 (0.0)

- -

-1.2 (3.6)

- -

Czech Rep.

44.9 (3.8)

538.3 (3.7)

51.9 (3.8)

527.9 (3.1)

3.2 (1.6)

539.9 (10.1)

10.4 (4.9)*

12.0 (10.6)

Finland

39.6 (3.0)

565.4 (3.3)

54.2 (3.3)

570.3 (2.6)

6.2 (1.7)

565.3 (12.4)

-4.9 (3.9)

-5.0 (13.1)

Georgia

53.7 (4.1)

449.4 (6.0)

46.3 (4.1)

451.8 (5.6)

0.0 (0.0)

- -

-2.4 (9.0)

- -

Germany

58.4 (3.5)

523.3 (3.1)

41.4 (3.5)

529.3 (3.4)

0.1 (0.1)

519.4 (11.7)

-6.0 (4.7)

-9.9 (12.4)

Hong Kong

44.3 (4.8)

543.7 (4.3)

55.0 (4.7)

532.1 (4.1)

0.7 (0.7)

548.1 (6.9)

11.6 (6.1)

16.0 (7.2)*

Hungary

70.3 (3.3)

528.9 (4.8)

27.7 (3.4)

531.2 (7.9)

2.0 (0.9)

537.9 (20.7)

-2.3 (10.3)

6.6 (23.2)

Iran

14.9 (3.3)

438.1 (11.5)

78.8 (3.5)

454.1 (4.5)

6.3 (1.5)

398.9 (20.6)

-16.0(13.5)

-55.2(21.4)*

Ireland

11.7 (2.3)

503.9 (8.9)

85.5 (2.6)

514.1 (3.4)

2.9 (1.2)

521.4 (11.5)

-10.2 (9.8)

7.3 (11.2)

Italy

23.3 (3.3)

505.7 (7.1)

76.0 (3.2)

528.1 (2.6)

0.7 (0.7)

458.4 (7.0)

-22.4 (7.5)*

-69.7 (0.0)

Lithuania

69.7 (3.6)

509.8 (2.8)

30.1 (3.5)

510.0 (5.1)

0.2 (0.2)

462.2 (8.4)

-0.2 (6.0)

-47.8(10.5)*

Malta

33.7 (0.1)

437.3 (2.6)

58.1 (0.1)

444.5 (2.1)

8.2 (0.1)

438.2 (3.8)

-7.3 (2.7)*

-6.3 (4.2)

Morocco

67.9 (3.3)

240.8 (5.6)

28.0 (3.3)

243.0 (10.3)

4.2 (1.5)

275.6 (35.9)

-2.2 (12.7)

32.7 (38.5)

North. Ireland

16.3 (3.0)

498.9 (6.7)

82.2 (3.2)

516.7 (3.3)

1.5 (1.1)

489.8 (22.6)

-17.8 (7.5)*

-26.8(23.4)

Norway

39.2 (5.2)

489.4 (4.2)

60.8 (5.2)

488.6 (2.4)

0.0 (0.0)

- -

0.8 (4.6)

- -

Oman

45.5 (3.1)

361.9 (5.6)

54.3 (3.1)

376.1 (5.7)

0.2 (0.1)

353.9(103.6)

-14.2 (8.1)

-22.2(103.4)

Poland

57.8 (3.8)

499.0 (3.1)

41.9 (3.9)

500.3 (3.0)

0.4 (0.4)

556.4 (7.3)

-1.3 (4.2)

56.0 (0.0)

Portugal

34.5 (4.0)

523.8 (4.8)

63.6 (4.1)

516.2 (5.4)

1.9 (1.5)

488.3 (7.0)

7.6 (7.0)

-27.8 (8.6)*

Qatar

57.0 (2.9)

359.2 (7.4)

41.5 (3.1)

419.4 (10.7)

1.5 (0.9)

492.3 (20.7)

-60.1(14.1)*

72.9 (25.9)*

Romania

35.8 (4.1)

499.0 (9.0)

64.0 (4.2)

501.2 (8.5)

0.1 (0.1)

438.9 (17.3)

-2.2 (12.5)

-62.3(19.1)*

Russian Fed.

47.7 (4.2)

551.5 (4.7)

50.8 (4.1)

545.8 (4.0)

1.4 (0.8)

548.7 (66.6)

5.8 (5.8)

3.0 (66.8)

Saudi Arabia

51.6 (4.0)

441.7 (7.3)

46.6 (3.9)

402.4 (9.0)

1.8 (0.9)

441.4 (46.5)

39.3(12.3)*

39.0 (48.2)

Singapore

68.6 (2.6)

574.6 (4.3)

31.4 (2.6)

589.4 (5.6)

0.0 (0.0)

- -

-14.8 (7.3)*

- -

Slovak Rep.

38.5 (3.0)

530.3 (5.3)

58.9 (3.2)

527.8 (3.8)

2.6 (1.3)

474.1 (22.7)

2.5 (5.7)

-53.7(23.4)*

Slovenia

50.3 (3.8)

512.5 (2.7)

48.2 (3.9)

520.3 (3.3)

1.5 (0.8)

504.8 (21.0)

-7.8 (4.6)

-15.5(20.6)

Spain

34.0 (3.7)

507.9 (5.2)

63.9 (3.7)

498.7 (3.6)

2.0 (0.9)

495.3 (17.7)

9.2 (6.5)

-3.5 (18.1)

Sweden

18.5 (3.8)

527.5 (8.0)

67.5 (4.7)

531.4 (4.0)

14.0 (3.6)

539.9 (4.6)

-3.9 (8.2)

8.6 (6.1)

Taiwan

44.2 (4.1)

549.0 (2.6)

55.8 (4.1)

547.6 (3.0)

0.0 (0.0)

- -

1.4 (4.3)

- -

UAE

53.5 (2.4)

406.5 (3.9)

45.9 (2.4)

442.9 (5.2)

0.6 (0.2)

440.2 (42.5)

-36.4 (7.7)*

-2.7 (42.9)

Int. Average

41.8 (0.6)

487.9 (1.0)

55.9 (0.6)

492.1 (0.9)

2.3 (0.2)

480.9 (5.8)

-4.2 (1.4)*

-6.3 (6.0)

Note: SA

WAB

=Average science achievement of students using worksheets as a basis; SA

WSP

= Average science achievement of students

using worksheets as supplements; SA

WNO

= Average science achievement of students not using worksheets; UAE = United Arab Emirates

(): Standard errors appear in parentheses. *p< .05

101

IJEMST (International Journal of Education in Mathematics, Science and Technology)

Table 2: Coefficients for multiple regression analysis

Country

Model

Worksheet

as a Basis

(WB)

Reading

Ach. (RA)

WB*RA

Lack of

Readiness

(LR)

WB*LR

Controlled Variable

EAS

SOS

CTS

IES

Australia

-13.2 (9.4)

-8.2 (9.6)

3.2 (2.1)

7.1(2.2)*

0.7 (1.5)

1.8 (1.9)

-5.7 (9.6)

0.76 (0.02)*

0.03 (0.05)

3.1 (2.1)

7.1(2.2)*

1.0 (1.4)

1.7 (1.9)

-14.0(12.5)

0.76 (0.02)*

0.03 (0.04)

-17.2(3.0)*

35.4(15.6)*

1.0 (2.1)

5.9(2.2)*

2.0 (1.3)

0.1 (1.9)

Austria

3.7 (4.4)

-0.1 (4.2)

4.8 (1.4)*

4.4(1.5)*

-2.4(1.2)*

-1.7 (1.3)

-0.6 (4.2)

0.85 (0.01)*

-0.04(0.03)

4.8 (1.4)*

4.4(1.6)*

-2.2 (1.2)

-1.6 (1.2)

-0.8 (4.2)

0.84 (0.01)*

-0.04(0.03)

-3.7 (2.4)

-2.7 (3.8)

4.3 (1.4)*

3.8(1.6)*

-2.5(1.2)*

-2.0 (1.3)

Azerbaijan

21.8 (11.5)

18.5 (11.5)

4.7 (2.6)

0.0 (3.1)

2.7 (4.9)

1.9 (3.4)

18.4 (11.4)

0.54 (0.04)*

-0.04(0.07)

4.7 (2.6)

0.0 (3.1)

2.6 (5.0)

1.7 (3.4)

21.0 (12.7)

0.52 (0.04)*

-0.03(0.08)

-7.8 (6.3)

5.0 (9.6)

3.4 (3.0)

-0.7 (3.2)

5.2 (4.7)

-0.3 (3.4)

Croatia

-1.2 (3.6)

-0.7 (3.6)

3.5 (1.4)*

-1.2 (1.2)

-1.6 (1.1)

1.3 (1.2)

-1.5 (3.7)

0.77 (0.02)*

-0.01(0.02)

0.5 (0.9)

-0.8 (1.2)

-1.5 (1.0)

1.2 (1.2)

0.8 (4.2)

0.77 (0.02)*

-0.01(0.02)

-2.2 (2.4)

5.6 (3.7)

0.6 (0.9)

-0.8 (1.2)

-1.5 (1.0)

1.4 (1.1)

Czech

Republic

9.7 (4.8)*

9.0 (4.9)

0.6 (0.9)

-1.8 (1.1)

0.7 (1.4)

1.4 (1.5)

9.9 (4.8)*

0.88 (0.03)*

0.01 (0.03)

2.3 (1.4)

-1.5 (1.2)

0.6 (1.4)

1.4 (1.5)

8.7 (4.9)

0.88 (0.03)*

0.01 (0.03)

-6.7 (3.8)

1.3 (5.4)

1.9 (1.4)

-1.7 (1.2)

0.3 (1.4)

1.5 (1.3)

Finland

-4.4 (3.9)

-4.4 (3.7)

2.2 (1.5)

5.0(1.0)*

-0.5 (1.1)

0.7 (1.3)

-4.7 (3.5)

0.73 (0.02)*

0.00 (0.03)

1.0 (1.4)

5.2(1.0)*

-0.2 (1.1)

0.2 (1.4)

-1.4 (3.8)

0.73 (0.02)*

0.00 (0.03)

-7.7 (2.2)*

10.0 (3.6)*

0.9 (1.3)

4.2(1.1)*

0.0 (1.0)

-0.6 (1.4)

Georgia

-2.4 (9.0)

-4.2 (9.1)

1.2 (1.4)

1.8 (2.8)

-0.9 (2.7)

-0.7 (2.2)

-4.5 (9.1)

0.82 (0.02)*

-0.03(0.02)

5.1 (2.2)*

2.0 (2.8)

-0.8 (2.6)

-0.5 (2.2)

-9.7 (9.2)

0.82 (0.02)*

-0.02(0.02)

2.2 (4.2)

-7.6 (5.7)

4.9 (2.2)*

2.0 (2.9)

-0.7 (2.6)

-0.4 (2.1)

Germany

-5.9 (4.7)

-8.8 (3.4)*

5.1 (2.2)*

2.3 (1.4)

-0.9 (1.4)

-4.3(1.3)*

-8.4 (3.3)*

0.78 (0.04)*

0.03 (0.03)

10.4(1.8)*

2.6 (1.3)

-0.8 (1.4)

-4.6(1.3)*

-7.2 (3.4)*

0.78 (0.04)*

0.03 (0.03)

-4.4 (3.3)

1.0 (4.5)

9.8 (1.8)*

3.0(1.3)*

-1.1 (1.4)

-4.8(1.3)*

Hong Kong

11.4 (6.1)

10.3 (6.0)

10.7(1.8)*

1.4 (1.9)

-3.3(1.4)*

-1.1 (1.6)

9.8 (6.0)

0.72 (0.03)*

0.01 (0.04)

-0.1 (1.6)

1.5 (2.0)

-3.5(1.4)*

-1.1 (1.6)

7.5 (6.1)

0.72 (0.03)*

0.01 (0.04)

-3.5 (3.7)

-6.8 (4.1)

-0.4 (1.5)

0.7 (2.0)

-3.8(1.3)*

-1.6 (1.6)

Hungary

-2.8 (9.8)

-8.7 (8.4)

0.0 (1.6)

4.6 (2.4)

-4.7 (1.8)*

-2.1 (1.9)

-10.1 (8.5)

0.82 (0.02)*

0.02 (0.03)

11.6(2.0)*

5.0(2.4)*

-5.0 (1.9)*

-1.9 (1.9)

-10.8 (8.4)

0.82 (0.02)*

0.02 (0.03)

-0.5 (5.9)

-5.1 (7.2)

11.3(2.0)*

4.1 (2.4)

-5.0 (1.9)*

-2.7 (1.9)

Iran

-11.9 (13.5)

-13.0 (14.3)

11.9(2.0)*

4.1 (2.4)

-2.5 (2.8)

3.9 (2.5)

-13.4 (14.3)

0.87 (0.02)*

-0.02(0.04)

7.4 (2.5)*

3.9 (2.4)

-2.5 (2.8)

3.9 (2.5)

-17.4 (13.8)

0.87 (0.02)*

-0.02(0.04)

-8.9 (5.7)

-15.2(11.9)

7.1 (2.5)*

1.9 (2.5)

-4.4 (3.0)

4.6 (2.4)

Note: EAS = Emphasis on Academic Success; SOS = Safety and Orderliness of School; CTS = Confidence in Teaching Science; IES =

Instructional Engagement of Students.

(): Standard errors appear in parentheses. *p< .05

102

Lee

Table 2 (cont.): Coefficients for multiple regression analysis

Country

Model

Worksheet

as a Basis

(WB)

Reading

Ach. (RA)

WB*RA

Lack of

Readiness

(LR)

WB*LR

Controlled Variable

EAS

SOS

CTS

IES

Ireland

-10.4 (9.8)

-4.8 (9.5)

7.3 (2.5)*

5.9(1.8)*

3.4(1.6)*

-4.7(1.7)*

-4.3 (9.8)

0.78 (0.02)*

0.04 (0.04)

5.9 (1.5)*

6.1(1.7)*

3.1 (1.6)

-4.9(1.6)*

-2.6 (9.6)

0.78 (0.02)*

0.04 (0.04)

-4.7 (2.5)

6.8 (8.0)

5.3 (1.5)*

5.5(1.7)*

2.7 (1.6)

-4.7(1.7)*

Italy

-21.8(7.5)*

-18.9(8.0)*

5.7 (1.5)*

3.0 (2.1)

-0.2 (1.9)

-1.8(1.4)

-19.0(7.9)*

0.78 (0.02)*

-0.02(0.04)

0.0 (1.7)

3.0 (2.1)

-0.3 (1.9)

-1.5 (1.4)

-11.7 (7.8)

0.78 (0.02)*

-0.01(0.04)

10.4 (4.2)*

-19.8(6.5)*

1.1 (1.7)

3.7 (2.1)

-0.7 (1.9)

-2.8(1.3)*

Lithuania

0.2 (5.9)

-2.7 (5.8)

0.2 (1.7)

1.2 (1.6)

-1.3 (1.7)

0.1 (1.6)

-3.1 (5.7)

0.75 (0.03)*

0.01 (0.04)

5.4 (2.0)*

1.1 (1.6)

-1.5 (1.7)

0.2 (1.6)

-4.0 (6.1)

0.75 (0.03)*

0.01 (0.04)

-7.1 (5.2)

1.2 (5.5)

4.7 (2.0)*

1.1 (1.6)

-1.9 (1.7)

-0.1 (1.6)

Malta

-6.5 (2.6)*

3.6 (2.6)

5.8 (1.9)*

4.4(0.7)*

2.8(0.6)*

-6.1(0.8)*

4.2 (1.6)*

0.74 (0.02)*

0.00 (0.04)

5.1 (0.5)*

4.2(0.5)*

2.8(0.4)*

-6.1(0.5)*

5.1 (1.5)*

0.74 (0.02)*

0.00 (0.04)

0.5 (0.9)

-3.9 (1.5)*

5.1 (0.5)*

4.1(0.6)*

2.9(0.4)*

-6.0(0.5)*

Morocco

-6.5 (12.4)

-9.5 (12.8)

4.8 (0.6)*

6.3(3.0)*

1.2 (4.1)

3.5 (3.1)

-7.4 (12.8)

0.66 (0.05)*

0.01 (0.06)

6.7 (4.3)

6.0(3.0)*

1.6 (4.1)

3.4 (3.0)

-39.7(21.5)

0.66 (0.05)*

0.01 (0.06)

-51.2(23.1)*

57.2(25.1)*

7.7 (4.3)

5.7(2.8)*

1.0 (4.1)

1.3 (2.9)

Northern

Ireland

-17.3(7.5)*

-18.6(6.9)*

7.1 (4.3)

5.8(1.8)*

-0.1 (1.5)

-1.7 (1.8)

-19.4(6.9)*

0.69 (0.01)*

-0.04(0.06)

3.3 (1.7)*

6.0(1.8)*

-0.1 (1.5)

-2.0 (1.8)

-22.5(6.9)*

0.70 (0.01)*

-0.04(0.05)

-6.8 (2.9)*

-10.7 (5.8)

2.2 (1.6)

4.7(1.7)*

0.2 (1.4)

-1.9 (1.9)

Norway

0.8 (4.6)

1.0 (4.0)

3.2 (1.7)

3.0 (1.6)

-2.5 (1.4)

1.4 (1.5)

0.7 (4.1)

0.78 (0.03)*

0.02 (0.04)

5.2 (1.5)*

3.1(1.6)*

-2.4 (1.3)

1.0 (1.4)

4.7 (4.8)

0.78 (0.03)*

0.02 (0.04)

-9.8 (2.3)*

11.2 (4.5)*

4.8 (1.4)*

3.2(1.5)*

-2.8 (1.4)

0.6 (1.4)

Oman

-14.1 (8.1)

-12.0 (7.1)

5.1 (1.5)*

5.8(2.8)*

-3.6 (2.7)

2.2 (2.5)

-11.4 (7.2)

0.92 (0.02)*

0.01 (0.02)

7.2 (2.3)*

5.5(2.8)*

-3.6 (2.7)

2.3 (2.5)

-14.4(7.1)*

0.92 (0.02)*

0.01 (0.02)

-2.3 (7.1)

3.6 (8.9)

6.8 (2.4)*

5.0 (2.9)

-2.5 (2.7)

2.3 (2.6)

Poland

-1.8 (4.1)

-4.4 (3.9)

7.2 (2.4)*

3.6(1.6)*

1.7 (1.5)

-1.3 (1.1)

-5.3 (3.9)

0.84 (0.02)*

0.01 (0.03)

3.6 (1.5)*

-3.2(1.6)*

1.6 (1.5)

-1.4 (1.1)

-5.3 (3.9)

0.83 (0.02)*

0.01 (0.03)

-1.8 (2.7)

-2.6 (4.2)

3.6 (1.5)*

-4.0(1.5)*

1.6 (1.5)

-1.5 (1.2)

Portugal

8.4 (7.0)

8.3 (7.8)

3.8 (1.4)*

0.6 (2.7)

-0.8 (2.1)

-1.6 (1.5)

7.5 (7.9)

0.77 (0.02)*

-0.05(0.03)

9.6 (2.2)*

0.9 (2.7)

-0.7 (2.1)

-1.5 (1.5)

7.6 (7.5)

0.77 (0.02)*

-0.05(0.03)

-0.4 (6.2)

1.0 (7.2)

9.6 (2.5)*

0.9 (2.7)

-0.8 (1.9)

-1.5 (1.5)

Qatar

-62.7(13.4)*

-61.4(14.0)*

9.4 (2.2)*

4.7 (3.1)

0.5 (4.2)

-3.4 (4.0)

-61.4(14.0)*

0.83 (0.03)*

0.05 (0.04)

5.5 (4.4)

4.9 (3.1)

0.8 (4.2)

-3.3 (4.0)

-64.3(13.2)*

0.83 (0.03)*

0.05 (0.04)

-38.1(8.0)*

26.2(9.8)*

3.3 (4.1)

3.7 (3.5)

0.1 (3.6)

-4.0 (3.7)

Note: EAS = Emphasis on Academic Success; SOS = Safety and Orderliness of School; CTS = Confidence in Teaching Science; IES =

Instructional Engagement of Students.

(): Standard errors appear in parentheses. *p< .05

103

IJEMST (International Journal of Education in Mathematics, Science and Technology)

Table 2 (cont.): Coefficients for multiple regression analysis

Country

Model

Worksheet

as a Basis

(WB)

Reading

Ach. (RA)

WB*RA

Lack of

Readiness

(LR)

WB*LR

Controlled Variable

EAS

SOS

CTS

IES

Romania

-2.0 (12.5)

-0.8 (12.5)

5.9 (4.3)

-2.6 (3.9)

-11.4(5.0)*

-0.8 (2.8)

-2.0 (12.5)

0.85 (0.03)*

-0.01(0.03)

10.2(3.1)*

-2.8 (3.9)

-11.8(5.0)*

-0.5 (2.9)

-3.7 (11.6)

0.85 (0.03)*

0.00 (0.03)

-25.4(8.5)*

5.9 (9.9)

7.7 (2.9)*

-5.3 (4.0)

-9.0 (4.8)

-0.8 (3.0)

Russian

Federation

5.7 (5.7)

5.1 (5.6)

9.8 (3.1)*

-0.3 (2.3)

0.0 (3.3)

2.3 (1.9)

4.5 (5.6)

0.71 (0.03)*

-0.01(0.03)

3.0 (2.0)

-0.3 (2.4)

-0.2 (3.3)

2.3 (1.9)

6.3 (6.7)

0.71 (0.03)*

-0.01(0.03)

-2.5 (4.5)

4.6 (6.1)

2.5 (2.3)

-0.3 (2.5)

-0.1 (3.3)

2.4 (1.9)

Saudi

Arabia

37.9(12.1)*

23.8(12.1)*

2.8 (2.0)

-1.6 (3.7)

4.1 (4.2)

0.5 (2.7)

23.2 (12.0)

0.82 (0.02)*

0.00 (0.04)

7.6 (3.0)*

-1.9 (3.6)

4.5 (4.3)

0.8 (2.7)

32.9(11.6)*

0.82 (0.02)*

-0.01(0.04)

8.0 (8.2)

-14.4(10.4)

7.3 (3.3)*

-2.8 (3.6)

3.5 (4.4)

1.3 (2.7)

Singapore

-14.8 (7.3)*

-12.1 (7.0)

8.1 (3.0)*

3.6 (2.4)

-2.2 (1.8)

-2.1 (1.5)

-13.1 (7.0)

0.81 (0.03)*

-0.01(0.04)

6.7 (2.4)*

3.3 (2.4)

-2.2 (1.8)

-1.9 (1.5)

-11.3 (6.3)

0.80 (0.03)*

0.00 (0.03)

-26.7(7.1)*

0.7 (7.8)

4.8 (2.2)*

1.0 (2.2)

-3.1 (1.7)

-3.4(1.5)*

Slovak

Republic

4.8 (5.8)

0.2 (5.8)

6.6 (2.4)*

-0.4 (2.8)

-1.3 (1.6)

-2.3 (1.9)

1.0 (5.8)

0.85 (0.02)*

-0.04(0.03)

9.8 (3.0)*

-0.4 (2.8)

-1.1 (1.6)

-2.4 (1.9)

-2.4 (8.1)

0.85 (0.02)*

-0.04(0.03)

-7.6 (3.4)*

-1.0 (5.7)

7.8 (3.1)*

-0.5 (3.0)

-1.0 (1.6)

-2.9 (2.0)

Slovenia

-7.3 (4.7)

-7.5 (4.9)

9.8 (3.0)*

1.2 (1.3)

0.1 (1.0)

-2.1 (1.3)

-7.5 (4.8)

0.83 (0.02)*

0.03 (0.03)

3.2 (1.4)*

1.1 (1.3)

0.2 (1.0)

-2.0 (1.2)

-8.2 (4.7)

0.83 (0.02)*

0.03 (0.03)

-12.3(4.3)*

7.6 (4.8)

2.1 (1.5)

0.7 (1.2)

0.4 (1.0)

-2.2 (1.2)

Spain

9.3 (6.5)

4.8 (6.6)

3.1 (1.4)*

4.6(1.7)*

1.0 (1.8)

-1.0 (1.6)

5.5 (6.5)

0.75 (0.03)*

0.04 (0.04)

3.0 (1.8)

5.0(1.6)*

0.9 (1.8)

-1.2 (1.6)

5.3 (7.1)

0.75 (0.02)*

0.04 (0.04)

-6.4 (3.3)

2.4 (5.1)

2.6 (1.8)

4.3(1.6)*

1.2 (1.9)

-1.4 (1.7)

Sweden

-5.4 (7.9)

-11.6 (7.3)

3.0 (1.8)

7.6(1.6)*

-3.0 (1.4)*

2.8 (1.4)*

-11.8 (7.3)

0.84 (0.03)*

0.05 (0.04)

3.7 (1.6)*

7.3(1.6)*

-3.1 (1.5)*

2.8 (1.4)*

-20.6 (10.7)

0.84 (0.03)*

0.05 (0.04)

2.5 (2.7)

-7.7 (8.2)

4.3 (1.5)*

7.1(1.5)*

-2.6 (1.4)

3.1 (1.4)*

Taiwan

1.4 (4.3)

-0.5 (4.2)

3.1 (1.5)*

-0.2 (1.4)

1.1 (1.4)

-0.2 (1.1)

-0.6 (4.2)

0.82 (0.02)*

-0.01(0.02)

3.7 (1.4)*

-0.2 (1.4)

1.1 (1.4)

-0.3 (1.1)

-1.0 (4.4)

0.82 (0.02)*

-0.01(0.02)

-4.2 (3.3)

2.9 (3.9)

3.3 (1.5)*

-0.3 (1.4)

1.1 (1.4)

-0.4 (1.1)

UAE

-36.4 (7.7)*

-34.1 (7.6)*

7.7 (2.0)*

2.2 (2.4)

2.1 (3.1)

-0.5 (1.7)

-33.4 (7.6)*

0.84 (0.02)*

0.02 (0.02)

7.8 (2.0)*

2.1 (2.4)

2.4 (3.1)

-0.7 (1.7)

-33.2 (7.3)*

0.84 (0.02)*

0.02 (0.02)

-30.7(4.9)*

22.7(7.0)*

5.3 (1.9)*

0.4 (2.2)

1.8 (3.0)

-1.8 (1.7)

Note: EAS = Emphasis on Academic Success; SOS = Safety and Orderliness of School; CTS = Confidence in Teaching Science; IES =

Instructional Engagement of Students; UAE = United Arab Emirates.

(): Standard errors appear in parentheses. *p< .05

Discussion and Conclusion

Instructional written materials play important roles as teachers’ agents in effective teaching practices.

Workbooks and worksheets are one of the most frequently used materials (Table 1). Based on the result of this

study, the association of worksheet usage and science achievement is found to be quite different across

104

Lee

countries. To sum up the result, there are five types of relationships among science achievement, worksheet

usage, and other related variables (Table 3).

 Type 1: The association between WB and SA remains the same regardless of whether or not teacher and

school variables are controlled, and no interaction is present between WB and LR in SA.

 Type 2: The association between WB and SA depends on whether or not teacher and school variables are

controlled, and no interaction is present between WB and LR in SA.

 Type 3: The association between WB and SA remains the same regardless of whether or not teacher and

school variables are controlled, and a positive interaction is present between WB and LR in SA.

 Type 4: The association between WB and SA remains the same regardless of whether or not teacher and

school variables are controlled and a negative interaction is present between WB and LR in SA.

 Type 5: The association between WB and SA depends on whether or not teacher and school variables are

controlled, and a negative interaction is present between WB and LR in SA.

In addition, there is no significant interaction between WB and RA in all participating countries.

Table 3: Relationships among worksheet usage, science achievement, and other variables

Type

Association btw WB & SA

after controlling variables

WB*LR

Country

Teacher and

school variables

Austria, Azerbaijan, Croatia, Czech Republic

Georgia, Hong Kong, Hungary, Iran, Ireland,

Lithuania, Oman, Poland, Portugal, Romania,

Russian Federation, Slovak Republic, Slovenia,

Spain, Sweden, Taiwan

Northern Ireland

Saudi Arabia

Singapore

Germany

Australia, Finland, Morocco, Norway

Qatar, United Arab Emirates

Italy

Malta

Note: ~S = Not significantly different from zero; P = Significantly positive; N = Significantly negative.

: Although the association between WB and SA is significantly different from zero, none of the teacher and school variables

are significantly related to SA.

Based upon the above findings, there are four directions of further investigation to identify important features of

designing and applying worksheets through comparisons across countries in future studies.

Firstly, the international comparison can be made among three groups of countries to identify the related factors

in predicting science achievement. In most countries, there is no association between WB and SA, including

countries of type 1a and 3a. Only in Saudi Arabia is the association positively different from zero. In the four

countries of type 1b, 3b, and 4, the association is negatively different from zero.

One explanation of the negative association between worksheet usage and science achievement is that teachers

tended to use worksheets in low-achievement classes, as Reid (1984) reported. If this is true, teachers’

perceptions of class achievement may be the cause of the negative association. However, after the introduction

of the variable of classes’ lack of readiness in Model 4, the association in the four countries, Northern Ireland,

Qatar, the United Arab Emirates, and Italy, remains the same (Table 2). Consequently, there are other factors

that have yet to be uncovered.

The second direction of future investigation is the relationship between worksheet usage and other teacher and

school factors. After controlling the teacher and school variables, EAS, SOS, CTS, and IES, for type 2 and 5

countries (Singapore, Germany, and Malta), the association between WB and SA changed. For Singapore, the

variable EAS is significantly related to SA in Model 2. For Germany, the significantly relative variables are

EAS and IES. For Malta, the relative variables are EAS, SOS, CTS, and IES (Table 2). These results imply that

these teacher and school variables, worksheet usage, and science achievement are correlated in these countries.

105

IJEMST (International Journal of Education in Mathematics, Science and Technology)

It is worth inquiring why these teacher and school variables and worksheet usage are correlated and how they

together influence students’ science achievement.

Thirdly, the mechanisms that make worksheets more effective for students in classes lacking readiness than

those in classes not lacking readiness are worth looking at further. It would be easier to find out the mechanisms

through the use of data from countries of type 3 (Australia, Finland, Morocco, Norway, Qatar, and the United

Arab Emirates), which have positive interactions between WB and LR in Model 4 (Table 2). Data from Italy

and Malta could also be used as contrast. To find the mechanisms, more data about worksheet design and about

teaching and learning with worksheets should be collected. For example, strategies that students use to complete

worksheets are important factors related to their achievement but are not well documented.

The last direction of further inquiry is identifying the factors that result in no interaction between worksheet

usage and reading achievement in science achievement. The result of no interaction between WB and RA may

be caused by appropriate matching of language levels of worksheets with students’ reading abilities, but it might

also be caused by teachers’ explanations before students starting to work on worksheets. The data collected by

TIMSS and PIRLS 2011 cannot help us to identify the cause. To resolve this problem, data about evaluation on

language demand of worksheets and the teaching methods accompanying worksheets need to be collected.

Acknowledgements or Notes

The author wish to thank the project (NSC 101-2511-S-003-021) funded by National Science Council of

Taiwan.

References

Anderson, R. C., Hiebert, E. H., Scott, J. A., & Wilkinson, I. A. G. (1985). Becoming a nation of readers: The

report of the Commission on Reading. Washington, DC: National Academy of Education, National Institute

of Education, Center for the Study of Reading.

Calderhead, W. J., Filter, K. J., & Albin, R. W. (2006). An investigation of incremental effects of interspersing

math items on task-related behavior. Journal of Behavioral Education, 15(1), 51–65.

Campbell, C. P. (1999). Instructional materials: Their preparation and evaluation. Journal of European

Industrial Training, 23(2), 55–107.

Department of Education and Science (2007). Inclusion of students with special educational needs: Post

primary guidelines. Dublin: Stationery Office.

Foy, P. (2013). TIMSS and PIRLS 2011 user guide for the fourth grade combined international database.

Chestnut Hill, MA: TIMSS & PIRLS International Study Center, Lynch School of Education, Boston

College and IEA.

Hayes, D. P., Wolfer, L. T., & Wolfe, M. F. (1996). Schoolbook simplification and its relation to the decline in

SAT-Verbal scores. American Educational Research Journal, 33(2), 489–508.

Hoener, A., Salend, S., & Kay, S. I. (1997). Creating readable handouts, worksheets, overheads, tests, review

materials, study guides, and homework assignments through effective typographic design. Teaching

Exceptional Children, 29(3), 32–35.

Hoffman, D. A., & Gavin, M. B. (1998). Centering decisions in hierarchical linear models: Implications for

research in organizations. Journal of Management, 23, 723–744.

IEA Data Processing and Research Center (2013). IDA analyzer. Amsterdam, Netherlands: IEA. Retrieved from

http://www.iea.nl/eula.html.

Kisiel, J. F. (2003). Teachers, museums and worksheets: A closer look at a learning experience. Journal of

Science Teacher Education, 14(1), 3–21.

Krombab, A., & Harms, U. (2008). Acquiring knowledge about biodiversity in a museum - Are worksheets

effective? Journal of Biological Education, 42(4), 157–163.

Lesley, M., & Labbo, L. D. (2003). A pedagogy of control: Worksheets and the special need child. Language

Arts, 80(6), 444.

Martin, M. O., & Mullis, I. V. S. (Eds.) (2013). TIMSS and PIRLS 2011: Relationships among reading,

mathematics, and science achievement at the fourth grade - Implications for early learning. Chestnut Hill,

MA: TIMSS & PIRLS International Study Center, Lynch School of Education, Boston College and IEA.

Martin, M. O., Mullis, I. V. S., Foy, P., & Stanco, G. M. (2012). TIMSS 2011 international results in science.

Chestnut Hill, M.A.: TIMSS & PIRLS International Study Center.

McDowell, E. T., & Waddling, R. E. L. (1985). Improving the design of laboratory worksheets. Journal of

106

Lee

Chemical Education, 62(11), 1037–1038.

Meyer, B. J. F. (2003). Text coherence and readability. Topics in Language Disorders, 23(3), 204-224.

O'Leary, S. (2011). The inclusive classroom: Effect of a readability intervention on student engagement and on-

task behaviour within two mixed-ability science classrooms. Science Education International, 22(2), 145–

151.

Reid, D. (1984). Readability and science worksheets in secondary schools. Research in Science and

Technological Education, 2(2), 153–165.

Rix, J. (2006). Simplified language materials: Their usage and value to teachers and support staff in mainstream

settings. Teaching and Teacher Education, 22(8), 1145–1156.

Rotter, K. (2006). Creating instructional materials for all pupils: Try COLA. Intervention in School and Clinic,

41(5), 273–282.

Sasmaz-Oren, F., & Ormanci, U. (2012). An application about pre-service teachers' development and use of

worksheets and an evaluation of their opinions about the application. Educational Sciences: Theory and

Practice, 12(1), 263–270.

Ueckert, C. W., & Gess-Newsome, J. (2008). Active learning strategies. Science Teacher, 75(9), 47–52.

Wolf, J., Stanton, M., & Gellott, L. (2010). Critical thinking in physical geography: Linking concepts of content

and applicability. Journal of Geography, 109(2), 43–53.