The editors of Research in the Teaching of English offer these guidelines to assist reviewers in evaluating manuscripts. Thank you to David Slomp (author) and Norbert Elliot (reviewer), who kindly agreed to allow their manuscript and comments to be used in the Example Review sections.STEP TWO
View a PDF of the Guidelines.
Briefly (in 2-3 sentences) describe your understanding of the author's/authors' project.
Does the project “fit” RTE
? State whether and why you feel the project is appropriate for RTE,
in terms of both quality and fit. To support your claims about the manuscript’s appropriateness for RTE,
please provide detailed feedback on the following:Significance to the Field
Does the manuscript identify a problem and is it a significant one for the field of language and literacy research? What ongoing conversations is it entering into? Is the problem identified, or the approach to the problem that is taken, fresh and timely? Do the findings or conclusions deliver new insights in relation to that problem?
Example ReviewMethodological Soundness
Problem Identification: Identification of diversity, impact of testing, and alignment of both—these are significant issues of enduring educational importance. The case at hand, Canada, presents a valuable sampling plan.
Original Approach: Attention to consequential validity is a good framework for the study, important because of frequent cosmetic attention to its presence in a testing situation.
Relationship to (and Extension of) Current Research and Theory: Attention to Messick and Kane are important in such an analysis, and the bases are covered. Yet, this research is 25 years old, and more theorization is needed for the paper to succeed.
Fresh Insight: As I will suggest below, a revision will provide needed insight for RTE readers.
Does the manuscript employ a methodology consistent with the theoretical orientation that informs the investigation and the goals of the manuscript? Does the manuscript clearly describe the research design and relevant methods used so knowledge gained on language and literacy research may be increased? To what extent is the methodological design aligned with the research questions? Does the methodology hold to the highest standards set by other relevant studies?
Congruence between Method and Theoretical Orientation: There is a basic congruence here that is good, but additional categories of validity conceptualization are needed if the theory and method are to match, resulting in fresh insight.
Information Analyzed with Concepts Based on Theory: In a recent presentation of his conceptual framework for validation, Kane (2013) again presented his 2006 validation categories: scoring, generalization, extrapolation, and implication. Little attention is paid to theory-based interpretation in the new paper, and implication remains a black box. Certainly, the authors could begin by reviewing that most recent work and calling for an expansion of categories significant to stakeholders beyond measurement specialists.
Analysis & Interpretation
Are the claims insightful? How well are the claims grounded in the evidence? How well is the evidence synthesized into the discussion?
Example ReviewScholarly Quality
Conclusions Grounded in Analysis: The authors write, "Our findings support Brennan's claims: while government documents made strong assertions about the value of standardized testing to monitor, maintain, or improve systems of education, the peer-reviewed research that collected directed evidence from within schools painted a much more negative picture of the effects of these assessment programs on literacy education and diversity in Canadian classrooms" (33). That may be indeed so. But whether this finding is dependent on consequential validity of other categories of validity evidence is unknown.
Now, to the methodological questions of analysis and interpretation as they related to the notes above: As a key research question, the authors ask, "What data or arguments are being used to justify/provide indicators of the impact of literacy assessments on teaching, learning, and achievement." Strictly speaking, this is not a consequential validity question but a validity argument question. And, the second question on page 11 is one of use. Again, this is why the categories of validity evidence need to be presented early on in the paper. Were these categories expanded, Tables 1 and 2 could be vastly improved to include categories of validity evidence identified in the document search.
To what extent does the article demonstrate strong scholarly grounding? How well is the theory that grounds the article extended or reconceived as a consequence of the analysis?
Example ReviewQuality of Writing
Strong Scholarship: The scholarship here is good but not sufficient for the analysis. I have suggested additional paths of inquiry in the references.
Theory Extended or Reconceived: As I have noted, a more exact presentation of consequential validity is needed.
To what extent is the writing clear, fluent, and engaging?
Example ReviewSTEP THREE
Clear, Fluent, and Engaging Presentation: The authors are very good on the page.
Offer the authors any advice you may have at this point about how to revise and develop the project. Note its strengths and points of interest; note its weaknesses both major and minor.
Example ReviewSTEP FOUR
Concluding Notes: The areas of revision I have suggested—addition of validity evidence categories, expansion of the definition of consequential validity based on shareholder theory, and design of Toulmin models to reveal document claims—are complex. I realize that. But you do have two sets of documents, and with more effort, something here can be produced of great value to RTE readers. My notes have been offered to improve the research, not to offer foolish barriers impossible to meet.
A final note: Consider abandoning the value dualism of standardization versus construct representation. Realistically, if test could be developed allowing fuller representation of the writing construct, is there evidence that new test would meet the other categories of validity evidence? If you can abandon value dualism and value hierarchy as a thought experiment, I think you would be able to re-design the tables and create the Toumin models more accurately. Then, once completed, I'll wager the final pages of the manuscript would look much different—and readers would be grateful for that indeed.
You are onto something important here. Go deeper. See Additional References, such as:
Fabian, S. B. (2009, Spring). Diverse Test Takers: Essential considerations for test developers and decision makers, Educational Measurement: Issues and Practice, 10-24.
Evaluate your comments and recommend a decision. To submit your decision, please log in to Editorial Manager
to upload your evaluation and comments and finalize your decision. We kindly ask that you refrain from uploading your review as an attachment. Instead, please copy and paste your comments into the review box.
This eases the review process for editors. If you have any difficulty, please email RTE
Assistant Editors at firstname.lastname@example.org
, as they will be able to assist in pasting your review. Recommendations to RTERejectReject and Resubmit as New Manuscript
This means you’re recommending the author completely re-write the manuscript and, when resubmitted, the editorial team will send it out to a completely new set of reviewers.Revise and Resubmit
This means you’re recommending the manuscript undergo major revisions and then be sent out to the same set of reviewers, if possible.Conditional Accept with Specified Revisions
This means you’re recommending a more minor set of revisions that might warrant one of the reviewers taking another look at the manuscript before it is officially “accepted.” Accept with Minor Editing
You have successfully reviewed for RTE. On behalf of the editorial team, thank you for your time and contribution.