Exploring evidence of higher order thinking skills in the writing of first year undergraduates

Research indicates that concern is often expressed about the language and discourse skills new students bring with them when they first enrol at university, which leads to assumptions being made about their academic abilities. In this paper, an argument is developed through detailed analysis of student writing, that many new first year students have nascent Higher Order Thinking Skills and the potential to be successful in their studies. The work of Robert Marzano and his associates (Marzano, 2001; Marzano & Kendall, 2007, 2008) is applied to student writing. Please cite this article as: Faragher, L. & Huijser, H. (2014). Exploring evidence of higher order thinking skills in the writing of first year undergraduates. The International Journal of the First Year in Higher Education, 5(2), 33-44. doi: 10.5204/intjfyhe.v5i2.230 This article has been peer reviewed and accepted for publication in Int J FYHE. Please see the Editorial Policies under the ‘About’ section of the Journal website for further information. © Copyright of articles is retained by author/s. As an open access journal, articles are free to use, with proper attribution, in educational and other non-commercial settings. ISSN: 1838-2959 Exploring evidence of higher order thinking skills in the writing of first year undergraduates 34 | The International Journal of the First Year in Higher Education, 5(2) August, 2014 Introduction The quality of writing in English by first year university students does not necessarily reflect the quality of their thinking. This paper considers the implications of that realisation for teaching and learning in higher education, especially as part of the first year experience. It will be shown that Higher Order Thinking Skills (HOTS) can be identified in the writing of first year university students and that the ability to use HOTS is not necessarily linked to the ability to express themselves well in academic language and in their ability to use academic discourse when they first enter the academy.1 There is further elaboration of this in the discussion section of the paper. The assumption that faulty language reflects faulty thinking is widespread and can lead to prejudicial attitudes by markers of first year writing. The report on Good Practice Principles (GPP) stated that ...with widening participation across tertiary education and the increasing numbers of international students, it can no longer be assumed that students enter their university study with the level of academic language proficiency required to participate effectively in their studies. (Australian Universities Quality Agency [AUQA], 2009, p. 2) 1 For the purpose of this paper the term academic discourse will be taken to mean “the language of a particular community...useful for specific purposes” (Bizzell, 1992, p. 20). In this case the community is broadly that of the university and the discourse appropriately used is what identifies individuals as “insiders” to the academic community (Worthman, 2008). This shows the assumption that language deficiency inhibits academic performance. In their analysis of the GPP, Harper, Prentice, & Wilson (2011) comment that Principle 7 demonstrates a deficit view of student writing and uses a pathological analogy when they suggest that students’ linguistic proficiency should be diagnosed and remediated (Harper et al.; Murray, 2012). Murray points to a growing body of research that reports concern among academics that student levels of linguistic ability are too low to enable them to “access program content, interact effectively and reach their full potential” (p. 3). This all demonstrates a perception that apparent insufficient linguistic ability is linked with insufficient academic ability. In related literature there are further suggestions that students entering university without the necessary language skills will be unable to access the curriculum successfully due to backgrounds that have not provided them with the language and behaviour required by the academy for success. This lack of linguistic and discourse skills has caused many mainstream academics to assume that their linguistic and discourse disadvantage also indicates an inability to display the required capacity for success in terms of thinking skills. Subsumed into all of this is the concept that language proficiency and the use of appropriate discourse are vital to short term academic success (Elder, Bright, & Bennett, 2007; Oliver, Vanderford, & Grote, 2012). However, Graham (1987) has suggested that with regard to international students it was not possible, in the short term, to claim correlations between English proficiency and academic success, which provides another perspective on the issue explored in this study. Furthermore, Bradley, Noonan, Nugent, & Scales (2008, p. 42) note that additional


Introduction
The quality of writing in English by first year university students does not necessarily reflect the quality of their thinking.This paper considers the implications of that realisation for teaching and learning in higher education, especially as part of the first year experience.
It will be shown that Higher Order Thinking Skills (HOTS) can be identified in the writing of first year university students and that the ability to use HOTS is not necessarily linked to the ability to express themselves well in academic language and in their ability to use academic discourse when they first enter the academy. 1 There is further elaboration of this in the discussion section of the paper.The assumption that faulty language reflects faulty thinking is widespread and can lead to prejudicial attitudes by markers of first year writing.The report on Good Practice Principles (GPP) stated that …with widening participation across tertiary education and the increasing numbers of international students, it can no longer be assumed that students enter their university study with the level of academic language proficiency required to participate effectively in their studies.
(Australian Universities Quality Agency [AUQA], 2009, p. 2) 1 For the purpose of this paper the term academic discourse will be taken to mean "the language of a particular community...useful for specific purposes" (Bizzell, 1992, p. 20).In this case the community is broadly that of the university and the discourse appropriately used is what identifies individuals as "insiders" to the academic community (Worthman, 2008).
This shows the assumption that language deficiency inhibits academic performance.In their analysis of the GPP, Harper, Prentice, & Wilson (2011) comment that Principle 7 demonstrates a deficit view of student writing and uses a pathological analogy when they suggest that students' linguistic proficiency should be diagnosed and remediated (Harper et al.;Murray, 2012).Murray points to a growing body of research that reports concern among academics that student levels of linguistic ability are too low to enable them to "access program content, interact effectively and reach their full potential" (p. 3).This all demonstrates a perception that apparent insufficient linguistic ability is linked with insufficient academic ability.In related literature there are further suggestions that students entering university without the necessary language skills will be unable to access the curriculum successfully due to backgrounds that have not provided them with the language and behaviour required by the academy for success.This lack of linguistic and discourse skills has caused many mainstream academics to assume that their linguistic and discourse disadvantage also indicates an inability to display the required capacity for success in terms of thinking skills.Subsumed into all of this is the concept that language proficiency and the use of appropriate discourse are vital to short term academic success (Elder, Bright, & Bennett, 2007;Oliver, Vanderford, & Grote, 2012).However, Graham (1987) has suggested that with regard to international students it was not possible, in the short term, to claim correlations between English proficiency and academic success, which provides another perspective on the issue explored in this study.Furthermore, Bradley, Noonan, Nugent, & Scales (2008, p. 42) note that additional development services cost the university extra expenditure, while they also identify a need for language support.The obvious other element of the impact of new students apparently lacking the academic and language skills needed for success in first year relates to the students themselves.Many have been encouraged by successive governments' inclusion agendas to enter the university but find themselves marginalised upon entry because they do not have control of the "discourse" required to demonstrate their ability to satisfy the requirements of assessment and become engaged with the curriculum.
Engagement is high on the agenda of most universities.Wilson (2009) has shown its importance as a predictor of success in the first year.The Australasian Survey of Student Engagement (AUSSE) data published in 2007 show that Australia is behind both the US and UK in areas of engagement (Coates, 2008).Priest (2009) discusses the alienation experienced by low SES students who do not have mastery of the language and discourse of the academy.She draws on a combination of Bourdieu and empirical research that shows that such students can find the world of the academy exclusionary because they do not know the language used that constitutes the dominant discourse.Wilson also makes the point that members of minority or disadvantaged groups who are often associated with low language proficiency are also more likely to drop out.These findings are supported in The First Year Experience in Australian Universities: Findings from 1994-2009(James, Krause & Jennings, 2010). Fernsten (2005) and Bizzell (1992) both demonstrate the alienating effect that the discourse of the academy can have.The students see themselves as "outsiders" and therefore have low expectations with regards to their own success (Worthman, 2008).
If assessors were provided with tools that would enable them to interpret the actual thinking in the faulty prose, they would be able to see the potential and separate the faulty writing from the thinking and give it suitable credit.This would provide beginning students with a more positive experience and the much-needed confidence to begin developing their linguistic skills.Assessment is still mostly based on written texts produced by students.
If assessors are distracted by the surface errors and are not able to interpret the thinking behind them, those students are likely to be marked down, lose confidence and be more likely to give up on their studies.This has serious potential implications for universities' retention rates.The development of a set of transparent descriptors that show Higher Order Thinking Skills (HOTS) will also assist students' metacognition and develop their awareness of how they are supposed to think and express that thinking in words.It will give them time to develop their writing skills while not losing confidence in their ability to succeed.In this paper, there is an analysis of a sample of first year education students' writing that uses a set of descriptors designed to identify HOTS shown in their writing.The results suggest that there is evidence of HOTS in the writing and which skills are most strongly evidenced.The results are discussed and recommendations made.

Relationship between thinking and writing
The relationship between thought and language has been hypothesised through many generations of philosophers and psychologists and is at the heart of this research.However little seems to be available about the relationship between thought and writing.The quality of the thinking is what this research aims to discover and often the only evidence available is in the writing.The idea that the quality of the thinking might not be immediately obvious but that it is nevertheless active "internally" is reflected in O' Brien and Opie's (2002) work about connectionism and how internalised language plays a role in co-ordinating and controlling cognitive activity.They also comment on resonances within Vygotsky's concept of inner speech, which is where problem-solving takes place (Daniels, 2001).Vygotsky's other theories about the relationships between thought and language, especially about concept formation, are also part of the underpinning theory for this research (Vygotsky, 1962).Slezak ( 2002) argues that we do not think in language and that thinking can happen independently of language.This has relevance for this study because the writing that will be analysed is the product of students' thinking about, and responding to, their assignment questions.If the markers are to give them credit for having tried to respond to the task, they are going to need to interpret the writing and evaluate the thinking behind it.Slezak concludes that we "should not be attributing properties of the world … to our internal, mental states … which might appear to be a silent soliloquy" (p.371).Thus, we cannot know what the students have actually thought but we do have evidence in the form of their writing which we must evaluate.Ultimately, there might be other ways in which students can provide evidence of their HOTS and debates about "authentic assessment" (Boud, 2010;Boud & Falchikov, 2006;Maclellan, 2004;Reeves, Herrington & Oliver, 2002) are interesting in that respect in that they suggest the use of other media, such as video or oral presentations, to communicate HOTS.While it is beyond the scope of this paper to discuss this in depth, it is important to consider in the context of our discussion here.
If the writing of first year students in Australian universities was subjected to a different lens, namely that of the quality of their thinking, it might be possible to develop curriculum and ways of assessing their thinking that would divert attention away from faulty expression and discourse without compromising pass criteria or rigorous grading.None of this is to deny the importance of academic writing, but it is rather to suggest that developing academic writing skills is a process over time, rather than a skill that can be expected to be fully formed at entry level.Moreover, de-coupling HOTS from academic writing skills potentially creates opportunities to develop such skills in such a way that it recognises and rewards already existing thinking skills, thereby building confidence in the process rather than alienation.Focussing on reasons for plagiarism Bretag's (2007) research indicates that academics do compromise "standards" where the writing is poor for a range of reasons.It is possible that if they had been able to evaluate the writing in terms of an analysis of thinking skills, their attitude might have been more positive and also the students might have benefited educationally.It might be possible to enable students to develop sensitivity and skill in academic writing and discourse by raising their metacognition and providing explicit teaching as to what is required of their writing at university level.
In a previous study conducted in South Africa the writing of African2 matriculants was analysed and it was argued, in terms of Vygotskyan thinking, that there was sufficient evidence in their writing for them to be described as self-regulated thinkers (Faragher, 1995).This description of the writers as self-regulated was based on the Vgotskyan definition of regulation (Kozulin, 1990;Vygotsky, 1962;Wertsch, 1985), and was argued on the basis that they had passed the writing examination and had been shown to have made rich and potent meaning in their writing.
A different approach is taken by Robert Marzano in Designing a New Taxonomy of Educational Objectives (Marzano, 2001).His research group worked with Bloom's Taxonomy of Learning Domains and developed it further.The hierarchy of Marzano's taxonomy works differently from Bloom's -starting from the top, the highest level, Level 6 -Self System Thinking, where students determine their motivation and self-efficacy, after which the metacognitive system comes into play to determine goals and processes, and finally the actual thinking systems where they start working on actual problems and issues.This is the most recent work on taxonomies of thinking skills, and it develops directly from Bloom and links to Bloom throughout.The original work was intended to be used by teachers to develop HOTS in their students.However here it has been adapted so it can be used to provide end point descriptors of the HOTS evident in the assessments.
Another quality of the Marzano Taxonomy is unusual, especially if Bloom's taxonomy is the usual standard: Marzano sees the process in terms of control rather than complexity and therefore the behaviour he places at the "top" is the Self-System, which is where the learner decides to engage with the task.Once that is achieved, the Metacognitive System sets goals and strategies and finally the Cognitive System processes the relevant information (Marzano 2001, p.11).In addition he considers how each processing level interacts with three domains of knowledge: Information, Mental procedures and Psychomotor procedures.As the students' texts are uni-dimensional, this aspect of his taxonomy is not included in this research.However one of the indicators of success identified by Wilson ( 2009) is that students should be "purposeful", that they should have a clear sense of why they are at university.This is what is indicated by strong "self-system" thinking.Using Marzano's taxonomy, it should be possible to classify what type of thinking students have done in the production of their assessments.

Method
In order to establish what thinking is being done in the virtual uni bag 3 that all students bring with them when they start at university, a pilot study was designed in which a sample of writing by first year Education students at the University of Southern Queensland was analysed.Mixed methods were used in the analysis (Dörnyei, 2007) and NVIVO was used to record and analyse the texts.The essays selected were analysed with a view to providing some preliminary idea of what thinking tools and strategies students have 3 A concept adapted from Pat Thompson's concept of the "virtual school bag", which suggests that new students bring with them a range of skills and abilities developed in their previous experience and education that could be utilised in their studies (Thomson & Hall, 2008).
brought with them in their virtual uni bags.Information about individual students was not factored into this particular study.The analytical categories are derived from Marzano's Taxonomy of Educational Objectives as evidenced in the students' writing and are used as the framework for analysis.The scripts were randomly selected on the basis of approximately 1:4 of a batch of 47, and they were written in response to an assessment item in a Literacy course in the Bachelor of Education programme at the University of Southern Queensland that required them to critically compare two examples of educational discourse with the help of a range of educational discourse analysis tools.They were expected to apply specific theory to their analysis and write in the genre of critical analysis.The scripts were graded for ability to apply theory to examples and make critical judgements as to what was most effective.
The scripts, referred to as "sources", were read and coded by one of the authors according to Marzano's descriptors.Instances of text conforming to the relevant descriptors were called "references", so there could be more than one reference in each source.Where a piece of text looked as if it fitted a descriptor but was not acceptable in terms of responding to the assessment task, it was not included.In addition, Marzano's Levels 1 and 2 (Retrieval and Comprehension) were not included as they refer to such basic processes that it was assumed that all students who qualify for university study would have them well under control.A Results section can be found on pages 42-44 and includes a table, Table 1, which summarises the results.The results of the research show that the majority of the students in the sample have some of the basic HOTS already in place but that very few reach the higher levels.A common weakness in the scripts was the students' tendency to describe and observe rather than analyse.Their observations were relevant to the task but to qualify for the descriptors used here they had to state problems and expand on their observations.To qualify for Self-System Thinking they had to demonstrate a sense of personal commitment and this only showed in the Empathy category which is explained further below.The analysis results were entered into NVIVO, which then performed the aggregations and the numbers were subsequently extrapolated and fed into Excel graphs.The table is a summary only, as showing the full Excel graphs is beyond the scope of this paper.

Discussion
The purpose of doing this research was to establish if first year university students do have HOTS in their virtual uni bag when they enter the academy and to discover what those thinking skills were and how they could be identified.It is intended that lecturers teaching first year students should become aware that non-standard, non-academic English writing skills do not necessarily indicate poor thinking and discourse skills.As mentioned earlier, a distinction is made between the terms language and discourse: language is the medium used to communicate the students' thoughts and ideas, which in turn constitute the discourse.Discourse is therefore not used here in the Foucauldian sense of the term.Rather, in this context, the use of the term is intended to cover issues like grammaticality, and it indicates a range of behaviours including correct academic language for the particular discipline, as well as issues such as: vocabulary, logical connectors, hedging, nominalisation, transition, framing and endophoric markers, evidentials and code glosses, abstractions and metaphoric grammar, long complicated noun phrases, words of Latin or Greek origin, passive voice and specific sets of words (Coxhead & Byrd, 2007).In addition Knoch (2008) adds certainty markers, attributors, attitude markers and commentaries.All of the above properties are not regularly and usually present in day to day speech and writing so they would be unlikely to be present in the writing of students who have not had the "training" required to alert them to the need for including them in their academic writing.Once they have been carefully inducted into the discourse and language of the academy, and their particular discipline, their ability to demonstrate their HOTS will be enhanced.Furthermore if assessors are alert to their attempts to use academic discourse and to other evidence of HOTS in their writing, which this research shows, they may be less likely to assume deficiency.
The results do show that in all the scripts there was some evidence of HOTS according to the Marzano descriptors.It was anticipated that the stronger numbers would show in the lower "levels" of the taxonomy, but this was not the case.The results do not show conformity with the idea of a hierarchical taxonomy.The stronger numbers of Sources are all concentrated in Knowledge Utilisation (20), although Matching (7) in Analysis shows equal frequency with Decision Making (7) in Knowledge Utilisation.However the number of References is different -Matching (13) and Decision Making (9).Yet Experimental Enquiry, which matches Matching at the top of the References (13) might have been expected to have fewer as it is included in the higher level of the hierarchy.
However, perhaps the somewhat generous interpretation of the descriptor might account for the numbers.In this case, wherever a student indicated the ability to hypothesise and suggest improvement or suggest reasons for a situation, that text was counted.
It is also possible that the nature of the assessment task would generate particular responses and require specific HOTS.Decision Making in Knowledge Utilisation is the next highest in the References which would be expected in an assignment requiring comparison, which in turn would support the theory that the assessment task will generate particular sorts of evidence of HOTS.That the next highest result is Empathy in Self System Thinking (in both Sources and References) is surprising until we consider the fact that these are teachers in training.It is to be expected that members of this profession would show empathy with teachers as well as learners.However, if that assumption is to be seen as appropriate then it is disappointing that only one of 12 showed thinking that took them "out of the box"!It was anticipated that the results would show that there is evidence of HOTS in the writing in this sample but not that it would vary so strongly from the hierarchy used in the framework.For a richer result, it would be necessary to include descriptors that would include attempts to use academic discourse as well as a word count and analysis of idiosyncratic expressions or Australian English (AusE).In a previous pilot study (Faragher, 2011), idiosyncratic expressions were used as well as academic discourse which gave further evidence of the manner in which writers were attempting to make meaning in ways acceptable to the academy.The word count has not been tested in this context but has generated a lot of research in other contexts (Coxhead, 2000;Simpson-Vlach & Ellis, 2010) and should be followed up.At this stage it is clear that the students in the above sample did bring some HOTS with them in their virtual uni bags and that it is possible to identify them using the descriptors mentioned above and that adding other descriptors will give a richer result.This is an important outcome and it has potential implications for Transition Pedagogy (Kift, Nelson & Clarke, 2010) and dealing with increasing diversity in incoming student cohorts.However, further research is required to see if Marzano's framework and set of descriptors is still applicable if the writing is much more problematic.Each transcript will be analysed and with assistance of theorists, essential parts will be highlighted to prove that the home provides a solid foundation, which is of a greater benefit to learning language and literacies.Furthermore, four key points will be explored in greater depth including: building on prior knowledge/experience at home is easier, home is self-paced, interest-based, school is generalised learning which makes it much harder to connect to individuals and then recommending what pedagogy is best implemented in the transcript.Metacognition/Process Monitoring -'tasks must be designed …that students can think about and monitor a skill or process while engaged in its execution…' *Specialised function that monitors "effectiveness of algorithm, tactic, or process as it is being used in a task" (p.49) *Applies to mental and physical procedural knowledge but not information.

4
4 This essay will compare the pedagogies applied in each transcript, discuss the influence that these, as well as any other factors, may have on the quality of each learning environment, and make recommendations for strategies which may be implemented to improve the quality of the learning environment.
Self The parent's prior knowledge of the child's past experiences guides and helps the parent into knowing which ways the child learns the most effectively in.Even though the parent from transcript one (O'Neil, 2009) simplifies everything, they still attach emotions and personalise the learning giving the child the opportunity to attempt answering the questions by thinking comfortably.
Self-System thinking/ Generate new ways of viewing a situation outside the bounds of standard conventions 1 1 This is nowhere else more demonstrated than in the classroom discourse where the fusion of the plurality of cultures takes place between that of the teacher's and his/her students (Wertsch, 2000, p. 20).The child's immediate encounter and experiences with his/her microsystem (Bronfenbrenner, 1979as cited in Gutierrez & Rogoff, 2003, p. 23) is mediated by the teacher whose expertise or lack thereof in bridging or facilitating classroom talk will determine that child's literacy development.

Table 1 :
Application of Marzano descriptorsThe table below is a summary, as showing the full Excel graphs is beyond the scope of this paper.