Phase 1 (first-year students):
From students:
- Freshman Class Survey
- Pre- and Post-Test Process Narratives
- Synthesis Essays
The following survey questions investigate first-semester freshmen’s perceptions of “research,” their self-efficacy (see Kurbanoglu et al), and their experiences with research. The survey was administered using a five-point likert scale:
- Strongly Disagree
- Disagree
- Neither Agree Nor Disagree
- Agree
- Strongly Agree
Demographic questions included:
- What is your gender?
- What is your age?
- What is your major?
- What is your second major (if applicable)?
- What is your third major (if applicable)?
- Are you a freshman or a transfer student?
- Is English your native language?
- What is your ethnic background?
(Kurbanoglu, S. Serap, Buket Akkoyunlu, and Aysun Umay. “Developing the Information Literacy Self-Efficacy Scale.” Journal of Documentation 62:6 (2006): 730-43.)
- When completing papers, I am required to use outside sources.
- I regularly use websites as sources for my papers.
- I regularly use print or online magazines and newspapers as sources for my papers.
- I regularly use scholarly sources for my papers, like academic journal articles.
- I do not have confidence in my academic research skills.
- I talk to others about possible topics before making a final choice for my papers.
- When I need research assistance, I contact a librarian.
- I tend to give up in my research when I can’t find an answer easily.
- I persist in making sense of my research, even when my sources present different views.
- I can effectively summarize, paraphrase, or quote from a source.
- I can cite sources appropriately in my writing.
- I look for sources after I write my papers.
- I regularly change my way of thinking about a topic by reading sources.
- I regularly choose the first sources I find for my papers.
- I use sources that challenge my point of view.
- When I find a source for my writing, I retrieve the full text of the source (e.g., by downloading, copying, or requesting from another library – “Interlibrary Loan”).
- As long as you let your reader know that you’ve done your research, it doesn’t matter what you say about your sources.
- Coming up with research questions helps me figure out how to think about my papers.
- I develop a variety of keywords to use when searching for sources.
- I think about what kind of sources I need for a paper (e.g., websites, newspaper articles, journal articles, etc.) before searching for them.
- I do not think about where to find sources (e.g., library catalogue, academic databases, etc.) before searching for them.
- I take notes when reading and thinking about sources I might use for my papers.
- I feel confident and competent that I can learn from my research experience and improve my research skills.
- I feel confident and competent critiquing the quality of my research.
Imagine that you have been assigned a 1500-word essay for this course. The essay must develop an argument about a current social issue and must use at least three outside sources. Explain how you would go about completing this assignment. Be as specific and detailed as possible.
A FYW student’s written response to the Synthesis Essay Assignment composed in the context of the student’s assigned section of ENG 1103. Synthesis Essays were analyzed by researchers using a coding scheme for each of two dimensions: source referentiality and source engagement
From faculty:
- Synthesis Essay Assignments
- Scores Assigned to Synthesis Essays on the Learning Outcome of Integration (as part of program assessment)
The Synthesis Essay Assignment was an FYW assignment introduced in fall 2012 following the adoption of “integration of sources” as a program learning outcome. The assignment varied among instructors and sections and had just two requirements:
- an essay of ~1200-1400 words
- integrate at least three sources
Instructors were encouraged to design and to share their assignments in order to generate ideas about what the assignment would accomplish and why in the context of ENG 1103.
At the time of data collection, “integration of sources” was one among several learning outcomes for ENG 1103. One aim of this outcome was to invite students to conduct research as inquiry, in the sense of integrating others’ ideas and information with their own. In Spring 2013, FYW instructors engaged in the process of dynamic criteria mapping (per the work of Bob Broad), developing a criteria map and glossary that would be further refined in assessment. For the assessment procedure, the eight members of the assessment committee were paired up and each pair scored the same packet. Readers were asked to assign each essay in their packet an Integration Score using a 1-6 scale, where 1-3 signified degrees of failing to meet expectations for the outcome and 4-6 signified degrees of meeting or exceeding expectations for the outcome. As a result, each essay was associated with what we call a Total Integration Score: a score from 2-12 that represents the sum of the two Integration Scores assigned by the essay’s faculty readers.
From first-year writing program assessment:
- Dynamic Criteria Mapping Rubric
- 2012-2013 Assessment Report
During the process of dynamic criteria mapping, faculty developed a rubric (including worksheet and glossary) that would be used in assessment.
The Composition Program 2012-2013 Assessment Report describes in detail the process of dynamic criteria mapping (per the work of Bob Broad). It also includes results, conclusions, and recommendations.
Phase 2 (senior students):
- Senior Class Survey
- Process Narratives
- Information Literacy Narratives
- Completed Coursework (General Education Literature Course, Major Project, Other Coursework)
- Faculty Feedback on Completed Coursework (e.g., grade, marginalia)
- Semistructured Document Interview (with Student)
The Senior Class Survey employed the same survey questions as the Freshman Class Survey. These questions investigated seniors’ perceptions of “research,” their self-efficacy (see Kurbanoglu et al), and their experiences with research by the late fall or early spring of students’ fourth year. As before, the survey was administered using a five-point likert scale. The same demographic questions were included, with the addition of the following questions:
- What is your minor (if applicable)?
- What is your second minor (if applicable)?
- Did you come to [university] as a freshman or a transfer student?
- What was your major when you first entered [university] as a new student?
In addition, the Senior Class Survey included several “growth mindset” questions, informed by the university’s recent Quality Enhancement Plan. These questions were based on the Implicit Theories of Intelligence Scale (for Adults) (Dweck). Consistent with that measure, which has been tested for reliability and validity (Dweck, Chiu, & Hong), the survey was administered using a six-point likert scale:
- Strongly Disagree
- Disagree
- Somewhat Disagree
- Somewhat Agree
- Agree
- Strongly Agree
These questions included the following:
- You have a certain amount of intelligence, and you can’t really do much to change it.
- Your intelligence is something about you that you can’t change very much.
- To be honest, you can’t really change how intelligent you are.
- You can learn new things, but you can’t really change your basic intelligence.
- You have a certain ability to do research, and you can’t really do much to change it.
- Your ability to do research is something about you that you can’t change very much.
- To be honest, you can’t really change your ability to do research.
- You can learn new things, but you can’t really change your basic ability to do research.
Sources
Dweck, Carol S. Self-Theories: Their Role in Motivation, Personality, and Development. Philadelphia, PA: Psychology Press, 1999. Print.
Dweck, Carol S., Chi-yue Chiu, and Ying-yi Hong. “Implicit Theories and Their Role in Judgments and Reactions: A World from Two Perspectives.” Psychological Inquiry 6.4 (1995): 267–285. Print.
Kurbanoglu, S. Serap, Buket Akkoyunlu, and Aysun Umay. “Developing the Information Literacy Self-Efficacy Scale.” Journal of Documentation 62:6 (2006): 730-43.
Imagine that you have been assigned a research essay in one of the upper-division courses for your major. This essay must develop an argument about some problem or question significant to your major and must use several outside sources. Explain how you would go about completing this assignment. Be as specific and detailed as possible.
A literacy narrative is simply a story about how you learned to read, write, or compose. Because we are interested in how students conduct research–a form of information literacy–we are interested in how you might tell a story about that type of literacy development. Write a short narrative focused on one of the prompts below:
- Tell a story about learning the kinds of research skills or values important to your major.
- Tell a story about your most significant learning experience doing research. Where was it? Who else was involved and how?
To create a layered understanding of students’ development, we asked each student to provide essays, projects, and other significant assignments from different moments in the college curriculum. Specifically, we requested coursework from students’ required general education literature course, typically experienced mid-way through their college careers (ENG 2200, or an approved substitute). They also were asked for materials associated with substantial projects completed as part of their respective majors. Finally, students were prompted to share any other coursework that they considered crucial to their development as researchers and writers.
Students and faculty were asked to share any feedback from faculty on completed coursework. Such feedback might include grades, scores associated with rubrics, marginal comments within student essays, verbal feedback, etc.
General Perceptions, Experiences, and Self-Efficacy With Respect to Academic Research and Writing
- What is your major? Do you have a second major? What is it? Do you have a minor? What is it? Second minor? When did you declare your major?
- Briefly, what is “research”? What is the role of research with respect to writing?
- How do you understand research now as compared with how you understood it in your first year of college? If something has changed, what? Why did it change?
- To what extent is your ability to do research something that you can or cannot change? That is, to what extent can you change how good of a researcher you are? How so? Why?
- To what extent are you confident as far as conducting research? Has your confidence changed since your first year? How so? Why?
- When you have chosen a topic for an essay (or project), to what extent does reading sources change your way of thinking about a topic? If your thinking changes, how does it change? Why does your thinking change (or why not)?
- To what extent do you regularly choose the first sources you find for your essay (or project)? Why (not)?
- To what extent do you use sources that challenge your point of view? Why (not)?
- To what extent can research be planned? Why (not)? Do you see research as more “linear” (one step after another) or “recursive” (going back and redoing something when necessary, e.g., when you find out something new)?
- To what extent do you think about what kind of sources you need for an essay (or project) (e.g., websites, newspaper articles, journal articles, etc.) before searching for them? Why (not)?
- To what extent do you think about where to find sources (e.g., library catalogue, academic databases, etc.) before searching for them? Why (not)?
- When you find a source for your writing, to what extent do you retrieve the full text of the source (e.g., by downloading, copying, or requesting from another library–”Interlibrary Loan”)? Why (not)?
- To what extent do you take notes when reading and thinking about sources you might use for essays (or projects)? Why (not)?
- To what extent do you look for sources after you write your essays (or projects)? Why (not)?
- To what extent do you persist with your research when it is frustrating? How so? Has this persistence changed since your first year? How so? Why?
- Who (if anyone) helps you with your research? How so? Has this changed since your first year? How so? Why?
- Do you talk to others about possible topics before making a final choice for your essays (or projects)? Why (not)?
- What more do you feel you have to learn about research? Could something have been done differently to help you learn about research while in college? Do you anticipate learning more? How so?
Exploring “Source Engagement” (with Documents)
- In general, what does it mean to you to engage with sources in an essay or project of this sort?
- Where do you see yourself as engaging with sources in these essays (or projects)?
- What was the assignment for this essay (or project)?
- What kind of essay (or project) is this [e.g., research paper]?
- How would you describe your engagement with sources?
- What is happening at that moment in the essay?
- Is your engagement with the source effective? Why (not)?
- Do you engage with sources differently from moment to moment? How so?
- Imagine an essay or project that exceeds all expectations for the assignment. How does your engagement with sources compare with respect to this exemplary essay or project?
- What is similar? Different? How?
- What is effective by comparison? Ineffective by comparison? How? (Followup: In general, what does it mean to you to engage with sources in an essay or project of this sort?)
- What grade did you receive on these essays (or projects) (i.e., Critical Reading & Interpretation (“CRI,” or ENG2200) Final Essay and Assignment; Major Capstone or Other Project and Assignment; and/or Other Course Work)?
- To what extent do you think that the grade was based on how you engaged with sources in these essays (or projects)? That is, how important is it that students engage with sources in essays (or projects)? What other criteria may be more (or less) important?
- How did you learn to engage with sources while at college? For this essay (or project)?
- How confident are you as far as engaging with sources in projects such as these? Has your confidence changed since your first year? How so? Why?
- What more do you feel you have to learn about engaging with sources? Could something have been done differently to help you learn about engaging with sources while in college? Do you anticipate learning more? How so?
Other (Nonacademic) Research and Writing
- In what areas of your life do you conduct research most often?
- In what areas of your life do you conduct research that is important to you? What situations or contexts prompt you to do research?
- What courses, people, or experiences prepared you to do this kind of research?
Phase 3 (faculty):
- Coursework Assignment
- Faculty Feedback on Completed Coursework (e.g., marginalia, exemplary artifact, etc.)
- Assessment Documents (e.g., rubric, department outcomes related to research, etc.)
- Supplemental Assignments or Teaching Materials
- Syllabus
- Semistructured Document Interview (with Faculty)
Faculty were asked to share assignments materials–the the extent available–associated with the completed coursework provided to us by students.
Students and faculty were asked to share any feedback from faculty on completed coursework. Such feedback might include grades, scores associated with rubrics, marginal comments within student essays, verbal feedback, etc. Faculty also were asked to share an exemplary artifact of their choosing that represented the kind of student work their assignments ideally would elicit.
Sometimes smaller assignments or pedagogical strategies were noted by students and faculty as being important to students accomplishing their larger essays or projects. Faculty were asked to share related documentation to the extent it was available.
In order to better understand the course context for other data, we collected syllabi from faculty for courses for which students shared completed coursework.
Semistructured Document Interview (with Faculty)
- What do you remember about the student? In what context did you work with the student?
- Briefly, what is “research”? What is the role of research with respect to writing?
- In general, what does it mean to you when a student engages with sources? (In an essay or project of this sort?)
- What was the assignment for this essay (or project)?
- What kind of essay or project is this [e.g., research paper]? (genre)
- Do you understand the student as engaging with sources in this essay or project? If so, where?
- How would you describe the student’s engagement with sources?
- What is happening at that moment in the essay or project?
- Does the student engage with sources differently from moment to moment? How so?
- Is the student’s engagement with sources effective? Why (not)?
- Imagine an essay or project that exceeds all expectations for the assignment. Do you have an example of an essay or project that represents what you are aiming for with this assignment (e.g., another student’s essay, a scholarly article)?
- How does the student’s engagement with sources compare with respect to the exemplary artifact you have chosen (i.e., the kind of essay or project your assignment ideally elicits)?
- What is similar? Different? How?
- What is effective by comparison? Ineffective by comparison? How?
- What grade did you (or would you) give this essay or project?
- To what extent is that grade based on how the student engaged with sources in this essay (or project)? That is, how important is this criteria? What other criteria may be more (or less) important?
- Is source engagement reflected in your program or department’s outcomes or assessment materials? How so?
- How does the student’s engagement with sources compare with your program or department’s expectations regarding assessment of this assignment (if any)? Does the student’s engagement with sources meet expectations? Not meet expectations? Exceed expectations? Why? Followup: What is “research”? Role of research with respect to writing? In general, what does it mean to you when a student engages with sources in an essay or project of this sort?
- How did this student (and other students like him or her) learn to engage with sources? For this essay (or project)?
- What is your role in helping students with their research? Specifically, how do you teach students to engage with sources?
- Given the essay (or project) the student completed for you, what more does this student have to learn about research? Engaging with sources?
- How might you teach students to engage with sources differently? (Does our conversation prompt any new ideas?)
- Do you have a syllabus for this course? Share?
- Do you have the assignment for this essay (or project)? Share?
- Do you have any supplementary materials for this assignment (e.g., preparatory assignments, assignment guidelines, reading questions, other teaching materials, etc.)? Share?
- Do you have any notes or feedback regarding this essay (or project) (e.g., marginalia, rubric)? Share?
- Do you have the “exemplary artifact” (example of an essay or project that represents what you are aiming for with this assignment)? Share?
- Do you have a copy of your program or department’s outcomes or assessment materials? Share?