“There is a huge value in learning with instant feedback,” Dr. Agarwal said. “Students are telling us they learn much better with instant feedback.”
But skeptics say the automated system is no match for live teachers. One longtime critic, Les Perelman, has drawn national attention several times for putting together nonsense essays that have fooled software grading programs into giving high marks. He has also been highly critical of studies that purport to show that the software compares well to human graders.
“My first and greatest objection to the research is that they did not have any valid statistical test comparing the software directly to human graders,” said Mr. Perelman, a retired director of writing and a current researcher at M.I.T.
He is among a group of educators who last month began circulating a petition opposing automated assessment software. The group, which calls itself Professionals Against Machine Scoring of Student Essays in High-Stakes Assessment, has collected nearly 2,000 signatures, including some from luminaries like .
“Let’s face the realities of automatic essay scoring,” the group’s statement reads in part. “Computers cannot ‘read.’ They cannot measure the essentials of effective written communication: accuracy, reasoning, adequacy of evidence, good sense, ethical stance, convincing argument, meaningful organization, clarity, and veracity, among others.”
But EdX expects its software to be adopted widely by schools and universities. EdX offers free online classes from Harvard, M.I.T. and the ; this fall, it will add classes from Wellesley, Georgetown and the . In all, 12 universities participate in EdX, which offers certificates for course completion and has said that it plans to continue to expand next year, including adding international schools.
The EdX assessment tool requires human teachers, or graders, to first grade 100 essays or essay questions. The system then uses a variety of machine-learning techniques to train itself to be able to grade any number of essays or answers automatically and almost instantaneously.
The software will assign a grade depending on the scoring system created by the teacher, whether it is a letter grade or numerical rank. It will also provide general feedback, like telling a student whether an answer was on topic or not.
Dr. Agarwal said he believed that the software was nearing the capability of human grading.
“This is machine learning and there is a long way to go, but it’s good enough and the upside is huge,” he said. “We found that the quality of the grading is similar to the variation you find from instructor to instructor.”
EdX is not the first to use automated assessment technology, which dates to early mainframe computers in the 1960s. There is now a range of companies offering commercial programs to grade written test answers, and four states — , , and — are using some form of the technology in secondary schools. A fifth, , has experimented with it. In some cases the software is used as a “second reader,” to check the reliability of the human graders.
But the growing influence of the EdX consortium to set standards is likely to give the technology a boost. On Tuesday, Stanford announced that it would work with EdX to develop a joint educational system that will incorporate the automated assessment technology.
Two start-ups, Coursera and Udacity, recently founded by Stanford faculty members to create “massive open online courses,” or MOOCs, are also committed to automated assessment systems because of the value of instant feedback.
“It allows students to get immediate feedback on their work, so that learning turns into a game, with students naturally gravitating toward resubmitting the work until they get it right,” said Daphne Koller, a computer scientist and a founder of Coursera.
Last year the Hewlett Foundation, a grant-making organization set up by one of the founders and his wife, sponsored two $100,000 prizes aimed at improving software that grades essays and short answers. More than 150 teams entered each category. A winner of one of the Hewlett contests, Vik Paruchuri, was hired by EdX to help design its assessment software.
“One of our focuses is to help kids learn how to think critically,” said Victor Vuchic, a program officer at the Hewlett Foundation. “It’s probably impossible to do that with multiple-choice tests. The challenge is that this requires human graders, and so they cost a lot more and they take a lot more time.”
Mark D. Shermis, a professor at the University of Akron in , supervised the Hewlett Foundation’s contest on automated essay scoring and wrote a paper about the experiment. In his view, the technology — though imperfect — has a place in educational settings.
With increasingly large classes, it is impossible for most teachers to give students meaningful feedback on writing assignments, he said. Plus, he noted, critics of the technology have tended to come from the nation’s best universities, where the level of pedagogy is much better than at most schools.
“Often they come from very prestigious institutions where, in fact, they do a much better job of providing feedback than a machine ever could,” Dr. Shermis said. “There seems to be a lack of appreciation of what is actually going on in the real world.”Continue reading the main story
Not many students would admit to enjoying taking exams or writing essays, but if you want to get a degree, they're an ordeal you have to survive.
So we've worked out how to make the whole thing a little less stressful. We've persuaded four academics from a range of subject areas to tell us the top 10 things students get wrong in exams and coursework. This is what they've told us:
Panic and procrastination
Sometimes a task can feel so overwhelming that it's difficult to begin, says Amber Regis, lecturer in 19th century literature at the University of Sheffield. Procrastination takes over and you just can't seem to get anything done. The bare white page is a formidable foe when it stares back at you, untouched, from the library desk. Try not to panic, protect and manage your preparation time, and don't put off getting started.
Lack of analysis
It can be tempting to parrot everything you know when writing essays and exam answers. But to demonstrate your understanding you should engage critically with your source material. Always assume an informed reader — they do not need a plot summary or biographies of key figures. Read through the marking scheme used by your department. You will notice some very telling words and phrases attached to the highest marks, for example: "originality of interpretation", "astute engagement" and "critical thought". To fulfil these criteria, you must favour analysis.
In exams it's vital that you don't jump the gun. Take the first five to 10 minutes to read through the paper and plan the questions you're going to answer in order of how confident you feel in that subject area, says Bhavik Patel, lecturer in physical and analytical chemistry at University of Brighton. Make sure you secure the marks on the questions that you find easiest to answer first, before attempting questions that are more difficult. The latter often make you lose confidence and time during exam conditions.
Not reading the question properly
When revising, students often rehearse answers in their head. says Roy Jackson, course leader in religion, philosophy and ethics at the University of Gloucestershire. "Although we don't deliberately intend to catch them out in exams, we do set questions that requires them to think and reflect under timed conditions. But instead students will often pick up key words in the question and write out a rehearsed response."
This can be avoided by taking some time to reflect upon the question, rather than seeing that as wasted time and rushing to fill the pages.
Focusing on word count
In both exam responses and coursework, students are often more concerned with quantity rather than quality. The best essays are those that demonstrate evidence of personal reflection and are not just trying to achieve a word limit.
Insufficient reading around a subject
During revision time, students are too selective in what they choose to read, selecting one or two books and remembering as much from those as possible. What comes across in a good essay is confidence, and this can only be achieved by demonstrating plenty of reading on a subject, so that you can be prepared for any question that you come across. This also requires giving yourself plenty of time to read, and not leaving it until a few days before an exam or assignment.
Regurgitating in-class or lecture material
In English we are looking for excitement and originality of thought backed up by evidence and we don't want you to take our formulations as gospel truth, says Martin Eve, lecturer in English literature at the University of Lincoln. Challenge – and think for yourself.
Always make sure your statements are specific and show self-awareness. Do say: "There is no one single representation of working-class life in post-50s British fiction". Don't ever go for something like: "Novels that feature the working class show us that these people..."
Getting characters' names or other basic factual details wrong just smacks of not caring. If you don't care enough to do this correctly when you're paying to be at university, what will an employer think when he or she is paying you?
Spelling, grammar and register
Universities have a standard academic English in which you should write. The best way to become proficient at this is to read a great number of academic journal articles and books and mirror the register, language and tone (but not the content: never plagiarise!). It can also help to write a small amount every day as a form of practice.