Automated Essay Scoring Updates

Today, September 23rd, 2015, we are rolling out the most significant change to our Automated Essay Scoring system in its history.  This involves many improvements summarized below:

  • Enhanced usage of grammar features in our predictive models
  • Ensemble methods for increased accuracy and reduced variability
  • Elimination of length bias
  • Additional predictors
  • More uniform distribution of scores
  • Incorporation of other Machine Learning and NLP techniques...
Although these changes represent an improvement in our AES technology, we recognize that classrooms as well as individuals may track changes in a score on a thesis or other written work over time, and that these changes could disrupt that process.  To mitigate this issue and ease the transition, we are blending the scores from our previous AES model with scores generated using our new scoring models.  As always, we welcome any feedback on the new scoring system.

We hope to continue with another round of major enhancements to the automated grader in the summer of 2016 when it will likely be less disruptive to most users of our service.

52 comments:

  1. Students report that the scores are higher all of a sudden! Intentional? Purposeful?

    ReplyDelete
    Replies
    1. The improved models do not "clump" to the center of the scoring distribution as they did in the past. This made it very difficult to get a solid A. So, "No", the intention is not to give higher grades, but "Yes", this is a side-effect of the improvements.

      Delete
  2. I have noticed that a paper can rate far below average in every measured category and still have a projected grade of "B". Why is that?

    ReplyDelete
    Replies
    1. Good question. The reason is that only a few categories are shown to the user with a percentile, and the scoring algorithm uses many additional features. For example, users are not shown a percentile ranking in terms of grammar, but clearly grammar has a major impact on their grade.

      Delete
  3. I have been using PaperRater for about a year and love it.

    ReplyDelete
  4. Previously, my work received A and B-plus as a rule. Now I am consistently getting C and C minus. Was the rater that wrong before, or has my skill level slipped that badly in the last month or so?

    ReplyDelete
    Replies
    1. The accuracy of our service has increased with our recent release, but please understand that it's still common for the score to be off by 10 points. Most writers will see an increase in their score after the recent updates, but some will see a decrease. We reduced the length bias, which means that papers no longer get a boost simply for being longer. It could be that your previous submissions benefited from this but now they do not. Feel free to visit our contact page and send us the specific paper in question.

      Delete
  5. I got a moderately good grade on my paper, but fixed the errors and my grade actually went down. There were no errors created by the added improvements of my paper, in fact it showed no errors at all. How is this possible?

    ReplyDelete
    Replies
    1. Generally, making the suggested improvements will cause your predicted grade to be higher, but this is not always the case. The prediction is based off many things, only a fraction of which are shown in the feedback sections.

      Delete
  6. I love paper rater. I feel more confident when I turn in my assignments.

    ReplyDelete
  7. I have been using paper rater for 3 year now and it has helped me improve my essay writing skills.

    ReplyDelete
    Replies
    1. Thanks for the feedback! That's exactly why we created this tool.

      Delete
  8. Thanks so much for creating this creative tool! It has improved my vocabulary and grammar, and has caused me to get much better grades on papers.

    ReplyDelete
  9. what does 'more uniform distribution of scores' mean, exactly?

    ReplyDelete
    Replies
    1. In this instance it means that the scores don't "clump" together in the B to C range as much as they once did.

      Delete
  10. I love paper rater! It has really helped my papers improve. Keep up the good work!

    ReplyDelete
  11. I made two entries and it confuses me how they are scored. On one I was below average in every aspect of the writing, and on the other I was above average in all of them, but my score barely moved.

    ReplyDelete
    Replies
    1. In general, improving the metrics that are displayed will boost your score, but there are many factors involved in generating the score, most of which are not displayed to the end user.

      Delete
  12. Three edits later my C turns into an A and thats ignoring what it says about transitional words and phrases. Most of them are far to stuffy for a fantasy novel xD I think the grading part i still off a but as I'm guessing it must be giving me a higher grade for vocabulary. I consider words like: muted, ambush, forward, stifled, dwindled, sheer, nostalgia, wisps, wafted, wizened to be average. :P

    ReplyDelete
  13. I would like to let you know of some flaws that I found in your vocabulary words section. I'm not sure what the measure of a vocabulary word is, but I'm assuming it means having a good grasp of the English vocabulary and using a more advanced vocabulary. Paper Rater noted my use of appreciated, diverse, academic, and disheartened as vocabulary words, but ignored unorthodox, intricacies, ingenuity, and arduous. Why is that? Personally I think that the latter set of words demonstrates a firmer grasp of the English vocabulary.

    ReplyDelete
    Replies
    1. I would tend to agree with you. The list of words used is not exhaustive nor is it set in stone. We hope to incorporate a larger vocabulary set in the future.

      Delete
    2. Do you have a list of words that can be used as a reference? I would love to develop a set of "college words" for my students, so a list of some sort would be appreciated!

      Delete
    3. We are working on developing a list for public consumption right now. The "PaperRater 500" as we call it will contain 500 words that every high school and college student would benefit from knowing. Please contact us at support@PaperRater.com in a couple of weeks and we will gladly send it to you.

      Delete
    4. @Christina Bowman: We have recently updated our vocabulary module to incorporate 10X the number of words. Please give it a try now and I think you'll be pleased.

      Delete
  14. an academic paper must differ substantially from a short story or a novel, for instance lines of dialogue with be shorter, and be crafted to suit the character and the occasion. How does this effect the rating.

    ReplyDelete
    Replies
    1. As of this writing, the rating does not differ based on the type of work submitted, but the feedback sometime does. This may change in the future.

      Delete
  15. If a paper is considered "possibly plagiarized" with only 50% original content, how does it manage to get an A? Is the plagiarism not factored in?

    ReplyDelete
    Replies
    1. The grade and the originality score are completely separate and independent of one another. So, it's possible to have a completely plagiarized paper that is scored an A+.

      Delete
  16. This comment has been removed by the author.

    ReplyDelete
  17. I'll chime in here s well. I have been using the Rater (PR) for a few years. Got in on the ground floor. I have noticed the recent changes resulting in lower grades. I was consistently getting A's and now getting B's. The biggest struggle I'm having with the new grading is the tendency to over-emphasize "big words" over concise and readable language. I have been writing for many years, and more complex words (exchanging commonly used words for fancy words I dig out in a thesaurus) is not always a good thing. It might impress a professor, but the final product is not always better. My verdict is still out on the changes...I will continue to use the objective tools, but the subjective feedback has me in a quandary.

    ReplyDelete
    Replies
    1. Thanks for the feedback, Kent! We hope to continue making improvements to our tools. The automated grading system is simpler than what you would find in a high-stakes testing service, so you are correct in your critique. With that said, our internal tests do confirm that it gets things close to the mark most of the time. We welcome any additional feedback!

      Delete
  18. I checked my paper . I was given 92 A as overall rate. What is this suppose to mean ?

    ReplyDelete
    Replies
    1. It means that our automated scoring system (Grendel) felt that your paper was good enough to receive an A rather than another score (B, C, D, F). However, it also means that there is still some room for improvement, so keep at it! :)

      Delete
  19. I use your services and find it a useful tool.
    Thanks a lot

    ReplyDelete
  20. Love your program. Could you add the letter grade feature for 7th and 8th grades? Most 7th graders at this time of year are 13, so COPA should not be a problem.

    ReplyDelete
  21. Awesome site. Thanks for helping me with my College English papers. :)

    ReplyDelete
  22. How do you check the accuracy of the grades. Are they compared to other works that have been scored by live graders?

    ReplyDelete
    Replies
    1. Yes, our predictive models are built and tested against documents scored by human graders.

      Delete
    2. The Paper rater is by far very good compared to others. My score is within a B, unfortunately, however I understood how to modify my B to A! I was trying to understand the trick, why my essays are not A right away?

      Delete
  23. im a 9th grade student and paper rater rates all my essays at 98 or 99. I feel like its grading too leniently

    ReplyDelete
    Replies
    1. You could be right the Paper Rater is too lenient in its grading because from your comment it appears you confuse its[a pronoun/adjective] and it's[a contracted form of it is/it has]. With this, I dont think your essays could go close to 98 or 99 as you are likely to come up with other confused words. Anyway, this is my observation.

      Delete
  24. The paper rater rates all of my essays at 98 or 99. I think it grades too leniently

    ReplyDelete
    Replies
    1. That sounds like a great problem to have! Trust me when I say that is not the norm.

      Delete
  25. Shouldn't grades "clump"? I mean there's probably a Gaussian distribution of some kind; more people get C+'s and B-'s than A's, D's or F's on average anyways, right?

    ReplyDelete
    Replies
    1. You are absolutely right! There's a reason a Guassian distribution is called the "normal distribution". But when modeling any process you want the distribution of predictions to match the distribution you are modeling. In data science terms, the distribution of your predictions should match the distribution of your training labels. In our previous predictive models this was not the case.

      Delete
  26. I have always loved using paper rater, especially now that they have Ginger. I no longer need to install GInger for Office because it tends to cause the program to launch slowly. Thanks Paper Rater.

    ReplyDelete
  27. When I type my essays in word, I use spell check. That however, doesn't catch all of the grammar and spelling errors the paper rater does. Also, it doesn't give me a rough idea of how I'm doing or what I can improve on. Thank you so much paper rater.

    ReplyDelete
  28. Paper rater catches all the errors that word doesn't and tells me where I can improve. Biggest life saver ever.

    ReplyDelete

All comments are reviewed by the moderator BEFORE they appear on this page. Spam will be deleted, so don't waste your time or my time.