- Enhanced usage of grammar features in our predictive models
- Ensemble methods for increased accuracy and reduced variability
- Elimination of length bias
- Additional predictors
- More uniform distribution of scores
- Incorporation of other Machine Learning and NLP techniques...
Although these changes represent an improvement in our AES technology, we recognize that classrooms as well as individuals may track changes in a score on a thesis or other written work over time, and that these changes could disrupt that process. To mitigate this issue and ease the transition, we are blending the scores from our previous AES model with scores generated using our new scoring models. As always, we welcome any feedback on the new scoring system.
We hope to continue with another round of major enhancements to the automated grader in the summer of 2016 when it will likely be less disruptive to most users of our service.
Related
- Automated Essay Scoring Myths - Part 1
- Automated Essay Scoring Myths - Part 2
- Free grammar checker
Posts
Students report that the scores are higher all of a sudden! Intentional? Purposeful?
ReplyDeleteThe improved models do not "clump" to the center of the scoring distribution as they did in the past. This made it very difficult to get a solid A. So, "No", the intention is not to give higher grades, but "Yes", this is a side-effect of the improvements.
DeleteI have noticed that a paper can rate far below average in every measured category and still have a projected grade of "B". Why is that?
ReplyDeleteGood question. The reason is that only a few categories are shown to the user with a percentile, and the scoring algorithm uses many additional features. For example, users are not shown a percentile ranking in terms of grammar, but clearly grammar has a major impact on their grade.
DeleteI have been using PaperRater for about a year and love it.
ReplyDeleteThanks!
DeletePreviously, my work received A and B-plus as a rule. Now I am consistently getting C and C minus. Was the rater that wrong before, or has my skill level slipped that badly in the last month or so?
ReplyDeleteThe accuracy of our service has increased with our recent release, but please understand that it's still common for the score to be off by 10 points. Most writers will see an increase in their score after the recent updates, but some will see a decrease. We reduced the length bias, which means that papers no longer get a boost simply for being longer. It could be that your previous submissions benefited from this but now they do not. Feel free to visit our contact page and send us the specific paper in question.
DeleteI got a moderately good grade on my paper, but fixed the errors and my grade actually went down. There were no errors created by the added improvements of my paper, in fact it showed no errors at all. How is this possible?
ReplyDeleteGenerally, making the suggested improvements will cause your predicted grade to be higher, but this is not always the case. The prediction is based off many things, only a fraction of which are shown in the feedback sections.
DeleteI love paper rater. I feel more confident when I turn in my assignments.
ReplyDeleteI have been using paper rater for 3 year now and it has helped me improve my essay writing skills.
ReplyDeleteThanks for the feedback! That's exactly why we created this tool.
DeleteThanks so much for creating this creative tool! It has improved my vocabulary and grammar, and has caused me to get much better grades on papers.
ReplyDeletewhat does 'more uniform distribution of scores' mean, exactly?
ReplyDeleteIn this instance it means that the scores don't "clump" together in the B to C range as much as they once did.
DeleteI love paper rater! It has really helped my papers improve. Keep up the good work!
ReplyDeleteI made two entries and it confuses me how they are scored. On one I was below average in every aspect of the writing, and on the other I was above average in all of them, but my score barely moved.
ReplyDeleteIn general, improving the metrics that are displayed will boost your score, but there are many factors involved in generating the score, most of which are not displayed to the end user.
DeleteThree edits later my C turns into an A and thats ignoring what it says about transitional words and phrases. Most of them are far to stuffy for a fantasy novel xD I think the grading part i still off a but as I'm guessing it must be giving me a higher grade for vocabulary. I consider words like: muted, ambush, forward, stifled, dwindled, sheer, nostalgia, wisps, wafted, wizened to be average. :P
ReplyDeleteThat's exactly what we like to see! :)
DeleteI would like to let you know of some flaws that I found in your vocabulary words section. I'm not sure what the measure of a vocabulary word is, but I'm assuming it means having a good grasp of the English vocabulary and using a more advanced vocabulary. Paper Rater noted my use of appreciated, diverse, academic, and disheartened as vocabulary words, but ignored unorthodox, intricacies, ingenuity, and arduous. Why is that? Personally I think that the latter set of words demonstrates a firmer grasp of the English vocabulary.
ReplyDeleteI would tend to agree with you. The list of words used is not exhaustive nor is it set in stone. We hope to incorporate a larger vocabulary set in the future.
DeleteDo you have a list of words that can be used as a reference? I would love to develop a set of "college words" for my students, so a list of some sort would be appreciated!
DeleteWe are working on developing a list for public consumption right now. The "PaperRater 500" as we call it will contain 500 words that every high school and college student would benefit from knowing. Please contact us at support@PaperRater.com in a couple of weeks and we will gladly send it to you.
Delete@Christina Bowman: We have recently updated our vocabulary module to incorporate 10X the number of words. Please give it a try now and I think you'll be pleased.
DeleteThe PaperRater 500 vocabulary word list is live.
Deletean academic paper must differ substantially from a short story or a novel, for instance lines of dialogue with be shorter, and be crafted to suit the character and the occasion. How does this effect the rating.
ReplyDeleteAs of this writing, the rating does not differ based on the type of work submitted, but the feedback sometime does. This may change in the future.
DeleteIf a paper is considered "possibly plagiarized" with only 50% original content, how does it manage to get an A? Is the plagiarism not factored in?
ReplyDeleteThe grade and the originality score are completely separate and independent of one another. So, it's possible to have a completely plagiarized paper that is scored an A+.
DeleteThis comment has been removed by the author.
ReplyDeleteI'll chime in here s well. I have been using the Rater (PR) for a few years. Got in on the ground floor. I have noticed the recent changes resulting in lower grades. I was consistently getting A's and now getting B's. The biggest struggle I'm having with the new grading is the tendency to over-emphasize "big words" over concise and readable language. I have been writing for many years, and more complex words (exchanging commonly used words for fancy words I dig out in a thesaurus) is not always a good thing. It might impress a professor, but the final product is not always better. My verdict is still out on the changes...I will continue to use the objective tools, but the subjective feedback has me in a quandary.
ReplyDeleteThanks for the feedback, Kent! We hope to continue making improvements to our tools. The automated grading system is simpler than what you would find in a high-stakes testing service, so you are correct in your critique. With that said, our internal tests do confirm that it gets things close to the mark most of the time. We welcome any additional feedback!
DeleteI checked my paper . I was given 92 A as overall rate. What is this suppose to mean ?
ReplyDeleteIt means that our automated scoring system (Grendel) felt that your paper was good enough to receive an A rather than another score (B, C, D, F). However, it also means that there is still some room for improvement, so keep at it! :)
DeleteI use your services and find it a useful tool.
ReplyDeleteThanks a lot
Thanks for the kind words!
DeleteLove your program. Could you add the letter grade feature for 7th and 8th grades? Most 7th graders at this time of year are 13, so COPA should not be a problem.
ReplyDeleteAwesome site. Thanks for helping me with my College English papers. :)
ReplyDeleteHow do you check the accuracy of the grades. Are they compared to other works that have been scored by live graders?
ReplyDeleteYes, our predictive models are built and tested against documents scored by human graders.
DeleteThe Paper rater is by far very good compared to others. My score is within a B, unfortunately, however I understood how to modify my B to A! I was trying to understand the trick, why my essays are not A right away?
Deleteim a 9th grade student and paper rater rates all my essays at 98 or 99. I feel like its grading too leniently
ReplyDeleteYou could be right the Paper Rater is too lenient in its grading because from your comment it appears you confuse its[a pronoun/adjective] and it's[a contracted form of it is/it has]. With this, I dont think your essays could go close to 98 or 99 as you are likely to come up with other confused words. Anyway, this is my observation.
DeleteThe paper rater rates all of my essays at 98 or 99. I think it grades too leniently
ReplyDeleteThat sounds like a great problem to have! Trust me when I say that is not the norm.
DeleteShouldn't grades "clump"? I mean there's probably a Gaussian distribution of some kind; more people get C+'s and B-'s than A's, D's or F's on average anyways, right?
ReplyDeleteYou are absolutely right! There's a reason a Guassian distribution is called the "normal distribution". But when modeling any process you want the distribution of predictions to match the distribution you are modeling. In data science terms, the distribution of your predictions should match the distribution of your training labels. In our previous predictive models this was not the case.
DeleteI have always loved using paper rater, especially now that they have Ginger. I no longer need to install GInger for Office because it tends to cause the program to launch slowly. Thanks Paper Rater.
ReplyDeleteWhen I type my essays in word, I use spell check. That however, doesn't catch all of the grammar and spelling errors the paper rater does. Also, it doesn't give me a rough idea of how I'm doing or what I can improve on. Thank you so much paper rater.
ReplyDeletePaper rater catches all the errors that word doesn't and tells me where I can improve. Biggest life saver ever.
ReplyDelete