Question

SmartScore -- An Explanation [Please read if you have any initial questions, then feel free to ask more!] @Laura* @cshalvey @shadowfiend @mattfeury @farmdawgnation @chris @darthsid chime on in

Answer 1

YAYYY!!!

Answer 2

Your SmartScore is a new concept that attempts to reflect the effort and ability you apply not only to learning, but also, to helping and to being supportive of fellow people who share a desire to share knowledge. We analyze the actions and behaviors of all users on the site. This analysis is meant to capture the things that grades cannot such as your ability to collaborate with other people, your willingness to be a compassionate helper of a person in need, or your natural urge to ask questions in order to understand new concepts. The score itself is determined relative to the scores of other users.

The variables, categories, and mathematics behind the score will always be improved and altered as we add new features to the site. Your score may change due to these improvements, but ultimately you have the most power to change and to improve your own score simply by using OpenStudy to study with your peers and to aid in the effort to make the world one big studygroup.

Answer 3

You’ll notice in the new profile pages there is a graph to display the progress of your SmartScore for the past few weeks. This represents your overall SmartScore which is currently comprised of your scores in the three categories we’ve decided to focus upon (for now):

Teamwork - Which focuses upon your willingness and aptitude to communicate and interact with others in group settings, very small groups like those that are formed within a question and large groups like all users studying mathematics. You interact in chat, but your actions within questions -- asking, answering, helping, guiding are considered more heavily as they are core the principles of the site

Problem Solving - The ability to identify a situation/question to which you are willing to exert your own effort to construct a coherent, thoughtful, and respectfully toned solution to the benefit of another and even yourself.

Engagement - which focuses primarily upon the actions and effort you exert in your own questions that you have asked because you want to learn a new concept and your willingness to interact with people who are helping you.

You’re surely wondering what each variable actually means in the category boxes. Below are the descriptions. Keep in mind, these are not all the variables that we are analyzing. There’s a lot more to the score, and we’ll keep you informed about the progress.

Answer 4

Problem Solving
- Questions Answered - These are the questions that other people have asked and you decided to answer.

- Medals for Answers - These are the medals you’ve received for helping someone in need, specifically someone who asked a question. You took the time to help the asker understand a new concept, and the asker thanked you with a medal.

Answer 5

Teamwork
- Fans
These are the people who think you’re awesome at helping others learn hard concepts or at asking good questions or even just at being friendly and telling a good joke.

- Testimonials
These are written by the people who really think you’re awesome, and they just had to tell you.

- People Helped
There are the people whose questions you’ve answered when they really needed help, you signed-on, saw their question and swooped in with a solution. Primarily, it's unique, meaning once per user.

Answer 6

Engagement
- Medals for Asking
These are the medals other people have given you because you were being involved and collaborative in the questions that you ask, really showing a desire to learn.

- Days Studied
This tracks the days you’ve been on OpenStudy, either helping or being helped.

Answer 7

woowww..very thorough :DD

Answer 8

Some FAQs:

1. If I have a SmartScore of 50, is that bad?

The score itself may be out of 0 to 100, but don’t get stuck in the old ways like thinking that a 50 as SmartScore is like receiving a 50% on a test in Biology. The score is a measure of your efforts on the site relative to those of your peers. Actually, a 50 SmartScore is a good score in our system! Be proud and share it, and of course, improve it!


2. How do I improve my SmartScore?

You improve your score by participating on OpenStudy by helping others, asking questions, and supporting the open learning-based community you’ve helped to create. We haven’t listed all the metrics and variables that we use to determine the score, but some of the major variables are listed in your profile and explained (see above).


3. There are differences between the numbers on my profile, what is up with that?

The score and variables and queries and analysis we use will always be changing. Discrepancies are thoroughly examined. Please inform us of any discrepancies that you find. If you have any feedback related to SmartScore please either post a question in the OpenStudy Feedback section (perhaps this question/thread). We’d love your feedback!

Answer 9

what is invited users? I saw the term in saifoo's profile.

Answer 10

maybe the ones he has recruited?

Answer 11

You can share you questions and SmartScore now, primarily you share these on social media, this sharing can be seen as an invitation. This then refers to people whom you invited and who joined OpenStudy.

Answer 12

ahhh exciting >:)))

Answer 13

i see.. thanks for the info! I am going to share my smartscore on fb.

Answer 14

these administrators never cease to surprise us lol

Answer 15

"This analysis is meant to capture the things that grades cannot such as your ability to collaborate with other people, your willingness..."

It's not that Universities are going to ask my smart score on OpenSudy, is it? So, why did you create it?

Answer 16

@Ishaan94 This is a new approach, and with a new approach, adoption takes time. We created it because we felt that it was a necessary to help accurately represent learners. Because this is a social sight, with dynamic content created by all you users, we get to see aspects of your personality that perhaps an A in Chemistry does not accurately portray.

Answer 17

could you delete my question from yesterday?

Answer 18

List of Relevant Links:

http://blog.openstudy.com/2012/04/09/to-boldly-go-where-no-grades-have-gone-before/

http://blog.openstudy.com/2012/04/17/redefining-%E2%80%9Csmart%E2%80%9D-openstudy-beyond-badges-and-points/

Answer 19

And how about problem solving respective to question's difficulty? Solving one FoolForMath's problem is way tougher than solving hundreds of slope, line problems. Maybe you guys can allow some users on openstudy or you yourself to increase problem solving score on some problems. Problems posted by KingGeorge are solvable by none on OpenStudy almost none. So, problem solving score for KingGeorge's problems should be much much higher. I don't know if you have already accounted for that.

Answer 20

We do indirectly consider this: we consider receiving medals from high level users to be more significant/valuable

Answer 21

And one thing more, how do you calculate smart score, my smart score is 99 but my problem solving is 93, teamwork is 75 and engagement 71. How did my smart score got 99? It shouldn't exceed 93, right? I can be wrong and I think I'm wrong.

Answer 22

Each category is first evaluated individually. The overall score is comprised of each category, but necessarily the score that you see which is between [0,100]. In the system you have a raw score of points accrued for each category, which we then use to determine your depicted score, that in the range [0,100].

Answer 23

*not necessarily

Answer 24

Oh, hmm I don't think I still understand it, but Thank you for your time :-)

Answer 25

Awesome.
PS you posted same paragraph twice. ;)

Answer 26

haha, thanks @saifoo.khan got carried away!

Answer 27

@dpflan . ;)

@ujjwal , where are those "invited users" thing? i can't see it

Answer 28

I saw it here.. see in TEAMWORK.. http://openstudy.com/users/saifoo.khan

Answer 29

Oh yes, that's 2. which seems wrong.

Answer 30

wrong??

Answer 31

Yes, i invited more. @ujjwal

Answer 32

but nvm.

Answer 33

Saifoo, all of those users you've been signing via the link up for your internship now give you points :)

Answer 34

"Invited Users" refers to those whom you've sent invites to and who then actually sign up and use OpenStudy.

Answer 35

I have never seen so many users viewing a single question all at a time with full patience...

Answer 36

and how do we send invitations?

Answer 37

The formula is (still) a secret isn't?

Answer 38

I still don't understand how the score is calculated. It's not the sum of your subscores and it's also not the average...

Answer 39

You can share your questions and SmartScore now, primarily you share these on social media, this sharing can be seen as an invitation. You share by posting to Facebook or Twitter. This is great because you are reaching out to people you know who could help on OpenStudy. This then refers to people whom you invited and who joined OpenStudy.

Answer 40

@FoolForMath Correct, the actual math behind the calculation has not been released, like that secret formula for Coca-Cola.

Answer 41

Notably, we haven't actually migrated old invitations, so right now it's just people who you have invited since we deployed SmartScore. Working on the migration now :)

Answer 42

I like the way you defined: "The score is a measure of your efforts on the site relative to those of your peers"

Answer 43

Virtual (+1) for that.

Answer 44

virtual *high five* @FoolForMath

Answer 45

so we just post this in fb/twitter and if someone joins coz of it our Invited Users will rise?

Answer 46

haha, Cheers mate! :) @dpflan

Answer 47

That's cool.
i just invited someone yesterday as well. ;D

Answer 48

^show off :P

Answer 49

Ohhhhh... so this is what I've been coding on for the past two weeks.... it all makes sense now. ;)

Answer 50

Ok, now the invites are up to date, but there weren't many. Keep in mind that to be counted in this, you have to use one of the share buttons. Either that or you have to include ?inviter=<your user id> on the end of the URL you send. Otherwise we have no way of seeing who sent you.

Answer 51

@Ishaan94: I have never seen KG's problems before. I just solved a couple of them now.

Answer 52

Thanks for pointing it out.

Answer 53

lol @FoolForMath never refuses achallenge =))))

Answer 54

Oh hmm well you're FoolForMath, users like you and zarkon aren't supposed to be included under general observation.

Answer 55

Thanks for looking at my questions :)

Answer 56

I think it would be nice to see the smart score change on the subject that you are logged into. I always looked at the ranking of the person answering questions as a badge of merit of sorts giving more credit to the answers of the higher ranked person. If someone is great at math and achieves all his medals and fans there then it doesn't seem that we should automaticaly assume he is amazing at biology or another subject. I like the idea of the smart score and maybe it will take me some time to get used to it but I would change that small aspect of it if given the chance.

Answer 57

Do you know what's the sad part?

"You took the time to help the asker understand a new concept, and the asker thanked you with a medal."

Yeah, this doesn't happen. It's usually passing users who thank the answerer, not the asker.

Okay, minor nitpick. It's not really important.

Answer 58

We're looking into how we can deal with the subject part of the score. Nothing's been decided yet.

Answer 59

How does "Medals for Asking" if the medal box from questions has been removed?

Answer 60

My smartscore is 60.. But I am probably zero in biology.. However, even in biology group, my smartscore remains the same.. And I think something should be done regarding this.. we are not equally good in all subjects and smartscore doesn't make it distinct.

Answer 61

@ujjwal Your score is not determined by subject specific performance. This score is meant to capture those elements of you as a learner that are present with respect any subject you can study. We are not measuring your specific ability to answer the questions in a given subject, we are trying to establish your skills and talents related to the overlying idea of each category [Problem Solving. Engagement, Teamwork]. For instance, there are many types of problems to solve, mathematics, biology, history, etc, you may prefer to answer questions in certain subjects over those in other subjects, but you are still exhibiting actions and behaviors that display the willingness and aptitude to solve problems. We are trying to remove the subject specificity. You can still see in your profile your top 3 three groups where you have participated, and people can see all the question you have asked and answered and all the medals you've received in your profile too.

Answer 62

But, you're correct in a sense, and @zbay raises a valid point: when there were group specific scores you could actually see whether a user was established as a solid source of help within the given subject. @shadowfiend mentions above how, while we develop this concept of SmartScore, we will look into determining subject specific metrics.

Answer 63

@arcticf0x (or should I say Federer?) With regard to your question: I'll refer you to this question:

http://openstudy.com/study#/updates/4f8df682e4b000310fac6912

Answer 64

@badreferences You mentioned the infrequency of askers awarding answerers. This has been considered, and it represents the ideal scenario: asker and answerer(s) engaged in a common goal of elucidation and understanding of a concept. There many possibilities for the reasoning behind the fact that perhaps you've noticed that askers do not award as frequently -- unaccustomed to the system, time constrained, distracted, etc. But when the dynamics of a question are one where the asker and answerer(s) are fully interacting, have you noticed the same issue you mentioned? In the end, your answer is still being recognized by other users which is awesome. But yes, that is an issue to look into.

Answer 65

What about giving medals to questions?

Answer 66

@nickymarden We currently look at questions you asked where you gave a medal. Expect modifications and improvements to the score that consider your role when you give a medal to someone: you could have: (1.) asked the question, (2.) participated in another person's question, or (3.) you could have just observed another person's question

Answer 67

So the more fans you have the higher your Teamwork?

Answer 68

@dpflan as far as the major-specific scoring goes, why don't you just add an option for users to choose for themselves?

Like "Go to profile and select a couple specialties so that other users know where you excel."

Then, give the avatars a color code or a little icon stamp to identify.

Answer 69

@Carniel As stated above, fans contribute partially to teamwork, and not exclusively.

Answer 70

(P.S. Mods, this thread needs to be sticky)
perhaps create a seperate Help and FAQ group...

Answer 71

@Dyiliq What you suggest is good in order yield a more accurate representation of the user. You can gauge subject competency/interest by looking at the "Most Active Subjects." This only lists 3 subjects so far. This is information is not directly available when that user is helping you with your question in a given subject. As we continue to develop the SmartScore and related concepts, having more fine grained and representative information easily accessible will be crucial.

Thanks for the suggestions/critiques.

Answer 72

@sasogeek I'll refer your point of interest to the following question:

http://openstudy.com/study#/updates/4f8da4ffe4b000310faa6f70

Site performance is taken very seriously as you see @mattfeury @shadowfiend @farmdawgnation discuss in the above question.

Answer 73

- Testimonials
These are written by the people who really think you’re awesome, and they just had to tell you.

messages is for that now! :D

Answer 74

Hi saso, as always I'm disappointed to hear that you're disappointed. ;) As I mentioned in the thread that Dan was kind enough to link to - if we've stopped focusing on performance it's because we're not seeing any issues that jump out at us at the moment. That said, we've progressed leaps and bounds over the past month on the performance front. @shadowfiend and @mattfeury did a lot of good work in isolating and patching issues on the server related to our ability to garbage collect efficiently. Full GCs (the ones that cause everything to pause) have gone from taking ~70+ seconds after 24 hours to sitting at around ~7 seconds after 24 hours. We're not done on that front. That's a huge improvement that I personally have seen reflected on the site.

Additionally, I've been involved in a lot of client-side optimizations. Specifically focused on IE8, but the improvements are visible in other browsers as well. If it's taking you 7 minutes to type something it would sound like your issues are on the client-side realm that I have been optimizing. I'm working on some more changes today.

As always, if you're using Internet Explorer as your browser, we encourage you to use Firefox 11 or Chrome. These browsers provide superior performance for client-side issues over IE. If you are having issues in FF or Chrome, then we can try to diagnose those. I've got some plans for some client side testing code that we can run to remotely diagnose issues like this, but it's not in place yet so we can only see so far into how our code runs on your machine.

Answer 75

@Tomas.A "Messages is that now" what do you mean by that?

Answer 76

I suggest that a site performance focused discussion take place/continue in the question I linked to, so that this question remains on topic related to SmartScore ;)

Discuss:
http://openstudy.com/study#/updates/4f8da4ffe4b000310faa6f70

Answer 77

i said messages is for that, to write to people that they are awesome..

Answer 78

Indeed folks, please keep the performance discussion out of this fine (and informative) question.

@Tomas.A While messages can be used to tell someone they're awesome, fan testimonials announce it to the world, as well.

Answer 79

I appreciate all of the efforts being taken to improve open study. Guys like @Dpflan , @shadowfiend, @farmdawgnation , and @cshalvey and all the other mods/staff who are doing all of the behind the scenes stuff.

Answer 80

That being said, I really see a full clear explanation as to what components go into measuring/calculating qualities such as teamwork and engagement.

Answer 81

Basically, I would like to know precisely how such things are calculated.

Answer 82

@Hero we don't get to know that, it's proprietary.
Especially since it is an incomplete beta feature (am I wrong, @shadowfiend ?)

Answer 83

I said it before, and I say it again: we need to methodically lower our scores to reverse engineer the formula!

:P

Answer 84

I checked my new SmartScore a few days ago and saw that it was 96. After a few days absence because of illness, I logged back in to OpenStudy and now I'm only a 60. ???

Answer 85

@badreferences How can you make that work if the scores are modified on the individual user level by moderators?

Answer 86

@Atam Pleez view the rest of this thread. Scores are beta and subject to change.

Answer 87

@Dyiliq - I read through the 50+ posts and my question was not answered. Obviously if I'm on this thread I understand that SmartScore is in beta. I'm evaluating OpenStudy from a student perspective for possible use by my institution and curious about why my score would drop nearly 40 points after a few days of absence. I think that's a reasonable questions for this thread, and am not sure telling a user to read through 50 posts for an answer is helpful! Just sayin...

I have a conference call with Preetha next week so I will ask my questions then.

Answer 88

@atam it was answered in other thread

they changed formula that's why your score dropped because too much people had to big score (I guess) and it's absolutely not because you were absent, to make you more happy i was 99 and now 77 lol

Answer 89

Thanks @Tomas.A , that makes sense.

Answer 90

So a machine learning algorithm is behind SmartScore?

Answer 91

For now, it's all human learning and statistics ;)

Answer 92

@Atam in particular you saw the very first draft, whose purpose was for us to see if there was a skew to the scores. It turns out, there was a very big one ;)

@Hero We do intend on being a bit clearer once things settle down a bit :)

Answer 93

smartscore sounds like good stuff, a novel way of quantifying individual learners' attributes.

can't wait until its open-sourced :-D