Question

I keep thinking that there should be a more graded form of giving recognition.

Answer 1

Medals are nice. It's great to be able to acknowledge the help that someone has given you.
However, it's quite rough that it's currently a binary thing. Medal or no medal. Here are some problems with it.
No way to reward someone MORE for helping you MORE. A two word answer receives the same amount of recognition as 30 minutes of careful explanation.
No way to reward multiple people for helping you.
It encourages giving MANY answers to receive MANY medals, rather than giving GOOD answers to receive more recognition.
Since the asker can only give one medal, it creates a situation where answerers are racing to answer first, rather than taking their time and hoping to give a quality explanation.

Answer 2

I realize there's not necessarily a simple solution to this situation, but I think it's obvious that some form of graded recognition would be very helpful. I have a couple of thoughts about this.
One idea would be simply a gold-silver-bronze system. I don't actually love this idea, but it's better than a single medal.
Another idea would be to allow people to give multiple medals for a single question, including the option to give multiple medals to a single person.

Answer 3

The bottom line is that somehow the system needs to put more value on quality of explanation than on quantity of answers.

Answer 4

Medals are just one component of your score. Other factors, like the amount of time you spend working with someone and the engagement with other people in the reply are also measured. Also, not all medals are weighted equally: medals from people just passing through are not as important as medals from people who worked in the problem to, a medal from the person who asked, or medals from highly ranked users who are experts in their subject.

Answer 5

Well, I'm glad to hear that. That's legit.
BUT, my concern actually has very little to do with smart score at all. I don't care about smart score or how it is calculated except insofar as it affects the behavior of the people on this site. I think that desire to raise one's smart score probably tends to motivate people to spend more time on the site (lgba is case in point), but I don't think that the average user's actual behavior is motivated by raising smart score, especially since they don't know how it is calculated.

No, the decisions people make are much more affected by the individual medals they receive. Each time someone receives a medal, it is positive feedback that tends to motivate them to repeat the behavior that earned them that medal.

So, I'm glad that the smart score takes those things into account, but in terms of actually encouraging better behavior, I don't think smart score is doing the job that graded medals could.

Answer 6

The more I think about it, the more I like the idea of a gold, silver, bronze medal system. Give people a limited daily allotment of gold medals and silver medals so that it FEELS like a big deal to give or receive one.
As for smart score, I don't think it matters that much how these medals affect how it is calculated, for the reasons I explained in my last post. They really could work exactly the same as the current medals, and I think it would still make a big difference in peoples' behavior.

Answer 7

What prevents people from always giving a gold medal?

Answer 8

I'll go ahead and quote myself.

"Give people a limited daily allotment of gold medals and silver medals"

So you can't give more than 3 gold medals per day and 5 silver medals per day, for example.

Answer 9

My primary purpose in biology right now is to go through the answered questions, make sure the answers are in fact right and not copied and pasted from other sources, correct the wrong ones and award medals as appropriate. Capping the number of medals I could give would severely inhibit my ability to do this...

Answer 10

Bronze would be unlimited. -_-

Answer 11

Such feeble nay-saying.

Answer 12

But I am certainly not going to give bronze medals to people who go to pains to tutor other people and work them through the concepts. And we have quite a few of those. Nor do I have time to read all fifty or so answered questions, keep track of who answered each one well and apportion out a shallow cache of gold and silvers, or go hunting back through my medal awarded questions and rescind gold/silvers, replace them with bronzes, and reallocate later on. Not feeble nay-saying - practical time constraints on a busy person.

Answer 13

Two points.
1) You're not the average user of this site. The site has to be designed for its users, not for those running it. Your complaint is unique to you and a few other mods.

2)
"But I am certainly not going to give bronze medals to people who go to pains to tutor other people and work them through the concepts."
There are two problems with this argument. The first is that you are taking a stance that, on the surface, appears to be defending those who work hard to give good answers. However, in the current system, those people are slighted by being rewarded in the same way that someone who types 2 characters and hits "post" would be rewarded. So, in actuality, you are arguing against what is good for those people. The second problem is that you talk about awarding bronze medals as if it's a punishment. Medals of any kind are a reward, and will be viewed that way. However, some answers are better than others and deserve a better reward. I think that you and your users are capable of discerning which answers those are.

Answer 14

THis is the third time someone has criticized/suggested changes to the medal system. 0.o

Answer 15

Um... it's definitely much more than the 3rd. People complain about the medal system regularly.

Answer 16

I'm just offering my suggestion for a fix of what I consider to be the salient problem with it in its current form.

Answer 17

I'm an advocate for simply changing best answer to best reply. It's the easiest change, and will see difference.

Answer 18

Some difference, perhaps.

Answer 19

Yes, but it is such an easy change!

Answer 20

i also find it disturbing that people who just answers questions directly have higher scores than those who give quality answers (in other words long explanations)...teamwork should be evaluated more in the score rather than problem solving because the mission/vision of this site is to *help* people...i find no sense in giving priority to the ability of a person to see good problems and be able to answer them rather than the people who actually *helps*

i for one used to encourage people to give quality answers and to explain very well as it is what is in the code of conduct and the mission/vision of the site but i have seen a lot of users rise to the top by just answering questions and seeing these makes me sad as it seems quality answers are not appreciated since people can rise through the ranks without worrying about their teamwork. i honestly think something should be done about this or else other people who still give quality answers will just lose motive to explain and will just answer directly.

if the gold-silver-bronze medal is the solution to this dilemma then i am a yes for it. however, i would also like to suggest that it be harder to reach 95 for the answer givers..i suggest that starting from 95 teamwork and engagement should be prioritized rather than the problem solving because the score should favor the people who are interested to learn and teach rather than the people who just come here to test their abilities by answering questions...i think something like this would be a better idea rather than classifying smart people with the smart teachers..an alternative i thought of may be there's a certain range of the score where engagement will be prioritized; a range where problem solving will be prioritized; and a range where teamwork is prioritized (i.e. 0 -60 engagement is prioritized; 61 - 90 problem solving; 91 - 99 teamwork; 100 mixed) a downside is just that it's gonna get harder but just like they say...the harder a gauge is the more credible it is

Answer 21

There are regular suggestions for how to improve the medal system. The only ones that will truly eliminate problems would rely on content analysis of some sort (be it content length or what have you).

The problem with the suggestion of applying scarcity to the issue (and we've run into it with the current one-medal-per-question approach, as well) is that scarcity is only a good solution if people use the scarce resource correctly. In this case, you're suggesting you can only give three gold medals. The likely result is that most users will give gold medals to the first 3 users who give them the answer, followed by 5 silvers to the next 5 who give them an answer, followed by bronzes for everyone who is unlucky enough to answer them after they've asked their first 8 questions for the day. It isn't a solution because the problem isn't the value of the medals, it's the value that people place on “just answers” vs actual walkthroughs.

You're right that blues isn't most users, but neither are you. We've decided to apply moderation as a sort of human content analysis which, in collaboration with a community that reports abuse on just-answer-style-questions, tries to give more credit to the better answerers. This is why medals from high-ranking members and moderators are worth more than medals from “regular” users: we trust their judgement more when it comes to this.

While we're not done experimenting with the medal system yet, and several of the other OpenStudy team members agree with a tiered medal system like what you propose, I personally haven't yet seen one that actually convinces me it will be in any way tamper-proof to the issues we've seen without simply working on the community's concept of what is a good answer and reinforcing that somehow.

What the next iteration of the rewards system ends up looking like will, of course, ultimately be a consensus decision inside the team.

Answer 22

First of all, thank you for a well thought out answer.

I appreciate the effort your team puts into this site, even when I disagree with the decisions you make. I think I tend to state things a little harshly, but please know that I recognize that you have a tough job and that these aren't easy decisions to make. You're doing a great job with a difficult task.

That said, here are some considerations that I think should weigh heavily into any changes to the medal system that you make in the future.
At root, I think that the focus should be on what the PURPOSE of the medal system and the smartscore is intended to be. This has to be your foundation for any interface you set up. My vision for that purpose could be quite different from yours, and I expect there are at least minor differences in opinion for what that purpose is among the team members. So let me outline what I think the purpose should be and should not be.

I believe that the smart score and medal system should have the primary purpose of being a sort of "carrot" promoting good behavior. We have mods as a stick to deal with particularly egregious offenders, but on a large scale, an incentive system will be more effective to promote the kind of behavior you want. Important to note is that whatever works to encourage the kind of behavior you want on the site is what you should implement.

I view this stated purpose as different in a nuanced but critical way from another view that is much more common, and seems to be the purpose that Lgba, blues, and you subscribe to. The view that seems to be most commonly held about the purpose of the medal/ smartscore system is that it is primarily there as a measuring stick of a user's standing and contributions. User's who contribute a lot should be REWARDED with lots of medals and a high smart score. And it is an INJUSTICE if certain users have a lower medal count or smart score than other users. I believe that this view is not entirely wrong, but it is missing the point somewhat. The system should not be primarily for ranking and comparing users, but for turning the site into what you want it to be by promoting certain behaviors. It is great if those more contributing users end up with a higher smartscore etc, but I do not view it as essential or an injustice.

Answer 23

If you spend your efforts trying to tweak the system so that it is a fair measure of contribution or method of comparing users, then I think your efforts are not entirely wasted, but a little bit misguided.

For one thing, any system will be exploitable. We're seeing this system be exploited, and the next system will be exploited too, at least to some extent. It's something that you just can't worry too much about. Sure, your users are going to care a lot about their smart score, but as a moderating team, I think that shouldn't be high on your list of what to care about.

The results that actually matter most are how behavior is affected. If you see an increase in quality explanation and a decrease in quick answering, then that is success, regardless of what happens with smart score trends or any perceived injustice in how the system ranks people.

Answer 24

Aaand, I apologize for my apparent inability to state things concisely.

Answer 25

that is why i suggest that the problem solving be not focused much because the criteria for it leans towards giving answers

Answer 26

but that is only my idea...i do not speak for anybody else

Answer 27

Well, Lgba, I'm fine with that proposition, but it's still only a suggestion for how to balance smart scores and rank users. It's a behind the scenes tweak not affecting the interface or the day to day experience of users, so I see it having only a minor affect on actual behavior.

Which is the main point I'm trying to make. Encouraging learning-facilitating behavior should be the priority over balancing smart-scores and ranking users.

Answer 28

hw do u guys manage write so long paragraphs :P

Answer 29

I actually agree with your stated purpose of the SmartScore—I think properly reflecting that metric and rewarding desired behavior are inextricably linked. But the thing is, we're relying on the community, an imperfect community, to use our rewards mechanisms to furnish the actual rewards. So the community is one of the important variables we can and are trying to tweak above and beyond the medal system itself.

Answer 30

Aravind, usually I have trouble writing for length, but if you ask me a good enough question, then I find myself struggling to keep it short. I think this is a very good question.

Shadowfiend, they are certainly linked. I definitely think that the distinction is an important one, however. Your point about the community is such a great one. If the community is one of learning, offering good help, and seeking it out, then as people join in, I think they tend to naturally fit in and mimic the behavior they are seeing.

Answer 31

lol @SmoothMath be impressed ;D I just read the whole thing. *like a boss*

Answer 32

So boss-esque.

Answer 33

I have to agree with shadowfiend about the likely results of a graded medal system. The average user of this site will very likely give out their gold and silver medals first, then bronze for everyone else, without much care or concern for gradations between answers. Remember, one of the major problems that we currently have is that askers will give their medals to whomever gives them the answer first. That type of behavior will not be changed by tiered medals.

There are really two things at issue here. One is the medal system, the other is the SmartScore system. We have been discussing the distinction between rewards and metrics; I think it is clear that to at least some extent, the medals are the rewards, and the SmartScore is the metric (though many view the score as a reward as well). Both systems can be improved, but I think we will benefit from keeping the distinction between their purposes in mind when trying to think of improvements.

I've been thinking about the idea of downvotes for quite a while. This is obviously not a new concept, and we've seen it implemented in a variety of ways (StackExchange, Reddit, others). Here at OpenStudy, we have a very different type of community and a very different purpose than those other sites, but the more I think about it, the more I think a downvote functionality could be beneficial. It would be useful in the obvious way (downvoting answers that are blatantly incorrect), but I think it would be far more useful as a method for communicating the OS culture. The perfect example would be downvoting answer sniping. We suffer quite a bit from the 'eternal September' phenomenon here on OS, in that we have a constant influx of new users who are not familiar with the culture or rules or code of conduct, and who probably won't stick around long enough to find out about them. Communicating culture is always difficult. I believe a downvote system (probably set up such that you only gain the ability to downvote after a certain smartscore?) would help with that task, and would help address the answer sniping problem.

While I'm typing an essay, I might as well address the metric issue as well. I see the value of a one-number SmartScore that summarizes a user's status in the community. This is a good thing. That said, I think that we need more, and I don't think that the Problem Solving / Teamwork / Engagement subscores address the need. I think we need a way to see, to some extent, where a user is at in their own progress learning a subject. Again, I am coming from a biased perspective of only using the math study group. That said, in math, we have some very different types of users. We have Parth as a great example of the helpful teacher who gives good answers to relatively lower-level problems, but who is unable to help with more advanced concepts (that's not to say he isn't ridiculously advanced for his age). On the other end of the spectrum, we have a few people who are very advanced in their knowledge, and who tend to answer more questions that are past a certain level, but fewer questions overall (I think eliassaab might be a good example here). I claim that both of these types of users should be encouraged, because they both benefit the community, but it is clear that the distinction between them ought to be recognized as well. It is worth pointing out that currently, the Parth-type user benefits more from both the reward and metric systems, in large part due to the quantity of questions they can answer.

The difference between these types of users does not fit into the current scoring system, and it is also very hard to measure. Shadowfiend mentions content analysis, and I think this is clearly one of our biggest hurdles. I think there is not just one solution, but I think that we will benefit by recognizing one of our biggest assets: the community and culture that we already have, and the users who help define that culture. I know that we already adjust the value of medals based on the smartscore of the person awarding them, and I think that is a very good idea. I think we can take that idea further.

In retrospect, I probably should have just made this my own post, but I suppose here is as good a place as any to put these thoughts. Thanks again to the moderators for continually working to improve the site.

Answer 34

Great points, Nbouscal.
I think that downvoting/upvoting is a pretty proven system, and I'd like to see it implemented. I'm curious to know whether management would consider it. I can't speak for them, but my impression has always been that they are super careful to cater to a really young audience and also to make everything as positive as possible. I believe they would reject this idea out of fear of how users would react to being downvoted.

Answer 35

We actually have considered the impact of downvoting several times, and SmoothMath is correct that our fear has always been that we're trying to keep the site as positive as possible. That said, there's definitely a community guidance aspect to downvotes, and it's been particularly successful in places like Hacker News. We don't have a final decision on downvotes, just the decision we've held to until now, and we are definitely willing to hear criticism on that point.

Answer 36

That was my initial concern as well, and why I haven't suggested downvotes until this point. Thinking about it more, though, I do think the benefits outweigh any detriments. I think this is especially true considering the significant amount of research that we have seen in recent years regarding the effects of the "positivity regime" that we've been under since the early 70s. This article ( http://nymag.com/news/features/27840/ ) does a decent job of summarizing what I'm referring to; I'm sure many of you are familiar with the idea. Positivity is a tool, and like many tools it can be helpful or harmful depending on its use.

Ultimately, the effects of a downvote system depend heavily on the implementation. I think that there is an implementation that could work for this site. I have not yet been able to think of an alternative that would be as effective at curtailing the answer sniping problem or at communicating OS culture. I think there is a way we can present it such that it doesn't look as negative as the term downvote implies, and is really more of a suggestive/corrective action that leads to the user being more informed about the CoC and the culture here at OS.

Answer 37

Essay wriring contest!!!!

Sorry i couldnt resist i just had to say it

Answer 38

i *think* the original system should be brought back along with smartscore....maybe we can take the problem solving away from smartscore (because it doesnt really measure interactivity) and instead measure problem solving using the old levels eh @shadowfiend ?

Answer 39

lol @lgbasallote I thought the same thing xD

Answer 40

Well, I was thinking that downvoting wouldn't decrease one's smartscore (if you got 10 downvotes and 5 upvotes or a score of negative 5, it wouldn't do anything to your score) - I do realize that there is still negativity associated with being able to downvote an answer, and maybe we could solve the problem by limiting downvotes to the amount of time you were on OS (say you get a downvote every hour)

Answer 41

I agree that downvotes, if implemented, should not decrease your smartscore. If anything, the most they should do is counteract any medals received for the same answer. I don't necessarily like the idea of limiting the number of available downvotes; I prefer only giving the ability to downvote anyone to people with a high enough smartscore.

Answer 42

nbouscal, that's a good idea, but I think it would be better to do both. Remember those people with a high smart score that still abuse OS (answer fiend it)? We should limit downvotes in some sort of way to keep out trolls, and maybe limit upvotes if really necessary.

Answer 43

I think people with a high smartscore that abuse OS should not have a high smartscore, and that should be addressed on the metric side.

Answer 44

Turning this into stackoverflow here, are we? I want mod when I get 99! :3

Answer 45

That's actually very interesting. The thought that the most downvotes could do is counteract medal earnings for this question, but never decrease your SmartScore beyond that.

Answer 46

i was thinking maybe downvoting can be a fancy way of telling people not to answer snipe :S that could be good to..everytime you try to downvote a person..it will be like a report abuse..you can add a description as to why you downvoted and then the person will get notified and read the description so he'll change yay!

Answer 47

I think downvoting is no bueno. I think that it would be abused and most members would just whine and complain about it. I'm also in a weird mood and sleep deprivation doesn't. Perhaps, I am not the one to give advice at this moment. I don't know though, it could be much more entertaining. xD

Answer 48

There is always a concern that a downvote system will be abused, and it is good to keep that concern in mind when developing the specifics of the system. There are various measures that can be taken to prevent abuse. The ones I have in mind so far are limiting which users are able to downvote, and limiting the impact that a downvote has. There may be other methods that would be useful as well. For example, on SE a user decreases their own rep (very slightly) by downvoting another user. I don't think that method would work here, because SmartScore is in such a black box. Other ideas might be useful, though.

In the end, the question comes down to whether the benefits outweigh the detriments or not. Personally, I find it likely that they would, but obviously have no way to know for certain.

I would also like to mention that I think the metric issue that I raised in my first reply is actually more important than the downvote idea, which has been the focus of all the subsequent discussion.

Answer 49

Here's an idea for a tweak to the downvoting system that might help people feel a little less disrespected, and at the same time help them know how to not be downvoted in the future.

Instead of just a pure "Thumb down" button, there could be a downvote button with a menu attached. To downvote someone, you must select from a finite list of reasons that you are downvoting them. On this list would be things that are just generally not beneficial to the learning community. Then, depending on the downvote option chosen, the user would get a notification like
"Your post has been downvoted because..."
"Someone thinks that the answer you've given needs more explanation."
"Someone thinks that you are seeking only answers and ignoring responses that give good explanation."
"Someone thinks that you are being disrespectful of the other users of this site."

I think that with most people, after they receive such a message once or twice, they are going to try and modify their behavior so that they don't receive it again. And simply having the option to report these kinds of behaviors makes it obvious to users that they are looked down upon.

The system would also give you options for dealing with those people who do not change their behavior. You could quickly identify answer givers simply by looking at who has a high percentage of their posts reported for answer giving. It's up to you if you'd want to do something about it, but you could suspend them, limit their site privileges somehow, or put them on probation.

Answer 50

so in a way @SmoothMath agrees with my suggestion that downvote be a fancy way of telling people not to answer snipe or something

Answer 51

right?

Answer 52

Right. Of course, downvoting is always a way to communicate to someone that you don't approve of their post. I think the system I'm suggesting has the advantage of communicating WHY.

The situation I think the owners of this site want to avoid and should want to avoid goes something like this. A user joins up, starts to post, and gets downvoted to oblivion. Not knowing much about the site, they don't know why they are being downvoted, and they feel like it's just because they are new and not liked. =( They whine, feel bad, and don't know how to stop getting downvoted, so they just leave.

Theoretically, I think giving a reason attached would result in this kind of situation instead: A new user joins up, gets downvoted a couple of times, and immediately starts to understand what kind of posting is valued and accepted in this community based on the feedback they're receiving. They alter their behavior slightly, and feel a sense of pride at receiving medals instead of downvotes =D They love this feeling! They stay FOREVER! RAINBOWS AND COOKIES AND UNICORNS ABOUND IN THIS LAND OF OVERFLOWING JOY.

Answer 53

but what if the old users get downvoted? some old users (not saying there are) are primadonnas..when they get downvoted they'll probably make a scene and whine about it...ideas to prevent this?

Answer 54

Maybe
"Someone thinks you should suck it up, you big baby."
should be another downvote option.

Answer 55

How would you guys feel about a tiered medal system that you only have full access to after achieving a certain SmartScore? For example, you can only award gold medals after reaching SmartScore 60 or something like that.

Answer 56

I like that idea @farmdawgnation It makes it fair for everyone involved.

Answer 57

if that's how it's going to be please make it so answer fiends cant reach 60 easily

Answer 58

Oh, yeah that.

Answer 59

Do keep in mind as we suggest this that implementing such a system would likely cause most smart scores to shift downward.

Answer 60

Users like satellite won't be affected (because he has an insane raw score) but many users would see their score go down if we implement this.

Answer 61

I've decided a well implemented downvoting system might be more effective, but I would definitely like an unlockable tiered medal system more than the current system. And I still think if we implement a tiered system, there should be a limit on number of higher tier medals.
I also agree with lgba. Asking a lot a lot of questions and not doing much else shouldn't get you to 60 very quickly.

Answer 62

insane is an understatement and you know it @farmdawgnation

Answer 63

Another reminder is that SmartScore, like everything else on this site, is an experiment. We're constantly measuring metrics on how it's used and whatnot. And, FWIW, I agree that we need to start weighting some things differently.

Answer 64

You can't make a system with no possible abuses. But maybe you can give some moderation tools to high ranking users or sensible users (the ones typing essays on this thread) to limit abuse. Moderation tools not moderator status. OS community isn't gigantic. I am sure you can give moderation tools to some sensible users manually. Every down-vote could notify the users with access to moderation tools or the moderators online. If the down-vote isn't legit the moderators or the users with moderation tools can undo the down-vote (and punish the down-voter as well lol). Maybe not every down-vote but if some user down-votes another users for 3rd time or 2nd time or if some posts gets down-voted more than once or twice.