Reflection on the recent Site-Promotion

Question

I think, now that the dust has settled and the winners notified from the recent site-promotion campaign, it would be very useful for the organisers to see our thoughts on it all (and indeed have their say on how they saw it working). Note that this is not a chance to complain about individual users/questions/answers/outcomes, but a chance to have your say on what you thought worked really well, and what if anything you think could be changed or improved in any future site promotions.

Please try to keep your answers constructive. If you have anything negative to say, feel free to say it, but try to also make suggestions as to what could be done in future to alleviate your concerns without simply complaining. Again, let's keep from linking to specific contest content, and talk about the structure.

Answer 1

Rankings, prize assignments, and competitive behavior

I have 1 change that I would like to see incorporated into future competitions, and that is essentially having a way of selecting winners that does not result in users directly competing against each other, rather than themselves. We should be trying to do the best we can achieve, not simply better than next guy. The benchmark should be where your personal pride and motivation takes you, and not be judged against the next users aims.

I think that the original intent was to get people to promote the use of Ask Different to an expanded audience and increase general participation levels, and initially this happened very well, but towards the end of the competition this resulted in the voting percentages dropping hugely (anecdotaly, at least for the Level 3 users) as users realised that complying with the spirit of the competition might have an affect on their chances of winning the iPads. This has a knock on affect that latter Q&As were top heavy on answers, often similar answers, and low on votes which could produce the outlook to the new users we were trying to attract that no-one was providing genuinely good answers, which is obviously contrary to the aims of the site in general, never mind the competition. You can look at pretty much anyones rep curve for the 3 week period and see this marked "Hump" affect as the patterns altered during the period.

In order to prevent this, I have 2 recommendations:

• Provide a greater element of Randomness when selecting winners. There were 10 Level 3 users, and if the rules had been something like "The Top Score wins an iPad for being amazing, everyone else is in a raffle to win the remaining 4" it would have produced odds of 4 in 9 to win an iPad - this is still really really great odds that would inspire people to work hard, and means that there is no difference to coming 2nd or 10th, so long as you worked hard enough to get into that level in the first place. "Tactical" voting would be greatly reduced, as it would not really be possible to affect the result by doing it.

• Either cut down on the ability to follow other users scores, or go the whole hog and provide the whole story in the contest progress pages. Looking at the lists (and knowing that there was a big difference between 5th and 6th) made it very tempting to compare your results against other users and use that as a basis for choosing who to selectively fail to vote for. During the last few hours of the contest it turned out that at least 4 people in the top 10 had written scripts to query the APIs and rank the current contestant scores. Whilst the data is public, better to either stop it happening (possibly by not explicitely starting the exact tie-break mechanism from the start) or level the playing field and prepare them for everyone up front.

Lastly, a big thanks to everyone at SE that helped to organise the thing - We know that @Lauren and @balpha were heavily involved, but I am sure that there are more names behind the doors that also deserve thanks.

Answer 2

Score thresholds and post quality

I love that the metrics to qualify for the higher levels included votes received, votes cast, and shares. Encouraging people to participate across the board is great, and it worked well (especially in the first few weeks).

That said, I think that increasing the vote thresholds for posts would increase quality.

A score of two isn't very high for a post to receive, especially when people are being motivated to vote more than ever. It is entirely conceivable that a low quality or borderline post could receive a score of two. I would love to see a slightly higher requirement, perhaps around 3 or 4. To keep the contest from getting harder, the required number of posts could be made lower, perhaps so that the total number of votes one was required to receive didn't increase.

Higher thresholds for votes received would make sure that low quality posts didn't contribute to someone's contest qualifications, and it would discourage the behavior of posting something not great in the hopes that it somehow gets a score of two.

Lower quantity requirements would also help quality: people simply wouldn't feel motivated to post in as high volume, which would discourage the "Well, I'll just post everything I can think of!" mentality.

Answer 3

This answer is an extension of my thoughts on both of @stuffe's and @NathanGreestein's posts.

Promotion vs Competition

Given the ability to take influence on who gets votes or not, any method of achieving goals by votes is likely to result in users competing against each other.

This is not necessarily bad, it can motivate users to multiply their efforts in providing good answers. This only turns the other way if users restrict themselves from voting. (When does this happen?) Therefore, changing the rules must ultimately take away the contestants ability to have that much influence on who wins the prize. A randomization of winners is a good way out of this dilemma.

Consequences: In order for the randomization of winners to work, there needs to be a set of potential winners that is large enough, otherwise the direct competition between contestants remains. Future contests need to have thresholds for levels that are easy enough to reach. If reaching a level is so difficult such that only about as many contestants reach a level as there are prizes for that level, users will still compete against each other and may try to avoid pushing each other up.

Quality matters

Do "low thresholds" encourage low quality posts? The requirement to achieve thirty-five posts with a score of two did two things: It promoted voting in the beginning of the contest because voting on posts did not directly decrease your own chances of winning - there were simply so many required. However, it also discouraged contestants from putting more effort into their answers for the same reason.

If the minimum requirement to reach a level would require posts with more that two votes, users are pushed to provide answers with greater content. But, at the same time contestants should not be required to achieve too many of such posts because then the threshold to reach a level would be to high and contestants begin to compete against each other.

Summary

If you agree with my arguments, you will see that one needs to (partly) randomize the selection of winners and find the right balance between...

1. ...the minimum requirements to reach a level and
2. ...the maximum requirements to reach a level

If the requirements are low, the randomization of winners works well, voting is generally promoted but more low quality posts emerge as well.

If the requirements are too high, the randomization of winners does not work, voting drops and contestants become rivals.

Answer 4

My answer extends on that of @gentmatt, which in turn extends on the answers from @stuffe and @NathanGreenstein.

I'm impressed with this community, and how we didn't let crazy incentives destroy our sense of community. The promotion wasn't designed as a competition, and it didn't get competitive until people blew the thresholds out of the water to a degree unanticipated in the design of the promotion. I think we did a good job staying non-evil, but, of course, incentives do matter, so perhaps we can tweak them in any future promotions

How could we have done this had we known then what we know now?

In hindsight, I think it turns out that the winning bars should have been higher in terms of quality (more posts with at least 6 or even more votes), perhaps lower in terms of quantity (encouraging a lot of two vote questions/answers maybe isn't a great thing for the site), and shorter in terms of the length of time the promotion runs.

If the winning bar for this one were so high that only one crazy person were likely to be able to make it there, I suspect it wouldn't have been a competition; it would have been a lot of people cheering him on. But then the promotion wouldn't have drawn in as many people.

I think the idea of some fixed prizes and some randomized is a good thing, but I'd like to see a different mix, with a smoother gradient between the deterministic portion (you will win this prize) and the purely stochastic (anyone who meets the minimum threshold has an equal chance). No abrupt cutoffs mean no cutthroat competition to be the one right above, and not below, the cutoff, but improved odds for better performance encourages everyone to do their best.

Perhaps something like

• the high scorer in the promotion wins an iPad.
• for people who complete x shares and vote at least y times, each question or answer with 8 (10?) or more votes gives you ten tickets in a drawing for one of 5 iPads.
• for people who complete w shares and vote at least z times, each question or answer with 4 (5?) or more votes gives you ten tickets in a drawing for one of 25 iPod Touch devices.
• No one can win more than one prize.
• If fewer than 5/25 people meet some minimum threshold, fewer prizes are given out.

Even better if the random drawing algorithm could be sophisticated enough that if you cast upvote #1-8 for a question or answer that ends up reaching 8 votes, it slightly improves your chances of winning yourself (things that would give you ten tickets give you 11 tickets, for instance) -- wouldn't it be great if we could reward constructive edits and constructive voting, given that they're what makes the site work?

Then, there's still an incentive to vote for quality. There's an incentive to post good stuff, and not just anything. And there's a link between merit and success, but some element of chance also.

It's not just luck, but it isn't cutthroat competition, either. Helping along your fellow participants is rewarded, but there's a direct link between posting great things and improving your own chance of winning.

Answer 5

Rule clarifications and tie-breaking

Some clarifications were made to the contest rules in a meta post. Next time around, if these particular rules still apply, they can simply be added to the initial ruleset to reduce confusion.

And a probably not crucial point: the rules (taking into account the later clarifications) were clear about what happened if more people than prizes were in a given level (the top users in a level, by score on contest posts, would get the prizes) — but they weren't clear on tie-breaking among the top users in a level. (During this contest, there was a two-way tie for 5th place in Level 3 — while probably unusual, it would be nice to see this case explicitly covered in the rules.)

Answer 6

Good:

• Goals were achievable
• Prizes were guaranteed for the top 30 contestants
• Prizes were substantial (I'm not going to put in hours of effort for a tshirt, but I'll put in a lot of time for an iPod and the possibility of an iPad)
• I really don't like contests where the odds are determined by the number of entrants, but this had a good balance of difficulty and large number of prizes that it was reasonable.

Improvements:

• It sometimes took more than a day to get various clarifications regarding the contest
• Clarifications that were made occurred on a number of posts, and it appears that those who hung out in chat had somewhat of a better grasp on the contest than those relying solely on the contest rules and meta posts. I don't know that any clarifications were made in chat, but perhaps some were.
• While I applaud simple rules, and while I understand no one believed the tie-breaker would be needed, it was obvious that some were confused as to how exactly the tie-breaker would be computed (specifically which posts counted) and that this confusion persisted until nearly the end of the contest. I had stopped contributing posts because I thought that the first 35 posts that reached 2+ were locked in - so this misunderstanding resulted in me contributing less in order to improve my final contest score, until I finally understood it was the top 35 posts during the contest. Such misunderstandings, if not corrected early in the contest, can work against the point of the contest.
• As an outsider coming in primarily to compete in the contest, I was surprised at how easy it was to compete, and how the contest was not biased towards regular contributors. It doesn't need to overtly balance towards regular contributors, and part of the contest is to bring in new regular users, but perhaps the rules can be set up so that those who "live" here find it easier to compete.
• The mix between random and guaranteed awards was imbalanced. 15% of the prizes went to the drawing, and 85% to the higher levels. For those who knew they could get into the higher levels without difficulty it's not an issue, but if I was at all unsure, then I wouldn't have participated even to reach the lowest level. I think it would have been better to have something like 1 ipad and 5 ipods for the level one drawing.
• I generally agree with the others that each level should be drawn - otherwise you should simply do away with the levels altogether. Once you have a "bucket" where you are either present or not, sub-ranking within the bucket is more complex and leaves the system open to more gaming.
• If you don't want to do a drawing inside each bucket, then consider downplaying the buckets, and using "contest points" instead. Each 2+ post gives you 1 point per upvote, each share gives you 1 point for every 5 people who click on it, each day you spend all 35 votes you get another point, etc. Then everyone can see where they are in the rankings on a daily basis, and know exactly what they need to do in order to be in the top 5 for the ipad, or top 30 for the ipod. You can still have a minimum participation requirement that needs to be fulfilled before you're eligible for each level - the bucket is still there - it's just not emphasized. I'm not a big fan of this as it adds complexity, but it allows contest administrators to finely tune the balance between the different parts of the contest (posts/shares, for instance) and encourages users to continue to contribute beyond the minimum.
• Part of the contest, at the end, depended on one's ability to write a script to determine one's ranking. Either provide that information up front, or use a random draw so the playing field is level on a non-programming site. I could understand this being part of the contest for a programming site, but it doesn't make sense on other sites.

Answer 7

Share links on @StackApple Twitter

IMHO, the @StackApple Twitter account shouldn’t tweet or retweet a contest entrant’s personalised share links while the contest is still running. This is unfair towards other people who participate in the contest.

Here’s an example of such a tweet: http://twitter.com/stackapple/status/185739657777135616 It links to http://goo.gl/q01yJ, which in turn redirects to http://apple.stackexchange.com/q/46077/11610?stw=1, which is the personalised twitter share link for one of the contest participants.

When tweeting during a promotion whomever is in charge of the twitter account simply needs to ensure that they do not use the linking buttons, or remembers to strip the userID off the end of the URL.

Answer 8

What if, for tie-breaker purposes only, votes up or down from other competitors at the same level simply did not count? They count to get you into the level but they do not count as tiebreaker votes.

This would not harm anyone's eligibility for a prize and if there was no tiebreaker would not affect anything. It has a downside in that legitimate and disinterested appreciation or otherwise from peers will not be counted as tiebreaker votes- but like a judge who has interest in an entity in court, recusal may be the lesser evil. It does allow votes to be cast in the out-of-competition sense without effect on the competition.

And it also maintains the suspense in chat next time, as nobody's script can tell just who cast the votes and so nobody knows the final outcome until official results appear >:-)