tag:blogger.com,1999:blog-7890764972166411105.post627623158171188845..comments2024-03-29T06:02:41.835+01:00Comments on Nick Brown's blog: Mechanical Turk: Amazon's new charges are not the biggest problemNick Brownhttp://www.blogger.com/profile/00172030184497186082noreply@blogger.comBlogger12125tag:blogger.com,1999:blog-7890764972166411105.post-88924279494133717102023-05-06T13:25:15.623+02:002023-05-06T13:25:15.623+02:00The situation with Mturk has worsened since the in...The situation with Mturk has worsened since the increase in the requester commission percentage from 20% to 40%. As a result, many requesters have left, leaving only a few who have no other choice. It would be beneficial for Mturk to consider reducing the requester commission percentage to 10%, particularly for those requesters who provide thousands and even millions of hits.Rama Raohttps://www.blogger.com/profile/17561532910723024526noreply@blogger.comtag:blogger.com,1999:blog-7890764972166411105.post-426445360233421822019-07-26T09:03:55.951+02:002019-07-26T09:03:55.951+02:00This comment has been removed by the author.Rama Raohttps://www.blogger.com/profile/17561532910723024526noreply@blogger.comtag:blogger.com,1999:blog-7890764972166411105.post-22642221155884433912017-09-18T14:24:02.188+02:002017-09-18T14:24:02.188+02:00Thanks for your information, it was really very he...Thanks for your information, it was really very helpfull..<br /><a href="https://www.atlretail.com/" rel="nofollow">Amazon</a><br />Anonymoushttps://www.blogger.com/profile/16361753546104244498noreply@blogger.comtag:blogger.com,1999:blog-7890764972166411105.post-13110400223230857542015-06-27T00:08:41.113+02:002015-06-27T00:08:41.113+02:00Shooting the messenger doesn't change what wor...Shooting the messenger doesn't change what workers report in their TurkOpticon reviews, nor does it do anything to move a conversation forward. <br /><br />Amazon raised its commission structure. Researchers believe it's an attempt to force them out of the marketplace. The 55 requesters who have committed to conduct ethical research (http://wiki.wearedynamo.org/index.php/Guidelines_for_Academic_Requesters) with Amazon Mechanical Turk are a drop in the bucket.<br /><br />Again, TurkOpticon tells the story. Attacking me doesn't change that.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-7890764972166411105.post-44201427436197638602015-06-26T19:59:25.732+02:002015-06-26T19:59:25.732+02:00The Anon who states . "The requesters who wil...The Anon who states . "The requesters who will pay the highest commissions are the worst abusers of workers and the Mechanical Turk platform." is in dream land. <br /><br />Look at the bottom of the list of 55 requesters who have signed this document supporting ethical treatment of study participants. The only reason why this person sees garbage is because they are a "casual turker" and they do not know how to turk properly.<br />Good fair paying work is taken quickly. Any casual turker will never see it and will only see the work that is left sitting because good non-casual turkers will not touch it.<br /><br />Go away Anon, you are hurting the cause with your ignorance. Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-7890764972166411105.post-65324488002272082392015-06-26T03:14:30.494+02:002015-06-26T03:14:30.494+02:00This is probably the most rational discussion on t...This is probably the most rational discussion on this topic I've seen. Like Anthony, I am also a (casual) Amazon Mechanical Turk worker (~5 years, Masters). When Amazon confirmed the rate changes, I did a little happy dance. The requesters who will pay the highest commissions are the worst abusers of workers and the Mechanical Turk platform.<br /><br />TurkOpticon (https://turkopticon.ucsd.edu/reports) is the only collective place workers have to rate and discuss requesters. At any given moment, TurkOpticon reviews paint a picture of the hideous treatment of workers by researchers: broken HITs (with no compensation), unpaid screeners, missing completion codes, phishing, woefully underpaid, deliberately misleading completion times, horrific setup, disingenuous rejections, failure to communicate with workers, etc. <br /><br />I can't imagine why any worker would take to the internet to defend their abusers.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-7890764972166411105.post-53625927392917969802015-06-24T05:10:06.669+02:002015-06-24T05:10:06.669+02:00Amazon doesn't care about Mturk from a revenue...Amazon doesn't care about Mturk from a revenue and profit perspective. That's why they've made basically no updates in years. The price increase was probably to justify the few resources they provide to it, or to get nuisance usage down.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-7890764972166411105.post-9880130135321310432015-06-23T22:38:52.295+02:002015-06-23T22:38:52.295+02:00A lot of the concerns about MTurk need to be conte...A lot of the concerns about MTurk need to be contextualized within the larger issue of using nonprobability samples or other sources of data. <br /><br />The quality of MTurk data in terms of attentiveness and test-retest reliability is quite good. In fact, consistency across self-reported demographics is equal to or higher than that observed on the General Social Survey or other sources of self-report data. This suggests that if you want to know where your workers are coming from, you can probably just ask them. <br /><br />Effects are typically consistent across MTurk and other samples. For example, see <a href="javascript:void(0);" rel="nofollow">here</a>. Not to say they are inevitably consistent, but the available evidence says this is often the case. <br /><br />I would be careful about how you frame the technical problems related to non-independence of data. All available evidence suggests that this attenuates true effects and can be offset through increased sample size. Repeated participation is a huge issue in all online panels. For a discussion see <a href="https://books.google.com/books?hl=en&lr=&id=RymGgxN3zD4C&oi=fnd&pg=PA110&dq=Online+Marketing+Research+miller&ots=a2oXFtY8pc&sig=ftmt1oCaCRu9GWJEersiyRnFZ4Q#v=onepage&q=Online%20Marketing%20Research%20miller&f=false" rel="nofollow">here</a>. Also, the paper you link to notes there are many technical solutions within the platform to limit duplicate responses across studies. <br /><br />I don't think anyone seriously makes the claim that MTurk workers are a representative sample, but I think you can make a good argument that they are about as representative as many of the samples already used within the field (e.g., student samples, snowball convenience samples. random digit dialing of the 55% of the population that still has a landline etc.). Truly representative samples are simply not attainable for anyone without a substantial grant. <br /><br />I can't speak as to why everyone else is irritated by this, but I feel that Amazon has been less than transparent about its motives. It claims this is to increase revenue for product development, but that is clearly false, because it is targeting the price increases at what it claims is a negligible part of its business. I would not be surprised if the hike was targeted at the segment of the requester population that is responsible for putting upward pressure on worker wages (through IRB requirements and an effort by individual researchers to be somewhat less mercenary). Academics have fixed budgets, and so the net effect of this will be to depress the payment rates they offer to workers.Jessehttp://www.jessechandler.comnoreply@blogger.comtag:blogger.com,1999:blog-7890764972166411105.post-65480346608842462062015-06-23T19:53:31.770+02:002015-06-23T19:53:31.770+02:00Oh yes, I don't deny there is bias and skewed ...Oh yes, I don't deny there is bias and skewed results. In fact, I'm sure of it. These surveys are completed as fast as humanly possible with almost no thought to the content. Most of them are done while doing something like watching TV. There are attention checks and similar manipulations inside the surveys, but seasoned Turkers don't have a problem with these.<br /><br />I'd venture to say that the results from almost all of these surveys are practically useless. Not to mention that they are almost all the same. I answer the same questions many times a day from different requesters doing psychological surveys. I have my answers on muscle memory at this point.Anonymoushttps://www.blogger.com/profile/06593693373498850775noreply@blogger.comtag:blogger.com,1999:blog-7890764972166411105.post-27584367685505047302015-06-23T19:45:01.573+02:002015-06-23T19:45:01.573+02:00In the case of the study that took 20 minutes, the...In the case of the study that took 20 minutes, the write-up explicitly states that 20 minutes was the amount of time that it actually took, not the advertised time.<br /><br />But I'm sure that "seasoned Turkers" can do way better than the planned time, and in stating that, you kind of make my point for me. MTurk is kind of the ultimate example of instant-gut-feeling self-report. Sure, there's a time limit to complete a questionnaire in a lab setting, but there's not usually any benefit to going quicker because the next part of the experiment is coming up in 20 minutes, so answering all 50 questions in 10 minutes doesn't gain you anything. Contrast that with the online situation where you have a rational motive to fill in the form as quickly as possible. This is already going to be a source of bias in much psychological research, even if it's mostly irrelevant to non-academic surveys where you have to identify pictures of dogs versus cats or whatever.Nick Brownhttps://www.blogger.com/profile/18266307287741345798noreply@blogger.comtag:blogger.com,1999:blog-7890764972166411105.post-35983652849770561612015-06-23T18:47:04.855+02:002015-06-23T18:47:04.855+02:00Sorry to double post, but there is one other point...Sorry to double post, but there is one other point I'd like to make. Just because a study gives 20 minutes as it's expected completion time, or gives that long before the HIT expires doesn't mean that is how long it will take. Seasoned Turkers regularly complete surveys in as little as 25% or less of the estimated completion time as given by the requester. <br /><br />If you browse HWTF on Reddit when someone posts a HIT for a survey, they always post how long it took them to complete, and it's always MUCH less than what the requester estimated it would take. Anonymoushttps://www.blogger.com/profile/06593693373498850775noreply@blogger.comtag:blogger.com,1999:blog-7890764972166411105.post-61221281393692816542015-06-23T18:44:37.130+02:002015-06-23T18:44:37.130+02:00When you talk about workers working for sub minimu...When you talk about workers working for sub minimum wage rates, you don't accurately take into account all the reasons one might want to do such a thing. You state they are probably "desperately poor". I'm here to tell you that that is most likely NOT the case, and I speak from experience. <br /><br />I work full time as a mid level manager at a nation retail company. I make 45,000 per year. It's not a lot, but it's I'm not desperately poor, but any means. My wife brings in as much as I do, so together we make close to 100,000 per year. <br /><br />I don't work on MTurk because I'm desperately poor, or because I just can't get enough of academic research. I work on MTurk because of the convenience that it provides. The fact I can work from home any hour of any day, for as short or as long a time as I want. No commute. No customer interaction. Can do it in my pajamas. There is a trade off here most people don't recognize. People are willing to work for less than they are "worth" because of the ease of use and convenience factor. <br /><br />Another thing most people don't realize is academic surveys are a small part of most Turker's income. At least if you're a smart Turker. Batches are where it's at. Plenty of startups and other companies post batches that pay anywhere from $10-20 per hour if you are skilled at completing them. You don't have to work for sub minimum wage unless you want to. And the key word is "want". Nobody is being exploited here. Take it from someone who knows. Personally.Anonymoushttps://www.blogger.com/profile/06593693373498850775noreply@blogger.com