The Evolving Ecosystem of Crowd Workers

Faced with the asymmetric market place on the crowdsourcing platforms, workers often seek external sources to leverage their experience on Amazon Mechanical Turk (MTurk) and optimize their productivity rates [15]. From forums, where they exchange experiences and knowledge, to developing software extensions with which they can optimize their work, they have created an ecosystem. Only through this ecosystem can workers become super turkers. This ecosystem comprises of: online forums, facebook groups, scripts and extensions, communal sharing and activist platforms.

Scripts are basically computer programs that are developed to automate cumbersome tasks. For supporting the workers to do their crowdsourcing work in a more optimized fashion, many scripts were developed and made available online to be freely used. These scripts can be found in central repositories, which comprise aggregated lists of useful scripts .e.g. greasyfork, indiaturkers, etc. Other repositories comprising manually curated lists of links to scripts can be found in forum threads e.g. Mturkgrind forum has a thread[1] dedicated for scripts that’s manually curated and maintained by the forum user: clickhappier.

These scripts were either created by the academic community or by super-turkers with programming skills. Most scripts require either the preinstallation of Scriptish[2] or Greasemonkey[3] for Firefox or Tapermonkey[4] for Chrome for smoother installations. Next, we introduce a very small sample of those scripts and roughly organize them based on their functionalities.

 

1. Filtering Requesters Scripts:

These scripts are developed to filter the available HITs based on the requesters. For example:

  • Block Requesters script, which filters out HITs from unwanted requesters. Other similar script extensions are available for Firefox[1] and Google Chrome[2].
  • HIT Scraper[3], which colors each HIT based on the Requester’s Turkopticon score (see section IV.D) or can filter out HITs based on various parameters e.g. Masters qualification, minimum pay threshold, worker’s corresponding qualifications, etc.
  • Mmmturkeybacon Color Coded Search with Checkpoints[4], which not only colors HITs based on the Requester’s Turkopticon score (see section IV.D), but also allows you to mark a HIT as completed or viewed so you won’t attempt the same HIT twice.

 

2. Search Optimizing Scripts:

These scripts are intended to speed up the search process. For example:

  •  Auto Pager[5], which automatically loads the next search page as the worker scrolls down.
  • Turk Master[6], which pushes alerts through a worker’s Dashboard when their favorite requesters post HITs. Again, plugins for both Firefox and Chrome are available.
  • mmmturkeybacon Enhanced HIT Information Capsule[7], which converts a worker’s favorite requester’s name into a link, with which the worker can search with to find other HITs posted by the same requester.

 

3. Earnings Monitoring Scripts:

Many scripts are available to manage the earnings a worker is making. For example:

  • Pending Earnings script, which calculates the accumulated amount of pending earning for a particular worker.
  • Today’s Project Earnings script, which calculates the accumulated amount of earnings for the current day.
  • Amazon Payments mTurk Details, which keeps track of the worker’s transactions on MTurk and adds them to the worker’s account activity page for easy monitoring.

 

4. HIT Organizer Scripts:

These enable the user to organize HITs within their different stages: active, pending, rejected, accepted. A few representative examples of such scripts include:

  • Mturk Dashboard HIT status links, which provides the workers with direct links to both rejected and pending HITs.
  • Requester ID & Auto Approval Time, which keeps track of the requesters ID, since requesters can freely change their account names. Moreover, it also displays the requester’s corresponding HIT autoapproval time (how long it will be before the HIT automatically approves) to help the workers in making a more informed decision about working with that particular requester.
  • mmmturkeybacon Queue Order Fix[8], which automatically detects when a worker finishes a HIT and then opens the next HIT with the shortest time left before it expires from their queue for the worker to instantly start working on it.
  • Mturk HIT Database, which keeps track of all the HITs a worker has completed by storing them on a local database hosted on the client side. Accordingly, workers can from their own first-hand experience with the requesters make an informed decision about whether to work for the same requesters again in the future or not.

 

5. Reputation scripts :

These scripts help the workers to keep track of their approval percentage rate as well as the requesters’ reputation.

  • Mturk Worst Case Scenario Calculator, which computes for the worker how many rejections they can receive before their approval rating drops below a significant level.
  • Turkopticon, which allows the user to see the community ratings of requesters on the MTurk search page or quickly rate a requester. See section IV.D for a more detailed description.

 

6. Work Efficiency scripts:

These scripts help the workers do work faster and better, or potentially even automates most of the work completion for them.

  • AutoHotKey, which is a program that automates keystrokes and mouse movements. This allows workers to complete an entire task by solely using their keyboard, saving them hundreds or thousands of mouse clicks which can be damaging to their wrist and arm. It also makes work completion much faster and more accurate as the program always enters the correct key strokes or mouse presses, and can do so with only milliseconds between commands.
  • mTurk - Focus IFrame[9], which focuses the cursor in the iFrame, so as soon as the HIT loads the worker can start typing or hitting the Tab button on their keyboard to get to the correct area of the work iFrame. Without this script the worker may find themselves hitting Tab a hundred times before they can get their cursor into the work iFrame, decreasing their efficiency.
  • ZSMTurker's Larger Radio Buttons and Checkboxes[10], which increases the size of radio buttons and checkboxes so workers can easily click them. This saves them from misclicking or not clicking an option on a survey, which can take up precious time when they submit the page only to be told they missed a question, which they then have to scroll through the page to find and answer again.

 

7. Warning scripts:

Workers need alerts when something has gone wrong, and these scripts provide them such warnings.

  • Mturk Hit Not Accepted[11], which warns the worker that the HIT they are viewing has not been accepted by turning the background pink. Workers may find themselves completing an exceptionally long HIT, but when they attempt to submit it they realize there is no submit button, only an accept button, as they never accepted it in the first place.
  • mmmturkeybacon Logged Out Alert[12], which alerts the worker that they have been logged out. Recently, MTurk began logging out workers every 12 hours, so this notice alerts a worker in case they are running a Search Optimizing script in an effort to find new work.
  • MTurk Captcha Alert[13], which displays an alert when a CAPTCHA, an image the worker has to transcribe to prove they are human, appears on a HIT. If a worker is engaging in “HIT hoarding”, a term that describes the behaviour of grabbing 25 HITs to maximally fill their work queue before they begin working on the HITs one by one, they may reach a CAPTCHA as MTurk displays one after every 35th HIT the worker accepts. If they do not complete it and the page refreshes, or they have multiple tabs open each with the CAPTCHA displaying, their account will be temporarily suspended from the site for five minutes. This script allows the worker to be alerted that a CAPTCHA has appeared so they can stop their other activities and complete it, saving them from a five minute halt.

 

8. Refreshing and Page Monitoring extensions:

In order to get work, one must find their favourite Requesters and HIT groups and refresh the pages that list that work continually. In fact, MTurk has a built in “preview and accept” mode that allows the worker to constantly refresh a HIT group URL and grab a new HIT each time they do. Without automated refreshing tools, a worker would have to manually refresh multiple pages to find new work. If the specific URL of new work isn’t known, a worker can instead use a page monitoring extension to alert them when the Requester’s HIT listing page is updated.

  • Some refreshing extensions require you to keep the page open in a tab, such as Easy Auto Refresh, ChromeReload, AutoRefresh, and ReloadEvery. These are good to use when you are reloading “preview and accept” URLs as they will continue refreshing even if you accept a HIT, meaning you can quickly fill your queue.
  • Other extensions monitor the page in the background, sending you a visual or audio alert when some content on the page changes. For the Requester’s HIT list page this is more appropriate as refreshing the page won’t accept HITs for you. Instead, an extension such as Page Monitor Plus, Distill Web Monitor, Update Scanner, or Check4Change alert to changes on the page in general, which the worker can then visit to see if the new work is of interest to them.

 

9. Layout scripts:

Amazon is in the process of changing the Worker website layout as we write this paper, but previous to that the Worker GUI has not been changed significantly in 10 years, despite workers begging for change[14]. Workers have been forced to create scripts that alter the layout in ways that help them to work more efficiently. We include a large number of scripts here to demonstrate how much the workers find is wrong with the MTurk layout, although it is only a small portion of the total number of scripts available.

  • mmmturkeybacon Butt-In Buttons[15] , which adds many features to the site, including a “preview and accept” button that allows the worker to accept a HIT from the search page, a “return and accept” button so the worker can return this HIT and automatically accept the next one available, and moves all buttons closer to the workspace so the page is more efficiently laid out
  • MTurk Max Job Window Height[16], which makes the work iFrame a more acceptable size in the browser window, giving workers the ability to see all of the HIT at one time.
  • Move Mechanical Turk Timer[17], which puts the work timer at the bottom of the HIT so the worker can see if as they near the bottom of the work space. This timer indicates how long the worker has to complete the HIT, and if it counts down before they are done, the HIT automatically returns to the worker pool to be completed by someone else. The worker cannot reaccess it to use the work they’ve already done to complete it, nor will they be paid for a partially completed HIT, so their time will have been wasted.
  • CH Remaining HITs Replicator[18] , which allows the worker to keep tabs on how many HITs remain in the batch as they make their way through a HIT by placing this number at the bottom of the HIT. This indicates for how long they will continue to be tied to their computer before the work runs out.
  • MTurk Queue Count[19], which displays how many HITs the worker has in their personal work queue at the top of the HIT, allowing them to keep tabs on their progress.
  • ID Copy/Paste[20], which shows the worker their Worker ID on more pages, giving them easy access to cut and paste it into one of the many academic surveys posted to the platform so their data can be tied to their MTurk account.
  • Show wage on HIT submit[21], which indicates the general hourly wage of the HITs currently being worked on based on how fast that individual worker is doing them. MTurk itself provides no information to the worker on how much a HIT will pay hourly, either on the search page or as they are completing the HITs.
  • Keep Mturk Autoaccept Box Checked[22], which automatically checks the AutoAccept Checkbox on every HIT, saving the worker from having to click the “Accept” button on individual HITs. This increases their efficiency, as well as allows them to use a refreshing script to automatically accept HITs without any manual intervention.
     

Authors: Kinda El Maarry, Kristy Milland, Wolf-Tilo Balke