Pavilion

The Pavilion algorithm handles asynchronous arrival/departure of crowd workers from MTurk.com in the waiting and active queue for enabling real-time crowdsourcing (RTC)

Introduction

Pavilion tries to retain workers in the active queue -- a queue which contains workers who are engaged with the task-- in addition to waiting queue based on turnover conditions (e.g. when workers leave the task either from the waiting or active queue). A worker can leave the task either when she submits a task, returns it, or abandons it. Following conditions are handled in Pavilion:

When worker leaves from waiting queue by submitting Human Intelligence Task (HIT) --> Pavilion hires a new worker
When worker leaves from the active queue by submitting HIT --> a) Pavilion moves worker from waiting to active queue, b) Pavilion hires a new worker to fullfil the deficiency in the waiting queue
When worker leaves from the active queue by returning HIT --> Pavilion moves worker from waiting to the active queue
When worker leaves from the waiting queue by returning HIT --> NONE

Main Logic for Pavilion

The main logic for Pavilion is contained in Pavilion\app\Pavilion\events.py file and Pavilion\app\Pavilion\routes.py.

expireHIT event handler expires a running HIT from admin --> EXPIRE HIT button on admin interface

@socketio.on('expireHIT', namespace='/chat')

stop_this_job event handler allows clients (crowd interfaces) to submit HITs automatically --> STOP button on admin interface

@socketio.on('stop_this_job', namespace='/chat')

Connected is the most important event handler.

@socketio.on('connected', namespace='/chat')

Connected event handler does number of things;

Checks whether a worker is a duplicate or new.
adds workers in the waiting queue until they reach to minimum threshold.
moves workers from waiting to active queue when they reach to minimum threshold.

IAmReady event handler is called when a worker is pushed from waiting to active queue. You need to write a logic to store waiting task related bonuses.

@socketio.on('IAmReady', namespace='/chat')

submit_waiting event handler is called when a worker submits HIT while in the waiting queue. It changes the status of worker from waiting to submitted and also posts a new job. You may also need to write your own logic to store bonus related details.

@socketio.on('submit_waiting', namespace='/chat')

submit_active event handler is called when a worker submits HIT while in the active queue. It changes the status of worker from active to submitted and also posts a new job. You may also need to write your own logic to store bonus related details.

@socketio.on('submit_active', namespace='/chat')

Now if you see Pavilion\app\Pavilion\routes.py, you will find this /api/from_mturk event handler.

@Pavilion.route('/api/from_mturk', methods = ['GET', 'POST', 'PUT'])

This uses Amazon Simple Notification System (SNS). For setting up SNS, see following links;

https://blog.mturk.com/tutorial-using-amazon-sns-with-amazon-mturk-a0c6562717cb (FOR NECESSARY STEPS)
https://gist.github.com/iMilnb/bf27da3f38272a76c801 (FOR CONFIRMATION)

SNS will take care of all HITS that were either returned or abandoned and then updates queues accordingly as explained before.

Admin Controls

In addition to managing workers in the active and waiting queue, we have also provided basic admin controls for;

Creating a new project
Creating a HIT/task on Amazon Mechanical Turk (MTurk)
Stopping a current job manually --> this will auto-submit HITs from all workers
Expiring a HIT
Migrating workers manually if needed from waiting to active queue

What is missing?

You need to write your own code for the following;

Handling payments of workers
you need to write task related logic (Pavilion\app\Pavilion --> events.py) and database tables (Pavilion\app\Pavilion --> models.py)

Requirements

See requirements.txt for details about required dependencies.

Local Testing

Uncomment the following code from run.py containing in the root folder. Make sure that create_app contains 'dev' as an argument. For production settings, change this to 'prod'

#--------local testing----------------
if __name__ == "__main__":
    flask_app = create_app('dev')
    with flask_app.app_context():
        db.create_all()
    flask_app.run()
#------------------------------------------

Then simply type: python run.py

To register yourself as Admin: http://127.0.0.1:5000/register
To create a new project: http://127.0.0.1:5000/create_project
See waiting sample page: http://127.0.0.1:5000/waiting_task
See main task sample page: http://127.0.0.1:5000/main_task

Steps for deployment on Heroku

clone our project and set it up in local git repository.
Sign up for Heroku
download and install Heroku CLI
create an app first, give it a unique name
then create a database --> click on resources tab under your newly created app --> search postgres --> and add hobby-dev (it's free)
click on your database name -->click on settings -->credentials --> you can copy the URI to the prod.py file but its NOT RECOMMENDED, you don't need to do anything. Everything is setup in prod.py under config folder.
now come back to console and type: heroku login
go back to app, and click on deploy and then run the following commands(you must be in the root folder of your project)

git add .
git commit -am "make it better"
git push heroku master

you can open the website using heroku open or use provided url after successful installtion

AMAZON S3 storage

login to AWS console (https://aws.amazon.com/console/)
select Amazon S3 from storage
click on create new bucket
click next --> in manage users page, select 'public read acces' becuase Heroku needs to access our Database
when you click on the bucket name, it will lead you to a page where you can add files
upload "Pavilion_DB.backup" --> it is inside the root folder
click on the uploaded file --> copy the link and go to console again
type this: heroku pg:backups:restore "https://s3.eu-central-1.amazonaws.com/mybookcatalogdatabase/local_db.backup"
once done, restart the service using : heroku restart

tahir80/Pavilion