Connecting to MechanicalTurk

[alex-birch]

Hi,

How do we connect this to mechanical turk? Is there some missing documentation?

Here's what we've done so far:

1. Created an AWS account
2. Created an AWS user with full turk access only
3. Created environment variables with the AWS_ID and AWS_KEY set
4. Created a fullvideotask in the admin site: /admin/mturk/fullvideotask/

I can see that the Task class in mturk/models.py actually calls publish and creates the HIT with AWS, but I can't see where the publish method is invoked.

Can you please provide some more instuctions?

Thanks for your help.

[clayshieh]

As of right now the task of publishing HIT's is done manually via Django shell. To run django shell, run python manage.py shell, then import fullvideotask(fvt), query the fvt model that you want, and run its publish method. Similarly you can also use the class function, bath_create_and_publish() to create multiple task objects and publish them all in one go. Admin dashboard HIT status/management functionality is in the works but feel free to build one!

[alex-birch]

Thanks for this - that worked. For those people who aren't that familiar with Django shell (like me) you'll need to do this:

python manage.py shell < script.py

where script.py is a script something along the lines of:

from mturk.models import FullVideoTask

tasks = [x for x in FullVideoTask.objects.all() if x.hit_id = ""]

for task in tasks :
    task.publish()

[clayshieh]

@alex-birch take a look at the new branch, mturk. Its a dashboard like functionality for users to publish tasks as HITs

[hocherie]

Hi @clayshieh , where can I find the dashboard to publish HIT tasks?

[antingshen]

There's a dropdown in /admin under Video objects to create the task and publish it in one step. I forget if there's another dashboard that just publishes HITs

[clayshieh]

@hocherie to expand on that, go to /admin, click Videos, select the check boxes next to the videos that you want to publish, and in the action menu drop down select "Publish to Turk" and then press the Go button.

[uzbhutta]

@clayshieh @antingshen when selecting "Publish to Turk", I get the error:

Request Method: | POST
-- | --
http://ec2-<link to server>:5000/admin/annotator/video/
1.10
CommunicationError
XML malformed
./mturk/mturk_api.py in validate, line 315
/home/ubuntu/beaverdam/venv/bin/uwsgi
3.5.2
['.',  '',  '/home/ubuntu/beaverdam/venv/lib/python35.zip',  '/home/ubuntu/beaverdam/venv/lib/python3.5',  '/home/ubuntu/beaverdam/venv/lib/python3.5/plat-x86_64-linux-gnu',  '/home/ubuntu/beaverdam/venv/lib/python3.5/lib-dynload',  '/usr/lib/python3.5',  '/usr/lib/python3.5/plat-x86_64-linux-gnu',  '/home/ubuntu/beaverdam/venv/lib/python3.5/site-packages']
Thu, 1 Nov 2018 13:13:50 -0700

I have set the AWS key and ID on my ubuntu server via:
export AWS_KEY="myiamkey"
export AWS_ID="myiamid"
and have tried this with both sandbox set to true and false

Thanks!

[antingshen]

It's hard to say. There's many reasons why AWS could be rejecting your request and they don't give you a clear error. See cvondrick/vatic#12

[antingshen]

Another note for future users, when BeaverDam was made, boto didn't support mturk in Python 3, so BeaverDam makes custom requests. Now, using boto3 might provide additional functionality and is the official way to connect to mturk.

[uzbhutta]

@antingshen thanks for your note. Before I consider refactoring the code to use boto3 to connect, I have a follow up question:

As aforementioned, when trying to "Publish to Turk", when I have this config:

web client <-> nginx <-> socket <-> uWSGI <-> beaverdam

Along with the DEBUG=True error I showed above about XML malformed, I get the following error in my uwsgi logs when running uwsgi --ini /etc/uwsgi/sites/beaverdam.ini:

'<?xml version="1.0"?>\n<CreateHITResponse><OperationRequest><RequestId>b17faffb-544c-44d4-a946-c0278f627623</RequestId><Errors><Error><Code>AWS.NotAuthorized</Code><Message>The identity contained in the request is not authorized to use this AWSAccessKeyId (1542143607840)</Message></Error></Errors></OperationRequest></CreateHITResponse>'
HTTP Response = b'<?xml version="1.0"?>\n<CreateHITResponse><OperationRequest><RequestId>b17faffb-544c-44d4-a946-c0278f627623</RequestId><Errors><Error><Code>AWS.NotAuthorized</Code><Message>The identity contained in the request is not authorized to use this AWSAccessKeyId (1542143607840)</Message></Error></Errors></OperationRequest></CreateHITResponse>'

However, publishing works fine and the AWS key and ID are accepted when going directly from browser to uwsgi, that is:

web client <-> uWSGI <-> beaverdam

Any help would be great. Thanks.

[antingshen]

Hmm I haven't seen that error before. Did you check what AWS says that error means?