Bots

Note

Bots are not yet supported in oTree Studio.

You can write “bots” that simulate participants playing your app, so that you can test that it functions properly.

A lot of oTree users skip writing bots because they think it’s complicated or because they are too busy with writing the code for their app. But bots are possibly the easiest part of oTree. For many apps, writing the bot just takes a few minutes; you just need to write one yield statement for each page in the app, like this:

class PlayerBot(Bot):

    def play_round(self):
        yield (pages.Contribute, {'contribution': 10})
        yield (pages.Results)

Note

Prior to January 2018, pages was called views. See oTree 2.0 for more information.

Then, each time you make a change to your app, you can run bots automatically, rather than repetitively clicking through. This will save you much more time than it initially took to write the bot.

Also, you can run dozens of bots simultaneously, to test that your game works properly even under heavy traffic and with different inputs from users, preventing any surprises on the day of the study.

Running tests

Let’s say you want to test the session config named ultimatum in settings.py. To test, open your terminal and run the following command from your project’s root folder:

$ otree test ultimatum

This command will test the session, with the number of participants specified in num_demo_participants in settings.py.

To run tests for all sessions in settings.py, run:

$ otree test

Exporting data

Use the --export flag to export the data generated by the bots to a CSV file:

$ otree test ultimatum --export

This will put the CSV in a folder whose name is autogenerated. To specify the folder name, do:

$ otree test ultimatum --export=myfolder

Writing tests

Submitting pages

Tests are contained in your app’s tests.py. Fill out the play_round() method of your PlayerBot. It should simulate each page submission. For example:

class PlayerBot(Bot):
    def play_round(self):
        yield (pages.Start)
        yield (pages.Offer, {'offer_amount': 50})

Here, we first submit the Start page, which does not contain a form. The next page is Offer, which contains a form whose field is called offer_amount, which we set to 50.

We use yield, because in Python, yield means to produce or generate a value. You could think of the bot as a machine that yields (i.e. generates) submissions.

If a page contains several fields, use a dictionary with multiple items:

yield (pages.Offer, {'first_offer_amount': 50, 'second_offer_amount': 150, 'third_offer_amount': 150})

The test system will raise an error if the bot submits invalid input for a page, or if it submits pages in the wrong order.

Rather than programming many separate bots, you program one bot that can play any variation of the game, using if statements. For example, here is how you can make a bot that can play either as player 1 or player 2.

if self.player.id_in_group == 1:
    yield (pages.Offer, {'offer': 30})
else:
    yield (pages.Accept, {'offer_accepted': True})

Your if statements can depend on self.player, self.group, self.subsession, etc.

You should ignore wait pages when writing bots. Just write a yield for every page that is submitted. After executing each yield statement, the bot will pause until any wait pages are cleared, then it will execute up to (and including) the next yield, and so on.

Asserts

You can use assert statements to ensure that your code is working properly.

For example:

class PlayerBot(Bot):

    def play_round(self):
        assert self.player.money_left == c(10)
        yield (pages.Contribute, {'contribution': c(1)})
        assert self.player.money_left == c(9)
        yield (pages.Results)

In Python, assert statements are used to check statements that should hold true. If the asserted condition is wrong (e.g. self.player.money_left is 11 initially), an error will be raised.

In the above example, we expect that initially, self.player.money_left should be 10, but after the user submits their contribution, money_left will be updated to 9.

The assert statements are executed immediately before submitting the following page. For example, let’s imagine the page_sequence for the game in the above example is [Contribute, ResultsWaitPage, Results]. The bot submits pages.Contribute, is redirected to the wait page, and is then redirected to the Results page. At that point, the Results page is displayed, and then the line assert self.player.money_left == c(9) is executed. If the assert passes, then the user will submit the Results page.

Testing form validation

If you use form validation, you should test that your app is correctly rejecting invalid input from the user, by using SubmissionMustFail().

For example, let’s say you have this page:

class MyPage(Page):

    form_model = 'player'
    form_fields = ['int1', 'int2']

    def error_message(self, values):
        if values["int1"] + values["int2"] != 100:
            return 'The numbers must add up to 100'

You can test that it is working properly with a bot that does this:

from . import pages
from otree.api import Bot, SubmissionMustFail

class PlayerBot(Bot):

    def play_round(self):
        yield SubmissionMustFail(pages.MyPage, {'int1': 0, 'int2': 0})
        yield SubmissionMustFail(pages.MyPage, {'int1': 101, 'int2': 0})
        yield (pages.MyPage, {'int1': 99, 'int2': 1})
        ...

The bot will submit MyPage 3 times. If one of the first 2 submissions passes (i.e. the input is accepted), an error will be raised, because they are marked as containing invalid input. Only the 3rd yield must succeed.

error_fields

When using SubmissionMustFail on forms with multiple fields, you can verify that all fields are rejecting invalid input, using error_fields.

For example, let’s say we a submit a valid age, but an invalid weight and height:

yield SubmissionMustFail(
    pages.Survey,
    {
        'age': 20,
        'weight': -1,
        'height': -1,
    }
)

What’s missing is that the bot system doesn’t tell us exactly why the submission fails. Is it an invalid weight, height, or both? To be sure that both fields are rejecting invalid inputs, add error_fields:

yield SubmissionMustFail(
    pages.Survey,
    {
        'age': 20,
        'weight': -1,
        'height': -1,
    },
    error_fields=['weight', 'height']
)

This will verify that weight and height contained errors, but age did not.

If error_message returns an error, then error_fields will be ['__all__'].

Test cases

You can define an attribute cases on your PlayerBot class that lists different test cases. For example, in a public goods game, you may want to test 3 scenarios:

  • All players contribute half their endowment

  • All players contribute nothing

  • All players contribute their entire endowment (100 points)

We can call these 3 test cases “basic”, “min”, and “max”, respectively, and put them in cases. Then, oTree will execute the bot 3 times, once for each test case. Each time, a different value from cases will be assigned to self.case in the bot, so you can have conditional logic that plays the game differently.

For example:

from . import pages
from otree.api import Bot, SubmissionMustFail


class PlayerBot(Bot):

    cases = ['basic', 'min', 'max']

    def play_round(self):
        yield (pages.Introduction)

        if self.case == 'basic':
            assert self.player.payoff == None

        if self.case == 'basic':
            if self.player.id_in_group == 1:
                for invalid_contribution in [-1, 101]:
                    yield SubmissionMustFail(pages.Contribute, {'contribution': invalid_contribution})
        contribution = {
            'min': 0,
            'max': 100,
            'basic': 50,
        }[self.case]

        yield (pages.Contribute, {"contribution": contribution})
        yield (pages.Results)

        if self.player.id_in_group == 1:

            if self.case == 'min':
                expected_payoff = 110
            elif self.case == 'max':
                expected_payoff = 190
            else:
                expected_payoff = 150
            assert self.player.payoff == expected_payoff

cases needs to be a list, but it can contain any data type, such as strings, integers, or even dictionaries. Here is a trust game bot that uses dictionaries as cases.

from . import pages
from otree.api import Bot, SubmissionMustFail


class PlayerBot(Bot):

    cases = [
        {'offer': 0, 'return': 0, 'p1_payoff': 10, 'p2_payoff': 0},
        {'offer': 5, 'return': 10, 'p1_payoff': 15, 'p2_payoff': 5},
        {'offer': 10, 'return': 30, 'p1_payoff': 30, 'p2_payoff': 0}
    ]

    def play_round(self):
        case = self.case
        if self.player.id_in_group == 1:
            yield (pages.Send, {"sent_amount": case['offer']})

        else:
            for invalid_return in [-1, case['offer'] * Constants.multiplication_factor + 1]:
                yield SubmissionMustFail(pages.SendBack, {'sent_back_amount': invalid_return})
            yield (pages.SendBack, {'sent_back_amount': case['return']})

        yield (pages.Results)


        if self.player.id_in_group == 1:
            expected_payoff = case['p1_payoff']
        else:
            expected_payoff = case['p2_payoff']

        assert self.player.payoff == expected_payoff

Checking the HTML

In the bot, self.html will be a string containing the HTML of the page you are about to submit. So, you can do assert statements to ensure that the HTML does or does not contain some specific substring.

Linebreaks and extra spaces are ignored.

For example, here is a “beauty contest” game bot that ensures that results are reported correctly:

from . import pages
from otree.api import Bot, SubmissionMustFail

class PlayerBot(Bot):

    cases = ['basic', 'tie']

    def play_round(self):
        case = self.case

        # start game
        yield (pages.Introduction)

        if case == 'basic':
            if self.player.id_in_group == 1:
                for invalid_guess in [-1, 101]:
                    yield SubmissionMustFail(pages.Guess, {"guess_value": invalid_guess})
            if self.player.id_in_group == 2:
                guess_value = 9
            else:
                guess_value = 10
        else:
            if self.player.id_in_group in [2, 4]:
                guess_value = 9
            else:
                guess_value = 10

        yield (pages.Guess, {"guess_value": guess_value})

        if case == 'basic':
            if self.player.id_in_group == 2:
                assert self.player.is_winner
                assert 'you were the winner' in self.html
            else:
                assert not self.player.is_winner
                assert 'you were not the winner' in self.html
            expected_winners = 1
        else:
            if self.player.id_in_group in [2, 4]:
                assert self.player.is_winner
                assert 'you were one of them' in self.html
            else:
                assert not self.player.is_winner
                assert 'you were not one of them' in self.html
            expected_winners = 2

        if self.player.id_in_group == 1:
            num_winners = sum([1 for p in self.group.get_players() if p.is_winner])
            assert num_winners == expected_winners
            if num_winners > 1:
                assert self.group.tie == True

        yield (pages.Results)

self.html is updated with the next page’s HTML, after every yield statement.

Automatic HTML checks

Before the bot submits a page, oTree ensures that any form fields the bot is trying to submit are actually found in the page’s HTML, and that there is a submit button on the page. Otherwise, an error will be raised.

However, these checks may not always work, because they are limited to scanning the page’s static HTML on the server side, whereas maybe your page uses JavaScript to dynamically add a form field or submit the form.

In these cases, you should disable the HTML check by using Submission with check_html=False. For example, change this:

class PlayerBot(Bot)
    def play_round(self):
        yield (pages.MyPage, {'foo': 99})

to this:

from otree.api import Submission

class PlayerBot(Bot)
    def play_round(self):
        yield Submission(pages.MyPage, {'foo': 99}, check_html=False)

(If you used Submission without check_html=False, the two code samples would be equivalent.)

If many of your pages incorrectly fail the static HTML checks, you can bypass these checks globally by setting BOTS_CHECK_HTML = False in settings.py.

Testing timeouts

You can simulate a timeout on a page by using Submission with timeout_happened=True:

from otree.api import Submission

class PlayerBot(Bot)
    def play_round(self):
        yield Submission(pages.MyPage, {'foo': 99}, timeout_happened=True)

Browser bots

Bots can run in the browser. They run the same way as command-line bots, by executing the submits in your tests.py.

However, the advantage is that they test the app in a more full and realistic way, because they use a real web browser, rather than the simulated command-line browser. Also, while it’s playing you can briefly see each page and notice if there are visual errors.

Basic use

  • Make sure you have programmed a bot in your tests.py as described above (preferably using yield rather than self.submit).

  • In settings.py, set 'use_browser_bots': True for your session config(s).

  • Run your server and create a session. The pages will auto-play with browser bots, once the start links are opened.

  • If using Heroku, make sure the timeoutworker dyno is enabled.

Command-line browser bots (running locally)

For more automated testing, you can use the otree browser_bots command, which launches browser bots from the command line.

  • Make sure Google Chrome is installed, or set BROWSER_COMMAND in settings.py (more info below).

  • Run your server (e.g. otree devserver)

  • Close all Chrome windows.

  • Run this (substituting the name of your session config):

    otree browser_bots public_goods

This should automatically launch several Chrome tabs, which will play the game very quickly. When finished, the tabs will close, and you will see a report in your terminal window of how long it took.

If Chrome doesn’t close windows properly, make sure you closed all Chrome windows prior to launching the command.

Command-line browser bots on a remote server (e.g. Heroku)

Let’s say you want to test your public_goods session config on a remote server, such as http://lit-bastion-5032.herokuapp.com/. It could be Heroku or any other server.

First, read the instructions above for running the command-line launcher locally.

Deploy your code to the server. Then close all Chrome windows, and then run this command:

otree browser_bots public_goods --server-url=http://lit-bastion-5032.herokuapp.com

(Don’t use heroku run, just execute the command as written above.)

Command-line browser bots: tips & tricks

(If the server is running on a host/port other than the usual http://localhost:8000, you need to pass --server-url as shown above.)

You will get the best performance if you use PostgreSQL or MySQL rather than SQLite, and use runprodserver rather than runserver or devserver.

On my PC, running the default public_goods session with 3 participants takes about 4-5 seconds, and with 9 participants takes about 10 seconds.

Choosing session configs and sizes

You can specify the number of participants:

otree browser_bots ultimatum 6

To test all session configs, just run this:

otree browser_bots

It defaults to num_demo_participants (not num_bots).

Browser bots: misc notes

You can use a browser other than Chrome by setting BROWSER_COMMAND in settings.py. Then, oTree will open the browser by doing something like subprocess.Popen(settings.BROWSER_COMMAND).

(Optional) To make the bots run more quickly, disable most/all add-ons, especially ad-blockers. Or create a fresh Chrome profile that you use just for browser testing. When oTree launches Chrome, it should use the last profile you had open.