The process of creating a problem at Algotester

1. In the top right corner of the site, open the tab «Problems (Writer)». In this section you will see all problems you have created, as well as problems whose author has designated you as a contributor. Also, you will see 5 buttons: «Create», «Delete», «Reorder», «Rejudge» and «Submit».

2. Press the button «Create». You will see a problem creation tab. It is necessary to specify 4 mandatory fields for creating a problem: «Name», «Time (sec.)», «Memory (MiB)» and «Output (MiB)». Enter the name of the problem in the «Name» field. Name requirements:

   • Avoid using any special characters in problem names ($&*%)

   • Must not be too long (7 words or fewer)

   • The problem name must be unique. There should be no problem with the same name in the archive.

   Note that the problem statement will have the name that you specify when creating the statement. In the time (sec.) field, enter the maximum time in seconds for one test to run. Standard time limit is 2 seconds. The time limit should be up to 7 seconds.

   In the "Memory (MiB)" field, enter the RAM usage limit for one test. The standard memory limit is 256 MB. The memory limit should be from 128 MB to 1024 MB. The memory limit must be a power of two.

   If the time or memory limits are less than standard, then make sure that the problem is solvable in Python and Java.

   In the "Output (MiB)" field, enter the maximum output size of the program on one test. The default output value is 1 Mebibyte.

   Limits to the problem should be chosen so that they are at least 2 times greater than the author’s solution requires.

   All other fields can be filled in later.

   After creating a problem, you will return to the «Problems (Writer)» tab. Your newly created problems will be there. By clicking on the name of the problem, you will go to the settings page.

3. In the problem settings page, you will see 9 tabs:

   • In the home tab (is denoted by a hut) you can change time and memory constraints, change problem name, choose sources and other. After writing the sources, you need to select them in this tab. The checker for the problem can be selected immediately.

   • In the tab «Contributors» you can add contributors to the problem. You can choose a contributor and assign him a role that he will perform during the preparation of this problem.

   • In the «Statements» tab, you can create statements in Ukrainian and English. Read more about creating statements in the «Statements» section.

   • In the «Resources» tab, you can add resources to the problem, for example, pictures to the statement.

   • In the «Sources» tab, you can create generators, validators, checkers, scorers, and solutions. You can read more about each type in the corresponding clause of the next section.

   • In the «Tests» tab, you can create manual tests, create tests using a generator, or add test archives. When creating tests, you need to specify their name, cost and the tests themselves (in the case of creating manual tests), or the parameters of the generator. If you want to make this test as sample, which will be displayed in the statement, you need to check the «sample» checkbox. Read more about creating a test generator in the generator section. Read more about the cost of tests in the «Scorer» section.

   • In the «Tags» tab, you can create tags to this problem.

   • In the «Result» tab, after submiting the problem, you can view the tests you created and the result of your solution.

   • In the «User solutions» tab, you can view all solutions of other users.

   To the right of the tabs, there will be 2 buttons: «Update» and «Submit».

Stages of problem creation

1. Statement

   The first step in creating a problem is to create a statement. The statement is a mandatory part of the problem. In the «Statement» tab, you will see three buttons: «Create», «Delete» and «Submit». To create a statement, click on the «Create» button.

   The statement is a very important part of the problem, where you must clearly explain the essence of the problem. Statements on Algotester are written in the LaTeX data markup language. When creating a statement, you will see 6 tabs:

   • In the home tab (indicated by a hut) you can choose the language and change the name of the problem.

   • In the «Description» tab, you must describe the essence of the problem, tell what is given, explain what the user needs to do to solve it.

   • In the «Input» tab, you must describe what data the user will receive and in what order and format it is given.

   • In the «Output» tab, you should describe what response you expect from the user and in what format he should output it.

   • In the «Constraints» tab, you must give constraints on all incoming data. If in the statement you guaranteed some special constraints, for example, the absence of loops or multi-edges in the graph, they should be mentioned here as well.

   • In the «Notes» tab, you can explain the samples, write an important note that may not be obvious in the description, or specify details that will help the user understand what exactly he needs to do.

   Follow our guidelines for writing terms and statements. After writing the statement, you have to wait until system will generate it. You can find examples of statements in the archive.

2. Solution

   The next step is to write the solution to the problem. Problem solution is the source code that the system will compile and run on the test set, to check the answer with the answers of the participants. The problem should have one main solution. The solution must read the input data in the format described in the statement. You can specify the main solution for the problem on the home tab (indicated by a hut). We recommend that each problem also have other solutions written by testers to avoid mistakes in the main solution.

3. Algotester libraries

   Read our article about Algotester library. You can find examples of its usage in this article in next sections.

4. Validator

   The next mandatory component of the problem is the validator. A validator is a program that reads a single test and verifies that it conforms to the format and constraints specified in the statement. Use our library for writing validators for problems. Validator reads input data from the standard input stream (STDIN), and outputs to the standard output stream (STDOUT). If it finds an incorrect test, it should output a short text explanation of an error and return 0. A test is considered correct, if validator did not output anything and returned 0. Validator must check not only constraints on the size of input data, array elements, but also graph connectivity, absence of loops in it, or any other guaranteed conditions and constraints. There must be one main validator in the problem. You need to specify the main validator for the problem in the home tab (indicated by a hut). Here are some examples of validators: validator for the A plus B problem, validator for the Marichka and cookies problem, validator for the Mykola, Vasyl and a maze problem.

5. Checker

   The last mandatory component for creating problems is the checker. Checker is a program that checks the correctness of the answer. If the answer to the problem is unambiguous, then Token Checker exists for this. If you need to check the answer in real numbers, use Double Checker with the precision you need. If there is no single answer to the problem, then you need to write a checker yourself. Use our library for writing checkers. Use the «initChecker» function. It returns an array of three elements: the test data reader, the user output reader, and the author solution output reader. If the user’s answer is incorrect, then the checker must output an error message to STDOUT and return 0. The answer is considered correct if the checker did not output anything and returned 0. The checker accepts 4 parameters: paths to files with input data, user output, author output, user code. There must be one main checker in the problem. You can specify the main checker for the problem in the home tab (indicated by a hut). Here are some examples of checkers: Token Checker, Double Checker 1e-4, checker for the Doubled Sum of Squares problem, checker for the Sweet Candies problem.

6. Tests

   At this stage, your problem already has all the mandatory components. You can go back to the home tab (indicated by a hut), select all the main sources for the problem and submit it. If you did everything correctly, the problem will receive the status «Ready».

   Despite the fact that the problem is ready, there are no tests in it yet. Tests are sets of input data on which user solutions will be tested. There are three ways to add a test to a problem:

   • Create a manual test. A manual test is a test that is created manually. Such tests can be used as samples that will be displayed in the statement.

   • Write a test generator (more details in the next paragraph).

   • Add archived tests. To add archived tests, you need to send an archive in zip format, and specify the extension of the input and output files. All input and output files must have extensions that match the ones you specified. Files with input and output data that relate to the same test must have the same name.

7. Generator

   The generator is a program that allows you to automate the test generation process. Use our library when writing generators for problems. Parameters can be passed to the generator, for example, the maximum size of the test or the constraints of the problem parameters. The first parameter (with index 1) is always the test number. You must specify the following parameters yourself. As the last parameter, you can add hash. This is a randomly generated string. It needs to be added so that it is not possible to recreate the tests even with the generator code. The pseudorandom number generator is initialized from all parameters. The generator must output the result to the standard output stream(STDOUT).

   Here are some examples of generators: generator for the A plus B problem, generator for the Marichka and cookies problem.

   One problem can have several generators that generate different types of input data. Here is one example and another example generators for the Maksym, Petro and Bohdan problem. The first generator forms a random connected graph, and the second a star of a given size. Since these are 2 different types of graphs, more than 1 generator had to be used.

8. Submitting a problem

   After writing all the main sources and adding tests, the problem has to be submitted. To submit a problem, click on the «Submit» button located to the right of the problem setting tabs. After submitting the problem, the following will happen:

   • Tests will be created. The generator will generate tests, and together with manual and archived tests, a set of tests will be formed.

   • The validator will check the tests for correctness.

   • The solution will run on a set of tests, and the author’s answers will be generated.

   If an error occurs at any of these stages, the problem will receive the status Has Errors, and in the «Result» tab you will see what exactly went wrong. Otherwise, the problem will have a status of Ready. If the problem has the status Ready, then you will be able to see the tests in the «Result» tab.

9. Interactive checker

   An interactive checker is a checker for interactive problems. Use the «initInteractiveChecker» function. It returns an array of three elements: a test data reader, a user input stream (you need to write input to this stream), and a user output reader (read the user’s responses from it). The generator and validator for interactive problems must generate and validate only the input data for the checker, but not the solutions to the problem. Problem solutions interact only with the interactive checker. Note that the interactive checker is responsible not only for checking the correctness of the solution, but also for providing the user with input. Here is an example of an interactive problem. Checker for the Interactive Number problem. Examples of a validator and a generator to it: validator for the problem and generator for the problem.

10. Scorers and cost of tests

    If you also want to take into account partially correct solutions for the problem, then you can score points for them. Then the user will be able to score a certain part of the points for the problem, if he did not come up with a fast enough or absolutely correct solution. The easiest way to do this is to assign a certain value to each group of tests. The user will receive points for a group of tests only if his solution is correct for all tests from the group.

    You can also get a certain number of points for a problem using scorer. Before receiving points for a solution, it is checked by a checker. The scorer scores only those solutions that have passed the checker. The scorer accepts 6 parameters, which are file paths:

    • Input file.

    • User output.

    • Author output.

    • Folder with processed code files (STDIN, STDOUT, STDERR files, user executable).

    • Folder for writing. This folder is for internal use by the scorer. It will be cleared between tests.

    • User code.

    The scorer must output the non-negative number of points that the user will receive for the test to the standard output stream. Here is an example of a scorer: scorer for the Zenyk and sorting problem.

11. Tests execution logs

    To see the result of the solution on one specific test, you need:

    • In the «Sources» tab, for the appropriate solution, click on «View», located in the rightmost «Actions» column.

    • Go to the «Result» tab and see the logs of the corresponding test.

    Note that the very large tests in this tab are incomplete, but you can download the ZIP archive (in the upper right corner) with the complete tests.