GrajaGrader for java programs     Documentation     Publications     Downloads     Contact

Graja is a JUnit-PMD-Checkstyle-based grader for java programs submitted by students in formative and summative assessment. This site provides publications, software, and information about Graja.

Develop assignments

Develop assignments

We use gradle and eclipse to develop assignments for Graja. A packed assignment that can be used by a professor / lecturer consists of a ProFormA zip file. The zip file must have a specific internal structure specified by the ProFormA format. In order to reach that structure, you may use the sample eclipse project and the gradle build script provided with the distribution (${graja.home}/devasgmt/SampleGrajaAssignments.zip).

The settings needed to create a deliverable assignment that can be used with Graja are specified in a so-called “descriptor” - an xml file with all settings for the assignment.

The following image illustrates that an author or developer of an assignment starts with a descriptor.xml and several implementation files in an eclipse project. The implementation usually consists of a JUnit test driver, a sample solution, a pmd rule file, a checkstyle configuration file, and maybe even more configuration files. When the author has finished his work, he want’s to deliver the assignment to a professor or lecturer, so that he can use the assignment. Therefore the author “builds” the assignment, which creates a ProFormA compatible zip-File, the so called “task.zip” file.

Descriptors

Every assignment has a descriptor.xml file, which defines settings of the assignment. The following picture shows the domain model. The Graja development tools’ build procedure converts descriptors to the ProFormA task.zip format in a completely automatic process.

Examples of descriptor.xml files can be found in the SampleGrajaAssignments example project.

Descriptor

The top level object has a few attributes describing an assignment:

  • policy: an inlined file with a java security policy file. The syntax is not exactly like the Sun Policy file format (see below). Maps to a File in the ProFormA format.
  • variabilityProviderFqn: for an assignment template here you specify the fully qualified class name of the de.hsh.graja.variability.gen.AssignmentVariabilityProvider subclass.

policy

The policy file has up to two sections defining security permissions for the grader code and for the student’s code. An example:

grantsubmission {
  permission java.util.PropertyPermission "write.file", "read";
};
grantgrader {
  permission java.util.PropertyPermission "write.file", "write";
};

The file format is leaned against the default policy file format. Instead of the usual grant entries with signedBy, codeBase and principal, here the only allowed entries are grantsubmission and grantgrader. Inside the curly braces the format is identical to the policy file syntax.

AssignmentSettingsTO

This object is the same as described as part of a FrontendRequestTO / BackendRequestTO object (see here).

There are however differences.

  • As part of a descriptor the AssignmentSettingsTO does not specify a variablityInfo. Instead the descriptor has a variabilityProviderFqn attribute, which references a class, that is used by the build prcoess to create a VariabilityInfoTO object.
  • As part of a descriptor the AssignmentSettingsTO does not specify a description nor an internalDescription. Instead the build process will search for files description.md and gradinghints.md in the same directory as the descriptor.xml and convert these to HTML.

Dependency

A dependency maps directly to an external resource entry in ProformaClassPathRefsTO in the Graja ProFormA task.zip format.

Attachment

An attachment of an assignment can be an instructional text for students, a file with input data, a java archive, …

  • type is one of { library, data, template, instruction }.
  • audience is a comma separated list of values from { student, grader, teacher }.
  • file is a relative or absolute filename in the local file system of the eclipse project

An attachment of type library with audience containing grader will be appended to the classpath on Graja startup. In contrast an attachment of type library with audience=”student” will only be used by students and not by the JVM running Graja. If you have a library that is used by students and also by Graja, then set audience=”student,grader”.

The build procedure will automatically include an additional attachment for “Grader.jar” of type “library” with audience “grader”. It is not necessary to configure that attachment manually in the descriptor.

ModelSolution

A ProFormA formatted task includes one or several so-called model solutions. In the development environment, model solutions are called sample solutions. All sample solutions are expected below the folder samplesolutions. Every subfolder of samplesolutions is considered a separate solution. The subfolder names will be mapped by the automatic build process to the ProFormA’s ModelSolution.id attribute.

Let us consider a single sample solution and it’s folder F. All files in F will be transferred to ProFormA’s File objects. If F contains more than one file, the build process automatically creates a zip file from F’s content. This will also happen, if F contains file(s) in subfolders.

The descriptor.xml optionally can specify additional information about the sample solutions. The attributes of a ModelSolution object are:

  • folder: identifies the folder F of the sample solution
  • comment: an optional comment (plain text) about the sample solution. This maps to ProFormA’s ModelSolution.comment attribute.

FileComment

A sample solution may consist of several files. Each file can get an individual comment. The attributes of a FileComment object are:

  • fileRef: the relative path name of the referenced file. Forward slashes are used as directory separator.
  • comment: an optional comment (plain text) about the file. This maps to ProFormA’s File.comment attribute.

As mentioned above, if a sample solution consists of several files, the automatic build process creates a zip archive for that sample solution. The zip archive is transferred as a ProFormA’s File object into the task.zip. Since in the descriptor.xml there might be individual FileComments for several files of the sample solution, all these FileComments get concatenated (each prefixed with the relative path and a colon) and then transferred to the ProFormA’s File object’s comment attribute.

ModelSolution Example

For example an assignment has the following sample solutions:

fibonacci
+- grader
   +- descriptor.xml
   +- description.md
   +- gradinghints.md
   +- Grader.java
   +- samplesolutions
      +- correct
      |  +- Fibonacci.java
      +- wrong_subfolder
      |  +- fib
      |     +- Fibonacci.java
      +- correct_withothers
         +- Fibonacci.java
         +- util
            +- MyHelper.java

The descriptor.xml contains the following XML fragment:

<modelSolutions>
  <modelSolution folder="correct" comment="Standard solution">
    <fileComment fileref="Fibonacci.java" comment="This is the boring standard solution."/>
  </modelSolution>
  <modelSolution folder="wrong_subfolder" comment="Wrong solution">
    <fileComment fileref="fib/Fibonacci.java" comment="This file should not be in the subfolder 'fib'."/>
  </modelSolution>
  <modelSolution folder="correct_withothers" comment="Correct solution with some functions delegated to a helper class">
    <fileComment fileref="Fibonacci.java" comment="This is the main class required by the assignment's description."/>
    <fileComment fileref="util/MyHelper.java" comment="Helper class"/>
  </modelSolution>
</modelSolutions>

From this the automatic build procedure will create the following ProFormA File and ModelSolution objects:

<p:files>
  <p:file id="id004" 
          filename="Fibonacci.java" 
          comment="Fibonacci.java: This is the boring standard solution." 
          class="internal" 
          type="file">samplesolutions/correct/Fibonacci.java</p:file>
  <p:file id="id005" 
          filename="wrong_subfolder.zip" 
          comment="fib/Fibonacci.java: This file should not be in the subfolder 'fib'." 
          class="internal" 
          type="file">samplesolutions/wrong_subfolder/wrong_subfolder.zip</p:file>
  <p:file id="id006" 
          filename="correct_withothers.zip" 
          comment="Fibonacci.java: This is the main class required by the assignment's description.; util/MyHelper.java: Helper class" 
          class="internal" 
          type="file">samplesolutions/correct_withothers/correct_withothers.zip</p:file>
</p:files>
<p:model-solutions>
  <p:model-solution id="correct" comment="Standard solution">
    <p:filerefs><p:fileref refid="id004"></p:fileref></p:filerefs>
  </p:model-solution>
  <p:model-solution id="wrong_subfolder" comment="Wrong solution">
    <p:filerefs><p:fileref refid="id005"></p:fileref></p:filerefs>
  </p:model-solution>
  <p:model-solution id="correct_withothers" comment="Correct solution with some functions delegated to a helper class">
    <p:filerefs><p:fileref refid="id006"></p:fileref></p:filerefs>
  </p:model-solution>
</p:model-solutions>

As you can see, the first sample solution was created as a java source file, because the solution consists of a single file only. The other sample solutions were transformed into zip archives.

Default descriptor and assignment descriptor

The SampleGrajaAssignments project has one descriptor for every assignment. There is an additional descriptor.default.xml that can contain defaults for all assignments. The default descriptor saves from repeating the same settings in every assignment. The overriding rules are as follows:

  • settings.assignmentMetaData.assignmentVersion: descriptor.xml overrides descriptor.default.xml
  • settings.title: descriptor.xml overrides descriptor.default.xml
  • description.md: default content gets appended to assignment’s content
  • gradinghints.md: is only read from the assignment’s folder and should not be included in the default folder (ignored).
  • policy: descriptor.xml content gets appended to descriptor.default.xml content
  • settings.javaVersion: descriptor.xml overrides descriptor.default.xml
  • settings.naturalLanguage: descriptor.xml overrides descriptor.default.xml
  • settings.assignmentMetaData.targetGroup: descriptor.xml overrides descriptor.default.xml
  • settings.grajaMetaData.grajaVersionCompatibility: descriptor.xml overrides descriptor.default.xml
  • settings.maxSubmissionSize: descriptor.xml overrides descriptor.default.xml
  • dependencies: descriptor.xml content gets appended to descriptor.default.xml content
  • settings.computingResources: child elements present in descriptor.xml override the corresponding child elements in descriptor.default.xml
  • settings.grajaCfg: child elements present in descriptor.xml override the corresponding child elements in descriptor.default.xml
  • settings.submissionRules: child elements present in descriptor.xml override the corresponding child elements in descriptor.default.xml
  • attachments: descriptor.xml content gets appended to descriptor.default.xml content
  • settings.modules: are only read from descriptor.xml and should not be included with descriptor.default.xml (ignored).
  • settings.gradingScheme: are only read from descriptor.xml and should not be included with descriptor.default.xml (ignored).
  • variabilityProviderFqn: is only read from descriptor.xml and should not be included with descriptor.default.xml (ignored).

The SampleGrajaAssignments build process will replace Gradle properties found in SampleGrajaAssignments/gradle.properties and %userhome%/.gradle/gradle.properties in the descriptor files before starting the conversion to the ProFormA task.zip format.

Markdown Files description.md and gradinghints.md

In addition to the descriptor, you may create two markdown files, which can be used to display more information to the students and tutors. These files are named description.md and gradinghints.md, and should be located in the same directory as the descriptor. During the assignment build process the Markdown files will be converted to HTML by Pandoc. You may use Pandoc specific Markdown features like including images and inline LaTeX. If you want to include images, please use the assignments’ attachments/images/ directory as the image location, as this one is checked to decide if the Markdown files should be converted again.

The description.md file should contain a textual explanation of the tasks contained in the assignment. A LMS may display this to the students.

gradinghints.md should contain information, that helps the tutors correct or grade the submitted solutions.

It is also possible to provide a description.default.md. The description.default.md is also converted by the build process. The resulting HTML of the default description is appended to the assignments description. This can be used to generate a common footer below all assignments’ descriptions.

Converter service

Graja provides converters to create requests and to consume results.

Convert a descriptor to a task zip file

The SampleGrajaAssignment project uses Gradle imports to call a sophisticated assignmentBuild.gradle file, that will convert the descriptor and all referenced files to a task.zip format.

A typical assignments project has a one-liner build.gradle file:

apply from: "$grajaHome/devasgmt/assignmentBuild.gradle"

Convert a ProFormA task.zip to a RequestTO

You can convert an assignment in the ProFormA task.zip format to a RequestTO file. The converter uses the JVM’s default locale for submissionNaturalLanguage and makes simplifying assumptions about the requested rdKinds (= {INFO,SEVERE} x {TEACHER,STUDENT} x {html,plain}). The converter can be started by calling the command create_request on the Graja CLI.

Convert result to HTML

  • The xml schema description used inside request.xml and result.xml can be viewed by calling command query_graja_xsd on the Graja CLI.

The result received from CmdlineStarter can be converted to a HTML message by calling the command convert_result_to_html on the Graja CLI.

GUI

Below we will describe the Graja Gui client that is capable of converting descriptor representations to requests and of converting a XML or JSON result to HTML. Both can be accomplished for single and multi requests.

Sample eclipse project

Graja comes with a sample project “SampleGrajaAssignments” that can be used to develop assignment graders. In a single project you can develop many assignments. The project comes with a Gradle builder that builds the deliverables above, namely a task.zip. The sample project helps you to not repeat yourself in that it defines e.g. default settings that hold for all your assignments at a single location. Each assignment in that project gets a unique label - we call it a “token”. The token is the fully qualified package name, where your grader code lives. Each assignment’s sources are organized in a predefined structure, which is documented in detail in a README.txt file in the SampleGrajaAssignments project.

The README.txt file inside the sample project also contains additional information on how to configure eclipse to successfully build the project. The sample project references the local Graja installation, i. e. you must install Graja on the development computer.

Creating an AssignmentGrader subclass

The entry point to your assignment is the class in Grader.java (or a different class name). Here you must implement a subclass of de.hsh.graja.graderapi.AssignmentGrader. The name of your class must be included in the descriptor in the JUnit module configuration’s graderFqn attribute.

In the sample project there are sample assignment graders. A Grader class acts as a JUnit test. It can have several instance methods annotated with @Test. Each test method will be executed by Graja. If a test method fails (org.junit.Assert.fail(…)), the corresponding score (declared in the descriptor.xml) won’t get awarded. The score from the descriptor for a test method is an absolute score, but this might change in the near feature, when the ProFormA format will be extended by a grading scheme using relative weights.

As usual in JUnit tests you may define methods that run @Before and @After any test and also methods that run once before all test (@BeforeClass) and once after all tests (@AfterClass).

When writing your grader, remember that all your Grader’s test methods may run concurrently and every test method gets its own Grader-instance.

A usual setup is as follows:

  • In a @BeforeClass method one or more classes from the submission are loaded and stored in static attributes. You will use methods of the Graja library to lookup the class. If the lookup fails, the respective org.junit.Assert.fail call already was done by the Graja library.
  • In a @Test method you call a method of the submission class again by using the Graja library. Example:

    ReflectionSupport.invokeStatic(submissionClass, String.class, "method", param1, param2, param3);

    This call may fail with a corresponding hint to the student, if the submitted class does not declare a matching method “method”. If the call succeeds, the returned value usually is checked against an expected result. A org.junit.Assert.assertEquals(…) call could be used to check the result.

  • A different path is to compare the behaviour of the submitted classes and a sample solution. The classes of the sample solution must be programmed by you, the assignment grader author. The Graja library provides the class DiffHelper with the possibility to compare the two outputs (e. g. console output) and to throw an AssertionError, when a difference has been noticed. The Graja library produces a message with a detailed diff view of the expected and the observed output. The DiffHelper can be configured to normalize the two outputs before comparison takes place.
  • For more details please have a look at the SampleGrajaAssignments project, where you can find a few example assignments.

The Grader class accepts the following annotations at the method level:

  • @Test specifies, that this is a test method which will be executed to grade the submission
  • @Hint(“text”) can be used to let Graja add the text to the grading comments, when the test method fails. If the annotation is missing, no hint will be appended.

Distribute the grading code among several classes

When your grader code is complex, it can be advantageous to split it into several classes. The descriptor.xml can point to many grader classes in a comma separated list.

Using the checkstyle module

The checkstyle module is based on a checkstyle configuration file as it is described elsewhere. Graja relies on the “metadata” extension feature of the checkstyle configuration. Every check configured in the configuration file can be annotated by these four metadata annotations:

<metadata name="de.hsh.graja.citeLinesBefore" value="..."/>
<metadata name="de.hsh.graja.citeLinesAfter" value="..."/>
<metadata name="de.hsh.graja.replaceTabs" value="..."/>
<metadata name="de.hsh.graja.maxReportsPerFile" value="..."/>

We explain all of these metadata:

  • citeLinesBefore and citeLinesAfter must be integer values >= 0. Graja will cite a fragment of the submitted source, which contains the line of the check violation and a few lines before and after that line. If both values are 0, then only the erroneous line will be cited. Default values are: citeLinesBefore=2, citeLinesAfter=1.
  • replaceTabs: if this is ‘true’ then the checkstyle module will print the source code fragment with tabs replaced by symbols. This is especially useful for checks like Indentation or FileTabCharacter. The default is false.
  • maxReportsPerFile must be an integer >= 1. This is the maximum number of error messages reported by the Checkstyle module per source file. This could be useful for rules like FileTabCharacter, that are likely to produce one error per line. If this attribute is missing, the default value is Integer.MAX_VALUE

An example checkstyle configuration file is contained in the SampleGrajaAssignments project in the “…dateclassv01” assignment.

Testing a Grader

During grader development you will want to test your grader. As test cases you can anticipate one or more solutions of your assignments. You put these test cases in the “samplesolutions” subfolder below your grader’s code. There is a single subfolder below “samplesolutions” for every test case.

Submitting a test case using the GrajaGui

Now, how can you submit the test case to your Grader? For this, Graja comes with a little GUI client that lets you select the test case(s) to run and that shows the results on the screen. This client is the “GrajaGui” and can be started using the Graja CLI. The command is:

java -jar /path/to/GrajaCli.jar --command start_gui

For additional options try

java -jar /path/to/GrajaCli.jar --command start_gui --help

Another way to start the GrajaGui from an Eclipse IDE is to call the launcher “LaunchGui.launch” that is contained in the SampleGrajaAssignments project. See the README.txt file for more information. The client does not need to be configured on the command line. All configuration can be done in the dialog, that looks like this:

The GUI client is capable of producing single and multi requests. On the left of the dialog you can choose the request type (single or multi).

In the first line you select the ProFormA task file. Recently opened task files are available in a drop down.

On the right below “Submission” you can choose a sample solution (Model solution) as the submission. You can run a single sample solution or all at once. Also you can choose a file or folder from disk.

When pushing one of the “Run…” buttons in the lower right, the selected test case(s) get packaged in the background as a submission (possibly zipped) and passed to Graja. The results will pop up in a browser window.

If you choose the request type “Multi”, you can include an assignment that is currently selected in the upper part into the table below “Multi”. In the table every line represents one assignment of the multi request. You may edit the table manually or you rely on the button “Add from above”.

A multi request is built automatically by merging the descriptors of the individual assignments.

Both single and multi requests can be executed directly using one of the “Run …” buttons. Or they can be saved as a request file using the “Save …” buttons. Currently only two file formats (XML and JSON) are supported. A result file from Graja can be loaded using the “Load …” button. The result will be converted to a html document that will pop up after loading has finished.

Result documents

Graja is capable of producing detailed comments dedicated to the teacher or the student or both. The comments are passed back as several separate documents. For example Graja could produce the following four documents:

  audience=TEACHER audience=STUDENT
level=SEVERE a document with error messages only, dedicated to the teacher a document with error messages only, dedicated to the student. Usually this is as subset of the messages that the teachers receives. Sometimes the messages are reformulated in order to be more comprehensible to students.
level=INFO a document with informational messages, dedicated to the teacher. In this document error messages are included, too. a document with informational messages, dedicated to the student. In this document error messages are included, too.

Every document can be delivered by Graja in one of two representations:

  • a html document,
  • a plain text.

Graja can be called with a list of desired documents. E. g. you could instruct Graja in a RequestTO object to deliver SEVERE documents only for students but INFO documents only for teachers. For testing purpose you can control the created documents in detail in the input field “rdKinds” at the lower left of the GUI.

Run in separate JVM

The button entitled “Run backend (…)” will grade the test case inside the same JVM. This makes debugging easier but prevents using the security policy from the assignment’s sources. The buttons “Run … frontend” run Graja in a separate JVM and configure that JVM with the security policy specified in the associated assignments. This makes it possible to test security issues.

Preferences of the GrajaGui

The GUI client stores the last input in a file .grajagui.xml below the user home directory. That way, for ongoing tests of a single assignment you don’t have to enter the same data over and over again.

Graja library

TODO: describe the library

Mocking

TODO: describe MockitoWrap

Execute Graja ←    → API documentation