Execute Graja

The Graja CLI

Graja provides a central command line interface (CLI) for all functions available in Graja. The functions reach from the usual processing of a grading request to developer oriented commands like creating a ProFormA zip file or querying an xml schema.

All commands are accessible by a command line of the following form:

java --module-path /path/to/Graja;/path/to/Graja/extjar --module de.hsh.graja/de.hsh.graja.Cli --command <command> [options]

The CLI provides logging to the console, when you specify a log level like this:

java -Dde.hsh.graja.cli.logging.level=<level> --module-path /path/to/Graja;/path/to/Graja/extjar --module de.hsh.graja/de.hsh.graja.Cli --command <command> [options]

The <level> controls the console output when running the command. Supported <level>s are ALL, DEBUG, INFO, WARN, ERROR, OFF. The default is ERROR.

The following will list all available <command>s:

java --module-path /path/to/Graja;/path/to/Graja/extjar --module de.hsh.graja/de.hsh.graja.Cli --help

The list of commands varies depending on additional modules, that can be added on the command line:

java --module-path /path/to/Graja;/path/to/Graja/extjar;/path/to/Graja/devasgmt --add-modules de.hsh.graja.devasgmt --module de.hsh.graja/de.hsh.graja.Cli --help

java --module-path /path/to/Graja;/path/to/Graja/extjar;/path/to/javafx/sdk/lib --add-modules de.hsh.graja.gui --module de.hsh.graja/de.hsh.graja.Cli --help

Command line service

The main grader can be called on the command line like follows:

java --module-path /path/to/Graja;/path/to/Graja/extjar --module de.hsh.graja/de.hsh.graja.Cli --command grade_request --format json --request /path/to/request.json --result /path/to/result.json 

The command line starter accepts three parameters:

a file format (currently json and xml are accepted)
a path to a file containing a grading request
a path to a file, where graja should store the result.

A single Graja run processes a single submission. A submission may contain solutions to one or more assignments (which depends on the contents of the request file, see below). If you want to process more than one submission (each solving one ore more assignments), you need to start a separate Graja instance for every submission.

If the request file requests Graja to grade several assignments for a single submission in one go, we call this a “multi request”. In that case, Graja will collect the individual results for every assignment and write the collected results (a “multi result”) to the specified output file.

A ProFormA 2.1 compatible way to start the main grader is:

java --module-path /path/to/Graja;/path/to/Graja/extjar --module de.hsh.graja/de.hsh.graja.Cli --command submit_proforma --submission /path/to/submission.zip --response /path/to/response.xml

Please note that as submission.zip Graja does not expect the student’s solution but instead a file in the ProFormA submission format. The returned response file is in ProFormA 2.1 response format.

Currently the response file is always an xml file. A zip response file is not supported, but might be in the near future. Also the submission file is always a zip file. A xml submission file is not supported, which also might change in the near future.

API service

Besides the main command line interface, there is another service class: de.hsh.graja.starter.apcli.WithinJVMStarter. This class serves a RequestTO object and returns a ResultTO object (see below). This class is used by libraries like the GrajaPlugin inside Grappa in order to start Graja conveniently from a running java program.

What makes up an assignment?

As stated above, a request consists of a submission and a specification of one or several assignments to be graded. A single assignment is perferably stored in a so-called ProFormA-task file. A task is a configuration of one or more tools or grading processes together with a grading scheme. Since Graja is using the ProFormA standard format for tasks, it has to adapt to that standard.

ProFormA terms

The ProFormA format defines (besides many more) two central areas:

tests: a Test object defines a check by specifying a title, a test-type and an id. Also a Test object contains a more or less complex TestConfiguration object, which can contain Graja-specific data. The ProFormA standard lists several test-types, e. g. compilation, unittest, style checker.
grading-hints: A GradingHints object defines scores and a hierarchy of grading categories and grading aspects.

Graja terms

Modules

Graja executes various tools as backend “modules”. Currently Graja uses the modules Compiler, JUnit, PMD, and Checkstyle. Also it provides a “Human” module, that defines the execution of a manual grading process.

Each module execution produces results. A module execution can be split into partial executions. E. g. a JUnit run can be seen as an execution of many individual JUnit test methods. Every test method is a partial test or a “submodule”. Every test method produces a score and a comment for the submitting student. In the sense of the two central ProFormA areas, a JUnit test method execution is a part of a “Test” (title, technical configuration) and it also is a “grading aspect” (score, category).

Currently the number of modules per type is limited in Graja as follows:

Compile: max. 1
JUnit: max. 1
Human: max. 1
PMD: unlimited
Checkstyle: 1

Grading categories and aspects

The grading scheme in Graja is a hierarchy of grading nodes at various levels: a “grading root” node is at the top, “grading categories” are at the intermediate levels, and “grading aspects” represent the leaves. A “grading category” aggregates several grading aspects or grading categories below a common headline. Common headlines are “Functional correctness” or “Maintainability”. Examples for “grading aspects” are: “The method x should calculate y”. Or “The class Z should get a leading comment”. Each leaf of the hierarchy refers to a corresponding ProFormA-Test object. That way any grading aspect directly refers to its associated submodule execution like a single JUnit test method, a single PMD rule, a single Checkstyle check, or a single grading aspect that is reviewed manually by humans.

The nodes of the Graja hierarchy directly correspond to nodes of the ProFormA grading-hints hierarchy. Inner nodes of that hierarchy correspond to grading categories, leaves of the hierarchy correspond to grading aspects.

Example

The following image shows an example Graja output for a task, that defines seven grading categories subsumed under a root category “Overall result”. Each category has got an individual number of grading aspects. The aspects are fed behind the scenes from different modules. E. g. the aspects below “Functional correctness” are calculated by a JUnit test, while the aspectes below “Coding style” stem from a Checkstyle execution.

Realization of Graja modules and grading categories/aspects as ProFormA tests and grading-hints

The following image shows, how Graja maps it’s concepts (modules, submodules, grading categories, grading aspects) to the ProFormA concepts (tests and grading-hints):

Legend:

At the top of the image we see the grading scheme, that is stored below the ProFormA grading-hints section as node objects. On the left the image shows the ProFormA Test objects.

Graja modules are realized as ProFormA Test objects (orange nodes Compile, JUnit, PMD, Checkstyle [not in the image], Human). As can be seen, some modules might be specified more than once, because it might be advantageous to run the same tool (PMD in the above example) twice with different parameterization, in order to produce results for different grading aspects. E. g. PMD rule 2 has been executed twice. In the second PMD run, that rule was evaluated again (PMD rule 2b), but with a different parameterization. The first evaluation (PMD rule 2) contributes to the grading aspect C.2 while the second evaluation (PMD rule 2b) contributes to a different aspect D.4.

We can also see, that some submodules (brown nodes on the left) are referenced by more than one grading aspect (dark green nodes at the top). The JUnit method 2 is evaluated only once during a single JUnit execution. Its results contribute to two grading aspects C.1 and E.1, possibly with different score weights.

The grading-hints hierarchy is rooted at a single root node (light green “Root” at the top), which recursively contains one or more category nodes or aspect nodes.

The Test objects on the left are not hierarchically arranged in the ProFormA format. Hence, the orange nodes are listed in no specific order in the task.xml file.

Request and Result

Graja expects requests and produces results. In this section we will describe requests and results in detail.

Domain model

Here we describe a domain model of requests accepted by Graja and results produced by Graja. For now we abstract from the representation of domain objects (XML or JSON).

Request

Graja can serve requests of different types. Most of these types are legacy formats. The preferred way to start Graja is by using the ProFormA assignment format (also called the task.zip format).

The ProFormA task.zip format

The task.zip format is described in detail on the ProFormA GitHub repository site and in this journal paper. Graja 1.8 and 1.9 support the ProFormA format version 1.1. Graja 1.7 supports ProFormA format version 1.0.1. Graja 2.0 and 2.1 support the ProFormA format 2.0.1. Graja 2.2 and 2.3 support the ProFormA format 2.1.

A task.zip file contains the Proforma exchange format as a file named “task.xml” together with several attachments.

Task

This is the root object of the proforma specification. It describes the contents of the task.xml inside the task.zip file.

A Task object has the following attributes:

uuid: a unique identifier. Currently, when using the Graja development tools, every newly built task.zip gets a new random uuid.
parentUuid: not used by Graja
lang: natural language. Default is “en”, which can be overridden by an assignment developer in the assignment.properties file.
title: The title of the assignment as seen by students.
description: This is HTML content with the assignment’s text seen by students.
internalDescription: This is HTML content with additional assignment’s text seen by teachers only.

Proglang

The value simple content should be “java”. The version attribute describes the assignment’s compatibility with Java versions. The assignment is considered compatible with any Java version equal to or higher than the specified attribute value.

SubmissionRestrictionsType

Currently Graja expects zero or more FileRestrType objects in this object.

Attributes of a SubmissionRestrictionsType object are:

maxSize: specifies the maximum size of a submission. Maps to AssignmentSettingsTO.maxSubmissionSizeBytes.
description: This optional attribute can be used for additional information in order to explain the restrictions to students.
internalDescription: These descriptions are for teachers only.

FileRestrType

Attributes of this object are:

value: defines an allowed path (relative to the root of the submission zip) or an allowed filename. The format (literally specified or regular expression) depends on the attribute patternFormat.
patternFormat: can be “none” or “posix-ere” (see ProFormA whitepaper).
use: Elements with use=”required” must be contained in a submission. Otherwise the Graja starter will not start the backend. Elements with use=”optional” may be contained in a submission. If the submission has even more files, these are skipped by the Graja starter, but the rest of the submission will be graded. Elements with use=”prohibited” are not allowed in a submission. Such a submission will be rejected by Graja before starting the backend.

MetaData

Inside the MetaData object an AssignmentMetaDataTO object has meta information in a Graja specific namespace. For an explanation of AssignmentMetaDataTO see below.

Further there could be a CVVp object inside the MetaData object for a materialized task (explanation see below).

AssignmentMetaDataTO

Attributes of this object are:

assignmentId: an identifier for the assignment
grajaVersionCompatibility: the minimum Graja version needed for the assignment. If missing, no graja compatibility check will be performed.

Files

Is a collection of files (TaskFileType).

TaskFileType

A file’s content can either be embedded in the xml or it can be attached as a separate file. Also a file can have binary or text content.

A file has the following attributes:

id: an identifier
mimeType: not used by Graja.
usedByGrader: if this is true, the respective file will be added to the classpath of the Graja backend JVM
visible: specifies (yes, no, delayed) whether the file can be seen by students. Since this is managed by the LMD, Graja won’t use this attribute.
usageByLms: not used by Graja
internalDescription: currently not used by Graja
fileChoiceGroup: selects between four file types.

Graja usually expects:

a file with the filename “policy” of the EmbeddedTxtFileType with visible=no and usedByGrader=true. This is the security policy file, which contains “grantgrader” and/or “grantsubmission” sections.
a file with the filename “javatypesel” of the EmbeddedTxtFileType with visible=no and usedByGrader=true. This is a file defining restrictions for the Junit, PMD and Checkstyle modules when loading classes, interfaces and enums as part from otherwise unrestricted file paths. The file contains key value pairs with the keys “include-package” and “exclude-type”.
one or more files of the AttachedBinFileType with visible=no and usedByGrader=true. These files can be referenced as the compile classpath or the JUnit classpath. Usually these are a jar file hosting the JUnit tests and maybe additional jar files.
one or more files of the AttachedTxtFileType with visible=no and usedByGrader=true. These files can contain configuration for PMD and Checkstyle modules.
one or more files of one of the Attached… types with usedByGrader=false and visible=no. These files are referenced by ModelSolutions because they hold sample solutions of the task.

The classpath and the external resources below together must include all classes, that the JUnit test classes are going to use.

The javatypesel file

This file specifies restrictions on java types (classes, enums and interfaces) to be processed by the JUnit, Pmd and Checkstyle modules. E. g. the JUnit module restricts java types to be loaded into the JVM during the JUnit test. PMD and Checkstyle restrict the java types to be statically analyzed.

The file is formatted as follows.

First there are lines with the key “include-package=…” defining allowed packages. If a task allows several packages, then there is a single line for each allowed package. Packages specified by this file are not interpreted hierarchically. I. e. if a submission is allowed to submit classes in package a.b and in package a.b.c, then there must be two include-package line for these two packages. A line specifying “include-package=”, i. e. an empty (but not missing) attribute, denotes the default (aka anonymous) package. If there is no “include-package” line, then all packages are allowed.
Second there are lines with the key “exclude-type=…” defining java types not to be loaded by the JUnit, PMD and Checkstyle modules. This key specifies a fully qualified class, interface or enum name. If a submission contains source code of this java type, then the JUnit, PMD and Checkstyle modules will ignore that source code. This can be useful, when students use classes provided by the teacher during programming. When it comes to grading, the grader might want to replace the provided class by a specifically instrumented version, so the submitted class should be ignored. Since FileRestrType / FileRestrictionTO (see below) work on file level only, and since several java types might be implemented into one submitted file, with “exclude-type=…” you can effectively prevent loading of unwanted java types. If there is no “exclude-type=…” line, no java type will be excluded.

The java type selection is a mechanism that unfolds its effect AFTER the submissions-restrictions. The submission-restrictions define exclusions by the FileRestrType / FileRestrictionTO specifications. All non-excluded files will be processed by the Graja modules. With the javatypesel mechanism you can define additional exclusions on java type level instead of file level.

Currently the Compile module does not accept javatypesel, so the java types excluded by javatypesel will be compiled.

The JUnit module currently emits a message for students, when excluding a submitted java type that has not been excluded by submission-restrictions.

The policy file

A policy file defines a java security policy. The syntax is not exactly like the Sun Policy file format (see below).

The policy file has up to two sections defining security permissions for the grader code and for the student’s code. An example:

grantsubmission {
  permission java.util.PropertyPermission "write.file", "read";
};
grantgrader {
  permission java.util.PropertyPermission "write.file", "write";
};

The file format is leaned against the default policy file format. Instead of the usual grant entries with signedBy, codeBase and principal, here the only allowed entries are grantsubmission and grantgrader. Inside the curly braces the format is identical to the policy file syntax.

ExternalResourcesType

Is a collection of external resources.

ExternalResourceType

An external resource basically is a reference. Graja expects:

zero or more external resources. These resources can be added to the JUnit classpath.

The classpath formed by files together with external resources must include all classes, that the JUnit test classes are going to use.

ModelSolutionsType

Is a collection of ModelSolution objects.

ModelSolutionType

A ModelSolution object holds descriptions and references to files.

id: an identifier of the model solution. When creating Graja assignments with the gradle script included with the Graja development tools, then the id corresponds to a folder name below the sources of the assignment. Example: ‘correct’ oder ‘wrong_constructorMissing’.
description: This optional attribute can be used for additional information, for example if more than one sample solution is provided it can be explained why there are several solutions.
internalDescription: These descriptions are for teachers only. Currently Graja only uses internal descriptions for model solutions.

A ModelSolutionType references a FileRefsType object. Currently for a given ModelSolution Graja expects a single file reference in that FileRefsType object. If a solution consists of several files, these are expected as a zip archive.

Currently Graja uses ModelSolutionsType only for documentation and testing purposes. The grading process is independent. The GrajaGui (see below) lets the user choose one of the ModelSolutions as input for the grader.

Tests

Graja expects several tests inside this collection. The test objects are recognized from the value of the testType attribute of a Test object.

As described in the ProFormA white paper, a test is an “automatic check and test for the task”. In Graja we use the ProFormA test element to express distinct kinds of checks. A Graja “module” is a ProFormA-test, that specifies the execution of a tool (Compiler, JUnit, PMD, Checkstyle) or the execution of a manual process (Human).

In terms of the example above a Test object is an orange object on the left.

Test

Graja expects a single TestConfiguration object inside the Test object. Further we have the following attributes in a Test:

Every Test object has an id that is used by elements of the grading scheme hierarchy to reference Test objects.
Every Test object has a title. The title can be seen as the headline of a test.
The attribute testType is one of the following (compare this with the orange nodes in the example above):
- testType=java-compilation : This is a test configuration of a compiler execution.
- testType=unittest : This is a test configuration of one or more JUnit test classes and methods.
- testType=java-pmd : This is a test configuration of a PMD execution.
- testType=java-checkstyle : This is a test configuration of a Checkstyle execution.
- testType=graja-human : This is a test configuration of a placeholder for manually assigned scores.
The attribute validity is not used by Graja.

FileRefsType

Graja expects an optional FilerefsType object in the various Test objects. E. g. in a JUnit test a Fileref to the jar file containing the JUnit test code, a Fileref to the security policy file and a Fileref to the javatypesel file are expected.

ExternalresourcerefsType

Graja allows for ExternalresourcerefsType object in the Test object of testType=unittest. It is expected to contain exactly the external resource ids that are delivered inside the ExternalResourcesType object. Currently external resources are only used for JUnit classpath entries.

Test objects of other testType are not expected to have a ExternalresourcerefsType element.

TestMetaData

Graja currently does not support test meta data in any Test object.

TestConfiguration

A TestConfiguration object may contain different information depending on the testType of the enclosing Test object.

A Test object of testType=unittest accepts a special configuration of the JUnit test. The configuration object is specific to JUnit tests. Hence it is declared in the org::proforma::tests::unittest package. Further, four objects in the Graja namespace are accepted:
- Unittest: configures a Test of testType=unittest
- ComputingResourcesTO: optional, explanation see below
- SystemLocaleTO: optional, explanation see below
Some Test objects accept a Filerefs object to include references to configuration files of the module:
- A Test object of testType=java-compilation does accept a Filerefs object referencing the compile classpath. This can be used when students are expected to compile their code against some library like apache-commons.
- A Test object of testType=unittest accepts a Filerefs object with one or many Fileref objects in it, referencing the jar files that hold the byte code of the JUnit test driver and possibly additional byte code. Also the security policy file and the javatypesel file is expected in this test.
- A Test object of testType=java-pmd accepts a Filerefs object with one or many Fileref objects in it. The referenced files are PMD-specific files with rule set definitions. Also the javatypesel file is expected in this test.
- A Test object of testType=java-checkstyle accepts a Filerefs object with a single Fileref object in it. The referenced file is a Checkstyle-specific file with check definitions. Also the javatypesel file is expected in this test.
- A Test object of testType=graja-human does not accept a Filerefs object.

SystemLocaleTO

Attributes of this object are:

defaultLocale: This locale is used by the JVM as the default locale when executing the backend and also when executing the student’s submission during JUnit testing. The name of the requested locale is built from an IETF BCP 47 language tag string, e. g. en-US or de-DE. The language tag string is the same that is processed by the JavaSE standard class java.util.Locale#forLanguageTag(String).

ComputingResourcesTO

This object specifies limits on resources during a Junit run. If not specified, Graja will assume default values.

maxDiscQuotaKib: maximum disc space in kibibytes allowed to occupy by the running grader and the observed submission
maxRuntimeSecondsWallclockTime: maximum number of seconds. After this number of seconds elapsed (wall clock time), the grader and a submission are aborted.
maxMemMib: maximum ram in mebibytes allowed to get occupied by the JVM running the grader and the observed submission

For multi requests, the values are summed up by Graja (maxDiscQuotaKib, maxRuntimeSecondsWallclockTime) or the maximum value over all assignments is taken (maxMemMib).

The specifies values are meant for the JUnit test only. Graja will reserve some fixed extra time and space for the other tests (compile, pmd, checkstyle).

TemplateSpec / CVVp / VarSpecRoot / MatSpec / CVp / CV

This object stores information for assignments with built in variability. The variability features of Graja are quite new and might change frequently.

Variability is a concept that allows giving each student an individual variant of an assignment. That way we possible can reduce cheating.

Basically an assignment could either serve as a template or as an instance. Only instance assignments can be used for grading. Templates first must be instantiated. The TemplateSpec object specifies a template, the CVVp object specifies an instance. The term CVVp means composite variation resolution (composite, because we potentially have several variation points composed in a vector).

A TemplateSpec defines so-called variation points (Vp) that are bundled in a so-called composite variation point (CVp). Variation point values are called variants (V) that are bundled in a so-called composite variant (CV). The variants chosen for each Vp might depend on each other. All valid combinations of variants are described in a CVSpec (composite variation specification). The number of valid variant combinations can be very large, so usually the set of all combinations is not enumerated but described by rules.

Also a variation template defines so-called materialization instructions in a MatSpec object. A MatSpec consists of:

pointers to artifacts (such as titles or descriptions in various task elements, files, etc.)
methods (search and replace methods, etc.). A method usually manipulates artifacts of the task. Also a method might delete or create parts of a task.
coulings of artifacts and methods called materializations (in fact a cartesian product of a subset of artifacts and a subset of methods).

A task template is materialized by looping through all materializations and by applying all methods of a materialization to all artifacts of that materialization.

Unittest

Attributes of this object are:

framework: this must contain “junit”
version: this must contain “4 with Graja customizations”
entryPoint: the fully qualified class name of the class Grader (the JUnit test class). If your JUnit test code is split among several test classes, there is a separate entryPoint for each of your your test classes.

GradingHints

Inside the GradingHints object a single “root” element of type GradingNode specifies the scores for the various aspects of the assignment. For an explanation of GradingNode see below.

GradingNode

A GradingNode corresponds to inner nodes of the grading scheme hierarchy. There are two kinds of GradingNodes:

A “root” element represents the root of the grading scheme (green box at the top of above example).
A “combine” element represents one of the “category” nodes on the next level.

Each GradingNode has the following attributes:

id: a unique id among all GradingNodes
function: one of “min”, “max”, “sum”
title: to be shown to student and teacher (plain text)
description: to be shown to student (HTML formatted)
internalDescription: to be shown to teacher (HTML formatted)

Each GradingNode references child nodes by their id. There are two kinds of child references:

CombineRef: references another inner node
TestRef: references a leaf node.

CombineRef

A CombineRef is a link in the grading scheme hierarchy.

The attribute ref corresponds to the attribute id of the referenced node.
The weight will be multiplied to the result of the referenced node.

TestRef

A TestRef object is a link from the grading scheme hierarchy to a Test object.

The attribute ref corresponds to the attribute id of the referenced test element.
The weight will be multiplied to the result of the referenced test or subtest.

In Graja we always do reference partial tests (so-called submodules). The attribute subRef means the following:

when referencing a Compile module, subRef is not used.
when referencing a JUnit module, subRef is the method name in a test class. The method name is built from the fully qualified class name, followed by a hash (#), followed by the method name.
when referencing a PMD module, subRef is the name of a PMD rule.
when referencing a Checkstyle module, subRef is the name of a Checkstyle check. If the Checkstyle configuration has Checkstyle modules with id, then this attribute must append the id after a hash, e. g. ‘Indentation#4711’. Currently we only support checknames from the standard checkstyle checks, that do not require the full package path (com.puppycrawl…..). The attribute check must contain the name of the check, optionally followed by the id.
when referencing a Human module, subRef is not used.

Last but not least, when referencing partial tests, usually we need to specify descriptions and a title that will be printed for the respective grading aspect:

title: to be shown to student and teacher (plain text)
description: to be shown to student (HTML formatted)
internalDescription: to be shown to teacher (HTML formatted)

Nullify*

TODO

Request types

A request object is meant as carrying data from an LMS to Graja as “transfer objects” (suffix “TO”). As a typical request Graja accepts an instance of FrontendProformaSubmissionRequestTO containing a single zip file. This is a single request which is comparable with an instance of FrontendGrajaSingleRequestTO containing a single assignment of the proforma type (FrontendProformaAssignmentTO). Both requests are shown in the following diagram.

Before going into the details of that diagram, we have a look at the other request types supported by Graja. First Graja distinguishes single and multi requests:

a single request is a request to grade a single assignment
a multi request is a request to grade multiple assignments during a single Graja execution run. Multi requests can be used, when students create programs for a worksheet with several assignments on it.

Multi requests are deprecated. They may be removed from Graja soon.

Furthermore Graja distinguishes frontend and backend requests. Frontend requests may point to files in the Graja resource database instead of local files. Also frontend assignments can be in the proforma assignment format. In backend requests resource database references have been resolved to local files, a zipped submission has been unzipped to a local folder, and proforma task.zip files have been broken down into their contents. Furthermore backend requests always concern a single assignment only.

Graja’s request domain model outdates that of the ProFormA submission format. A special request type FrontendProformaSubmissionRequestTO consists of a single file in the ProFormA submission format. All other requests are Graja specific request formats.

Last but not least Graja distinguishes “proforma assignments” and “plain old assignments” at the frontend. At the Graja frontend proforma assignments are the recommended assignment format while plain old assignments at the frontend are considered legacy.

All requests are inherited from a common base request object. Graja differentiates between requests by checking the object type. The following picture shows the request domain model:

We will now go into the details of the top level RequestTO object.

RequestTO

A RequestTO is the base class of all requests.

FrontendProformaSubmissionRequestTO

This request holds a single zip file. The data contained in that zip file is described in the ProFormA White paper. An UML model of the data is shown in the above diagram on the left (type SubmissionType in package org::proforma). Graja does only allow tasks to be included as an attached file. Submissions must be present as files. External submissions or tasks are not supported.

The objects DebugSwitchesTO and ProformaSubmissionDataTO contain Graja-specific data that is also represented in a FrontendGrajaRequestTO object in a slightly different layout.

A FrontendProformaSubmissionRequestTO is always a single request.

If a FrontendProformaSubmissionRequestTO contains a GradingHints object inside the SubmissionType object, then Graja will use these GradingHints instead of the grading scheme inside the task. The Grappa middleware and it’s Graja plugin use this in order to scale a task’s score to a given maximum.

ResultSpecTO

A ResultSpecTO object contains many RDKindTO objects. These objects instruct Graja, what kind of comments (“result documents”) it will create.

Also a ResultSpecTO object has the following attributes:

mergeSubresultDocuments: if this is true, then Graja will deliver one ResultNodeTO object per assignments. If false, then Graja will deliver additional ResultNodeTO objects hierarchically per category and per aspect. Default: true. This maps to the ProFormA attribute ResultSpecType.structure (true is equivalent with “merged-test-feedback”, false is equivalent with “separate-test-feedback”).
includeScores: This switch is in effect only, if mergeSubresultDocuments is true. In that case some headlines will contain scores only if this switch is true. Default: true.
includeConsoleOutput: This switch controls, whether the output of the Graja backend will be included into the teacher’s comments. Default: true.
useJavascript: This switch controls, whether HTML output includes Javascript snippets. Default: false.

RDKindTO

A RequestTO object contains many RDKindTO objects. These objects instruct Graja, what kind of comments (“result documents”) it will create. It is a wishlist prepared by the caller. Later in the result object, Graja will return a single document per result node for each single wish in the list.

Graja’s comments may be either scarce or comprehensive. In a RDKindTO object a request defines the kind of document, that it is going to receive.

level: is one of possible values of a java.util.logging.Level object. A SEVERE document consists of error messages only. WARNING and INFO provide more detailed information. The levels CONFIG, FINE, FINER, FINEST are reserved for debugging purposes. This attribute maps to the ProFormA attributes studentFeedbackLevel and teacherFeedbackLevel.
audience: possible values: TEACHER, STUDENT, BOTH. TEACHER is a label for a document targeted to the teacher, STUDENT targets the student, BOTH is a document that contains all the details intended for student and teacher. This attribute maps to the ProFormA attributes studentFeedbackLevel and teacherFeedbackLevel.
representation: possible values: html, plain. When grading a FrontendProformaSubmissionRequestTO, only html is used.

DebugSwitchesTO

keepStarterWorkspace: if true, the folder where the submission gets unzipped and where a temporary security policy file is generated, won’t be deleted after grading completed
keepBackendWorkspace: if true, the workspace folder, where the student’s program is executed, won’t be deleted after grading completed
javaSecurityDebug: a JVM -D switch passed to Graja on backend startup, that can be used to debug security issues
agentlibJdwp: a JVM switch passed to Graja on backend startup, that can be used to debug Graja remotely
overrideMaxRuntimeSecondsWallclockTime: if set, this overrules any runtime settings in the assignments. The specified value will be used for every single assignment. Hence, a multi request will get at most “num of assignments” times “overrideMaxRuntimeSecondsWallclockTime” seconds to complete.

ResourceTO

This is an abstract base class for all resources. Every resource must have a role attribute. Currently Graja uses the following roles:

proformasubmission: a file in the ProFormA submission format
task: this describes the ProFormA task zip file
submission: the submitted file
policy: the security policy file used by the JUnit module
javatypesel: the file with key value pairs to define restrictions on java types loaded by JUnit, PMD and Checkstyle modules
pmdcfg: configuration file of the PMD module
checkstylecfg: configuration file of the Checkstyle module
classpath: file to be added to the classpath of the Compile or JUnit module.

Other roles like model solution files, that are present in a ProFormA task.xml file, are not relevant for ResourceTO objects at the Grader frontend or backend. A ResourceTO object represents a file or external resource, that is referenced by a Graja module.

ResourceLocalFileTO

This object has several attributes:

path: the (usually absolute) path to the local file
encoding: optional encoding
submissionNaturalLanguage: specifies the ISO 639-1 code of the language that is used inside the submitted files. This will be used by Graja to detect the encoding of the submitted file if not provided in the encoding attribute. If the submitted file is UTF-8, the language safely can be omitted. If missing, the default is the natural language inside the assignment. In case of a multi request with contradicting assignment natural languages, the default is “en”.

IdentifiableResourceTO

Graja can reuse resources. Therefore Graja will install resources together with an id for example on the local file system. An IdentifiableResourceTO object references such a resource installed by Graja.

FrontendGrajaRequestTO / BackendRequestTO

The usual interface to communicate with Graja is the frontend interface. Frontend and backend interfaces differ in the following aspects:

At the frontend any files can be either local files or references to the Graja Resource database. Graja can reuse resources. Therefore Graja will install resources together with an id for example on the local file system. A file object referenced as an IdentifiableResourceTO references such a resource installed by Graja.
At the frontend there can be assignments in Proforma task.zip format. This is not possible at the backend. I. e. the frontend module is responsible for unpacking a Proforma assignment and to generate a plain old backend assignment.
At the frontend the submission can be passed as a zip file. The backend expects the submission expanded to a local folder.
The frontend can process a multi request. The backend grades a single assignment only.
The frontend accepts several debug switches.

The FrontendGrajaRequestTO has a hierarchical structure. It can be single request or a multi request.

In a FrontendGrajaRequestTO we have a submission resource. Usually this will be a local file (zip or plain text) and not an IdentifableResourceTO (but it could be). In a BackendRequestTO the submission always is a local folder.

The following diagrams show frontend request and backend request with the details contained:

The class FrontendGrajaRequestTO contains data used for debugging purposes (DebugSwitchesTO) and data that steers the result document generation (ResultSpecTO).

A FrontendGrajaRequestTO object has the following attributes:

maxSubmissionUnpackSizeKib: specifies the maximum number of KiB that a submission may reach when unzipped to memory or disc. The default is 100,000 KiB. This property does not override an assignment specific configuration of the maximum memory or disc quota of the Graja backend process. Instead this value controls maximum resource consumption of the starter process when unzipping a submission. This value should be large in order to allow for any imaginable assignment requirements. The configuration is meant as defense against malicious zip files (https://en.wikipedia.org/wiki/Zip_bomb).

FrontendSingleRequestTO / BackendRequestTO

This object consists of a single FrontendAssignmentTO or BackendAssignmentTO object.

FrontendAssignmentTO

This could be either a FrontendPlainOldAssignmentTO or a FrontenProformaAssignmentTO object.

FrontendPlainOldAssignmentTO / BackendAssignmentTO

A FrontendPlainOldAssignmentTO / BackendAssignmentTO object holds an association with an AssignmentSettingsTO object. That object contains modules with links to several resources. At the frontend Graja can handle local files or identifiable resources. At the backend all resources must be local files.

FrontendProformaAssignmentTO

In this assignment type only a single resource is referenced: a task.zip file in ProFormA format. Above we described the format of that zip file in more detail. The task.zip contains at least the same information as a FrontendPlainOldAssignmentTO.

Moreover a FrontendProformaAssignmentTO defines the following attributes:

FrontendMultiRequestTO

In contrast to a single request this object contains many assignment objects. Each assignment is labelled with a key (see FrontendKeyedAssignmentTO).

Moreover a multi request defines the following attribute:

expectedRootFolderInSubmissionZip: a folder name that Graja will search for as the root folder in the submission zip file. Multi assignment submissions always are expected as a zip file.

FrontendKeyedAssignmentTO

This is an assignment object together with an additional attribute to identify an assignment within a multi request:

key: a multi request may have several assignments. Every assignment inside the request gets a unique key (unique with respect to its siblings).

AssignmentSettingsTO

An AssignmentSettingsTO serves as a bundle of several configuration options that are common in frontend and backend assignments. The AssignmentSettingsTO object is a container for almost all configuration data of a Graja assignments. Some of these configuration options (e. g. classes AssignmentMetaDataTO, ComputingResourcesTO, SystemLocaleTO) are also contained in a ProFormA task file, but in a different order and structure. The AssignmentSettingsTO object is not part of a ProFormA task, but some of the referenced objects are. In the class diagram above the package ending with “extendproforma” holds classes that are also contained in a ProFormA task.

Attributes of an AssignmentSettingsTO object are:

javaVersion: the minimum Java version needed for the assignment. If missing, the minimum java version is assumed 8. The value of this attribute has the same meaning as the attribute version of the ProFormA ProgLang class. ProgLang is used in the task.zip format, AssignmentSettings is used in internal and legacy external formats.
naturalLanguage: an ISO 639-1 string specifying the natural language of the assignment’s description. Maps to the lang attribute of the ProFormA task root element.
maxSubmissionSizeBytes: specifies the maximum size of a submission. Maps to ProFormA’s SubmissionRestrictionsType.maxSize.
submissionRestrictionsDescription: specifies an explanation of the submission restrictions directed to students. Maps to ProFormA’s SubmissionRestrictionsType.description.
submissionRestrictionsInternalDescription: specifies an explanation of the submission restrictions directed to teachers. Maps to ProFormA’s SubmissionRestrictionsType.internalDescription.
title: the title of the assignment. Maps to ProFormA’s MetaData.title.
description: The assignment’s HTML formatted description as seen by students. Maps to ProFormA’s Task.description.
internalDescription: HTML formatted content seen by teachers. Maps to ProFormA’s Task.internalDescription.

Additionally, an AssignmentSettingsTO object might store a reference to a CVVp object that defines the resolution of some variation points. This object is used by the JUnit module when running JUnit test methods. In the ProFormA format this CVVp object is contained in the MetaData object.

ModulesTO / ModuleTO

A ModulesTO object stores the configurations of the modules that should be started when grading the assignment. Each module has an id, a title and possibly a specific configuration. Currently there are five modules: Compile, JUnit, PMD, Checkstyle, and Human.

Each module has the following attributes:

id: a module’s id. This is equivalent to the respective ProFormA Test object’s id attribute.
type: the module’s type, i. e. COMPILE, JUNIT, PMD, CHECKSTYLE or HUMAN.
title: a module’s title. This is equivalent to the respective ProFormA Test object’s title attribute. Currently we often use titles like “Compilation” or “JUnit dynamic analysis” or “PMD static analysis” or “Checkstyle static analysis” or “Manually assigned scores”.
resources: configuration files of the module. This attribute is equivalent to the ProFormA Filerefs object inside a TestConfiguration object. Typical resources are elements on the JUnit classpath or the compile classpath or a security policy file.

Optionally each module references a ModuleConfigurationTO object.

ModuleConfigurationTO

This is an abstract base class that currently does not have any attributes.

JunitModuleConfigurationTO

The attributes of this object are corresponding to the attributes of the Unittest object in a TestConfiguration of a ProFormA Test:

graderFqn: the fully qualified class name of the JUnit test class. If your JUnit test code is split among several test classes, this attribute must hold a comma separated list of all your test classes.

The JunitModuleConfigurationTO holds references to the same SystemLocaleTO and ComputingResourcesTO objects as the corresponding ProFormA Test of test-type=unittest.

FileRestrictionTO

This object holds the same data as the ProFormA FileRestrType.

GradingCategoryTO

This object corresponds to the “root” element and the “combine” elements in the ProFormA GradingHints. It serves as a container for a collection of GradingNodeTO objects. The container stores the following attributes:

id (inherited from GradingNodeTO): a mandatory identifier that is unique among all identifiers of objects of subclasses of GradingNodeTO.
title (inherited from GradingNodeTO): the headline. E. g. “Total result” (for the root) or “Functionality” or “Maintainability” for subsequent inner nodes.
weight (inherited from GradingNodeTO): If the weight value is given, the achieved score for a given submission is multiplied by the given weight.
description (inherited from GradingNodeTO): an optional HTML formatted description shown to student. This description targets the overall grading category. Every child node has it’s own description for the details.
internalDescription (inherited from GradingNodeTO): an optional HTML formatted description shown to teachers.
function: This corresponds to the ProFormA “combine” element’s function attribute. Currently supported: “min”, “max”, “sum”.

Additionally a GradingCategoryTO inherits a reference to a NullifyConditionTO object. If the nullifyCondition association is present, the calculated score for this GradingCategoryTO node might get reduced to zero when the specified condition is met.

GradingAspectTO

This object corresponds to a “TestRef” element in the ProFormA GradingHints hierarchy. This class stores a title of a grading aspect, the weight, and possibly more specific configuration data:

id (inherited from GradingNodeTO): a mandatory identifier that is unique among all identifiers of objects of subclasses of GradingNodeTO.
title (inherited from GradingNodeTO): the aspect’s headline, describing the purpose of the aspect. A title is optional. Missing titles are replaced by standard titles, e. g. built from the corresponding submodule.
weight (inherited from GradingNodeTO): maximum score awarded by the aspect. If the weight is missing, Graja assumes a weight of 1.
description (inherited from GradingNodeTO): an optional HTML formatted description shown to student.
internalDescription (inherited from GradingNodeTO): an optional HTML formatted description shown to teacher.
moduleRefId: id of a specific grading aspect’s Test object. This attribute is used to attach a grading aspect to a specific module.
moduleSubRefId: This identifies the submodule. For an explanation of the valid contents see “TestRef” above.

Additionally a GradingAspectTO inherits a reference to a NullifyConditionTO object. If the nullifyCondition association is present, the calculated score for this GradingAspectTO node might get reduced to zero when the specified condition is met.

In terms of the example above a GradingAspectTO combines is a green node at the top.

Nullify*

TODO

Implications for submissions

If Graja receives a single request, the submitted file could be plain text file (usually a java source file) or a zip file containing several files. A zip file is expected to have the following content:

The solution files are located at the root of the archive. If a solution itself is divided into subdirectories, these subdirectory entries are rooted at the root of the zip archive.
This is also the expected content of the expanded submission folder passed to the backend.

A multi request assumes a zip file of the following content:

The archive has a single root folder named expectedRootFolderInSubmissionZip.
Below that there are one or more subdirectories, each one containing the solution for a single assignment. The names of the subdirectories are equal to the key attribute of the respective FrontendKeyedAssignmentTO.
Below that the solution files are expected. If a solution itself is divided into subdirectories, every such subdirectory entries are rooted at <expectedRootFolderInSubmissionZip>/<key>/<subdirectory>.

Result

The following picture shows the result domain model:

ResultTO

The result transfer object is either a Graja specific result object or an object containing a ProFormA response object.

ProformaResponseResultTO

In the responseContent attribute there is a serialized form of an object of the type ResponseType. The data contained in a ResponseType object is described in the ProFormA White paper. An UML model of the data is shown in the above diagram.

GrajaResultTO

Attributes of a GrajaResultTO are:

grajaVersion: the version number (usually a date/time tag) of Graja
error: will be true, if the submission could not be graded in detail. We refer to this as a “class level failure”, where the submission either could not be unzipped or compiled or where the Grader was not able to call individual methods of the submission. The reason might be a wrong class name in the submission, a wrong package, but the reason might also be an error in the Grader itself. When the Grader was able to call individual methods of the submission and some or all methods behaved wrong, the Grader assigns less than 100%, but this not an “error”, i. e. the attribute error gets false.
messagePreformattedPlain: Here Graja may include error messages, that usually are targeted to the teacher or the administrator of the software installation. Examples are: Graja cannot find some configuration files, Graja cannot find a bash, Graja cannot find the request file passed as parameter to CmdlineStarter, etc.

The Graja backend always delivers an object of type GrajaSingleResultTO.

The Graja frontend usually delivers an object of type GrajaSingleResultTO, when it received a FrontendGrajaSingleRequestTO and it delivers a GrajaMultiResultTO as an answer to a FrontendGrajaMultiRequestTO. But, if things went severely wrong, Graja might not be able to read the request file, so it doesn’t know, which result object type to deliver. In this case, Graja delivers an instance of GrajaResultTO, the base class.

GrajaSingleResultTO

A single result consists of a single AssignmentResultTO object.

AssignmentResultTO

In this object Graja delivers:

graderVersion/graderName: information about the grader (date/time tag and the “token” of the Grader).
structure: is either “merged” or “separate”.
root: a reference to the root result node with scores and result documents
nullifyRefdOrphaneTestResult: an optional list of references to test/sub-test results that are referenced by nullify conditions but that are not reachable through the node hierarchy rooted at root. This is only used if mergeSubresultDocuments is false.

ResultNodeTO

In this object Graja delivers:

id: this id references a corresponding node in the grading scheme of the request, i. e. the respective GradingNodeTO object inside the AssignmentSettingsTO object.
headline: a title text, that is built from the GradingNodeTO title and the achieved result score.
scoreAchieved: the achieved score (maximum value is the referenced grading scheme)
scoreMax: the maximum score copied here from the grading scheme

Inside a ResultNodeTO Graja includes documents with an explanation of the achieved score. For every requested RDKindTO a single DocumentTO object will be delivered. Also a ResultNodeTO includes the descriptions of the respective GradingNodeTO as separate documents.

Further a ResultNodeTO usually has a hierarchy of subResults that corresponds the the grading scheme hierarchy. If the request’s ResultSpecTO object specified to merge subresults, then there will be only one ResultNodeTO object per assignment and the subResults list will be empty.

DocumentTO

The documents differ in three dimensions:

level: legal values are the same as in RDKindTO.
audience: see RDKindTO.
content: here the document is contained as a byte sequence. The bytes are to be interpreted as plain text (if representation points to a DocReprPlainTextTO object) or as html (if representation points to a DocReprHtmlTO object).

DocRepresentationTO

This object holds information about the representation of the document.

If it is an object of class DocReprHtmlTO, the content must be interpreted as UTF-8 encoded text containing a html document.
If it is an object of class DocReprPlainTextTO, the content must be interpreted as text that is encoded in the specified charset. Currently Graja always delivers UTF-8.

GrajaMultiResultTO

In a multi result Graja delivers a single AssignmentResultTO for every AssignmentTO object from the request. The keys in the KeyedAssignmentResultTO refer to the keys in the request.

Graja will create comments not only for every single assignment, but also some comments that conceptually belong to the whole result. E. g. error messages regarding the unzipping of the submission belong to the whole result. A GrajaMultiResultTO contains a set of DocumentTO objects, where each DocumentTO object corresponds to one of the RDKindTO objects in the FrontendGrajaMultiRequestTO object.

KeyedAssignmentResultTO

This contains simply an AssignmentResultTO together with a key, that references a corresponding FrontendKeyedAssignmentTO in the request.

Representing a request in XML or JSON

An example of a single request file with a ProFormA assignment in it is the following:

{
  "@class" : ".FrontendGrajaSingleRequestTO",
  "maxSubmissionUnpackSizeKib" : 100000,
  "debugSwitches" : {
    "keepStarterWorkspace" : true,
    "keepBackendWorkspace" : true,
    "overrideMaxRuntimeSecondsWallclockTime" : 30
  },
  "submission" : {
    "@class" : ".ResourceLocalFileTO",
    "role" : "submission",
    "path" : "C:\\Users\\john\\Documents\\submission.zip",
    "encoding" : null,
    "naturalLanguage" : null
  },
  "resultSpec" : {
    "includeScores" : true,
    "mergeSubresultDocuments" : true,
    "includeConsoleOutput" : true,
    "useJavascript" : false,
    "rdKind" : [ {
      "level" : "INFO",
      "audience" : "STUDENT",
      "representation" : "html"
    }, {
      "level" : "INFO",
      "audience" : "TEACHER",
      "representation" : "html"
    } ]
  },
  "assignment" : {
    "@class" : ".FrontendProformaAssignmentTO",
    "taskZip" : {
      "@class" : ".ResourceLocalFileTO",
      "role" : "task",
      "path" : "C:\\Users\\john\\Documents\\task.zip",
      "encoding" : null,
      "naturalLanguage" : null
    }
  }
}

Schema descriptions

There is a helper service in Graja, that prints the XML schema description for requests and results:

The xml schema description used inside request.xml and result.xml can be viewed by calling command query_graja_xsd on the Graja CLI.
Not functional yet: The json schema used in request.json and result.json can be viewed by calling: java -cp /path/to/GrajaStarter.jar de.hsh.graja.transform.QueryJsonSchema

Converter services

Graja provides converters to create requests and to consume and show results in a human readable form. See the next section Develop assignments for details.

Variability

The variability features of Graja are quite new and might change frequently.

Variability is a concept that allows giving each student an individual variant of an assignment. That way we possible can reduce cheating.

Basically an assignment could either serve as a template or as an instance. Only instance assignments can be used for grading. Templates first must be instantiated.

Data model

Variability information of a template or an instance is contained in the TemplateSpec object described above. A TemplateSpec object is generated by the Graja development tools from a so-called VariabilityProvider. A VariabilityProvider is a subclass of de.hsh.graja.devasgmt.variability.AssignmentVariabilityProvider and implements the template. This subclass must be implemented in the subpackage grader, e. g. for an assignment in the package org.example, the variability provider must be implemented in the class org.example.grader.VariabilityProvider.

A variation template defines so-called variation points (Vp) that are bundled in a so-called composite variation point (CVp). Variation point values (called variants) might depend on each other. All valid combinations of variants are described in a CVSpec (composite variation specification). The number of valid variant combinations can be very large, so usually the set of all combinations is not enumerated but described by rules. The rules are implemented in the VariabilityProvider class. The VariabilityProvider defines the set of variation points together with their data types and it defines the total set of valid variants as something like a union of cartesian products. Some Vp might not be specified by their variants, but instead by a javascript fragment that calculates the possible variants on the fly.

Creating instances from a template

There is a command line service to create an assignment instance from an assignment template:

java -Dde.hsh.graja.cli.logging.level=INFO --module-path /path/to/Graja;/path/to/Graja/extjar --module de.hsh.graja/de.hsh.graja.Cli 
   --command instantiate_template
   --input /path/to/task_template.zip
   --output /path/to/task.zip

See option --help for more informations.

Currently the automatic build procedure creates a task template and additionally a task instance with default values for the variables. The task instance is named …dis.zip (short for default instance).

The GrajaGui (see next section) let’s the user interactively choose variants for all variation points. When starting Graja from the Graja GUI, a new task instance will be generated for the selected variants.

Example

In SampleGrajaAssignments there is an example (vowels), that demonstrates a few features of variability. More examples are currently being prepared.

Page contents

Documentation Overview

Execute Graja

The Graja CLI

Command line service

API service

What makes up an assignment?

ProFormA terms

Graja terms

Modules

Grading categories and aspects

Example

Realization of Graja modules and grading categories/aspects as ProFormA tests and grading-hints

Request and Result

Domain model

Request

The ProFormA task.zip format

Task

Proglang

SubmissionRestrictionsType

FileRestrType

MetaData

AssignmentMetaDataTO

Files

TaskFileType

The javatypesel file

The policy file

ExternalResourcesType

ExternalResourceType

ModelSolutionsType

ModelSolutionType

Tests

Test

FileRefsType

ExternalresourcerefsType

TestMetaData

TestConfiguration

SystemLocaleTO

ComputingResourcesTO

TemplateSpec / CVVp / VarSpecRoot / MatSpec / CVp / CV

Unittest

GradingHints

GradingNode

CombineRef

TestRef

Nullify*

Request types

RequestTO

FrontendProformaSubmissionRequestTO

ResultSpecTO

RDKindTO

DebugSwitchesTO

ResourceTO

ResourceLocalFileTO

IdentifiableResourceTO

FrontendGrajaRequestTO / BackendRequestTO

FrontendSingleRequestTO / BackendRequestTO

FrontendAssignmentTO

FrontendPlainOldAssignmentTO / BackendAssignmentTO

FrontendProformaAssignmentTO

FrontendMultiRequestTO

FrontendKeyedAssignmentTO

AssignmentSettingsTO

ModulesTO / ModuleTO

ModuleConfigurationTO

JunitModuleConfigurationTO

FileRestrictionTO

GradingCategoryTO

GradingAspectTO

Nullify*

Implications for submissions

Result

ResultTO

ProformaResponseResultTO

GrajaResultTO

GrajaSingleResultTO

AssignmentResultTO

ResultNodeTO

DocumentTO

DocRepresentationTO