GrajaGrader for java programs     Documentation     Publications     Downloads     Contact

Graja is a JUnit-PMD-Checkstyle-based grader for java programs submitted by students in formative and summative assessment. This site provides publications, software, and information about Graja.

Execute Graja

Execute Graja

The Graja CLI

Graja provides a central command line interface (CLI) for all functions available in Graja. The functions reach from the usual processing of a grading request to developer oriented commands like creating a ProFormA zip file or querying an xml schema.

All commands are accessible by a command line of the following form:

java -jar /path/to/GrajaCli.jar --command <command> [options]

The CLI provides logging to the console, when you specify a log level like this:

java -Dde.hsh.graja.cli.logging.level=<level> -jar /path/to/GrajaCli.jar --command <command> [options]

The <level> controls the console output when running the command. Supported <level>s are ALL, DEBUG, INFO, WARN, ERROR, OFF. The default is ERROR.

The following will list all available <command>s:

java -jar /path/to/GrajaCli.jar --help

Command line service

The main grader can be called on the command line like follows:

java -jar /path/to/GrajaCli.jar --command grade_request --format json --request /path/to/request.json --result /path/to/result.json 

The command line starter accepts three parameters:

  • a file format (currently json and xml are accepted)
  • a path to a file containing a grading request
  • a path to a file, where graja should store the result.

A single Graja run processes a single submission. A submission may contain solutions to one or more assignments (which depends on the contents of the request file, see below). If you want to process more than one submission (each solving one ore more assignments), you need to start a separate Graja instance for every submission.

If the request file requests Graja to grade several assignments for a single submission in one go, we call this a “multi request”. In that case, Graja will collect the individual results for every assignment and write the collected results (a “multi result”) to the specified output file.

API service

Besides the main command line interface, there is another service class: de.hsh.graja.starter.apcli.WithinJVMStarter. This class serves a RequestTO object and returns a ResultTO object (see below). This class is used by libraries like the GrajaPlugin inside Grappa in order to start Graja conveniently from a running java program.

What makes up an assignment?

As stated above, a request consists of a submission and a specification of one or several assignments to be graded. A single assignment is perferably stored in a so-called ProFormA-task file. A task is a configuration of one or more tools or grading processes together with a grading scheme. Since Graja is using the ProFormA standard format for tasks, it has to adapt to that standard.

ProFormA terms

The ProFormA format defines (besides many more) two central areas:

  • tests: a Test object defines a check by specifying a title, a test-type and an id. Also a Test object contains a more or less complex TestConfiguration object, which can contain Graja-specific data. The ProFormA standard lists several test-types, e. g. compilation, unittest, style checker.
  • grading-hints: A GradingHints object in the ProFormA standard is mainly an extension point for grader specific enhancements of the format. Graja here defines scores and a hierarchy of grading categories and grading aspects.

Graja terms

Modules

Graja executes various tools as backend “modules”. Currently Graja uses the modules Compiler, JUnit, PMD, and Checkstyle. Also it provides a “Human” module, that defines the execution of a manual grading process.

Each module execution produces results. A module execution can be split into partial executions. E. g. a JUnit run can be seen as an execution of many individual JUnit test methods. Every test method is a partial test or a “submodule”. Every test method produces a score and a comment for the submitting student. In the sense of the two central ProFormA areas, a JUnit test method execution is a “Test” (title, technical configuration) and it also is a “grading aspect” (score, category).

Currently the number of modules per type is limited in Graja as follows:

  • Compile: max. 1
  • JUnit: max. 1
  • Human: max. 1
  • PMD: unlimited
  • Checkstyle: 1

Grading categories and aspects

The grading scheme in Graja is a hierarchy of three levels: a “grading root” node at the top, “grading categories” at the intermediate level, and “grading aspects” as leaves. A “grading category” aggregates several grading aspects below a common headline. Common headlines are “Functional correctness” or “Maintainability”. Examples for “grading aspects” are: “The method x should calculate y”. Or “The class Z should get a leading comment”. The three-leveled hierarchy is realized by “group” objects (inner nodes of the hierarchy) and “element” objects (leaves). Each group and each element refers to a corresponding ProFormA-Test object. That way any grading aspect directly refers to its associated submodule execution like a single JUnit test method, a single PMD rule, a single Checkstyle check, or a single grading aspect that is reviewed manually by humans.

The “group” and “element” objects from the previous paragraph are name “TestGroupTO” and “TestElementTO”. The term “Test” in these names reminds us, that every grading aspect and every grading aspect group refers to real test executions.

Example

The following image shows an example Graja output for a task, that defines five grading categories with an individual number of grading aspects in every category. The “Source” column states the module used to realize the aspect. The example was invented when there was no Checkstyle module yet, so Checkstyle is missing. But in all places where PMD is stated or executed, you equivalently could use Checkstyle as a replacement.

Realization of Graja modules and grading categories/aspects as ProFormA tests and grading-hints

The following image shows, how Graja maps it’s concepts (modules, submodules, grading categories, grading aspects) to the ProFormA concepts (tests and grading-hints):


Legend:


At the top of the image we see the grading scheme, that is stored below the ProFormA grading-hints section as TestGroupTO and TestElementTO objects. On the left the image shows the ProFormA Test objects. Since we wanted to use as much of the standard as possible, we define the title of inner nodes of the grading scheme hierarchy as part of dedicated Test objects. The inner nodes of the grading scheme hierarchy are TestGroupTO objects (e. g. green node “Category D”), each storing a score and child nodes and a reference to a corresponding Test object (e. g. orange node “Category D”).

Graja modules are realized as ProFormA Test objects (orange nodes Compile, JUnit, PMD, Checkstyle [not in the image], Human). As can be seen, some modules might be specified more than once, because it might be advantageous to run the same tool (PMD in the above example) twice with different parameterization, in order to produce results for different grading aspects. E. g. PMD rule 2 has been executed twice. In the second PMD run, that rule was evaluated again (PMD rule 2b), but with a different parameterization. The first evaluation (PMD rule 2) contributes to the grading aspect C.2 while the second evaluation (PMD rule 2b) contributes to a different aspect D.4.

We can also see, that some submodules (brown nodes on the left) are referenced by more than one grading aspect (dark green nodes at the top). The JUnit method 2 is evaluated only once during a single JUnit execution. Its results contribute to two grading aspects C.1 and E.1, possibly with different score weights.

The grading-hints hierarchy is rooted at a single root node (light green “Root” at the top), which contains one or more category nodes, which contain one or more aspect nodes.

The Test objects on the left cannot be hierarchically arranged in the ProFormA format, because the standard does not allow the grouping of tests. Hence, the orange nodes are listed in no specific order in the task.xml file. A special orange “Root” test (left side) holds global configuration data, that is relevant to all other tests.

Request and Result

Graja expects requests and produces results. In this section we will describe requests and results in detail.

Domain model

Here we describe a domain model of requests accepted by Graja and results produced by Graja. For now we abstract from the representation of domain objects (XML or JSON).

Request

Graja can serve requests of different types. Most of these types are legacy formats. The preferred way to start Graja is by using the ProFormA assignment format (also called the task.zip format).

The ProFormA task.zip format

The task.zip format is described in detail on the ProFormA GitHub repository site and in this journal paper. Graja 1.8 and 1.9 support the ProFormA format version 1.1. Graja 1.7 supports ProFormA format version 1.0.1.

A task.zip file contains the Proforma exchange format as a file named “task.xml” together with several attachments.

Task

This is the root object of the proforma specification. It describes the contents of the task.xml inside the task.zip file.

A Task object has the following attributes:

  • uuid: a unique identifier. Currently, when using the Graja development tools, every newly built task.zip gets a new random uuid.
  • parentUuid: not used by Graja
  • lang: natural language. Default is “en”, which can be overridden by an assignment developer in the AssignmentSettingsTO in the so called descriptor.xml.
  • description: This is HTML content with the assignment’s text seen by students.

Proglang

The value simple content should be “java”. The version attribute describes the assignment’s compatibility with Java versions. The assignment is considered compatible with any Java version equal to or higher than the specified attribute value.

SubmissionRestrictions

Graja does not interpret the data in this object.

GradingHints

Inside the GradingHints object a single TestGroupTO object specifies the scores for the various aspects of the assignment. For an explanation of TestGroupTO see below.

MetaData

Inside the MetaData object an AssignmentMetaDataTO object has meta information in a Graja specific namespace. For an explanation of AssignmentMetaDataTO see below.

AssignmentMetaDataTO

Attributes of this object are:

  • assignmentId: an identifier for the assignment
  • assignmentVersion: the version of the assignment. If the assigment is modified, it should get a new version.
  • targetGroup: a description of the target student audience. Defines the ideal student for this assignment in natural language plain text.

Files

Is a collection of files.

File

A file’s content can either be embedded in the xml or it can be attached as a separate file. A file has the following attributes:

  • id: an identifier
  • type: if “file”, then the file is attached to the zip file. If “embedded”, the file content is embedded into the task.xml.
  • filename: the name of the file (optional if embedded).
  • comment: a plain text comment about the file
  • class: the type of the file, e. g. “internal” for files not visible to students.

Graja expects:

  • a file which is referenced by ProformaPolicyRefTO. This is the security policy file, which contains “grantgrader” and/or “grantsubmission” sections.
  • one or more files with class=internal and referenced by ProformaClasspathRefsTO. These files will be used as the classpath (a list of jar files) when starting Graja.
  • one or more files with class=internal and referenced by ModelSolutions. These files hold sample solutions of the task.

The classpath and the external resources below together must include all classes, that the Grader class (the AssignmentGrader subclass) is going to use.

ExternalResources

Is a collection of external resources.

ExternalResource

An external resource basically is a reference. Graja expects:

  • zero or more external resources, that can be referenced by ProformaPolicyRefTO of ProformaClasspathRefsTO. The latter resources will be used as the classpath (a list of jar files) when starting Graja.

The classpath formed by files together with external resources must include all classes, that the Grader class (the AssignmentGrader subclass) is going to use.

ModelSolutions

Is a collection of ModelSolution objects.

ModelSolution

A ModelSolution object holds a comment and references to files.

  • id: an identifier of the model solution. When creating Graja assignments with the gradle script included with the Graja development tools, then the id corresponds to a folder name below the sources of the assignment. Example: ‘correct’ oder ‘wrong_constructorMissing’.
  • comment: This optional attribute can be used for additional information, for example if more than one sample solution is provided it can be explained why there are several solutions.

A ModelSolution references a FileRefs object. Currently for a given ModelSolution Graja expects a single file reference in that FileRefs object. If a solution consists of several files, these are expected as a zip archive.

Currently Graja uses ModelSolutions only for documentation and testing purposes. The grading process is independent. The GrajaGui (see below) lets the user choose one of the ModelSolutions as input for the grader.

Tests

Graja expects several tests inside this collection. The test objects are recognized from the value of the testType attribute of a Test object.

As described in the ProFormA white paper, a test is an “automatic check and test for the task”. In Graja we use the ProFormA test element to express distinct kinds of checks. Some checks depend on other checks:

  • A Graja “module” is a ProFormA-test, that specifies the execution of a tool (Compiler, JUnit, PMD, Checkstyle) or the execution of a manual process (Human).
  • A Graja “submodule” is a ProFormA-test, that specifies a partial execution of a tool or of a manual process. A submodule can be a single JUnit test method, a single PMD rule, a single Checkstyle check, or a single grading aspect that is reviewed manually by humans. Of course a submodule depends on the corresponding module. For the compiler currently Graja supports only a single default submodule that represents all compilation results.

Furthermore Graja expects tests that define the inner nodes of a grading hierarchy:

  • A Graja “grading root” test specifies the root of the grading scheme hierarchy and also global configuration data, common to all other tests.
  • A Graja “grading category” is a ProFormA-test, that specifies an inner node of the grading scheme hierarchy with a headline. Common headlines are “Functional correctess” or “Maintainability”.

Since the ProFormA Test object is not prepared for a hierarchical specificiation of tests and subtests, that hierarchy information is stored as part of the GradingHints object (see below) in additional objects TestGroupTO, TestGroupMembersTO and TestElementTO. These three objects can be seen as a data enrichment for the ProFormA Test objects

So, we can look at a Test object in Graja as a small piece of configuration data, that has to be arranged and put to its place in a bigger picture. The bigger picture is the GradingHints object (see below). In terms of the example above a Test object is an orange/brown object on the left that has to be arranged into the bigger picture of the GradingHints objects (green at the top).

Test

Graja expects a single TestConfiguration object inside the Test object. Further we have the following attributes in a Test:

  • Every Test object has an id that is used by elements of the grading scheme hierarchy to reference Test objects. The classes TestGroupTO and TestElementTO (see below) represent nodes of the grading scheme hierarchy and have got a testrefId-attribute that references one of the Test objects.
  • Every Test object has a title. The title can be seen as the headline of a grading category or module or as a short description of an individual grading aspect.
  • The attribute testType is one of the following (compare this with the orange/brown nodes in the example above):
    • testType=graja-grading-root : this is the so called “main test” with files and data for an overall configuration of Graja.
    • testType=graja-grading-category : this is an inner node of the grading scheme hierarchy. There can be several Test objects with this testType.
    • testType=graja-module-compile : This is a test configuration of a compiler execution.
    • testType=graja-submodule-compile-default : this is the configuration of a single part of a compiler execution. Currently we do not configure anything at all here. In the future e. g. we could configure a single part for every submitted source file.
    • testType=graja-module-junit : This is a test configuration of a JUnit based AssignmentGrader class.
    • testType=graja-submodule-junit-method : This is a configuration of a single JUnit test method.
    • testType=graja-module-pmd : This is a test configuration of a PMD execution.
    • testType=graja-submodule-pmd-rule : This is a configuration of a single PMD rule.
    • testType=graja-module-checkstyle : This is a test configuration of a Checkstyle execution.
    • testType=graja-submodule-checkstyle-check : This is a configuration of a single Checkstyle check.
    • testType=graja-module-human : This is a test configuration of a placeholder for manually assigned scores.
    • testType=graja-submodule-human-aspect : This is a configuration of a single aspect for manually assigned scores.
  • The attribute validity is not used by Graja.

TestGroupTO, TestGroupMembersTO and TestElementTO

These classes represent information that cannot be stored directly inside Test objects, because the ProFormA format does not provide attributes for that. We see these three classes as a data enrichment for the ProFormA Test objects. The following are the four kinds of data that we need in Graja that cannot be stored in proforma Test objects:

  • maximum scores
  • parent-children associations (i. e. grouping of grading aspects)
  • description to be shown to student (HTML formatted)
  • internal description to be shown to teacher (HTML formatted)

Maximum scores are stored in a TestElementTO object (representing a grading aspect that references a submodule like a JUnit test method, a PMD rule, a Checkstyle check, or a manual grading aspect) or in a TestGroupTO object (representing an aggregate of individual grading aspects). If Graja receives scoreMax values at inner nodes, these must be exactly the sum of all child nodes. If inner nodes have no scoreMax attribute value, Graja will derive that value from the sum of the child node’s scores. Leave nodes (TestElementTO) without a scoreMax value are considered a scoreMax=1.

In the example above the green nodes at the top are realized by TestGroupTO, TestGroupMembersTO and TestElementTO objects.

FileRefs

Graja expects a single FileRefs object only in the main Test object of testType=graja-grading-root. This Test object is considered the father of all tests and hence it is the place, where references to exactly those file ids are expected, that are delivered inside the Files object.

Test objects of other testType are not expected to have a FileRefs element.

Externalresourcerefs

Graja expects a single Externalresourcerefs object only in the main Test object of testType=graja-grading-root. It is expected to contain exactly the external resource ids that are delivered inside the ExternalResources object. Currently external resources are only used for classpath entries.

Test objects of other testType are not expected to have a Externalresourcerefs element.

TestMetaData

Graja expects a single TestMetaData object only in the main Test object of testType=graja-grading-root. Inside this Graja expects a single GrajaMetaDataTO object. The GrajaMetaDataTO object is specific to Graja. Hence it is declared in a graja package. For an explanation of GrajaMetaDataTO see below.

GrajaMetaDataTO

Attributes of this object are:

  • grajaVersionCompatibility: the minimum Graja version needed for the assignment. If missing, no graja compatibility check will be performed.

TestConfiguration

A TestConfiguration object may contain different information depending on the testType of the enclosing Test object.

  • The main Test object of testType=graja-grading-root ist expected to have the following objects inside TestConfiguration. All these objects are specific to Graja. Hence they are declared in a graja package.

    • ProformaClasspathRefsTO: explanation see below
    • ProformaPolicyRefTO: explanation see below
    • ComputingResourcesTO: optional, explanation see below
    • GrajaConfigurationTO: optional, explanation see below
    • SubmissionRulesTO: optional, explanation see below
    • VariabilityInfoTO: optional, explanation see below
  • A Test object of testType=graja-grading-category currently does not accept special configuration.

  • A Test object of testTypes=graja-module-* accepts special configuration of the respective test. All these configuration objects are specific to Graja. Hence they are declared in a graja package. For an explanation of these objects see below. The configuration data is different for the different testTypes:

    • CompileModuleConfigurationTO: configures a Test of testType=graja-module-compile
    • JunitModuleConfigurationTO: configures a Test of testType=graja-module-junit
    • PmdModuleConfigurationTO: configures a Test of testType=graja-module-pmd
    • CheckstyleModuleConfigurationTO: configures a Test of testType=graja-module-checkstyle
    • HumanModuleConfigurationTO: configures a Test of testType=graja-module-human
  • A Test object of testType=graja-submodule-* accepts special configuration of the respective test. All these configuration objects are specific to Graja. Hence they are declared in a graja package. For an explanation of these objects see below. The configuration data is different for the different testTypes:

    • CompileSubmoduleMethodConfigurationTO: configures a Test of testType=graja-submodule-compile-default
    • JunitSubmoduleMethodConfigurationTO: configures a Test of testType=graja-submodule-junit-method
    • PmdSubmoduleRuleConfigurationTO: configures a Test of testType=graja-submodule-pmd-rule
    • CheckstykeSubmoduleCheckConfigurationTO: configures a Test of testType=graja-submodule-checkstyle-check
    • HumanSubmoduleAspectConfigurationTO: configures a Test of testType=graja-submodule-human-aspect

GrajaConfigurationTO

Attributes of this object are:

  • defaultCharset: This charset is used by the JVM as the default charset when executing the backend and also when executing the student’s submission during JUnit testing. The name of the requested charset may be either a canonical name or an alias. Legal values are defined by java.nio.charset.Charset#forName(String).
  • defaultLocale: This locale is used by the JVM as the default locale when executing the backend and also when executing the student’s submission during JUnit testing. The name of the requested locale is built from an IETF BCP 47 language tag string, e. g. en-US or de-DE. The language tag string is the same that is processed by the JavaSE standard class java.util.Locale#forLanguageTag(String).

CompileModuleConfigurationTO

The compile module currently does not accept any configuration.

JunitModuleConfigurationTO

Attributes of this object are:

  • graderFqn: the fully qualified class name of the class Grader that subclasses de.hsh.graja.graderapi.AssignmentGrader. If your JUnit test code is split among several subclasses of AssignmentGrader, this attribute must hold a comma separated list of all your subclasses.

PmdModuleConfigurationTO

Attributes of this object are:

  • rulesetRef: name of a PMD ruleset.

CheckstyleModuleConfigurationTO

Attributes of this object are:

  • configRef (mandatory): file name of a Checkstyle configuration file.

HumanModuleConfigurationTO

This object has no attributes, since the respective module currently does not allow any specific configuration. This might change in future versions.

ModuleConfigurationTO

This is an abstract base class that currently does not have any attributes.

CompileSubmoduleDefaultConfigurationTO

The compile submodule currently does not accept any configuration.

JunitSubmoduleMethodConfigurationTO

Attributes of this object are:

  • method: method name in a Grader class that subclasses de.hsh.graja.graderapi.AssignmentGrader. The method name is built from the fully qualified class name, followed by a hash (#), followed by the method name.

PmdSubmoduleRuleConfigurationTO

Attributes of this object are:

  • rule: name of a PMD rule.

CheckstyleSubmoduleCheckConfigurationTO

Attributes of this object are:

  • check: name of a Checkstyle check. If the Checkstyle configuration has modules with id, then this attribute must append the id after a hash, e. g. ‘Indentation#4711’. Currently we only support checknames from the standard checkstyle checks, that do not require the full package path (com.puppycrawl…..). The attribute check must contain the name of the check, optionally followed by the id.

HumanSubmoduleAspectConfigurationTO

This submodule currently does not accept any configuration.

SubmoduleConfigurationTO

Attributes of this object are:

  • moduleRefId: id of a specific module’s Test object. This attribute can be used to attach a submodule to a specific module. This is especially useful, when there is more than one module of a kind like in the above example where we had two PMD modules. If there is no moduleRefId or it is empty, then Graja will associate the submodule with any module of matching type, e. g. a graja-submodule-junit-method will be associated to a graja-module-junit.

ProformaClasspathRefsTO

This object contains a list of references to Files and/or ExternalResources. All these references are used in the classpath when starting Graja.

ProformaPolicyRefTO

This object contains an optional reference to a File or an ExternalResource. The referenced resource is used as the security policy.

ProformaRefTO

This object represents a reference to a File or an ExternalResource. Attributes of this object are:

  • refid: The id of the referenced object
  • externalOrFile: if this attribute contains ‘file’, a File object is referenced. If this attribute contains ‘external’, an ExternalResource is referenced.

ComputingResourcesTO

This object specifies limits on resources during a Graja run. If not specified, Graja will assume default values.

  • maxDiscQuotaKib: maximum disc space in kibibytes allowed to occupy by the running grader and the observed submission
  • maxRuntimeSecondsWallclockTime: maximum number of seconds. After this number of seconds elapsed (wall clock time), the grader and a submission are aborted.
  • maxMemMib: maximum ram in mebibytes allowed to get occupied by the JVM running the grader and the observed submission

For multi requests, the values are summed up by Graja (maxDiscQuotaKib, maxRuntimeSecondsWallclockTime) or the maximum value over all assignments is taken (maxMemMib).

SubmissionRulesTO

This object specifies rules on student submissions like allowed java packages and submitted classes that will be ignored by Graja.

IgnoreClassTO

This object specifies a fully qualified class name. If a submission contains source code of this class, then Graja will ignore it. This can be useful, when students use classes provided by the teacher during programming. When it comes to grading, the grader might want to replace the provided class by a specifically instrumented version, so the submitted class should be ignored.

Currently skipped classes will be compiled, even if specified in this object. So this annotation will be effective when loading classes only. In order to prevent also compiling a file, use RestrictFileTO below. If there is no IgnoreClassTO object, no class will be ignored.

RestrictPackageTO / RestrictFileTO

These objects specify restrictions on submitted files. Graja builds the union set of all specified RestrictPackageTO and RestrictFileTO objects. Let us call this union set U.

The modules Compile, JUnit, PMD, and Checkstyle restrict their actions to those submitted files, that are contained in U. If a submitted file is not contained in U, the modules will act as follows:

  • JUnit will abort the module, if the submissions contains classes in other packages than specified. File restrictions are not accounted for.
  • Compile will simply skip a submitted file that is neither contained in the specified packages nor in the specified files.
  • PMD will simply skip the submitted file that is neither contained in the specified packages nor in the specified files.
  • Checkstyle will simply skip the submitted file that is neither contained in the specified packages nor in the specified files.

A RestrictPackageTO objects specifying an empty (but not missing) name attribute denotes the default (aka anonymous) package.

Packages specified by a RestrictPackageTO object are not interpreted hierarchically. I. e. if a submission is allowed to submit classes in package a.b and in package a.b.c, then there must be two RestrictPackageTO objects for these two packages.

If there is no RestrictPackageTO AND no RestrictFileTO object, then all packages and files are allowed.

VariabilityInfoTO

This object stores information for assignments with built in variability. The variability features of Graja are quite new and might change frequently.

Variability is a concept that allows giving each student an individual variant of an assignment. That way we possible can reduce cheating.

Basically an assignment could either serve as a template or as an instance. Only instance assignments can be used for grading. Templates first must be instantiated. The VarSpec object specifies a template, the CVr object specifies an instance. The term CVr means composite variation resolution (composite, because we potentially have several variation points composed in a vector). VariabilityInfoTO can reference exactly one of these two objects.

A variation template defines so-called variation points (Vp) that are bundled in a so-called composite variation point (CVp). Variation point values are called variants (V). The variants chosen for each Vp might depend on each other. All valid combinations of variants are described in a CVSpec (composite variation specification). The number of valid variant combinations can be very large, so usually the set of all combinations is not enumerated but described by rules.

Request types

A request object is meant as carrying data from an LMS to Graja as “transfer objects” (suffix “TO”). As a typical request Graja accepts an instance of FrontendSingleRequestTO containing a single assignment of the proforma type (FrontendProformaAssignmentTO). Such a request is shown in the following diagram.

Before going into the details of that diagram, we have a look at the other request types supported by Graja. First Graja distinguishes single and multi requests:

  • a single request is a request to grade a single assignment
  • a multi request is a request to grade mutliple assignments during a single Graja execution run. Multi requests can be used, when students create programs for a worksheet with several assignments on it.

Furthermore Graja distinguishes frontend and backend requests. Frontend requests may point to files in the Graja resource database instead of local files. Also frontend assignments can be in the proforma assignment format. In backend requests resource database references have been resolved to local files, a zipped submission has been unzipped to a local folder, and proforma task.zip files have been broken down into their contents. Furthermore backend requests always concern a single assignment only.

Last but not least Graja distinguishes “proforma assignments” and “plain old assignments” at the frontend. At the Graja frontend proforma assignments are the recommended assignment format while plain old assignments at the frontend are considered legacy.

All requests are inherited from a common base request object. Graja differentiates between requests by checking the object type. The following picture shows the request domain model:

We will now go into the details of the top level RequestTO object.

RequestTO

A RequestTO is the base class of all requests. It carries data that steers the result document generation.

The subclass FrontendRequestTO contains data used for debugging purposes and data that will be interpreted by the client only.

A RequestTO object has the following attributes:

  • submissionNaturalLanguage: specifies the ISO 639-1 code of the language that is used inside the submitted files. This will be used by Graja to detect the encoding of the submitted file. If the submitted file is UTF-8, the language safely can be omitted. If missing, the default is the natural language inside the assignment. In case of a multi request with contradicting assignment natural languages, the default is “en”.

ResultSpecTO

A ResultSpecTO object contains many RDKindTO objects. These objects instruct Graja, what kind of comments (“result documents”) it will create.

Also a ResultSpecTO object has the following attributes:

  • mergeSubresultDocuments: if this is true, then Graja will deliver one ResultNodeTO object per assignments. If false, then Graja will deliver deliver additional ResultNodeTO objects hierarchically per category and per aspect. Default: true.
  • includeScores: This switch is in effect only, if mergeSubresultDocuments is true. In that case some headlines will contain scores only if this switch is true. Default: true.
  • includeConsoleOutput: This switch controls, whether the output of the Graja backend will be included into the teacher’s comments. Default: true.

RDKindTO

A RequestTO object contains many RDKindTO objects. These objects instruct Graja, what kind of comments (“result documents”) it will create. It is a wishlist prepared by the caller. Later in the result object, Graja will return a single document per result node for each single wish in the list.

Graja’s comments may be either scarce or comprehensive. In a RDKindTO object a request defines the kind of document, that it is going to receive.

  • level: is one of possible values of a java.util.logging.Level object. A SEVERE document consists of error messages only. WARNING and INFO provide more detailed information. The levels CONFIG, FINE, FINER, FINEST are reserved for debugging purposes.
  • audience: possible values: TEACHER, STUDENT, BOTH. TEACHER is a label for a document targeted to the teacher, STUDENT targets the student, BOTH is a document that contains all the details intended for student and teacher.
  • representation: possible values: html, plain

DebugSwitchesTO

  • keepStarterWorkspace: if true, the folder where the submission gets unzipped and where a temporary security policy file is generated, won’t be deleted after grading completed
  • keepBackendWorkspace: if true, the workspace folder, where the student’s program is executed, won’t be deleted after grading completed
  • javaSecurityDebug: a JVM -D switch passed to Graja on backend startup, that can be used to debug security issues
  • agentlibJdwp: a JVM switch passed to Graja on backend startup, that can be used to debug Graja remotely
  • overrideMaxRuntimeSecondsWallclockTime: if set, this overrules any runtime settings in the assignments. The specified value will be used for every single assignment. Hence, a multi request will get at most “num of assignments” times “overrideMaxRuntimeSecondsWallclockTime” seconds to complete.

ClientDataTO

This class can transport any data as key value pairs to Graja, that the caller wishes to get back as part of the result. Graja does not interpret nor process the client data. It simply attaches the client data to the result.

ResourceTO

This is an abstract base class for all resources

ResourceLocalFileTO

This object has a single attribute:

  • path: the (usually absolute) path to the local file

IdentifiableResourceTO

Graja can reuse resources. Therefore Graja will install resources together with an id for example on the local file system. An IdentifiableResourceTO object references such a resource installed by Graja.

FrontendRequestTO / BackendRequestTO

The usual interface to communicate with Graja is the frontend interface. Frontend and backend interfaces differ in the following aspects:

  • At the frontend any files can be either local files or references to the Graja Resource database. Graja can reuse resources. Therefore Graja will install resources together with an id for example on the local file system. A file object referenced as an IdentifiableResourceTO references such a resource installed by Graja.
  • At the frontend there can be assignments in Proforma task.zip format. This is not possible at the backend. I. e. the frontend module is responsible for unpacking a Proforma assignment and to generate a plain old backend assignment.
  • At the frontend the submission can be passed as a zip file. The backend expects the submission expanded to a local folder.
  • The frontend can process a multi request. The backend grades a single assignment only.
  • The frontend handles client data (ClientDataDO).
  • The frontend accepts several debug switches.

The FrontendRequestTO has a hierarchical structure. It can be single request or a multi request.

In a FrontendRequestTO we have a submission resource. Usually this will be a local file (zip or plain text) and not an IdentifableResourceTO (but it could be). In a BackendRequestTO the submission always is a local folder.

The following diagrams show frontend request and backend request with the details contained:

FrontendSingleRequestTO / BackendRequestTO

This object consists of a single FrontendAssignmentTO or BackendAssignmentTO object.

FrontendAssignmentTO

This could be either a FrontendPlainOldAssignmentTO or a FrontenProformaAssignmentTO object.

FrontendPlainOldAssignmentTO / BackendAssignmentTO

A FrontendPlainOldAssignmentTO / BackendAssignmentTO object has associations with several resources. At the frontend Graja can handle local files or identifiable resources. At the backend all resources must be local files. There are the following resources in an assignment object:

  • classPath: a classpath with one or more elements. Each element usually is a jar file. This classpath must include all classes, that the Grader class (the AssignmentGrader subclass) is going to use.
  • securityPolicy: the (usually absolute) path to the security policy file, which contains “grantgrader” and/or “grantsubmission” sections.

FrontendProformaAssignmentTO

In this assignment type only a single resource is referenced: a task.zip file in proforma format. Above we described the format of that zip file in more detail. The task.zip contains at least the same information as a FrontendPlainOldAssignmentTO.

FrontendMultiRequestTO

In contrast to a single request this object contains many assignment objects. Each assignment is labelled with a key (see FrontendKeyedAssignmentTO).

Moreover a multi request defines the following attribute:

  • expectedRootFolderInSubmissionZip: a folder name that Graja will search for as the root folder in the submission zip file. Multi assignment submissions always are expected as a zip file.

FrontendKeyedAssignmentTO

This is an assignment object together with an additional attribute to identify an assignment within a multi request:

  • key: a multi request may have several assignments. Every assignment inside the request gets a unique key (unique with respect to its siblings).

AssignmentSettingsTO

An AssignmentSettingsTO serves as a bundle of several configuration options that are common in frontend and backend assignments. The AssignmentSettingsTO object is a container for almost all configuration data of a Graja assignments. Many of these configuration options (e. g. classes AssignmentMetaDataTO, GrajaMetaDataTO, ComputingResourcesTO, SubmissionRulesTO, ModuleConfigurationTO, SubmoduleConfigurationTO) is also contained in a ProFormA task file, but in a different order and structure. The AssignmentSettingsTO object is not part of a ProFormA task, but the referenced objects are. In the class diagram above the packages ending with “extendproforma” hold classes that are also contained in a ProFormA task.

Attributes of an AssignmentSettingsTO object are:

  • javaVersion: the minimum Java version needed for the assignment. If missing, the minimum java version is assumed 8. The value of this attribute has the same meaning as the attribute version of the ProFormA ProgLang class. ProgLang is used in the task.zip format, AssignmentSettings is used in internal and legacy external formats.
  • naturalLanguage: an ISO 639-1 string specifying the natural language of the assignment’s description. Maps to the lang attribute of the ProFormA task root element.
  • maxSubmissionSizeBytes: specifies the maximum size of a submission. Maps to ProFormA’s RegExpRestriction.maxSize.
  • title: the title of the assignment. Maps to ProFormA’s MetaData.title.
  • description: The assignment’s HTML formatted description as seen by students. Maps to ProFormA’s Task.description.
  • internalDescription: HTML formatted content seen by teachers. Maps to ProFormA’s GradingHints->InternalDescriptionTO.

ModulesTO / ModuleTO

A ModulesTO object stores the configurations of the modules that should be started when grading the assignment. Each module has an id, a title and possibly a specific configuration. Currently there are five modules: Compile, JUnit, PMD, Checkstyle, and Human.

Each module has an id and a title:

  • id: a module’s id. This is equivalent to the ProFormA Test object’s id attribute for Tests of test-type=graja-module-*.
  • title: a module’s title. This is equivalent to the ProFormA Test object’s title attribute for Tests of test-type=graja-module-*. Currently we often use titles like “Compilation” or “JUnit dynamic analysis” or “PMD static analysis” or “Checkstyle static analysis” or “Manually assigned scores”.

Also each module references a ModuleConfigurationTO object, that is also used as TestConfiguration in a ProFormA Test.

GradingSchemeTO

This object corresponds to the ProFormA Test object of testType=graja-grading-root (id=graja-root) and also the root TestGroupTO object below the ProFormA GradingHints. It serves as a container for a collection of GradingCategoryTO objects. The grading scheme stores the total maximum score:

  • id (inherited from GradingNodeTO): a mandatory identifier that is unique among all identifiers of objects of subclasses of GradingNodeTO.
  • title (inherited from GradingNodeTO): an optional headline.
  • scoreMax (inherited from GradingNodeTO): If the scoreMax value is given, it must be consistent with the child grading category’s maximum scores. If the value is missing, Graja will itself sum up the scores of the child grading categories.
  • description (inherited from GradingNodeTO): an optional HTML formatted description shown to student. This description targets the overall grading scheme. Every grading category and every grading aspect has it’s own description for the details.
  • internalDescription (inherited from GradingNodeTO): an optional HTML formatted description shown to teachers.

GradingCategoryTO

This object corresponds to a ProFormA Test object of testType=graja-grading-category. There can be several categories in a single assignment. A category is a headline such as “Functional correctness” or “Maintainability” together with its maximum score.

  • id (inherited from GradingNodeTO): a mandatory identifier that is unique among all identifiers of objects of subclasses of GradingNodeTO.
  • title (inherited from GradingNodeTO): the headline.
  • scoreMax (inherited from GradingNodeTO): If the scoreMax value is given, it must be consistent with the child grading aspect’s maximum scores. If the value is missing, Graja will itself sum up the scores of the child grading aspects.
  • description (inherited from GradingNodeTO): an optional HTML formatted description shown to student.
  • internalDescription (inherited from GradingNodeTO): an optional HTML formatted description shown to teacher.

GradingAspectTO

This class stores a title of a grading aspect, the maximum score, and possibly more specific configuration data:

  • id (inherited from GradingNodeTO): a mandatory identifier that is unique among all identifiers of objects of subclasses of GradingNodeTO.
  • title (inherited from GradingNodeTO): the aspect’s headline, describing the purpose of the aspect. A title is optional. Missing titles are replaced by standard titles, e. g. built from the corresponding submodule.
  • scoreMax (inherited from GradingNodeTO): maximum score awarded by the aspect. If the score is missing, Graja assumes a maximum score of 1.
  • description (inherited from GradingNodeTO): an optional HTML formatted description shown to student.
  • internalDescription (inherited from GradingNodeTO): an optional HTML formatted description shown to teacher.

Also each grading aspect references a SubmoduleConfigurationTO object, that is also used as TestConfiguration in a ProFormA Test.

In terms of the example above a GradingAspectTO combines the data of a green node at the top and the data of a brown submodule node on the left. In the marshalled form this looks like this:

<graja:gradingCategory id="cat_1" title="Maintainability" scoreMax="25.0">
    <graja:pmdAspect id="asp_1_1" moduleRefId="pmd1" title="Variable naming conventions" scoreMax="2" rule="VariableNamingConventions"/>
    <graja:pmdAspect id="asp_1_2" moduleRefId="pmd2" title="Method naming conventions" scoreMax="2" rule="MethodNamingConventions"/>
    ...
    <graja:junitAspect id="asp_1_3" title="Attributes in class Fraction should be private" scoreMax="5.0" method="org.myinstitution.myassignments.fractionv01.grader.Grader#attributesShouldBePrivate"/>
    <graja:humanAspect id="asp_1_4" title="Readability" scoreMax="10">
      <graja:description>A grading assistant will manually assign scores for readability of your code.</graja:description>
    </graja:humanAspect>
</graja:gradingCategory>

Seen this way a GradingAspectTO blurs the border between submodules and grading aspects. Currently every submodule configuration is not more than a simple identifier (a JUnit method name or a PMD rule name or a Checkstyle check name). We use this singular configuration data both as a configuration of the submodule and as an id referenced by a grading aspect. If future modules included into Graja will demand submodule configurations that consist of more than a simple identifier, then we will break up the marshalling of a grading aspect into its pure grading data (scoreMax, title) and its referenced submodule configuration as a separate object.

Implications for submissions

If Graja receives a single request, the submitted file could be plain text file (usually a java source file) or a zip file containing several files. A zip file is expected to have the following content:

  • The solution files are located at the root of the archive. If a solution itself is divided into subdirectories, these subdirectory entries are rooted at the root of the zip archive.
  • This is also the expected content of the expanded submission folder passed to the backend.

A multi request assumes a zip file of the following content:

  • The archive has a single root folder named expectedRootFolderInSubmissionZip.
  • Below that there are one or more subdirectories, each one containing the solution for a single assignment. The names of the subdirectories are equal to the key attribute of the respective FrontendKeyedAssignmentTO.
  • Below that the solution files are expected. If a solution itself is divided into subdirectories, every such subdirectory entries are rooted at <expectedRootFolderInSubmissionZip>/<key>/<subdirectory>.

Result

The following picture shows the result domain model:

ResultTO

The result transfer object first contains a copy of the client data received as part of the request.

Attributes of a ResultTO are:

  • grajaVersion: the version number (usually a date/time tag) of Graja
  • error: will be true, if the submission could not be graded in detail. We refer to this as a “class level failure”, where the submission either could not be unzipped or compiled or where the Grader was not able to call individual methods of the submission. The reason might be a wrong class name in the submission, a wrong package, but the reason might also be an error in the Grader itself. When the Grader was able to call individual methods of the submission and some or all methods behaved wrong, the Grader assigns less than 100%, but this not an “error”, i. e. the attribute error gets false.
  • messagePreformattedPlain: Here Graja may include error messages, that usually are targeted to the teacher or the administrator of the software installation. Examples are: Graja cannot find some configuration files, Graja cannot find a bash, Graja cannot find the request file passed as parameter to CmdlineStarter, etc.

The Graja backend always delivers an object of type SingleResultTO.

The Graja frontend usually delivers an object of type SingleResultTO, when it received a FrontendSingleRequestTO and it delivers a MultiResultTO as an answer to a FrontendMultiRequestTO. But, if things went severely wrong, Graja might not be able to read the request file, so it doesn’t know, which result object type to deliver. In this case, Graja delivers an instance of ResultTO, the base class.

SingleResultTO

A single result consists of a single AssignmentResultTO object.

AssignmentResultTO

In this object Graja delivers:

  • graderVersion/graderName: information about the grader (date/time tag and the “token” of the Grader).
  • a reference to the root result node with scores and result documents

ResultNodeTO

In this object Graja delivers:

  • id: this id references a corresponding node in the grading scheme of the request, i. e. the respective GradingNodeTO object inside the AssignmentSettingsTO object.
  • scoreAchieved: the achieved score (maximum value is the referenced grading scheme)

Inside a ResultNodeTO Graja includes documents with an explanation of the achieved score. For every requested RDKindTO a single DocumentTO object will be delivered.

Further a ResultNodeTO usually has a hierarchy of subResults that corresponds the the grading scheme hierarchy. If the request’s ResultSpecTO object specified to merge subresults, then there will be only one ResultNodeTO object per assignment and the subResults list will be empty.

DocumentTO

The documents differ in three dimensions:

  • level: legal values are the same as in RDKindTO.
  • audience: see RDKindTO.
  • content: here the document is contained as a byte sequence. The bytes are to be interpreted as plain text (if representation points to a DocReprPlainTextTO object) or as html (if representation points to a DocReprHtmlTO object).

DocRepresentationTO

This object holds information about the representation of the document.

  • If it is an object of class DocReprHtmlTO, the content must be interpreted as UTF-8 encoded text containing a html document.
  • If it is an object of class DocReprPlainTextTO, the content must be interpreted as text that is encoded in the specified charset. Currently Graja always delivers UTF-8.

MultiResultTO

In a multi result Graja delivers a single AssignmentResultTO for every AssignmentTO object from the request. The keys in the KeyedAssignmentResultTO refer to the keys in the request.

Graja will create comments not only for every single assignment, but also some comments that conceptually belong to the whole result. E. g. error messages regarding the unzipping of the submission belong to the whole result. A MultiResultTO contains a set of DocumentTO objects, where each DocumentTO object corresponds to one of the RDKindTO objects in the FrontendMultiRequestTO object.

KeyedAssignmentResultTO

This contains simply an AssignmentResultTO together with a key, that references a corresponding FrontendKeyedAssignmentTO in the request.

Representing a request in XML or JSON

An example of a single request file with a ProFormA assignment in it is the following:

{
  "@class" : ".FrontendSingleRequestTO",
  "resultSpec" : {
    "includeScores" : true,
    "mergeSubresultDocuments" : true,
    "includeConsoleOutput" : true,
    "rdKind" : [ {
      "level" : "SEVERE",
      "audience" : "STUDENT",
      "representation" : "html"
    }, {
      "level" : "INFO",
      "audience" : "STUDENT",
      "representation" : "html"
    }, {
      "level" : "INFO",
      "audience" : "TEACHER",
      "representation" : "html"
    } ]
  },
  "debugSwitches" : {
    "keepStarterWorkspace" : true,
    "keepBackendWorkspace" : true
  },
  "submission" : {
    "@class" : ".ResourceLocalFileTO",
    "path" : "C:\\Users\\john\\Documents\\submission.zip"
  },
  "assignment" : {
    "@class" : ".FrontendProformaAssignmentTO",
    "taskZip" : {
      "@class" : ".ResourceLocalFileTO",
      "path" : "C:\\Users\\john\\Documents\\org.myinstitution.myassignments.dateclassv01.zip"
    }
  }
}

Schema descriptions

There is a helper service in Graja, that prints the XML schema description for requests and results:

  • The xml schema description used inside request.xml and result.xml can be viewed by calling command query_graja_xsd on the Graja CLI.
  • Not functional yet: The json schema used in request.json and result.json can be viewed by calling: java -cp /path/to/GrajaStarter.jar de.hsh.graja.starter.transform.QueryJsonSchema

Converter services

Graja provides converters to create requests and to consume and show results in a human readable form. See the next section Develop assignments for details.

Variability

The variability features of Graja are quite new and might change frequently.

Variability is a concept that allows giving each student an individual variant of an assignment. That way we possible can reduce cheating.

Basically an assignment could either serve as a template or as an instance. Only instance assignments can be used for grading. Templates first must be instantiated.

Data model

Variability information of a template or an instance is contained in the VariabilityInfoTO object described above. A VariabilityInfoTO object is generated by the Graja development tools from a so-called VariabilityProvider. A VariabilityProvider is a subclass of de.hsh.graja.variability.gen.AssignmentVariabilityProvider and implements the template.

A variation template defines so-called variation points (Vp) that are bundled in a so-called composite variation point (CVp). Variation point values (called variants) might depend on each other. All valid combinations of variants are described in a CVSpec (composite variation specification). The number of valid variant combinations can be very large, so usually the set of all combinations is not enumerated but described by rules. The rules are implemented in the VariabilityProvider class. The VariabilityProvider defines the set of variation points together with their data types and it defines the total set of valid variants as something like a union of cartesian products. Some Vp might not be specified by their variants, but instead by a javascript fragment that calculates the possible variants on the fly.

Creating instances from a template

There is a command line service to create an assignment instance from an assignment template:

java -Dde.hsh.graja.cli.logging.level=INFO -jar /path/to/GrajaCli.jar
   --command instantiate_template
   --input /path/to/task_template.zip
   --output /path/to/task.zip

See option --help for more informations.

Currently the automatic build procedure, that converts a descriptor to a ProFormA task, creates a task template and additionally a task instance with default values for the variables. The task instance is named …vr.zip (short for variation resolution).

The GrajaGui (see next section) let’s the user interactively choose variants for all variation points. When starting Graja from the Graja GUI, a new task instance will be generated for the selected variants.

Example

In SampleGrajaAssignments there is an example (vowelsv01), that demonstrates a few features of variability. More examples are currently being prepared.

Installation ←    → Develop assignments