CARML

A pretty sweet RML engine

Table of Contents

Introduction
Getting started
Reactive streams
Support for Apache Jena
Input stream extension
Function extension
XML namespace extension
CARML in RML Test Cases
Projects using CARML
About CARML

Introduction

CARML is a java library that transforms structured sources to RDF as declared in an RML mapping, in accordance with the RML spec.

The best place to start learning about RML is at the source, but basically RML is defined as a superset of R2RML which is a W3C recommendation that describes a language for expressing mappings from relational databases to RDF datasets. RML allows not only the expression of mappings for relational databases, but generalizes this to any structured source. All you need is a way to iterate over and query the source.

NOTE: If you have questions about RML in general, the best place to ask them is at RML questions.

Getting started

CARML is available as a runnable jar with CLI: CARML jar.

CARML is also available from the Central Maven Repository.

<dependency>
    <groupId>io.carml</groupId>
    <artifactId>carml-engine</artifactId>
    <version>${carml.version}</version>
</dependency>

    <!-- Choose the resolvers to suit your need -->
<dependency>
    <groupId>io.carml</groupId>
    <artifactId>carml-logical-source-resolver-jsonpath</artifactId>
    <version>${carml.version}</version>
    </dependency>
<dependency>
    <groupId>io.carml</groupId>
    <artifactId>carml-logical-source-resolver-xpath</artifactId>
    <version>${carml.version}</version>
</dependency>
<dependency>
    <groupId>io.carml</groupId>
    <artifactId>carml-logical-source-resolver-csv</artifactId>
    <version>${carml.version}</version>
</dependency>

Example usage:

Set<TriplesMap> mapping = RmlMappingLoader.build()
    .load(RDFFormat.TURTLE, Paths.get("path-to-mapping-file"));

RdfRmlMapper mapper = RdfRmlMapper.builder()
    // add mappings
    .triplesMaps(mapping)
    // Add the resolvers to suit your need
    .setLogicalSourceResolver(Rdf.Ql.JsonPath, JsonPathResolver::getInstance)
    .setLogicalSourceResolver(Rdf.Ql.XPath, XPathResolver::getInstance)
    .setLogicalSourceResolver(Rdf.Ql.Csv, CsvResolver::getInstance)
    //-- optional: --
    // specify base IRI to use for relative IRIs in mapping results
    // default is "http://example.com/base/"
    .baseIri("http://foo.bar")
    // specify IRI unicode normalization form (default = NFC)
    // see http://www.unicode.org/unicode/reports/tr15/tr15-23.html
    .iriUnicodeNormalization(Form.NFKC)
    // set file directory for sources in mapping
    .fileResolver("/some/dir/")
    // set classpath basepath for sources in mapping
    .classPathResolver("some/path")
    // specify casing of hex numbers in IRI percent encoding (default = true)
    // added for backwards compatibility with IRI encoding up until v0.2.3
    .iriUpperCasePercentEncoding(false)
    // Specify a custom value factory supplier
    .valueFactorySupplier(ValidatingValueFactory::new)
    //---------------

    .build();

Model result = mapper.mapToModel();

Reactive Streams

CARML leverages Project Reactor's implementation of reactive streams to achieve streaming and (potentially) non-blocking processing of mappings.

CARML exposes Reactor's Flux data structure. When you execute a RML mapping using one of the Flux returning methods you get a Flux<Statement> as result. This allows for further processing with the many reactive operators available for Flux.

Flux<Statement> statements = mapper.map();
// do some operations
Model result = statements.collect(ModelCollector.toModel())
    .block();

Support for Apache Jena

As CARML is built on RDF4J, the default output is either a Flux of RDF4J Statements, or an RDF4J Model.

However, CARML provides utilities to transform this output to equivalent Jena datastructures.

To use these utilities one needs to import the io.carml.carml-converters-jena dependency.

<dependency>
    <groupId>io.carml</groupId>
    <artifactId>carml-converters-jena</artifactId>
    <version>${carml.version}</version>
</dependency>

The following example shows the streaming transformation of RDF Statements to Jena Quads and subsequent collection into a Jena Dataset using the provided utilities.

Flux<Statement> statements = mapper.map();

Dataset jenaDataset = statements.map(JenaConverters::toQuad)
    .collect(JenaCollectors.toDataset());

Input stream extension

When it comes to non-database sources, the current RML spec only supports the specification of file based sources in an rml:LogicalSource. However, it is often very useful to be able to transform stream sources.

To this end CARML introduces the notion of 'Named Streams'. Which follows the ontology defined here.

So now, you can define streams in your mapping like so:

:SomeLogicalSource
  rml:source [
    a carml:Stream ;
    # NOTE: name is not mandatory and can be left unspecified, when working with a single stream
    carml:streamName "stream-A" ;
  ];
  rml:referenceFormulation ql:JSONPath;
  rml:iterator "$"
.

Then the input stream can be mapped by providing a map of named input streams.

RdfRmlMapper mapper = RdfRmlMapper.builder()
    .triplesMaps(mapping)    
    .setLogicalSourceResolver(Rdf.Ql.JsonPath, JsonPathResolver::getInstance)
    .build();

mapper.map(Map.of("stream-A", inputStream));

Note that it is possible to map several input streams. When combining named input streams with an unnamed input stream, the constant RmlMapper.DEFAULT_STREAM_NAME can be used as the name for the unnamed input stream.

RdfRmlMapper mapper = RdfRmlMapper.builder()
    .triplesMaps(mapping)
    .setLogicalSourceResolver(Rdf.Ql.JsonPath, JsonPathResolver::getInstance)
    .build();

mapper.map(Map.of("stream-A", inputStreamA, RmlMapper.DEFAULT_STREAM_NAME, defaultInputStream));

Function extension

A recent development related to RML is the possibility of adding functions to the mix. This is a powerful extension that increases the expressivity of mappings immensely. Do note that the function ontology is still under development at UGhent.

Because we believe that this extension can already be of great value, we've already adopted it in CARML.

The way it works is, you describe the execution of a function in terms of the Function Ontology (FnO) .

Take for example the SumFunction example of the FnO spec. This defines an instance ex:sumFunction of class fno:Function that is able to compute the sum of two values provided as parameters of the function at execution time.

It also describes an instance ex:sumExecution of fno:execution, which fno:executes ex:sumFunction which descibes an instance of an execution of the defined sum function. In this case with parameters 2 and 4.

To be able to use this in RML mappings we use executions of instances of fno:Function to determine the value of a term map. The execution of a function can be seen as a post-processing step in the evaluation of a term map.

@prefix rr: <http://www.w3.org/ns/r2rml#> .
@prefix rml: <http://semweb.mmlab.be/ns/rml#> .
@prefix fnml: <http://semweb.mmlab.be/ns/fnml#> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
@prefix fno: <https://w3id.org/function/ontology#> .
@prefix ex: <http://example.org/> .

ex:sumValuePredicateObjectMap
  rr:predicate ex:total ;
  rr:objectMap [
    a fnml:FunctionMap ;
    fnml:functionValue [
      rml:logicalSource ex:LogicalSource ;
      rr:subjectMap [
        rr:template "functionExec";
        rr:termType rr:BlankNode ;
        rr:class fno:Execution
      ] ;
      rr:predicateObjectMap
        [
          rr:predicate fno:executes ;
          rr:objectMap [
            rr:constant ex:sumFunction ;
          ]
        ] ,
        [
          rr:predicate ex:intParameterA ;
          rr:objectMap [ rml:reference "foo" ]
        ] ,
        [
          rr:predicate ex:intParameterB  ;
          rr:objectMap [ rml:reference "bar" ]
        ]
    ] ;
    rr:datatype xsd:integer ;
  ]
.

A function can be created in any .java class. The function should be annotated with @FnoFunction, providing the string value of the function IRI, and the function parameters with @FnoParam, providing the string value of the function parameter IRIs.

public class RmlFunctions {

  @FnoFunction("http://example.org/sumFunction")
  public int sumFunction(
      @FnoParam("http://example.org/intParameterA") int intA,
      @FnoParam("http://example.org/intParameterB") int intB
  ) {
    return intA + intB;
  }

}

The class or classes containing the annotated functions can then be registered on the mapper via the RmlMapper#addFunctions method.

RdfRmlMapper mapper = RdfRmlMapper.builder()
    .triplesMaps(mapping)
    .setLogicalSourceResolver(Rdf.Ql.JsonPath, JsonPathResolver::getInstance)
    .addFunctions(new YourRmlFunctions())
    .build();

Model result=mapper.mapToModel();

It is recommended to describe and publish new functions in terms of FnO for interpretability of mappings, and, possibly, reuse of functions, but it's not mandatory for use in CARML.

Note that it is currently possible to specify and use function executions as parameters of other function executions in CARML, although this is not (yet?) expressible in FnO.

XML namespace extension

When working with XML documents, it is often necessary specify namespaces to identify a node's qualified name. Most XPath implementations allow you to register these namespaces, in order to be able to use them in executing XPath expressions. In order to convey these expressions to the CARML engine, CARML introduces the class carml:XmlDocument that can be used as a value of rml:source. An instance of carml:XmlDocument can, if it is a file source, specify a location via the carml:url property, and specify namespace declarations via the carml:declaresNamespace property.

For example, given the following XML document:

<?xml version="1.0" encoding="UTF-8"?>
<ex:bookstore xmlns:ex="http://www.example.com/books/1.0/">
    <ex:book category="children">
        <ex:title lang="en">Harry Potter</ex:title>
        <ex:author>J K. Rowling</ex:author>
        <ex:year>2005</ex:year>
        <ex:price>29.99</ex:price>
    </ex:book>
</ex:bookstore>

one can now use the following mapping, declaring namespaces, to use them in XPath expressions:

@prefix rr: <http://www.w3.org/ns/r2rml#>.
@prefix rml: <http://semweb.mmlab.be/ns/rml#>.
@prefix ql: <http://semweb.mmlab.be/ns/ql#> .
@prefix carml: <http://carml.taxonic.com/carml/> .
@prefix ex: <http://www.example.com/> .

<#SubjectMapping> a rr:TriplesMap ;
  rml:logicalSource [
    rml:source [
      a carml:Stream ;
      # or in case of a file source use:
      # carml:url "path-to-source" ;
      carml:declaresNamespace [
        carml:namespacePrefix "ex" ;
        carml:namespaceName "http://www.example.com/books/1.0/" ;
      ] ;
    ] ;
    rml:referenceFormulation ql:XPath ;
    rml:iterator "/ex:bookstore/*" ;
  ] ;

  rr:subjectMap [
    rr:template "http://www.example.com/{./ex:title}" ;
    rr:class ex:Book ;
    rr:termType rr:IRI ;
  ] ;
.

which yields:

<http://www.example.com/Harry%20Potter> a <http://www.example.com/Book> .

CARML in RML Test Cases

See the RML implementation Report for how CARML does in the RML test cases.

Note: currently we've raised issues for some of the test cases which we believe are incorrect, or have an adverse effect on mapping data.

Projects using CARML

These projects are using CARML.

carml-service - A project by Zazuko which creates a web service around CARML.
barnard59 - A project by Zazuko which provides a toolkit to automate extract, transform and load (ETL) tasks. Its main focus is on creating Linked Data.
DotWebStack - A Framework for publishing rich data services. DotWebStack provides an extension to provide support for Linked data URI dereferencing, which uses CARML to transform data on the fly.

Are you using CARML? Let us know, so we can add you to this list. Or better yet, open a PR.

About CARML

CARML was first developed by Taxonic in cooperation with Kadaster . And is now being maintained and developed further by Skemu.

License

carml/carml

About

Topics

Resources

License

Stars

Watchers

Forks

Releases 23

Packages 0

Contributors 10

Languages

Packages