DDSL - Data Domain Specific Language

Introduction
Getting Started
Writing Apps
Versioning
Contributing Code
License

Introduction

DDSL provides a way to work with strongly typed data in Lua, which itself is dynamically typed and does not enforce data structure constraints.

Here is a quick illustration using a ShapeType datatype.

IDL:

struct ShapeType {
    long x;
    long y;
    double shapesize;
    string<128> color; //@Key
};

DDSL:

local ShapeType = xtypes.struct{
    ShapeType = {
      { x = { xtypes.long } },
      { y = { xtypes.long } },
      { shapesize = { xtypes.long } },
      { color = { xtypes.string(128), xtypes.Key } },
    }
  }

An instance created from a DDSL datatype will have a form that adheres to the the underlying datatype.

 local shape = xtypes.new_instance(ShapeType)

The shape instance of ShapeType will have the default value of:

shape = {
    x           = 'x',
    y           = 'y',
    shapesize   = 'shapesize',
    color       = 'color'
}

The fields can be used to retrive the instances values from some storage system. The are correctly constructed for deeply nested datatypes.

The new instance can be used to store a shape object:

-- Fields in the underlying datatype are allowed:
shape.color     = 'GREEN'
shape.shapesize = 44
shape.x         = 99
shape.y         = 77

-- But, fields NOT in the datatype are NOT allowed:
shape.z         = 111  -- will get dropped
assert(shape.z == nil) -- 'z' not allowed

The shape instance can be queried for the underlying datatypes of its fields:

-- color:
assert(shape('color')[1] == xtypes.string(128))
print(table.unpack(shape('color')) -- string<128>   @Key

-- number of members
print(#shape) -- 4

We can iterate through all the member names, values, and datatypes:

for name, value in pairs(shape) do
  --    name  value   :   datatype
  print(name, value, ':', table.unpack(shape(name)))
end

DDSL brings the following capabilities to Lua (and therefore to platforms where Lua can be embedded).

DDSL is a language for describing datatypes in Lua. It can be used as replacement for data description in an IDL. In particular:
DDS X-Types can be defined in DDSL.
Datatypes can be imported from XML.
Datatypes can be exported to IDL.

Unlike static formats (such as IDL or XML or JSON), DDSL brings the full power of the Lua programming language to constructing and manipulating datatypes. The datatypes form a connected graph whose integrity is maintained as new datatypes are defined, existing ones removed, or altered.

DDSL provides a way to create instances from datatypes. The datatype structural constraints are enforced on the instances. For example, only fields that are allowed by the datatype can be present in an instance. Collection bounds are enforced.
Note that for efficiency reasons, non-structural constraints are not enforced on the instances. Thus, an instance field whose underlying datatype is a boolean can be assigned a string value. However, such constraints can easily be enforced by user code, if so desired.
DDSL provides a way to modify the datatypes dynamically at anytime. All the aspects of a datatype, except its KIND can be changed. For example, datatype members can be added, removed, or their datatype changed. The datatype name, enclosing scopes (namespaces) can be altered. This makes DDSL ideal for datatype modeling, synthesis and transformation.
DDSL keeps all the instances of a datatype in "sync" with the datatype. Thus, if a member is removed from a datatype, the corresponding field is removed from all the instances of the datatype. When a new member is added, all the instances are updated with the new field--- initialized to a default value. If a member's datatype is structurally changed, the corresponding field is reset to the default value for the new structure.
The default value of a field in an instance is a dot ('.') separated string formed by navigating to that field from the instance. The default value can be used as an index into some storage system.
Datatypes can be introspected, examined, and traversed. For example, enumerations can be be looked by name or ordinal values. Collection bounds can be looked up, aliases can be resolved, and so on. This makes it easy to work with structured data in a dynamic language such as Lua, or any environment where Lua can be embedded (almost every platform).
All of the above make DDSL ideally suited for for writing generators. In particular:
Data generators that produce instances confirming to some data space or data generation rules/constraints, while adhering to an underlying datatype.
Code generators that produce behavior the context of some operational scenario, while adhering to an underlying datatype.
DDSL is designed to integrate nicely with the RTI Connector for Connext DDS which provides a quick and easy way to access the power and functionality of RTI Connext DDS from a variety of different scripting languages including JavaScript, Python and Lua. RTI Connext DDS is a popular software Connectivity Databus for the Industrial Internet of Things based on the Data Distribution Service, a.k.a. DDS open standard for IoT data connectivity.

Datatype Algebra Diagram

Getting Started

Minimum requirement: Lua 5.1+

# check the installed lua version
lua -v

Install an updated Lua version if needed.

Get DDSL

#--- Download the latest bundle:
wget http://rticommunity.github.io/rticonnextdds-ddsl/ddsl.zip
#--- unzip the release:
unzip ddsl.zip   # This will create a directory: ddsl/
#--- We refer to the location of the release as DDSL_HOME, e.g.
cd /path/to/ddsl/
export DDSL_HOME=$(pwd -P)

Read the ddsl.xtypes module overview.
Browse the ddsl.xtypes API and usage examples.
Setup Lua's package.path to include lib/lua. Assuming DDSL_HOME is the location of the directory where you installed DDSL, set the LUA_PATH environment variable as follows.

export LUA_PATH+="${DDSL_HOME}/lib/lua/?.lua;${DDSL_HOME}/lib/lua/?/init.lua;"
export LUA_PATH+="${DDSL_HOME}/lib/lua/?.lc;${DDSL_HOME}/lib/lua/?/init.lc;"

Step through the tutorial examples.

cd tutorial/
export LUA_PATH+="${DDSL_HOME}/tutorial/?.lua;"
lua ddsl_tutorial.lua

Try out the scripts: xml2idl.

cd test/
bin/run xml2idl xml-test-simple.xml

or, with debugging on:

bin/run xml2idl -d xml-test-simple.xml

Look at the the unit tests for more advanced examples.

Writing Apps

Create datatypes directly using the ddsl.xtypes module.

require 'ddsl.xtypes'

OR, Import datatypes from XML using the ddsl.xtypes.xml module.

require 'ddsl.xtypes.xml'

Output IDL using the ddsl.xtypes.utils module.

require 'ddsl.xtypes.utils'

Create instances and use them in application code. Use the logger logger module to log messages at different verbosity levels.

require 'logger'

Versioning

DDSL uses semantic versioning.
Annotated tags specify the release version numbers.
The master branch is always the latest stable version.
The develop branch is the work in progress.

Contributing Code

Ensure dependencies are available on the development host
Lua 5.1+
LDoc 1.4.3+ for documentation

Install the above, if not present on your system.

Fork and clone the repository.
Setup the client side git hooks in you local clone of the repository (fork)

# setup client side hooks
cd .git/hooks/
ln -s ../../build/scripts/pre-commit.lua pre-commit

[OPTIONAL] Build the public API documentation.

cd doc/
ldoc .
# Browse the output: out/html/index.html
open out/html/index.html

[OPTIONAL] Build all the documentation, both public and private. This may be helpful if you intend to create new type-systems or data model, or just want to understand the inner workings.

cd doc/
ldoc . -all
# Browse the output: out/html/index.html
open out/html/index.html

Review the Documentation Conventions and Module Organization.
[OPTIONAL: Eclipse Users] If you use the Eclipse IDE, Import... the rticonnextdds-ddsl project located in build/eclipse.

#--- Eclipse > File > Import... > Existing Projects into Workspace
build/eclipse/

Eclipse Lua Development Tools (LDT) provides an excellent IDE for editing Lua code.

Setup Lua's package.path to include src/ (before lib/lua). Assuming DDSL_HOME is the location of the directory where you cloned DDSL, set the LUA_PATH environment variable as follows.

export LUA_PATH+="${DDSL_HOME}/src/?.lua;${DDSL_HOME}/src/?/init.lua;"
export LUA_PATH+="${DDSL_HOME}/lib/lua/?.lua;${DDSL_HOME}/lib/lua/?/init.lua;"
export LUA_PATH+="${DDSL_HOME}/lib/lua/?.lc;${DDSL_HOME}/lib/lua/?/init.lc;"

Add/Modify/Update Code
Add unit tests
Pass all the unit tests

cd test/
lua ddsl-xtypes-tester.lua

Update CHANGELOG.md to add a section describing the contribution

edit CHANGELOG.md

Create a candidate build locally.

./build/scripts/new-build.sh
# Output goes to out/{bin,lib, html}.

Spot check the candidate build.

cd test/
../out/bin/run xml2idl xml-test-connector.xml
# Browse the documentation: out/html/index.html
open out/html/index.html

If everything looks great, send a pull request; otherwise continue iterating on the previous steps.

Create a new release.

# update the version number and tag the sources
./build/scripts/new-release.lua

License

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

DDSL

Contents

Topics

Modules

Scripts