TAME – Key Features – Schema and Entities

In the previous post we discussed the features that we would like to have in a modern day MVC framework. In this post let us discuss some of the key problems in designing such a framework and how we have approached it in TAME.

Data is stored in storage devices as bits. However humans don’t quite understand this, so have come up with mechanisms to store and retrieve data from a storage device using a high level interface. This high level interface is the notion of files and on top of files, we have built the notion of databases.

A database makes this storage and retrieval more user friendly rather than working with raw files. Databases are stores of “entities”. Entities may be related to other entities. Interaction with the database involves performing CRUD operations on the entities.

Entities contain fields. Fields have data types. Data types can be broadly categorized as “primitive data types” and “composite data types“. When considering relational databases, we normalize data into primitive data types. Composite data types are handled using primary and foreign key relationships across multiple tables. However, in a NoSQL store, we may store composite data as a sub-record within a single record itself and save an additional join.

So we start our discussion by asking, what are the minimum set of primitive and composite data types we need? If we do a comparative analysis of different programming languages and database technologies, we will soon realize that almost all languages have some way of representing strings, numbers and booleans. There are many other primitive types, but these are the minimum. The reasoning behind why this is so, is beyond the scope of this post.

Composites are ways of grouping data together. Composites are made up of primitives and other composites. At a minimum, we need one composite data type that has sequential properties (list like) and at least one that can be used to store non-sequential (key/value) form of data (map like).

In order to perform CRUD operations on entities, MVC frameworks expect you to first come up with the model definitions. The frameworks then provide you with a default CRUD form to interact with these models using an ORM. How about when we are working with NoSQL database? Or, in general, how can we define a “generic schema” which can work with any kind of database backend (relational or NoSQL or mixed)?

The JSON data format is generic enough to handle the above cases of primitive and composite data types, and it is widely supported in almost all popular languages. So, in TAME, we came up with a mechanism to define the schema of an entity using JSON as the data format.

In this post, we shall be using the term “entity” to refer to a single “table” of a relational database (or the equivalent of collections of MongoDB). We shall use the term “instance” to refer to a “record” in a database (or the equivalent of documents in MongoDB).

Further, since JSON is serializable, we can also store this schema in a database and this will help us with a whole lot of things like modifying schemata without requiring a redeployment (DDL becomes DML when schema is also considered as data), schema versioning and schema migrations.

Let us take an example:

Consider the following page showing testimonials written by various learners who have attended jnaapti’s training in the past.

If this data were stored in a database, what would be the fields of the entity called “testimonial”?

Every testimonial has a description, an author, and some of the author’s experience details like the company he is a part of and his designation.

In a relational database world we store this in a table and each testimonial is a record. The company details could be a separate table. In a NoSQL store we can store the testimonial along with all related entities in a single record.

In a relational database we typically start by defining tables and then start adding records. In a NoSQL database, defining the schema is not mandatory. However, it is preferred to define a schema for the purposes of validation, automatic form creation and all the benefits mentioned above.

So let us assume that defining a schema is a good thing! We now need a language for the schema that is generic enough to support primitive data types and composite types.

In TAME we provide mechanisms to add entities via a Schema Editor. Here is what it looks like:

Define the fields that are a part of the testimonial entity. TAME now generates a schema and stores it in the database.

The schema stored in the database looks like this:

{
  "name": "testimonial",
  "fields": [
    {
      "name": "description",
      "type": "String"
    },
    {
      "name": "author",
      "type": "String"
    },
    {
      "name": "experience",
      "type": "Object",
      "schema": {
        "fields": [
          {
            "name": "company",
            "type": "String"
          },
          {
            "name": "designation",
            "type": "String"
          }
        ]
      }
    }
  ]
}

A simple way to understand the above is:

  • testimonial is the name of the entity
  • It has 3 fields
    • description of type String
    • author of type String
    • experience of type Object (composite type) – which in turn has 2 fields:
      • company of type String
      • designation of type String

Once a schema is added, we can now start performing CRUD on instances of the entity.

A basic UI for this would be somewhat like as shown below:

The JSON structure of the created instance would look like this:

{
 "description": "The examples were concise and very useful. The slide material was good. We had very specific requirements for the training and Gautham was able to deliver what we were looking for. The training was well structured.",
 "author": "Rajesh",
 "experience": {
 "company": "ACME Corp",
 "designation": "Senior Product Engineer"
 }
}

Note that the JSON structure has no extra bells and whistles. So this makes it easy for us to import existing entities into TAME. We also have a schema deducer that can help us define schema for existing entities. This allows us to consume data from a REST service and define a schema for it and we can then start performing CRUD operations on that entity in less than 5 minutes!

Suppose we also want to store a list of all the technologies that we trained this person on as part of each testimonial.

{
  "name": "testimonial",
  "fields": [
    {
      "name": "description",
      "type": "String"
    },
    {
      "name": "author",
      "type": "String"
    },
    {
      "name": "experience",
      "type": "Object",
      "schema": {
        "fields": [
          {
            "name": "company",
            "type": "String"
          },
          {
            "name": "designation",
            "type": "String"
          }
        ]
      }
    },
    {
      "name": "technologies",
      "type": "ArrayOfStrings"
    }
  ]
}

This will now show up in our editor as follows

And here is the corresponding instance in the database

{
  "description": "Training was awesome.",
  "author": "Jay",
  "experience": {
    "company": "ACME Corp",
    "designation": "Product Engineer"
  },
  "technologies": [
    "HTML5",
    "JavaScript",
    "CSS3"
  ]
}

As you can see above, TAME allows us to add denormalized composite types. TAME has Array and Object Renderers for list-like and map-like composite types. Needless to say, Objects can have Objects can have Objects…

Let us now look at a cool feature of TAME. One nice thing about TAME is that CRUD on schema (adding new entity definitions) is done using the same editor that we use to perform CRUD on instances. Adding a new entity is basically about adding a new instance to the schema table/collection. The form is rendered using a schema’s schema definition which is stored in the database along with other schemata.

Here is what the schema’s schema looks like (simplified for the sake of readability):

{
  "name": "schema",
  "fields": [
    {
      "name": "name",
      "type": "String",
      "optional": true
    },
    {
      "name": "fields",
      "type": "ArrayOfObjects",
      "schema": {
        "name": "fields",
        "fields": [
          {
            "name": "name",
            "type": "String"
          },
          {
            "name": "type",
            "type": "String"
          }
        ]
      }
    }
  ]
}

The schema called “schema” consists of 2 fields

  • name of type String
  • fields of type ArrayOfObjects where each Object in the Array has 2 fields
    • name of type String
    • type of type String

So we can do all this without writing a single line of code!

This is just the beginning of the capabilities of TAME.