TAME – Key Features – Schema and Entities

In the previous post we discussed the features that we would like to have in a modern day MVC framework. In this post let us discuss some of the key problems in designing such a framework and how we have approached it in TAME.

Data is stored in storage devices as bits. However humans don’t quite understand this, so have come up with mechanisms to store and retrieve data from a storage device using a high level interface. This high level interface is the notion of files and on top of files, we have built the notion of databases.

A database makes this storage and retrieval more user friendly rather than working with raw files. Databases are stores of “entities”. Entities may be related to other entities. Interaction with the database involves performing CRUD operations on the entities.

Entities contain fields. Fields have data types. Data types can be broadly categorized as “primitive data types” and “composite data types“. When considering relational databases, we normalize data into primitive data types. Composite data types are handled using primary and foreign key relationships across multiple tables. However, in a NoSQL store, we may store composite data as a sub-record within a single record itself and save an additional join.

So we start our discussion by asking, what are the minimum set of primitive and composite data types we need? If we do a comparative analysis of different programming languages and database technologies, we will soon realize that almost all languages have some way of representing strings, numbers and booleans. There are many other primitive types, but these are the minimum. The reasoning behind why this is so, is beyond the scope of this post.

Composites are ways of grouping data together. Composites are made up of primitives and other composites. At a minimum, we need one composite data type that has sequential properties (list like) and at least one that can be used to store non-sequential (key/value) form of data (map like).

Continue reading TAME – Key Features – Schema and Entities

Rethinking MVC Frameworks

In the previous post I explained the need for an application framework that allows us to evolve the functionality as our startup processes evolve. In this post, we will go into the technical details of what we expect from such a framework.

Almost all server side MVC frameworks provide a standard set of functionalities:

  • A way to receive a HTTP Request from the client, examine the Request and determine what the user wants to do. A URL mapper transfers control to a Controller which then does the request processing.
  • The controller fetches data from backend services (eg: databases).
  • Along the way, data transforms from relational form to object form with the help of an Object Relational Mapper (we call this a model).
  • The model data is then transformed into a view with the help of templates.
  • Several other functionalities are provided by mature MVC frameworks – sessions, caching, REST endpoints etc.

In the last decade there have been some trends that are changing the way we work with MVC frameworks. For one, clients are becoming more intelligent and thicker (logic moving from server to client where possible). This has resulted in the emergence of client side MVC frameworks.

Continue reading Rethinking MVC Frameworks

Scaling processes for a growing startup

Jnaapti completed 6 years this May. What started as a 1 person company in 2011, is now a growing business with more than a dozen people working on different aspects of the business. We have conducted training in over 35 organizations in over 30 technologies in the last 6 years.

Like many other tech startups, I started off writing the entire code for our main product, The Virtual Coach, single handedly. The first version of the product took me 2.5 months to build from concept to deployment.

We are now a team of 8-10 engineers working on different aspects of engineering. We now have multiple products in the education space and are targeting multiple customer segments.

When it was just one or two people, we didn’t need a formal project management process. It doesn’t mean a process didn’t exist; rather, the process was in the heads of the individuals who were part of the development team.

If 2 of us are programming, we can easily share tasks, arrive on deadlines and decide our commitments without requiring any elaborate processes as long as we trust each others’ commitments. We may resort to using simple tools like Spreadsheets or may even use pen and paper or a whiteboard.

However, as the organization grows, the number of cross-communication in the team increases. If every person communicates with every other, there is chaos and this soon starts reflecting in the overall efficiency of the organization. This is not a new problem and has been elaborately discussed in some excellent literature.

Continue reading Scaling processes for a growing startup

The Jnaapti Journey – Learning From Kids

One of the most exciting moments for us was to watch 12 year olds programming in Python. Our record is being able to teach the use of a “for-loop” in Python to a 6-year old.

While we see Engineering students and Software professionals struggling with the syntax of programming languages despite having a degree in Computer Science, we were able to teach these kids basic programming constructs like looping and conditional within less than 3 hours. So when professionals say “Python is too hard”, we retaliate saying, “No, the syntax is not that hard. 10 year old kids were able to pick it up in less than 3 hours! May be you need to unlearn.”

Here we describe our experiences of teaching kids and learning about learning from kids.

It all started with our Fast Trackers Workshop. We had a student from 1st PUC (11th standard) attending this program. We found him to be self-motivated, sharp and he picked up concepts as easily as the engineering students.

This got us wondering. If school students can learn programming, can we try with students of a younger age. What is the age at which students “get” programming? Is programming everyone’s cup of tea? Should we teach programming to everyone? When I asked these questions, Shreelakshmi told that she can help. Shreelakshmi used to run her startup in Mangaluru and as a part of this, used to teach in a school in Mangaluru. So we decided to run an experiment in her school.

Continue reading The Jnaapti Journey – Learning From Kids

The Jnaapti Journey – Meetup Stories

It was 2014, and Shreelakshmi had just moved to Bangalore. She had just assumed her new role as “Head of Training & Operations” at jnaapti in February 2014. One of her goals was to bring more visibility to jnaapti. This is an account of her experience of growing a community with technical meetup sessions.

Shreelakshmi’s memories go back to the days when jnaapti was still a single person organization and was hardly known in the community. jnaapti grew to be a four member team in early 2014. It was during this phase that jnaapti was taking baby steps towards a bigger future that we are living in today.

We brainstormed on how to make people around us know jnaapti and the work we do. We started looking for various ways to conduct events which will help spread our learning philosophies. That is when we came across the site Meetup.com.

The First Meetup

They say the first step is always the hardest and so it was with our meetup journey as well.

Continue reading The Jnaapti Journey – Meetup Stories

The Jnaapti Journey – Helping Companies Succeed

Jnaapti has conducted training in over 30 companies in over 30 technologies in the last 6 years. We are the proud technology training partners to some of the best minds in the industry. Our training programs have been attended by employees of organizations like VMware, Akamai, Citrix, Ericsson, Practo, SAP Labs etc. We have trained people in organizations of varied sizes (startups, SMBs, MNCs) and with varied experience levels – freshers to people with over 20 years experience in the industry.

Our youngest student is 6 years old and our oldest student is a retired doctor who was building his own startup. Professionals that we have trained play various roles in organizations; we have trained software developers, architects, test engineers, system administrators, database administrators, Devops engineers, performance engineers, full-stack developers, sales people and engineering managers.

Our vision is to be able to teach anything to anyone starting from anywhere. With this in mind, we have evolved a carefully thought through training process which caters to organizations of different sizes and varied training needs.

jnaapti’s 4 step approach to training

Learning Objectives

When companies plan for a training, they have various objectives in mind. Sometimes, the training is planned to meet organizational learning goals, sometimes for project specific needs, sometimes for anticipated project needs or sometimes for employee motivation. The way a training is delivered depends on these objectives.

The experience level of the target audience differs. People may be learning the same technology, but they have very diverse backgrounds. The way you teach a learner fresh out of college is different from an architect who has 10 years of experience in the industry. To a fresher, you need to relate more to the concepts that they have learnt in their curriculum and why these technologies exist and what to expect from each. To an experienced person, you need to be able to relate to the technologies they already know and what they are learning now. The way you teach MongoDB to a DBA who has worked in Relational Database technology all his career is very different from a fresher who will perhaps find MongoDB more natural than a relational database!

Continue reading The Jnaapti Journey – Helping Companies Succeed

The Jnaapti Journey – Virtual Coach

What it is

While the idea of Virtual Coach has always been in Gautham’s mind since the inception of jnaapti, a startup does not have the luxury to build the entire vision and then launch. Rather, we have to start building the minimum set of features which we can validate with our users and then iterate frequently and let usage and metrics drive the product’s evolution. Jnaapti has been doing this ever since the launch of the first version of the product.

The vision of “Virtual Coach” is to eventually replace a human coach with a “Virtual” coach but in such a way that the learner does not even realize that he/she is being coached by a virtual entity. It follows the Turing test for Artificial Intelligence, but applies to learning and coaching. While we have started with software technology training, we intend to build a generic platform that has the ability to teach anything (any skill) to anyone (no bias) starting from anywhere (irrespective of what they know today).

The problem that keeps us awake at night is, “How do we scale good quality coaching so that it reaches the maximum number of people?”

Technical details

Version 0

The MVP of Jnaapti’s Virtual Coach was not a “product” at all. 🙂

The initial idea of the training process was tested out using Email as a form of communication between the learners and the coach. As Gautham coached people, he carefully noted down the pains of using Email as the medium of communication. The requirements for the first version of the product was to replace Email as the mode of communication.

Version 1 – 2011

The first version of the product was built single handedly by Gautham using Web2Py. It was a server-heavy product. There was minimal Javascript used and a lot of the backend data entry was done using ready-made views that Web2Py provides out of the box. The entire conceptualization, design, development and deployment of the product took 2.5 months.

The first version of the product was called the “Jnaapti Virtual Learning Environment”.

Continue reading The Jnaapti Journey – Virtual Coach

The Jnaapti Journey – Engineering Education

While, on one side, we see students who are in the system for all the wrong reasons, there are several we come across who have a genuine interest and curiosity about the world around them. They are passionate about using technology to solve real world problems. Jnaapti tries to identify such potential students and connect them to the right places in the industry.

Fast Trackers Workshop

We conduct a workshop called the “Fast Trackers Workshop” for engineering students. This workshop is a unique blend of technical and life skills delivered in an intense 5-7 days session. Students learn what aspects of their engineering is relevant to the industry needs, what else the industry expects, how to quickly learn a new technology. We show the intricate connection between the various concepts that the students learn during their engineering and how they are required to solve problems in the industry. We use the “Swimming pool philosophy – If you gotta learn swimming, jump into the swimming pool while a coach is watching you”. Within no time we make learners independent of needing a technology trainer to learn new concepts and technologies. In jnaapti, our engineers pick up a new technology within a day or two and we teach these skills to the participants of the Fast Trackers Workshops.

Continue reading The Jnaapti Journey – Engineering Education

The Jnaapti Journey – Prologue

Jnaapti was founded in May 2011 with the intention of providing quality education to the masses. We intend to build life-long learners, helping them understand the learning process and making the whole process of learning, fun and engaging. It’s been 6 years since we started and it feels like we have just started! We have learnt a lot and it has been an exciting journey so far.

Our post on The Story of Engineering Education in India is one of the most widely read blog posts. Many have praised us for succinctly capturing the key issues, but some have criticized us saying, “We know the issues. What’s the solution?”

Very little is known about what jnaapti has done or is in the process of doing and this series of posts is an attempt to capture some of our salient experiments and our success stories.

29-may-2011

Here are some posts in this series:

Continue reading The Jnaapti Journey – Prologue

A Docker Based Development Environment

At jnaapti, we have been using containerization technologies since almost the very early days. In 2011, when I was evaluating a solution to provide light-weight containers to our learners in the Virtual Coach, I was told that the only mature solution was to use Virtualization solutions. But Virtualization was too slow (in terms of boot up time) and I didn’t have enough resources to keep stand-by nodes running all the time. So after some evaluation, I decided to use LXC and it served its purpose. However, there were several features missing and I was in the verge of building a few of them myself.

So it’s not surprising that when I discovered Docker, I just fell in love with it. One of the first things we did was we moved our LXC based learner containers to Docker. We then slowly started to migrate portions of our infrastructure to Docker. We achieved full Docker migration last November, and then also moved our staging/testing systems in the cloud to Docker.

The last in the list was to migrate our development environments. The initial migration wasn’t too hard, because of 2 things:

  1. We already had Docker in production – so it was a matter of working off our production Dockerfiles
  2. We were already using KVM in our development – so we already had a clear idea about what containers our system should be running.

Why the move away from KVM based development environments? Simple! KVM’s disk usage is too much to suit our requirements. We had our development images of 5G each and you have 10’s of such images and are soon out of disk space. I am not sure how many Docker fans will approve some of the things I discuss here but I believe that this is a much leaner solution than using Virtualization and I don’t see any issues in the way I am doing it.

So with the migration from KVM to Docker, there were a few additional things that I wanted to handle:

  1. Can we use Desktop tools like text-editors with data in Docker? Imagine I am writing a NodeJS application. I want to use Atom installed in my host to write code. However, I want to run NodeJS inside the container.
  2. It is very easy to work with command line utilities (which don’t need X) in containers, but how about Desktop utilities like Eclipse? Can we run this in a Docker container and still have the same user experience as a regular app? What are some best practices to do so?
  3. Is it possible to expose devices into Docker containers – this is required for eg, if we are doing Android development and want to debug our app in an actual Android device

The first 2 were rather easy and I sailed through it. The third one, I struggled a little, but I finally made some head-way.

This post attempts to capture some of my learnings in this entire process in case you want to build a similar environment. So let me answer these questions:

Using Desktop tools with Docker containers

This one is easy. A practice we follow is all the code that we write is inside a mounted volume. Containers are used to run processes in a contained fashion, but the processes are manipulating files that are in our host (and not in the underlying diff file system). We can as well remove our containers and we don’t lose anything.

Here is a sample run to demonstrate this:

Start a Docker container that has NodeJS installed in it. Make sure that this container has access to a local host directory (in this case /home/gautham/Desktop/node-data):

gautham@ananya:~|? docker run -d -P -v /home/gautham/Desktop/node-data:/data --name "node-example" ananya-nodejs:0.0.1
820e105db5061b380e6117e42a0cabad5f00c54e54f5016aefc18399e2a2eb25

Check if the container is running

gautham@ananya:~|? docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
820e105db506 ananya-nodejs:0.0.1 "/usr/sbin/sshd -D" 9 seconds ago Up 8 seconds 0.0.0.0:49155->22/tcp node-example

Using Atom to edit files in a volume shared with a Docker container

Inside the container (which is accessible via SSH):
gautham@ananya:~|? ssh -p 49155 ubuntu@localhost
ubuntu@localhost's password:
Last login: Tue Apr 14 14:13:12 2015 from 172.17.42.1
ubuntu@820e105db506:~$ cd /data/
ubuntu@820e105db506:/data$ ls
hello.js
ubuntu@820e105db506:/data$ node hello.js
Hello World!

Using Desktop tools inside Docker containers

This one initially seemed a little difficult, but I figured out soon.

Create an image that has lxde-core package installed in it.

Now, there are 2 options:

Connect to lxde running inside the Docker container

Run these commands in host:

docker run -d -P ananya-desktop:0.0.1
sudo su
xinit -- :1 &

This will now switch you to a different terminal (accessible at Ctrl+Alt+F8). You will also see a white terminal. Type the following in this terminal to start LXDE:

docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
38280dd66b98 ananya-android:0.0.1 "/usr/sbin/sshd -D" 4 minutes ago Up 4 minutes 0.0.0.0:49156->22/tcp, 0.0.0.0:49157->5901/tcp hopeful_lumiere
ssh -X -p 49156 ubuntu@localhost

In the Docker container run

startlxde

You should now see a full fledged Desktop running like this!

LXDE running in a Docker container

So your host is running at Ctrl+Alt+F7 and your Docker container is at Ctrl+Alt+F8. Use this option in case you are running many Desktop applications and you want to be totally isolated from the host when working with the applications in the container (i.e you are not using any host applications in conjunction with the applications in the Docker container).

Only run the application that you are interested in

I found this option to be better in some ways. I have my Android Studio setup using this option now.

gautham@ananya:~|? docker run -d -P ananya-desktop:0.0.1
gautham@ananya:~|? ssh -X -p 49156 ubuntu@localhost
ubuntu@localhost's password:
Welcome to Ubuntu 14.04.2 LTS (GNU/Linux 3.16.0-34-generic x86_64)

* Documentation: https://help.ubuntu.com/
Last login: Tue Apr 14 14:25:47 2015 from 172.17.42.1
ubuntu@38280dd66b98:~$ cd android-studio/
ubuntu@38280dd66b98:~/android-studio$ ls
ubuntu@38280dd66b98:~/android-studio$ bin/studio.sh

And lo and behold!

Anroid Studio Running in a Docker Container

Accessing devices within Docker container

A final requirement was whether we can get Docker to detect USB devices. I found that if you pass a –privileged flag and mount the /dev/ device appropriately, you can then access it in the Docker container. I was able to successfully use adb along with my Docker container.

docker run --privileged -v /data:/data -v /dev/bus/usb:/dev/bus/usb -d -P ananya-android:0.0.1

Docker has been a boon and although there are several area of improvement, I see that it has a future. It has become an indispensable tool in our software tools arsenal in jnaapti.

Cross-posted here: http://buzypi.in/2015/04/14/a-docker-based-development-environment/