Notes from NoSQL meetup in ATLSWC

Some notes from "Overview of NoSQL Database systems" hosted by Thoughtworks, Atlanta.

Presenter: Noah Kriegel

Starters: Awesome pizza and drinks.

Presentation gist:

• Beautiful prezi layout (check below image - Courtesy: James Brechtel).

• Most of the content is inspired from NoSQL Distilled book, but Noah did a good job in extracting the necessary stuff and supplementing it with his own experience.

Why we love RDBMS

• ACID compliant
• Standard, Expressive and Powerful SQL.

Why the hate?

•  Challenges with data replication and multi node setup resulting in expensive infrastructure.
•  Application object and db data type mismatch. Tools like ActiveRecord and Hibernate plays a major part in overcoming this but has their own pitfalls.

NoSQL movement

• Inspired by two papers on Google Big Table and Amazon DynamoDB
• Gain traction with the developers unlike failures like Object databases in the past.

Some important facts to remember

• CAP theorem
• Quorum-based voting techniques

NoSQL types

•  Key Store
• Document oriented database
• Column oriented DBMS 
• Graph databases

Each of the type was discussed on the following terms:

• Introduction and key feature
• When to use
• When not to use
• Sample code

I'll put some more details on each of these data types in the upcoming posts.

Some key TIL:

• CQL closely mirrors SQL in its syntax.
• Redis stands for REmote DIctionary Server.
• Mongo is a strip-off from the word - humongous

Hadoop installation in AWS EC2

I was using the CDH's vm image on my local for a hands-on experience with Hadoop. I thought let's try it out in AWS and see how smooth is the process.

So, I started following the cloudera's blog post on the same. But the blog had a lot of issues and things didn't work as outlined. I received a little more help from this blog.

So, here's the brief setup after creating the instance. As told in both the posts, I used whirr to install the cluster to avoid manual setup.

Step 1: Get the latest whirr binary

	wget http://apache.cs.utah.edu/whirr/whirr-0.8.1/whirr-0.8.1.tar.gz
	tar -xvf whirr-0.8.1.tar.gz

view raw wget hosted with ❤ by GitHub

Step 2: Setup the whirr config file. You can copy the below contents and update the AWS Access Key ID and Secret Access Key accordingly.

	vi hadoop.properties
	________________________________________________________________________________________________________________________
	whirr.cluster-name=<cluster_name_all_lowercase>
	whirr.instance-templates=1 hadoop-namenode+yarn-resourcemanager+mapreduce-historyserver,2 hadoop-datanode+yarn-nodemanager
	whirr.provider=aws-ec2
	whirr.identity=<AWS Access Key ID>
	whirr.credential=<AWS Secret Access Key>
	whirr.private-key-file=/home/ubuntu/.ssh/id_rsa
	whirr.public-key-file=/home/ubuntu/.ssh/id_rsa.pub
	whirr.env.mapreduce_version=2
	whirr.env.repo=cdh4
	whirr.hadoop.install-function=install_cdh_hadoop
	whirr.hadoop.configure-function=configure_cdh_hadoop
	whirr.mr_jobhistory.start-function=start_cdh_mr_jobhistory
	whirr.yarn.configure-function=configure_cdh_yarn
	whirr.yarn.start-function=start_cdh_yarn
	whirr.hardware-id=m1.large
	whirr.image-id=eu-west-1/ami-81c5fdf5
	whirr.location-id=eu-west-1
	________________________________________________________________________________________________________________________

view raw hadoop_prop hosted with ❤ by GitHub

Step 3: Install java

	sudo apt-get update
	sudo apt-get install openjdk-6-jre-headless

view raw jdk-install hosted with ❤ by GitHub

Step 4: Generate public key. I just entered on the first prompt.

	ssh-keygen -t rsa -P ''
	_______
	Enter file in which to save the key (/home/ubuntu/.ssh/id_rsa): <Press Enter>
	Your identification has been saved in /home/ubuntu/.ssh/id_rsa.
	Your public key has been saved in /home/ubuntu/.ssh/id_rsa.pub.
	The key fingerprint is:
	_______

view raw keygen hosted with ❤ by GitHub

Step 5: Launch cluster. Wait till you get the instruction to ssh to the nodes.

bin/whirr launch-cluster --config hadoop.properties

view raw whirr-instal hosted with ❤ by GitHub

Step 6: ssh to the nodes.

ssh -i /home/ubuntu/.ssh/id_rsa -o "UserKnownHostsFile /dev/null" -o StrictHostKeyChecking=no ubuntu@xx.xxx.xx.xx

view raw ssh hosted with ❤ by GitHub

Step 7: verify if hadoop installation works. We'll look more on this later.

Currently reading: NoSQL Distilled

When the rought cut version of "NoSQL Distilled: A Brief Guide to the Emerging World of Polyglot Persistence" was released in Safari books, I dived in and pointed out some comments. Those were totally useless to the authors and the publishers as they were either incomplete sentences or grammatical mistakes which were already caught by the proof readers. However, Martin Fowler was kind enough to respond to each and every comment.

Later on, when the book was released, I made a joke to one of my friends who was looking to buy the book that its better to get the Table of Contents from Amazon website and read the topic in wikipedia and save a few bucks. Last week, I thought I should seriously give it a read. So far, I've completed 3 chapters and have learnt a lot of stuff. Not only did I realize that I made a fool of myself by making that joke but also I'm learning many things a little late.

My upcoming blogs in the near future will definitely reflect a few things that I'm learning from this book.