Notes from NoSQL meetup in ATLSWC

Some notes from "Overview of NoSQL Database systems" hosted by Thoughtworks, Atlanta.

Presenter: Noah Kriegel

Starters: Awesome pizza and drinks.

Presentation gist:

• Beautiful prezi layout (check below image - Courtesy: James Brechtel).

• Most of the content is inspired from NoSQL Distilled book, but Noah did a good job in extracting the necessary stuff and supplementing it with his own experience.

Why we love RDBMS

• ACID compliant
• Standard, Expressive and Powerful SQL.

Why the hate?

•  Challenges with data replication and multi node setup resulting in expensive infrastructure.
•  Application object and db data type mismatch. Tools like ActiveRecord and Hibernate plays a major part in overcoming this but has their own pitfalls.

NoSQL movement

• Inspired by two papers on Google Big Table and Amazon DynamoDB
• Gain traction with the developers unlike failures like Object databases in the past.

Some important facts to remember

• CAP theorem
• Quorum-based voting techniques

NoSQL types

•  Key Store
• Document oriented database
• Column oriented DBMS 
• Graph databases

Each of the type was discussed on the following terms:

• Introduction and key feature
• When to use
• When not to use
• Sample code

I'll put some more details on each of these data types in the upcoming posts.

Some key TIL:

• CQL closely mirrors SQL in its syntax.
• Redis stands for REmote DIctionary Server.
• Mongo is a strip-off from the word - humongous

Hadoop installation in AWS EC2

I was using the CDH's vm image on my local for a hands-on experience with Hadoop. I thought let's try it out in AWS and see how smooth is the process.

So, I started following the cloudera's blog post on the same. But the blog had a lot of issues and things didn't work as outlined. I received a little more help from this blog.

So, here's the brief setup after creating the instance. As told in both the posts, I used whirr to install the cluster to avoid manual setup.

Step 1: Get the latest whirr binary
Step 2: Setup the whirr config file. You can copy the below contents and update the AWS Access Key ID and Secret Access Key accordingly.
Step 3: Install java
Step 4: Generate public key. I just entered on the first prompt.
Step 5: Launch cluster. Wait till you get the instruction to ssh to the nodes.
Step 6: ssh to the nodes.
Step 7: verify if hadoop installation works. We'll look more on this later.

Currently reading: NoSQL Distilled

When the rought cut version of "NoSQL Distilled: A Brief Guide to the Emerging World of Polyglot Persistence" was released in Safari books, I dived in and pointed out some comments. Those were totally useless to the authors and the publishers as they were either incomplete sentences or grammatical mistakes which were already caught by the proof readers. However, Martin Fowler was kind enough to respond to each and every comment.

Later on, when the book was released, I made a joke to one of my friends who was looking to buy the book that its better to get the Table of Contents from Amazon website and read the topic in wikipedia and save a few bucks. Last week, I thought I should seriously give it a read. So far, I've completed 3 chapters and have learnt a lot of stuff. Not only did I realize that I made a fool of myself by making that joke but also I'm learning many things a little late.

My upcoming blogs in the near future will definitely reflect a few things that I'm learning from this book.

An open letter to Thoughtworks Tech Radar Team

Dear Thoughtworks Radar Team,

I attended your Tech Radar webinar this Tuesday. Its a great iniative from your side to come up with something like and sharing it with the rest of the world. I would like to share some of my thoughts on the same.

• One hour to cover all the quadrants was very less. Because of this, you had to rush through some points and there was very less time for Q&A. I know this is a free webinar and you are putting in a lot of effort and money into imparting your research and learning to the outside world. However, it would be great if you have it as a 4 or 2 part webinar covering either 1 or 2 quadrants respectively.
• In the techniques section, I feel we need to include 'Application Security as a first class citizen' on par with performance test. A security flaw in an application is the most embarrassing moment for any organization. Most of the times, they are implicit in the user stories. It is the duty of the development team to identify, test and code them as early as possible. I would like to know if you guys are using any tool for this specifically apart from including it in your Functional testing.

On the good side, there were a lot of take away for a developer like me. A lot of new ideas were presented; some of the holds category made a lot of sense after listening and reading through them thereafter. I'm eagerly looking forward for the next edition.

Yours truly,
A Developer still learning.

PS: For others, here's the link to their radar.

Upgrade to Play 2.0? - Not now

What is the first thing for a seasoned Java developer to leave Spring/J2EE and move to something cool like Play? Of course, its easy setup and improvement in productivity. When I tried play 1.x series, I was very happy. After a few months, I recommended and used it for one of our internal projects and also successfully hosted it to heroku. It had matured a lot with new and useful modules.

Then saw the announcement of Play 2.0 which was completely being rewritten in Scala. As every other person, I too was excited about it and the twitter feed was full of praise once its released. And add on top of it, its inclusion in the Typesafe stack that created more buzz. So, with lot of excitement, I wanted to evaluate so that I can upgrade my application to this latest and greatest version.

First, I tried the Play-Scala module with their todoList example. As promised, the setup time was negligibly small and it lived up to the promise. But the real problem started right after that. Any change to your source code and the next page load takes a lot of time. Its definitely a lot better than the old J2EE-RAD setup but very bad when compared with its predecessor. I tried it with the java module and faced with the same issue.

There are some promises on their groups that they are working on improving the performance which is good. The two main issues must be the compile time of the Scala code (this is pure guess considering similar issues in other projects) and new feature that compiles and watches more components. Hopefully, they figure out a solution soon.

Until then, I'm going to stick with Play 1.2.4 for the application that we developed. As of now, we don't have any specific need to upgrade it to the next version. But its a definitely good thing to upgrade the application to incorporate any such things in the future.

Anti-If Campaign for Cincy Clean Coders

Below is the presentation I used for my talk on Anti-If Campaign for Cincy Clean Coders today.

The source code used in the presentation is available in github.

Anti-IF Campaign on Prezi

Dumping the thread - Websphere Performance Analysis

There are numerous occasions when we land up working in a performance problem be it at a development box or a defect opened from Production or Performance testing environment. While there are good profiling tools to help us get to the root of the issue like JProbe or YourKit, they come with a price tag with them. If there's a constraint on the team budget you wouldn't be able to use these tools. You can still workaround and use the trial version but if you read through the fine prints in the license file, you and your project stakeholders can be in trouble. So, stay out if not officially working on evaluating on these tools.

Though it could be a handicap there are other options that can be of use. In this mail, I'm going to demonstrate how to get the thread dump for your Websphere (WAS) server for performance analysis.

What is a thread dump?
A thread dump is a list of all the Java threads that are currently active in a Java Virtual Machine (JVM). When the jvm receives the signal for the same, it collects all the thread statistics and outputs it to a .txt file.

How do I generate it?
The most reliable way to generate a thread dump in WAS is using wsadmin utility. The steps for the same are as follows:
1.Navigate to the bin directory
cd <was_root>/profiles/<PROFILE_NAME>/bin/

2. Connect to deployment manager using wsadmin script
wsadmin.bat -conntype SOAP -username -password

3. The above command opens a wsadmin prompt. Now set the object variable to be used for generating the dumps
wsadmin> set jvm [$AdminControl completeObjectName type=JVM,process=,node=,*]

4.Run this command:
wsadmin> $AdminControl invoke $jvm dumpThreads

5. If you want to force heap dump, run the following command:
wsadmin> $AdminControl invoke $jvm generateHeapDump

Besides, if you have unix based systems like Linux/Mac, you can generate threaddump by just running the command:
kill -3 <pid>.

use ps -ef | grep java or ps -ef | grep to get the process-id(pid).

If you run WAS in console mode in Windows, Ctrl+Break helps to generate the dump but I have never tried it before.

A sample for generating a thread dump in a machine with node 5184Node01 and hosted locally in port 9443 with user/pass (ADMIN/password) is as follows:
wsadmin.bat localhost 9443 -username ADMIN -password password
The following commands will be run in wsadmin prompt.
set jvm [$AdminControl completeObjectName type=JVM,process=server1,node=5184Node01,*]
$AdminControl invoke $jvm dumpThreads

How to read the logs?
There are several ways to do it. Doing it manually is one of the most painful thing. IBM alphaworks has a cool tool for the same - IBM Thread and Monitor Dump Analyzer for Java shortly called as JCA. The ReadMe html inside the jar and the FAQ section talk a lot about the usage and interpretation of the data. I have used this a lot in the past and it helped us fix a lot of problems.

All said and done, try them in your leisure or when you are dealing with performance problems.

Effective Enums - 1

I'm planning to start a small series of one of my favorite inclusions in Java 5 - Enums. Joshua Bloch in his book Effective Java has dedicated an entire chapter on the same. This series will also be loosely based on the same.

In short, I want to have a Loan constant that can help me get the interest rate, maximum repay period and to check if one is eligible provided you enter the age and credit score. Here is my implementation in Java 5 for the same using enum.

Now, there are some constraints for a developer on the environment he works. Lets say if you are forced to work on Java 1.4, how do we implement this feature.

Lets create a new interface for Enums


Lets now implement our Loan enum here:

Phew! Its done.

Answer:Why is toArray() a generic method in Java Collections

At the outset, a special thanks to Joshua Marotti and Kevin from CinJUG to help me get to the solution. This is the solution to the question of my earlier post: Why is toArray() a generic method in Java Collections.

Consider the following scenario:

class Bar{}

class Foo extends Bar{}

List foos = new ArrayList();
foos.add(new Foo());
foos.add(new Foo());

Bar[] bars = new Bar[ foos.size()];
bars = foos.toArray(bars);

This works fine with the current code but will fail at compile if the method use the class level parameterized type as I had mentioned in the previous post. This is the use case why the method is declared generic. However, it doesn't prevent us from writing the below code which would compile just fine but fail at runtime with ArrayStoreException.

String[] strings = new String[foos.size()];
strings = foos.toArray(strings);

Per Joshua Bloch, we should follow PECS while using generics. Producer Extends, Consumer Super. Its probably the reason, the constructor for classes implementing collections has the extends parameter.

public ArrayList(Collection<? extends E> c)

Unfortunately, the language allows only wildcards for super. If it had supported type declaration with super, we could've solved the ArrayStoreException with the following usage.

<T super E> T[] toArray(T[] a)

On the other hand, I see Scala offers something on this line in scala.collection.immutable.List
def copyToArray [B >: A] (xs: Array[B]): Unit

There's a fabulous explanation on codeidol that discusses generics and collections in a very deep sense (suggested by Joshua Marotti). The chapter about "reification" that has some talks on toarrays and why it is the way it is and is a great read: http://codeidol.com/java/javagenerics/Reification/

Diamond Problem and Scala Traits

In object-oriented programming languages with multiple inheritance and knowledge organization, the diamond problem is an ambiguity that arises when two classes B and C inherit from A, and class D inherits from both B and C. If a method in D calls a method defined in A (and does not override the method), and B and C have overridden that method differently, then from which class does it inherit: B, or C?



Java's approach to this problem was to prevent developers to extend from multiple classes but provide them options to implement multiple Interface(s). Scala provides a more clean implementation similar to the Ruby's mixin concept (I'm 0% into Ruby) called Traits. So, how does Scala solve this problem. Check the code below:
TraitLearn.scala

abstract class A{
        def isWhat():Boolean
}

trait  B extends A{
        override def isWhat():Boolean = true;
}

trait C extends A{
        override def isWhat():Boolean = false;
}

val a = new A with C with B;
println(a.isWhat());

The output is:

scala TraitLearn.scala
true

So, instead of getting confused to identify which method to execute it has left it to the discretion of the developer. The actual purpose of traits is not what is shown above but what I like it more than the interface is its ability to have partial implementation.

By the way, I bought a new MacBook Pro and the very first thing to try in it was Scala. :-)

Why is toArray() a generic method in Java Collections?

Probably some java enthusiast can answer this question for me. I was referring to the Collections API to convert a list to array. I was shocked to find that the method was generic.

<T> T[] toArray(T[] a)

I was wondering why the method was declared generic when the type is itself declared with one. So, in order to convince myself, I wrote the following code and a test case. Please excuse if the code looks odd and isn't following any standard; its written quickly for the purpose of this blog.


public class AList<E> {

private final List<E> list;

public AList(List<E> list) {
super();
this.list = list;
}

public boolean add(E value){
return this.list.add(value);
}

public int size(){
return this.list.size();
}

public E[] toArray(E[] a) {
System.arraycopy(list.toArray(), 0, a, 0, size());
return a;
}

}


The first part of the test case is for pre-Java5 collection and the next for the type safe Java 5 way.


@Test
public void testAListToArray() {
AList list = new AList(new ArrayList());
list.add("Not type safe");
String[] sl = (String[]) list.toArray(new String[list.size()]);
assertEquals(sl.length, list.size());

AList<String> list2 = new AList<String>(new ArrayList<String>());
list2.add("Type safe");
String[] s2 = list2.toArray(new String[list.size()]);
assertEquals(s2.length, list2.size());
}


Both worked perfectly fine and the test passed. So, back to square one --> I'm unable to figure out why they method is declared generic with an additional parameterized type.

Just for gigs...

This is a sequel to my earlier post - Numbers: Not everything is magic.

If I show the numbers 1024, 2048, 4096... to a software geek, the immediate answer would be around bits and bytes. But these numbers have another peculiar pattern.

1024 is the first positive integer to satisfy the condition.

aⁿ⁺² + aⁿ = (2a)ⁿ

It is followed by the remaining numbers. Check the following:

1024²² + 1024²⁰ = 2048²⁰

2048²⁴ + 2048²² = 4096²²

4096²⁶ + 4096²⁴ = 8192²⁴

8192²⁸ + 8192²⁶ = 16384²⁶

16384³⁰ + 16384²⁸ = 32768²⁸

……

Now the hack to find it.

aⁿ⁺² + aⁿ = (2a)ⁿ

=>aⁿ (a² + 1) = 2 ⁿ aⁿ

=>a² + 1 = 2 ⁿ

=>a= (2 ⁿ - 1)^1/2

Now, let us find out a way to solve the below equation:

a^n+x + aⁿ = (2a)ⁿ

=>aⁿ (a^x + 1) = 2 ⁿ aⁿ

=>a^x + 1 = 2 ⁿ

=>a= (2 ⁿ - 1)^1/x

So, for n and n+4, the first positive integer to satisfy is 64.

Last but not the least for the post.

a^n+x + aⁿ = (k*a)ⁿ

=>aⁿ (a^x + 1) = k ⁿ aⁿ

=>a^x + 1 = k ⁿ

=>a= (k ⁿ - 1)^1/x

So for k=3 and x = 1, the numbers are 2, 8, 26, 80...

useEmma=openjpa.isInUse()?false:true

OpenJPA is a JPA framework very light compared to the heavyweight Hibernate. In order to provide optimal runtime performance, flexible lazy loading, and efficient, immediate dirty tracking, OpenJPA can use an enhancer . An enhancer is a tool that automatically adds code to your persistent classes after you have written them. The enhancer post-processes the bytecode generated by your Java compiler, adding the necessary fields and methods to implement the required persistence features. This bytecode modification perfectly preserves the line numbers in stack traces and is compatible with Java debuggers.

This is my third stint with EMMA and I simply love it even over the much detailed Clover. Apart from the free factor, its pretty fast and has less memory overhead. In one of my project, our nightly CI job with clover used to crash with clover which forced us to stick with EMMA. Instrumentation is an important step to get the coverage. The instrumentor adds bytecode instrumentation to all classes found in an instrumentation path that also pass through user-provided coverage filters. Additionally, it produces the class metadata file necessary for associating runtime coverage data with the original class definitions during coverage report generation.

Unfortunately, OpenJPA enhancer and EMMA instrumentor don't jell together. Yes, there's a conflict between them. As a result, your coverage will be incomplete. So, options left out will be

1. To switch over to Cobertura(sic) or Clover.

2. To filter the Dao and JPA entity classes from the coverage (sic again).

Depending upon your flexibility and need, you can choose one of them.

Reference: OpenJPA Enhancer, Emma

Numbers: Not everything is magic

In my previous post - Number Magic, I scribbled that I found a pattern as follows...
- For any number n from 1 to 10 and any number k that satisfy the condition - k^n + k^(n+1) = (k+k)^n the next such condition is satisfied by (2k+1). But actually, there was an interesting solution. Lets put it in plain equation.

k^(n+1)+k^n = (2k)^n

=>k^n(k+1)=(2^n)*(k^n)

=>k=(2^n)-1

So, the number thats satisfies are 1, 3, 7, 15...1267650600228229401496703205375...

What is my learning here? - Get back to the basics... Or should I say to myself KISS - Keep it Simple, Stupid!!!

Number Magic

Thanks to a post from one of my old manager, mentor and friend - Anand Mohanram - 10 (Square + Cube pattern - Interesting Number Pattern), I was able to find an interesting pattern. Some were commented to his facebook posts.

1. The difference between the cube numbers over their previous square partners is an arithmetic progression with common difference of 1. See them below:

2+1=3

3+3=6

4+6=10

5+10=15

6+15=21

7+21=28

8+28=36

9+36=45

10+45=55

11+55=66

12+66=78

13+78=91

14+91=105

15+105=120

16+120=136

17+136=153

18+153=171

19+171=190

20+190=210

21+210=231

22+231=253

23+253=276

24+276=300

25+300=325

26+325=351

27+351=378

28+378=406

29+406=435

30+435=465

31+465=496

32+496=528

33+528=561

34+561=595

35+595=630

36+630=666

37+666=703

38+703=741

39+741=780

40+780=820

41+820=861

42+861=903

43+903=946

44+946=990

45+990=1035

Actually, we can easily find the numbers matching the pattern by solving the equation:
a^3+b^2 = (a+b)^2;
=>a^3+b^2 = a^2+b^2+2*a*b;
=>b = a*(a-1)/2

This explains the AP of constant 1 between the numbers for every value of a.

Java Source code that helped to find this (though its of no use now) :

public void findSquareCubeNumbers(int count){
 for (int i = 1; i < count; i++){
  int num = i*i*i;
  for (int j = 1; j < count; j++){
   int sum = i+j;
   if (num+(j*j)==sum*sum){
    System.out.println(i+"+"+j+"="+sum);
   }
  }
 }
}

2. For any number n from 1 to 10 and any number k that satisfy the condition - k^n + k^(n+1) = (k+k)^n

the next such condition is satisfied by (2k+1).

The below illustration is the proof:

(k= 1((2*0)+1); n= 1) 1^1+1^2=2^1

(k= 3((2*1)+1); n= 2) 3^2+3^3=6^2

(k= 7((2*3)+1); n= 3) 7^3+7^4=14^3

(k= 15((2*7)+1); n= 4) 15^4+15^5=30^4

(k= 31((2*15)+1); n= 5) 31^5+31^6=62^5

(k= 63((2*31)+1); n= 6) 63^6+63^7=126^6

(k= 127((2*63)+1); n= 7) 127^7+127^8=254^7

(k= 255((2*127)+1); n= 8) 255^8+255^9=510^8

(k= 511((2*255)+1); n= 9) 511^9+511^10=1022^9

(k= 1023((2*511)+1); n= 10) 1023^10+1023^11=2046^10

Java Source code that helped to find this:

public void findNextPowerNumbers(int powVal, int count){
 double prevK = 0;
 for(double i = 1; i < powVal; i++){
  double power = i+1;
  for (double k = 1; k < count; k++){
   double num = Math.pow(k, power);
   double num1 = Math.pow(k, i);
   double sum = 2*k;
   double num3 = Math.pow(sum, i);
   if (num+num1 == num3){
    System.out.println("(k= "+k+"((2*"+prevK+")+1); n= "+i+") "+k+"^"+i+"+"+k+"^"+power+"="+sum+"^"+i);
    prevK=k;
   }
  }
 }
}

Cincinnati Day of Agile - Mar 26, 2011

BeforeClass Over Static

The JUnit API states the following:
"Sometimes several tests need to share computationally expensive setup (like logging into a database). While this can compromise the independence of tests, sometimes it is a necessary optimization. Annotating a public static void no-arg method with @BeforeClass causes it to be run once before any of the test methods in the class. The @BeforeClass methods of superclasses will be run before those the current class."

Can we not avoid the repetition of this expensive setup using a static initializer which comes with Java itself? We can; but the BeforeClass offers one more additional feature. You can tailor-call the required setup call when you want to modularize your test cases by introducing inheritance. The below code and output will help you understand it more clearly.

BaseTest.java



Test1.java


The output if you run the test:

inside static of class org.sudhin.statictest.BaseTest
inside @BeforeClass of class org.sudhin.statictest.BaseTest
inside static of class org.sudhin.statictest.Test1
inside @BeforeClass setupForTest of class org.sudhin.statictest.Test1
inside test method of class org.sudhin.statictest.Test1

Came back...

I somehow felt blospot is better than wordpress. So, I'm back to business here. I've provided a link to my old posts from this blog.

One + One = One Strong Output

Post can be found here.

EJB in a nutshell

Post can be found here.