Scaling Data tier strategies

Kaushik pointed me to an excellent presentation of the eBay Architecture. The intriguing aspect has been scaling the Data tier and some of the interesting strategies suggested are a) No business logic in the DB, quite agreed but the strategy says not to use stored procedures, very interesting and recommends usage of extensive Prepared Statements with bind variables b) Auto commit for vast majority of DB writes.

It is PDF with lot of insights. Read it!

Sweetheart, I'm missing you

It has been a while since I programmed in Ruby. Almost 8 months! I'm doing lot of enterprise Java ;-). Not that I hate Java, it is just to let you know how much I love Ruby. Busy schedules, tight timelines didn't allow me much to allocate time for learning Ruby. I almost forgot it, sob! I wanted to start afresh with Ruby and was searching for some good tutorials and landed at this place - Why's (poignant) guide to Ruby. And I must tell you, it is unputdownable. Go grab the book and start reading. I really wish I work with Ruby and Rails on a real time project soon.

Coding Horror - blog

If you have read Code Complete by Steve McConnell, you must have noticed the "Coding Horror" side bars that show you how not to write code. I found out today that there is actually a blog for this. Check the Coding Horror Blog.

**** UPDATE *****
The link is not actually related to Code Horror or Steve McConnell. I realized this after publishing the blog and didn't update the entry. Point to www.thedailywtf.com as suggested in the comments to this entry. Thanks to the person for the comment, btw.

Software and Manufacturing - Nice analogy

This article talks of what the software and manufatcuring have in common and how can the lessons learnt in manufacturing be applied to software. Good one!

Btw, a friend has asked me why my recent blog entries had few lines and links to other sites. Dude, I'm in reading mode ;-) and blog is the best place for quick dissemination of the learnings :-)

Stress Management

Found some good tips to manage stress at this linkCome Out of Stress. Nice entry. Go read it if you have time.

Semblance project

If you are a pro-Struts chap and interested in using Struts for your projects, then check the Semblance project that has lot of goodies required to kick start a web based project.

JSF - what are you made of?

If you have been using JSF for a while, you'd understand my feelings better. JSF has failed to keep it's promise of making developers' life easy. There are so many instances where JSF would not scale to the requirements and made the lives of developers miserable.

So far in my current project, I focused more on the middleware and the infrastructural aspects and decided to stay away from the presentation layer for a while. My frustration levels have increased when I was asked to develop few components for the presentation layer using JSF. It simply did not scale. The same requirements could have been implemented very easily using Struts/Tiles.

Thought of Struts makes me happy, I began my career in Software almost the same time when Struts was born. My career grew along with Struts. 5 good years of working with Struts. The combination of Struts, Tiles met most of the challenging requirements, oh yeah, even with AJAX. What was the need to create a bloated API like JSF that sucks when it comes to simple tasks, even like downloading a file. And coming to the different implementations of JSF - if you are using MyFaces, you are little lucky and if you decide to go with the JSF-RI, expect the frustation level of your development to be on the higher side. My experience with JSF hasn't been an happy one. Going forward, I'd be happy to use Struts for my projects.

Technology compendium

Try this link. Closer to a technology compendium. Good work, though!

Synthesis - The little known and used tool

Today, I came home a little early after a very nice team outing, and was thrilled to see my engineering classmate. We have recollected our good old engineering days, whereabouts of our classmates etc etc. It was nice to know that most of the chaps are doing well and many have preferred Software industry or rather IT as it is called here. And we talked about how many are working as developers, business analysts, release engineers etc. [Btw, programmers, designers, architects all belong to the developer category, my way of grouping ;-)]. The next topic was how many turned out to be good developers (Salary wasn't considered as an yard stick)

One of the things that spurred interest is a subjective equation, in the context of development or assessing a developer, which my friend read in some book; simple but elusive

Good = Acumen * Dexterity

Acumen is the gamut of technologies the person is aware of and dexterity is the extent of deftness exhibited in applying the known/learnt stuff. Acumen can be enhanced through trainings, reading books, learning from colleagues/peers, on the other hand dexterity is something that does not get enhanced through traditional learning approaches. Dexterity cannot be taught and has to be learnt by the individual. The next question would be how to gauge the dexterity.

A simple analogy - any (genuine) mechanical engineer would have studied the subject "Design of Machine Elements". For designing a simple nut and bolt, the right metal has to be choosen, the thread length on the nut has to be calculated, the dimensions have to fixed and the list goes on. To arrive at an "optimal" design, one needs to consider the metallurgical aspects, the shear force and bending moment, production engineering aspects etc. This process of considering all the factors affecting a system is called Synthesis. Essentially, Synthesis is nothing more than looking at all possible angles of the problem and bringing in the knowledge from the related fields and making best use of it.

When designing something, people do some Analysis. Analysis would help in breaking up the problem into small and manageable chunks. This is essential. At the same time, applying Synthesis would add more flexibility, extensibility, and maintainability to the piece being designed. While interviewing candidates, select a person with high synthesis quotient.

Oh boy! I have jotted much stuff. If you feel that this entry is disorganized, put your comments, I'll correct it.

Ultra-Scalability

Had been reading an interview with Werner Vogels, CTO of Amazon. It has been very informative. Follow the link Link. I particulary liked the phrase "to be ultra-scalable while maintaining availability and performance".

Aren't you using Jakarta JMeter

My previous blog entry on "Debugging Webservices" spurred lot of interest and I received mails from many people with lots of questions, pointers, asking help etc.
Many seem to be happy using the SoapUI. Thanks guys!

One of my friends asked me a straight question - aren't you using JMeter to do performance testing of the web services. This was very flat and I wasn't aware that JMeter can be used to load test the WS. There is some tweaking that needs to be done - place mail.jar and activation.jar files in the lib of the JMeter to get the menus and the options displayed correctly. The rest of the setup is similar to any other JMeter configuration. Setup the number of threads, ramp-up period and loop count and hit start. The results will be displayed in a crisp graph.

If you haven't been using JMeter, it is high time you start using it. If you need any help in setting up JMeter WS, send me an email at ravisk@gmail.com.

Debugging Web Services

Ever wondered how to debug web services?
Debugging web services has been misconstrued by the folks, atleast the people I have been talking to. It is a normal practice to just debug the back end of the WS, example the EJB or an helper class that makes a call to the DB, and claim that the WS is debugged. This, apparantly, doesnot debug the actual service. I decided to come up with a check list for debugging the WS and here it is:

1. The primary stuff, the WSDL should be accessible from the browser
2. Validate request and response bodies against their schema definitions
3. Validate the definitions for WS-I (Web service interoperability) compliance.

The second and third points are of concern. I had been using the TcpMon, that is shipped with Apache Axis so far to see the SOAP messages. This would require "tunneling". Hmmm.. something non-intrusive would be better. I had been looking for some nice tools that does the true debugging of WS. Here is a list of tools and their descriptions:

1. WSUnit - Unit Testing a WS
2. WebService Studio - Invoking the WS, it a .NET program. Given the endpoint, generates the proxies and invokes the methods
3. SOAPUI - A very nice tool with all the required features. It doesn;t limit to just debugging, but provides features to load test and functional test the WS. Really cool!

I installed the SOAPUI and the experience is very good. I strongly recommend it to all folks using WS and wondering how to debug.

CLOB issues in weblogic

I noticed a strange behavior with CLOB in weblogic. The CLOBs would work fine with the main method, but when the same code is run on Weblogic, it throws an "No more data to read from socket" exception. The interesting aspect is, the row in the table gets inserted perfectly including the CLOB column.

The first thing I suspected is with the driver version of the ojdbc.jar shipped with WLS, but that wasn't the problem as the driver I used in the application is same as the one shipped with WLS.

Since, the infrastructure appeared fine, I decided to debug and track the root cause. Debugging the problem brought out some intriguing points. The hack is to create the CLOB column with the default as EMPTY_CLOB() and do a post process after the insertion. The sequence would be:

1. insert some dummy string first cs.setString(4," ");
2. Excecute the proc
3. post process the clob

Clob clob = null;
pstmt = conn.prepareStatement("SELECT clob_col from clob_tbl where id=? FOR UPDATE");
pstmt.setString(1,id);
rs = pstmt.executeQuery();
if(rs.next()) {
clob = rs.getClob(1);
Writer writer = ((OracleThinClob)clob).getCharacterOutputStream();
writer.write(msg);
writer.close();
}
st = conn.prepareStatement("UPDATE clob_tbl set clob_col=? where id=?");
st.setClob(1,clob);
st.setString(2,msgId);
st.executeUpdate();


That is all to be done. Though, there is an additional step in the flow of control, this solution is clean. Whew! some solace.

Script Debugger

Well, it has been a while since I coded something on the presentation layer. Infact, I had a very long term association with the Servlets, JSPs, and Struts and sort of bored with the web-db-web routines. The middleware is bit challenging and adventurous. Got some time yesterday, so explored the updates on the presentation layer including JSF. A friend pointed to a script debugger. This was really a cool tool that would literally eliminate the alert() in the javascript code. Progress!
If you'd like to try it out, the link to the tool is
Script Debugger. What more, it is from the Microsoft.

Listening to the PDF

This is nice and cool info from one of my colleagues. If you use Acrobat Reader 7.0, you can listen to the PDF. The following are the shortcuts:
ctrl + shift + b - To listen to the entire document
ctrl + shift + v - To listen to the current page
ctrl + shift + e - To Stop
ctrl + shift + c - To Resume

Very cool, isn't it. Try it out!!

Java Object to Oracle Object mapping

I had a requirement of passing large data to Oracle db and the various options I evaluated are:

1. Using PreparedStatements and batching the updates/inserts: Though this option seems apparent, I didn't consider it as a candidate for the reason that we need to use raw SQL statements and this violates our DB convention to use stored procs and functions. I'm not very sure if this would be fast either. So, dropped this option.

2. Using CLOBs: This would require a lot of change in the existing procs, so dropped it for the sake of reusing the existing procs. Someone, told me that CLOBs do not scale well.

3. Using Arrays: This is an elegant option. Most of the time, the data sent in bulk to DBs is primitive, ie, some numbers or strings or atmost encoded strings (NVARCHAR2). To evaluate sending info using array, I did the following:
a) created a table with just two columns first_name and last_name
b) created a oracle type to be a table of varchar2
create or replace type str_typ_t as table of varchar2(50);
c) created a stored proc to accept the string array as input and insert into the table.
d) The java class required some googling and experimentation, here is the code snippet:

ArrayDescriptor descriptor = ArrayDescriptor.createDescriptor("STR_TYP_T",conn);
ARRAY fnArray = new ARRAY(descriptor,conn,fnames);
That is all to be done. And I'm very happy to note the results. 300 inserts took just over a second!!!. This is Tres cool.

And if you would like to pass a java object to the stored proc and leave the mapping to the driver, the following lines would do the work:

StructDescriptor descriptor = StructDescriptor.createDescriptor("TEST_OBJ_T",conn);
Object[] attributes = {"ABCD","XYZ"};
STRUCT to = new STRUCT(descriptor,conn,attributes);


Some progress today. My PL/SQL skills have been really rusty, need more practice.

Some useful hacks

Well.. it has been a while since I blogged. The primary reason being, I'm very busy with the work and trying to get adjusted to the new project env. The project uses lot of latest technologies, Oracle AQ/Streams, Web Services etc. Some useful hacks I have found last month -

Web Services - This is applicable only for the Axis platform. Axis persists the info of deployed web services to a file by name server-config.wsdd that is created under the WEB-INF. This file, incidentally, has a structure similar to that of the deploy.wsdd file. Instead of creating a deploy file and calling AdminClient from the command prompt, the same entries can be given in the server-config.wsdd and bouncing the server. When the server is started the web services get deployed. This is useful if the deployer of the project is not much aware of the Web services and their deployment processes.
*** Update ***
This is an info I often forget,well, in rush to publish the WS and consume it ;-). If proper description is not given about the methods exposed by the WS, the input arguments are exposed as in0, in1, in2, etc. A few additional lines in the WSDD expose the appropriate and intuitive info about the input parameters:

<operation name="theMethodName" qname="ns:qnameFortheMethod" returnQName="ns:theReturnTypeQNameofTheMethod" returnType="ns:theReturnType">
<parameter name="nameOfTheParameter" type="example-xsd:string"/>
</operation>


I had to recollect this when one of our testing team members said that the WSDL aren't descriptive about the input parameters.

NVarchar2 issue - I observed a strange "No more data to read from socket error" if the return type of the strings is NVarchar2. This was noticed with the Oracle 10g and the latest jdbc driver. Two hacks to overcome this are: a) Change the data type to Varchar2, if you have DB control b) add a java VM argument "-Doracle.jdbc.defaultNChar=true".

Other than work, there seems to be little time for life. Trying best to establish the work-life balance, ;-)

Javascript trivia

One of my ex-colleagues was very passionate about Javascript and used to say that Javascript is one of the most misinterpreted and misused language by the developers. I had to recollect this while developing a trivial trim() function. Though there are many ways to implement the trim() functionality, I wanted to do it in a different (um, adventurous way). I explored a bit and found that there is something called "Prototype" for each of the data types used and new functions can be added dynamically. In this case, I wanted to add a trim function, so the following code would add a function called trim to the String data type

String.prototype.trim = function() { }

And now, filling up the function, I used Regular expressions, the leading spaces can be removed by

str=str.replace(/^\s*(.*)/, "$1");

and the trailing spaces by

str=str.replace(/(.*?)\s*$/, "$1");


The whole code would look like

String.prototype.trim = function() {
var str=this;
str=str.replace(/^\s*(.*)/, "$1");
str=str.replace(/(.*?)\s*$/, "$1");
return str;
}

This is may be a trivial discovery but now it has helped us a lot in the Javascipt code, instead of calling trim(str), we can just say str.trim().

JMS backed by Websphere MQ

This is a learning experience about a JMS queue backed by Websphere MQ. My initial understanding was that a message published on the MQ queue wouldn't invoke the onMessage method of a listener/MDB. The notion sounded logical as the Java/JMS equivalent of the message exposed by the MQ JMS is of type com.ibm.mq.JMSBytesMessage and the onMessage expects the argument to be of type javax.jms.Message. Yesterday, we were able to publish a message on MQ and get an MDB listen to the messages. The details follow...

Before getting into the nitty-gritty, some backdrop on the context - our requirement was to process messages put on the MQ "asynchronously". Our knowledge of MQ being naive, the task motivated us and we explored the MQI(java interface for MQ) and played with it by developing some sample code. This led to the roll out of a design using the MQI and the quartz scheduler. I named it the "Pull Approach". A quartz job would poll the MQ at a defined time intervals and retrieve any messages in the MQ queue. Developed the code for the Pull approach and gave a demo of it.

Intrigued by the challenge and exploring more, came up with a second design that would use the inbuilt features of the MQ. A trigger is invoked on publishing a message. The trigger "translates" the message to JMS compatible message and publishes the message on a Topic. This design was baptized as "Push Approach" as the MQ server was pushing the messages. A document has been prepared with both the approaches and with crisp state diagrams made using Rational Rose and circulated it across. There was an inherent flaw in the Push approach, it didn't give implicit support for transactions. There were mixed opinions amongst the folklore about the the approaches with offhsore favoring Pull and onsite for Push.

With adrenalin still gushing, we got more adventurous and explored the JMSAdmin of the MQ. We have been playing around with that for a while but couldnot hit the bush.We decided to give try it afresh from scratch. So, created a context in the JMSAdmin, bound the connection factory and queue to the JNDI, used the same bindings in the JMS configuration on Websphere App server. The prototype was ready for testing. And to our surprise, when we pubslished a message on the MQ, it did invoke the MDB, though there was a ClassCastException for com.ibm.jms.JMSBytesMEssage. The cast exception could be resolved by reading out the bytes out of the JMSBytesMessage and copying them onto the JMS TextMessage.

This was an interesting development, indeed. Although, the earlier two approaches would be promptly decommissioned in favor of the third approach, we are happy that we were able to suggest a robust design that leverages the inbuilt features of the product. The learning curve has taken off. The documentation and the articles found on the IBM developer works site didn't help us much and most of the examples appeared to ratify our initial understanding. Well, being li'l adventurous and thinking outta box does help sometimes. There will a sequel to this blog entry soon with gory technical details.

*** UPDATE ***
There is an article on IBM site that explains in detail about configuring Weblogic JMS for Websphere MQ. Read the article here. I wish the article would have been published earlier that would have saved a few days of my tussle .

Tussle with Websphere

Has been working with a variety of App. servers like Weblogic, JBoss, and JRun, but never got a chance to work with Websphere. Recently, got a chance to work with Websphere suit of products. First the websphere MQ and then the websphere server itself. The experience with MQ was bit adventurous, as the technology predates JMS. With some trial and error, intuition, googling, got the things up and running. Reb books helped a lot in understanding the internals for MQ.

To test the use cases against our design, we decided to develop a vertical slice of the application. One part of the slice was an MDB that listens to a JMS Queue on the websphere, backed by the MQ. The task is broken to two parts - have an MDB listening to the JMS queue on webspshere and once part 1 is in place, configure a MQ as a backend for the JMS queue. Developing an MDB was no much hassle and it got deployed on the websphere(The server had to be bounced everytime after configuring the Listener Ports and registering the Queue with the JMS server, and after deploying the jar file with MDB, duh, it sucks!). The next step was to publish some messages on the queue and see if the onMessage of the MDB is getting called.

I developed a suit of small utilities to help me in server-side testing, which I call as the fixture. The configuration of the fixture is externalized to facilitate portability. The messaging fixture worked without any errors when it was pointing to Weblogic, JBoss, JRun. But with Websphere, a lot of tweaking of the classpath is required. The following is the summary of the tweaking:

1. First of all, use the JDK that comes with the server, located at ${was_home}/java. If you don't, you'll waste a couple of hours like me.

2. Place these jars in the classpath:
com.ibm.mq.jar
com.ibm.mqjms.jar
dhbcore.jar
ecutils.jar
j2cImpl.jar
j2ee.jar
messagingClient.jar
naming.jar
namingClient.jar
namingserver.jar
webcontainer.jar
Properties folder. Place all or you'll repent.

3. Websphere works on iiop. So the JNDI props will be com.ibm.websphere.naming.WsnInitialContextFactory and iiop://localhost:2809.

To figure out what jars go in the classpath, it took me almost 6 hours. Extensive googling and some experimentation made the things work. It seems most of the people had a similar experience. Looks like I'm going to have some more tussles with websphere in the near future. Why can't the Websphere be as developer friendly as Weblogic.

Year.introspect(2005)

Yesterday,I was arranging my book-shelf to accomodate new books and a paper fell-out. It was a receipt of a book purchased from Gangarams book store, Bangalore, dated Jan 31, 2005. The book, Head First Design Patterns, is still my favorite book. After reading the book, there is a conspicuous change in the way I write code. I have undergone a transformation, hehehe. A friend calls it metamorphosis. I have recommended the book to all my colleagues and the next day there were 5 more purchases of the book.

Incidentally, on the same day Jan 31, 2005, I've put my papers at HP. The co-incidence of the events developed a retrospective urge and made me look back and introspect the last year. I had more >5 offers on hand from good companies at that moment. I've truned down all the offers from companies in Bangalore as the real estate prices skyrocketed and the traffic is horrendous. Subsequently, I decided to relocate to Hyderabad, my native place and narrowed down on a medium sized company doing product development for wealth management segment of the financial industry.

The year was of mixed results, sometimes working more than 14 hours a day and sometimes with absolutely no work for weeks together! oh yeah! there was an onsite trip for 3 months to San Francisco. Oflate, I have become a passionate reader[almost a bookworm ;-)] and mostly I'm reading tech books (currently reading Introduction to Algorithms by Thomas Cormen, Geeeee).Things worked out ,more or less,as expected and the year was a break-even, if not lucrative. Wondering what 2006 has in store for me, wink!

The Jaipur trip

A week long trip to Jaipur had been full of joy and happiness. I was wondering if I could make it to Vivek's wedding amidst so much work. My manager finally relented and approved a leave for 3 days. Weekend + republic day added 3 more days. 6 days of total fun and joy.

I have known Vivek since my engineering days. He is a Telugu speaking Marwadi :-).
The train journey to Jaipur and back was really entertaining. I travelled along with 40 other friends and relatives of Vivek. There was no need to turn on the mp3 player as the kids were playing anthakhasari non-stop. Though, the particpation from elders (including me) was meagre, it was nice to listen to the Hindi songs.

The marriage and the reception were very elegant and full of activity. I never got a chance to see a traditional north Indian marriage, except in the movies. The baraath was the ultimate. We danced and enjoyed the music played by the band. It was all fun and an unforgettable experience. Look at Vivek on the horse..