Vengatc techology logs

Framework – Enterprise Information Integration (EII) that I developed

Posted in Uncategorized by vengatc on October 10, 2008

Again a challenge in my current job , here I have a sandwidched role of an architect and a Senior Engineer .I have no scope for giving escuses but crack all the problem that come in the way in achieving what we want.

Requirement

Here we are building a datacenter based project where all the customers internal network information will recide in our company datacenter. So each customer will have his instance of database in the datacenter. Yeah i hear what you say… having instances of databases is a kill.. but we have to choose this so to cut down lot of development effort. So money (Vs) Design huristics??? Money wins.

Problem.

Everything goes good in having a seperate instances of a database until there is no need to aggregate information across multiple databases. We wanted to do BI on the available data. But consider the in numberable databases we have in the datacenter we need a way to run query across databases. I mean a mean to run a same query across databases. I evaluated ETL tools like Kettle, Talend etc. I also tried to hack MYSQL to have a datagreegation layer in easiest way.. But nothing helped, no of the technologies i explored was crisp and clear to my requirement.

In short. I need a tool that takes my query .. aggregates data from mulitple databases, the number of datasources will be increaseing in runtime … and stil have a unified facade kind of simple interface.

Solution

Again as usual… I never give up any oppurtunity in get for writing any innovative frameworks solving business problems. I choose to write the EII(Real time data aggregation framework)from scratch.

(More info on this work… will be updated… got a important work to attend…)