Established in 1991, CFT is one of the most successful and largest suppliers of banking and financial software in Russia and CIS. National banks and financial institutions use CFT products everyday to serve businesses and citizens and carry out millions of transactions.
CFT started development of its banking product line in 1995 using PL/SQL language and since that time products dramatically increased in size and complexity. Heterogeneous architecture, large number of different modules, Oracle database as a core, and constant improvement were requiring more and more powerful and expensive hardware on customer side to be able to run the products. Usage of PL/SQL technology limited the ability for CFT to run its products in clustered environments and required customers to purchase very expensive servers and hardware. I.e., in order to launch system for one of national banks, it needed to purchase 128-processor server that cost millions of dollars. To resolve that, CFT decided to port its products from PL/SQL to J2EE technology and used custom written PL/SQL-J2EE translators. As a result, stability and performance of the products was seriously damaged: test deployment in a national bank showed that the system crashed after 20 min of work while performance dropped substantially: a regular operation could take up to 5-7 minutes to get response from server. CFT started looking for a team who could help resolve the situation.
To increase stability and performance of banking product line and enable it run in clustered environment.
After initial analysis it was decided to split project in two phases:
1. Resolve stability issue making the system perform stably;
2. Look at performance optimization — according to contract it was required to raise performance not less than by 40%.
In work on first phase we started with profiling procedure to find weak points in configuration of IBM WebSphere that CFT used as middleware, in translated J2EE code, and in configuration of Oracle database.
For profiling we used a tool called JProfiler that allowed measuring response time for different stages of operations.
The first step was to check connections from application to Oracle database. Our team wrote custom tools and tests showed that this layer worked fine.
The second step in this phase was to check system for memory leaks. Our team again used JProfiler to make and compare memory stamps, and found memory leakage in business logic and in Hibernate library. Business logic was fixed and we found a patch for Hibernate to fix the issue.
The last step was in this phase was to check definitions of PL/SQL type wrappers that were written by CFT in order to allow PL/SQL types work properly in J2EE environment (basic types in PL/SQL and J2EE very differs). Analysis showed that type wrappers were written not optimally and we fixed that.
After these actions application began to work stably, but the system still required 5-7 minutes to response to an operator's initial connection.
To fix that we made multiple thread dumps and the tests revealed the following:
1. Configuration of IBM WebSphere server was done properly, but
2. Oracle drivers used crypto-function that was improperly implemented in IBM Java machine.
We used Apache BCEL (Byte Code Engineering Library) tool to replace crypto-function in Oracle JDBC drivers with another crypto-function, which was implemented much better in IBM Java machine.
Also, we optimized LIKE operation for Varchar type and that allowed to increase performance of basic operations on this type by 2.5-4 times. And optimization of fundamental type Number allowed for increase of its operations performance by 3-9 times. Additional improvements in business logic related to work with documents increased response time by 2 times.
After these improvements, final measurements showed increase of overall system performance by more than twice, and for some operations the increase was 300-900%.
CFT's major client — national bank — was satisfied with the improvement: system worked stably, without crashes, and gave fast responses.
All works were done by our team on-site and the whole project took only 1.5 months.