The Story:

         Recently, I’m working as a deployment engineer at customer site with my team members. The system under deployment consists of a cluster service, a NoSql database and some client applications which interact with the cluster/database by using the Dynamic Link Library (DLL) provided by the vender.

         We were requested to deploy one new module which collects the database usage and health statistics such as running time, average read/write rates, database response delays and etc. Let’s call it Statistics Service as brief. We were also asked to upgrade the DLL for each client programs. I didn’t think too much even a big question mark in my mind: “Why we also need to upgrade the DLL for client for the Statistics Service?”

         Everything went well at the beginning, but after deployment, we found that some statistics items were still zero. We re-checked our upgrade process including checking the configuration files, the file md5sums, DLL version numbers. It seems nothing was wrong. We asked the developer in the other city for help but no valuable information still. Then we did the deployment again and still no luck.

         After several days, problems were still there. We started to analysis the log of the Statistics Service line by line. We happened to recognize a phenomenon that if the client program creates the connection to the database before the Statistics Service is ready, the Statistics Service won’t initiate statistics data collection for this client. The question popped up in my mind again: “Why we also need to upgrade the DLL for client for the statistics service?”

         I made a call to the developer to check why the system has such weird behavior. To my surprise, he said that the clients notify the statistics service via the DLL method calls quietly. And the DLL’s cluster connection method creates a TCP connection to the Statistics Service in the background for the clients. I asked why? He says: “I’m a newcomer; I failed to persuade the developers of the cluster service to monitor the statistics data and notify the Statistics Service. So we added code for collecting the statistics data in the DLL file for client programs which is an easiest and fastest way.” And later I asked another question: “why we still have problems that some statistics items are still zero? We already checked that DLL version is the latest one. ” Again, he managed to surprise me again: “We just found that we might released different DLL versions with the same version numbers before.” Later that day, we received a new DLL and after upgrading, everything went OK!

         No, it’s still far away from OK!!!

The Questions:

         Why such a version control pitfall happened?

         Why does the DLL care about statistics and does weird work furtively?

         Why did the vender choose this easiest and fastest design without carefully thinking about the whole system and the consequences?

Further Reading:  

         What is DLL? http://zh.wikipedia.org/wiki/%E5%8A%A8%E6%80%81%E9%93%BE%E6%8E%A5%E5%BA%93

Lazy Makes Others Busy – a bad experience with DLL