Creating Thread Safe Data Engines with Spring using composition and the "template" pattern

In 2003, I had the privilege of working with an extremely talented architect, Jim Loverde.  Jim worked for NVisia and had designed a framework (along with other very talented NVisia developers) for batch processing that was the engine for an accounting back end that serviced multiple installations.  It was in the dark ages before Spring, we were wrapping our services with stateless session beans to get transactional behavior.

Shortly after I started, I began studying the framework code and began to see the similarities that the batch framework had with the implementation of the Java Servlet API.  Data was flowing through the methods via predefined paths and client code could be developed to extend the framework.   It was like a light went off in my head, I realized that larger patterns existed and that they could be adapted to fit completely different solutions.

A few years later, I was again working with the same group of developers, now at a different company and we started using Spring.  Osman Mazhar developed a set of services patterned after Loverde's service based architecture which laid the foundation for our team (Venu Gogineni, Mike Pasacrita and others) to develop a wide array of products by combining loosely coupled services into a set of enterprise products facilitating payment processing, event organization, and donation tracking.  What I started to see was that, with Spring, you could build almost anything from small pieces.

A few years after that, I was at Verizon, and I had the chance to develop my own framework and services.  I started with two basic best practices that were observed in the two preceding frameworks:

• Services should be built to have no shared state - caching solutions can be created to share data between threads - but as a basic rule - no class level variables are used.  This is not to say services don't delegate method calls into other objects that are injected into them, its just to say that data itself is never stored in the services. (Class level variables aside.. the main idea is to avoid mutable state)
• Services work as data driven engines and can be linked together with one service calling another.  All the while, thread safety is ensured as data flows from service method to service method without anything being shared at the class level.  Many levels of delegation can be created without worrying about any issue due to shared state.  (All spring services were designed to be singletons) 

In addition to learning from the two preceding frameworks, I also had started to read code from the Spring Framework itself.   In particular, the JdbcTemplate class was something that I studied.  I was working to develop my own library to facilitate Spring integration with Apache Torque.  In reading this code, I found that the template pattern was probably the most useful of all the patterns for standardizing data flow.  This fact, the value of the template pattern, had been highlighted to me by another co-worker Edward Bridges; Edward pointed out that many frameworks depend on the pattern to establish control in base layers while not limiting expressiveness in extending classes.  When I say template pattern, I mean the process of defining a common pipeline for data flow with built in hooks that client code can utilize to customize the generalized flow.

So in the new framework, I decided that, in addition to combining services with composition, I would standardize data flows in my engines using this pattern.  As it turns out, this flow abstraction can be applied at several levels at once.  Each layer in the structure has the ability to be customized in client code while maintaining central control over data flow via the template method in the base class.  Base classes define common flow and the children implement specific use cases.

As it turns out, this structure makes it very easy to generate stubbed out implementations that conform to a base design via modeling and Domain Specific Languages.  The base classes define the accepted relationships between services and that information is used a blueprint for future implementations.  The result is that via Emergent Design, a system evolves into one that can be easily replicated and even generated.

NOTE: the final structure is evolved... to a large degree, the design did not start out with all the final base classes and template methods.  In the beginning there was only a system conforming to the first two principles, it was via the process of Emergent design that patterns were abstracted and data flows were standardized.  Every enterprise system defines its own way of computing, and these patterns need to be first defined and then refined.  Moving from definition to refined state is the process of Emergent Design.  You are not going to find the "real" patterns for your enterprise systems word for work in any books - the "real" patterns are how your organization transforms and processes data.  OO design patterns are great, but they are not the final layer of pattern. You have to have working processes before you can see the "real" patterns. 

See:
http://c2.com/cgi/wiki?MakeItWorkMakeItRightMakeItFast
http://www.ibm.com/developerworks/library/j-eaed1/

In a nutshell,  You can't expect version 1 to be a finished product.  And you can't move from version 1 to version 3 without going though the refinement process.  The refinement process is defined via Emergent Design principles

NOTE:  the template pattern is not the only pattern available, I am not saying that ALL software should be created using this pattern.  I am just saying the pattern fits well with object hierarchies and provides a better solution that just overriding base class methods.