Session and Clustered Java Web Apps

Session can be a headache to work with in Java web applications. For that reason, most developers now use MVC frameworks, such as Java Server Faces, that hide the use of session and allow you to work with simple Java beans and configuration instead. But, that’s not always the case, especially where you have plain servlets that you have to maintain. With servlets, you have the power to put objects into and take objects out of session yourself. Alone, this presents thread-safety issues, but manually managing session in an application that is clustered presents more problems.

There is a jump that has to be made when moving from a single server environment to a clustered one. In a clustered environment, session management becomes complex because it has to be shared by all the servers in a cluster. This sharing of session becomes a distribution of objects strategy, which is an advanced topic you should familiarize yourself with before clustering your application. A high-end application server (i.e. IBM WebSphere) will handle most of this complexity for you, but you have to educate yourself on the myriad configuration options and their affect on your application.

In my recent excursion into running an existing Java web app in a clustered environment, I found that there was one major stumbling block from my programmer point-of-view—understanding the mechanism for distributing session objects between servers in a cluster. If you’re about to do the same for your application, the first thing to understand is that distribution of session objects is done through Java object serialization. That method is prescribed in the Servlet specification in section “SRV.7.7.2 Distributed Environments”, in fact. How application servers actually manage that so session is consistent is left up to the vendors and is one way in which they can compete.

Understanding that object serialization is the mechanism for sharing session is important to know. To start, it means that every single object that gets put into session has to do two things: 1.) it has to implement the class java.io.Serializable and 2.) it has to actually be serializable. An object is not really serializable unless its entire object graph is also serializable. So think about this, if you didn’t design your application with this requirement in mind, you could be in serious trouble when moving to a cluster. At the very least, you will have to identify all objects you put into session in your servlets and modify them to implement java.io.Serializable (which doesn’t require you to override any methods). At the worst, you will have to write custom serialization code for objects that don’t serialize naturally. The transient keyword can help with this by marking fields of objects that should be skipped during serialization.

The bright side to this discussion is that there is ample documentation out there on the topic, although it is spread out. Below I have annotated a few sources I found very helpful when I started researching this topic. I am working with an IBM WebSphere cluster, so most of the documentation is from IBM, however, all of the advice applies to any application server. Moving to a clustered environment is probably harder work than you realized, so you’ll be well served (and appreciated by your boss) if you check out these resources before making the jump to a clustered environment.

I’m still actively researching this topic myself. As such, I’ll keep updating this post as I learn more (read: I might be wrong on a detail or two ).

Best practices for using HTTP sessions

http://www14.software.ibm.com/webapp/wsbroker/redirect?version=compass&product=was-nd-mp&topic=cprs_best_practice

This is IBM’s advice on the best way to handle various aspects of session. It’s broadly applicable advice that, honestly, I wish I would have seen about 5 years ago when I first started working with web apps

Java Theory and Practice: State Replication in the Web Tier

http://www.ibm.com/developerworks/java/library/j-jtp07294/index.html

This is an older article by Brian Goetz that contains similar knowledge to the previous link.

WebSphere Application Server V7 Administration and Configuration Guide

http://www.redbooks.ibm.com/redbooks/pdfs/sg247615.pdf

Overall, this book is very WebSphere specific. However, Chapter 12 does describe how WebSphere handles distributed session and it’s very enlightening to read. On a guess, I’d say that other app servers do things similarly. There is actually a good bit of general discussion in Chapter 12, much more than there is explanation of what buttons to click to set it up so it’s worth the time to skim.

Designing and Coding Applications for Performance and Scalability in WebSphere Application Server

http://www.redbooks.ibm.com/redbooks/pdfs/sg247497.pdf

This book is far less specific to WebSphere than it sounds. Chapter 3 “General coding considerations” is a fantastic overview of how to deal with topics such as garbage collection, synchronization and database access. Most of the other chapters deal with issues that are similarly applicable to any application server, not just WebSphere.

The big bonus for downloading this RedBook is that it has a section called “Cluster considerations”. This section walks you through all of the things you should consider when you want to run your application on a cluster of servers.

Serializing Access to Session

http://www14.software.ibm.com/webapp/wsbroker/redirect?version=compass&product=was-nd-mp&topic=tprs_serializing_access

This link explains how to serialize access to objects in session for IBM WebSphere application server 7.0. I threw this link in to point out that session is basically a shared memory that multiple threads have access to concurrently. This is true whether you are running on a single server or a cluster of servers. This document is specific to WebSphere, but I’m guessing the other commercial app servers have similar options.

Getting Jython Modules to Work with the WebSphere wsadmin Tool.

I just finished wrestling with a problem for a day and a half – I’m writing down the solution for your benefit.  I’m not going into excruciating detail because I don’t understand it all.  This is just a quick look at the problem and what worked for me.  I hope it saves a little bit of time for a few people.

The Context

I am working on a set of scripts that I will use to automatically deploy a JEE EAR file to an IBM WebSphere version 7.0 (WAS 7) application server.  Even so, I believe this blog applies equally well to deploying a WAR file and on version 6.1 – 7.0 of WAS.  I am using a combination of Windows batch scripts, Ant scripts and Jython scripts.  The Jython scripts are being run through the wsadmin tool to do the WAS admin work.  I am setting the deployment scripts up in both a test and production environment.  The test environment is deploying from an automated build, which uses Ant to start the deployment process.  The production environment does not use Ant.  In both environments, there is a batch script that passes a Jython script to the wsadmin tool, where the Jython script does the actual WAS admin work (e.g. starting/stopping servers, deploying code and saving the configuration).

So to summarize, there are two deployment scenarios:

1. An Ant script, via the exec task, calls a Windows batch script, which runs the wsadmin tool with the necessary parameters, one of those parameters being a Jython script that does the deployment work.
2. A Windows batch script (which is part of another automated scheduling system) calls another Windows batch script, which runs the wsadmin tool with the necessary parameters, one of those parameters being a Jython script that does the deployment work.

The Problem

Once you’ve gotten jaded enough on IBM products and other such “enterprise-level tools”, the above setup is “basic”, so I’m assuming you’re just as twisted as I am and am not going to explain it further.  My actual problem is that I have written a Jython module that I want to use in my Jython deployment scripts.  Python, and thus Jython, is just like Java in that it has a specific route it follows when searching for referenced modules.  In regular Python programs the environment variable that is analogous to Java’s CLASSPATH is called PYTHONPATH.  The idea is exactly the same in both languages, the Python runtime has to know about all the directories where you expect Python modules to be in order for it to find them.  The same is true for Jython, although you have to setup the path a little differently since you’re running on the JVM.

All this isn’t complicated, but the wsadmin tool does make it harder to deal with.  The wsadmin tool sets up a very particular environment for itself, which is what tripped me up for so long.  Not to mention the muddled mess of running Python, via Jython, via the wsadmin tool, via a batch script, via an Ant script.  I was having a lot of trouble in actually importing my module because I did not have the Python path setup correctly, even though all of my Jython files were in the same directory.  If you have a Python file in the same directory as another module you are importing, you don’t have to do any special setup for Python to resolve the import.  However, with Jython and the wsadmin tool, your Jython files are being run out of the directory where either the JVM or wsadmin tool are being executed from, respectively.  This means that you have to setup the Python path, even if all of your Jython files are in the same directory.

The Solution(s)

It turns out that this is easy to solve, you just have to setup the Python path.  That seems obvious now, but since Python seems to handle module resolution fine in the same situation, it was confusing that Jython doesn’t.  An alternate solution is to be explicit about where the wsadmin tool should execute.  I ended up using each solution, one for the test environment and one for production.  In the test environment, where I’m starting with an Ant script, I’m using the dir attribute of the exec command to specify where to execute the command from.  This keeps the working directory for the wsadmin tool in the same place where my Jython scripts reside and allows the Jython runtime to find my referenced modules.  By “working directory” I mean the directory from which the wsadmin tool is actually executed. Without the dir attribute specified, the wsadmin tool will run in a different directory (the directory where the wsadmin.bat file is installed, I believe) and my Jython files will not be on the Python path.  Here’s an example of the Ant task:

<!-- The dir attribute is VERY IMPORTANT and allows the Jython scripts to import other scripts from this directory.-->
<exec executable="${env.WORKSPACE}/wasscripts/DeployWebServiceToCluster.bat"
      failonerror="true"
      failifexecutionfails="true"
      dir="${env.WORKSPACE}/wasscripts"
>
  <env key="WSADMIN_HOME" value="${env.WSADMIN.7.HOME}"/>
  <arg value="${wsadmin.props.file}"/>
  <arg value="${env.WORKSPACE}/wasscripts"/>
  <arg value="${env.WORKSPACE}/dist/${ear.file.name}"/>
  <arg value="${was.dmgr.name}" />
  <arg value="${was.cell.name}" />
  <arg value="${was.cluster.name}" />
  <arg value="${ant.project.name}"/>
  <arg value="${was.user}"/>
  <arg value="${was.user.password}"/>
</exec>

For the production environment I took the simpler route of passing in the Python path to the script that starts the wsadmin tool.  The wsadmin tool allows you to do this with the command line parameter -javaoption.  Here’s an example of what I did in the Windows batch script:

@rem %1 - Absolute path to the properties file.
@rem %2 - User with admin access to the WAS deployment manager.
@rem %3 - Password for the admin user.
@rem %4 - Absolute path to the Jython scripts folder.
@rem %5 through %8 - Parameters passed to Deploy_To_Production.py (see file for details).

call "%WSADMIN_HOME%\wsadmin.bat" -p %1 -user %2 -password %3 -javaoption "-Dpython.path=%~4" -f %4/deploy/Deploy_To_Production.py %5 %6 %7 %8
set ERRORLEVEL=%MYERRORLEVEL%
exit /B %MYERRORLEVEL%

The most crucial part is the -javaoption “-Dpython.path=%~4″.  Notice first that the entire value of the -javaoption parameter is enclosed in quotes.  It seems that you have to do this to get it to work correctly when passed to the JVM that gets started to run your Jython script. Since the entire value for the -javaoption parameter is in quotes, I’m using the Windows batch script trick (%~4 as opposed to just %4) to remove quotes from the fourth parameter to the batch script.  As an example, if the path passed into the batch script above as parameter four is

C:\temp

%4 will produce

"C:\temp"

whereas %~4 will output

C:\temp

This is also necessary to get things to work correctly.

Conclusion

I realize that there are many holes in my explanation.  The point is to briefly convey how to solve this problem if you should get hung up on it too.  I’m continuing to learn more about how to use Jython with the wsadmin tool for WAS scripting, so feel free to ask for clarification or about related problems.  Helping you will give me a chance to learn more as well.  Also, I found a pretty good blog about wsadmin scripting while I was researching this problem.

How to Enable MTOM Support for JAX-WS Web Services Running on WebSphere Application Server

I’m going to keep this post short today.  This is something I ran across today and it was rather irritating that I couldn’t find any good documentation on the subject. This is a highly specific problem but it might help a few of you out there.

Here’s the scenario.  I’m working on building a web service using JAX-WS 2.1 that is being deployed to WebSphere Application Server (WAS) 7.0.  I’m using Rational Application Developer (RAD) version 7.5.5.1 as my IDE. It’s a very simple web service that has to return moderate sized binary data chunks in the SOAP response (PDF files).  My team have decided to use the Message Transmission Optimization Mechanism (MTOM) as the method for attaching the bytes of the PDF files.  This is a relatively painless process if you have all the information you need.  However, the IBM WAS 7 InfoCenter has an incomplete description of how to enable MTOM on a web service.  IBM’s documentation tells you how to annotate your web service properly, but it doesn’t tell you that you also need to make a change to the IBM-specific webservices.xml file.  webservices.xml can be found in your Web Content\WEB-INF folder in your web application.

The change you have to make to webservices.xml is simple.  You just have to find the <webservice-description> tag for the web service on which you are enabling MTOM then change the value of the <enable-mtom> tag from false to true.  An example (a fragment of the webservices.xml file) of what this should look like is given below.

<webservice-description>
   <webservice-description-name>MyWebService</webservice-description-name>
   <port-component>
     <port-component-name>MyWebService</port-component-name>
     <wsdl-service xmlns:pfx="http://www.comapny.com/MyWebService/">
       pfx:DocumentService</wsdl-service>
     <wsdl-port xmlns:pfx="http://www.company.com/MyWebService/">
       pfx:DocumentServiceSOAP</wsdl-port>
     <enable-mtom>true</enable-mtom>
     <service-endpoint-interface>com.comapny.MyWebService</service-endpoint-interface>
     <service-impl-bean>
       <servlet-link>com.comapny.MyWebServiceSOAPImpl</servlet-link>
     </service-impl-bean>
   </port-component>
 </webservice-description>

Once your web service has been properly annotated and you make the above change to webservices.xml your web service will be setup to support MTOM.

Summary

To summarize, you have to follow two steps to get MTOM support in your JAX-WS web service:

1. Follow IBM’s documentation on annotating your web services implementation classes properly for MTOM support.
2. Set the <enable-mtom> tag to true in the webservices.xml file.

One more note is that the problem of IBM’s incomplete documentation is exasperated by the fact that their “Generate Java bean skeleton” wizard also neglects to update the webservices.xml file properly.  Even if you check the “Enable MTOM Support” check box on the “WebSphere JAX-WS Top Down Web Service Configuration” screen the webservices.xml file will remain unchanged after the wizard completes.  So whether you are using the wizard to create your web service code or doing it manually, you will have to update the webservices.xml afterward to fully enable MTOM support.

Explaining the Confusion

The webservices.xml file is actually completely optional — at least according to This WAS 7 InfoCenter page. Supposedly you can remove the webservices.xml file from your application and the annotations will be enough to enable MTOM support and you won’t have to go through any extra, manual steps.  If I sound unsure it’s because I am.  I tried removing the webservices.xml file from my web service and, after a re-deploy to the server, it was no longer accessible (received this error: “Error 404: SRVE0190E: File not found: /MyWebService”).  Once I restored the webservices.xml file and re-deployed I could reach the web service.

What I can verify is that the file is used on the server side to override any annotations that are found in the web service code.  This explains why MTOM doesn’t work unless you have <enable-mtom> set to true in webservices.xml.  Unfortunately, this is also probably why the file is not kept in sync when generating the code although I have no proof to back up that statement.  Please comment if you have a better or more correct explanation!

References

It seems that none of the official IBM references really catch this problem (otherwise I would not be writing this post ).  As mentioned earlier in this post, the IBM WAS 7 InfoCenter has some basic directions on enabling MTOM support that are incomplete.  Two other references of high quality, but also incomplete, with respect to MTOM support, are the Redbooks IBM WebSphere Application Server V7.0 Web Services Guide and Rational Application Developer V7.5 Programming Guide.  The Redbooks offer different levels of detail about MTOM support, so it’s worth your time to research MTOM in both books.

How to share session between web modules in WebSphere 6.1.

I recently had the *pleasure* of attempting to integrate a second Java web application module into an existing Enterprise application.  Throughout what became a mere exercise, I ran into several problems that were not well documented on the web. I found enough documentation to wade my way through to a solution but thought it would be helpful to document my experience for others’ benefit.  This post is a brief discussion of the actions you have to perform to share session amongst multiple web modules within a single Java Enterprise application.

Environment

The environment is a run-of-the-mill Java Enterprise business web application.  The JEE version used is 1.4, Java SDK 1.5 (IBM implementation), running on IBM WebSphere 6.1 application server (fixpack 17 in case that matters).  The application is deployed as an EAR with a single WAR.  Also important to note is that the application server and web module class loaders are set to load from the application up (inverted, or “parent last”).  The IDE being used is the WebSphere Development Studio Client version 7.

Requirements

The desire was to separate a web service client from the one WAR in the EAR and deploy the client as a second WAR inside the EAR.  It was necessary for servlets in the two web modules to share session state.   In other words, I wanted to have all the web modules in my EAR use one, and only one, session.  The two web modules would also be sharing some common code where each module maintained its own copies of the JAR files.

Solution

To meet these requirements there are multiple steps you must take.  The requirement to focus on here, especially if your situation is not exactly like mine, is sharing session amongst multiple web modules.  Sharing session is not a part of the servlet specification and so leads to some traps.  Each step is explained below.

Step 1: Turn on the Shared Session Context IBM extension in the enterprise application configuration.

A big problem with wanting to share session state between web modules or applications in a servlet container is that the official servlet specification forbids it.  You can read it for yourself in the Session Scope section of the version of the servlet specification you are using.  IBM overcomes this limitation with a non-standard (a.k.a. proprietary) extension that you can toggle called Shared Session Context.  How to toggle this extension is documented here.  You simply flip on this switch and WebSphere starts sharing session state across web modules.

Remember that, if you decide to use this IBM extension, it is non-standard and it is then unlikely that your web application will be portable to other application servers.  For sure, you will not be able to run your application on Apache Tomcat since that container adheres strictly to the servlet specification.

Step 1a: Work around a bug in Rational Application Developer (RAD) IDE for WebSphere.

NOTE: If you are using another IDE such as Eclipse, NetBeans, or IntelliJ IDEA, you can probably skip this step.

There is a bug in certain versions of Rational Application Developer (RAD) and WebSphere Developer Studio Client (WDSC) that must be worked around in order to perform Step 2 without error.  Before you continue you should read the IBM bug report and determine if your IDE is affected.  If not, skip straight to Step 2.  If you are affected, follow the directions in the bug report.  If you do have to work around the bug you may notice some side effects that I noticed after doing so.  The side effects were acute with my application because it has a large number of files in it (approximately 150MB).  You may not experience these problems with your application.

Side effects I experienced after using the work-around in the IDE bug:

1. The Integrated Solutions Console for WebSphere took minutes to load some pages.
2. Publishing my application to my development server took ~90 minutes each time.
3. I lost the option to change class loading order from Parent_First to Parent_Last.

Step 2: Change the class loader policy from Multiple to Single.

Turning on Shared Session Context in Step 1 immediately reveals another problem related to class loading.  By default, WebSphere uses unique class loaders for each WAR within an application. So even though two WARs might instantiate the exact same class, the Java runtime does not consider the resulting objects as compatible because they were loaded by different class loaders. The problem when trying to share session across WARs is that an object placed into session by one WAR will not be usable by another WAR. IBM has documented this part of their WebSphere architecture here.

For example, consider the situation where a web application installed in WebSphere is comprised of two web modules: AppA and AppB.  Assume that the WebSphere server has been set up as in Step 1 above.  If the two web modules are sharing session then it’s going to be common for a servlet in one to place an object in session for a servlet in the other module to retrieve.  For instance, consider that AppA places an instance of java.lang.String into session for AppB to use as shown below.

String stringPutIntoSessionByAppA = “appA stuff.”;

session.setAttribute(stringPutIntoSessionByAppA);

When AppB pulls this object out of session as below,

String appBString = (String) session.getAttribute(“stringPutIntoSessionByAppA”);

which only works because of turning on the non-standard option Shared Session Context, an exception like this will be thrown:

ClassCastException: java.lang.String is incompatible with java.lang.String.

This error illustrates that the two classes, whose package and class name match exactly, are not considered to be the same class by the JVM at runtime because each one was loaded by a different class loader.

To fix this problem you have configure the web application to use a single class loader for all web modules within it.  This class loader setting is changed through the WebSphere Integrated Solutions Console (a.k.a. the administration console).  With the server running, log into the Integrated Solutions Console and click on Applications –> Enterprise Applications and then click on the name of the application whose settings you are changing. Under the Detail Properties section click on Class loading and update detection.  Finally, you have two choice as shown below.

Class Loading Properties for a Web Application in the WebSphere 6.1 Integrated Solutions Console.

In the section WAR Class Loader Policy choose the option Single class loader for application.  Click the OK button and then save the configuration.  This setting will cause everything (all WARs and EJBs) in the EAR to be loaded under one class loader and allow session to be truly shared between two WARs.

Summary

If you want to write a web application for WebSphere 6.1 consisting of multiple web modules and you want all of the modules to share session you will have to do some configuration.  The first step, turning on Shared Session Context, is an IBM proprietary extension to the servlet specification.  Turning on this feature causes other problems and forces you to change your class loader settings on the server in order to get session sharing fully functional.