Configure WebLogic application server to shutdown itself if fatal errors are detected while handling application lifecycle events

This article explains how to configure the WebLogic application server to shutdown itself if fatal errors are detected while handling application lifecycle events.

Note: the steps mentioned in this article were tested with WebLogic 12.1.1


Problem statement

WebLogic application server allows us to register listeners for handling application’s lifecycle events. This allows us to control the behavior during deployment, undeployment and redeployment of the application. (See reference #1)

Recently, my colleagues mentioned a problem they were facing in production. There was a load balancer between the client and the server. The load balancer would start sending requests to the server as soon as it found the server’s listen port open. The application had a lifecycle listener which initialized the application. If an exception was thrown from a lifecycle method, the application initialization would be interrupted but the server still opened the listen port. The load balancer would find the listen port open and start sending requests to the server. The requests would fail because the application was not initialized properly. The operators would find the server to be running and incorrectly conclude that the application was initialized properly; the operators were not expected to monitor the log files of hundreds of servers ! An hour or so after starting the server, the monitoring team would realize that the request failure rate is abnormally high and it would take another hour or so to fix the problem and the restart the application. Until then, the client requests served by this server would keep on failing.

Had the server not opened its listen port, the load balancer would not have sent requests to this server; instead, the load balancer would have redirected the requests to other live servers. The request failures would have been avoided.

So, what they desired is that (somehow) the WebLogic server should shutdown automatically if fatal errors are detected while handling application lifecycle events.


Solution

The documentation of WebLogic application lifecycle event listeners (see reference #1) does not explain how to achieve this.

But I found another way.

WebLogic also supports server-level startup and shutdown classes. (see reference #2).

The server-level startup class has a configuration option called ‘Failure is Fatal’. (see reference #3). If this option is selected, a failure in this startup class causes the server to shutdown itself.

So, all that I had to do was:

  1. Create a server-level startup class which throws an exception when it is informed that a fatal error has occured.
  2. Deploy the startup class to the server.
  3. Select the ‘Failure is Fatal’ option for this startup class.
  4. Configure the startup class to execute after the application lifecycle events are handled.
  5. Let the application lifecycle event handler inform the startup class when fatal error occurs while handling the lifecycle events.

Step 1: Create a server-level startup class which throws an exception when it is informed that fatal error has occurred

Example:

 
public class FailOnFatalErrorStartupClass {

public static boolean hasFatalFailures = false;

public static void main(String[] args) throws Exception {

if (hasFatalFailures) {

throw new Exception("Fatal error had occurred during application
initialization");

}

}


Step 2: Deploy the startup class to the server

Do the following:

  1. Configure the the startup class in the server. (see reference # 4)
  2. Deploy the startup class to the server. (see reference # 5)
  3. Add the startup class to the server’s classpath. (see reference # 6)

See reference # 7 for complete details.


Step 3: Select the ‘Failure is Fatal’ option for the startup class

See reference #4, #8 and #9 for more information.


Step 4: Configure the startup class to execute after the application lifecycle events are handled

Uncheck the “Run Before Application Deployments” and “Run Before Application Activations” options for the startup class.

See reference #8 for the meaning of these options.

See reference #5 for more information on how to configure these options.


Step 5: Make the application lifecycle event handler inform the startup class when fatal error occurs while handling the lifecycle events

 
public class ApplicationLifecycleListener extends
weblogic.application.ApplicationLifecycleListener {

@Override

public void postStart(ApplicationLifecycleEvent evt) {

try {

initialize();

} catch (Exception ex) {

//check if the exception is fatal

FailOnFatalErrorStartupClass.hasFatalFailures = true;

}

}


How it works

When the lifecycle event handler detects a fatal error, it sets the value of the boolean field, hasFatalFailures, of the startup class to true. After the application has been activated, but before the listen port is opened, the server executes the startup class. If the field hasFatalFailures is true, the startup class’s main method throws an exception. On catching the exception, the server checks the value of ‘Failure is Fatal’ option. Because this option is selected, the server shuts down itself.

Following are some logs which are printed on the standard output during this process:

 

<Apr 21, 2013 12:38:53 AM ICT> <Info> <WebLogicServer> <BEA-000377> <Starting WebLogic Server with Java HotSpot(TM) 64-Bit Server VM Version 23.7-b01

...

<Apr 21, 2013 12:39:02 AM ICT> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STANDBY.>

<Apr 21, 2013 12:39:02 AM ICT> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STARTING.>

...

[13] [Sun Apr 21 00:39:03 ICT 2013] App LifeCycle Event preStart

[13] [Sun Apr 21 00:39:05 ICT 2013] App LifeCycle Event postStart

java.lang.Exception: lock file already exists

at name.rahulb.sample.startup.ApplicationLifecycleListener.someValidation(ApplicationLifecycleListener.java:44)

at name.rahulb.sample.startup.ApplicationLifecycleListener.postStart(ApplicationLifecycleListener.java:23)

at weblogic.application.internal.flow.BaseLifecycleFlow$PostStartAction.run(BaseLifecycleFlow.java:309)

...

FailOnFatalErrorStartupClass : enter

<Apr 21, 2013 12:39:05 AM ICT> <Critical> <WebLogicServer> <BEA-000362> <Server failed. Reason:

There are 1 nested errors:

java.lang.Exception: Fatal error had occurred during application initialization

at name.rahulb.sample.startup.FailOnFatalErrorStartupClass.main(FailOnFatalErrorStartupClass.java:12)

 

<Apr 21, 2013 12:39:07 AM ICT> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FAILED.>

<Apr 21, 2013 12:39:07 AM ICT> <Error> <WebLogicServer> <BEA-000383> <A critical service failed. The server will shut itself down.>

<Apr 21, 2013 12:39:10 AM ICT> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FORCE_SHUTTING_DOWN.>

[1] [Sun Apr 21 00:39:10 ICT 2013] App LifeCycle Event preStop

[1] [Sun Apr 21 00:39:10 ICT 2013] App LifeCycle Event postStop


References

  1. http://docs.oracle.com/cd/E24329_01/web.1211/e24368/lifecycle.htm
  2. http://docs.oracle.com/cd/E24329_01/web.1211/e21048/overview.htm#i1075539
  3. http://download.oracle.com/docs/cd/E24329_01/apirefs.1211/e24403/mbeans/StartupClassMBean.html#FailureIsFatal
  4. http://docs.oracle.com/cd/E24329_01/apirefs.1211/e24401/taskhelp/startup_shutdown/ConfigureStartupAndShutdownClasses.html
  5. http://docs.oracle.com/cd/E24329_01/apirefs.1211/e24401/taskhelp/startup_shutdown/AssignStartupAndShutdownClasses.html
  6. http://docs.oracle.com/cd/E24329_01/apirefs.1211/e24401/taskhelp/startup_shutdown/AddStartupAndShutdownClassesToTheClasspath.html
  7. http://docs.oracle.com/cd/E24329_01/apirefs.1211/e24401/taskhelp/startup_shutdown/UseStartupAndShutdownClasses.html
  8. http://docs.oracle.com/cd/E24329_01/apirefs.1211/e24401/pagehelp/Corecoreclassesstartupclassconfigtitle.html#attributes
  9. http://docs.oracle.com/cd/E24329_01/apirefs.1211/e24403/core/index.html
Advertisements
This entry was posted in Application Server, Java, Java EE, WebLogic and tagged , , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s