Issues with Connection Pooling in Tomcat/JBoss

Problem Statement:

One of the application is showing the below error message while accessing the application.  The application is raising this bug for every three or four months.

Ok, what is the bug here?

Cannot get a connection, pool exhausted error.

How the problem was identified?

First of all, I came to know that the problem is related to connection pooling, by observing the above specified error message details in log file.  I wanted to replicate the problem to identify the origin of the bug.  Then, I have gone through the project to verify the settings related to connection pooling.  I have identified an entry maxActive related to Connection Pooling as part of DB configuration settings.  The entry specifies the maximum number of connections that are available in connection pool.  To identify the bug, I have minimized its value from 200 to 5.  Then, I have tested the each and every operation at least 6 times.  Out of all operations, I found the above error at one operation.  The operation is throwing error as the operation is not getting an open connection.  So, I felt that the operation might be missed the close_connection statement. 

Then, I have gone through the code in that specific flow.  I found the closeConnection statement, but, it is referring the class/global variable.  In fact, the usage of class/global variable is very less in that java program.  Then, I removed the class/global variable and created the local variables when ever required.

Solution provided:

Quick solution provided:

*) Restart the web/application server. [and/or]

*) Enable removeAbandoned flag which is related to Connection Pooling as part of DB configuration settings.

<!-- To configure a DBCP DataSource so that abandoned dB connections are removed and recycled add the following paramater to the ResourceParams configuration for your DBCP DataSource Resource: -->

<parameter>

<name>removeAbandoned</name>

<value>true</value>

</parameter>

<!--Use the removeAbandonedTimeout parameter to set the number of seconds a dB connection has been idle before it is considered abandoned.   -->

<parameter>

<name>removeAbandonedTimeout</name>

<value>60</value> 

</parameter>                                

Suggested solution:

Removed the class/global variable and created the local variable.

Issues with Tomcat 5.0 Start/Restart

Tomcat _5.0  Start/Restart

Problem Statement :  This document describes features and drawbacks of various options of Starting/Restarting Tomcat Webserver.

We can Start/Restart the Tomcat Web server in the following two ways. 

1)      Through the services.msc,

2)    Through the startup.bat. 

Through the services.msc :-

Features :-

1)      It’s a GUI interface.  So, performing Tomcat Start/Stop/Restart operations are very easy.

2)    This process is going on background.  So, it won’t open another window.

3)    Tomcat Service progress information automatically redirects to logs directory’s stdout.log file.  So, at any time, we can review the service status. 

Note:- Tomcat overrides stdout.log file on Service Start-up.  If you want previous information, then you need to take backup.  This file also gives clear information on Tomcat Startup failures.

4)    We need not worry in case of Windows’ user logoff/restart the system.  It means, this process works with any user and starts automatically as a windows background service while windows Start/Restart.

Drawbacks :-

5)    But, only one problem with this feature, i.e., services.msc is also an application.  It starts from C:\WINNT\system32 directory.  So, by default, this application is pointing to this directory.  So, we need to place application dependent files (like properties files) into this directory.

6)    We should specify the path environment variables for JAVA_HOME and CATALINA_HOME.

Through the startup.bat :-

Drawbacks :-

1)      This is command-line application.  There are different batch files for Start/Shutdown the Tomcat Web Server.  To Restart the Tomcat Web Server, we have to do Shutdown and Start.

2)    This process opens a new console window.  This window is very sensitive.  We should not close it.  If we close this window, Tomcat gets shutdown automatically.  And also, we should not click on this window.  If you left-click on this window, Tomcat stops the service temporarily.  Then, all applications that are working on this web server gets hang.  Then, applications won’t work.

3)    This window displays progress information about Tomcat Service.  But, this information scrolls out of the screen after some time.  We can’t get it back and not possible to take backup.

4)    We should take care in case of Windows’ user logoff/restart the system.  In this case, Windows closes this window automatically.  We need to restart the tomcat again.

Features :-

5)    This batch file runs from <TOMCAT_HOME>/BIN directory.  We need to place application dependent files (like properties files) into this directory.

6)    The only one main advantage is, this start-up batch file sets JAVA_HOME and CATALINA_HOME path settings automatically for that instant.

Conclusion:- Start/Restart the Tomcat Web Server through services.msc is the best way.  But, we need to take care of dependent files as specified in   case 4. 

Liferay : Adding https feature (Secured Socket Layer - SSL) - Default port 443

·         (1) Configuration of https (Secured Socket Layer-SSL) feature : -

·         Stop the Tomcat

·         To prepare the keystore certificate, please follow the below steps: -

·         Type the following command and enter the values specified as below

C:\keytool –genkey –alias tomcat –validity 1000 –keyalg RSA  <press ENTER>

(Note: - “your firstname and lastname” MUST be hostname of your server and cannot be a IP address; this is very important as an IP address will fail client hostname verification even if it is correct.)

·         Enter keystore password:   changeit

·         What is your first and last name?

[Unknown]:   www.company_name.com

·         What is the name of your organizational unit?

[Unknown]:   Infra

·         What is the name of your organization?

[Unknown]:   Company Name

·         What is the name of your City or Locality?

[Unknown]:   Bengaluru

·         What is the name of your State or Province?

[Unknown]:   Karnataka

·         What is the two-letter country code for this unit?

[Unknown]:   IN

·         Is CN=www.company_name.com, OU=Infrastructure, O=Company Name, L=Bangalore, ST=Karnataka, C=IN correct? [no]:  y

·         Enter key password for <tomcat>

·         (RETURN if same as keystore password): <press ENTER>

·         Search for .keystore file and copy it to C:\Documents and Settings\Default User folder.  Otherwise, you will get an error message on Tomcat restart.  Tomcat searches for .keystore file at this path.

·         Uncomment the connector element for port No. 8443 and comment the connector element for port No. 8080 in the file

·         C:\training\liferay\tomcat\conf\server.xml

·         Look for the tag <Security-Constraint> and its sub-tag <user-data-constraint> in the file C:\training\liferay\tomcat\webapps\ROOT\WEB-INF\web.xml and configure their sub-tag value as

<transport-guarantee>CONFIDENTIAL</transport-guarantee>

·         Start the Tomcat

·         Check for the Tomcat’s log files in C:\training\liferay\tomcat\logs

·         Test the application using the URL: - https://localhost:8443

·         Login the application with Administrator’s User-Id and Password.  In Admin portlet, click on Enterprise menu and update the port number for the Portal URL and Home URL.   

·         Click the Save button.

·      (2) To convert the https port number of application from 8443 to default 443:

·         If Microsoft’s IIS was installed in the same system, create one more IP address for that system and configure the IIS with the new IP address.

·         Replace the port number 443 in place of 8443 in the file:

·         C:\training\liferay\tomcat\conf\server.xml

·         Test the application using the URL: - https://localhost

·         Login the application with Administrator’s User-Id and Password.  In Admin portlet, click on Enterprise menu and remove the port number for the Portal URL and Home URL.

• Click the Save button.

To make Tomcat Webserver start automatically on system startup

·         Open <TOMCAT-HOME>\BIN\service.bat  file

·         Look for this line:

"%EXECUTABLE%" //US//%SERVICE_NAME% --JvmOptions "-Dcatalina.base=%CATALINA_BASE%;-Dcatalina.home=%CATALINA_HOME%;-Djava.endorsed.dirs=%CATALINA_HOME%\common\endorsed" --StartMode jvm --StopMode jvm

·         Replace it with something like this:

"%EXECUTABLE%" //US//%SERVICE_NAME% --JvmOptions "-Dcatalina.base=%CATALINA_BASE%;-Dcatalina.home=%CATALINA_HOME%;-Djava.endorsed.dirs=%CATALINA_HOME%\common\endorsed;-Dfile.encoding=UTF8;-Duser.timezone=GMT;-Djava.security.auth.login.config=%CATALINA_HOME%\conf\jaas.config;" --StartMode jvm --StopMode jvm

·         Also change the maximum heap size of the JVM: (It should be at least 512)

"%EXECUTABLE%" //US//%SERVICE_NAME% ++JvmOptions "-Djava.io.tmpdir=%CATALINA_BASE%\temp" --JvmMs 128 --JvmMx 256

·         Open up a command prompt and navigate to tomcat/bin:

tomcat/bin>service.bat install

·         Type the command services.msc in Start -> Run window

·         Start the Apache Tomcat service and make its startup type as Automatic.

Tomcat and JBoss co-existence

Introduction

The purpose of this note is to make Tomcat and JBoss co-existence in one system.  If Tomcat has been installed in your system and you wanted to install JBoss in the same system, you may face some problems due to conflict of resources such as ports.  This note should help you solve them.

Scenario: -

Let’s say, you have installed JDK 1.4/1.5 and Tomcat 5.0/5.5 (on Port No. 8085) to deploy some applications.  Now, you want to install JBoss 3.2.6 (on Port No. 8080) to deploy some other EJB application.  You follow the normal procedure to install JBoss.  After installing, during, you may face the following error:

11:59:03,017 ERROR [Http11Protocol] Error initializing endpoint

java.net.BindException: Address already in use: JVM_Bind:8080

at org.apache.tomcat.util.net.PoolTcpEndpoint.initEndpoint(PoolTcpEndpoint.java:264)

at org.apache.coyote.http11.Http11Protocol.init(Http11Protocol.java:137)

.

.

at org.jboss.system.server.ServerImpl.doStart(ServerImpl.java:407)

at org.jboss.system.server.ServerImpl.start(ServerImpl.java:311)

at org.jboss.Main.boot(Main.java:145)

at org.jboss.Main$1.run(Main.java:399)

at java.lang.Thread.run(Thread.java:534)

Solution: -

The search suggested me to change the webserver Port Number from 8080 to Something-Else (for example, 8090).  8080 is a favorite port number for a lot of different applications these days.  Please search the below mentioned files and replace 8080 to another port, say, 8090 -

'...deploy/http-invoker.sar/META-INF/jboss-service.xml' and

'...deploy/jbossweb-tomcat50.sar/server.xml'

After doing these settings restart the web server. You may still face the following console message on JBoss startup: -

11:20:00,168 INFO  [ChannelSocket] Port busy 8009 java.net.BindException:

Address already in use: JVM_Bind

This appears to be a known issue with JBoss. You can read more about this in bugs.sun.com  and bug_id=5098261

As per this information, it is not a bug.

Try the below links to ensure everything is fine:

http://localhost:8090

http://localhost:8090/jmx-console

Conclusion: - 

Tomcat and JBoss can co-exist by a simple configuration of port settings at appropriate places.

Java Plug-in: Single Sign-On for Intranet Applications: JCIFS (Java CIFS client Library)

Java Plug-in to make Single Sign-On for Intranet Applications

JCIFS is an open source client library that implements the CIFS/SMB networking protocol in 100% Java.  CIFS is the standard file sharing protocol on the Microsoft Windows platform (Ex: Map Network Drive ...).  This client is used extensively in production on large Intranets.  The JCIFS SMB client library enables Java applications to remotely access shared files and directories on SMB file servers (i.e. a Microsoft Windows "share") in addition to domain, workgroup, and server enumeration of NetBIOS over TCP/IP networks.  It is an advanced implementation of the CIFS protocol supporting Unicode, batching, multiplexing of threaded callers, encrypted authentication, transactions, the Remote Access Protocol (RAP), and much more.

Tomcat Service is consuming 100% CPU memory

Update the server.xml (tomcatRoot\conf) as follows: 

Existing Settings:

<!-- Define an AJP 1.3 Connector on port 8009 -->
<Connector port="8009"
    enableLookups="false"  redirectPort="8443"  protocol="AJP/1.3"/>


Proposed Settings:

<!-- Define an AJP 1.3 Connector on port 8009 -->
<Connector port="8009"
      enableLookups="false"  redirectPort="8443"  protocol="AJP/1.3"
      maxThreads="50"  minSpareThreads="5"  maxSpareThreads="20"
      acceptCount="100"  connectionTimeout="2000"/>