Tom Janofsky Consulting
Here are the PowerPoint slides for a presentation I gave for the Delaware Valley BEA Users' Group, entitled "Weblogic Admin Best Practices".
This presentation covered a broad range of best practices, hints and tips, and general advice for the use of WebLogic in a production environment. [Disclaimer] Some of the following advice adheres closely to WebLogic recommendations (e.g. the security section) others are not BEA documented approaches (.wlnotdelete, how to hassle BEA support). Many statements are generalizations representing a personal point of view, and your mileage may vary.
Process
Integrate early and often on a cluster. There are almost always issues in clustered deployment and testing.
Test as early as possible on the deployment platform and configuration. There are frequently minor issues in compatibility between minor revisions.
Make sure that the development team has a designated build manager who is on call for deployments. WebLogic validation can be picky, failing to deploy if JSPs don’t precompile, or if the EJB validation doesn’t like the look of the schema.
When deploying a new release of an application, make sure you save the old ear file, and that the DBA exports the database before making any changes. That way, if the application does not deploy, you can 1) roll back to the previous database snapshot, and 2) reinstall the previous ear. Depending on the changes in database structure, it cannot be guaranteed that an older EAR file will deploy against a new schema.
Make sure that a development environment exists that models the QA and production environments as closely as possible. Just because new development works on a standalone WebLogic with an exploded deployment is no guarantee that it will work in a clustered EAR deployment. “It works on my PC” may well be true, but doesn’t mean it works in production.
Deployment Architecture
Strive for as much simplicity as possible. The fewer moving parts and servers you have and the smaller your clusters the easier they are to maintain.
Have a really good reason before splitting the presentation tier and the object tier. In CRUD style (create-read-update-delete) database-centric apps the communication between these tiers is extensive.
If you have webapp only applications, consider deployment on WebLogic Express Premium – it is basically WebLogic server, but without EJBs and JMS. It is cluster-able, and at a much lower cost. It can participate in a cluster with WebLogic server, if that cluster only hosts web apps (i.e., you can use you full WebLogic servers to backup Express by creating a cluster for your webapps across all servers, then using your load balancer to stay away from the WebLogic instances unless there is a failure.)
Prove a real improvement in putting static content on Apache before taking on the management overhead. (I.e., don’t split the web tier from the presentation tier unless it really helps. This adds another moving piece to administration, requires deployment of more than just an EAR, makes rollback harder, etc.)
Beware giving your production servers different internal and external names. http://edocs.bea.com/wls/docs70/cluster/planning.html#1111820
To evaluate an application before deploying it to a cluster, deploy it temporarily to the admin server. Test it on the admin server. If the application tests successfully, undeploy it from the admin server, and deploy it to the cluster. (Note that this requires targeting your connection pools and other required resources to the admin server as well as the cluster).
Domains, when to create – domains are best thought of as an administrative constraint. Create new domains when the roles of administrators do not intersect. (E.g., if your QA lead will have administrative rights for QA servers, establish a QA domain.) Also create a new domain for different product levels.
Debugging WebLogic Administration
The only time you ever need the WebLogic ".wlnotdelete" folders is when you need to delete them. If you have problems with deleted classes still being present in your application, or a bean that has been removed from an application EAR still trying to deploy and failing, try 1) shutting down the server and 2) deleting the not-delete (and if that fails, 3) delete the stage directories as well). This is the ‘hit it with the hammer’ solution to some strange WebLogic deployment problems. Undeploying and redeploying an application through the console can sometimes help, but does not delete all of these files. If all else fails, it is frequently faster to go through the steps to create a new domain and cluster and deploy in that than to diagnose a really fouled up configuration.
If you end up with a firewall between your application server and your database, set the connection pool initial size to 0, and set a shrink parameter. This will make sure that when the sockets are inactive (e.g. at night) the app server will close the connections instead of the firewall shutting them down.
Sometimes schema changes require a server restart. If new objects are created in a schema (or an existing object is modified via a drop and create) you may need to log out and log back in again to get permissions, which is easiest done in our case by bouncing the server(s).
As best as possible stay on one WL version in QA and production.
At this point (mid-2003), don’t be too afraid of upgrading WebLogic. Their QA process seems to have been yielding fairly stable software and migration paths since 6 have been fairly painless. In particular, moving from JDK 1.3.x to 1.4.x is fairly important, particularly for any deployment on Linux.
Backup the config.xml file, and checkpoint it before every configuration change. Config.booted is only a copy of the config.xml from the last time you started it up, and that may be too long ago to be useful.
For WebLogic 7 SP2, the console domain wizard is problematic. Use the X version if possible.
WebLogic licenses are IP bound. Make sure you plan for getting new licenses if changing network addresses or machines.
WebLogic licenses are always available via http://elicense.bea.com. Depending on contract terms, what you see in the license browse here can be very misleading. Call license support to help sort it out. The products listed and the number of CPUs might not reflect the contract. The number of CPUs in the downloaded license may not accurately reflect the CPUs you selected when you retrieved the license. It will still work.
If you are getting ClassCastExceptions or NoClassDefFoundErrors in a production environment, the most common cause (although not the only possibility) is a problem with the arrangement of classes in the deployment architecture. This is particularly common when someone makes changes to security related classes which are put on the system classpath. They occur not because the class isn’t in the EAR or the JAR, but because it was loaded higher up in the classloader hierarchy than the class it is looking for is located. They will require changes to the build packaging to fix.
If you get NotSerializableExceptions in production, this is most likely due to objects being placed in the HttpSession that were not tested in a clustered environment. It will require programming changes to fix.
Try, as much as possible, to shutdown running WebLogic servers as opposed to kill –9. This can occasionally lead to problems in the config.xml.
Try searching http://eSupport.bea.com and http://groups.google.com/groups?hl=en&lr=&ie=UTF-8&oe=UTF-8&group=weblogic.developer when help is needed.
When contacting BEA support, make sure you provide all the needed info upfront (even if it is irrelevant, e.g. you are having a startup problem and the form wants to know your database). Get past first level support as quickly as you can if the problem is tricky. Don’t be afraid to call your sales rep if you’re not happy with the turnaround.
Deadlocks can happen at runtime for various reasons (stale row locks being one example). To assist developers, it can be useful to create a thread dump while the system is in this state. To do this, find the offending java process, and send it a kill –3 (SIGQUIT). This will send a full thread dump to stdout (which is being logged to a file). WebLogic recommends doing it twice to eliminate the possibility that things are just happening slowly.
The parameters supplied for bean pool values in weblogic-ejb-jar.xml can cause problems if the values are far too small or far too large. See http://edocs.bea.com/wls/docs70/perform/WLSTuning.html#1115602. The default sizes are often preferable to very small or vary large numbers. Object creation is not usually a binding factor in these applications.
Some EJB monitoring information can be obtained through the console: Select EJB in the left pane, then the deployment you are interested in, then click the monitoring tab in the right pane. The select either stateless session beans or entity beans. This will show you info such as how many TX’s have been rolled back. Similarly monitoring the connection pools can show you if anyone has been waiting for a database connection.
TX timeout values can be set in the administration console under JTA. IF the application has any long-running (bulk) operations, it may be necessary to increase this value. Increasing it too much though may lead to situations where deadlock scenarios and harder to debug and become more difficult to tie back to user activity. See http://edocs.bea.com/wls/docs70/ConsoleHelp/domain_domain_config_jta.html#1104722
Performance
The first rule of performance improvement is you cannot improve unless you measure!
Focus on the quantity & quality of object tier <-> DB communications as the first place to benchmark when there are performance problems. For many CRUD style J2EE apps this is the usual culprit.
Throwing hardware at CRUD style (create-read-update-delete) database-centric apps frequently makes more sense on the database machine and network connectivity than WebLogic server machines themselves.
Though YMMV, these CRUD apps tend to be (from the perspective of the app server) network bound on database access, then memory bound, then CPU bound.
Make sure headers are set properly and images are cached at the browser. Almost all the images tend to be static. This can frequently alleviate the need for a proxy web tier in an intranet application.
Use replication groups if you add more servers to a cluster to cut down on unnecessary communication.
If you can, size the max of your connection pool to the max number of concurrent users you expect. This is a frequent throttle.
Security
Keep an eye on http://dev2dev.bea.com/advisories , or sign up for the advisory emails.
The user running the WebLogic server needs only to have read/write/execute privileges to the BEA home directory, the product installation directory, and the domain directory.
Do not use the SSL demo identity and trust in a production environment.
Restrict size and time limit of requests in production to limit DOS attacks. In the console Servers -> ServerName -> Connections->Protocols. Limit for HTTP, T3, IIOP.
Password protect the JDBC connection pools to a role. Otherwise they are available via JNDI to remote clients.
Enable security auditing. http://edocs.bea.com/wls/docs70/ConsoleHelp/security_7x.html#auditprovider
Don’t allow JSPs in production to have keepgenerated = true. Don’t allow builds in this environment. No source on this machine.
Don’t enable the Servlet servlet in production. http://edocs.bea.com/wls/docs70/webapp/components.html#configuring-servlets
Do not leave the FileServlet as the default servlet. http://edocs.bea.com/wls/docs70/webapp/components.html#default-servlet
Scripting
The attached script exposes the following capabilities of WebLogic Administration to the command line:
#!/bin/bash
#
# This script exposes some WebLogic maintenance functions on the command line
# In particular, it allows for stopping servers, and starting and stopping applications
# from the command line
#
# Upkeep: This script will need new servers added as they are added to the environment
#
# Author: Tom Janofsky
#
CLASSPATH=${BEA_HOME}/weblogic700/server/lib/weblogic.jar
#
# First, a menu to get the server we want to operate on
#
echo
echo "Server List"
echo "-----------"
echo
PS3="Select Server >"
OPTIONS="server1 server2 Quit"
select opt in $OPTIONS; do
if [ "$opt" = "server1" ]; then
ADMIN_URL="t3://server1:7001"
SERVER_URL="t3://server1:7001"
SERVER="server1"
break
elif [ "$opt" = "server2" ]; then
ADMIN_URL="t3://server2:7001"
SERVER_URL="t3://server2:7004"
SERVER="serv er2"
break
elif [ "$opt" = "Quit" ]; then
exit 0
else
echo "Not a valid server"
fi
done
#
#Get the admin password
#
echo
echo -n "Please enter the admin password for this server: "
read -s PA
echo
#
# Pick an operation
#
JAVA_CMD="${BEA_HOME}/jdk131_06/bin/java"
ADMIN_CMD="${JAVA_CMD} -classpath ${CLASSPATH} weblogic.Admin -url ${ADMIN_URL} -username weblogic -password ${PASSWORD}"
SERVER_CMD="${JAVA_CMD} -classpath ${CLASSPATH} weblogic.Admin -url ${SERVER_URL} -username weblogic -password ${PASSWORD}"
DEPLOY_CMD="${JAVA_CMD} -classpath ${CLASSPATH} weblogic.Deployer -adminurl ${ADMIN_URL} -username weblogic -password ${PASSWORD}"
PS3="Select Command >"
while [ true ]; do
echo
echo "Operations"
echo "----------"
echo
OPTIONS="Ping Shutdown View-License List-JNDI View-Log Thread-Dump Get-Server-State List-Installed-Applications Stop-Application Start-Application Pick-Another-Server Quit"
select opt in $OPTIONS; do
if [ "$opt" = "Ping" ]; then
${ADMIN_CMD} PING
break
elif [ "$opt" = "Shutdown" ]; then
${ADMIN_CMD} SHUTDOWN ${SERVER}
break
elif [ "$opt" = "View-License" ]; then
${ADMIN_CMD} LICENSES
break
elif [ "$opt" = "List-JNDI" ]; then
${SERVER_CMD} LIST /
break
elif [ "$opt" = "View-Log" ]; then
${SERVER_CMD} SERVERLOG
break
elif [ "$opt" = "Thread-Dump" ]; then
${SERVER_CMD} THREAD_DUMP
break
elif [ "$opt" = "Get-Server-State" ]; then
${SERVER_CMD} GET -pretty -type ServerRuntime -property State
break
elif [ "$opt" = "List-Installed-Applications" ]; then
${SERVER_CMD} GET -pretty -type ApplicationRuntime -property ApplicationName
break
elif [ "$opt" = "Start-Application" ]; then
echo "Enter application name:"
read APP_NAME
${DEPLOY_CMD} -undeploy -name ${APP_NAME}
break
elif [ "$opt" = "Stop-Application" ]; then
echo "Enter application name:"
read APP_NAME
${DEPLOY_CMD} -activate -name ${APP_NAME}
break
elif [ "$opt" = "Pick-Another-Server" ]; then
exit
elif [ "$opt" = "Quit" ]; then
exit 0
else
echo "Not a valid operation"
fi
done
done
And don't forget about direct programmatic access to MBeans - you can have complete control this way. Also be sure to check out WLShell. The attached program shows how to create and manipulate JDBC connection pools and data sources.
import org.w3c.dom.Document;
import org.w3c.dom.Element;
import org.w3c.dom.Node;
import org.w3c.dom.NodeList;
import org.xml.sax.SAXException;
import weblogic.management.Helper;
import weblogic.management.MBeanHome;
import weblogic.management.MBeanCreationException;
import weblogic.management.DistributedManagementException;
import weblogic.management.configuration.JDBCConnectionPoolMBean;
import weblogic.management.configuration.JDBCDataSourceMBean;
import weblogic.management.configuration.ServerMBean;
import javax.xml.parsers.DocumentBuilder;
import javax.xml.parsers.DocumentBuilderFactory;
import javax.xml.parsers.ParserConfigurationException;
import javax.management.InstanceNotFoundException;
import javax.management.InvalidAttributeValueException;
import java.io.*;
import java.util.ArrayList;
import java.util.List;
import java.util.Properties;
/**
* This class will create connection pools and data sources in a running WebLogic server
* from an XML file of the format:
*
*
<datasources>
<server name="myserver"
url="t3://localhost:7001"
adminUser="weblogic"
adminPassword="password">
<pool driver="oracle.jdbc.driver.OracleDriver"
url="jdbc:oracle:thin:@host:1521:SID"
user="USER"
password="PWD"
pool_name="pool_name"
ds_name="ds_name"
/>
<pool driver="oracle.jdbc.driver.OracleDriver"
url="jdbc:oracle:thin:@host2:1521:SID2"
user="user"
password="PWD"
pool_name="pool_name2"
ds_name="ds_name2"
/>
</server>
</datasources>
*
*
* Each pool entry will create its own connection pool and JDBC data source. If a connection pool
* of the given name already exists, it's parameters will be updated to the params in the file. If a
* data source of the given ds_name already exists, it will also be updated.
*
* @author Tom Janofsky
*
*/
public class CreateWebLogicJDBC {
public static void main(String[] args) throws Exception {
new CreateWebLogicJDBC().execute(args);
}
private void execute(String[] args) throws ParserConfigurationException, SAXException, IOException, InstanceNotFoundException, MBeanCreationException, InvalidAttributeValueException, DistributedManagementException {
InputStream is = null;
if (args.length != 1) { //try loading it from the classpath, if not, show usage
is = this.getClass().getResourceAsStream("datasources.xml");
if (is == null) usage();
} else {
is = new FileInputStream(args[0]);
}
//load and parse XML
WeblogicJDBCData[] dataSources = getDataSources(is);
BufferedReader br = new BufferedReader(new InputStreamReader(System.in));
System.out.print("WebLogic host URL is ("+WeblogicJDBCData.serverurl+") Hit enter to accept, or type new URL:");
String line = br.readLine();
if (line != null && line.trim().length() > 0) WeblogicJDBCData.serverurl = line;
System.out.print("WebLogic server name is ("+WeblogicJDBCData.serverName+") Hit enter to accept, or type new name:");
line = br.readLine();
if (line != null && line.trim().length() > 0) WeblogicJDBCData.serverName = line;
System.out.print("WebLogic username is ("+WeblogicJDBCData.adminUser+") Hit enter to accept, or type new user:");
line = br.readLine();
if (line != null && line.trim().length() > 0) WeblogicJDBCData.adminUser = line;
System.out.print("WebLogic password is ("+WeblogicJDBCData.adminPassword+") Hit enter to accept, or type new pwd:");
line = br.readLine();
if (line != null && line.trim().length() > 0) WeblogicJDBCData.adminPassword = line;
//create a datasource for each entry
for (int i = 0; i < dataSources.length; i++) {
WeblogicJDBCData source = dataSources[i];
//Get the admin home and the server bean
MBeanHome adminHome = Helper.getAdminMBeanHome(source.adminUser, source.adminPassword, source.serverurl);
ServerMBean serverMBean = (ServerMBean) adminHome.getAdminMBean(source.serverName, "Server");
JDBCConnectionPoolMBean poolBean = (JDBCConnectionPoolMBean) adminHome.findOrCreateAdminMBean(source.pool_name, "JDBCConnectionPool");
Properties pros = new Properties();
pros.put("user", source.user);
// Set DataSource attributes
poolBean.setURL(source.dsurl);
poolBean.setDriverName(source.driver);
poolBean.setProperties(pros);
poolBean.setPassword(source.password);
poolBean.setLoginDelaySeconds(1);
poolBean.setInitialCapacity(0);
poolBean.setMaxCapacity(2);
poolBean.setCapacityIncrement(1);
poolBean.setShrinkingEnabled(true);
poolBean.setShrinkPeriodMinutes(10);
poolBean.setRefreshMinutes(10);
poolBean.setTestTableName("dual");
poolBean.addTarget(serverMBean);
// Create DataSource MBean
JDBCDataSourceMBean dsMBeans = (JDBCDataSourceMBean) adminHome.findOrCreateAdminMBean(source.pool_name, "JDBCDataSource");
// Set DataSource attributes
dsMBeans.setJNDIName(source.ds_name);
dsMBeans.setPoolName(source.pool_name);
// Startup datasource
dsMBeans.addTarget(serverMBean);
is.close();
System.out.println("Set datasource = " + dataSources[i].dsurl);
}
}
/**
* Parse the XML file, and build data sources
* @param file
* @return
* @throws ParserConfigurationException
* @throws SAXException
* @throws IOException
*/
private static WeblogicJDBCData[] getDataSources(InputStream file) throws ParserConfigurationException, SAXException, IOException {
List list = new ArrayList();
DocumentBuilderFactory builder = DocumentBuilderFactory.newInstance();
builder.setValidating(false);
builder.setExpandEntityReferences(false);
DocumentBuilder db1 = builder.newDocumentBuilder();
DocumentBuilder db = db1;
Document doc = db.parse(file);
Element root = doc.getDocumentElement();
NodeList beans = root.getChildNodes();
for (int i = 0; i < beans.getLength(); i++) {
Node node = beans.item(i);
if (node.getNodeType() == Node.ELEMENT_NODE && node.getNodeName().equals("server")) {
Element element = (Element) node;
WeblogicJDBCData.serverurl = element.getAttribute("url");
WeblogicJDBCData.serverName = element.getAttribute("name");
WeblogicJDBCData.adminUser = element.getAttribute("adminUser");
WeblogicJDBCData.adminPassword = element.getAttribute("adminPassword");
NodeList children = node.getChildNodes();
for (int j = 0; j < children.getLength(); j++) {
Node child = children.item(j);
if (child.getNodeType() == Node.ELEMENT_NODE && child.getNodeName().equals("pool")) {
Element pool = (Element) child;
WeblogicJDBCData data = new WeblogicJDBCData();
data.driver = pool.getAttribute("driver");
data.ds_name = pool.getAttribute("ds_name");
data.user = pool.getAttribute("user");
data.password = pool.getAttribute("password");
data.dsurl = pool.getAttribute("url");
data.pool_name = pool.getAttribute("pool_name");
list.add(data);
}
}
}
}
return (WeblogicJDBCData[]) list.toArray(new WeblogicJDBCData[list.size()]);
}
/**
* Display usage and exit
*/
private static void usage() {
System.err.println("Usage: java CreateWebLogicJDBC ");
System.exit(1);
}
/**
* Inner class representing the data source
*/
private static class WeblogicJDBCData {
//deliberately static since these are the same for one run
static String serverurl;
static String serverName;
static String adminUser;
static String adminPassword;
String driver;
String dsurl;
String user;
String password;
String ds_name;
String pool_name;
}
}