今天兄弟单位的一台应用服务器需要从测试环境移交到机房并修改IP，在移交前关闭了应用服务器，在移交后启动weblogic时出现了问题，weblogic进程起来了，但是控制台进不了不能修改数据源设置(也就是修改jdbc连接串)。操作下如下:
手动执行启动脚本(但这里其实另一个同事已经执行过一次)
[root@cdydtest bin]# ./startWebLogic.sh . . JAVA Memory arguments: -Xms300m -Xmx300m -XX:CompileThreshold=8000 -XX:PermSize=256m -XX:MaxPermSize=256m . WLS Start Mode=Development . CLASSPATH=/usr/bea/patch_wls1036/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/usr/bea/jdk1.6.0_20/lib/tools.jar:/usr/bea/wlserver_10.3/server/lib/weblogic_sp.jar:/usr/bea/wlserver_10.3/server/lib/weblogic.jar:/usr/bea/modules/features/weblogic.server.modules_10.3.6.0.jar:/usr/bea/wlserver_10.3/server/lib/webservices.jar:/usr/bea/modules/org.apache.ant_1.7.1/lib/ant-all.jar:/usr/bea/modules/net.sf.antcontrib_1.1.0.0_1-0b2/lib/ant-contrib.jar:/usr/bea/wlserver_10.3/common/derby/lib/derbyclient.jar:/usr/bea/wlserver_10.3/server/lib/xqrl.jar . PATH=/usr/bea/wlserver_10.3/server/bin:/usr/bea/modules/org.apache.ant_1.7.1/bin:/usr/bea/jdk1.6.0_20/jre/bin:/usr/bea/jdk1.6.0_20/bin:/usr/lib64/qt-3.3/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin . *************************************************** * To start WebLogic Server, use a username and * * password assigned to an admin-level user. For * * server administration, use the WebLogic Server * * console at http://hostname:port/console * *************************************************** starting weblogic with Java version: java version "1.6.0_20" Java(TM) SE Runtime Environment (build 1.6.0_20-b02) Java HotSpot(TM) 64-Bit Server VM (build 16.3-b01, mixed mode) Starting WLS with line: /usr/bea/jdk1.6.0_20/bin/java -client -Xms300m -Xmx300m -XX:CompileThreshold=8000 -XX:PermSize=256m -XX:MaxPermSize=256m -Dweblogic.Name=AdminServer -Djava.security.policy=/usr/bea/wlserver_10.3/server/lib/weblogic.policy -Xverify:none -da -Dplatform.home=/usr/bea/wlserver_10.3 -Dwls.home=/usr/bea/wlserver_10.3/server -Dweblogic.home=/usr/bea/wlserver_10.3/server -Dweblogic.management.discover=true -Dwlw.iterativeDev= -Dwlw.testConsole= -Dwlw.logErrorsToConsole= -Dweblogic.ext.dirs=/usr/bea/patch_wls1036/profiles/default/sysext_manifest_classpath weblogic.Server
因为输出日志被重定向到了日志文件(47ggzj.log)中，所以这里没有显示完整的日志信息。
抓取java进程来判断是否weblogic已经启动
[root@cdydtest bin]# ps -ef | grep java root 11933 1 0 14:23 ? 00:00:17 /usr/bea/jdk1.6.0_20/bin/java -client -Xms300m -Xmx300m -XX:CompileThreshold=8000 -XX:PermSize=256m -XX:MaxPermSize=256m -Dweblogic.Name=AdminServer -Djava.security.policy=/usr/bea/wlserver_10.3/server/lib/weblogic.policy -Xverify:none -da -Dplatform.home=/usr/bea/wlserver_10.3 -Dwls.home=/usr/bea/wlserver_10.3/server -Dweblogic.home=/usr/bea/wlserver_10.3/server -Dweblogic.management.discover=true -Dwlw.iterativeDev= -Dwlw.testConsole= -Dwlw.logErrorsToConsole= -Dweblogic.ext.dirs=/usr/bea/patch_wls1036/profiles/default/sysext_manifest_classpath weblogic.Server root 14675 1 20 15:10 pts/3 00:00:01 /usr/bea/jdk1.6.0_20/bin/java -client -Xms300m -Xmx300m -XX:CompileThreshold=8000 -XX:PermSize=256m -XX:MaxPermSize=256m -Dweblogic.Name=AdminServer -Djava.security.policy=/usr/bea/wlserver_10.3/server/lib/weblogic.policy -Xverify:none -da -Dplatform.home=/usr/bea/wlserver_10.3 -Dwls.home=/usr/bea/wlserver_10.3/server -Dweblogic.home=/usr/bea/wlserver_10.3/server -Dweblogic.management.discover=true -Dwlw.iterativeDev= -Dwlw.testConsole= -Dwlw.logErrorsToConsole= -Dweblogic.ext.dirs=/usr/bea/patch_wls1036/profiles/default/sysext_manifest_classpath weblogic.Server root 14695 14567 0 15:10 pts/3 00:00:00 grep java
从上面的信息看出现了两个weblogic进程(pid为11933,14675)在运行，这是因为两个同事都手动执行了一次启动脚本。但登录不了weblogic控制台，如是查看weblogic日志文件。
[root@cdydtest base_domain]# cat 47ggzj.log
<nov 19, 2015 3:10:34 PM CST> <info> <security> <bea -090905> <disabling CryptoJ JCE Provider self-integrity check for better startup performance. To enable this check, specify -Dweblogic.security.allowCryptoJDefaultJCEVerification=true>
<nov 19, 2015 3:10:34 PM CST> <info> <security> <bea -090906> <changing the default Random Number Generator in RSA CryptoJ from ECDRBG to FIPS186PRNG. To disable this change, specify -Dweblogic.security.allowCryptoJDefaultPRNG=true>
<nov 19, 2015 3:10:35 PM CST> <info> <weblogicserver> <bea -000377> <starting WebLogic Server with Java HotSpot(TM) 64-Bit Server VM Version 16.3-b01 from Sun Microsystems Inc.>
<nov 19, 2015 3:10:45 PM CST> <info> <management> <bea -141281> <unable to get file lock, will retry …>
<nov 19, 2015 3:10:55 PM CST> <info> <management> <bea -141281> <unable to get file lock, will retry …>
<nov 19, 2015 3:11:05 PM CST> <info> <management> <bea -141281> <unable to get file lock, will retry …>
<nov 19, 2015 3:11:15 PM CST> <info> <management> <bea -141281> <unable to get file lock, will retry …>
<nov 19, 2015 3:11:25 PM CST> <info> <management> <bea -141281> <unable to get file lock, will retry …>
<nov 19, 2015 3:11:35 PM CST> <info> <management> <bea -141281> <unable to get file lock, will retry …>
<nov 19, 2015 3:11:45 PM CST> <info> <management> <bea -141281> <unable to get file lock, will retry …>
<nov 19, 2015 3:11:55 PM CST> <info> <management> <bea -141281> <unable to get file lock, will retry …>
<nov 19, 2015 3:12:05 PM CST> <info> <management> <bea -141281> <unable to get file lock, will retry …>
<nov 19, 2015 3:12:15 PM CST> <info> <management> <bea -141281> <unable to get file lock, will retry …>
<nov 19, 2015 3:12:25 PM CST> <info> <management> <bea -141281> <unable to get file lock, will retry …>
<nov 19, 2015 3:12:35 PM CST> <info> <management> <bea -141281> <unable to get file lock, will retry …>
<nov 19, 2015 3:12:45 PM CST> <info> <management> <bea -141281> <unable to get file lock, will retry …>
<nov 19, 2015 3:12:55 PM CST> <info> <management> <bea -141281> <unable to get file lock, will retry …>
<nov 19, 2015 3:13:06 PM CST> <info> <management> <bea -141281> <unable to get file lock, will retry …>
从上面消息可以看到不能获得文件锁，虽然weblogic进程已经在运行，但是不能执行任何操作。这里出现这个问题的原因是因为需要修改jdbc连接，因为数据库服务器的IP地址修改了，而原来在weblogic中配置的jdbc并没有修改，那么在启动weblogic时，就会一直尝试连接，在这时weblogic服务是没有成功启动的,也登录不了控制台,但尝试连接达到weblogic缺的次数后，就会放弃尝试连接而执行后续的启动操作，但这需要等待一定的时间，而这时业务人员说不能登录系统，一位同事如是登录weblogic控制不能登录，如是再次执行了一次启动脚本，所以出现了不能获得文件锁的问题。现在的处理方法是kill掉这两个weblogic进程，并删除被锁定的AdminServer.lok文件，再次执行weblogic启动脚本就能正常启动。
删除被锁定的AdminServer.lok文件
[root@cdydtest /]#cd /usr/bea/user_projects/domains/base_domain/servers/AdminServer/tmp [root@cdydtest tmp]# ls AdminServer.lok WebServiceUtils.ser _WL_internal _WL_user [root@cdydtest tmp]# rm AdminServer.lok rm: remove regular empty file `AdminServer.lok'? y
手动执行启动脚本
[root@cdydtest bin]# ./startWebLogic.sh . . JAVA Memory arguments: -Xms300m -Xmx300m -XX:CompileThreshold=8000 -XX:PermSize=256m -XX:MaxPermSize=256m . WLS Start Mode=Development . CLASSPATH=/usr/bea/patch_wls1036/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/usr/bea/jdk1.6.0_20/lib/tools.jar:/usr/bea/wlserver_10.3/server/lib/weblogic_sp.jar:/usr/bea/wlserver_10.3/server/lib/weblogic.jar:/usr/bea/modules/features/weblogic.server.modules_10.3.6.0.jar:/usr/bea/wlserver_10.3/server/lib/webservices.jar:/usr/bea/modules/org.apache.ant_1.7.1/lib/ant-all.jar:/usr/bea/modules/net.sf.antcontrib_1.1.0.0_1-0b2/lib/ant-contrib.jar:/usr/bea/wlserver_10.3/common/derby/lib/derbyclient.jar:/usr/bea/wlserver_10.3/server/lib/xqrl.jar . PATH=/usr/bea/wlserver_10.3/server/bin:/usr/bea/modules/org.apache.ant_1.7.1/bin:/usr/bea/jdk1.6.0_20/jre/bin:/usr/bea/jdk1.6.0_20/bin:/usr/lib64/qt-3.3/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin . *************************************************** * To start WebLogic Server, use a username and * * password assigned to an admin-level user. For * * server administration, use the WebLogic Server * * console at http://hostname:port/console * *************************************************** starting weblogic with Java version: java version "1.6.0_20" Java(TM) SE Runtime Environment (build 1.6.0_20-b02) Java HotSpot(TM) 64-Bit Server VM (build 16.3-b01, mixed mode) Starting WLS with line: /usr/bea/jdk1.6.0_20/bin/java -client -Xms300m -Xmx300m -XX:CompileThreshold=8000 -XX:PermSize=256m -XX:MaxPermSize=256m -Dweblogic.Name=AdminServer -Djava.security.policy=/usr/bea/wlserver_10.3/server/lib/weblogic.policy -Xverify:none -da -Dplatform.home=/usr/bea/wlserver_10.3 -Dwls.home=/usr/bea/wlserver_10.3/server -Dweblogic.home=/usr/bea/wlserver_10.3/server -Dweblogic.management.discover=true -Dwlw.iterativeDev= -Dwlw.testConsole= -Dwlw.logErrorsToConsole= -Dweblogic.ext.dirs=/usr/bea/patch_wls1036/profiles/default/sysext_manifest_classpath weblogic.Server
查看日志信息
[root@cdydtest base_domain]# cat 47ggzj.log 2015-11-19 15:23:46 Initializing Insur_CHANGDE from init-parameters: PowerSI Version v0.2.9(Build20140901) starttime:2015-11-19 15:23:46 trace_busiconn:1 trace_dbconn:0 hostname:cdydtest jdbclogger:/usr/bea/user_projects/domains/base_domain/applications/Insur_CHANGDE/WEB-INF/jdbclogger.properties homedir:/usr/bea/user_projects/domains/base_domain/applications/Insur_CHANGDE companyname:???? jdbclogger.maxBatchCount:10 jdbclogger.minRuntime:1000 jdbclogger.needCaller:true service_centerid: applicationname:Insur_CHANGDE scheduler_flag:0 logger:/usr/bea/user_projects/domains/base_domain/applications/Insur_CHANGDE/WEB-INF/log4j.properties log_level:0 instancename:cdydtest.Insur_CHANGDE <nov 19, 2015 3:23:46 PM CST> <notice> <loggingservice> <bea -320400> <the log file /usr/bea/user_projects/domains/base_domain/servers/AdminServer/logs/base_domain.log will be rotated. Reopen the log file if tailing has stopped. This can happen on some platforms like Windows.>
<nov 19, 2015 3:23:46 PM CST> <notice> <loggingservice> <bea -320401> <the log file has been rotated to /usr/bea/user_projects/domains/base_domain/servers/AdminServer/logs/base_domain.log00027. Log messages will continue to be logged in /usr/bea/user_projects/domains/base_domain/servers/AdminServer/logs/base_domain.log.>
<nov 19, 2015 3:23:46 PM CST> <notice> <log Management> <bea -170027> <the Server has established connection with the Domain level Diagnostic Service successfully.>
<nov 19, 2015 3:23:46 PM CST> <notice> <weblogicserver> <bea -000365> <server state changed to ADMIN>
<nov 19, 2015 3:23:46 PM CST> <notice> <weblogicserver> <bea -000365> <server state changed to RESUMING>
<nov 19, 2015 3:23:46 PM CST> <notice> <server> <bea -002613> <channel "Default" is now listening on 10.138.130.251:7001 for protocols iiop, t3, ldap, snmp, http.>
<nov 19, 2015 3:23:46 PM CST> <notice> <server> <bea -002613> <channel "Default[1]" is now listening on 127.0.0.1:7001 for protocols iiop, t3, ldap, snmp, http.>
<nov 19, 2015 3:23:46 PM CST> <notice> <weblogicserver> <bea -000331> <started WebLogic Admin Server "AdminServer" for domain "base_domain" running in Development Mode>
<nov 19, 2015 3:23:46 PM CST> <notice> <weblogicserver> <bea -000365> <server state changed to RUNNING>
<nov 19, 2015 3:23:46 PM CST> <notice> <weblogicserver> <bea -000360> <server started in RUNNING mode>
从上面信息可知weblogic成功启动。从这个故障的原因来看就是操作人员在处理问题时不够细心，在不能登录weblogic控制台时并没有检查当前已经启动weblogic服务，而就执行了启动脚本才产生的问题。处理问题时一定要搞清状况，了解必要的信息，弄清原因才能操作。