hp rx6600两台oracle数据库双机互备服务器其中一台经常自动关机，刚好在做巡检时遇到了就顺便检查一下原因.检查经常出故障的一台小机日志信息如下:
rx6600-1:[/]#cat /var/adm/syslog/syslog.log Nov 6 10:40:35 rx6600-1 syslogd: restart Nov 6 10:40:35 rx6600-1 vmunix: Found adjacent data tr. Growing size. 0x32a6000 -> 0x72a6000. Nov 6 10:40:35 rx6600-1 vmunix: Pinned PDK malloc pool: base: 0xe000000100d5a000 size=117400K Nov 6 10:40:35 rx6600-1 vmunix: Loaded ACPI revision 2.0 tables. Nov 6 10:40:35 rx6600-1 vmunix: MMIO on this platform supports Write Coalescing. Nov 6 10:40:35 rx6600-1 vmunix: Nov 6 10:40:35 rx6600-1 vmunix: MFS is defined: base= 0xe000000100d5a000 size= 5084 KB Nov 6 10:40:35 rx6600-1 vmunix: Unpinned PDK malloc pool: base: 0xe000000108000000 size=393216K Nov 6 10:40:35 rx6600-1 vmunix: NOTICE: cachefs_link(): File system was registered at index 5. Nov 6 10:40:35 rx6600-1 vmunix: emcp:GPX:Info: GPX emcpgpx_install() success. Nov 6 10:40:35 rx6600-1 vmunix: Nov 6 10:40:35 rx6600-1 above message repeats 2 times Nov 6 10:40:35 rx6600-1 vmunix: emcp:GPX:Info: DM emcpgpx_dm_install() success. Nov 6 10:40:35 rx6600-1 vmunix: emcp:GPX:Info: VLUMD emcpgpx_vlumd_install() success. Nov 6 10:40:35 rx6600-1 vmunix: emcp:GPX:Info: XCRYPT emcpgpx_xcrypt_install() success. Nov 6 10:40:35 rx6600-1 vmunix: NOTICE: nfs3_link(): File system was registered at index 8. Nov 6 10:40:35 rx6600-1 vmunix: NOTICE: mod_fs_reg: Cannot retrieve configured loading phase from KRS for module: cifs. Setting to load at INIT Nov 6 10:40:35 rx6600-1 vmunix: Nov 6 10:40:35 rx6600-1 vmunix: 0 sba Nov 6 10:40:35 rx6600-1 vmunix: 0/0 lba Nov 6 10:40:35 rx6600-1 vmunix: 0/0/1/0 rmp3f01 Nov 6 10:40:35 rx6600-1 vmunix: 0/0/1/1 rmp3f01 Nov 6 10:40:35 rx6600-1 vmunix: 0/0/1/2 asio0 Nov 6 10:40:35 rx6600-1 vmunix: 0/0/2/0 UsbOhci Nov 6 10:40:35 rx6600-1 vmunix: NOTICE: USB device attached. Identification String: Nov 6 10:40:35 rx6600-1 vmunix: Devices/Device/USB/Standard/hp/Unknown/0_1 Nov 6 10:40:35 rx6600-1 vmunix: <2.1.3.10.1008.4390.1> Nov 6 10:40:35 rx6600-1 vmunix: 0/0/2/0.0 UsbMiniBus Nov 6 10:40:35 rx6600-1 vmunix: Devices/Keyboard/USB/Boot/hp/Unknown/0_1 Nov 6 10:40:35 rx6600-1 vmunix: <2.305.3.100.1008.4390.1> Nov 6 10:40:35 rx6600-1 vmunix: 0/0/2/0.0.0 UsbBootKeyboard Nov 6 10:40:35 rx6600-1 vmunix: Devices/Mouse/USB/Standard/hp/Unknown/0_1 Nov 6 10:40:35 rx6600-1 vmunix: <2.307.3.10.1008.4390.1> Nov 6 10:40:35 rx6600-1 vmunix: 0/0/2/1 UsbOhci Nov 6 10:40:35 rx6600-1 vmunix: Devices/Device/USB/Standard/hp/Multibay/0_a1 Nov 6 10:40:35 rx6600-1 vmunix: <2.1.3.10.1008.294.161> Nov 6 10:40:35 rx6600-1 vmunix: 0/0/2/1.0 UsbMiniBus Nov 6 10:40:35 rx6600-1 vmunix: Devices/MassStorage-SCSI/USB/BulkOnly/hp/Multibay/0_a1 Nov 6 10:40:35 rx6600-1 vmunix: <2.310.3.150.1008.294.161> Nov 6 10:40:35 rx6600-1 vmunix: 0/0/2/1.0.0 UsbBulkOnlyMS Nov 6 10:40:35 rx6600-1 vmunix: Devices/ScsiControllerAdaptor/USB/BulkOnly/hp/Multibay Nov 6 10:40:35 rx6600-1 vmunix: <2.1000.3.150.1008.294> Nov 6 10:40:35 rx6600-1 vmunix: 0/0/2/1.0.16 UsbScsiAdaptor Nov 6 10:40:35 rx6600-1 vmunix: NOTICE: USB device attached. Identification String: Nov 6 10:40:36 rx6600-1 above message repeats 5 times Nov 6 10:40:35 rx6600-1 vmunix: 0/0/2/1.0.16.0 tgt Nov 6 10:40:35 rx6600-1 vmunix: 0/0/2/1.0.16.0.0 sdisk Nov 6 10:40:35 rx6600-1 vmunix: 0/0/2/1.0.16.7 tgt Nov 6 10:40:35 rx6600-1 vmunix: 0/0/2/1.0.16.7.0 sctl Nov 6 10:40:35 rx6600-1 vmunix: NOTICE: USB device attached. Identification String: Nov 6 10:40:35 rx6600-1 vmunix: Devices/Device/USB/Standard/Avocent/KVMAdaptor/1_0 Nov 6 10:40:35 rx6600-1 vmunix: <2.1.3.10.1572.833.256> Nov 6 10:40:35 rx6600-1 vmunix: 0/0/2/1.1 UsbMiniBus Nov 6 10:40:35 rx6600-1 vmunix: Devices/Keyboard/USB/Boot/Avocent/KVMAdaptor/1_0 Nov 6 10:40:35 rx6600-1 vmunix: <2.305.3.100.1572.833.256> Nov 6 10:40:35 rx6600-1 vmunix: 0/0/2/1.1.0 UsbBootKeyboard Nov 6 10:40:35 rx6600-1 vmunix: Devices/Mouse/USB/Boot/Avocent/KVMAdaptor/1_0 Nov 6 10:40:35 rx6600-1 vmunix: <2.307.3.100.1572.833.256> Nov 6 10:40:35 rx6600-1 vmunix: 0/0/2/1.1.1 UsbBootMouse Nov 6 10:40:35 rx6600-1 vmunix: NOTICE: USB device attached. Identification String: Nov 6 10:40:36 rx6600-1 above message repeats 2 times Nov 6 10:40:35 rx6600-1 vmunix: 0/0/2/2 UsbEhci Nov 6 10:40:35 rx6600-1 vmunix: 0/0/4/0 gvid_core Nov 6 10:40:35 rx6600-1 vmunix: 0/1 lba Nov 6 10:40:35 rx6600-1 vmunix: 0/2 lba Nov 6 10:40:35 rx6600-1 vmunix: 0/2/1/0 PCItoPCI Nov 6 10:40:35 rx6600-1 vmunix: fcd: Claimed HP AD193-60001 4Gb Fibre Channel port at hardware path 0/2/1/0/4/0 (FC Port 1 on HBA) Nov 6 10:40:35 rx6600-1 vmunix: 0/2/1/0/4/0 fcd Nov 6 10:40:35 rx6600-1 vmunix: 0/2/1/0/6/0 iether Nov 6 10:40:35 rx6600-1 vmunix: 0/3 lba Nov 6 10:40:35 rx6600-1 vmunix: 0/3/1/0 PCItoPCI Nov 6 10:40:35 rx6600-1 vmunix: fcd: Claimed HP AD193-60001 4Gb Fibre Channel port at hardware path 0/3/1/0/4/0 (FC Port 1 on HBA) Nov 6 10:40:35 rx6600-1 vmunix: 0/3/1/0/4/0 fcd Nov 6 10:40:35 rx6600-1 vmunix: 0/3/1/0/6/0 iether Nov 6 10:40:35 rx6600-1 vmunix: 0/4 lba Nov 6 10:40:35 rx6600-1 vmunix: sasd: Claimed HP PCI/PCI-X SAS MPT adapter at hardware path 0/4/1/0 Nov 6 10:40:35 rx6600-1 vmunix: 0/4/1/0 sasd Nov 6 10:40:35 rx6600-1 vmunix: 0/4/2/0 iether Nov 6 10:40:35 rx6600-1 vmunix: 0/4/2/1 iether Nov 6 10:40:35 rx6600-1 vmunix: 0/5 lba Nov 6 10:40:35 rx6600-1 vmunix: 0/5/1/0 PCItoPCI Nov 6 10:40:35 rx6600-1 vmunix: fcd: Claimed HP AD193-60001 4Gb Fibre Channel port at hardware path 0/5/1/0/4/0 (FC Port 1 on HBA) Nov 6 10:40:35 rx6600-1 vmunix: 0/5/1/0/4/0 fcd Nov 6 10:40:35 rx6600-1 vmunix: 0/5/1/0/6/0 iether Nov 6 10:40:35 rx6600-1 vmunix: 0/6 lba Nov 6 10:40:35 rx6600-1 vmunix: 0/6/1/0 PCItoPCI Nov 6 10:40:35 rx6600-1 vmunix: fcd: Claimed HP AD193-60001 4Gb Fibre Channel port at hardware path 0/6/1/0/4/0 (FC Port 1 on HBA) Nov 6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0 fcd Nov 6 10:40:35 rx6600-1 vmunix: 0/6/1/0/6/0 iether Nov 6 10:40:35 rx6600-1 vmunix: 0/7 lba Nov 6 10:40:35 rx6600-1 vmunix: Initializing the Ultra320 SCSI Controller at 0/7/1/0. Controller firmware version is 01.03.35.70 Nov 6 10:40:35 rx6600-1 vmunix: 0/7/1/0 mpt Nov 6 10:40:35 rx6600-1 vmunix: Initializing the Ultra320 SCSI Controller at 0/7/1/1. Controller firmware version is 01.03.35.70 Nov 6 10:40:35 rx6600-1 vmunix: 0/7/1/1 mpt Nov 6 10:40:35 rx6600-1 vmunix: 120 processor Nov 6 10:40:35 rx6600-1 vmunix: 121 processor Nov 6 10:40:35 rx6600-1 vmunix: 122 processor Nov 6 10:40:35 rx6600-1 vmunix: 123 processor Nov 6 10:40:35 rx6600-1 vmunix: 124 processor Nov 6 10:40:35 rx6600-1 vmunix: 125 processor Nov 6 10:40:35 rx6600-1 vmunix: 126 processor Nov 6 10:40:35 rx6600-1 vmunix: 127 processor Nov 6 10:40:35 rx6600-1 vmunix: 250 pdh Nov 6 10:40:35 rx6600-1 vmunix: 250/0 ipmi Nov 6 10:40:35 rx6600-1 vmunix: 250/1 asio0 Nov 6 10:40:35 rx6600-1 vmunix: 250/2 acpi_node Nov 6 10:40:35 rx6600-1 vmunix: 0/7/1/0.7 tgt Nov 6 10:40:35 rx6600-1 vmunix: 0/7/1/0.7.0 sctl Nov 6 10:40:35 rx6600-1 vmunix: 0/5/1/0/4/0.1 fcd_fcp Nov 6 10:40:35 rx6600-1 vmunix: 0/5/1/0/4/0.1.9.0.0 fcd_vbus Nov 6 10:40:35 rx6600-1 vmunix: 0/5/1/0/4/0.1.9.255.0 fcd_vbus Nov 6 10:40:35 rx6600-1 vmunix: 0/5/1/0/4/0.1.13.255.0 fcd_vbus Nov 6 10:40:35 rx6600-1 vmunix: 0/5/1/0/4/0.1.13.255.0.0 tgt Nov 6 10:40:35 rx6600-1 vmunix: 0/5/1/0/4/0.1.13.255.0.0.0 sdisk Nov 6 10:40:35 rx6600-1 vmunix: 0/5/1/0/4/0.1.9.0.0.0 tgt Nov 6 10:40:35 rx6600-1 vmunix: 0/5/1/0/4/0.1.9.255.0.0 tgt Nov 6 10:40:35 rx6600-1 vmunix: 0/5/1/0/4/0.1.9.0.0.0.0 sdisk Nov 6 10:40:35 rx6600-1 vmunix: 0/5/1/0/4/0.1.9.255.0.0.0 sctl Nov 6 10:40:35 rx6600-1 vmunix: 0/5/1/0/4/0.1.9.0.0.0.1 sdisk Nov 6 10:40:35 rx6600-1 vmunix: 0/5/1/0/4/0.1.9.0.0.0.2 sdisk Nov 6 10:40:35 rx6600-1 vmunix: 0/5/1/0/4/0.1.9.0.0.0.3 sdisk Nov 6 10:40:35 rx6600-1 vmunix: 0/5/1/0/4/0.1.9.0.0.0.4 sdisk Nov 6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0.1 fcd_fcp Nov 6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0.1.9.0.0 fcd_vbus Nov 6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0.1.9.255.0 fcd_vbus Nov 6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0.1.13.0.0 fcd_vbus Nov 6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0.1.13.255.0 fcd_vbus Nov 6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0.1.13.0.0.0 tgt Nov 6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0.1.13.255.0.0 tgt Nov 6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0.1.9.0.0.0 tgt Nov 6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0.1.13.0.0.0.0 sdisk Nov 6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0.1.9.255.0.0 tgt Nov 6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0.1.13.255.0.0.0 sctl Nov 6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0.1.9.0.0.0.0 sdisk Nov 6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0.1.9.255.0.0.0 sctl Nov 6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0.1.13.0.0.0.1 sdisk Nov 6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0.1.13.0.0.0.2 sdisk Nov 6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0.1.13.0.0.0.3 sdisk Nov 6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0.1.9.0.0.0.1 sdisk Nov 6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0.1.13.0.0.0.4 sdisk Nov 6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0.1.9.0.0.0.2 sdisk Nov 6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0.1.9.0.0.0.3 sdisk Nov 6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0.1.9.0.0.0.4 sdisk Nov 6 10:40:35 rx6600-1 vmunix: 0/7/1/1.7 tgt Nov 6 10:40:35 rx6600-1 vmunix: 0/7/1/1.7.0 sctl Nov 6 10:40:35 rx6600-1 vmunix: 0/4/1/0.0.0 sasd_vbus Nov 6 10:40:35 rx6600-1 vmunix: 0/4/1/0.0.0.0 tgt Nov 6 10:40:35 rx6600-1 vmunix: 0/4/1/0.0.0.0.0 sdisk Nov 6 10:40:35 rx6600-1 vmunix: Boot device's HP-UX HW path is: 0/4/1/0.0.0.0.0 Nov 6 10:40:35 rx6600-1 vmunix: Nov 6 10:40:35 rx6600-1 vmunix: System Console is on the Built-In Serial Interface Nov 6 10:40:35 rx6600-1 vmunix: iether0: INITIALIZING HP AD193-60001 PCI/PCI-X 1000Base-T 4Gb FC/1000B-T Combo Adapter at hardware path 0/2/1/0/6/0 Nov 6 10:40:35 rx6600-1 vmunix: iether1: INITIALIZING HP AD193-60001 PCI/PCI-X 1000Base-T 4Gb FC/1000B-T Combo Adapter at hardware path 0/3/1/0/6/0 Nov 6 10:40:35 rx6600-1 vmunix: iether2: INITIALIZING HP AB352-60003 PCI/PCI-X 1000Base-T Dual-port Core at hardware path 0/4/2/0 Nov 6 10:40:35 rx6600-1 vmunix: iether4: INITIALIZING HP AD193-60001 PCI/PCI-X 1000Base-T 4Gb FC/1000B-T Combo Adapter at hardware path 0/5/1/0/6/0 Nov 6 10:40:35 rx6600-1 vmunix: iether5: INITIALIZING HP AD193-60001 PCI/PCI-X 1000Base-T 4Gb FC/1000B-T Combo Adapter at hardware path 0/6/1/0/6/0 Nov 6 10:40:35 rx6600-1 vmunix: iether3: INITIALIZING HP AB352-60003 PCI/PCI-X 1000Base-T Dual-port Core at hardware path 0/4/2/1 Nov 6 10:40:35 rx6600-1 vmunix: Logical volume 64, 0x3 configured as ROOT Nov 6 10:40:35 rx6600-1 vmunix: Logical volume 64, 0x2 configured as SWAP Nov 6 10:40:35 rx6600-1 vmunix: Logical volume 64, 0x2 configured as DUMP Nov 6 10:40:35 rx6600-1 vmunix: Swap device table: (start & size given in 512-byte blocks) Nov 6 10:40:35 rx6600-1 vmunix: entry 0 - major is 64, minor is 0x2; start = 0, size = 16777216 Nov 6 10:40:35 rx6600-1 vmunix: Dump device table: (start & size given in 1-Kbyte blocks) Nov 6 10:40:35 rx6600-1 vmunix: entry 0000000000000000 - major is 31, minor is 0x30000; start = 2349940, size = 8388604 Nov 6 10:40:35 rx6600-1 vmunix: Starting the STREAMS daemons-phase 1 Nov 6 10:40:35 rx6600-1 vmunix: Create STCP device files Nov 6 10:40:35 rx6600-1 vmunix: Starting the STREAMS daemons-phase 2 Nov 6 10:40:35 rx6600-1 vmunix: $Revision: vmunix: B11.23_LR FLAVOR=perf Fri Aug 29 22:35:38 PDT 2003 $ Nov 6 10:40:35 rx6600-1 vmunix: Memory Information: Nov 6 10:40:35 rx6600-1 vmunix: physical page size = 4096 bytes, logical page size = 4096 bytes Nov 6 10:40:35 rx6600-1 vmunix: Physical: 25133536 Kbytes, lockable: 18994328 Kbytes, available: 22051156 Kbytes Nov 6 10:40:35 rx6600-1 vmunix: Nov 6 10:40:36 rx6600-1 nettl[832]: nettl starting up. Nov 6 10:40:48 rx6600-1 sshd[986]: Server listening on :: port 22. Nov 6 10:40:48 rx6600-1 sshd[986]: Server listening on 0.0.0.0 port 22. Nov 6 10:40:49 rx6600-1 rpcbind: check_netconfig: Found CLTS loopback transport Nov 6 10:40:49 rx6600-1 rpcbind: check_netconfig: Found COTS loopback transport Nov 6 10:40:49 rx6600-1 rpcbind: check_netconfig: Found COTS ORD loopback transport Nov 6 10:40:49 rx6600-1 rpcbind: init_transport: check binding for udp Nov 6 10:40:49 rx6600-1 rpcbind: init_transport: check binding for tcp Nov 6 10:40:49 rx6600-1 rpcbind: init_transport: check binding for ticlts Nov 6 10:40:49 rx6600-1 rpcbind: init_transport: check binding for ticotsord Nov 6 10:40:49 rx6600-1 rpcbind: init_transport: check binding for ticots Nov 6 10:40:50 rx6600-1 inetd[1100]: Reading configuration Nov 6 10:40:50 rx6600-1 inetd[1100]: ftp/tcp: Added service, server /usr/lbin/ftpd Nov 6 10:40:50 rx6600-1 inetd[1100]: telnet/tcp: Added service, server /usr/lbin/telnetd Nov 6 10:40:50 rx6600-1 inetd[1100]: tftp/udp: Added service, server /usr/lbin/tftpd Nov 6 10:40:50 rx6600-1 inetd[1100]: login/tcp: Added service, server /usr/lbin/rlogind Nov 6 10:40:50 rx6600-1 inetd[1100]: shell/tcp: Added service, server /usr/lbin/remshd Nov 6 10:40:50 rx6600-1 inetd[1100]: exec/tcp: Added service, server /usr/lbin/rexecd Nov 6 10:40:50 rx6600-1 inetd[1100]: ntalk/udp: Added service, server /usr/lbin/ntalkd Nov 6 10:40:50 rx6600-1 inetd[1100]: auth/tcp: Added service, server /usr/lbin/identd Nov 6 10:40:50 rx6600-1 inetd[1100]: printer/tcp: Added service, server /usr/sbin/rlpdaemon Nov 6 10:40:51 rx6600-1 inetd[1100]: daytime/tcp: Added service, server internal Nov 6 10:40:51 rx6600-1 inetd[1100]: daytime/udp: Added service, server internal Nov 6 10:40:51 rx6600-1 inetd[1100]: time/tcp: Added service, server internal Nov 6 10:40:51 rx6600-1 inetd[1100]: echo/tcp: Added service, server internal Nov 6 10:40:51 rx6600-1 inetd[1100]: echo/udp: Added service, server internal Nov 6 10:40:51 rx6600-1 inetd[1100]: discard/tcp: Added service, server internal Nov 6 10:40:51 rx6600-1 inetd[1100]: discard/udp: Added service, server internal Nov 6 10:40:51 rx6600-1 inetd[1100]: chargen/tcp: Added service, server internal Nov 6 10:40:51 rx6600-1 inetd[1100]: chargen/udp: Added service, server internal Nov 6 10:40:51 rx6600-1 inetd[1100]: kshell/tcp: Added service, server /usr/lbin/remshd Nov 6 10:40:51 rx6600-1 inetd[1100]: klogin/tcp: Added service, server /usr/lbin/rlogind Nov 6 10:40:51 rx6600-1 inetd[1100]: dtspc/tcp: Added service, server /usr/dt/bin/dtspcd Nov 6 10:40:51 rx6600-1 inetd[1100]: recserv/tcp: Added service, server /usr/lbin/recserv Nov 6 10:40:51 rx6600-1 inetd[1100]: swat/tcp: Added service, server /opt/samba/bin/swat Nov 6 10:40:51 rx6600-1 inetd[1100]: registrar/tcp: Added service, server /etc/opt/resmon/lbin/registrar Nov 6 10:40:51 rx6600-1 inetd[1100]: hacl-probe/tcp: Added service, server /opt/cmom/lbin/cmomd Nov 6 10:40:51 rx6600-1 inetd[1100]: hacl-cfg/udp: Added service, server /usr/lbin/cmclconfd Nov 6 10:40:51 rx6600-1 inetd[1100]: hacl-cfg/tcp: Added service, server /usr/lbin/cmclconfd Nov 6 10:40:51 rx6600-1 inetd[1100]: instl_boots/udp: Added service, server /opt/ignite/lbin/instl_bootd Nov 6 10:40:51 rx6600-1 inetd[1100]: omni/tcp: Added service, server /opt/omni/lbin/inet Nov 6 10:40:51 rx6600-1 inetd[1100]: rpc.cmsd/udp: Added service, server /usr/dt/bin/rpc.cmsd Nov 6 10:40:51 rx6600-1 inetd[1100]: rpc.ttdbserver/tcp: Added service, server /usr/dt/bin/rpc.ttdbserver Nov 6 10:40:51 rx6600-1 inetd[1100]: Configuration complete Nov 6 10:40:53 rx6600-1 EMCPP: emcpAudit: Info: cmd=powermt: restore (user ID real=0 effective=0) Nov 6 10:40:53 rx6600-1 EMCPP: emcpAudit: Info: cmd=powermt: config (user ID real=0 effective=0) Nov 6 10:40:53 rx6600-1 EMCPP: emcpAudit: Info: cmd=powermt: save (user ID real=0 effective=0) Nov 6 10:40:54 rx6600-1 su: + tty?? root-sfmdb Nov 6 10:41:06 rx6600-1 cimserver[1706]: starting Nov 6 10:41:29 rx6600-1 cimserver[1707]: PGS10026: THE CIM SERVER IS LISTENING ON HTTPS PORT 5,989. Nov 6 10:41:29 rx6600-1 cimserver[1707]: PGS10028: THE CIM SERVER IS LISTENING ON THE LOCAL CONNECTION SOCKET. Nov 6 10:41:29 rx6600-1 cimserver[1707]: PGS10030: STARTED HP-UX WBEM Services VERSION A.02.07. Nov 6 10:41:32 rx6600-1 FontServer[1755]: Warning: Bad font path element: "/usr/lib/X11/fonts/hp_japanese/100dpi/" Nov 6 10:41:32 rx6600-1 FontServer[1755]: Warning: Bad font path element: "/usr/lib/X11/fonts/hp_japanese/75dpi/" Nov 6 10:41:32 rx6600-1 FontServer[1755]: Warning: Bad font path element: "/usr/lib/X11/fonts/hp_korean/75dpi/" Nov 6 10:41:32 rx6600-1 FontServer[1755]: Warning: Cannot initialize font path element: "/usr/lib/X11/fonts/hp_chinese_t/75dpi/" Nov 6 10:41:32 rx6600-1 FontServer[1755]: Warning: Bad font path element: "/usr/lib/X11/fonts/ttfjpn.st" Nov 6 10:41:32 rx6600-1 FontServer[1755]: Warning: Bad font path element: "/usr/lib/X11/fonts/ifojpn.st" Nov 6 10:41:34 rx6600-1 pwgrd: Started at Thu Nov 6 10:41:34 2014, pid = 1798 Nov 6 10:41:34 rx6600-1 diagmond[1833]: started Nov 6 10:41:34 rx6600-1 /usr/sbin/envd[1837]: VXPBFt6/, 2"6A3vEdVCND< ~ Nov 6 10:41:50 rx6600-1 /usr/sbin/nfsd[2180]: Setting STREAMS-HEAD high water value to 131072. Nov 6 10:41:50 rx6600-1 /usr/sbin/nfsd[2185]: nfsd do_one mpctl succeeded: ncpus = 8. Nov 6 10:41:50 rx6600-1 /usr/sbin/nfsd[2185]: nfsd do_one pmap 2 Nov 6 10:41:50 rx6600-1 /usr/sbin/nfsd[2185]: nfsd do_one pmap 3 Nov 6 10:41:50 rx6600-1 /usr/sbin/nfsd[2190]: nfsd do_one bind 0 Nov 6 10:41:50 rx6600-1 /usr/sbin/nfsd[2191]: nfsd do_one bind 1 Nov 6 10:41:50 rx6600-1 /usr/sbin/nfsd[2192]: nfsd do_one bind 2 Nov 6 10:41:50 rx6600-1 /usr/sbin/nfsd[2193]: nfsd do_one bind 3 Nov 6 10:41:50 rx6600-1 /usr/sbin/nfsd[2194]: nfsd do_one bind 4 Nov 6 10:41:50 rx6600-1 /usr/sbin/nfsd[2195]: nfsd do_one bind 5 Nov 6 10:41:50 rx6600-1 /usr/sbin/nfsd[2185]: nfsd do_one bind 7 Nov 6 10:41:50 rx6600-1 /usr/sbin/nfsd[2195]: Return from t_optmgmt(XTI_DISTRIBUTE) 0 Nov 6 10:41:50 rx6600-1 /usr/sbin/nfsd[2195]: nfsd 5 1 sock 4 Nov 6 10:41:50 rx6600-1 /usr/sbin/nfsd[2197]: nfsd 5 0 sock 4 Nov 6 10:41:50 rx6600-1 /usr/sbin/nfsd[2193]: Return from t_optmgmt(XTI_DISTRIBUTE) 0 Nov 6 10:41:50 rx6600-1 /usr/sbin/nfsd[2192]: Return from t_optmgmt(XTI_DISTRIBUTE) 0 Nov 6 10:41:50 rx6600-1 /usr/sbin/nfsd[2193]: nfsd 3 1 sock 4 Nov 6 10:41:50 rx6600-1 /usr/sbin/nfsd[2192]: nfsd 2 1 sock 4 Nov 6 10:41:50 rx6600-1 /usr/sbin/nfsd[2200]: nfsd 2 0 sock 4 Nov 6 10:41:50 rx6600-1 /usr/sbin/nfsd[2191]: Return from t_optmgmt(XTI_DISTRIBUTE) 0 Nov 6 10:41:50 rx6600-1 /usr/sbin/nfsd[2194]: Return from t_optmgmt(XTI_DISTRIBUTE) 0 Nov 6 10:41:50 rx6600-1 /usr/sbin/nfsd[2191]: nfsd 1 1 sock 4 Nov 6 10:41:50 rx6600-1 /usr/sbin/nfsd[2201]: nfsd 1 0 sock 4 Nov 6 10:41:50 rx6600-1 /usr/sbin/nfsd[2199]: nfsd 3 0 sock 4 Nov 6 10:41:50 rx6600-1 /usr/sbin/nfsd[2194]: nfsd 4 1 sock 4 Nov 6 10:41:50 rx6600-1 /usr/sbin/nfsd[2202]: nfsd 4 0 sock 4 Nov 6 10:41:50 rx6600-1 /usr/sbin/nfsd[2185]: Return from t_optmgmt(XTI_DISTRIBUTE) 0 Nov 6 10:41:50 rx6600-1 /usr/sbin/nfsd[2185]: nfsd 7 1 sock 4 Nov 6 10:41:50 rx6600-1 /usr/sbin/nfsd[2219]: nfsd 7 0 sock 4 Nov 6 10:41:50 rx6600-1 /usr/sbin/nfsd[2196]: nfsd do_one bind 6 Nov 6 10:41:50 rx6600-1 /usr/sbin/nfsd[2190]: Return from t_optmgmt(XTI_DISTRIBUTE) 0 Nov 6 10:41:50 rx6600-1 /usr/sbin/nfsd[2196]: Return from t_optmgmt(XTI_DISTRIBUTE) 0 Nov 6 10:41:50 rx6600-1 /usr/sbin/nfsd[2190]: nfsd 0 1 sock 4 Nov 6 10:41:50 rx6600-1 /usr/sbin/nfsd[2220]: nfsd 0 0 sock 4 Nov 6 10:41:50 rx6600-1 /usr/sbin/nfsd[2196]: nfsd 6 1 sock 4 Nov 6 10:41:50 rx6600-1 /usr/sbin/nfsd[2221]: nfsd 6 0 sock 4 Nov 6 10:41:53 rx6600-1 krsd[2300]: Delay time is 300 seconds Nov 6 10:41:53 rx6600-1 sfd[2301]: daemon already running. Nov 6 10:41:54 rx6600-1 sfd[2314]: starting the daemon. Nov 6 10:41:54 rx6600-1 emcp_mond: PP daemon: Info: New event pair [0] (2,4,60) Nov 6 10:41:54 rx6600-1 emcp_mond: PP daemon: Info: New event pair [1] (20,40,300) Nov 6 10:41:54 rx6600-1 emcp_mond: PP daemon: Info: SetLogMask:: EventLogMask set to 0x66 Nov 6 10:41:54 rx6600-1 emcp_mond: PP daemon: Info: Using hostname localhost community public debug 0 Nov 6 10:41:54 rx6600-1 emcp_mond: PP daemon: Info: Daemon created successfully. Starting it now Nov 6 10:41:54 rx6600-1 emcp_mond: PP daemon: Info: SNMP trap processing disabled. Nov 6 10:41:54 rx6600-1 emcp_mond: PP daemon: Info: PP Remote Management disabled. Nov 6 10:45:17 rx6600-1 vmunix: emcp:Mpx:Info: PowerPath Auto Host Registration on VNX-FCN00125000137 is unavailable: incompatible initiator information received from the array Nov 6 10:45:42 rx6600-1 /usr/sbin/envd[1837]: ***** 9} HH AY =g >/ 8f ***** Nov 6 10:45:42 rx6600-1 /usr/sbin/envd[1837]: NB6H3,9}U}3#9$WwAY=gV5, P^U}9}HHLu< ~!# Nov 6 10:45:42 rx6600-1 EMS [2970]: ------ EMS Event Notification ------ Value: "MAJORWARNING (3)" for Resource: "/system/events/ia64_corehw/core_hw" (Threshold: >= " 3") Execute the following command to obtain event details: /opt/resmon/bin/resdata -R 194641922 -r /system/events/ia64_corehw/core_hw -n 194641921 -a Nov 6 10:49:14 rx6600-1 EMS [2928]: ------ EMS Event Notification ------ Value: "CRITICAL (5)" for Resource: "/system/events/ipmi_fpl/ipmi_fpl" (Threshold: >= " 3") Execute the following command to obtain event details: /opt/resmon/bin/resdata -R 191889410 -r /system/events/ipmi_fpl/ipmi_fpl -n 191889409 -a Nov 6 18:48:12 rx6600-1 EMS [2970]: ------ EMS Event Notification ------ Value: "CRITICAL (5)" for Resource: "/system/events/ia64_corehw/core_hw" (Threshold: >= " 3") Execute the following command to obtain event details: /opt/resmon/bin/resdata -R 194641922 -r /system/events/ia64_corehw/core_hw -n 194641922 -a Nov 6 19:00:00 rx6600-1 su: + tty?? root-oracle Nov 7 08:00:00 rx6600-1 su: + tty?? root-root
从如下信息看到服务器已经出问题了，且信息已经指出可以执行
/opt/resmon/bin/resdata -R 194641922 -r /system/events/ia64_corehw/core_hw -n 194641921 -a 命令来查看详细信息
Nov 6 10:41:54 rx6600-1 emcp_mond: PP daemon: Info: SNMP trap processing disabled. Nov 6 10:41:54 rx6600-1 emcp_mond: PP daemon: Info: PP Remote Management disabled. Nov 6 10:45:17 rx6600-1 vmunix: emcp:Mpx:Info: PowerPath Auto Host Registration on VNX-FCN00125000137 is unavailable: incompatible initiator information received from the array Nov 6 10:45:42 rx6600-1 /usr/sbin/envd[1837]: ***** 9} HH AY =g >/ 8f ***** Nov 6 10:45:42 rx6600-1 /usr/sbin/envd[1837]: NB6H3,9}U}3#9$WwAY=gV5, P^U}9}HHLu< ~!# Nov 6 10:45:42 rx6600-1 EMS [2970]: ------ EMS Event Notification ------ Value: "MAJORWARNING (3)" for Resource: "/system/events/ia64_corehw/core_hw" (Threshold: >= " 3") Execute the following command to obtain event details: /opt/resmon/bin/resdata -R 194641922 -r /system/events/ia64_corehw/core_hw -n 194641921 -a
执行/opt/resmon/bin/resdata -R 194641922 -r /system/events/ia64_corehw/core_hw -n 194641921 -a 命令来查看详细信息
rx6600-1:[/]#/opt/resmon/bin/resdata -R 194641922 -r /system/events/ia64_corehw/core_hw -n 194641921 -a
ARCHIVED MONITOR DATA:
Event Time..........: Thu Nov 6 10:45:42 2014
Severity............: MAJORWARNING
Monitor.............: ia64_corehw
Event #.............: 101011
System..............: rx6600-1
Summary:
System temperature is out of normal range.
Description of Error:
The system temperature is not within normal operating range. It is higher
than required operating range.
这个错误描述是说系统的温度超出了正常范围，下面信息说明了可能的原因
Probable Cause / Recommended Action:
Something may be blocking the cooling intakes of the fans. Check for
obstruction.
One or more fans may be operating at lower speed than normal. Check the
fan performance.
Check for problems with the room air conditioning.
If the problem is not fixed, the operating temperature may become
non-recoverable, in which case there are chances that the hardware may be
damaged. At that temperature level, on Integrity servers, the firmware
will shutdown the system automatically. However on HP 9000 servers, the
action specified in the envd config file will be taken - which may be to
shutdown the system automatically.
For information on the sensor that generated this event, refer to FRU ID
in Event Details section.
上面的信息是说，可能需要清理一下风机，或者风机性能出现问题，或者检查空调情况，如果不是这些原因造成那么可能是硬件出现问题了。下面的论断事件的数据:
Additional Event Data:
System IP Address...: 10.138.129.5
Event Id............: 0x545ae0d600000000
Monitor Version.....: B.01.00
Event Class.........: System
Client Configuration File...........:
/var/stm/config/tools/monitor/default_ia64_corehw.clcfg
Client Configuration File Version...: A.01.00
Qualification criteria met.
Number of events..: 1
Associated OS error log entry id(s):
None
Additional System Data:
System Model Number.............: ia64 hp server rx6600
EMS Version.....................: A.04.20
STM Version.....................: C.58.00
System Serial Number............: SGH48045VY
Latest information on this event:
http://docs.hp.com/hpux/content/hardware/ems/ia64_corehw.htm#101011
v-v-v-v-v-v-v-v-v-v-v-v-v D E T A I L S v-v-v-v-v-v-v-v-v-v-v-v-v
Event Details :
Event Date .............: Thu Nov 6 10:44:08 2014
Sensor Number ..........: 0xdb
Sensor Type ............: Temperature
Sensor Class ...........: Threshold based
Sensor Reading/Offset...: 0x07 (Offset)
Event Type.............: Assertion
Entity ID ..............: 3
Generic Message.........:
Temperature : Upper non-critical - going high
Entity FRU Id Info......:
processor (Sensor ID: Processor 2)
从上面的Event Details信息可以看到，传感器类型是温度方面的问题，传感器类别是基于阈值，事件类型是断言，是说2号cpu的温度已经超过了阈值.经过检查不是机房空调，通风口堵塞问题，需要联系小机厂商来进行一步检查是什么原因造成cpu温度超过阈值，平时cpu使用率只有10%。