DMDSC动态扩展节点
DMDSC集群支持动态扩展节点，每次扩展可以在原有基础上增加一个节点。动态扩展节点要求当前DMDSC集群的所有节点都为OK状态，所有dmserver实例都处于OPEN状态，且可以正常访问。
注意:扩展节点过程中，不应该有修改数据库状态或模式的操作
一. 环境说明
新增节点环境为： 操作系统：RedHat Linux 64位。
网络配置：eth0网卡为10.10.13.0外网网段，该机器为10.10.13.203；eth1为11.11.11.0内网网段，该机器为11.11.11.5内网网段用于MAL通讯。
DM各种工具位于目录：/dm8/bin。
配置文件位于目录：/dm8/data。
二、操作系统配置
1、关闭防火墙和SELINUX
[root@dm8rac3 ~]# systemctl stop firewalld
[root@dm8rac3 ~]# systemctl disable firewalld
Removed symlink /etc/systemd/system/multi-user.target.wants/firewalld.service.
Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.
[root@dm8rac3 ~]# systemctl status firewalld
firewalld.service - firewalld - dynamic firewall daemon
Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
Active: inactive (dead)
Docs: man:firewalld(1)
Dec 09 17:28:02 dm8rac3 systemd[1]: Starting firewalld - dynamic firewall daemon...
Dec 09 17:28:04 dm8rac3 systemd[1]: Started firewalld - dynamic firewall daemon.
Dec 10 08:29:20 dm8rac3 systemd[1]: Stopping firewalld - dynamic firewall daemon...
Dec 10 08:29:21 dm8rac3 systemd[1]: Stopped firewalld - dynamic firewall daemon.
[root@dm8rac3 ~]# setenforce 0
[root@dm8rac3 ~]# sed -i s:^SELINUX=.*$:SELINUX=disabled:g /etc/selinux/config
[root@dm8rac3 ~]# cat /etc/selinux/config
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
# enforcing - SELinux security policy is enforced.
# permissive - SELinux prints warnings instead of enforcing.
# disabled - No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of three values:
# targeted - Targeted processes are protected,
# minimum - Modification of targeted policy. Only selected processes are protected.
# mls - Multi Level Security protection.
SELINUXTYPE=targeted
2、修改主机名
[root@dm8rac3 ~]# hostnamectl set-hostname dm8rac3 [root@dm8rac3 ~]# sed -i s:^HOSTNAME=.*$:HOSTNAME=dm8rac3:g /etc/sysconfig/network [root@dm8rac3 ~]# echo " > 10.10.13.201 dm8rac1 > 10.10.13.202 dm8rac2 > 10.10.13.203 dm8rac3" >> /etc/hosts [root@dm8rac3 ~]# cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 10.10.13.201 dm8rac1 10.10.13.202 dm8rac2 10.10.13.203 dm8rac3 [root@dm8rac1 ~]# vi /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 10.10.13.201 dm8rac1 10.10.13.202 dm8rac2 10.10.13.203 dm8rac3 [root@dm8rac2 ~]# vi /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 10.10.13.201 dm8rac1 10.10.13.202 dm8rac2 10.10.13.203 dm8rac3
三、 安装达梦软件
3.1检查Linux(Unix)系统信息
[root@dm8rac3 ~]# getconf LONG_BIT 64
查询操作系统release信息
[root@dm8rac3 ~]# cat /etc/redhat-release Red Hat Enterprise Linux Server release 7.8 Beta (Maipo)
3.2创建安装用户
为了减少对操作系统的影响，用户不应该以root系统用户来安装和运行DM。用户可以在安装之前为DM创建一个专用的系统用户。
1. 创建安装用户组dinstall。
[root@dm8rac3 ~]# groupadd dinstall
2. 创建安装用户dmdba。
[root@dm8rac3 ~]# useradd -g dinstall -m -d /home/dmdba -s /bin/bash dmdba
3. 初始化用户密码。
[root@dm8rac3 ~]# passwd dmdba Changing password for user dmdba. New password: Retype new password: passwd: all authentication tokens updated successfully.
3.3 Linux(Unix)下检查操作系统限制
在Linux(Unix)系统中，因为ulimit命令的存在，会对程序使用操作系统资源进行限制。为了使DM能够正常运行，建议用户检查当前安装用户的ulimit参数。
运行ulimit -a进行查询。如下图所示：
[root@dm8rac3 ~]# ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 31152 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 31152 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited
参数使用限制：
1.data seg size
data seg size (kbytes, -d)
建议用户设置为1048576(即1GB)以上或unlimited(无限制)，此参数过小将导致数据库启动失败。
2. file size
file size(blocks, -f)
建议用户设置为unlimited(无限制)，此参数过小将导致数据库安装或初始化失败。
3. open files
open files(-n)
建议用户设置为65536以上或unlimited(无限制)。
4.virtual memory
virtual memory (kbytes, -v)
建议用户设置为1048576(即1GB)以上或unlimited(无限制)，此参数过小将导致数据库启动失败。
如果用户需要为当前安装用户更改ulimit的资源限制，请修改文件/etc/security/limits.conf。
[root@dm8rac3 ~]# vi /etc/security/limits.conf dmdba soft data unlimited dmdba hard data unlimited dmdba soft fsize unlimited dmdba hard fsize unlimited dmdba soft nofile 65536 dmdba hard nofile 65536 [root@dm8rac3 ~]# su - dmdba [dmdba@dm8rac3 ~]$ ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 31152 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 65536 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 4096 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited
3.4.检查系统内存与存储空间
1.检查内存
为了保证DM的正确安装和运行，要尽量保证操作系统至少1GB的可用内存(RAM)。如果可用内存过少，可能导致DM安装或启动失败。用户可以
使用以下命令检查操作内存：
#获取内存总大小
[root@dm8rac3 ~]# grep MemTotal /proc/meminfo
MemTotal: 8009068 kB
#获取交换分区大小
[root@dm8rac3 ~]# grep SwapTotal /proc/meminfo
SwapTotal: 8257532 kB
#获取内存使用详情
[root@dm8rac3 ~]# free -g
total used free shared buff/cache available
Mem: 7 0 6 0 0 6
Swap: 7 0 7
2.检查存储空间
1) DM完全安装需要1GB的存储空间，用户需要提前规划好安装目录，预留足够的存储空间。用户在DM安装前也应该为数据库实例预留足够的存储空间，规划好数据路径和备份路径。用户可使用以下命令检查存储空间:
#查询目录/dm8可用空间
[root@dm8rac3 ~]# mkdir /dm8 [root@dm8rac3 ~]# chown -R dmdba:dinstall /dm8 [root@dm8rac3 ~]# chmod -R 775 /dm8 [root@dm8rac3 ~]# df -h /dm8 Filesystem Size Used Avail Use% Mounted on /dev/mapper/rhel-root 48G 5.6G 43G 12% /
2) DM安装程序在安装时将产生临时文件，临时文件需要1GB的存储空间，临时文件目录默认为/tmp。用户可以使用以下命令检查存储空间。
如下图所示：
[root@dm8rac3 ~]# df -h /tmp Filesystem Size Used Avail Use% Mounted on /dev/mapper/rhel-root 48G 5.6G 43G 12% /
3.5.安装DM
创建目录/soft/dmsoft用来存储挂载iso文件后来显示软件包中的文件
[root@dm8rac3 ~]# cd /soft [root@dm8rac3 soft]# unzip dm8_20211021_x86_rh6_64_ent.zip Archive: dm8_20211021_x86_rh6_64_ent.zip creating: dm8_20211021_x86_rh6_64_ent/ inflating: dm8_20211021_x86_rh6_64_ent/dm8_20211021_x86_rh6_64_ent_8.1.2.84.iso extracting: dm8_20211021_x86_rh6_64_ent/dm8_20211021_x86_rh6_64_ent_8.1.2.84.iso_SHA256.txt extracting: dm8_20211021_x86_rh6_64_ent/verinfo.txt [root@dm8rac3 soft]# ls -lrt total 778320 drwxr-xr-x. 2 root root 132 Nov 12 13:43 dm8_20211021_x86_rh6_64_ent -rw-r--r--. 1 root root 796998047 Nov 29 10:25 dm8_20211021_x86_rh6_64_ent.zip [root@dm8rac3 soft]# mv dm8_20211021_x86_rh6_64_ent dm8 [root@dm8rac3 soft]# mkdir dmsoft [root@dm8rac3 soft]# mount -t iso9660 -o loop dm8/dm8_20211021_x86_rh6_64_ent_8.1.2.84.iso /soft/dmsoft mount: /dev/loop0 is write-protected, mounting read-only [root@dm8rac3 soft]# cd dmsoft [root@dm8rac3 dmsoft]# ls -lrt total 790160 -r-xr-xr-x. 1 root root 2802503 Oct 21 14:04 DM8 Install.pdf -r-xr-xr-x. 1 root root 806320703 Oct 21 14:11 DMInstall.bin
在/soft/dmsoft目录下存在DMInstall.bin文件， DMInstall.bin文件就是DM的安装程序。在运行安装程序前，需要赋予DMInstall.bin文件执行权限。具体命令如下所示：
[root@dm8rac3 dmsoft]# chmod 755 DMInstall.bin chmod: changing permissions of ‘DMInstall.bin’: Read-only file system
在现实中，许多Linux(Unix)操作系统上是没有图形化界面的，为了使DM能够在这些操作系统上顺利安装，DM提供了命令行的安装方式。在终端进入到安装程序所在文件夹，执行以下命令进行命令行安装：
[dmdba@dm8rac3 dmsoft]$ ./DMInstall.bin -i Please select the installer's language (E/e:English C/c:Chinese) [E/e]:e Extract install files......... Welcome to DM DBMS Installer Whether to input the path of Key File? (Y/y:Yes N/n:No) [Y/y]:n Whether to Set The TimeZone? (Y/y:Yes N/n:No) [Y/y]:y TimeZone: [ 1]: GTM-12=West Date Line [ 2]: GTM-11=Samoa [ 3]: GTM-10=Hawaii [ 4]: GTM-09=Alaska [ 5]: GTM-08=Pacific(America and Canada) [ 6]: GTM-07=Arizona [ 7]: GTM-06=Central(America and Canada) [ 8]: GTM-05=East(America and Canada) [ 9]: GTM-04=Atlantic(America and Canada) [10]: GTM-03=Brasilia [11]: GTM-02=Middle Atlantic [12]: GTM-01=Azores [13]: GTM=Greenwich Mean Time [14]: GTM+01=Sarajevo [15]: GTM+02=Cairo [16]: GTM+03=Moscow [17]: GTM+04=AbuDhabi [18]: GTM+05=Islamabad [19]: GTM+06=Dakar [20]: GTM+07=BangKok,Hanoi [21]: GTM+08=China [22]: GTM+09=Seoul [23]: GTM+10=Guam [24]: GTM+11=Solomon [25]: GTM+12=Fiji [26]: GTM+13=Nukualofa [27]: GTM+14=Kiribati Please Select the TimeZone [21]:21 Installation Type: 1 Typical 2 Server 3 Client 4 Custom Please Input the number of the Installation Type [1 Typical]:4 1 Server component 2 Client component 2.1 Manager 2.2 Monitor 2.3 DTS 2.4 Console 2.5 Analyzer 2.6 DISQL 3 DM Drivers 4 Manual component 5 DBMS Service 5.1 Realtime Audit Service 5.2 Job Service 5.3 Instance Monitor Service 5.4 Assistant Plug-In Service Please Input the number of the Installation Type [1 2 3 4 5]:1 2 3 4 5 Require Space: 1242M Please Input the install path [/home/dmdba/dmdbms]:/dm8 Available Space:39G Please Confirm the install path(/dm8)? (Y/y:Yes N/n:No) [Y/y]:y Pre-Installation Summary Installation Location: /dm8 Require Space: 1242M Available Space: 39G Version Information: Expire Date: Installation Type: Custom Confirm to Install? (Y/y:Yes N/n:No):y 2021-12-10 08:52:34 [INFO] Installing DM DBMS... 2021-12-10 08:52:34 [INFO] Installing BASE Module... 2021-12-10 08:52:36 [INFO] Installing SERVER Module... 2021-12-10 08:52:39 [INFO] Installing CLIENT Module... 2021-12-10 08:52:54 [INFO] Installing DRIVERS Module... 2021-12-10 08:52:55 [INFO] Installing MANUAL Module... 2021-12-10 08:52:55 [INFO] Installing SERVICE Module... 2021-12-10 08:53:03 [INFO] Move log file to log directory. 2021-12-10 08:53:04 [INFO] Installed DM DBMS completely. Please execute the commands by root: /dm8/script/root/root_installer.sh End
以root用户来执行上面的脚本
[root@dm8rac3 ~]# /dm8/script/root/root_installer.sh
Move /dm8/bin/dm_svc.conf to /etc
Modify the files' mode of DM Server
[dmdba@dm8rac3 ~]$ vi .bash_profile
# .bash_profile
# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi
# User specific environment and startup programs
PATH=$PATH:$HOME/.local/bin:$HOME/bin
export PATH
export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/dm8/bin"
export DM_HOME="/dm8"
export PATH=$PATH:$LD_LIBRARY_PATH
四、绑定UDEV
编辑/etc/udev/rules.d/99-dm-devices.rules文件
[root@dm8rac3 ~]# vi /etc/udev/rules.d/99-dm-devices.rules ACTION=="add", KERNEL=="sdb", RUN+="/bin/raw /dev/raw/raw1 %N" ACTION=="add", KERNEL=="sdc", RUN+="/bin/raw /dev/raw/raw2 %N" ACTION=="add", KERNEL=="sdd", RUN+="/bin/raw /dev/raw/raw3 %N" ACTION=="add", KERNEL=="sde", RUN+="/bin/raw /dev/raw/raw4 %N" ACTION=="add", KERNEL=="sdf", RUN+="/bin/raw /dev/raw/raw5 %N" ACTION=="add", KERNEL=="sdg", RUN+="/bin/raw /dev/raw/raw6 %N" ACTION=="add", KERNEL=="raw[1-6]", OWNER="dmdba", GROUP="dinstall", MODE="660"
如果不能正常显示就重启操作系统
[root@dm8rac3 ~]# ls -lrt /dev/raw/ total 0 crw-rw---- 1 root disk 162, 0 Dec 10 09:01 rawctl crw-rw---- 1 dmdba dinstall 162, 6 Dec 10 09:01 raw6 crw-rw---- 1 dmdba dinstall 162, 5 Dec 10 09:01 raw5 crw-rw---- 1 dmdba dinstall 162, 4 Dec 10 09:01 raw4 crw-rw---- 1 dmdba dinstall 162, 3 Dec 10 09:01 raw3 crw-rw---- 1 dmdba dinstall 162, 1 Dec 10 09:01 raw1 crw-rw---- 1 dmdba dinstall 162, 2 Dec 10 09:01 raw2
可以通过blockdev –getsize64 /dev/raw/raw1命令查看裸设备大小
[root@dm8rac3 ~]# blockdev --getsize64 /dev/raw/raw1 2147483648 [root@dm8rac3 ~]# blockdev --getsize64 /dev/raw/raw2 2147483648 [root@dm8rac3 ~]# blockdev --getsize64 /dev/raw/raw3 10737418240 [root@dm8rac3 ~]# blockdev --getsize64 /dev/raw/raw4 10737418240 [root@dm8rac3 ~]# blockdev --getsize64 /dev/raw/raw5 10737418240 [root@dm8rac3 ~]# blockdev --getsize64 /dev/raw/raw6 10737418240
四. 操作流程
1. 在10.10.13.201机器上使用DMASMCMD工具export出备份dmdcr_cfg_bak.ini
[dmdba@dm8rac1 ~]$ dmasmcmd DMASMCMD V8 ASM>export dcrdisk '/dev/raw/raw1' to '/dm8/data/dmdcr_cfg_bak.ini' ASMCMD export DCRDISK success. Used time: 59.153(ms).
2. 为新增节点准备日志文件
1) 使用DIsql登录任意一个节点执行添加日志文件操作：
至少两个日志文件，路径必须是ASM文件格式，大小可以参考其他两个活动节点。
SQL> alter database add node logfile '+DMLOG/log/rac2_log01.log' size 256,'+DMLOG/log/rac2_log02.log' size 256; executed successfully used time: 00:00:01.419. Execute id is 208.
2)使用dmctlcvt工具将dm.ctl转换为文本文件dmctl.txt，查看dmctl.txt，新增节点的日志文件信息已经添加进dm.ctl。
[dmdba@dm8rac1 data]$ dmctlcvt TYPE=1 SRC=+DMDATA/data/rac/dm.ctl DEST=/dm8/data/dmctl.txt DCR_INI=/dm8/data/dmdcr.ini DMCTLCVT V8 convert ctl to txt success! [dmdba@dm8rac1 data]$ ls -lrt total 28 -rw-r--r-- 1 dmdba dinstall 956 Dec 8 09:24 dmdcr_cfg.ini -rw-r--r-- 1 dmdba dinstall 143 Dec 8 09:29 dmasvrmal.ini -rw-r--r-- 1 dmdba dinstall 328 Dec 8 09:37 dmdcr.ini -rw-r--r-- 1 dmdba dinstall 864 Dec 8 10:13 dminit.ini drwxr-xr-x 2 dmdba dinstall 55 Dec 8 10:17 rac1_config drwxr-xr-x 3 dmdba dinstall 100 Dec 8 10:36 rac0_config -rw-r--r-- 1 dmdba dinstall 1647 Dec 10 09:06 dmdcr_cfg_bak.ini -rw-r--r-- 1 dmdba dinstall 8158 Dec 10 09:23 dmctl.txt [dmdba@dm8rac1 data]$ more dmctl.txt # file path fil_path=+DMLOG/log/rac2_log01.log # mirror path mirror_path= # file id fil_id=0 # whether the file is auto extend autoextend=1 # file create time fil_create_time=DATETIME '2021-12-10 9:20:50' # file modify time fil_modify_time=DATETIME '2021-12-10 9:20:50' # the max size of file fil_max_size=0 # next size of file fil_next_size=0 # file path fil_path=+DMLOG/log/rac2_log02.log # mirror path mirror_path= # file id fil_id=1 # whether the file is auto extend autoextend=1 # file create time fil_create_time=DATETIME '2021-12-10 9:20:50' # file modify time fil_modify_time=DATETIME '2021-12-10 9:20:50' # the max size of file
3)使用dmasmtool工具登录ASM文件系统，也可以看到新增的节点日志文件
[dmdba@dm8rac1 ~]$ dmasmtool DCR_INI=/dm8/data/dmdcr.ini
DMASMTOOL V8
ASM>ls +DMLOG/log
file : rac0_log01.log
file : rac0_log02.log
file : rac1_log01.log
file : rac1_log02.log
file : rac2_log01.log
file : rac2_log02.log
total count 6.
Used time: 6.598(ms).
3. 为新增节点准备config_path
将10.10.13.201机器/dm8/data/rac0_config目录拷贝到10.10.13.203机器相同目录下，修改名字为/dm8/data/rac2_config。
[dmdba@dm8rac1 data]$ scp -r rac0_config 10.10.13.203:`pwd` The authenticity of host '10.10.13.203 (10.10.13.203)' can't be established. ECDSA key fingerprint is SHA256:6O8c9WEeEYPbL4ncdRR1RsrjxxmfzPq9Tkq4/6uLSP4. ECDSA key fingerprint is MD5:e1:73:3e:8d:79:be:5c:82:0f:c7:58:79:45:ad:df:86. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added '10.10.13.203' (ECDSA) to the list of known hosts. dmdba@10.10.13.203's password: dmmal.ini 100% 200 8.5KB/s 00:00 dminit20211208101727.log 100% 1144 70.0KB/s 00:00 dm.ini 100% 53KB 3.2MB/s 00:00 sqllog.ini 100% 481 19.3KB/s 00:00 [dmdba@dm8rac1 data]$ [dmdba@dm8rac3 data]$ mv rac0_config rac2_config [dmdba@dm8rac3 data]$ ls -lrt total 0 drwxr-xr-x 2 dmdba dinstall 6 Dec 10 09:54 trace drwxr-xr-x 3 dmdba dinstall 100 Dec 10 09:56 rac2_config
修改dsc2_config文件夹下的配置文件：
1) 修改dm.ini
[dmdba@dm8rac3 rac2_config]$ vi dm.ini CONFIG_PATH = /dm8/data/rac2_config #config path INSTANCE_NAME = RAC2 #Instance name
2) 如果打开了归档参数，修改dmarch.ini
4. 新建dmdcr.ini配置文件，保存到节点10.0.2.103的/dm8/data/目录下面注意设置dmdcr_seqo为2，修改dm.ini路径。
[dmdba@dm8rac3 data]$ vi dmdcr.ini DMDCR_PATH = /dev/raw/raw1 DMDCR_MAL_PATH =/dm8/data/dmasvrmal.ini DMDCR_SEQNO = 2 #ASM DMDCR_ASM_RESTART_INTERVAL = 0 DMDCR_ASM_STARTUP_CMD = /dm8/bin/dmasmsvr dcr_ini=/dm8/data/dmdcr.ini #DB DMDCR_DB_RESTART_INTERVAL = 0 DMDCR_DB_STARTUP_CMD = /dm8/bin/dmserver path=/dm8/data/rac2_config/dm.ini dcr_ini=/dm8/data/dmdcr.ini
5. 修改当前环境的MAL配置文件
直接修改当前环境的dmasvrmal.ini文件，添加新增节点信息，使用DMASM的所有节点都要配置，内容完全一样，并且将新增信息后的dmasvrmal.ini文件拷贝到节点10.10.13.203的/dm8/data目录下。
[dmdba@dm8rac1 data]$ vi dmasvrmal.ini [MAL_INST1] MAL_INST_NAME = ASM0 MAL_HOST = 11.11.11.1 MAL_PORT = 7236 [MAL_INST2] MAL_INST_NAME = ASM1 MAL_HOST = 11.11.11.2 MAL_PORT = 7236 [MAL_INST3] MAL_INST_NAME = ASM2 MAL_HOST = 11.11.11.5 MAL_PORT = 7236 [dmdba@dm8rac2 data]$ vi dmasvrmal.ini [MAL_INST1] MAL_INST_NAME = ASM0 MAL_HOST = 11.11.11.1 MAL_PORT = 7236 [MAL_INST2] MAL_INST_NAME = ASM1 MAL_HOST = 11.11.11.2 MAL_PORT = 7236 [MAL_INST3] MAL_INST_NAME = ASM2 MAL_HOST = 11.11.11.5 MAL_PORT = 7236 [dmdba@dm8rac3 data]$ vi dmasvrmal.ini [MAL_INST1] MAL_INST_NAME = ASM0 MAL_HOST = 11.11.11.1 MAL_PORT = 7236 [MAL_INST2] MAL_INST_NAME = ASM1 MAL_HOST = 11.11.11.2 MAL_PORT = 7236 [MAL_INST3] MAL_INST_NAME = ASM2 MAL_HOST = 11.11.11.5 MAL_PORT = 7236
直接修改dmserver三个实例的dmmal.ini，添加新增节点信息，所有节点都要配置相同内容，保存到各自的rac_config目录下
[dmdba@dm8rac1 rac0_config]$ vi dmmal.ini
[mal_inst0]
mal_inst_name = RAC0
mal_host = 11.11.11.1
mal_port = 9340
[mal_inst1]
mal_inst_name = RAC1
mal_host = 11.11.11.2
mal_port = 9340
[mal_inst2]
mal_inst_name = RAC2
mal_host = 11.11.11.5
mal_port = 9340
[dmdba@dm8rac2 rac1_config]$ vi dmmal.ini
[mal_inst0]
mal_inst_name = RAC0
mal_host = 11.11.11.1
mal_port = 9340
[mal_inst1]
mal_inst_name = RAC1
mal_host = 11.11.11.2
mal_port = 9340
[mal_inst2]
mal_inst_name = RAC2
mal_host = 11.11.11.5
mal_port = 9340
[dmdba@dm8rac3 rac2_config]$ vi dmmal.ini
[mal_inst0]
mal_inst_name = RAC0
mal_host = 11.11.11.1
mal_port = 9340
[mal_inst1]
mal_inst_name = RAC1
mal_host = 11.11.11.2
mal_port = 9340
[mal_inst2]
mal_inst_name = RAC2
mal_host = 11.11.11.5
mal_port = 9340
后续实例会重新读MAL配置文件，更新内存信息。
6. 修改dmdcr_cfg_bak.ini，添加新增节点信息，CSS/ASMSVR/DB都要配置
所有组信息修改：
DCR_GRP_N_EP = 3
DCR_GRP_EP_ARR = {0,1,2}
每个组增加一个节点信息，注意DCR_EP_SHM_KEY、端口号不能冲突；各组信息要放在各自的后面，即[GRP_CSS]中CSS2放在CSS1后面，[GRP_ASM]中ASM2放在ASM1后面，RAC2放在RAC1后面。
[dmdba@dm8rac1 data]$ cat dmdcr_cfg_bak.ini
# the file is auto-created by system, self edit is invalid!
#DCR HDR
DCR_N_GRP = 3
DCR_VTD_PATH = /dev/raw/raw2
DCR_OGUID = 63635
[GRP]
DCR_GRP_TYPE = CSS
DCR_GRP_NAME = GRP_CSS
DCR_GRP_N_EP = 3
DCR_GRP_EP_ARR = {0,1,2}
DCR_GRP_N_ERR_EP = 0
DCR_GRP_ERR_EP_ARR = {}
DCR_GRP_DSKCHK_CNT = 60
[GRP]
DCR_GRP_TYPE = ASM
DCR_GRP_NAME = GRP_ASM
DCR_GRP_N_EP = 3
DCR_GRP_EP_ARR = {0,1,2}
DCR_GRP_N_ERR_EP = 0
DCR_GRP_ERR_EP_ARR = {}
DCR_GRP_DSKCHK_CNT = 60
[GRP]
DCR_GRP_TYPE = DB
DCR_GRP_NAME = GRP_RAC
DCR_GRP_N_EP = 3
DCR_GRP_EP_ARR = {0,1,2}
DCR_GRP_N_ERR_EP = 0
DCR_GRP_ERR_EP_ARR = {}
DCR_GRP_DSKCHK_CNT = 60
[GRP_CSS]
DCR_EP_NAME = CSS0
DCR_EP_HOST = 11.11.11.1
DCR_EP_PORT = 9541
[GRP_CSS]
DCR_EP_NAME = CSS1
DCR_EP_HOST = 11.11.11.2
DCR_EP_PORT = 9541
[GRP_CSS]
DCR_EP_NAME = CSS2
DCR_EP_HOST = 11.11.11.5
DCR_EP_PORT = 9541
[GRP_ASM]
DCR_EP_NAME = ASM0
DCR_EP_SHM_KEY = 93360
DCR_EP_SHM_SIZE = 10
DCR_EP_HOST = 10.10.13.201
DCR_EP_PORT = 9641
DCR_EP_ASM_LOAD_PATH = /dev/raw
[GRP_ASM]
DCR_EP_NAME = ASM1
DCR_EP_SHM_KEY = 93361
DCR_EP_SHM_SIZE = 10
DCR_EP_HOST = 10.10.13.202
DCR_EP_PORT = 9641
DCR_EP_ASM_LOAD_PATH = /dev/raw
[GRP_ASM]
DCR_EP_NAME = ASM2
DCR_EP_SHM_KEY = 93362
DCR_EP_SHM_SIZE = 10
DCR_EP_HOST = 10.10.13.203
DCR_EP_PORT = 9641
DCR_EP_ASM_LOAD_PATH = /dev/raw
[GRP_RAC]
DCR_EP_NAME = RAC0
DCR_EP_SEQNO = 0
DCR_EP_PORT = 5236
DCR_CHECK_PORT = 9741
[GRP_RAC]
DCR_EP_NAME = RAC1
DCR_EP_SEQNO = 1
DCR_EP_PORT = 5236
DCR_CHECK_PORT = 9741
[GRP_RAC]
DCR_EP_NAME = RAC2
DCR_EP_SEQNO = 2
DCR_EP_PORT = 5236
DCR_CHECK_PORT = 9741
7. 使用DMASMCMD工具将新增节点信息写回磁盘，新增节点作为error节点
[dmdba@dm8rac1 ~]$ dmasmcmd DMASMCMD V8 ASM>extend dcrdisk '/dev/raw/raw1' from '/dm8/data/dmdcr_cfg_bak.ini' ASMCMD extend node for dcr disk success. ASMCMD extend node for vote disk success. Used time: 00:00:02.362.
8. 在dmcssm控制台执行扩展节点命令
先配置dmcssm.ini文件
[dmdba@dm8rac1 data]$ vi dmcssm.ini CSSM_OGUID = 63635 #配置所有CSS的连接信息， #和dmdcr_cfg.ini中CSS配置项的DCR_EP_HOST和DCR_EP_PORT保持一致 CSSM_CSS_IP = 11.11.11.1:9541 CSSM_CSS_IP = 11.11.11.2:9541 CSSM_CSS_IP = 11.11.11.5:9541 CSSM_LOG_PATH =/dm8/dmcssm/log #监视器日志文件存放路径 CSSM_LOG_FILE_SIZE = 32 #每个日志文件最大32M CSSM_LOG_SPACE_LIMIT = 0 #不限定日志文件总占用空间
启动dmcssm监视器
[dmdba@dm8rac1 ~]$ dmcssm INI_PATH=/dm8/data/dmcssm.ini
[monitor] 2021-12-10 10:46:32: CSS MONITOR V8
[monitor] 2021-12-10 10:46:32: CSS MONITOR SYSTEM IS READY.
[monitor] 2021-12-10 10:46:32: Wait CSS Control Node choosed...
[monitor] 2021-12-10 10:46:33: Wait CSS Control Node choosed succeed.
show
monitor current time:2021-12-10 10:46:57, n_group:3
=================== group[name = GRP_CSS, seq = 0, type = CSS, Control Node = 0] ========================================
[CSS0] auto check = TRUE, global info:
[ASM0] auto restart = FALSE
[RAC0] auto restart = FALSE
[CSS1] auto check = TRUE, global info:
[ASM1] auto restart = FALSE
[RAC1] auto restart = FALSE
ep: css_time inst_name seqno port mode inst_status vtd_status is_ok active guid ts
2021-12-10 10:46:57 CSS0 0 9541 Control Node OPEN WORKING OK TRUE 738576474 738751954
2021-12-10 10:46:57 CSS1 1 9541 Normal Node OPEN WORKING OK TRUE 738606319 738781682
=================== group[name = GRP_ASM, seq = 1, type = ASM, Control Node = 0] ========================================
n_ok_ep = 2
ok_ep_arr(index, seqno):
(0, 0)
(1, 1)
sta = OPEN, sub_sta = STARTUP
break ep = NULL
recover ep = NULL
crash process over flag is TRUE
ep: css_time inst_name seqno port mode inst_status vtd_status is_ok active guid ts
2021-12-10 10:46:57 ASM0 0 9641 Control Node OPEN WORKING OK TRUE 738588988 738764427
2021-12-10 10:46:57 ASM1 1 9641 Normal Node OPEN WORKING OK TRUE 738624010 738799326
=================== group[name = GRP_RAC, seq = 2, type = DB, Control Node = 0] ========================================
n_ok_ep = 2
ok_ep_arr(index, seqno):
(0, 0)
(1, 1)
sta = OPEN, sub_sta = STARTUP
break ep = NULL
recover ep = NULL
crash process over flag is TRUE
ep: css_time inst_name seqno port mode inst_status vtd_status is_ok active guid ts
2021-12-10 10:46:57 RAC0 0 5236 Control Node OPEN WORKING OK TRUE 2383210041 2383382787
2021-12-10 10:46:57 RAC1 1 5236 Normal Node OPEN WORKING OK TRUE 2383231629 2383404339
==================================================================================================================
extend node
[monitor] 2021-12-10 10:47:13: Extend node
[monitor] 2021-12-10 10:47:15: Notify current active CSS to do clear
[monitor] 2021-12-10 10:47:16: Clean request of CSS(0) success
[monitor] 2021-12-10 10:47:17: Clean request of CSS(1) success
[monitor] 2021-12-10 10:47:17: Command EXTENT NODE execute success
show
monitor current time:2021-12-10 10:47:22, n_group:3
=================== group[name = GRP_CSS, seq = 0, type = CSS, Control Node = 0] ========================================
[CSS0] auto check = TRUE, global info:
[ASM0] auto restart = FALSE
[RAC0] auto restart = FALSE
[CSS1] auto check = TRUE, global info:
[ASM1] auto restart = FALSE
[RAC1] auto restart = FALSE
[CSS2] auto check = FALSE, global info:
Connect to [CSS2] failed, please check the network or the CSSM_CSS_IP config in [/dm8/data/dmcssm.ini] .
ep: css_time inst_name seqno port mode inst_status vtd_status is_ok active guid ts
2021-12-10 10:47:22 CSS0 0 9541 Control Node OPEN WORKING OK TRUE 738576474 738751979
2021-12-10 10:47:22 CSS1 1 9541 Normal Node OPEN WORKING OK TRUE 738606319 738781707
2021-12-10 10:47:22 CSS2 2 9541 Normal Node SHUTDOWN UNKNOWN OK FALSE 0 0
=================== group[name = GRP_ASM, seq = 1, type = ASM, Control Node = 0] ========================================
n_ok_ep = 2
ok_ep_arr(index, seqno):
(0, 0)
(1, 1)
sta = OPEN, sub_sta = STARTUP
break ep = NULL
recover ep = NULL
crash process over flag is TRUE
ep: css_time inst_name seqno port mode inst_status vtd_status is_ok active guid ts
2021-12-10 10:47:22 ASM0 0 9641 Control Node OPEN WORKING OK TRUE 738588988 738764452
2021-12-10 10:47:22 ASM1 1 9641 Normal Node OPEN WORKING OK TRUE 738624010 738799351
2021-12-10 10:47:22 ASM2 2 9641 Normal Node SHUTDOWN UNKNOWN ERROR FALSE 0 0
=================== group[name = GRP_RAC, seq = 2, type = DB, Control Node = 0] ========================================
n_ok_ep = 2
ok_ep_arr(index, seqno):
(0, 0)
(1, 1)
sta = OPEN, sub_sta = STARTUP
break ep = NULL
recover ep = NULL
crash process over flag is TRUE
ep: css_time inst_name seqno port mode inst_status vtd_status is_ok active guid ts
2021-12-10 10:47:22 RAC0 0 5236 Control Node OPEN WORKING OK TRUE 2383210041 2383382812
2021-12-10 10:47:22 RAC1 1 5236 Normal Node OPEN WORKING OK TRUE 2383231629 2383404364
2021-12-10 10:47:22 RAC2 2 5236 Normal Node SHUTDOWN UNKNOWN ERROR FALSE 0 0
==================================================================================================================
程序会通知所有实例(CSS/ASMSVR/dmserver)更新信息，在CSS控制台执行SHOW命令，能看到新增节点信息，ASMSVR/dmserver是error节点，程序会通知ASMSVR/dmserver更新MAL信息。
注意:如果由于配置文件错误，动态扩展节点失败，只能停掉所有实例，重新init dcr磁盘，不影响dmserver数据
9. 启动新的DMCSS、DMASM服务程序
在10.10.13.203节点启动dmcss、dmasmsvr程序。
手动启动新的dmcss，dcr_ini指向新的dmdcr.ini文件：
[/opt/dmdbms/bin]# ./dmcss DCR_INI=/home/data/dmdcr.ini
手动启动新的dmasmsvr，dcr_ini指向新的dmdcr.ini文件，asmsvr启动故障重加入流程：
[/opt/dmdbms/bin]# ./dmasmsvr DCR_INI=/home/data/dmdcr.ini
如果DMCSS配置有自动拉起dmasmsvr的功能，可以等待DMCSS自动拉起dmasmsvr程序，不需要手动启动。
我这里将服务注册成服务
[root@dm8rac3 system]# /dm8/script/root/dm_service_installer.sh -t dmcss -dcr_ini /dm8/data/dmdcr.ini -p rac3
Created symlink from /etc/systemd/system/multi-user.target.wants/DmCSSServicerac3.service to /usr/lib/systemd/system/DmCSSServicerac3.service.
Finished to create the service (DmCSSServicerac3)
[root@dm8rac3 system]# /dm8/script/root/dm_service_installer.sh -t dmasmsvr -dcr_ini /dm8/data/dmdcr.ini -p rac3 -y DmCSSServicerac3
Created symlink from /etc/systemd/system/multi-user.target.wants/DmASMSvrServicerac3.service to /usr/lib/systemd/system/DmASMSvrServicerac3.service.
Finished to create the service (DmASMSvrServicerac3)
[root@dm8rac3 system]#
[root@dm8rac3 system]# service DmCSSServicerac3 start
Redirecting to /bin/systemctl start DmCSSServicerac3.service
[root@dm8rac3 system]# service DmCSSServicerac3 status
Redirecting to /bin/systemctl status DmCSSServicerac3.service
DmCSSServicerac3.service - DM Cluster Synchronization Services Service(DmCSSServicerac3).
Loaded: loaded (/usr/lib/systemd/system/DmCSSServicerac3.service; enabled; vendor preset: disabled)
Active: active (running) since Fri 2021-12-10 10:59:41 CST; 19s ago
Process: 4340 ExecStart=/dm8/bin/DmCSSServicerac3 start (code=exited, status=0/SUCCESS)
Main PID: 4368 (dmcss)
Tasks: 12
CGroup: /system.slice/DmCSSServicerac3.service
└─4368 /dm8/bin/dmcss dcr_ini=/dm8/data/dmdcr.ini
Dec 10 10:59:26 dm8rac3 systemd[1]: Starting DM Cluster Synchronization Services Service(DmCSSServicerac3)....
Dec 10 10:59:41 dm8rac3 DmCSSServicerac3[4340]: [38B blob data]
Dec 10 10:59:41 dm8rac3 systemd[1]: Started DM Cluster Synchronization Services Service(DmCSSServicerac3)..
[root@dm8rac3 system]# service DmASMSvrServicerac3 start
Redirecting to /bin/systemctl start DmASMSvrServicerac3.service
[root@dm8rac3 system]# service DmASMSvrServicerac3 status
Redirecting to /bin/systemctl status DmASMSvrServicerac3.service
DmASMSvrServicerac3.service - DM Auto Storage Manager Service(DmASMSvrServicerac3).
Loaded: loaded (/usr/lib/systemd/system/DmASMSvrServicerac3.service; enabled; vendor preset: disabled)
Active: active (running) since Fri 2021-12-10 11:00:23 CST; 7s ago
Process: 4441 ExecStart=/dm8/bin/DmASMSvrServicerac3 start (code=exited, status=0/SUCCESS)
Main PID: 4467 (dmasmsvr)
Tasks: 18
CGroup: /system.slice/DmASMSvrServicerac3.service
└─4467 /dm8/bin/dmasmsvr dcr_ini=/dm8/data/dmdcr.ini
Dec 10 11:00:08 dm8rac3 systemd[1]: Starting DM Auto Storage Manager Service(DmASMSvrServicerac3)....
Dec 10 11:00:23 dm8rac3 DmASMSvrServicerac3[4441]: [41B blob data]
Dec 10 11:00:23 dm8rac3 systemd[1]: Started DM Auto Storage Manager Service(DmASMSvrServicerac3)..
show
monitor current time:2021-12-10 11:01:08, n_group:3
=================== group[name = GRP_CSS, seq = 0, type = CSS, Control Node = 0] ========================================
[CSS0] auto check = TRUE, global info:
[ASM0] auto restart = FALSE
[RAC0] auto restart = FALSE
[CSS1] auto check = TRUE, global info:
[ASM1] auto restart = FALSE
[RAC1] auto restart = FALSE
[CSS2] auto check = TRUE, global info:
[ASM2] auto restart = FALSE
[RAC2] auto restart = FALSE
ep: css_time inst_name seqno port mode inst_status vtd_status is_ok active guid ts
2021-12-10 11:01:08 CSS0 0 9541 Control Node OPEN WORKING OK TRUE 738576474 738752802
2021-12-10 11:01:08 CSS1 1 9541 Normal Node OPEN WORKING OK TRUE 738606319 738782529
2021-12-10 11:01:08 CSS2 2 9541 Normal Node OPEN WORKING OK TRUE 790591989 790592089
=================== group[name = GRP_ASM, seq = 1, type = ASM, Control Node = 0] ========================================
n_ok_ep = 3
ok_ep_arr(index, seqno):
(0, 0)
(1, 1)
(2, 2)
sta = OPEN, sub_sta = STARTUP
break ep = NULL
recover ep = NULL
crash process over flag is TRUE
ep: css_time inst_name seqno port mode inst_status vtd_status is_ok active guid ts
2021-12-10 11:01:08 ASM0 0 9641 Control Node OPEN WORKING OK TRUE 738588988 738765275
2021-12-10 11:01:08 ASM1 1 9641 Normal Node OPEN WORKING OK TRUE 738624010 738800174
2021-12-10 11:01:08 ASM2 2 9641 Normal Node OPEN WORKING OK TRUE 790605125 790605178
=================== group[name = GRP_RAC, seq = 2, type = DB, Control Node = 0] ========================================
n_ok_ep = 2
ok_ep_arr(index, seqno):
(0, 0)
(1, 1)
sta = OPEN, sub_sta = STARTUP
break ep = NULL
recover ep = NULL
crash process over flag is TRUE
ep: css_time inst_name seqno port mode inst_status vtd_status is_ok active guid ts
2021-12-10 11:01:08 RAC0 0 5236 Control Node OPEN WORKING OK TRUE 2383210041 2383383635
2021-12-10 11:01:08 RAC1 1 5236 Normal Node OPEN WORKING OK TRUE 2383231629 2383405186
2021-12-10 11:01:08 RAC2 2 5236 Normal Node SHUTDOWN UNKNOWN ERROR FALSE 0 0
==================================================================================================================
10. 启动新的数据库服务器
如果DMCSS配置有自动拉起dmserver的功能，可以等待DMCSS自动拉起实例，不需要手动启动。
如果需要手动启动，可参考下面的操作步骤：
10.10.13.203机器：
./dmserver /home/data/dsc2_config/dm.ini dcr_ini=/home/data/dmdcr.ini
我这里注册成服务来启动数据库
[root@dm8rac3 system]# /dm8/script/root/dm_service_installer.sh -t dmserver -dm_ini /dm8/data/rac2_config/dm.ini -dcr_ini /dm8/data/dmdcr.ini -p rac3 -y DmASMSvrServicerac3
Created symlink from /etc/systemd/system/multi-user.target.wants/DmServicerac3.service to /usr/lib/systemd/system/DmServicerac3.service.
Finished to create the service (DmServicerac3)
[root@dm8rac3 system]# service DmServicerac3 start
Redirecting to /bin/systemctl start DmServicerac3.service
[root@dm8rac3 system]# service DmServicerac3 status
Redirecting to /bin/systemctl status DmServicerac3.service
DmServicerac3.service - DM Instance Service(DmServicerac3).
Loaded: loaded (/usr/lib/systemd/system/DmServicerac3.service; enabled; vendor preset: disabled)
Active: active (running) since Fri 2021-12-10 11:05:56 CST; 7s ago
Process: 4673 ExecStart=/dm8/bin/DmServicerac3 start (code=exited, status=0/SUCCESS)
Main PID: 4704 (dmserver)
Tasks: 105
CGroup: /system.slice/DmServicerac3.service
└─4704 /dm8/bin/dmserver path=/dm8/data/rac2_config/dm.ini dcr_ini=/dm8/data/dmdcr.ini -noconsole
Dec 10 11:05:40 dm8rac3 systemd[1]: Starting DM Instance Service(DmServicerac3)....
Dec 10 11:05:41 dm8rac3 DmServicerac3[4673]: Starting DmServicerac3: connnect dmasmtool successfully.
Dec 10 11:05:56 dm8rac3 DmServicerac3[4673]: [11B blob data]
Dec 10 11:05:56 dm8rac3 systemd[1]: Started DM Instance Service(DmServicerac3)..
show
monitor current time:2021-12-10 11:06:09, n_group:3
=================== group[name = GRP_CSS, seq = 0, type = CSS, Control Node = 0] ========================================
[CSS0] auto check = TRUE, global info:
[ASM0] auto restart = FALSE
[RAC0] auto restart = FALSE
[CSS1] auto check = TRUE, global info:
[ASM1] auto restart = FALSE
[RAC1] auto restart = FALSE
[CSS2] auto check = TRUE, global info:
[ASM2] auto restart = FALSE
[RAC2] auto restart = FALSE
ep: css_time inst_name seqno port mode inst_status vtd_status is_ok active guid ts
2021-12-10 11:06:09 CSS0 0 9541 Control Node OPEN WORKING OK TRUE 738576474 738753101
2021-12-10 11:06:09 CSS1 1 9541 Normal Node OPEN WORKING OK TRUE 738606319 738782829
2021-12-10 11:06:09 CSS2 2 9541 Normal Node OPEN WORKING OK TRUE 790591989 790592389
=================== group[name = GRP_ASM, seq = 1, type = ASM, Control Node = 0] ========================================
n_ok_ep = 3
ok_ep_arr(index, seqno):
(0, 0)
(1, 1)
(2, 2)
sta = OPEN, sub_sta = STARTUP
break ep = NULL
recover ep = NULL
crash process over flag is TRUE
ep: css_time inst_name seqno port mode inst_status vtd_status is_ok active guid ts
2021-12-10 11:06:09 ASM0 0 9641 Control Node OPEN WORKING OK TRUE 738588988 738765574
2021-12-10 11:06:09 ASM1 1 9641 Normal Node OPEN WORKING OK TRUE 738624010 738800474
2021-12-10 11:06:09 ASM2 2 9641 Normal Node OPEN WORKING OK TRUE 790605125 790605478
=================== group[name = GRP_RAC, seq = 2, type = DB, Control Node = 0] ========================================
n_ok_ep = 3
ok_ep_arr(index, seqno):
(0, 0)
(1, 1)
(2, 2)
sta = OPEN, sub_sta = STARTUP
break ep = NULL
recover ep = NULL
crash process over flag is TRUE
ep: css_time inst_name seqno port mode inst_status vtd_status is_ok active guid ts
2021-12-10 11:06:09 RAC0 0 5236 Control Node OPEN WORKING OK TRUE 2383210041 2383383934
2021-12-10 11:06:09 RAC1 1 5236 Normal Node OPEN WORKING OK TRUE 2383231629 2383405486
2021-12-10 11:06:09 RAC2 2 5236 Normal Node OPEN WORKING OK TRUE 2557760476 2557760496
==================================================================================================================
五.注意事项
1. 扩展节点前由用户保证所有dmcss/dmasmsvr/dmserver节点都是OK的，且都是活动的；
2. 每次扩展节点只能扩一个节点，扩展完成后可以再继续扩展节点；
3. 扩展节点的过程中不能出现修改实例状态或模式的操作；
4. 扩展节点的过程中，如果发生dmcss/dmasmsvr/dmserver实例故障，会导致扩展失败；
5. 扩展过程中操作失误（比如未修改dmmal.ini、asmsvrmal.ini，未增加日志文件），会导致扩展失败；
6. 执行完extend node命令，用户需要查看log文件，确认扩展操作是否成功；
7. 扩展失败可能会导致集群环境异常，需要退出所有dmcss/dmasmsvr/dmserver，重新init dcr磁盘。