前言
本文主要介绍了关于PostgreSQL中Slony-I同步复制部署的相关内容，分享出来供大家参考学习，下面话不多说了，来一起看看详细的介绍吧
本次测试环境
|
IP |
10.189.102.118 |
10.189.100.195 |
10.189.100.226 |
|
PGHOME |
/usr/local/pgsql |
/usr/local/pgsql |
/usr/local/pgsql |
|
Role |
Origin,Providers |
Subscribers,Providers |
Subscribers |
|
Database |
region_il |
region_il |
region_il |
|
Table |
il_longlive.entity il_longlive.entity_result il_longlive.result il_longlive.result_logbook |
il_longlive.entity il_longlive.entity_result il_longlive.result il_longlive.result_logbook |
il_longlive.entity il_longlive.entity_result il_longlive.result il_longlive.result_logbook |
|
Sequences |
il_longlive.entity_entity_id_seq il_longlive.entity_result_id_seq il_longlive.result_id_seq il_longlive.result_logbook_record_id_seq |
il_longlive.entity_entity_id_seq il_longlive.entity_result_id_seq il_longlive.result_id_seq il_longlive.result_logbook_record_id_seq |
il_longlive.entity_entity_id_seq il_longlive.entity_result_id_seq il_longlive.result_id_seq il_longlive.result_logbook_record_id_seq |
Slony-I一般要求表有主键,或者唯一键,Slony-I会自动识别主键,如果表没有主键,需要手工指定唯一键,如果没有指定唯一键,则不能复制,下图为Slony-I架构图.
1. 查看主库表信息
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 |
region_il=# \d List of relations Schema | Name | Type | Owner -------------+------------------------------+----------+---------- il_longlive | entity | table | postgres il_longlive | entity_entity_id_seq | sequence | postgres il_longlive | entity_result | table | postgres il_longlive | entity_result_id_seq | sequence | postgres il_longlive | result | table | postgres il_longlive | result_id_seq | sequence | postgres il_longlive | result_logbook | table | postgres il_longlive | result_logbook_record_id_seq | sequence | postgres il_longlive | result_vendor_schema | table | postgres il_longlive | result_vendor_schema_id_seq | sequence | postgres il_longlive | snapshot | table | postgres il_longlive | snapshot_id_seq | sequence | postgres (12 rows )
region_il=# \d entity Table "il_longlive.entity" Column | Type | Modifiers --------------+------------------------+------------------------------------------------------------ entity_id | integer | not null default nextval( 'entity_entity_id_seq' ::regclass) country_code | character varying (3) | state_code | character varying (50) | name | character varying (128) | islive | boolean | Indexes: "entity_pkey" PRIMARY KEY , btree (entity_id)
region_il=# select count (*) from entity; count --------- (1 row) |
2. 在所有节点安装Slony-I
|
1 2 3 4 5 6 |
$ wget http: //www .slony.info /downloads/2 .2 /source/slony1-2 .2.4. tar .bz2 $ tar -jxf slony1-2.2.4. tar .bz2 $ cd slony1-2.2.4 $ . /configure --prefix= /usr/local/pgsql --with-pgconfigdir= /usr/local/pgsql/bin --with-perltools $ make $ make install |
3. 在所有节点创建Slony-I使用的专用用户
|
1 |
$ psql -c "CREATE USER slonyadmin WITH SUPERUSER REPLICATION LOGIN PASSWORD '888888';" |
4. 在所有节点设置允许Slony-I用户远程登录
|
1 2 |
$ grep slonyadmin /usr/local/pgsql/data/pg_hba .conf host all slonyadmin 0.0.0.0 /0 md5 |
5. 在所有节点的对应数据库创建plpgsql语言
|
1 |
$ createlang plpgsql -d region_il |
6. 在主节点创建Slony-I配置文件,并复制到所有其他节点
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 |
$ cat /usr/ local /pgsql/etc/slon_tools.conf if ($ENV{ "SLONYNODES" }) { require $ENV{ "SLONYNODES" }; } else { $CLUSTER_NAME = 'region_il' ; $PIDFILE_DIR = '/usr/local/pgsql/data' ; $LOGDIR = '/usr/local/pgsql/data/pg_log' ; $MASTERNODE = 1; $DEBUGLEVEL = 0; $START_WATCHDOG = 1; $SLEEP_TIME = 60; $WATCHDOG_VERBOSE = 0;
add_node(node => 1, host => '10.189.102.118' , dbname => 'region_il' , port => 5432, user => 'slonyadmin' , password => '888888' ,);
add_node(node => 2, parent => 1, host => '10.189.100.195' , dbname => 'region_il' , port => 5432, user => 'slonyadmin' , password => '888888' ,);
add_node(node => 3, parent => 2, host => '10.189.100.226' , dbname => 'region_il' , port => 5432, user => 'slonyadmin' , password => '888888' ,); }
$SLONY_SETS = { "il_longlive_schema" => { "set_id" => 1, "table_id" => 101, "sequence_id" => 101, "pkeyedtables" => [ 'il_longlive.entity' , 'il_longlive.entity_result' , 'il_longlive.result' , 'il_longlive.result_logbook' , ],
"keyedtables" => {},
"sequences" => [ 'il_longlive.entity_entity_id_seq' , 'il_longlive.entity_result_id_seq' , 'il_longlive.result_id_seq' , 'il_longlive.result_logbook_record_id_seq' , ], },
};
if ($ENV{ "SLONYSET" }) { require $ENV{ "SLONYSET" }; }
# Please do not add or change anything below this point. 1; $ scp /usr/ local /pgsql/etc/slon_tools.conf postgres@10.189.100.195:/usr/ local /pgsql/etc/ $ scp /usr/ local /pgsql/etc/slon_tools.conf postgres@10.189.100.226:/usr/ local /pgsql/etc/ |
7. 将需要复制的表结构从主节点导出并导入到所有辅节点中
|
1 2 |
$ pg_dump -U slonyadmin -s -t il_longlive.entity -t il_longlive.entity_result -t il_longlive.result -t il_longlive.result_logbook region_il | psql -h 10.189.100.195 -U slonyadmin region_il $ pg_dump -U slonyadmin -s -t il_longlive.entity -t il_longlive.entity_result -t il_longlive.result -t il_longlive.result_logbook region_il | psql -h 10.189.100.226 -U slonyadmin region_il |
8. 在主节点初始化Slony-I集群
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 |
$ /usr/local/pgsql/bin/slonik_init_cluster
# INIT CLUSTER cluster name = region_il; node 1 admin conninfo= 'host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888' ; node 2 admin conninfo= 'host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888' ; node 3 admin conninfo= 'host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888' ; init cluster ( id = 1, comment = 'Node 1 - region_il@10.189.102.118' );
# STORE NODE store node ( id = 2, event node = 1, comment = 'Node 2 - region_il@10.189.100.195' ); store node ( id = 3, event node = 1, comment = 'Node 3 - region_il@10.189.100.226' ); echo 'Set up replication nodes' ;
# STORE PATH echo 'Next: configure paths for each node/origin' ; store path (server = 1, client = 2, conninfo = 'host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888' ); store path (server = 1, client = 3, conninfo = 'host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888' ); store path (server = 2, client = 1, conninfo = 'host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888' ); store path (server = 2, client = 3, conninfo = 'host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888' ); store path (server = 3, client = 1, conninfo = 'host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888' ); store path (server = 3, client = 2, conninfo = 'host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888' ); echo 'Replication nodes prepared' ; echo 'Please start a slon replication daemon for each node' ; $ /usr/local/pgsql/bin/slonik_init_cluster | slonik <stdin>:7: Possible unsupported PostgreSQL version (90505) 9.5, defaulting to 8.4 support <stdin>:10: Possible unsupported PostgreSQL version (90505) 9.5, defaulting to 8.4 support <stdin>:11: Possible unsupported PostgreSQL version (90505) 9.5, defaulting to 8.4 support <stdin>:12: Set up replication nodes <stdin>:15: Next: configure paths for each node /origin <stdin>:22: Replication nodes prepared <stdin>:23: Please start a slon replication daemon for each node |
9. 在所有节点启动slon进程
在主节点启动slon进程
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
$ slon_start 1 Invoke slon for node 1 - /usr/local/pgsql/bin/slon -p /usr/local/pgsql/data/region_il_node1 .pid -s 1000 -d0 region_il 'host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888' > /usr/local/pgsql/data/pg_log/node1/region_il-2017-06-14 .log 2>&1 & Slon successfully started for cluster region_il, node node1 PID [2454] Start the watchdog process as well... $ slon_status 1 Slon is running for the 'region_il' cluster on node1. $ ps -ef | grep slon postgres 2454 1 0 22:54 pts /1 0 /usr/local/pgsql/bin/slon -p /usr/local/pgsql/data/region_il_node1 .pid -s 1000 -d0 region_il host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888 postgres 2455 2454 0 22:54 pts /1 0 /usr/local/pgsql/bin/slon -p /usr/local/pgsql/data/region_il_node1 .pid -s 1000 -d0 region_il host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888 postgres 2459 24883 0 22:54 ? 0 postgres: slonyadmin region_il 10.189.102.118(24667) idle postgres 2467 24883 0 22:54 ? 0 postgres: slonyadmin region_il 10.189.102.118(24669) idle postgres 2468 24883 0 22:54 ? 0 postgres: slonyadmin region_il 10.189.102.118(24673) idle postgres 2469 24883 0 22:54 ? 0 postgres: slonyadmin region_il 10.189.102.118(24675) idle postgres 2470 24883 0 22:54 ? 0 postgres: slonyadmin region_il 10.189.102.118(24677) idle postgres 2471 24883 0 22:54 ? 0 postgres: slonyadmin region_il 10.189.102.118(24681) idle postgres 2473 1 0 22:54 pts /1 0 /usr/bin/perl /usr/local/pgsql/bin/slon_watchdog --config= /usr/local/pgsql/etc/slon_tools .conf node1 60 |
在辅节点1启动slave进程
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
$ slon_start 2 Invoke slon for node 2 - /usr/local/pgsql/bin/slon -p /usr/local/pgsql/data/region_il_node2 .pid -s 1000 -d0 region_il 'host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888' > /usr/local/pgsql/data/pg_log/node2/region_il-2017-06-14 .log 2>&1 & Slon successfully started for cluster region_il, node node2 PID [15535] Start the watchdog process as well... $ slon_status 2 Slon is running for the 'region_il' cluster on node2. $ ps -ef | grep slon postgres 15515 12843 0 22:54 ? 0 postgres: slonyadmin region_il 10.189.102.118(38804) idle postgres 15535 1 0 22:57 pts /0 0 /usr/local/pgsql/bin/slon -p /usr/local/pgsql/data/region_il_node2 .pid -s 1000 -d0 region_il host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888 postgres 15536 15535 0 22:57 pts /0 0 /usr/local/pgsql/bin/slon -p /usr/local/pgsql/data/region_il_node2 .pid -s 1000 -d0 region_il host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888 postgres 15540 12843 0 22:57 ? 0 postgres: slonyadmin region_il 10.189.100.195(54629) idle postgres 15546 12843 0 22:57 ? 0 postgres: slonyadmin region_il 10.189.100.195(54631) idle postgres 15547 12843 0 22:57 ? 0 postgres: slonyadmin region_il 10.189.100.195(54633) idle postgres 15548 12843 0 22:57 ? 0 postgres: slonyadmin region_il 10.189.100.195(54635) idle postgres 15549 12843 0 22:57 ? 0 postgres: slonyadmin region_il 10.189.100.195(54639) idle postgres 15552 12843 0 22:57 ? 0 postgres: slonyadmin region_il 10.189.100.195(54643) idle postgres 15555 1 0 22:57 pts /0 0 /usr/bin/perl /usr/local/pgsql/bin/slon_watchdog --config= /usr/local/pgsql/etc/slon_tools .conf node2 60 postgres 15559 12843 0 22:57 ? 0 postgres: slonyadmin region_il 10.189.102.118(38812) idle |
在辅节点2启动slave进程
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 |
$ slon_start 3 Invoke slon for node 3 - /usr/local/pgsql/bin/slon -p /usr/local/pgsql/data/region_il_node3 .pid -s 1000 -d0 region_il 'host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888' > /usr/local/pgsql/data/pg_log/node3/region_il-2017-06-14 .log 2>&1 & Slon successfully started for cluster region_il, node node3 PID [19212] Start the watchdog process as well... $ slon_status 3 Slon is running for the 'region_il' cluster on node3. $ ps -ef | grep slon postgres 6033 1 0 Jun02 ? 00:00:02 /usr/local/pgsql/bin/slon -p /usr/local/pgsql/data/region_il_node2 .pid -s 1000 -d0 region_il host=ec2t-pgtest-03 dbname=region_il user=slonyadmin port=5432 password=mypna123 postgres 6050 1 0 Jun02 ? 00:00:02 /usr/bin/perl /usr/local/pgsql/bin/slon_watchdog --config= /usr/local/pgsql/etc/slon_tools .conf node2 60 postgres 19059 15237 0 22:54 ? 0 postgres: slonyadmin region_il 10.189.102.118(63615) idle postgres 19109 15237 0 22:57 ? 0 postgres: slonyadmin region_il 10.189.100.195(26781) idle postgres 19212 1 0 23:00 pts /2 0 /usr/local/pgsql/bin/slon -p /usr/local/pgsql/data/region_il_node3 .pid -s 1000 -d0 region_il host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888 postgres 19213 19212 0 23:00 pts /2 0 /usr/local/pgsql/bin/slon -p /usr/local/pgsql/data/region_il_node3 .pid -s 1000 -d0 region_il host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888 postgres 19217 15237 0 23:00 ? 0 postgres: slonyadmin region_il 10.189.100.226(41213) idle postgres 19225 15237 0 23:00 ? 0 postgres: slonyadmin region_il 10.189.100.226(41215) idle postgres 19226 15237 0 23:00 ? 0 postgres: slonyadmin region_il 10.189.100.226(41219) idle postgres 19227 15237 0 23:00 ? 0 postgres: slonyadmin region_il 10.189.100.226(41221) idle postgres 19228 15237 0 23:00 ? 0 postgres: slonyadmin region_il 10.189.100.226(41223) idle postgres 19229 15237 0 23:00 ? 0 postgres: slonyadmin region_il 10.189.100.226(41227) idle postgres 19232 1 0 23:00 pts /2 0 /usr/bin/perl /usr/local/pgsql/bin/slon_watchdog --config= /usr/local/pgsql/etc/slon_tools .conf node3 60 postgres 19234 15237 0 23:00 ? 0 postgres: slonyadmin region_il 10.189.102.118(63635) idle postgres 19235 6033 0 23:00 ? 0 /usr/local/pgsql/bin/slon -p /usr/local/pgsql/data/region_il_node2 .pid -s 1000 -d0 region_il host=ec2t-pgtest-03 dbname=region_il user=slonyadmin port=5432 password=mypna123 postgres 19239 15237 0 23:00 ? 0 postgres: slonyadmin region_il 10.189.102.118(63637) idle |
10. 在主节点创建复制集
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 |
$ /usr/local/pgsql/bin/slonik_create_set 1 cluster name = region_il; node 1 admin conninfo= 'host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888' ; node 2 admin conninfo= 'host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888' ; node 3 admin conninfo= 'host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888' ;
# TABLE ADD KEY
# CREATE SET create set ( id = 1, origin = 1, comment = 'Set 1 (il_longlive_schema) for region_il' );
# SET ADD TABLE echo 'Subscription set 1 (il_longlive_schema) created' ; echo 'Adding tables to the subscription set' ; set add table ( set id = 1, origin = 1, id = 101, full qualified name = 'il_longlive.entity' , comment = 'Table il_longlive.entity with primary key' ); echo 'Add primary keyed table il_longlive.entity' ; set add table ( set id = 1, origin = 1, id = 102, full qualified name = 'il_longlive.entity_result' , comment = 'Table il_longlive.entity_result with primary key' ); echo 'Add primary keyed table il_longlive.entity_result' ; set add table ( set id = 1, origin = 1, id = 103, full qualified name = 'il_longlive.result' , comment = 'Table il_longlive.result with primary key' ); echo 'Add primary keyed table il_longlive.result' ; set add table ( set id = 1, origin = 1, id = 104, full qualified name = 'il_longlive.result_logbook' , comment = 'Table il_longlive.result_logbook with primary key' ); echo 'Add primary keyed table il_longlive.result_logbook' ;
# SET ADD SEQUENCE echo 'Adding sequences to the subscription set' ; set add sequence ( set id = 1, origin = 1, id = 101, full qualified name = 'il_longlive.entity_entity_id_seq' , comment = 'Sequence il_longlive.entity_entity_id_seq' ); echo 'Add sequence il_longlive.entity_entity_id_seq' ; set add sequence ( set id = 1, origin = 1, id = 102, full qualified name = 'il_longlive.entity_result_id_seq' , comment = 'Sequence il_longlive.entity_result_id_seq' ); echo 'Add sequence il_longlive.entity_result_id_seq' ; set add sequence ( set id = 1, origin = 1, id = 103, full qualified name = 'il_longlive.result_id_seq' , comment = 'Sequence il_longlive.result_id_seq' ); echo 'Add sequence il_longlive.result_id_seq' ; set add sequence ( set id = 1, origin = 1, id = 104, full qualified name = 'il_longlive.result_logbook_record_id_seq' , comment = 'Sequence il_longlive.result_logbook_record_id_seq' ); echo 'Add sequence il_longlive.result_logbook_record_id_seq' ; echo 'All tables added' ; $ /usr/local/pgsql/bin/slonik_create_set 1 | slonik <stdin>:12: Subscription set 1 (il_longlive_schema) created <stdin>:13: Adding tables to the subscription set <stdin>:17: Add primary keyed table il_longlive.entity <stdin>:21: Add primary keyed table il_longlive.entity_result <stdin>:25: Add primary keyed table il_longlive.result <stdin>:29: Add primary keyed table il_longlive.result_logbook <stdin>:32: Adding sequences to the subscription set <stdin>:36: Add sequence il_longlive.entity_entity_id_seq <stdin>:40: Add sequence il_longlive.entity_result_id_seq <stdin>:44: Add sequence il_longlive.result_id_seq <stdin>:48: Add sequence il_longlive.result_logbook_record_id_seq <stdin>:49: All tables added |
11. 在主节点创建复制订阅者
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
$ slonik_subscribe_set 1 2 cluster name = region_il; node 1 admin conninfo= 'host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888' ; node 2 admin conninfo= 'host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888' ; node 3 admin conninfo= 'host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888' ;
subscribe set ( id = 1, provider = 1, receiver = 2, forward = yes ); echo 'Subscribed nodes to set 1' ; $ slonik_subscribe_set 1 2 | slonik <stdin>:7: Subscribed nodes to set 1 $ slonik_subscribe_set 1 3 cluster name = region_il; node 1 admin conninfo= 'host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888' ; node 2 admin conninfo= 'host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888' ; node 3 admin conninfo= 'host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888' ;
subscribe set ( id = 1, provider = 2, receiver = 3, forward = yes ); echo 'Subscribed nodes to set 1' ; $ slonik_subscribe_set 1 3 | slonik <stdin>:7: Subscribed nodes to set 1 |
12.查看各节点Slony-I日志信息
主节点查看Slony-I日志信息
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 |
$ cat region_il-2017-06-14.log 2017-06-14 22:54:42 PDT CONFIG main: slon version 2.2.4 starting up 2017-06-14 22:54:42 PDT INFO slon: watchdog process started 2017-06-14 22:54:42 PDT CONFIG slon: watchdog ready - pid = 2454 2017-06-14 22:54:42 PDT CONFIG slon: worker process created - pid = 2455 2017-06-14 22:54:42 PDT CONFIG main: Integer option vac_frequency = 3 2017-06-14 22:54:42 PDT CONFIG main: Integer option log_level = 0 2017-06-14 22:54:42 PDT CONFIG main: Integer option sync_interval = 1000 2017-06-14 22:54:42 PDT CONFIG main: Integer option sync_interval_timeout = 10000 2017-06-14 22:54:42 PDT CONFIG main: Integer option sync_group_maxsize = 20 2017-06-14 22:54:42 PDT CONFIG main: Integer option syslog = 0 2017-06-14 22:54:42 PDT CONFIG main: Integer option quit_sync_provider = 0 2017-06-14 22:54:42 PDT CONFIG main: Integer option remote_listen_timeout = 300 2017-06-14 22:54:42 PDT CONFIG main: Integer option monitor_interval = 500 2017-06-14 22:54:42 PDT CONFIG main: Integer option explain_interval = 0 2017-06-14 22:54:42 PDT CONFIG main: Integer option tcp_keepalive_idle = 0 2017-06-14 22:54:42 PDT CONFIG main: Integer option tcp_keepalive_interval = 0 2017-06-14 22:54:42 PDT CONFIG main: Integer option tcp_keepalive_count = 0 2017-06-14 22:54:42 PDT CONFIG main: Integer option apply_cache_size = 100 2017-06-14 22:54:42 PDT CONFIG main: Boolean option log_pid = 0 2017-06-14 22:54:42 PDT CONFIG main: Boolean option log_timestamp = 1 2017-06-14 22:54:42 PDT CONFIG main: Boolean option tcp_keepalive = 1 2017-06-14 22:54:42 PDT CONFIG main: Boolean option monitor_threads = 1 2017-06-14 22:54:42 PDT CONFIG main: Real option real_placeholder = 0.000000 2017-06-14 22:54:42 PDT CONFIG main: String option cluster_name = region_il 2017-06-14 22:54:42 PDT CONFIG main: String option conn_info = host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888 2017-06-14 22:54:42 PDT CONFIG main: String option pid_file = /usr/local/pgsql/data/region_il_node1 .pid 2017-06-14 22:54:42 PDT CONFIG main: String option log_timestamp_format = %Y-%m-%d %H:%M:%S %Z 2017-06-14 22:54:42 PDT CONFIG main: String option archive_dir = [NULL] 2017-06-14 22:54:42 PDT CONFIG main: String option sql_on_connection = [NULL] 2017-06-14 22:54:42 PDT CONFIG main: String option lag_interval = [NULL] 2017-06-14 22:54:42 PDT CONFIG main: String option command_on_logarchive = [NULL] 2017-06-14 22:54:42 PDT CONFIG main: String option syslog_facility = LOCAL0 2017-06-14 22:54:42 PDT CONFIG main: String option syslog_ident = slon 2017-06-14 22:54:42 PDT CONFIG main: String option cleanup_interval = 10 minutes 2017-06-14 22:54:42 PDT CONFIG main: local node id = 1 2017-06-14 22:54:42 PDT INFO main: main process started 2017-06-14 22:54:42 PDT CONFIG main: launching sched_start_mainloop 2017-06-14 22:54:42 PDT CONFIG main: loading current cluster configuration 2017-06-14 22:54:42 PDT CONFIG storeNode: no_id=2 no_comment= 'Node 2 - region_il@10.189.100.195' 2017-06-14 22:54:42 PDT CONFIG storeNode: no_id=3 no_comment= 'Node 3 - region_il@10.189.100.226' 2017-06-14 22:54:42 PDT CONFIG storePath: pa_server=2 pa_client=1 pa_conninfo= "host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" pa_connretry=10 2017-06-14 22:54:42 PDT CONFIG storePath: pa_server=3 pa_client=1 pa_conninfo= "host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888" pa_connretry=10 2017-06-14 22:54:42 PDT CONFIG storeListen: li_origin=2 li_receiver=1 li_provider=2 2017-06-14 22:54:42 PDT CONFIG storeListen: li_origin=3 li_receiver=1 li_provider=3 2017-06-14 22:54:42 PDT CONFIG main: last local event sequence = 5000000007 2017-06-14 22:54:42 PDT CONFIG main: configuration complete - starting threads 2017-06-14 22:54:42 PDT INFO localListenThread: thread starts 2017-06-14 22:54:42 PDT CONFIG version for "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505 2017-06-14 22:54:42 PDT CONFIG enableNode: no_id=2 2017-06-14 22:54:42 PDT CONFIG enableNode: no_id=3 2017-06-14 22:54:42 PDT INFO main: running scheduler mainloop 2017-06-14 22:54:42 PDT INFO remoteWorkerThread_3: thread starts 2017-06-14 22:54:42 PDT INFO remoteListenThread_3: thread starts 2017-06-14 22:54:42 PDT CONFIG cleanupThread: thread starts 2017-06-14 22:54:42 PDT INFO syncThread: thread starts 2017-06-14 22:54:42 PDT INFO monitorThread: thread starts 2017-06-14 22:54:42 PDT INFO remoteListenThread_2: thread starts 2017-06-14 22:54:42 PDT INFO remoteWorkerThread_2: thread starts 2017-06-14 22:54:42 PDT CONFIG version for "host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505 2017-06-14 22:54:42 PDT CONFIG version for "host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505 2017-06-14 22:54:42 PDT CONFIG version for "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505 2017-06-14 22:54:42 PDT CONFIG version for "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505 2017-06-14 22:54:42 PDT CONFIG version for "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505 2017-06-14 22:54:42 PDT CONFIG version for "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505 2017-06-14 22:54:42 PDT CONFIG cleanupThread: bias = 60 2017-06-14 22:54:42 PDT CONFIG remoteWorkerThread_2: update provider configuration 2017-06-14 22:54:42 PDT CONFIG version for "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505 2017-06-14 22:54:42 PDT CONFIG remoteWorkerThread_3: update provider configuration 2017-06-14 22:54:42 PDT CONFIG storeListen: li_origin=2 li_receiver=1 li_provider=2 2017-06-14 22:54:42 PDT CONFIG storeListen: li_origin=3 li_receiver=1 li_provider=3 2017-06-14 22:54:42 PDT CONFIG storeListen: li_origin=3 li_receiver=1 li_provider=2 2017-06-14 22:54:42 PDT CONFIG remoteWorkerThread_2: update provider configuration 2017-06-14 22:54:42 PDT CONFIG storeListen: li_origin=2 li_receiver=1 li_provider=2 2017-06-14 22:54:42 PDT CONFIG storeListen: li_origin=3 li_receiver=1 li_provider=3 2017-06-14 22:54:42 PDT CONFIG storeListen: li_origin=3 li_receiver=1 li_provider=2 2017-06-14 22:54:42 PDT CONFIG storeListen: li_origin=2 li_receiver=1 li_provider=3 2017-06-14 22:54:42 PDT CONFIG remoteWorkerThread_3: update provider configuration 2017-06-14 22:54:42 PDT CONFIG storeListen: li_origin=2 li_receiver=1 li_provider=2 2017-06-14 22:54:42 PDT CONFIG storeListen: li_origin=3 li_receiver=1 li_provider=3 2017-06-14 22:54:42 PDT CONFIG storeListen: li_origin=3 li_receiver=1 li_provider=2 2017-06-14 22:54:42 PDT CONFIG storeListen: li_origin=2 li_receiver=1 li_provider=3 2017-06-14 22:54:42 PDT CONFIG remoteWorkerThread_2: update provider configuration 2017-06-14 22:54:42 PDT CONFIG storeListen: li_origin=2 li_receiver=1 li_provider=2 2017-06-14 22:54:42 PDT CONFIG storeListen: li_origin=3 li_receiver=1 li_provider=3 2017-06-14 22:54:42 PDT CONFIG storeListen: li_origin=3 li_receiver=1 li_provider=2 2017-06-14 22:54:42 PDT CONFIG storeListen: li_origin=2 li_receiver=1 li_provider=3 2017-06-14 22:54:42 PDT CONFIG remoteWorkerThread_3: update provider configuration 2017-06-14 22:54:42 PDT CONFIG remoteWorkerThread_2: update provider configuration 2017-06-14 22:57:50 PDT CONFIG remoteWorkerThread_2: update provider configuration 2017-06-14 22:57:50 PDT CONFIG version for "host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505 2017-06-14 22:57:50 PDT INFO remoteWorkerThread_2: SYNC 5000000003 done in 0.014 seconds 2017-06-14 22:57:58 PDT INFO remoteWorkerThread_2: SYNC 5000000004 done in 0.003 seconds 2017-06-14 22:58:11 PDT INFO remoteWorkerThread_2: SYNC 5000000005 done in 0.002 seconds 2017-06-14 22:58:19 PDT INFO remoteWorkerThread_2: SYNC 5000000006 done in 0.002 seconds 2017-06-14 22:58:27 PDT INFO remoteWorkerThread_2: SYNC 5000000007 done in 0.003 seconds 2017-06-14 22:58:35 PDT INFO remoteWorkerThread_2: SYNC 5000000008 done in 0.002 seconds 2017-06-14 22:58:43 PDT INFO remoteWorkerThread_2: SYNC 5000000009 done in 0.002 seconds 2017-06-14 22:59:01 PDT INFO remoteWorkerThread_2: SYNC 5000000010 done in 0.003 seconds 2017-06-14 22:59:09 PDT INFO remoteWorkerThread_2: SYNC 5000000011 done in 0.002 seconds 2017-06-14 22:59:17 PDT INFO remoteWorkerThread_2: SYNC 5000000012 done in 0.002 seconds 2017-06-14 22:59:25 PDT INFO remoteWorkerThread_2: SYNC 5000000013 done in 0.002 seconds 2017-06-14 22:59:38 PDT INFO remoteWorkerThread_2: SYNC 5000000014 done in 0.003 seconds 2017-06-14 22:59:51 PDT INFO remoteWorkerThread_2: SYNC 5000000015 done in 0.002 seconds 2017-06-14 22:59:54 PDT INFO remoteWorkerThread_2: SYNC 5000000016 done in 0.002 seconds 2017-06-14 23:00:05 PDT INFO remoteWorkerThread_2: SYNC 5000000017 done in 0.002 seconds 2017-06-14 23:00:19 PDT INFO remoteWorkerThread_2: SYNC 5000000018 done in 0.003 seconds 2017-06-14 23:00:32 PDT INFO remoteWorkerThread_2: SYNC 5000000019 done in 0.003 seconds 2017-06-14 23:00:35 PDT INFO remoteWorkerThread_2: SYNC 5000000020 done in 0.003 seconds 2017-06-14 23:00:43 PDT CONFIG remoteWorkerThread_3: update provider configuration 2017-06-14 23:00:43 PDT CONFIG version for "host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505 2017-06-14 23:00:43 PDT INFO remoteWorkerThread_3: SYNC 5000000003 done in 0.045 seconds 2017-06-14 23:00:47 PDT CONFIG remoteWorkerThread_2: update provider configuration 2017-06-14 23:00:47 PDT CONFIG version for "host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505 2017-06-14 23:00:47 PDT INFO remoteWorkerThread_2: SYNC 5000000021 done in 0.014 seconds 2017-06-14 23:00:55 PDT INFO remoteWorkerThread_3: SYNC 5000000004 done in 0.002 seconds 2017-06-14 23:00:55 PDT INFO remoteWorkerThread_2: SYNC 5000000022 done in 0.005 seconds 2017-06-14 23:01:08 PDT INFO remoteWorkerThread_3: SYNC 5000000005 done in 0.002 seconds 2017-06-14 23:01:08 PDT INFO remoteWorkerThread_2: SYNC 5000000023 done in 0.005 seconds 2017-06-14 23:01:16 PDT INFO remoteWorkerThread_3: SYNC 5000000006 done in 0.002 seconds 2017-06-14 23:01:20 PDT INFO remoteWorkerThread_3: SYNC 5000000007 done in 0.002 seconds 2017-06-14 23:01:20 PDT INFO remoteWorkerThread_2: SYNC 5000000024 done in 0.005 seconds 2017-06-14 23:01:24 PDT INFO remoteWorkerThread_2: SYNC 5000000025 done in 0.005 seconds 2017-06-14 23:01:38 PDT INFO remoteWorkerThread_3: SYNC 5000000008 done in 0.003 seconds 2017-06-14 23:01:38 PDT INFO remoteWorkerThread_2: SYNC 5000000026 done in 0.005 seconds 2017-06-14 23:01:41 PDT INFO remoteWorkerThread_3: SYNC 5000000009 done in 0.002 seconds 2017-06-14 23:01:44 PDT INFO remoteWorkerThread_2: SYNC 5000000027 done in 0.004 seconds 2017-06-14 23:01:52 PDT INFO remoteWorkerThread_3: SYNC 5000000010 done in 0.003 seconds 2017-06-14 23:01:55 PDT INFO remoteWorkerThread_2: SYNC 5000000028 done in 0.005 seconds 2017-06-14 23:02:08 PDT CONFIG remoteWorkerThread_3: update provider configuration 2017-06-14 23:02:08 PDT CONFIG version for "host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505 2017-06-14 23:02:08 PDT INFO remoteWorkerThread_2: SYNC 5000000029 done in 0.012 seconds 2017-06-14 23:02:08 PDT INFO remoteWorkerThread_3: SYNC 5000000011 done in 0.016 seconds 2017-06-14 23:02:11 PDT INFO remoteWorkerThread_3: SYNC 5000000012 done in 0.004 seconds 2017-06-14 23:02:19 PDT INFO remoteWorkerThread_2: SYNC 5000000030 done in 0.005 seconds 2017-06-14 23:02:22 PDT INFO remoteWorkerThread_3: SYNC 5000000013 done in 0.005 seconds 2017-06-14 23:02:25 PDT INFO remoteWorkerThread_2: SYNC 5000000031 done in 0.004 seconds 2017-06-14 23:02:38 PDT INFO remoteWorkerThread_2: SYNC 5000000032 done in 0.004 seconds 2017-06-14 23:02:38 PDT INFO remoteWorkerThread_3: SYNC 5000000014 done in 0.005 seconds 2017-06-14 23:02:46 PDT INFO remoteWorkerThread_3: SYNC 5000000015 done in 0.005 seconds 2017-06-14 23:02:50 PDT INFO remoteWorkerThread_3: SYNC 5000000016 done in 0.004 seconds 2017-06-14 23:02:50 PDT INFO remoteWorkerThread_2: SYNC 5000000033 done in 0.005 seconds 2017-06-14 23:02:54 PDT INFO remoteWorkerThread_2: SYNC 5000000034 done in 0.004 seconds 2017-06-14 23:03:08 PDT INFO remoteWorkerThread_3: SYNC 5000000017 done in 0.004 seconds 2017-06-14 23:03:08 PDT INFO remoteWorkerThread_2: SYNC 5000000035 done in 0.005 seconds 2017-06-14 23:03:11 PDT CONFIG storeSet: set_id=1 set_origin=1 set_comment= 'Set 1 (il_longlive_schema) for region_il' 2017-06-14 23:03:16 PDT INFO remoteWorkerThread_3: SYNC 5000000018 done in 0.005 seconds 2017-06-14 23:03:17 PDT INFO remoteWorkerThread_2: SYNC 5000000036 done in 0.005 seconds 2017-06-14 23:03:21 PDT INFO remoteWorkerThread_3: SYNC 5000000019 done in 0.004 seconds 2017-06-14 23:03:29 PDT INFO remoteWorkerThread_2: SYNC 5000000037 done in 0.016 seconds 2017-06-14 23:03:32 PDT INFO remoteWorkerThread_3: SYNC 5000000020 done in 0.012 seconds 2017-06-14 23:03:35 PDT INFO remoteWorkerThread_2: SYNC 5000000038 done in 0.004 seconds 2017-06-14 23:03:48 PDT INFO remoteWorkerThread_2: SYNC 5000000039 done in 0.004 seconds 2017-06-14 23:03:48 PDT INFO remoteWorkerThread_3: SYNC 5000000021 done in 0.005 seconds 2017-06-14 23:03:56 PDT INFO remoteWorkerThread_3: SYNC 5000000022 done in 0.005 seconds 2017-06-14 23:03:57 PDT INFO remoteWorkerThread_2: SYNC 5000000040 done in 0.005 seconds 2017-06-14 23:04:01 PDT INFO remoteWorkerThread_3: SYNC 5000000023 done in 0.005 seconds 2017-06-14 23:04:04 PDT INFO remoteWorkerThread_2: SYNC 5000000041 done in 0.005 seconds 2017-06-14 23:04:17 PDT INFO remoteWorkerThread_2: SYNC 5000000042 done in 0.005 seconds 2017-06-14 23:04:17 PDT INFO remoteWorkerThread_3: SYNC 5000000024 done in 0.005 seconds 2017-06-14 23:04:20 PDT INFO remoteWorkerThread_3: SYNC 5000000025 done in 0.004 seconds 2017-06-14 23:04:23 PDT INFO remoteWorkerThread_2: SYNC 5000000043 done in 0.008 seconds 2017-06-14 23:04:31 PDT INFO remoteWorkerThread_3: SYNC 5000000026 done in 0.005 seconds 2017-06-14 23:04:34 PDT INFO remoteWorkerThread_2: SYNC 5000000044 done in 0.004 seconds 2017-06-14 23:04:42 PDT INFO remoteWorkerThread_3: SYNC 5000000027 done in 0.005 seconds NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=2498 CONTEXT: SQL statement "SELECT " _region_il ".cleanupNodelock()" PL /pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=2499 CONTEXT: SQL statement "SELECT " _region_il ".cleanupNodelock()" PL /pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=2500 CONTEXT: SQL statement "SELECT " _region_il ".cleanupNodelock()" PL /pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=2526 CONTEXT: SQL statement "SELECT " _region_il ".cleanupNodelock()" PL /pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM NOTICE: Slony-I: Logswitch to sl_log_2 initiated CONTEXT: SQL statement "SELECT " _region_il ".logswitch_start()" PL /pgSQL function _region_il.cleanupevent(interval) line 97 at PERFORM 2017-06-14 23:04:43 PDT INFO cleanupThread: 0.011 seconds for cleanupEvent() 2017-06-14 23:04:45 PDT INFO remoteWorkerThread_2: SYNC 5000000045 done in 0.004 seconds 2017-06-14 23:04:53 PDT INFO remoteWorkerThread_2: SYNC 5000000046 done in 0.005 seconds 2017-06-14 23:04:53 PDT INFO remoteWorkerThread_3: SYNC 5000000028 done in 0.005 seconds 2017-06-14 23:05:06 PDT INFO remoteWorkerThread_3: SYNC 5000000029 done in 0.005 seconds 2017-06-14 23:05:06 PDT INFO remoteWorkerThread_2: SYNC 5000000047 done in 0.005 seconds 2017-06-14 23:05:19 PDT INFO remoteWorkerThread_2: SYNC 5000000048 done in 0.004 seconds 2017-06-14 23:05:19 PDT INFO remoteWorkerThread_3: SYNC 5000000030 done in 0.012 seconds 2017-06-14 23:05:22 PDT INFO remoteWorkerThread_3: SYNC 5000000031 done in 0.004 seconds 2017-06-14 23:05:25 PDT INFO remoteWorkerThread_2: SYNC 5000000049 done in 0.005 seconds 2017-06-14 23:05:38 PDT INFO remoteWorkerThread_2: SYNC 5000000050 done in 0.004 seconds 2017-06-14 23:05:38 PDT INFO remoteWorkerThread_3: SYNC 5000000032 done in 0.005 seconds 2017-06-14 23:05:41 PDT INFO remoteWorkerThread_3: SYNC 5000000033 done in 0.005 seconds 2017-06-14 23:05:45 PDT INFO remoteWorkerThread_2: SYNC 5000000051 done in 0.005 seconds 2017-06-14 23:05:53 PDT INFO remoteWorkerThread_3: SYNC 5000000034 done in 0.004 seconds 2017-06-14 23:05:53 PDT INFO remoteWorkerThread_2: SYNC 5000000052 done in 0.005 seconds 2017-06-14 23:06:06 PDT INFO remoteWorkerThread_3: SYNC 5000000035 done in 0.005 seconds 2017-06-14 23:06:06 PDT INFO remoteWorkerThread_2: SYNC 5000000053 done in 0.005 seconds 2017-06-14 23:06:14 PDT INFO remoteWorkerThread_2: SYNC 5000000054 done in 0.005 seconds 2017-06-14 23:06:14 PDT INFO remoteWorkerThread_3: SYNC 5000000036 done in 0.006 seconds 2017-06-14 23:06:22 PDT INFO remoteWorkerThread_3: SYNC 5000000037 done in 0.005 seconds 2017-06-14 23:06:25 PDT INFO remoteWorkerThread_2: SYNC 5000000055 done in 0.004 seconds 2017-06-14 23:06:30 PDT INFO remoteWorkerThread_3: SYNC 5000000038 done in 0.005 seconds 2017-06-14 23:06:36 PDT INFO remoteWorkerThread_2: SYNC 5000000056 done in 0.005 seconds 2017-06-14 23:06:40 PDT INFO remoteWorkerThread_3: SYNC 5000000039 done in 0.005 seconds 2017-06-14 23:06:48 PDT INFO remoteWorkerThread_2: SYNC 5000000057 done in 0.007 seconds 2017-06-14 23:06:56 PDT INFO remoteWorkerThread_3: SYNC 5000000040 done in 0.004 seconds 2017-06-14 23:06:56 PDT INFO remoteWorkerThread_2: SYNC 5000000058 done in 0.005 seconds 2017-06-14 23:07:09 PDT INFO remoteWorkerThread_3: SYNC 5000000041 done in 0.004 seconds 2017-06-14 23:07:09 PDT INFO remoteWorkerThread_2: SYNC 5000000059 done in 0.005 seconds 2017-06-14 23:07:10 PDT INFO remoteWorkerThread_3: SYNC 5000000042 done in 0.004 seconds 2017-06-14 23:07:19 PDT INFO remoteWorkerThread_2: SYNC 5000000060 done in 0.005 seconds 2017-06-14 23:07:23 PDT INFO remoteWorkerThread_3: SYNC 5000000043 done in 0.015 seconds 2017-06-14 23:07:23 PDT INFO remoteWorkerThread_2: SYNC 5000000061 done in 0.017 seconds 2017-06-14 23:07:31 PDT INFO remoteWorkerThread_3: SYNC 5000000044 done in 0.004 seconds 2017-06-14 23:07:39 PDT INFO remoteWorkerThread_2: SYNC 5000000062 done in 0.013 seconds 2017-06-14 23:07:42 PDT INFO remoteWorkerThread_3: SYNC 5000000045 done in 0.006 seconds 2017-06-14 23:07:43 PDT INFO remoteWorkerThread_2: SYNC 5000000063 done in 0.005 seconds 2017-06-14 23:07:51 PDT INFO remoteWorkerThread_3: SYNC 5000000046 done in 0.008 seconds 2017-06-14 23:07:56 PDT INFO remoteWorkerThread_2: SYNC 5000000064 done in 0.005 seconds 2017-06-14 23:08:02 PDT INFO remoteWorkerThread_3: SYNC 5000000047 done in 0.022 seconds 2017-06-14 23:08:07 PDT INFO remoteWorkerThread_2: SYNC 5000000065 done in 0.005 seconds 2017-06-14 23:08:13 PDT INFO remoteWorkerThread_3: SYNC 5000000048 done in 0.008 seconds 2017-06-14 23:08:18 PDT INFO remoteWorkerThread_2: SYNC 5000000066 done in 0.011 seconds 2017-06-14 23:08:24 PDT INFO remoteWorkerThread_3: SYNC 5000000049 done in 0.004 seconds 2017-06-14 23:08:24 PDT INFO remoteWorkerThread_2: SYNC 5000000067 done in 0.017 seconds 2017-06-14 23:08:35 PDT INFO remoteWorkerThread_3: SYNC 5000000050 done in 0.005 seconds 2017-06-14 23:08:39 PDT INFO remoteWorkerThread_2: SYNC 5000000068 done in 0.005 seconds 2017-06-14 23:08:40 PDT INFO remoteWorkerThread_3: SYNC 5000000051 done in 0.004 seconds 2017-06-14 23:08:49 PDT INFO remoteWorkerThread_2: SYNC 5000000069 done in 0.005 seconds 2017-06-14 23:08:53 PDT INFO remoteWorkerThread_2: SYNC 5000000070 done in 0.005 seconds 2017-06-14 23:08:54 PDT INFO remoteWorkerThread_3: SYNC 5000000052 done in 0.005 seconds 2017-06-14 23:09:02 PDT INFO remoteWorkerThread_3: SYNC 5000000053 done in 0.006 seconds 2017-06-14 23:09:07 PDT INFO remoteWorkerThread_2: SYNC 5000000071 done in 0.005 seconds 2017-06-14 23:09:13 PDT INFO remoteWorkerThread_3: SYNC 5000000054 done in 0.005 seconds 2017-06-14 23:09:18 PDT INFO remoteWorkerThread_2: SYNC 5000000072 done in 0.005 seconds 2017-06-14 23:09:24 PDT INFO remoteWorkerThread_3: SYNC 5000000055 done in 0.005 seconds 2017-06-14 23:09:29 PDT INFO remoteWorkerThread_2: SYNC 5000000073 done in 0.005 seconds 2017-06-14 23:09:33 PDT INFO remoteWorkerThread_2: SYNC 5000000074 done in 0.005 seconds 2017-06-14 23:09:34 PDT INFO remoteWorkerThread_3: SYNC 5000000056 done in 0.005 seconds 2017-06-14 23:09:48 PDT INFO remoteWorkerThread_3: SYNC 5000000057 done in 0.004 seconds 2017-06-14 23:09:48 PDT INFO remoteWorkerThread_2: SYNC 5000000075 done in 0.005 seconds 2017-06-14 23:09:52 PDT INFO remoteWorkerThread_3: SYNC 5000000058 done in 0.005 seconds 2017-06-14 23:09:55 PDT INFO remoteWorkerThread_2: SYNC 5000000076 done in 0.004 seconds 2017-06-14 23:10:00 PDT INFO remoteWorkerThread_3: SYNC 5000000059 done in 0.005 seconds 2017-06-14 23:10:06 PDT INFO remoteWorkerThread_2: SYNC 5000000077 done in 0.005 seconds 2017-06-14 23:10:11 PDT INFO remoteWorkerThread_3: SYNC 5000000060 done in 0.005 seconds 2017-06-14 23:10:17 PDT INFO remoteWorkerThread_2: SYNC 5000000078 done in 0.005 seconds 2017-06-14 23:10:22 PDT INFO remoteWorkerThread_3: SYNC 5000000061 done in 0.005 seconds 2017-06-14 23:10:28 PDT INFO remoteWorkerThread_2: SYNC 5000000079 done in 0.005 seconds 2017-06-14 23:10:33 PDT INFO remoteWorkerThread_3: SYNC 5000000062 done in 0.005 seconds 2017-06-14 23:10:37 PDT INFO remoteWorkerThread_2: SYNC 5000000080 done in 0.005 seconds 2017-06-14 23:10:41 PDT INFO remoteWorkerThread_3: SYNC 5000000063 done in 0.005 seconds 2017-06-14 23:10:46 PDT INFO remoteWorkerThread_2: SYNC 5000000081 done in 0.005 seconds 2017-06-14 23:10:52 PDT INFO remoteWorkerThread_3: SYNC 5000000064 done in 0.004 seconds 2017-06-14 23:10:57 PDT INFO remoteWorkerThread_2: SYNC 5000000082 done in 0.004 seconds 2017-06-14 23:11:03 PDT INFO remoteWorkerThread_3: SYNC 5000000065 done in 0.004 seconds 2017-06-14 23:11:08 PDT INFO remoteWorkerThread_2: SYNC 5000000083 done in 0.005 seconds 2017-06-14 23:11:14 PDT INFO remoteWorkerThread_3: SYNC 5000000066 done in 0.004 seconds 2017-06-14 23:11:19 PDT INFO remoteWorkerThread_2: SYNC 5000000084 done in 0.005 seconds 2017-06-14 23:11:23 PDT INFO remoteWorkerThread_2: SYNC 5000000085 done in 0.005 seconds 2017-06-14 23:11:25 PDT INFO remoteWorkerThread_3: SYNC 5000000067 done in 0.005 seconds 2017-06-14 23:11:34 PDT INFO remoteWorkerThread_2: SYNC 5000000086 done in 0.006 seconds 2017-06-14 23:11:38 PDT INFO remoteWorkerThread_3: SYNC 5000000068 done in 0.005 seconds 2017-06-14 23:11:42 PDT INFO remoteWorkerThread_3: SYNC 5000000069 done in 0.005 seconds 2017-06-14 23:11:46 PDT INFO remoteWorkerThread_2: SYNC 5000000087 done in 0.005 seconds 2017-06-14 23:11:54 PDT CONFIG storeListen: li_origin=2 li_receiver=1 li_provider=2 2017-06-14 23:11:54 PDT CONFIG storeListen: li_origin=3 li_receiver=1 li_provider=3 2017-06-14 23:11:54 PDT CONFIG storeListen: li_origin=2 li_receiver=1 li_provider=3 2017-06-14 23:11:54 PDT CONFIG storeListen: li_origin=3 li_receiver=1 li_provider=2 2017-06-14 23:11:54 PDT CONFIG remoteWorkerThread_2: update provider configuration 2017-06-14 23:11:54 PDT CONFIG remoteWorkerThread_2: connection for provider 2 terminated 2017-06-14 23:11:54 PDT CONFIG remoteWorkerThread_2: disconnecting from data provider 2 2017-06-14 23:11:54 PDT CONFIG remoteWorkerThread_2: connection for provider 3 terminated 2017-06-14 23:11:54 PDT CONFIG remoteWorkerThread_2: disconnecting from data provider 3 2017-06-14 23:11:54 PDT CONFIG storeListen: li_origin=2 li_receiver=1 li_provider=2 2017-06-14 23:11:54 PDT CONFIG storeListen: li_origin=3 li_receiver=1 li_provider=3 2017-06-14 23:11:54 PDT CONFIG storeListen: li_origin=2 li_receiver=1 li_provider=3 2017-06-14 23:11:54 PDT CONFIG storeListen: li_origin=3 li_receiver=1 li_provider=2 2017-06-14 23:11:54 PDT CONFIG remoteWorkerThread_2: update provider configuration 2017-06-14 23:11:54 PDT INFO remoteWorkerThread_3: SYNC 5000000070 done in 0.005 seconds 2017-06-14 23:11:54 PDT CONFIG remoteWorkerThread_3: update provider configuration 2017-06-14 23:11:54 PDT CONFIG remoteWorkerThread_3: connection for provider 3 terminated 2017-06-14 23:11:54 PDT CONFIG remoteWorkerThread_3: disconnecting from data provider 3 2017-06-14 23:11:54 PDT CONFIG remoteWorkerThread_3: connection for provider 2 terminated 2017-06-14 23:11:54 PDT CONFIG remoteWorkerThread_3: disconnecting from data provider 2 2017-06-14 23:11:58 PDT CONFIG remoteWorkerThread_2: update provider configuration 2017-06-14 23:11:58 PDT CONFIG version for "host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505 2017-06-14 23:11:58 PDT INFO remoteWorkerThread_2: SYNC 5000000088 done in 0.012 seconds 2017-06-14 23:12:02 PDT CONFIG remoteWorkerThread_3: update provider configuration 2017-06-14 23:12:02 PDT CONFIG version for "host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505 2017-06-14 23:12:02 PDT INFO remoteWorkerThread_3: SYNC 5000000071 done in 0.020 seconds 2017-06-14 23:12:06 PDT CONFIG remoteWorkerThread_2: update provider configuration 2017-06-14 23:12:06 PDT CONFIG version for "host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505 2017-06-14 23:12:07 PDT INFO remoteWorkerThread_2: SYNC 5000000089 done in 0.278 seconds 2017-06-14 23:12:13 PDT INFO remoteWorkerThread_3: SYNC 5000000072 done in 0.003 seconds 2017-06-14 23:12:24 PDT INFO remoteWorkerThread_3: SYNC 5000000073 done in 0.003 seconds 2017-06-14 23:12:33 PDT INFO remoteWorkerThread_2: SYNC 5000000090 done in 0.005 seconds 2017-06-14 23:12:37 PDT CONFIG remoteWorkerThread_3: update provider configuration 2017-06-14 23:12:37 PDT INFO remoteWorkerThread_2: SYNC 5000000091 done in 0.004 seconds 2017-06-14 23:12:37 PDT CONFIG version for "host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505 2017-06-14 23:12:38 PDT INFO remoteWorkerThread_3: SYNC 5000000074 done in 0.522 seconds 2017-06-14 23:12:41 PDT INFO remoteWorkerThread_3: SYNC 5000000075 done in 0.005 seconds 2017-06-14 23:12:52 PDT INFO remoteWorkerThread_3: SYNC 5000000076 done in 0.005 seconds 2017-06-14 23:12:55 PDT INFO remoteWorkerThread_2: SYNC 5000000092 done in 0.005 seconds 2017-06-14 23:13:03 PDT INFO remoteWorkerThread_2: SYNC 5000000093 done in 0.016 seconds 2017-06-14 23:13:09 PDT INFO remoteWorkerThread_3: SYNC 5000000077 done in 0.005 seconds 2017-06-14 23:13:10 PDT INFO remoteWorkerThread_3: SYNC 5000000078 done in 0.005 seconds 2017-06-14 23:13:14 PDT INFO remoteWorkerThread_2: SYNC 5000000094 done in 0.005 seconds 2017-06-14 23:13:21 PDT INFO remoteWorkerThread_3: SYNC 5000000079 done in 0.005 seconds 2017-06-14 23:13:25 PDT INFO remoteWorkerThread_2: SYNC 5000000095 done in 0.004 seconds 2017-06-14 23:13:32 PDT INFO remoteWorkerThread_3: SYNC 5000000080 done in 0.005 seconds 2017-06-14 23:13:36 PDT INFO remoteWorkerThread_2: SYNC 5000000096 done in 0.012 seconds 2017-06-14 23:13:43 PDT INFO remoteWorkerThread_3: SYNC 5000000081 done in 0.005 seconds 2017-06-14 23:13:47 PDT INFO remoteWorkerThread_2: SYNC 5000000097 done in 0.089 seconds 2017-06-14 23:13:54 PDT INFO remoteWorkerThread_3: SYNC 5000000082 done in 0.005 seconds 2017-06-14 23:13:58 PDT INFO remoteWorkerThread_2: SYNC 5000000098 done in 0.006 seconds 2017-06-14 23:14:06 PDT INFO remoteWorkerThread_3: SYNC 5000000083 done in 0.005 seconds 2017-06-14 23:14:10 PDT INFO remoteWorkerThread_3: SYNC 5000000084 done in 0.004 seconds 2017-06-14 23:14:11 PDT INFO remoteWorkerThread_2: SYNC 5000000099 done in 0.005 seconds 2017-06-14 23:14:21 PDT INFO remoteWorkerThread_3: SYNC 5000000085 done in 0.015 seconds 2017-06-14 23:14:21 PDT INFO remoteWorkerThread_2: SYNC 5000000100 done in 0.016 seconds 2017-06-14 23:14:32 PDT INFO remoteWorkerThread_3: SYNC 5000000086 done in 0.009 seconds 2017-06-14 23:14:35 PDT INFO remoteWorkerThread_2: SYNC 5000000101 done in 0.005 seconds NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=2529 CONTEXT: SQL statement "SELECT " _region_il ".cleanupNodelock()" PL /pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=2552 CONTEXT: SQL statement "SELECT " _region_il ".cleanupNodelock()" PL /pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=2567 CONTEXT: SQL statement "SELECT " _region_il ".cleanupNodelock()" PL /pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=2568 CONTEXT: SQL statement "SELECT " _region_il ".cleanupNodelock()" PL /pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM NOTICE: Slony-I: log switch to sl_log_2 complete - truncate sl_log_1 CONTEXT: PL /pgSQL function _region_il.cleanupevent(interval) line 95 at assignment 2017-06-14 23:14:43 PDT INFO cleanupThread: 0.086 seconds for cleanupEvent() 2017-06-14 23:14:49 PDT INFO remoteWorkerThread_3: SYNC 5000000087 done in 0.005 seconds 2017-06-14 23:14:50 PDT INFO remoteWorkerThread_3: SYNC 5000000088 done in 0.005 seconds 2017-06-14 23:14:54 PDT CONFIG storeListen: li_origin=2 li_receiver=1 li_provider=2 2017-06-14 23:14:54 PDT CONFIG storeListen: li_origin=3 li_receiver=1 li_provider=3 2017-06-14 23:14:54 PDT CONFIG storeListen: li_origin=2 li_receiver=1 li_provider=3 2017-06-14 23:14:54 PDT CONFIG storeListen: li_origin=3 li_receiver=1 li_provider=2 2017-06-14 23:14:54 PDT CONFIG remoteWorkerThread_2: update provider configuration 2017-06-14 23:14:54 PDT CONFIG remoteWorkerThread_2: connection for provider 3 terminated 2017-06-14 23:14:54 PDT CONFIG remoteWorkerThread_2: disconnecting from data provider 3 2017-06-14 23:14:54 PDT CONFIG remoteWorkerThread_2: connection for provider 2 terminated 2017-06-14 23:14:54 PDT CONFIG remoteWorkerThread_2: disconnecting from data provider 2 2017-06-14 23:14:54 PDT CONFIG remoteWorkerThread_3: update provider configuration 2017-06-14 23:14:54 PDT CONFIG remoteWorkerThread_3: connection for provider 3 terminated 2017-06-14 23:14:54 PDT CONFIG remoteWorkerThread_3: disconnecting from data provider 3 2017-06-14 23:14:54 PDT CONFIG remoteWorkerThread_3: connection for provider 2 terminated 2017-06-14 23:14:54 PDT CONFIG remoteWorkerThread_3: disconnecting from data provider 2 2017-06-14 23:14:54 PDT CONFIG remoteWorkerThread_2: update provider configuration 2017-06-14 23:14:54 PDT CONFIG storeListen: li_origin=2 li_receiver=1 li_provider=2 2017-06-14 23:14:54 PDT CONFIG storeListen: li_origin=3 li_receiver=1 li_provider=3 2017-06-14 23:14:54 PDT CONFIG storeListen: li_origin=2 li_receiver=1 li_provider=3 2017-06-14 23:14:54 PDT CONFIG storeListen: li_origin=3 li_receiver=1 li_provider=2 2017-06-14 23:14:54 PDT CONFIG remoteWorkerThread_3: update provider configuration 2017-06-14 23:14:54 PDT CONFIG version for "host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505 2017-06-14 23:14:54 PDT INFO remoteWorkerThread_2: SYNC 5000000102 done in 0.037 seconds 2017-06-14 23:14:54 PDT CONFIG remoteWorkerThread_2: update provider configuration 2017-06-14 23:14:54 PDT CONFIG remoteWorkerThread_2: connection for provider 2 terminated 2017-06-14 23:14:54 PDT CONFIG remoteWorkerThread_2: disconnecting from data provider 2 2017-06-14 23:15:02 PDT CONFIG remoteWorkerThread_2: update provider configuration 2017-06-14 23:15:02 PDT CONFIG version for "host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505 2017-06-14 23:15:02 PDT INFO remoteWorkerThread_2: SYNC 5000000103 done in 0.013 seconds 2017-06-14 23:15:06 PDT CONFIG remoteWorkerThread_3: update provider configuration 2017-06-14 23:15:06 PDT CONFIG version for "host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505 2017-06-14 23:15:06 PDT INFO remoteWorkerThread_3: SYNC 5000000089 done in 0.013 seconds 2017-06-14 23:15:14 PDT CONFIG remoteWorkerThread_3: update provider configuration 2017-06-14 23:15:14 PDT CONFIG version for "host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505 2017-06-14 23:15:17 PDT INFO remoteWorkerThread_3: SYNC 5000000090 done in 2.438 seconds 2017-06-14 23:15:20 PDT INFO remoteWorkerThread_2: SYNC 5000000104 done in 0.003 seconds 2017-06-14 23:15:21 PDT INFO remoteWorkerThread_2: SYNC 5000000105 done in 0.002 seconds 2017-06-14 23:15:25 PDT INFO remoteWorkerThread_3: SYNC 5000000091 done in 0.016 seconds 2017-06-14 23:15:32 PDT INFO remoteWorkerThread_2: SYNC 5000000106 done in 0.002 seconds 2017-06-14 23:15:36 PDT INFO remoteWorkerThread_3: SYNC 5000000092 done in 0.005 seconds 2017-06-14 23:15:43 PDT INFO remoteWorkerThread_2: SYNC 5000000107 done in 0.003 seconds 2017-06-14 23:15:47 PDT INFO remoteWorkerThread_3: SYNC 5000000093 done in 0.006 seconds 2017-06-14 23:15:55 PDT INFO remoteWorkerThread_2: SYNC 5000000108 done in 0.003 seconds 2017-06-14 23:15:55 PDT INFO remoteWorkerThread_3: SYNC 5000000094 done in 0.013 seconds 2017-06-14 23:16:06 PDT INFO remoteWorkerThread_2: SYNC 5000000109 done in 0.003 seconds 2017-06-14 23:16:10 PDT INFO remoteWorkerThread_3: SYNC 5000000095 done in 0.004 seconds 2017-06-14 23:16:11 PDT CONFIG remoteWorkerThread_2: update provider configuration 2017-06-14 23:16:11 PDT CONFIG version for "host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505 2017-06-14 23:16:11 PDT INFO remoteWorkerThread_2: SYNC 5000000110 done in 0.058 seconds 2017-06-14 23:16:12 PDT INFO remoteWorkerThread_3: SYNC 5000000096 done in 0.012 seconds 2017-06-14 23:16:27 PDT INFO remoteWorkerThread_2: SYNC 5000000111 done in 0.008 seconds 2017-06-14 23:16:31 PDT INFO remoteWorkerThread_2: SYNC 5000000112 done in 0.013 seconds 2017-06-14 23:16:31 PDT INFO remoteWorkerThread_3: SYNC 5000000097 done in 0.037 seconds 2017-06-14 23:16:42 PDT INFO remoteWorkerThread_3: SYNC 5000000098 done in 0.006 seconds 2017-06-14 23:16:42 PDT INFO remoteWorkerThread_2: SYNC 5000000113 done in 0.006 seconds 2017-06-14 23:16:53 PDT INFO remoteWorkerThread_3: SYNC 5000000099 done in 0.005 seconds 2017-06-14 23:16:53 PDT INFO remoteWorkerThread_2: SYNC 5000000114 done in 0.005 seconds 2017-06-14 23:17:04 PDT INFO remoteWorkerThread_3: SYNC 5000000100 done in 0.005 seconds 2017-06-14 23:17:04 PDT INFO remoteWorkerThread_2: SYNC 5000000115 done in 0.006 seconds 2017-06-14 23:17:08 PDT INFO remoteWorkerThread_3: SYNC 5000000101 done in 0.005 seconds 2017-06-14 23:17:14 PDT INFO remoteWorkerThread_2: SYNC 5000000116 done in 0.004 seconds 2017-06-14 23:17:19 PDT INFO remoteWorkerThread_3: SYNC 5000000102 done in 0.005 seconds 2017-06-14 23:17:25 PDT INFO remoteWorkerThread_2: SYNC 5000000117 done in 0.005 seconds 2017-06-14 23:17:30 PDT INFO remoteWorkerThread_3: SYNC 5000000103 done in 0.005 seconds 2017-06-14 23:17:34 PDT INFO remoteWorkerThread_2: SYNC 5000000118 done in 0.013 seconds 2017-06-14 23:17:38 PDT INFO remoteWorkerThread_3: SYNC 5000000104 done in 0.012 seconds 2017-06-14 23:17:43 PDT INFO remoteWorkerThread_2: SYNC 5000000119 done in 0.005 seconds 2017-06-14 23:17:49 PDT INFO remoteWorkerThread_3: SYNC 5000000105 done in 0.004 seconds 2017-06-14 23:17:53 PDT INFO remoteWorkerThread_2: SYNC 5000000120 done in 0.005 seconds 2017-06-14 23:18:01 PDT INFO remoteWorkerThread_2: SYNC 5000000121 done in 0.005 seconds 2017-06-14 23:18:01 PDT INFO remoteWorkerThread_3: SYNC 5000000106 done in 0.006 seconds 2017-06-14 23:18:14 PDT INFO remoteWorkerThread_2: SYNC 5000000122 done in 0.005 seconds 2017-06-14 23:18:14 PDT INFO remoteWorkerThread_3: SYNC 5000000107 done in 0.006 seconds 2017-06-14 23:18:27 PDT INFO remoteWorkerThread_2: SYNC 5000000123 done in 0.005 seconds 2017-06-14 23:18:27 PDT INFO remoteWorkerThread_3: SYNC 5000000108 done in 0.006 seconds 2017-06-14 23:18:30 PDT INFO remoteWorkerThread_3: SYNC 5000000109 done in 0.005 seconds 2017-06-14 23:18:33 PDT INFO remoteWorkerThread_2: SYNC 5000000124 done in 0.004 seconds 2017-06-14 23:18:38 PDT INFO remoteWorkerThread_3: SYNC 5000000110 done in 0.004 seconds 2017-06-14 23:18:44 PDT INFO remoteWorkerThread_2: SYNC 5000000125 done in 0.004 seconds 2017-06-14 23:18:49 PDT INFO remoteWorkerThread_3: SYNC 5000000111 done in 0.005 seconds 2017-06-14 23:18:55 PDT INFO remoteWorkerThread_2: SYNC 5000000126 done in 0.005 seconds 2017-06-14 23:19:00 PDT INFO remoteWorkerThread_3: SYNC 5000000112 done in 0.011 seconds 2017-06-14 23:19:04 PDT INFO remoteWorkerThread_2: SYNC 5000000127 done in 0.005 seconds 2017-06-14 23:19:08 PDT INFO remoteWorkerThread_3: SYNC 5000000113 done in 0.005 seconds 2017-06-14 23:19:12 PDT INFO remoteWorkerThread_2: SYNC 5000000128 done in 0.005 seconds 2017-06-14 23:19:20 PDT INFO remoteWorkerThread_3: SYNC 5000000114 done in 0.004 seconds 2017-06-14 23:19:21 PDT INFO remoteWorkerThread_2: SYNC 5000000129 done in 0.004 seconds 2017-06-14 23:19:34 PDT INFO remoteWorkerThread_3: SYNC 5000000115 done in 0.005 seconds 2017-06-14 23:19:34 PDT INFO remoteWorkerThread_2: SYNC 5000000130 done in 0.006 seconds 2017-06-14 23:19:47 PDT INFO remoteWorkerThread_2: SYNC 5000000131 done in 0.005 seconds 2017-06-14 23:19:47 PDT INFO remoteWorkerThread_3: SYNC 5000000116 done in 0.005 seconds 2017-06-14 23:19:48 PDT INFO remoteWorkerThread_3: SYNC 5000000117 done in 0.005 seconds 2017-06-14 23:19:51 PDT INFO remoteWorkerThread_2: SYNC 5000000132 done in 0.005 seconds 2017-06-14 23:20:02 PDT INFO remoteWorkerThread_3: SYNC 5000000118 done in 0.005 seconds 2017-06-14 23:20:02 PDT INFO remoteWorkerThread_2: SYNC 5000000133 done in 0.006 seconds 2017-06-14 23:20:16 PDT INFO remoteWorkerThread_3: SYNC 5000000119 done in 0.005 seconds 2017-06-14 23:20:16 PDT INFO remoteWorkerThread_2: SYNC 5000000134 done in 0.006 seconds 2017-06-14 23:20:19 PDT INFO remoteWorkerThread_3: SYNC 5000000120 done in 0.005 seconds 2017-06-14 23:20:22 PDT INFO remoteWorkerThread_2: SYNC 5000000135 done in 0.005 seconds 2017-06-14 23:20:35 PDT INFO remoteWorkerThread_3: SYNC 5000000121 done in 0.005 seconds 2017-06-14 23:20:35 PDT INFO remoteWorkerThread_2: SYNC 5000000136 done in 0.006 seconds 2017-06-14 23:20:38 PDT INFO remoteWorkerThread_3: SYNC 5000000122 done in 0.005 seconds 2017-06-14 23:20:41 PDT INFO remoteWorkerThread_2: SYNC 5000000137 done in 0.005 seconds |
辅节点1查看Slony-I日志信息
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 |
$ cat /usr/local/pgsql/data/pg_log/node2/region_il-2017-06-14 .log 2017-06-14 22:57:41 PDT CONFIG main: slon version 2.2.4 starting up 2017-06-14 22:57:41 PDT INFO slon: watchdog process started 2017-06-14 22:57:41 PDT CONFIG slon: watchdog ready - pid = 15535 2017-06-14 22:57:41 PDT CONFIG slon: worker process created - pid = 15536 2017-06-14 22:57:41 PDT CONFIG main: Integer option vac_frequency = 3 2017-06-14 22:57:41 PDT CONFIG main: Integer option log_level = 0 2017-06-14 22:57:41 PDT CONFIG main: Integer option sync_interval = 1000 2017-06-14 22:57:41 PDT CONFIG main: Integer option sync_interval_timeout = 10000 2017-06-14 22:57:41 PDT CONFIG main: Integer option sync_group_maxsize = 20 2017-06-14 22:57:41 PDT CONFIG main: Integer option syslog = 0 2017-06-14 22:57:41 PDT CONFIG main: Integer option quit_sync_provider = 0 2017-06-14 22:57:41 PDT CONFIG main: Integer option remote_listen_timeout = 300 2017-06-14 22:57:41 PDT CONFIG main: Integer option monitor_interval = 500 2017-06-14 22:57:41 PDT CONFIG main: Integer option explain_interval = 0 2017-06-14 22:57:41 PDT CONFIG main: Integer option tcp_keepalive_idle = 0 2017-06-14 22:57:41 PDT CONFIG main: Integer option tcp_keepalive_interval = 0 2017-06-14 22:57:41 PDT CONFIG main: Integer option tcp_keepalive_count = 0 2017-06-14 22:57:41 PDT CONFIG main: Integer option apply_cache_size = 100 2017-06-14 22:57:41 PDT CONFIG main: Boolean option log_pid = 0 2017-06-14 22:57:41 PDT CONFIG main: Boolean option log_timestamp = 1 2017-06-14 22:57:41 PDT CONFIG main: Boolean option tcp_keepalive = 1 2017-06-14 22:57:41 PDT CONFIG main: Boolean option monitor_threads = 1 2017-06-14 22:57:41 PDT CONFIG main: Real option real_placeholder = 0.000000 2017-06-14 22:57:41 PDT CONFIG main: String option cluster_name = region_il 2017-06-14 22:57:41 PDT CONFIG main: String option conn_info = host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888 2017-06-14 22:57:41 PDT CONFIG main: String option pid_file = /usr/local/pgsql/data/region_il_node2 .pid 2017-06-14 22:57:41 PDT CONFIG main: String option log_timestamp_format = %Y-%m-%d %H:%M:%S %Z 2017-06-14 22:57:41 PDT CONFIG main: String option archive_dir = [NULL] 2017-06-14 22:57:41 PDT CONFIG main: String option sql_on_connection = [NULL] 2017-06-14 22:57:41 PDT CONFIG main: String option lag_interval = [NULL] 2017-06-14 22:57:41 PDT CONFIG main: String option command_on_logarchive = [NULL] 2017-06-14 22:57:41 PDT CONFIG main: String option syslog_facility = LOCAL0 2017-06-14 22:57:41 PDT CONFIG main: String option syslog_ident = slon 2017-06-14 22:57:41 PDT CONFIG main: String option cleanup_interval = 10 minutes 2017-06-14 22:57:41 PDT CONFIG main: local node id = 2 2017-06-14 22:57:41 PDT INFO main: main process started 2017-06-14 22:57:41 PDT CONFIG main: launching sched_start_mainloop 2017-06-14 22:57:41 PDT CONFIG main: loading current cluster configuration 2017-06-14 22:57:41 PDT CONFIG storeNode: no_id=1 no_comment= 'Node 1 - region_il@10.189.102.118' 2017-06-14 22:57:41 PDT CONFIG storeNode: no_id=3 no_comment= '<event pending>' 2017-06-14 22:57:41 PDT CONFIG storePath: pa_server=1 pa_client=2 pa_conninfo= "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" pa_connretry=10 2017-06-14 22:57:41 PDT CONFIG storePath: pa_server=3 pa_client=2 pa_conninfo= "host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888" pa_connretry=10 2017-06-14 22:57:41 PDT CONFIG storeListen: li_origin=1 li_receiver=2 li_provider=1 2017-06-14 22:57:41 PDT CONFIG storeListen: li_origin=3 li_receiver=2 li_provider=3 2017-06-14 22:57:41 PDT CONFIG main: last local event sequence = 5000000002 2017-06-14 22:57:41 PDT CONFIG main: configuration complete - starting threads 2017-06-14 22:57:41 PDT INFO localListenThread: thread starts 2017-06-14 22:57:41 PDT CONFIG version for "host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505 2017-06-14 22:57:41 PDT CONFIG enableNode: no_id=1 2017-06-14 22:57:41 PDT INFO main: running scheduler mainloop 2017-06-14 22:57:41 PDT CONFIG cleanupThread: thread starts 2017-06-14 22:57:41 PDT INFO syncThread: thread starts 2017-06-14 22:57:41 PDT INFO monitorThread: thread starts 2017-06-14 22:57:41 PDT INFO remoteListenThread_1: thread starts 2017-06-14 22:57:41 PDT INFO remoteWorkerThread_1: thread starts 2017-06-14 22:57:41 PDT CONFIG version for "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505 2017-06-14 22:57:41 PDT CONFIG version for "host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505 2017-06-14 22:57:41 PDT CONFIG version for "host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505 2017-06-14 22:57:41 PDT CONFIG version for "host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505 2017-06-14 22:57:41 PDT CONFIG cleanupThread: bias = 60 2017-06-14 22:57:41 PDT CONFIG remoteWorkerThread_1: update provider configuration 2017-06-14 22:57:41 PDT CONFIG version for "host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505 2017-06-14 22:57:41 PDT CONFIG storeListen: li_origin=1 li_receiver=2 li_provider=1 2017-06-14 22:57:41 PDT CONFIG storeListen: li_origin=3 li_receiver=2 li_provider=3 2017-06-14 22:57:41 PDT CONFIG remoteWorkerThread_1: update provider configuration 2017-06-14 22:57:41 PDT CONFIG storeListen: li_origin=1 li_receiver=2 li_provider=1 2017-06-14 22:57:41 PDT CONFIG storeListen: li_origin=3 li_receiver=2 li_provider=3 2017-06-14 22:57:41 PDT CONFIG remoteWorkerThread_1: update provider configuration 2017-06-14 22:57:41 PDT CONFIG storeNode: no_id=3 no_comment= 'Node 3 - region_il@10.189.100.226' - update node 2017-06-14 22:57:41 PDT CONFIG storeListen: li_origin=1 li_receiver=2 li_provider=1 2017-06-14 22:57:41 PDT CONFIG storeListen: li_origin=3 li_receiver=2 li_provider=3 2017-06-14 22:57:41 PDT CONFIG remoteWorkerThread_1: update provider configuration 2017-06-14 22:57:41 PDT CONFIG enableNode: no_id=3 2017-06-14 22:57:41 PDT INFO remoteListenThread_3: thread starts 2017-06-14 22:57:41 PDT INFO remoteWorkerThread_3: thread starts 2017-06-14 22:57:41 PDT CONFIG storeListen: li_origin=1 li_receiver=2 li_provider=1 2017-06-14 22:57:41 PDT CONFIG storeListen: li_origin=3 li_receiver=2 li_provider=3 2017-06-14 22:57:41 PDT CONFIG remoteWorkerThread_1: update provider configuration 2017-06-14 22:57:41 PDT CONFIG version for "host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505 2017-06-14 22:57:41 PDT CONFIG version for "host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505 2017-06-14 22:57:41 PDT CONFIG remoteWorkerThread_3: update provider configuration 2017-06-14 22:57:41 PDT CONFIG storeListen: li_origin=1 li_receiver=2 li_provider=1 2017-06-14 22:57:41 PDT CONFIG storeListen: li_origin=3 li_receiver=2 li_provider=3 2017-06-14 22:57:41 PDT CONFIG storeListen: li_origin=3 li_receiver=2 li_provider=1 2017-06-14 22:57:41 PDT CONFIG remoteWorkerThread_1: update provider configuration 2017-06-14 22:57:41 PDT CONFIG remoteWorkerThread_3: update provider configuration 2017-06-14 22:57:41 PDT CONFIG storeListen: li_origin=1 li_receiver=2 li_provider=1 2017-06-14 22:57:41 PDT CONFIG storeListen: li_origin=3 li_receiver=2 li_provider=3 2017-06-14 22:57:41 PDT CONFIG storeListen: li_origin=3 li_receiver=2 li_provider=1 2017-06-14 22:57:41 PDT CONFIG remoteWorkerThread_1: update provider configuration 2017-06-14 22:57:41 PDT CONFIG remoteWorkerThread_3: update provider configuration 2017-06-14 22:57:41 PDT CONFIG remoteWorkerThread_1: update provider configuration 2017-06-14 22:57:41 PDT CONFIG version for "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505 2017-06-14 22:57:41 PDT INFO remoteWorkerThread_1: SYNC 5000000008 done in 0.015 seconds 2017-06-14 22:57:41 PDT CONFIG storeListen: li_origin=1 li_receiver=2 li_provider=1 2017-06-14 22:57:41 PDT CONFIG storeListen: li_origin=3 li_receiver=2 li_provider=3 2017-06-14 22:57:41 PDT CONFIG storeListen: li_origin=3 li_receiver=2 li_provider=1 2017-06-14 22:57:41 PDT CONFIG storeListen: li_origin=1 li_receiver=2 li_provider=3 2017-06-14 22:57:41 PDT CONFIG remoteWorkerThread_3: update provider configuration 2017-06-14 22:57:41 PDT INFO remoteWorkerThread_1: SYNC 5000000010 done in 0.002 seconds 2017-06-14 22:57:41 PDT CONFIG remoteWorkerThread_1: update provider configuration 2017-06-14 22:57:41 PDT CONFIG remoteWorkerThread_1: connection for provider 1 terminated 2017-06-14 22:57:41 PDT CONFIG remoteWorkerThread_1: disconnecting from data provider 1 2017-06-14 22:57:41 PDT CONFIG remoteWorkerThread_1: update provider configuration 2017-06-14 22:57:41 PDT CONFIG storeListen: li_origin=1 li_receiver=2 li_provider=1 2017-06-14 22:57:41 PDT CONFIG storeListen: li_origin=3 li_receiver=2 li_provider=3 2017-06-14 22:57:41 PDT CONFIG storeListen: li_origin=3 li_receiver=2 li_provider=1 2017-06-14 22:57:41 PDT CONFIG storeListen: li_origin=1 li_receiver=2 li_provider=3 2017-06-14 22:57:41 PDT CONFIG remoteWorkerThread_3: update provider configuration 2017-06-14 22:57:41 PDT CONFIG version for "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505 2017-06-14 22:57:41 PDT INFO remoteWorkerThread_1: SYNC 5000000014 done in 0.011 seconds 2017-06-14 22:57:41 PDT CONFIG remoteWorkerThread_1: update provider configuration 2017-06-14 22:57:41 PDT CONFIG remoteWorkerThread_1: connection for provider 1 terminated 2017-06-14 22:57:41 PDT CONFIG remoteWorkerThread_1: disconnecting from data provider 1 2017-06-14 22:57:41 PDT CONFIG remoteWorkerThread_1: update provider configuration 2017-06-14 22:57:41 PDT CONFIG version for "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505 2017-06-14 22:57:41 PDT INFO remoteWorkerThread_1: SYNC 5000000022 done in 0.012 seconds 2017-06-14 22:57:41 PDT INFO remoteWorkerThread_1: SYNC 5000000025 done in 0.002 seconds 2017-06-14 22:57:45 PDT INFO remoteWorkerThread_1: SYNC 5000000026 done in 0.003 seconds 2017-06-14 22:57:56 PDT INFO remoteWorkerThread_1: SYNC 5000000027 done in 0.003 seconds 2017-06-14 22:58:10 PDT INFO remoteWorkerThread_1: SYNC 5000000028 done in 0.003 seconds 2017-06-14 22:58:14 PDT INFO remoteWorkerThread_1: SYNC 5000000029 done in 0.003 seconds 2017-06-14 22:58:28 PDT INFO remoteWorkerThread_1: SYNC 5000000030 done in 0.003 seconds 2017-06-14 22:58:39 PDT INFO remoteWorkerThread_1: SYNC 5000000031 done in 0.003 seconds 2017-06-14 22:58:43 PDT INFO remoteWorkerThread_1: SYNC 5000000032 done in 0.003 seconds 2017-06-14 22:59:01 PDT INFO remoteWorkerThread_1: SYNC 5000000033 done in 0.003 seconds 2017-06-14 22:59:05 PDT INFO remoteWorkerThread_1: SYNC 5000000034 done in 0.003 seconds 2017-06-14 22:59:19 PDT INFO remoteWorkerThread_1: SYNC 5000000035 done in 0.003 seconds 2017-06-14 22:59:23 PDT INFO remoteWorkerThread_1: SYNC 5000000036 done in 0.002 seconds 2017-06-14 22:59:41 PDT INFO remoteWorkerThread_1: SYNC 5000000037 done in 0.003 seconds 2017-06-14 22:59:45 PDT INFO remoteWorkerThread_1: SYNC 5000000038 done in 0.002 seconds 2017-06-14 22:59:59 PDT INFO remoteWorkerThread_1: SYNC 5000000039 done in 0.003 seconds 2017-06-14 23:00:03 PDT INFO remoteWorkerThread_1: SYNC 5000000040 done in 0.003 seconds 2017-06-14 23:00:21 PDT INFO remoteWorkerThread_1: SYNC 5000000041 done in 0.003 seconds 2017-06-14 23:00:25 PDT INFO remoteWorkerThread_1: SYNC 5000000042 done in 0.002 seconds 2017-06-14 23:00:39 PDT INFO remoteWorkerThread_1: SYNC 5000000043 done in 0.003 seconds 2017-06-14 23:00:43 PDT CONFIG remoteWorkerThread_3: update provider configuration 2017-06-14 23:00:43 PDT INFO remoteWorkerThread_1: SYNC 5000000044 done in 0.003 seconds 2017-06-14 23:00:43 PDT CONFIG version for "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505 2017-06-14 23:00:43 PDT INFO remoteWorkerThread_3: SYNC 5000000003 done in 0.014 seconds 2017-06-14 23:00:56 PDT CONFIG remoteWorkerThread_3: update provider configuration 2017-06-14 23:00:56 PDT CONFIG version for "host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505 2017-06-14 23:00:56 PDT INFO remoteWorkerThread_3: SYNC 5000000004 done in 0.022 seconds 2017-06-14 23:01:00 PDT INFO remoteWorkerThread_3: SYNC 5000000005 done in 0.013 seconds 2017-06-14 23:01:01 PDT INFO remoteWorkerThread_1: SYNC 5000000045 done in 0.003 seconds 2017-06-14 23:01:04 PDT CONFIG remoteWorkerThread_1: update provider configuration 2017-06-14 23:01:04 PDT CONFIG version for "host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505 2017-06-14 23:01:04 PDT INFO remoteWorkerThread_1: SYNC 5000000046 done in 0.014 seconds 2017-06-14 23:01:15 PDT INFO remoteWorkerThread_3: SYNC 5000000006 done in 0.005 seconds 2017-06-14 23:01:15 PDT INFO remoteWorkerThread_1: SYNC 5000000047 done in 0.006 seconds 2017-06-14 23:01:19 PDT INFO remoteWorkerThread_3: SYNC 5000000007 done in 0.004 seconds 2017-06-14 23:01:26 PDT INFO remoteWorkerThread_1: SYNC 5000000048 done in 0.005 seconds 2017-06-14 23:01:30 PDT INFO remoteWorkerThread_3: SYNC 5000000008 done in 0.005 seconds 2017-06-14 23:01:37 PDT INFO remoteWorkerThread_1: SYNC 5000000049 done in 0.005 seconds 2017-06-14 23:01:41 PDT INFO remoteWorkerThread_3: SYNC 5000000009 done in 0.005 seconds 2017-06-14 23:01:49 PDT INFO remoteWorkerThread_1: SYNC 5000000050 done in 0.013 seconds 2017-06-14 23:01:53 PDT INFO remoteWorkerThread_3: SYNC 5000000010 done in 0.005 seconds 2017-06-14 23:01:53 PDT INFO remoteWorkerThread_1: SYNC 5000000051 done in 0.006 seconds 2017-06-14 23:02:01 PDT INFO remoteWorkerThread_3: SYNC 5000000011 done in 0.006 seconds 2017-06-14 23:02:07 PDT INFO remoteWorkerThread_1: SYNC 5000000052 done in 0.005 seconds 2017-06-14 23:02:11 PDT INFO remoteWorkerThread_3: SYNC 5000000012 done in 0.005 seconds 2017-06-14 23:02:15 PDT INFO remoteWorkerThread_1: SYNC 5000000053 done in 0.005 seconds 2017-06-14 23:02:19 PDT INFO remoteWorkerThread_3: SYNC 5000000013 done in 0.005 seconds 2017-06-14 23:02:26 PDT INFO remoteWorkerThread_1: SYNC 5000000054 done in 0.005 seconds 2017-06-14 23:02:30 PDT INFO remoteWorkerThread_3: SYNC 5000000014 done in 0.005 seconds 2017-06-14 23:02:37 PDT INFO remoteWorkerThread_1: SYNC 5000000055 done in 0.005 seconds 2017-06-14 23:02:41 PDT INFO remoteWorkerThread_3: SYNC 5000000015 done in 0.005 seconds 2017-06-14 23:02:45 PDT INFO remoteWorkerThread_1: SYNC 5000000056 done in 0.006 seconds 2017-06-14 23:02:49 PDT INFO remoteWorkerThread_3: SYNC 5000000016 done in 0.006 seconds 2017-06-14 23:02:55 PDT INFO remoteWorkerThread_1: SYNC 5000000057 done in 0.005 seconds 2017-06-14 23:03:00 PDT INFO remoteWorkerThread_3: SYNC 5000000017 done in 0.005 seconds 2017-06-14 23:03:06 PDT INFO remoteWorkerThread_1: SYNC 5000000058 done in 0.009 seconds 2017-06-14 23:03:11 PDT INFO remoteWorkerThread_3: SYNC 5000000018 done in 0.004 seconds 2017-06-14 23:03:17 PDT CONFIG storeSet: set_id=1 set_origin=1 set_comment= 'Set 1 (il_longlive_schema) for region_il' 2017-06-14 23:03:17 PDT CONFIG remoteWorkerThread_1: update provider configuration 2017-06-14 23:03:17 PDT CONFIG remoteWorkerThread_1: connection for provider 1 terminated 2017-06-14 23:03:17 PDT CONFIG remoteWorkerThread_1: disconnecting from data provider 1 2017-06-14 23:03:17 PDT CONFIG remoteWorkerThread_1: connection for provider 3 terminated 2017-06-14 23:03:17 PDT CONFIG remoteWorkerThread_1: disconnecting from data provider 3 2017-06-14 23:03:17 PDT CONFIG remoteWorkerThread_1: update provider configuration 2017-06-14 23:03:17 PDT CONFIG version for "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505 2017-06-14 23:03:17 PDT INFO remoteWorkerThread_1: SYNC 5000000060 done in 0.014 seconds 2017-06-14 23:03:17 PDT INFO remoteWorkerThread_1: SYNC 5000000062 done in 0.002 seconds 2017-06-14 23:03:17 PDT INFO remoteWorkerThread_1: SYNC 5000000064 done in 0.002 seconds 2017-06-14 23:03:17 PDT INFO remoteWorkerThread_1: SYNC 5000000066 done in 0.002 seconds 2017-06-14 23:03:17 PDT INFO remoteWorkerThread_1: SYNC 5000000068 done in 0.002 seconds 2017-06-14 23:03:17 PDT INFO remoteWorkerThread_1: SYNC 5000000070 done in 0.002 seconds 2017-06-14 23:03:17 PDT INFO remoteWorkerThread_1: SYNC 5000000072 done in 0.002 seconds 2017-06-14 23:03:17 PDT INFO remoteWorkerThread_1: SYNC 5000000074 done in 0.002 seconds 2017-06-14 23:03:17 PDT INFO remoteWorkerThread_1: SYNC 5000000076 done in 0.002 seconds 2017-06-14 23:03:21 PDT INFO remoteWorkerThread_3: SYNC 5000000019 done in 0.005 seconds 2017-06-14 23:03:25 PDT INFO remoteWorkerThread_1: SYNC 5000000077 done in 0.002 seconds 2017-06-14 23:03:29 PDT INFO remoteWorkerThread_3: SYNC 5000000020 done in 0.005 seconds 2017-06-14 23:03:36 PDT INFO remoteWorkerThread_1: SYNC 5000000078 done in 0.003 seconds 2017-06-14 23:03:40 PDT INFO remoteWorkerThread_3: SYNC 5000000021 done in 0.005 seconds 2017-06-14 23:03:44 PDT CONFIG remoteWorkerThread_1: update provider configuration 2017-06-14 23:03:44 PDT CONFIG version for "host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505 2017-06-14 23:03:44 PDT INFO remoteWorkerThread_1: SYNC 5000000079 done in 0.015 seconds 2017-06-14 23:03:54 PDT INFO remoteWorkerThread_1: SYNC 5000000080 done in 0.005 seconds 2017-06-14 23:03:58 PDT INFO remoteWorkerThread_3: SYNC 5000000022 done in 0.005 seconds 2017-06-14 23:04:01 PDT INFO remoteWorkerThread_3: SYNC 5000000023 done in 0.005 seconds 2017-06-14 23:04:06 PDT INFO remoteWorkerThread_1: SYNC 5000000081 done in 0.005 seconds 2017-06-14 23:04:12 PDT INFO remoteWorkerThread_3: SYNC 5000000024 done in 0.005 seconds 2017-06-14 23:04:16 PDT INFO remoteWorkerThread_1: SYNC 5000000082 done in 0.005 seconds 2017-06-14 23:04:20 PDT INFO remoteWorkerThread_3: SYNC 5000000025 done in 0.005 seconds 2017-06-14 23:04:24 PDT INFO remoteWorkerThread_1: SYNC 5000000083 done in 0.005 seconds 2017-06-14 23:04:32 PDT INFO remoteWorkerThread_3: SYNC 5000000026 done in 0.005 seconds 2017-06-14 23:04:33 PDT INFO remoteWorkerThread_1: SYNC 5000000084 done in 0.004 seconds 2017-06-14 23:04:44 PDT INFO remoteWorkerThread_3: SYNC 5000000027 done in 0.005 seconds 2017-06-14 23:04:44 PDT INFO remoteWorkerThread_1: SYNC 5000000085 done in 0.006 seconds 2017-06-14 23:04:54 PDT INFO remoteWorkerThread_3: SYNC 5000000028 done in 0.005 seconds 2017-06-14 23:04:58 PDT INFO remoteWorkerThread_1: SYNC 5000000086 done in 0.005 seconds 2017-06-14 23:05:01 PDT INFO remoteWorkerThread_3: SYNC 5000000029 done in 0.006 seconds 2017-06-14 23:05:05 PDT INFO remoteWorkerThread_1: SYNC 5000000087 done in 0.005 seconds 2017-06-14 23:05:13 PDT INFO remoteWorkerThread_3: SYNC 5000000030 done in 0.005 seconds 2017-06-14 23:05:14 PDT INFO remoteWorkerThread_1: SYNC 5000000088 done in 0.005 seconds 2017-06-14 23:05:25 PDT INFO remoteWorkerThread_3: SYNC 5000000031 done in 0.005 seconds 2017-06-14 23:05:25 PDT INFO remoteWorkerThread_1: SYNC 5000000089 done in 0.006 seconds 2017-06-14 23:05:29 PDT INFO remoteWorkerThread_3: SYNC 5000000032 done in 0.005 seconds 2017-06-14 23:05:36 PDT INFO remoteWorkerThread_1: SYNC 5000000090 done in 0.006 seconds 2017-06-14 23:05:40 PDT INFO remoteWorkerThread_3: SYNC 5000000033 done in 0.004 seconds 2017-06-14 23:05:47 PDT INFO remoteWorkerThread_1: SYNC 5000000091 done in 0.005 seconds 2017-06-14 23:05:51 PDT INFO remoteWorkerThread_3: SYNC 5000000034 done in 0.004 seconds 2017-06-14 23:05:58 PDT INFO remoteWorkerThread_1: SYNC 5000000092 done in 0.005 seconds 2017-06-14 23:06:02 PDT INFO remoteWorkerThread_3: SYNC 5000000035 done in 0.005 seconds 2017-06-14 23:06:09 PDT INFO remoteWorkerThread_1: SYNC 5000000093 done in 0.005 seconds 2017-06-14 23:06:13 PDT INFO remoteWorkerThread_1: SYNC 5000000094 done in 0.005 seconds 2017-06-14 23:06:13 PDT INFO remoteWorkerThread_3: SYNC 5000000036 done in 0.004 seconds 2017-06-14 23:06:21 PDT INFO remoteWorkerThread_3: SYNC 5000000037 done in 0.004 seconds 2017-06-14 23:06:27 PDT INFO remoteWorkerThread_1: SYNC 5000000095 done in 0.004 seconds 2017-06-14 23:06:31 PDT INFO remoteWorkerThread_3: SYNC 5000000038 done in 0.005 seconds 2017-06-14 23:06:35 PDT INFO remoteWorkerThread_1: SYNC 5000000096 done in 0.005 seconds 2017-06-14 23:06:39 PDT INFO remoteWorkerThread_3: SYNC 5000000039 done in 0.004 seconds 2017-06-14 23:06:46 PDT INFO remoteWorkerThread_1: SYNC 5000000097 done in 0.005 seconds 2017-06-14 23:06:50 PDT INFO remoteWorkerThread_3: SYNC 5000000040 done in 0.004 seconds 2017-06-14 23:06:57 PDT INFO remoteWorkerThread_1: SYNC 5000000098 done in 0.005 seconds 2017-06-14 23:07:01 PDT INFO remoteWorkerThread_3: SYNC 5000000041 done in 0.005 seconds 2017-06-14 23:07:08 PDT INFO remoteWorkerThread_1: SYNC 5000000099 done in 0.004 seconds 2017-06-14 23:07:12 PDT INFO remoteWorkerThread_3: SYNC 5000000042 done in 0.005 seconds 2017-06-14 23:07:13 PDT INFO remoteWorkerThread_1: SYNC 5000000100 done in 0.004 seconds 2017-06-14 23:07:22 PDT INFO remoteWorkerThread_3: SYNC 5000000043 done in 0.005 seconds 2017-06-14 23:07:27 PDT INFO remoteWorkerThread_1: SYNC 5000000101 done in 0.030 seconds 2017-06-14 23:07:31 PDT INFO remoteWorkerThread_3: SYNC 5000000044 done in 0.005 seconds 2017-06-14 23:07:35 PDT INFO remoteWorkerThread_1: SYNC 5000000102 done in 0.004 seconds 2017-06-14 23:07:40 PDT INFO remoteWorkerThread_3: SYNC 5000000045 done in 0.009 seconds NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=15654 CONTEXT: SQL statement "SELECT " _region_il ".cleanupNodelock()" PL /pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=15655 CONTEXT: SQL statement "SELECT " _region_il ".cleanupNodelock()" PL /pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=15658 CONTEXT: SQL statement "SELECT " _region_il ".cleanupNodelock()" PL /pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM NOTICE: Slony-I: Logswitch to sl_log_2 initiated CONTEXT: SQL statement "SELECT " _region_il ".logswitch_start()" PL /pgSQL function _region_il.cleanupevent(interval) line 97 at PERFORM 2017-06-14 23:07:42 PDT INFO cleanupThread: 0.019 seconds for cleanupEvent() 2017-06-14 23:07:46 PDT INFO remoteWorkerThread_1: SYNC 5000000103 done in 0.009 seconds 2017-06-14 23:07:51 PDT INFO remoteWorkerThread_3: SYNC 5000000046 done in 0.005 seconds 2017-06-14 23:07:57 PDT INFO remoteWorkerThread_1: SYNC 5000000104 done in 0.010 seconds 2017-06-14 23:08:02 PDT INFO remoteWorkerThread_3: SYNC 5000000047 done in 0.009 seconds 2017-06-14 23:08:08 PDT INFO remoteWorkerThread_1: SYNC 5000000105 done in 0.012 seconds 2017-06-14 23:08:13 PDT INFO remoteWorkerThread_1: SYNC 5000000106 done in 0.008 seconds 2017-06-14 23:08:13 PDT INFO remoteWorkerThread_3: SYNC 5000000048 done in 0.013 seconds 2017-06-14 23:08:24 PDT INFO remoteWorkerThread_3: SYNC 5000000049 done in 0.009 seconds 2017-06-14 23:08:24 PDT INFO remoteWorkerThread_1: SYNC 5000000107 done in 0.010 seconds 2017-06-14 23:08:35 PDT INFO remoteWorkerThread_3: SYNC 5000000050 done in 0.005 seconds 2017-06-14 23:08:39 PDT INFO remoteWorkerThread_1: SYNC 5000000108 done in 0.006 seconds 2017-06-14 23:08:39 PDT INFO remoteWorkerThread_3: SYNC 5000000051 done in 0.005 seconds 2017-06-14 23:08:43 PDT INFO remoteWorkerThread_1: SYNC 5000000109 done in 0.005 seconds 2017-06-14 23:08:54 PDT INFO remoteWorkerThread_3: SYNC 5000000052 done in 0.006 seconds 2017-06-14 23:08:54 PDT INFO remoteWorkerThread_1: SYNC 5000000110 done in 0.006 seconds 2017-06-14 23:09:04 PDT INFO remoteWorkerThread_3: SYNC 5000000053 done in 0.005 seconds 2017-06-14 23:09:08 PDT INFO remoteWorkerThread_1: SYNC 5000000111 done in 0.005 seconds 2017-06-14 23:09:11 PDT INFO remoteWorkerThread_3: SYNC 5000000054 done in 0.005 seconds 2017-06-14 23:09:16 PDT INFO remoteWorkerThread_1: SYNC 5000000112 done in 0.005 seconds 2017-06-14 23:09:22 PDT INFO remoteWorkerThread_3: SYNC 5000000055 done in 0.005 seconds 2017-06-14 23:09:27 PDT INFO remoteWorkerThread_1: SYNC 5000000113 done in 0.005 seconds 2017-06-14 23:09:33 PDT INFO remoteWorkerThread_3: SYNC 5000000056 done in 0.005 seconds 2017-06-14 23:09:34 PDT INFO remoteWorkerThread_1: SYNC 5000000114 done in 0.005 seconds 2017-06-14 23:09:45 PDT INFO remoteWorkerThread_1: SYNC 5000000115 done in 0.005 seconds 2017-06-14 23:09:48 PDT INFO remoteWorkerThread_3: SYNC 5000000057 done in 0.005 seconds 2017-06-14 23:09:49 PDT INFO remoteWorkerThread_3: SYNC 5000000058 done in 0.005 seconds 2017-06-14 23:09:59 PDT INFO remoteWorkerThread_1: SYNC 5000000116 done in 0.005 seconds 2017-06-14 23:10:00 PDT INFO remoteWorkerThread_3: SYNC 5000000059 done in 0.005 seconds 2017-06-14 23:10:04 PDT INFO remoteWorkerThread_1: SYNC 5000000117 done in 0.005 seconds 2017-06-14 23:10:10 PDT INFO remoteWorkerThread_3: SYNC 5000000060 done in 0.005 seconds 2017-06-14 23:10:14 PDT INFO remoteWorkerThread_1: SYNC 5000000118 done in 0.005 seconds 2017-06-14 23:10:22 PDT INFO remoteWorkerThread_3: SYNC 5000000061 done in 0.005 seconds 2017-06-14 23:10:28 PDT INFO remoteWorkerThread_1: SYNC 5000000119 done in 0.005 seconds 2017-06-14 23:10:33 PDT INFO remoteWorkerThread_3: SYNC 5000000062 done in 0.005 seconds 2017-06-14 23:10:39 PDT INFO remoteWorkerThread_1: SYNC 5000000120 done in 0.004 seconds 2017-06-14 23:10:43 PDT INFO remoteWorkerThread_3: SYNC 5000000063 done in 0.005 seconds 2017-06-14 23:10:43 PDT INFO remoteWorkerThread_1: SYNC 5000000121 done in 0.005 seconds 2017-06-14 23:10:52 PDT INFO remoteWorkerThread_3: SYNC 5000000064 done in 0.005 seconds 2017-06-14 23:10:57 PDT INFO remoteWorkerThread_1: SYNC 5000000122 done in 0.004 seconds 2017-06-14 23:11:03 PDT INFO remoteWorkerThread_3: SYNC 5000000065 done in 0.005 seconds 2017-06-14 23:11:08 PDT INFO remoteWorkerThread_1: SYNC 5000000123 done in 0.005 seconds 2017-06-14 23:11:14 PDT INFO remoteWorkerThread_3: SYNC 5000000066 done in 0.005 seconds 2017-06-14 23:11:19 PDT INFO remoteWorkerThread_1: SYNC 5000000124 done in 0.005 seconds 2017-06-14 23:11:25 PDT INFO remoteWorkerThread_3: SYNC 5000000067 done in 0.005 seconds 2017-06-14 23:11:30 PDT INFO remoteWorkerThread_1: SYNC 5000000125 done in 0.005 seconds 2017-06-14 23:11:34 PDT INFO remoteWorkerThread_1: SYNC 5000000126 done in 0.005 seconds 2017-06-14 23:11:36 PDT INFO remoteWorkerThread_3: SYNC 5000000068 done in 0.005 seconds 2017-06-14 23:11:40 PDT INFO remoteWorkerThread_3: SYNC 5000000069 done in 0.004 seconds 2017-06-14 23:11:45 PDT INFO remoteWorkerThread_1: SYNC 5000000127 done in 0.005 seconds 2017-06-14 23:11:51 PDT INFO remoteWorkerThread_3: SYNC 5000000070 done in 0.004 seconds 2017-06-14 23:11:56 PDT CONFIG storeSubscribe: sub_set=1 sub_provider=1 sub_forward= 't' 2017-06-14 23:11:56 PDT CONFIG storeListen: li_origin=1 li_receiver=2 li_provider=1 2017-06-14 23:11:56 PDT CONFIG storeListen: li_origin=3 li_receiver=2 li_provider=3 2017-06-14 23:11:56 PDT CONFIG storeListen: li_origin=3 li_receiver=2 li_provider=1 2017-06-14 23:11:56 PDT CONFIG storeListen: li_origin=1 li_receiver=2 li_provider=3 2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_1: update provider configuration 2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_1: connection for provider 1 terminated 2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_1: disconnecting from data provider 1 2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_1: connection for provider 3 terminated 2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_1: disconnecting from data provider 3 2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_3: update provider configuration 2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_3: connection for provider 1 terminated 2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_3: disconnecting from data provider 1 2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_3: connection for provider 3 terminated 2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_3: disconnecting from data provider 3 2017-06-14 23:11:56 PDT INFO copy_set 1 - omit=f - bool=0 2017-06-14 23:11:56 PDT INFO omit is FALSE 2017-06-14 23:11:56 PDT CONFIG version for "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505 2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_1: connected to provider DB 2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_1: prepare to copy table "il_longlive" . "entity" 2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_1: prepare to copy table "il_longlive" . "entity_result" 2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_1: prepare to copy table "il_longlive" . "result" 2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_1: prepare to copy table "il_longlive" . "result_logbook" 2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_1: all tables for set 1 found on subscriber 2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_1: copy sequence "il_longlive" . "entity_entity_id_seq" 2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_1: copy sequence "il_longlive" . "entity_result_id_seq" 2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_1: copy sequence "il_longlive" . "result_id_seq" 2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_1: copy sequence "il_longlive" . "result_logbook_record_id_seq" 2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_1: copy table "il_longlive" . "entity" 2017-06-14 23:11:57 PDT CONFIG remoteWorkerThread_1: Begin COPY of table "il_longlive" . "entity" NOTICE: truncate of "il_longlive" . "entity" succeeded 2017-06-14 23:12:02 PDT CONFIG remoteWorkerThread_3: update provider configuration 2017-06-14 23:12:02 PDT CONFIG version for "host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505 2017-06-14 23:12:02 PDT INFO remoteWorkerThread_3: SYNC 5000000071 done in 0.029 seconds 2017-06-14 23:12:13 PDT INFO remoteWorkerThread_3: SYNC 5000000072 done in 0.003 seconds 2017-06-14 23:12:28 PDT INFO remoteWorkerThread_3: SYNC 5000000073 done in 0.016 seconds 2017-06-14 23:12:33 PDT CONFIG remoteWorkerThread_1: 125045870 bytes copied for table "il_longlive" . "entity" 2017-06-14 23:12:35 PDT INFO remoteWorkerThread_3: SYNC 5000000074 done in 0.010 seconds 2017-06-14 23:12:43 PDT CONFIG remoteWorkerThread_1: 46.468 seconds to copy table "il_longlive" . "entity" 2017-06-14 23:12:43 PDT CONFIG remoteWorkerThread_1: copy table "il_longlive" . "entity_result" 2017-06-14 23:12:43 PDT CONFIG remoteWorkerThread_1: Begin COPY of table "il_longlive" . "entity_result" NOTICE: truncate of "il_longlive" . "entity_result" succeeded 2017-06-14 23:12:46 PDT INFO remoteWorkerThread_3: SYNC 5000000075 done in 0.003 seconds 2017-06-14 23:12:50 PDT INFO remoteWorkerThread_3: SYNC 5000000076 done in 0.003 seconds 2017-06-14 23:13:12 PDT CONFIG remoteWorkerThread_1: 66996342 bytes copied for table "il_longlive" . "entity_result" 2017-06-14 23:13:14 PDT CONFIG remoteWorkerThread_3: update provider configuration 2017-06-14 23:13:14 PDT CONFIG version for "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505 2017-06-14 23:13:14 PDT INFO remoteWorkerThread_3: SYNC 5000000078 done in 0.025 seconds 2017-06-14 23:13:23 PDT INFO remoteWorkerThread_3: SYNC 5000000079 done in 0.013 seconds 2017-06-14 23:13:32 PDT INFO remoteWorkerThread_3: SYNC 5000000080 done in 0.005 seconds 2017-06-14 23:13:41 PDT INFO remoteWorkerThread_3: SYNC 5000000081 done in 0.005 seconds 2017-06-14 23:13:55 PDT INFO remoteWorkerThread_3: SYNC 5000000082 done in 0.005 seconds 2017-06-14 23:13:57 PDT CONFIG remoteWorkerThread_1: 74.255 seconds to copy table "il_longlive" . "entity_result" 2017-06-14 23:13:57 PDT CONFIG remoteWorkerThread_1: copy table "il_longlive" . "result" 2017-06-14 23:13:57 PDT CONFIG remoteWorkerThread_1: Begin COPY of table "il_longlive" . "result" NOTICE: truncate of "il_longlive" . "result" succeeded 2017-06-14 23:14:07 PDT INFO remoteWorkerThread_3: SYNC 5000000083 done in 0.005 seconds 2017-06-14 23:14:20 PDT INFO remoteWorkerThread_3: SYNC 5000000084 done in 0.020 seconds 2017-06-14 23:14:23 PDT INFO remoteWorkerThread_3: SYNC 5000000085 done in 0.005 seconds 2017-06-14 23:14:29 PDT CONFIG remoteWorkerThread_1: 105961503 bytes copied for table "il_longlive" . "result" 2017-06-14 23:14:33 PDT INFO remoteWorkerThread_3: SYNC 5000000086 done in 0.005 seconds 2017-06-14 23:14:41 PDT INFO remoteWorkerThread_3: SYNC 5000000087 done in 0.005 seconds 2017-06-14 23:14:50 PDT CONFIG remoteWorkerThread_1: 52.781 seconds to copy table "il_longlive" . "result" 2017-06-14 23:14:50 PDT CONFIG remoteWorkerThread_1: copy table "il_longlive" . "result_logbook" 2017-06-14 23:14:50 PDT CONFIG remoteWorkerThread_1: Begin COPY of table "il_longlive" . "result_logbook" NOTICE: truncate of "il_longlive" . "result_logbook" succeeded 2017-06-14 23:14:50 PDT CONFIG remoteWorkerThread_1: 79 bytes copied for table "il_longlive" . "result_logbook" 2017-06-14 23:14:50 PDT CONFIG remoteWorkerThread_1: 0.322 seconds to copy table "il_longlive" . "result_logbook" 2017-06-14 23:14:50 PDT CONFIG remoteWorkerThread_1: set last_value of sequence 103 ( "il_longlive" . "result_id_seq" ) to 2611527 2017-06-14 23:14:50 PDT CONFIG remoteWorkerThread_1: set last_value of sequence 102 ( "il_longlive" . "entity_result_id_seq" ) to 2611527 2017-06-14 23:14:50 PDT CONFIG remoteWorkerThread_1: set last_value of sequence 104 ( "il_longlive" . "result_logbook_record_id_seq" ) to 1 2017-06-14 23:14:50 PDT CONFIG remoteWorkerThread_1: set last_value of sequence 101 ( "il_longlive" . "entity_entity_id_seq" ) to 2528612 2017-06-14 23:14:50 PDT INFO remoteWorkerThread_1: 0.002 seconds to copy sequences 2017-06-14 23:14:50 PDT INFO remoteWorkerThread_1: copy_set SYNC found, use event seqno 5000000130. 2017-06-14 23:14:50 PDT INFO remoteWorkerThread_1: 0.005 seconds to build initial setsync status 2017-06-14 23:14:50 PDT INFO copy_set 1 done in 173.913 seconds 2017-06-14 23:14:50 PDT CONFIG enableSubscription: sub_set=1 2017-06-14 23:14:50 PDT CONFIG storeListen: li_origin=3 li_receiver=2 li_provider=3 2017-06-14 23:14:50 PDT CONFIG storeListen: li_origin=3 li_receiver=2 li_provider=1 2017-06-14 23:14:50 PDT CONFIG storeListen: li_origin=1 li_receiver=2 li_provider=1 2017-06-14 23:14:50 PDT CONFIG remoteWorkerThread_1: update provider configuration 2017-06-14 23:14:50 PDT CONFIG remoteWorkerThread_1: added active set 1 to provider 1 2017-06-14 23:14:50 PDT CONFIG remoteWorkerThread_3: update provider configuration 2017-06-14 23:14:50 PDT CONFIG remoteWorkerThread_3: connection for provider 3 terminated 2017-06-14 23:14:50 PDT CONFIG remoteWorkerThread_3: disconnecting from data provider 3 2017-06-14 23:14:50 PDT CONFIG remoteWorkerThread_3: connection for provider 1 terminated 2017-06-14 23:14:50 PDT CONFIG remoteWorkerThread_3: disconnecting from data provider 1 2017-06-14 23:14:50 PDT CONFIG version for "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505 2017-06-14 23:14:50 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1 2017-06-14 23:14:50 PDT INFO remoteWorkerThread_1: SYNC 5000000131 done in 0.053 seconds 2017-06-14 23:14:50 PDT CONFIG remoteWorkerThread_1: update provider configuration 2017-06-14 23:14:50 PDT CONFIG remoteWorkerThread_1: added active set 1 to provider 1 2017-06-14 23:14:50 PDT CONFIG version for "host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505 2017-06-14 23:14:50 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1 2017-06-14 23:14:50 PDT INFO remoteWorkerThread_1: SYNC 5000000135 done in 0.016 seconds 2017-06-14 23:14:50 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1 2017-06-14 23:14:50 PDT INFO remoteWorkerThread_1: SYNC 5000000143 done in 0.005 seconds 2017-06-14 23:14:50 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1 2017-06-14 23:14:50 PDT INFO remoteWorkerThread_1: SYNC 5000000147 done in 0.005 seconds 2017-06-14 23:14:52 PDT CONFIG remoteWorkerThread_3: update provider configuration 2017-06-14 23:14:52 PDT CONFIG storeListen: li_origin=3 li_receiver=2 li_provider=3 2017-06-14 23:14:52 PDT CONFIG storeListen: li_origin=3 li_receiver=2 li_provider=1 2017-06-14 23:14:52 PDT CONFIG storeListen: li_origin=1 li_receiver=2 li_provider=1 2017-06-14 23:14:52 PDT CONFIG remoteWorkerThread_1: update provider configuration 2017-06-14 23:14:52 PDT CONFIG remoteWorkerThread_1: added active set 1 to provider 1 2017-06-14 23:14:52 PDT CONFIG remoteWorkerThread_1: connection for provider 3 terminated 2017-06-14 23:14:52 PDT CONFIG remoteWorkerThread_1: disconnecting from data provider 3 2017-06-14 23:14:52 PDT CONFIG version for "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505 2017-06-14 23:14:52 PDT CONFIG storeListen: li_origin=3 li_receiver=2 li_provider=3 2017-06-14 23:14:52 PDT CONFIG storeListen: li_origin=3 li_receiver=2 li_provider=1 2017-06-14 23:14:52 PDT CONFIG storeListen: li_origin=1 li_receiver=2 li_provider=1 2017-06-14 23:14:52 PDT CONFIG remoteWorkerThread_1: update provider configuration 2017-06-14 23:14:52 PDT CONFIG remoteWorkerThread_1: added active set 1 to provider 1 2017-06-14 23:14:52 PDT INFO remoteWorkerThread_3: SYNC 5000000088 done in 0.013 seconds 2017-06-14 23:14:52 PDT CONFIG remoteWorkerThread_3: update provider configuration 2017-06-14 23:14:52 PDT CONFIG remoteWorkerThread_3: connection for provider 1 terminated 2017-06-14 23:14:52 PDT CONFIG remoteWorkerThread_3: disconnecting from data provider 1 2017-06-14 23:15:00 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1 2017-06-14 23:15:00 PDT INFO remoteWorkerThread_1: SYNC 5000000150 done in 0.006 seconds 2017-06-14 23:15:05 PDT CONFIG remoteWorkerThread_3: update provider configuration 2017-06-14 23:15:05 PDT CONFIG version for "host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505 2017-06-14 23:15:05 PDT INFO remoteWorkerThread_3: SYNC 5000000089 done in 0.054 seconds 2017-06-14 23:15:11 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1 2017-06-14 23:15:11 PDT INFO remoteWorkerThread_1: SYNC 5000000151 done in 0.004 seconds 2017-06-14 23:15:15 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1 2017-06-14 23:15:15 PDT INFO remoteWorkerThread_1: SYNC 5000000152 done in 0.004 seconds 2017-06-14 23:15:16 PDT INFO remoteWorkerThread_3: SYNC 5000000090 done in 0.005 seconds 2017-06-14 23:15:26 PDT CONFIG remoteWorkerThread_3: update provider configuration 2017-06-14 23:15:26 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1 2017-06-14 23:15:26 PDT INFO remoteWorkerThread_1: SYNC 5000000153 done in 0.007 seconds 2017-06-14 23:15:26 PDT CONFIG version for "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505 2017-06-14 23:15:26 PDT INFO remoteWorkerThread_3: SYNC 5000000091 done in 0.025 seconds 2017-06-14 23:15:37 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1 2017-06-14 23:15:37 PDT INFO remoteWorkerThread_1: SYNC 5000000154 done in 0.005 seconds 2017-06-14 23:15:37 PDT INFO remoteWorkerThread_3: SYNC 5000000092 done in 0.006 seconds 2017-06-14 23:15:47 PDT INFO remoteWorkerThread_3: SYNC 5000000093 done in 0.005 seconds 2017-06-14 23:15:51 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1 2017-06-14 23:15:51 PDT INFO remoteWorkerThread_1: SYNC 5000000155 done in 0.004 seconds 2017-06-14 23:15:55 PDT INFO remoteWorkerThread_3: SYNC 5000000094 done in 0.005 seconds 2017-06-14 23:15:59 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1 2017-06-14 23:15:59 PDT INFO remoteWorkerThread_1: SYNC 5000000156 done in 0.005 seconds 2017-06-14 23:16:09 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1 2017-06-14 23:16:09 PDT INFO remoteWorkerThread_1: SYNC 5000000157 done in 0.004 seconds 2017-06-14 23:16:12 PDT INFO remoteWorkerThread_3: SYNC 5000000095 done in 0.009 seconds 2017-06-14 23:16:13 PDT INFO remoteWorkerThread_3: SYNC 5000000096 done in 0.015 seconds 2017-06-14 23:16:17 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1 2017-06-14 23:16:17 PDT INFO remoteWorkerThread_1: SYNC 5000000158 done in 0.004 seconds 2017-06-14 23:16:28 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1 2017-06-14 23:16:28 PDT INFO remoteWorkerThread_1: SYNC 5000000159 done in 0.004 seconds 2017-06-14 23:16:31 PDT INFO remoteWorkerThread_3: SYNC 5000000097 done in 0.005 seconds 2017-06-14 23:16:35 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1 2017-06-14 23:16:35 PDT INFO remoteWorkerThread_1: SYNC 5000000160 done in 0.004 seconds 2017-06-14 23:16:39 PDT INFO remoteWorkerThread_3: SYNC 5000000098 done in 0.005 seconds 2017-06-14 23:16:46 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1 2017-06-14 23:16:46 PDT INFO remoteWorkerThread_1: SYNC 5000000161 done in 0.004 seconds 2017-06-14 23:16:50 PDT INFO remoteWorkerThread_3: SYNC 5000000099 done in 0.005 seconds 2017-06-14 23:16:57 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1 2017-06-14 23:16:57 PDT INFO remoteWorkerThread_1: SYNC 5000000162 done in 0.004 seconds 2017-06-14 23:17:01 PDT INFO remoteWorkerThread_3: SYNC 5000000100 done in 0.008 seconds 2017-06-14 23:17:09 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1 2017-06-14 23:17:09 PDT INFO remoteWorkerThread_3: SYNC 5000000101 done in 0.006 seconds 2017-06-14 23:17:09 PDT INFO remoteWorkerThread_1: SYNC 5000000163 done in 0.005 seconds 2017-06-14 23:17:22 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1 2017-06-14 23:17:22 PDT INFO remoteWorkerThread_1: SYNC 5000000164 done in 0.005 seconds 2017-06-14 23:17:22 PDT INFO remoteWorkerThread_3: SYNC 5000000102 done in 0.005 seconds 2017-06-14 23:17:25 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1 2017-06-14 23:17:25 PDT INFO remoteWorkerThread_1: SYNC 5000000165 done in 0.004 seconds 2017-06-14 23:17:30 PDT INFO remoteWorkerThread_3: SYNC 5000000103 done in 0.005 seconds 2017-06-14 23:17:36 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1 2017-06-14 23:17:36 PDT INFO remoteWorkerThread_1: SYNC 5000000166 done in 0.004 seconds 2017-06-14 23:17:40 PDT INFO remoteWorkerThread_3: SYNC 5000000104 done in 0.013 seconds NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=15559 CONTEXT: SQL statement "SELECT " _region_il ".cleanupNodelock()" PL /pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=15656 CONTEXT: SQL statement "SELECT " _region_il ".cleanupNodelock()" PL /pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=15675 CONTEXT: SQL statement "SELECT " _region_il ".cleanupNodelock()" PL /pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=15683 CONTEXT: SQL statement "SELECT " _region_il ".cleanupNodelock()" PL /pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=15825 CONTEXT: SQL statement "SELECT " _region_il ".cleanupNodelock()" PL /pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=15838 CONTEXT: SQL statement "SELECT " _region_il ".cleanupNodelock()" PL /pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=15853 CONTEXT: SQL statement "SELECT " _region_il ".cleanupNodelock()" PL /pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=15939 CONTEXT: SQL statement "SELECT " _region_il ".cleanupNodelock()" PL /pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=15940 CONTEXT: SQL statement "SELECT " _region_il ".cleanupNodelock()" PL /pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=15941 CONTEXT: SQL statement "SELECT " _region_il ".cleanupNodelock()" PL /pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM NOTICE: Slony-I: log switch to sl_log_2 complete - truncate sl_log_1 CONTEXT: PL /pgSQL function _region_il.cleanupevent(interval) line 95 at assignment 2017-06-14 23:17:42 PDT INFO cleanupThread: 0.047 seconds for cleanupEvent() 2017-06-14 23:17:48 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1 2017-06-14 23:17:48 PDT INFO remoteWorkerThread_3: SYNC 5000000105 done in 0.005 seconds 2017-06-14 23:17:48 PDT INFO remoteWorkerThread_1: SYNC 5000000167 done in 0.005 seconds 2017-06-14 23:18:01 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1 2017-06-14 23:18:01 PDT INFO remoteWorkerThread_3: SYNC 5000000106 done in 0.005 seconds 2017-06-14 23:18:01 PDT INFO remoteWorkerThread_1: SYNC 5000000168 done in 0.005 seconds 2017-06-14 23:18:14 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1 2017-06-14 23:18:14 PDT INFO remoteWorkerThread_3: SYNC 5000000107 done in 0.006 seconds 2017-06-14 23:18:14 PDT INFO remoteWorkerThread_1: SYNC 5000000169 done in 0.006 seconds 2017-06-14 23:18:17 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1 2017-06-14 23:18:17 PDT INFO remoteWorkerThread_1: SYNC 5000000170 done in 0.005 seconds 2017-06-14 23:18:20 PDT INFO remoteWorkerThread_3: SYNC 5000000108 done in 0.005 seconds 2017-06-14 23:18:28 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1 2017-06-14 23:18:28 PDT INFO remoteWorkerThread_3: SYNC 5000000109 done in 0.005 seconds 2017-06-14 23:18:28 PDT INFO remoteWorkerThread_1: SYNC 5000000171 done in 0.005 seconds 2017-06-14 23:18:41 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1 2017-06-14 23:18:41 PDT INFO remoteWorkerThread_1: SYNC 5000000172 done in 0.005 seconds 2017-06-14 23:18:41 PDT INFO remoteWorkerThread_3: SYNC 5000000110 done in 0.005 seconds 2017-06-14 23:18:54 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1 2017-06-14 23:18:54 PDT INFO remoteWorkerThread_3: SYNC 5000000111 done in 0.005 seconds 2017-06-14 23:18:54 PDT INFO remoteWorkerThread_1: SYNC 5000000173 done in 0.005 seconds 2017-06-14 23:18:57 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1 2017-06-14 23:18:57 PDT INFO remoteWorkerThread_1: SYNC 5000000174 done in 0.013 seconds 2017-06-14 23:19:00 PDT INFO remoteWorkerThread_3: SYNC 5000000112 done in 0.009 seconds 2017-06-14 23:19:08 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1 2017-06-14 23:19:08 PDT INFO remoteWorkerThread_3: SYNC 5000000113 done in 0.005 seconds 2017-06-14 23:19:08 PDT INFO remoteWorkerThread_1: SYNC 5000000175 done in 0.005 seconds 2017-06-14 23:19:21 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1 2017-06-14 23:19:21 PDT INFO remoteWorkerThread_3: SYNC 5000000114 done in 0.006 seconds 2017-06-14 23:19:21 PDT INFO remoteWorkerThread_1: SYNC 5000000176 done in 0.005 seconds 2017-06-14 23:19:34 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1 2017-06-14 23:19:34 PDT INFO remoteWorkerThread_3: SYNC 5000000115 done in 0.006 seconds 2017-06-14 23:19:34 PDT INFO remoteWorkerThread_1: SYNC 5000000177 done in 0.006 seconds 2017-06-14 23:19:37 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1 2017-06-14 23:19:37 PDT INFO remoteWorkerThread_1: SYNC 5000000178 done in 0.005 seconds 2017-06-14 23:19:40 PDT INFO remoteWorkerThread_3: SYNC 5000000116 done in 0.005 seconds 2017-06-14 23:19:48 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1 2017-06-14 23:19:48 PDT INFO remoteWorkerThread_3: SYNC 5000000117 done in 0.005 seconds 2017-06-14 23:19:48 PDT INFO remoteWorkerThread_1: SYNC 5000000179 done in 0.005 seconds 2017-06-14 23:20:01 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1 2017-06-14 23:20:01 PDT INFO remoteWorkerThread_1: SYNC 5000000180 done in 0.006 seconds 2017-06-14 23:20:01 PDT INFO remoteWorkerThread_3: SYNC 5000000118 done in 0.006 seconds 2017-06-14 23:20:14 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1 2017-06-14 23:20:14 PDT INFO remoteWorkerThread_1: SYNC 5000000181 done in 0.005 seconds 2017-06-14 23:20:14 PDT INFO remoteWorkerThread_3: SYNC 5000000119 done in 0.005 seconds 2017-06-14 23:20:17 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1 2017-06-14 23:20:17 PDT INFO remoteWorkerThread_1: SYNC 5000000182 done in 0.005 seconds 2017-06-14 23:20:18 PDT INFO remoteWorkerThread_3: SYNC 5000000120 done in 0.005 seconds 2017-06-14 23:20:27 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1 2017-06-14 23:20:27 PDT INFO remoteWorkerThread_1: SYNC 5000000183 done in 0.005 seconds 2017-06-14 23:20:32 PDT INFO remoteWorkerThread_3: SYNC 5000000121 done in 0.006 seconds 2017-06-14 23:20:38 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1 2017-06-14 23:20:38 PDT INFO remoteWorkerThread_1: SYNC 5000000184 done in 0.004 seconds 2017-06-14 23:20:39 PDT INFO remoteWorkerThread_3: SYNC 5000000122 done in 0.005 seconds 2017-06-14 23:20:53 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1 2017-06-14 23:20:53 PDT INFO remoteWorkerThread_3: SYNC 5000000123 done in 0.006 seconds 2017-06-14 23:20:53 PDT INFO remoteWorkerThread_1: SYNC 5000000185 done in 0.006 seconds 2017-06-14 23:21:01 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1 2017-06-14 23:21:01 PDT INFO remoteWorkerThread_3: SYNC 5000000124 done in 0.006 seconds 2017-06-14 23:21:01 PDT INFO remoteWorkerThread_1: SYNC 5000000186 done in 0.006 seconds 2017-06-14 23:21:09 PDT INFO remoteWorkerThread_3: SYNC 5000000125 done in 0.005 seconds 2017-06-14 23:21:14 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1 2017-06-14 23:21:14 PDT INFO remoteWorkerThread_1: SYNC 5000000187 done in 0.004 seconds 2017-06-14 23:21:15 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1 2017-06-14 23:21:15 PDT INFO remoteWorkerThread_1: SYNC 5000000188 done in 0.004 seconds 2017-06-14 23:21:20 PDT INFO remoteWorkerThread_3: SYNC 5000000126 done in 0.005 seconds 2017-06-14 23:21:26 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1 2017-06-14 23:21:26 PDT INFO remoteWorkerThread_1: SYNC 5000000189 done in 0.004 seconds 2017-06-14 23:21:31 PDT INFO remoteWorkerThread_3: SYNC 5000000127 done in 0.005 seconds 2017-06-14 23:21:37 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1 2017-06-14 23:21:37 PDT INFO remoteWorkerThread_1: SYNC 5000000190 done in 0.005 seconds 2017-06-14 23:21:42 PDT INFO remoteWorkerThread_3: SYNC 5000000128 done in 0.005 seconds 2017-06-14 23:21:48 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1 2017-06-14 23:21:48 PDT INFO remoteWorkerThread_1: SYNC 5000000191 done in 0.008 seconds 2017-06-14 23:21:52 PDT INFO remoteWorkerThread_3: SYNC 5000000129 done in 0.006 seconds 2017-06-14 23:22:00 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1 2017-06-14 23:22:00 PDT INFO remoteWorkerThread_1: SYNC 5000000192 done in 0.005 seconds 2017-06-14 23:22:00 PDT INFO remoteWorkerThread_3: SYNC 5000000130 done in 0.006 seconds 2017-06-14 23:22:13 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1 2017-06-14 23:22:13 PDT INFO remoteWorkerThread_3: SYNC 5000000131 done in 0.005 seconds 2017-06-14 23:22:13 PDT INFO remoteWorkerThread_1: SYNC 5000000193 done in 0.005 seconds 2017-06-14 23:22:16 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1 2017-06-14 23:22:16 PDT INFO remoteWorkerThread_1: SYNC 5000000194 done in 0.004 seconds 2017-06-14 23:22:21 PDT INFO remoteWorkerThread_3: SYNC 5000000132 done in 0.005 seconds 2017-06-14 23:22:27 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1 2017-06-14 23:22:27 PDT INFO remoteWorkerThread_1: SYNC 5000000195 done in 0.005 seconds 2017-06-14 23:22:32 PDT INFO remoteWorkerThread_3: SYNC 5000000133 done in 0.005 seconds 2017-06-14 23:22:38 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1 2017-06-14 23:22:38 PDT INFO remoteWorkerThread_1: SYNC 5000000196 done in 0.006 seconds 2017-06-14 23:22:42 PDT INFO remoteWorkerThread_3: SYNC 5000000134 done in 0.005 seconds 2017-06-14 23:22:50 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1 2017-06-14 23:22:50 PDT INFO remoteWorkerThread_1: SYNC 5000000197 done in 0.005 seconds 2017-06-14 23:22:54 PDT INFO remoteWorkerThread_3: SYNC 5000000135 done in 0.005 seconds 2017-06-14 23:22:55 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1 2017-06-14 23:22:55 PDT INFO remoteWorkerThread_1: SYNC 5000000198 done in 0.004 seconds 2017-06-14 23:22:59 PDT INFO remoteWorkerThread_3: SYNC 5000000136 done in 0.005 seconds 2017-06-14 23:23:10 PDT INFO remoteWorkerThread_3: SYNC 5000000137 done in 0.005 seconds 2017-06-14 23:23:13 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1 2017-06-14 23:23:13 PDT INFO remoteWorkerThread_1: SYNC 5000000199 done in 0.005 seconds 2017-06-14 23:23:21 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1 2017-06-14 23:23:21 PDT INFO remoteWorkerThread_1: SYNC 5000000200 done in 0.004 seconds 2017-06-14 23:23:25 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1 2017-06-14 23:23:25 PDT INFO remoteWorkerThread_3: SYNC 5000000138 done in 0.005 seconds 2017-06-14 23:23:25 PDT INFO remoteWorkerThread_1: SYNC 5000000201 done in 0.005 seconds 2017-06-14 23:23:34 PDT INFO remoteWorkerThread_3: SYNC 5000000139 done in 0.005 seconds 2017-06-14 23:23:39 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1 2017-06-14 23:23:39 PDT INFO remoteWorkerThread_1: SYNC 5000000202 done in 0.005 seconds 2017-06-14 23:23:43 PDT INFO remoteWorkerThread_3: SYNC 5000000140 done in 0.005 seconds 2017-06-14 23:23:47 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1 2017-06-14 23:23:47 PDT INFO remoteWorkerThread_1: SYNC 5000000203 done in 0.004 seconds |
辅节点2查看Slony-I日志信息
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 |
$ cat /usr/local/pgsql/data/pg_log/node3/region_il-2017-06-14 .log 2017-06-14 23:00:38 PDT CONFIG main: slon version 2.2.4 starting up 2017-06-14 23:00:38 PDT INFO slon: watchdog process started 2017-06-14 23:00:38 PDT CONFIG slon: watchdog ready - pid = 19212 2017-06-14 23:00:38 PDT CONFIG slon: worker process created - pid = 19213 2017-06-14 23:00:38 PDT CONFIG main: Integer option vac_frequency = 3 2017-06-14 23:00:38 PDT CONFIG main: Integer option log_level = 0 2017-06-14 23:00:38 PDT CONFIG main: Integer option sync_interval = 1000 2017-06-14 23:00:38 PDT CONFIG main: Integer option sync_interval_timeout = 10000 2017-06-14 23:00:38 PDT CONFIG main: Integer option sync_group_maxsize = 20 2017-06-14 23:00:38 PDT CONFIG main: Integer option syslog = 0 2017-06-14 23:00:38 PDT CONFIG main: Integer option quit_sync_provider = 0 2017-06-14 23:00:38 PDT CONFIG main: Integer option remote_listen_timeout = 300 2017-06-14 23:00:38 PDT CONFIG main: Integer option monitor_interval = 500 2017-06-14 23:00:38 PDT CONFIG main: Integer option explain_interval = 0 2017-06-14 23:00:38 PDT CONFIG main: Integer option tcp_keepalive_idle = 0 2017-06-14 23:00:38 PDT CONFIG main: Integer option tcp_keepalive_interval = 0 2017-06-14 23:00:38 PDT CONFIG main: Integer option tcp_keepalive_count = 0 2017-06-14 23:00:38 PDT CONFIG main: Integer option apply_cache_size = 100 2017-06-14 23:00:38 PDT CONFIG main: Boolean option log_pid = 0 2017-06-14 23:00:38 PDT CONFIG main: Boolean option log_timestamp = 1 2017-06-14 23:00:38 PDT CONFIG main: Boolean option tcp_keepalive = 1 2017-06-14 23:00:38 PDT CONFIG main: Boolean option monitor_threads = 1 2017-06-14 23:00:38 PDT CONFIG main: Real option real_placeholder = 0.000000 2017-06-14 23:00:38 PDT CONFIG main: String option cluster_name = region_il 2017-06-14 23:00:38 PDT CONFIG main: String option conn_info = host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888 2017-06-14 23:00:38 PDT CONFIG main: String option pid_file = /usr/local/pgsql/data/region_il_node3 .pid 2017-06-14 23:00:38 PDT CONFIG main: String option log_timestamp_format = %Y-%m-%d %H:%M:%S %Z 2017-06-14 23:00:38 PDT CONFIG main: String option archive_dir = [NULL] 2017-06-14 23:00:38 PDT CONFIG main: String option sql_on_connection = [NULL] 2017-06-14 23:00:38 PDT CONFIG main: String option lag_interval = [NULL] 2017-06-14 23:00:38 PDT CONFIG main: String option command_on_logarchive = [NULL] 2017-06-14 23:00:38 PDT CONFIG main: String option syslog_facility = LOCAL0 2017-06-14 23:00:38 PDT CONFIG main: String option syslog_ident = slon 2017-06-14 23:00:38 PDT CONFIG main: String option cleanup_interval = 10 minutes 2017-06-14 23:00:38 PDT CONFIG main: local node id = 3 2017-06-14 23:00:38 PDT INFO main: main process started 2017-06-14 23:00:38 PDT CONFIG main: launching sched_start_mainloop 2017-06-14 23:00:38 PDT CONFIG main: loading current cluster configuration 2017-06-14 23:00:38 PDT CONFIG storeNode: no_id=1 no_comment= 'Node 1 - region_il@10.189.102.118' 2017-06-14 23:00:38 PDT CONFIG storeNode: no_id=2 no_comment= 'Node 2 - region_il@10.189.100.195' 2017-06-14 23:00:38 PDT CONFIG storePath: pa_server=1 pa_client=3 pa_conninfo= "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" pa_connretry=10 2017-06-14 23:00:38 PDT CONFIG storePath: pa_server=2 pa_client=3 pa_conninfo= "host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" pa_connretry=10 2017-06-14 23:00:38 PDT CONFIG storeListen: li_origin=1 li_receiver=3 li_provider=1 2017-06-14 23:00:38 PDT CONFIG storeListen: li_origin=2 li_receiver=3 li_provider=2 2017-06-14 23:00:38 PDT CONFIG main: last local event sequence = 5000000002 2017-06-14 23:00:38 PDT CONFIG main: configuration complete - starting threads 2017-06-14 23:00:38 PDT INFO localListenThread: thread starts 2017-06-14 23:00:38 PDT CONFIG version for "host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505 2017-06-14 23:00:38 PDT CONFIG enableNode: no_id=1 2017-06-14 23:00:38 PDT CONFIG enableNode: no_id=2 2017-06-14 23:00:38 PDT INFO main: running scheduler mainloop 2017-06-14 23:00:38 PDT INFO remoteWorkerThread_2: thread starts 2017-06-14 23:00:38 PDT INFO remoteListenThread_2: thread starts 2017-06-14 23:00:38 PDT CONFIG cleanupThread: thread starts 2017-06-14 23:00:38 PDT INFO syncThread: thread starts 2017-06-14 23:00:38 PDT INFO monitorThread: thread starts 2017-06-14 23:00:38 PDT INFO remoteListenThread_1: thread starts 2017-06-14 23:00:38 PDT INFO remoteWorkerThread_1: thread starts 2017-06-14 23:00:38 PDT CONFIG version for "host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505 2017-06-14 23:00:38 PDT CONFIG version for "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505 2017-06-14 23:00:38 PDT CONFIG version for "host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505 2017-06-14 23:00:38 PDT CONFIG version for "host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505 2017-06-14 23:00:38 PDT CONFIG version for "host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505 2017-06-14 23:00:38 PDT CONFIG version for "host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505 2017-06-14 23:00:38 PDT CONFIG version for "host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505 2017-06-14 23:00:38 PDT CONFIG remoteWorkerThread_2: update provider configuration 2017-06-14 23:00:38 PDT CONFIG cleanupThread: bias = 60 2017-06-14 23:00:38 PDT CONFIG remoteWorkerThread_1: update provider configuration 2017-06-14 23:00:38 PDT CONFIG storeListen: li_origin=1 li_receiver=3 li_provider=1 2017-06-14 23:00:38 PDT CONFIG storeListen: li_origin=2 li_receiver=3 li_provider=2 2017-06-14 23:00:38 PDT CONFIG storeListen: li_origin=1 li_receiver=3 li_provider=2 2017-06-14 23:00:38 PDT CONFIG remoteWorkerThread_2: update provider configuration 2017-06-14 23:00:38 PDT CONFIG storeListen: li_origin=1 li_receiver=3 li_provider=1 2017-06-14 23:00:38 PDT CONFIG storeListen: li_origin=2 li_receiver=3 li_provider=2 2017-06-14 23:00:38 PDT CONFIG storeListen: li_origin=1 li_receiver=3 li_provider=2 2017-06-14 23:00:38 PDT CONFIG remoteWorkerThread_1: update provider configuration 2017-06-14 23:00:38 PDT CONFIG storeListen: li_origin=1 li_receiver=3 li_provider=1 2017-06-14 23:00:38 PDT CONFIG storeListen: li_origin=2 li_receiver=3 li_provider=2 2017-06-14 23:00:38 PDT CONFIG storeListen: li_origin=1 li_receiver=3 li_provider=2 2017-06-14 23:00:38 PDT CONFIG remoteWorkerThread_2: update provider configuration 2017-06-14 23:00:38 PDT CONFIG remoteWorkerThread_2: update provider configuration 2017-06-14 23:00:38 PDT CONFIG storeListen: li_origin=1 li_receiver=3 li_provider=1 2017-06-14 23:00:38 PDT CONFIG storeListen: li_origin=2 li_receiver=3 li_provider=2 2017-06-14 23:00:38 PDT CONFIG storeListen: li_origin=1 li_receiver=3 li_provider=2 2017-06-14 23:00:38 PDT CONFIG remoteWorkerThread_1: update provider configuration 2017-06-14 23:00:38 PDT CONFIG version for "host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505 2017-06-14 23:00:38 PDT CONFIG storeListen: li_origin=1 li_receiver=3 li_provider=1 2017-06-14 23:00:38 PDT CONFIG storeListen: li_origin=2 li_receiver=3 li_provider=2 2017-06-14 23:00:38 PDT CONFIG storeListen: li_origin=2 li_receiver=3 li_provider=1 2017-06-14 23:00:38 PDT CONFIG storeListen: li_origin=1 li_receiver=3 li_provider=2 2017-06-14 23:00:38 PDT CONFIG remoteWorkerThread_1: update provider configuration 2017-06-14 23:00:38 PDT INFO remoteWorkerThread_2: SYNC 5000000003 done in 0.019 seconds 2017-06-14 23:00:38 PDT CONFIG remoteWorkerThread_2: update provider configuration 2017-06-14 23:00:38 PDT CONFIG remoteWorkerThread_2: connection for provider 2 terminated 2017-06-14 23:00:38 PDT CONFIG remoteWorkerThread_2: disconnecting from data provider 2 2017-06-14 23:00:38 PDT CONFIG remoteWorkerThread_2: update provider configuration 2017-06-14 23:00:38 PDT CONFIG storeListen: li_origin=1 li_receiver=3 li_provider=1 2017-06-14 23:00:38 PDT CONFIG storeListen: li_origin=2 li_receiver=3 li_provider=2 2017-06-14 23:00:38 PDT CONFIG storeListen: li_origin=2 li_receiver=3 li_provider=1 2017-06-14 23:00:38 PDT CONFIG storeListen: li_origin=1 li_receiver=3 li_provider=2 2017-06-14 23:00:38 PDT CONFIG remoteWorkerThread_1: update provider configuration 2017-06-14 23:00:38 PDT CONFIG remoteWorkerThread_1: update provider configuration 2017-06-14 23:00:38 PDT CONFIG version for "host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505 2017-06-14 23:00:38 PDT CONFIG version for "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505 2017-06-14 23:00:38 PDT INFO remoteWorkerThread_2: SYNC 5000000005 done in 0.011 seconds 2017-06-14 23:00:38 PDT INFO remoteWorkerThread_1: SYNC 5000000008 done in 0.011 seconds 2017-06-14 23:00:38 PDT CONFIG remoteWorkerThread_2: update provider configuration 2017-06-14 23:00:38 PDT CONFIG remoteWorkerThread_2: connection for provider 2 terminated 2017-06-14 23:00:38 PDT CONFIG remoteWorkerThread_2: disconnecting from data provider 2 2017-06-14 23:00:38 PDT CONFIG remoteWorkerThread_2: update provider configuration 2017-06-14 23:00:38 PDT CONFIG version for "host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505 2017-06-14 23:00:38 PDT INFO remoteWorkerThread_1: SYNC 5000000010 done in 0.002 seconds 2017-06-14 23:00:38 PDT INFO remoteWorkerThread_1: SYNC 5000000014 done in 0.002 seconds 2017-06-14 23:00:38 PDT INFO remoteWorkerThread_2: SYNC 5000000009 done in 0.011 seconds 2017-06-14 23:00:38 PDT INFO remoteWorkerThread_1: SYNC 5000000022 done in 0.002 seconds 2017-06-14 23:00:38 PDT INFO remoteWorkerThread_2: SYNC 5000000017 done in 0.002 seconds 2017-06-14 23:00:38 PDT INFO remoteWorkerThread_1: SYNC 5000000038 done in 0.002 seconds 2017-06-14 23:00:38 PDT INFO remoteWorkerThread_2: SYNC 5000000020 done in 0.002 seconds 2017-06-14 23:00:38 PDT INFO remoteWorkerThread_1: SYNC 5000000043 done in 0.002 seconds 2017-06-14 23:00:46 PDT CONFIG remoteWorkerThread_1: update provider configuration 2017-06-14 23:00:46 PDT INFO remoteWorkerThread_2: SYNC 5000000021 done in 0.003 seconds 2017-06-14 23:00:46 PDT CONFIG version for "host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505 2017-06-14 23:00:46 PDT INFO remoteWorkerThread_1: SYNC 5000000044 done in 0.017 seconds 2017-06-14 23:00:57 PDT INFO remoteWorkerThread_2: SYNC 5000000022 done in 0.002 seconds 2017-06-14 23:01:02 PDT INFO remoteWorkerThread_1: SYNC 5000000045 done in 0.005 seconds 2017-06-14 23:01:03 PDT INFO remoteWorkerThread_1: SYNC 5000000046 done in 0.004 seconds 2017-06-14 23:01:08 PDT INFO remoteWorkerThread_2: SYNC 5000000023 done in 0.003 seconds 2017-06-14 23:01:14 PDT INFO remoteWorkerThread_1: SYNC 5000000047 done in 0.004 seconds 2017-06-14 23:01:19 PDT INFO remoteWorkerThread_2: SYNC 5000000024 done in 0.003 seconds 2017-06-14 23:01:23 PDT INFO remoteWorkerThread_2: SYNC 5000000025 done in 0.003 seconds 2017-06-14 23:01:25 PDT INFO remoteWorkerThread_1: SYNC 5000000048 done in 0.004 seconds 2017-06-14 23:01:34 PDT INFO remoteWorkerThread_2: SYNC 5000000026 done in 0.002 seconds 2017-06-14 23:01:38 PDT INFO remoteWorkerThread_1: SYNC 5000000049 done in 0.004 seconds 2017-06-14 23:01:46 PDT INFO remoteWorkerThread_2: SYNC 5000000027 done in 0.003 seconds 2017-06-14 23:01:50 PDT INFO remoteWorkerThread_1: SYNC 5000000050 done in 0.005 seconds 2017-06-14 23:01:54 PDT INFO remoteWorkerThread_2: SYNC 5000000028 done in 0.002 seconds 2017-06-14 23:01:54 PDT INFO remoteWorkerThread_1: SYNC 5000000051 done in 0.005 seconds 2017-06-14 23:02:05 PDT INFO remoteWorkerThread_2: SYNC 5000000029 done in 0.003 seconds 2017-06-14 23:02:09 PDT INFO remoteWorkerThread_1: SYNC 5000000052 done in 0.005 seconds 2017-06-14 23:02:17 PDT INFO remoteWorkerThread_1: SYNC 5000000053 done in 0.005 seconds 2017-06-14 23:02:21 PDT CONFIG remoteWorkerThread_2: update provider configuration 2017-06-14 23:02:21 PDT CONFIG version for "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505 2017-06-14 23:02:21 PDT INFO remoteWorkerThread_2: SYNC 5000000030 done in 0.027 seconds 2017-06-14 23:02:25 PDT INFO remoteWorkerThread_1: SYNC 5000000054 done in 0.007 seconds 2017-06-14 23:02:25 PDT INFO remoteWorkerThread_2: SYNC 5000000031 done in 0.008 seconds 2017-06-14 23:02:36 PDT INFO remoteWorkerThread_1: SYNC 5000000055 done in 0.008 seconds 2017-06-14 23:02:39 PDT INFO remoteWorkerThread_2: SYNC 5000000032 done in 0.005 seconds 2017-06-14 23:02:43 PDT INFO remoteWorkerThread_1: SYNC 5000000056 done in 0.005 seconds 2017-06-14 23:02:47 PDT INFO remoteWorkerThread_2: SYNC 5000000033 done in 0.005 seconds 2017-06-14 23:02:54 PDT INFO remoteWorkerThread_1: SYNC 5000000057 done in 0.004 seconds 2017-06-14 23:02:54 PDT INFO remoteWorkerThread_2: SYNC 5000000034 done in 0.005 seconds 2017-06-14 23:03:05 PDT INFO remoteWorkerThread_1: SYNC 5000000058 done in 0.005 seconds 2017-06-14 23:03:08 PDT INFO remoteWorkerThread_2: SYNC 5000000035 done in 0.005 seconds 2017-06-14 23:03:16 PDT CONFIG storeSet: set_id=1 set_origin=1 set_comment= 'Set 1 (il_longlive_schema) for region_il' 2017-06-14 23:03:16 PDT INFO remoteWorkerThread_2: SYNC 5000000036 done in 0.005 seconds 2017-06-14 23:03:16 PDT CONFIG remoteWorkerThread_1: update provider configuration 2017-06-14 23:03:16 PDT CONFIG remoteWorkerThread_1: connection for provider 1 terminated 2017-06-14 23:03:16 PDT CONFIG remoteWorkerThread_1: disconnecting from data provider 1 2017-06-14 23:03:16 PDT CONFIG remoteWorkerThread_1: connection for provider 2 terminated 2017-06-14 23:03:16 PDT CONFIG remoteWorkerThread_1: disconnecting from data provider 2 2017-06-14 23:03:16 PDT CONFIG remoteWorkerThread_1: update provider configuration 2017-06-14 23:03:16 PDT CONFIG version for "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505 2017-06-14 23:03:16 PDT INFO remoteWorkerThread_1: SYNC 5000000060 done in 0.012 seconds 2017-06-14 23:03:16 PDT INFO remoteWorkerThread_1: SYNC 5000000062 done in 0.002 seconds 2017-06-14 23:03:16 PDT INFO remoteWorkerThread_1: SYNC 5000000064 done in 0.002 seconds 2017-06-14 23:03:16 PDT INFO remoteWorkerThread_1: SYNC 5000000066 done in 0.002 seconds 2017-06-14 23:03:16 PDT INFO remoteWorkerThread_1: SYNC 5000000068 done in 0.003 seconds 2017-06-14 23:03:16 PDT INFO remoteWorkerThread_1: SYNC 5000000070 done in 0.002 seconds 2017-06-14 23:03:16 PDT INFO remoteWorkerThread_1: SYNC 5000000072 done in 0.002 seconds 2017-06-14 23:03:16 PDT INFO remoteWorkerThread_1: SYNC 5000000074 done in 0.002 seconds 2017-06-14 23:03:16 PDT INFO remoteWorkerThread_1: SYNC 5000000076 done in 0.002 seconds 2017-06-14 23:03:29 PDT CONFIG remoteWorkerThread_1: update provider configuration 2017-06-14 23:03:29 PDT INFO remoteWorkerThread_2: SYNC 5000000037 done in 0.005 seconds 2017-06-14 23:03:29 PDT CONFIG version for "host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505 2017-06-14 23:03:29 PDT INFO remoteWorkerThread_1: SYNC 5000000077 done in 0.015 seconds 2017-06-14 23:03:42 PDT INFO remoteWorkerThread_2: SYNC 5000000038 done in 0.013 seconds 2017-06-14 23:03:42 PDT INFO remoteWorkerThread_1: SYNC 5000000078 done in 0.015 seconds 2017-06-14 23:03:43 PDT INFO remoteWorkerThread_1: SYNC 5000000079 done in 0.004 seconds 2017-06-14 23:03:43 PDT INFO remoteWorkerThread_2: SYNC 5000000039 done in 0.005 seconds 2017-06-14 23:03:56 PDT INFO remoteWorkerThread_1: SYNC 5000000080 done in 0.004 seconds 2017-06-14 23:03:56 PDT INFO remoteWorkerThread_2: SYNC 5000000040 done in 0.005 seconds 2017-06-14 23:04:04 PDT INFO remoteWorkerThread_2: SYNC 5000000041 done in 0.005 seconds 2017-06-14 23:04:08 PDT INFO remoteWorkerThread_1: SYNC 5000000081 done in 0.005 seconds 2017-06-14 23:04:16 PDT INFO remoteWorkerThread_1: SYNC 5000000082 done in 0.005 seconds 2017-06-14 23:04:16 PDT INFO remoteWorkerThread_2: SYNC 5000000042 done in 0.006 seconds 2017-06-14 23:04:29 PDT INFO remoteWorkerThread_2: SYNC 5000000043 done in 0.004 seconds 2017-06-14 23:04:29 PDT INFO remoteWorkerThread_1: SYNC 5000000083 done in 0.005 seconds 2017-06-14 23:04:37 PDT INFO remoteWorkerThread_2: SYNC 5000000044 done in 0.004 seconds 2017-06-14 23:04:37 PDT INFO remoteWorkerThread_1: SYNC 5000000084 done in 0.005 seconds 2017-06-14 23:04:48 PDT INFO remoteWorkerThread_2: SYNC 5000000045 done in 0.004 seconds 2017-06-14 23:04:48 PDT INFO remoteWorkerThread_1: SYNC 5000000085 done in 0.005 seconds 2017-06-14 23:05:02 PDT INFO remoteWorkerThread_1: SYNC 5000000086 done in 0.004 seconds 2017-06-14 23:05:02 PDT INFO remoteWorkerThread_2: SYNC 5000000046 done in 0.005 seconds 2017-06-14 23:05:03 PDT INFO remoteWorkerThread_1: SYNC 5000000087 done in 0.005 seconds 2017-06-14 23:05:05 PDT INFO remoteWorkerThread_2: SYNC 5000000047 done in 0.005 seconds 2017-06-14 23:05:14 PDT INFO remoteWorkerThread_1: SYNC 5000000088 done in 0.005 seconds 2017-06-14 23:05:19 PDT INFO remoteWorkerThread_2: SYNC 5000000048 done in 0.004 seconds 2017-06-14 23:05:23 PDT INFO remoteWorkerThread_2: SYNC 5000000049 done in 0.005 seconds 2017-06-14 23:05:25 PDT INFO remoteWorkerThread_1: SYNC 5000000089 done in 0.004 seconds 2017-06-14 23:05:34 PDT INFO remoteWorkerThread_2: SYNC 5000000050 done in 0.006 seconds 2017-06-14 23:05:38 PDT INFO remoteWorkerThread_1: SYNC 5000000090 done in 0.005 seconds 2017-06-14 23:05:46 PDT INFO remoteWorkerThread_1: SYNC 5000000091 done in 0.004 seconds 2017-06-14 23:05:46 PDT INFO remoteWorkerThread_2: SYNC 5000000051 done in 0.005 seconds 2017-06-14 23:05:59 PDT INFO remoteWorkerThread_1: SYNC 5000000092 done in 0.004 seconds 2017-06-14 23:05:59 PDT INFO remoteWorkerThread_2: SYNC 5000000052 done in 0.005 seconds 2017-06-14 23:06:12 PDT INFO remoteWorkerThread_2: SYNC 5000000053 done in 0.004 seconds 2017-06-14 23:06:12 PDT INFO remoteWorkerThread_1: SYNC 5000000093 done in 0.004 seconds 2017-06-14 23:06:15 PDT INFO remoteWorkerThread_2: SYNC 5000000054 done in 0.004 seconds 2017-06-14 23:06:15 PDT INFO remoteWorkerThread_1: SYNC 5000000094 done in 0.005 seconds 2017-06-14 23:06:28 PDT INFO remoteWorkerThread_2: SYNC 5000000055 done in 0.005 seconds 2017-06-14 23:06:28 PDT INFO remoteWorkerThread_1: SYNC 5000000095 done in 0.005 seconds 2017-06-14 23:06:36 PDT INFO remoteWorkerThread_1: SYNC 5000000096 done in 0.004 seconds 2017-06-14 23:06:36 PDT INFO remoteWorkerThread_2: SYNC 5000000056 done in 0.005 seconds 2017-06-14 23:06:47 PDT INFO remoteWorkerThread_1: SYNC 5000000097 done in 0.005 seconds 2017-06-14 23:06:48 PDT INFO remoteWorkerThread_2: SYNC 5000000057 done in 0.004 seconds 2017-06-14 23:06:58 PDT INFO remoteWorkerThread_1: SYNC 5000000098 done in 0.004 seconds 2017-06-14 23:06:58 PDT INFO remoteWorkerThread_2: SYNC 5000000058 done in 0.005 seconds 2017-06-14 23:07:12 PDT INFO remoteWorkerThread_1: SYNC 5000000099 done in 0.004 seconds 2017-06-14 23:07:12 PDT INFO remoteWorkerThread_2: SYNC 5000000059 done in 0.005 seconds 2017-06-14 23:07:15 PDT INFO remoteWorkerThread_2: SYNC 5000000060 done in 0.004 seconds 2017-06-14 23:07:15 PDT INFO remoteWorkerThread_1: SYNC 5000000100 done in 0.005 seconds 2017-06-14 23:07:28 PDT INFO remoteWorkerThread_1: SYNC 5000000101 done in 0.060 seconds 2017-06-14 23:07:28 PDT INFO remoteWorkerThread_2: SYNC 5000000061 done in 0.062 seconds 2017-06-14 23:07:36 PDT INFO remoteWorkerThread_2: SYNC 5000000062 done in 0.004 seconds 2017-06-14 23:07:36 PDT INFO remoteWorkerThread_1: SYNC 5000000102 done in 0.005 seconds 2017-06-14 23:07:47 PDT INFO remoteWorkerThread_1: SYNC 5000000103 done in 0.008 seconds 2017-06-14 23:07:47 PDT INFO remoteWorkerThread_2: SYNC 5000000063 done in 0.009 seconds 2017-06-14 23:07:58 PDT INFO remoteWorkerThread_1: SYNC 5000000104 done in 0.005 seconds 2017-06-14 23:07:58 PDT INFO remoteWorkerThread_2: SYNC 5000000064 done in 0.005 seconds 2017-06-14 23:08:09 PDT INFO remoteWorkerThread_1: SYNC 5000000105 done in 0.007 seconds 2017-06-14 23:08:09 PDT INFO remoteWorkerThread_2: SYNC 5000000065 done in 0.009 seconds 2017-06-14 23:08:13 PDT INFO remoteWorkerThread_1: SYNC 5000000106 done in 0.005 seconds 2017-06-14 23:08:19 PDT INFO remoteWorkerThread_2: SYNC 5000000066 done in 0.010 seconds 2017-06-14 23:08:23 PDT INFO remoteWorkerThread_2: SYNC 5000000067 done in 0.004 seconds 2017-06-14 23:08:24 PDT INFO remoteWorkerThread_1: SYNC 5000000107 done in 0.008 seconds 2017-06-14 23:08:37 PDT INFO remoteWorkerThread_2: SYNC 5000000068 done in 0.006 seconds 2017-06-14 23:08:41 PDT INFO remoteWorkerThread_1: SYNC 5000000108 done in 0.005 seconds 2017-06-14 23:08:45 PDT INFO remoteWorkerThread_1: SYNC 5000000109 done in 0.005 seconds 2017-06-14 23:08:49 PDT INFO remoteWorkerThread_2: SYNC 5000000069 done in 0.005 seconds 2017-06-14 23:08:57 PDT INFO remoteWorkerThread_1: SYNC 5000000110 done in 0.006 seconds 2017-06-14 23:08:57 PDT INFO remoteWorkerThread_2: SYNC 5000000070 done in 0.006 seconds 2017-06-14 23:09:08 PDT INFO remoteWorkerThread_2: SYNC 5000000071 done in 0.005 seconds 2017-06-14 23:09:08 PDT INFO remoteWorkerThread_1: SYNC 5000000111 done in 0.005 seconds 2017-06-14 23:09:19 PDT INFO remoteWorkerThread_1: SYNC 5000000112 done in 0.004 seconds 2017-06-14 23:09:19 PDT INFO remoteWorkerThread_2: SYNC 5000000072 done in 0.005 seconds 2017-06-14 23:09:29 PDT INFO remoteWorkerThread_2: SYNC 5000000073 done in 0.004 seconds 2017-06-14 23:09:29 PDT INFO remoteWorkerThread_1: SYNC 5000000113 done in 0.005 seconds 2017-06-14 23:09:33 PDT INFO remoteWorkerThread_2: SYNC 5000000074 done in 0.005 seconds 2017-06-14 23:09:34 PDT INFO remoteWorkerThread_1: SYNC 5000000114 done in 0.005 seconds 2017-06-14 23:09:43 PDT INFO remoteWorkerThread_2: SYNC 5000000075 done in 0.005 seconds 2017-06-14 23:09:47 PDT INFO remoteWorkerThread_1: SYNC 5000000115 done in 0.004 seconds 2017-06-14 23:09:55 PDT INFO remoteWorkerThread_2: SYNC 5000000076 done in 0.005 seconds 2017-06-14 23:10:01 PDT INFO remoteWorkerThread_1: SYNC 5000000116 done in 0.005 seconds 2017-06-14 23:10:05 PDT INFO remoteWorkerThread_1: SYNC 5000000117 done in 0.005 seconds 2017-06-14 23:10:06 PDT INFO remoteWorkerThread_2: SYNC 5000000077 done in 0.005 seconds 2017-06-14 23:10:14 PDT INFO remoteWorkerThread_1: SYNC 5000000118 done in 0.005 seconds 2017-06-14 23:10:18 PDT INFO remoteWorkerThread_2: SYNC 5000000078 done in 0.005 seconds 2017-06-14 23:10:26 PDT INFO remoteWorkerThread_2: SYNC 5000000079 done in 0.005 seconds 2017-06-14 23:10:30 PDT INFO remoteWorkerThread_1: SYNC 5000000119 done in 0.005 seconds 2017-06-14 23:10:34 PDT INFO remoteWorkerThread_2: SYNC 5000000080 done in 0.005 seconds NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=19256 CONTEXT: SQL statement "SELECT " _region_il ".cleanupNodelock()" PL /pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM NOTICE: Slony-I: Logswitch to sl_log_2 initiated CONTEXT: SQL statement "SELECT " _region_il ".logswitch_start()" PL /pgSQL function _region_il.cleanupevent(interval) line 97 at PERFORM 2017-06-14 23:10:39 PDT INFO cleanupThread: 0.021 seconds for cleanupEvent() 2017-06-14 23:10:39 PDT INFO remoteWorkerThread_1: SYNC 5000000120 done in 0.005 seconds 2017-06-14 23:10:45 PDT INFO remoteWorkerThread_1: SYNC 5000000121 done in 0.005 seconds 2017-06-14 23:10:45 PDT INFO remoteWorkerThread_2: SYNC 5000000081 done in 0.005 seconds 2017-06-14 23:10:56 PDT INFO remoteWorkerThread_2: SYNC 5000000082 done in 0.005 seconds 2017-06-14 23:10:57 PDT INFO remoteWorkerThread_1: SYNC 5000000122 done in 0.004 seconds 2017-06-14 23:11:07 PDT INFO remoteWorkerThread_1: SYNC 5000000123 done in 0.005 seconds 2017-06-14 23:11:08 PDT INFO remoteWorkerThread_2: SYNC 5000000083 done in 0.004 seconds 2017-06-14 23:11:18 PDT INFO remoteWorkerThread_2: SYNC 5000000084 done in 0.005 seconds 2017-06-14 23:11:19 PDT INFO remoteWorkerThread_1: SYNC 5000000124 done in 0.004 seconds 2017-06-14 23:11:23 PDT INFO remoteWorkerThread_2: SYNC 5000000085 done in 0.004 seconds 2017-06-14 23:11:29 PDT INFO remoteWorkerThread_1: SYNC 5000000125 done in 0.005 seconds 2017-06-14 23:11:34 PDT INFO remoteWorkerThread_1: SYNC 5000000126 done in 0.005 seconds 2017-06-14 23:11:34 PDT INFO remoteWorkerThread_2: SYNC 5000000086 done in 0.005 seconds 2017-06-14 23:11:45 PDT INFO remoteWorkerThread_1: SYNC 5000000127 done in 0.004 seconds 2017-06-14 23:11:45 PDT INFO remoteWorkerThread_2: SYNC 5000000087 done in 0.004 seconds 2017-06-14 23:11:56 PDT INFO remoteWorkerThread_2: SYNC 5000000088 done in 0.005 seconds 2017-06-14 23:11:56 PDT CONFIG storeListen: li_origin=1 li_receiver=3 li_provider=1 2017-06-14 23:11:56 PDT CONFIG storeListen: li_origin=2 li_receiver=3 li_provider=2 2017-06-14 23:11:56 PDT CONFIG storeListen: li_origin=2 li_receiver=3 li_provider=1 2017-06-14 23:11:56 PDT CONFIG storeListen: li_origin=1 li_receiver=3 li_provider=2 2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_1: update provider configuration 2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_1: connection for provider 1 terminated 2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_1: disconnecting from data provider 1 2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_1: connection for provider 2 terminated 2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_1: disconnecting from data provider 2 2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_2: update provider configuration 2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_2: connection for provider 2 terminated 2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_2: disconnecting from data provider 2 2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_2: connection for provider 1 terminated 2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_2: disconnecting from data provider 1 2017-06-14 23:12:00 PDT CONFIG storeListen: li_origin=1 li_receiver=3 li_provider=1 2017-06-14 23:12:00 PDT CONFIG storeListen: li_origin=2 li_receiver=3 li_provider=2 2017-06-14 23:12:00 PDT CONFIG storeListen: li_origin=2 li_receiver=3 li_provider=1 2017-06-14 23:12:00 PDT CONFIG storeListen: li_origin=1 li_receiver=3 li_provider=2 2017-06-14 23:12:00 PDT CONFIG remoteWorkerThread_1: update provider configuration 2017-06-14 23:12:00 PDT CONFIG remoteWorkerThread_2: update provider configuration 2017-06-14 23:12:00 PDT CONFIG remoteWorkerThread_1: update provider configuration 2017-06-14 23:12:00 PDT CONFIG version for "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505 2017-06-14 23:12:00 PDT INFO remoteWorkerThread_1: SYNC 5000000130 done in 0.012 seconds 2017-06-14 23:12:08 PDT CONFIG remoteWorkerThread_2: update provider configuration 2017-06-14 23:12:08 PDT INFO remoteWorkerThread_1: SYNC 5000000131 done in 0.003 seconds 2017-06-14 23:12:08 PDT CONFIG version for "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505 2017-06-14 23:12:08 PDT INFO remoteWorkerThread_2: SYNC 5000000089 done in 0.014 seconds 2017-06-14 23:12:19 PDT INFO remoteWorkerThread_1: SYNC 5000000132 done in 0.002 seconds 2017-06-14 23:12:30 PDT INFO remoteWorkerThread_1: SYNC 5000000133 done in 0.003 seconds 2017-06-14 23:12:34 PDT INFO remoteWorkerThread_1: SYNC 5000000134 done in 0.003 seconds 2017-06-14 23:12:34 PDT INFO remoteWorkerThread_2: SYNC 5000000090 done in 0.003 seconds 2017-06-14 23:12:38 PDT INFO remoteWorkerThread_2: SYNC 5000000091 done in 0.003 seconds 2017-06-14 23:12:49 PDT INFO remoteWorkerThread_1: SYNC 5000000135 done in 0.002 seconds 2017-06-14 23:12:53 PDT CONFIG remoteWorkerThread_2: update provider configuration 2017-06-14 23:12:53 PDT CONFIG version for "host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505 2017-06-14 23:12:56 PDT INFO remoteWorkerThread_1: SYNC 5000000136 done in 0.002 seconds 2017-06-14 23:13:02 PDT INFO remoteWorkerThread_2: SYNC 5000000092 done in 8.409 seconds 2017-06-14 23:13:07 PDT INFO remoteWorkerThread_1: SYNC 5000000137 done in 0.003 seconds 2017-06-14 23:13:07 PDT INFO remoteWorkerThread_2: SYNC 5000000093 done in 0.005 seconds 2017-06-14 23:13:18 PDT INFO remoteWorkerThread_1: SYNC 5000000138 done in 0.003 seconds 2017-06-14 23:13:18 PDT INFO remoteWorkerThread_2: SYNC 5000000094 done in 0.005 seconds 2017-06-14 23:13:29 PDT INFO remoteWorkerThread_1: SYNC 5000000139 done in 0.003 seconds 2017-06-14 23:13:29 PDT INFO remoteWorkerThread_2: SYNC 5000000095 done in 0.005 seconds 2017-06-14 23:13:40 PDT INFO remoteWorkerThread_1: SYNC 5000000140 done in 0.003 seconds 2017-06-14 23:13:40 PDT INFO remoteWorkerThread_2: SYNC 5000000096 done in 0.006 seconds 2017-06-14 23:13:44 PDT INFO remoteWorkerThread_1: SYNC 5000000141 done in 0.003 seconds 2017-06-14 23:13:48 PDT INFO remoteWorkerThread_2: SYNC 5000000097 done in 0.006 seconds 2017-06-14 23:13:58 PDT INFO remoteWorkerThread_2: SYNC 5000000098 done in 0.005 seconds 2017-06-14 23:14:02 PDT INFO remoteWorkerThread_1: SYNC 5000000142 done in 0.003 seconds 2017-06-14 23:14:06 PDT INFO remoteWorkerThread_1: SYNC 5000000143 done in 0.002 seconds 2017-06-14 23:14:16 PDT INFO remoteWorkerThread_2: SYNC 5000000099 done in 0.005 seconds 2017-06-14 23:14:20 PDT INFO remoteWorkerThread_2: SYNC 5000000100 done in 0.007 seconds 2017-06-14 23:14:20 PDT INFO remoteWorkerThread_1: SYNC 5000000144 done in 0.003 seconds 2017-06-14 23:14:28 PDT INFO remoteWorkerThread_1: SYNC 5000000145 done in 0.003 seconds 2017-06-14 23:14:34 PDT INFO remoteWorkerThread_2: SYNC 5000000101 done in 0.015 seconds 2017-06-14 23:14:39 PDT INFO remoteWorkerThread_1: SYNC 5000000146 done in 0.003 seconds 2017-06-14 23:14:50 PDT INFO remoteWorkerThread_1: SYNC 5000000147 done in 0.003 seconds 2017-06-14 23:14:51 PDT CONFIG storeSubscribe: sub_set=1 sub_provider=2 sub_forward= 't' 2017-06-14 23:14:51 PDT CONFIG remoteWorkerThread_2: update provider configuration 2017-06-14 23:14:51 PDT CONFIG remoteWorkerThread_2: connection for provider 1 terminated 2017-06-14 23:14:51 PDT CONFIG remoteWorkerThread_2: disconnecting from data provider 1 2017-06-14 23:14:51 PDT CONFIG remoteWorkerThread_2: connection for provider 2 terminated 2017-06-14 23:14:51 PDT CONFIG remoteWorkerThread_2: disconnecting from data provider 2 2017-06-14 23:14:51 PDT CONFIG storeListen: li_origin=1 li_receiver=3 li_provider=1 2017-06-14 23:14:51 PDT CONFIG storeListen: li_origin=2 li_receiver=3 li_provider=2 2017-06-14 23:14:51 PDT CONFIG storeListen: li_origin=2 li_receiver=3 li_provider=1 2017-06-14 23:14:51 PDT CONFIG storeListen: li_origin=1 li_receiver=3 li_provider=2 2017-06-14 23:14:51 PDT CONFIG remoteWorkerThread_1: update provider configuration 2017-06-14 23:14:51 PDT CONFIG remoteWorkerThread_1: connection for provider 1 terminated 2017-06-14 23:14:51 PDT CONFIG remoteWorkerThread_1: disconnecting from data provider 1 2017-06-14 23:14:51 PDT CONFIG remoteWorkerThread_2: update provider configuration 2017-06-14 23:14:51 PDT WARN remoteWorkerThread_1: copy set : data provider 2 only on sync 5000000128 - sleep 5 seconds 2017-06-14 23:14:55 PDT CONFIG remoteWorkerThread_2: update provider configuration 2017-06-14 23:14:55 PDT CONFIG version for "host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505 2017-06-14 23:14:55 PDT INFO remoteWorkerThread_2: SYNC 5000000102 done in 0.012 seconds 2017-06-14 23:14:56 PDT INFO copy_set 1 - omit=f - bool=0 2017-06-14 23:14:56 PDT INFO omit is FALSE 2017-06-14 23:14:56 PDT CONFIG version for "host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505 2017-06-14 23:14:56 PDT CONFIG remoteWorkerThread_1: connected to provider DB 2017-06-14 23:14:56 PDT CONFIG remoteWorkerThread_1: prepare to copy table "il_longlive" . "entity" 2017-06-14 23:14:56 PDT CONFIG remoteWorkerThread_1: prepare to copy table "il_longlive" . "entity_result" 2017-06-14 23:14:56 PDT CONFIG remoteWorkerThread_1: prepare to copy table "il_longlive" . "result" 2017-06-14 23:14:56 PDT CONFIG remoteWorkerThread_1: prepare to copy table "il_longlive" . "result_logbook" 2017-06-14 23:14:56 PDT CONFIG remoteWorkerThread_1: all tables for set 1 found on subscriber 2017-06-14 23:14:56 PDT CONFIG remoteWorkerThread_1: copy sequence "il_longlive" . "entity_entity_id_seq" 2017-06-14 23:14:56 PDT CONFIG remoteWorkerThread_1: copy sequence "il_longlive" . "entity_result_id_seq" 2017-06-14 23:14:56 PDT CONFIG remoteWorkerThread_1: copy sequence "il_longlive" . "result_id_seq" 2017-06-14 23:14:56 PDT CONFIG remoteWorkerThread_1: copy sequence "il_longlive" . "result_logbook_record_id_seq" 2017-06-14 23:14:56 PDT CONFIG remoteWorkerThread_1: copy table "il_longlive" . "entity" 2017-06-14 23:14:56 PDT CONFIG remoteWorkerThread_1: Begin COPY of table "il_longlive" . "entity" NOTICE: truncate of "il_longlive" . "entity" succeeded 2017-06-14 23:15:03 PDT CONFIG remoteWorkerThread_2: update provider configuration 2017-06-14 23:15:03 PDT CONFIG version for "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505 2017-06-14 23:15:03 PDT INFO remoteWorkerThread_2: SYNC 5000000103 done in 0.033 seconds 2017-06-14 23:15:17 PDT CONFIG remoteWorkerThread_1: 125045870 bytes copied for table "il_longlive" . "entity" 2017-06-14 23:15:19 PDT INFO remoteWorkerThread_2: SYNC 5000000104 done in 0.006 seconds 2017-06-14 23:15:22 PDT INFO remoteWorkerThread_2: SYNC 5000000105 done in 0.005 seconds 2017-06-14 23:15:25 PDT CONFIG remoteWorkerThread_1: 28.669 seconds to copy table "il_longlive" . "entity" 2017-06-14 23:15:25 PDT CONFIG remoteWorkerThread_1: copy table "il_longlive" . "entity_result" 2017-06-14 23:15:25 PDT CONFIG remoteWorkerThread_1: Begin COPY of table "il_longlive" . "entity_result" NOTICE: truncate of "il_longlive" . "entity_result" succeeded 2017-06-14 23:15:34 PDT INFO remoteWorkerThread_2: SYNC 5000000106 done in 0.004 seconds 2017-06-14 23:15:37 PDT CONFIG remoteWorkerThread_1: 66996342 bytes copied for table "il_longlive" . "entity_result" 2017-06-14 23:15:45 PDT INFO remoteWorkerThread_2: SYNC 5000000107 done in 0.012 seconds 2017-06-14 23:15:56 PDT INFO remoteWorkerThread_2: SYNC 5000000108 done in 0.004 seconds 2017-06-14 23:16:07 PDT INFO remoteWorkerThread_2: SYNC 5000000109 done in 0.005 seconds 2017-06-14 23:16:10 PDT CONFIG remoteWorkerThread_1: 45.106 seconds to copy table "il_longlive" . "entity_result" 2017-06-14 23:16:10 PDT CONFIG remoteWorkerThread_1: copy table "il_longlive" . "result" 2017-06-14 23:16:10 PDT CONFIG remoteWorkerThread_1: Begin COPY of table "il_longlive" . "result" NOTICE: truncate of "il_longlive" . "result" succeeded 2017-06-14 23:16:11 PDT INFO remoteWorkerThread_2: SYNC 5000000110 done in 0.015 seconds 2017-06-14 23:16:27 PDT CONFIG remoteWorkerThread_1: 105961503 bytes copied for table "il_longlive" . "result" 2017-06-14 23:16:29 PDT INFO remoteWorkerThread_2: SYNC 5000000111 done in 0.005 seconds 2017-06-14 23:16:32 PDT INFO remoteWorkerThread_2: SYNC 5000000112 done in 0.005 seconds 2017-06-14 23:16:35 PDT CONFIG remoteWorkerThread_1: 25.058 seconds to copy table "il_longlive" . "result" 2017-06-14 23:16:35 PDT CONFIG remoteWorkerThread_1: copy table "il_longlive" . "result_logbook" 2017-06-14 23:16:35 PDT CONFIG remoteWorkerThread_1: Begin COPY of table "il_longlive" . "result_logbook" NOTICE: truncate of "il_longlive" . "result_logbook" succeeded 2017-06-14 23:16:35 PDT CONFIG remoteWorkerThread_1: 79 bytes copied for table "il_longlive" . "result_logbook" 2017-06-14 23:16:35 PDT CONFIG remoteWorkerThread_1: 0.079 seconds to copy table "il_longlive" . "result_logbook" 2017-06-14 23:16:35 PDT CONFIG remoteWorkerThread_1: set last_value of sequence 101 ( "il_longlive" . "entity_entity_id_seq" ) to 2528612 2017-06-14 23:16:35 PDT CONFIG remoteWorkerThread_1: set last_value of sequence 103 ( "il_longlive" . "result_id_seq" ) to 2611527 2017-06-14 23:16:35 PDT CONFIG remoteWorkerThread_1: set last_value of sequence 102 ( "il_longlive" . "entity_result_id_seq" ) to 2611527 2017-06-14 23:16:35 PDT CONFIG remoteWorkerThread_1: set last_value of sequence 104 ( "il_longlive" . "result_logbook_record_id_seq" ) to 1 2017-06-14 23:16:35 PDT INFO remoteWorkerThread_1: 0.002 seconds to copy sequences 2017-06-14 23:16:35 PDT INFO remoteWorkerThread_1: copy_set SYNC found, use event seqno 5000000147. 2017-06-14 23:16:35 PDT INFO remoteWorkerThread_1: 0.015 seconds to build initial setsync status 2017-06-14 23:16:35 PDT INFO copy_set 1 done in 98.961 seconds 2017-06-14 23:16:35 PDT CONFIG enableSubscription: sub_set=1 2017-06-14 23:16:35 PDT CONFIG remoteWorkerThread_2: update provider configuration 2017-06-14 23:16:35 PDT CONFIG remoteWorkerThread_2: connection for provider 2 terminated 2017-06-14 23:16:35 PDT CONFIG remoteWorkerThread_2: disconnecting from data provider 2 2017-06-14 23:16:35 PDT CONFIG remoteWorkerThread_2: connection for provider 1 terminated 2017-06-14 23:16:35 PDT CONFIG remoteWorkerThread_2: disconnecting from data provider 1 2017-06-14 23:16:35 PDT CONFIG storeListen: li_origin=2 li_receiver=3 li_provider=2 2017-06-14 23:16:35 PDT CONFIG storeListen: li_origin=2 li_receiver=3 li_provider=1 2017-06-14 23:16:35 PDT CONFIG storeListen: li_origin=1 li_receiver=3 li_provider=2 2017-06-14 23:16:35 PDT CONFIG remoteWorkerThread_1: update provider configuration 2017-06-14 23:16:35 PDT CONFIG remoteWorkerThread_1: added active set 1 to provider 2 2017-06-14 23:16:35 PDT CONFIG remoteWorkerThread_2: update provider configuration 2017-06-14 23:16:35 PDT CONFIG remoteWorkerThread_1: update provider configuration 2017-06-14 23:16:35 PDT CONFIG remoteWorkerThread_1: added active set 1 to provider 2 2017-06-14 23:16:35 PDT CONFIG version for "host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505 2017-06-14 23:16:35 PDT CONFIG version for "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505 2017-06-14 23:16:35 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2 2017-06-14 23:16:35 PDT INFO remoteWorkerThread_1: SYNC 5000000151 done in 0.042 seconds 2017-06-14 23:16:35 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2 2017-06-14 23:16:35 PDT INFO remoteWorkerThread_1: SYNC 5000000155 done in 0.006 seconds 2017-06-14 23:16:35 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2 2017-06-14 23:16:35 PDT INFO remoteWorkerThread_1: SYNC 5000000159 done in 0.005 seconds 2017-06-14 23:16:37 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2 2017-06-14 23:16:37 PDT INFO remoteWorkerThread_1: SYNC 5000000160 done in 0.008 seconds 2017-06-14 23:16:41 PDT CONFIG remoteWorkerThread_2: update provider configuration 2017-06-14 23:16:41 PDT CONFIG version for "host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505 2017-06-14 23:16:41 PDT INFO remoteWorkerThread_2: SYNC 5000000113 done in 0.013 seconds 2017-06-14 23:16:55 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2 2017-06-14 23:16:55 PDT INFO remoteWorkerThread_2: SYNC 5000000114 done in 0.005 seconds 2017-06-14 23:16:55 PDT INFO remoteWorkerThread_1: SYNC 5000000161 done in 0.008 seconds 2017-06-14 23:16:59 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2 2017-06-14 23:16:59 PDT INFO remoteWorkerThread_1: SYNC 5000000162 done in 0.007 seconds 2017-06-14 23:17:03 PDT INFO remoteWorkerThread_2: SYNC 5000000115 done in 0.003 seconds 2017-06-14 23:17:14 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2 2017-06-14 23:17:14 PDT INFO remoteWorkerThread_2: SYNC 5000000116 done in 0.004 seconds 2017-06-14 23:17:14 PDT INFO remoteWorkerThread_1: SYNC 5000000163 done in 0.008 seconds 2017-06-14 23:17:28 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2 2017-06-14 23:17:28 PDT INFO remoteWorkerThread_2: SYNC 5000000117 done in 0.004 seconds 2017-06-14 23:17:28 PDT INFO remoteWorkerThread_1: SYNC 5000000165 done in 0.008 seconds 2017-06-14 23:17:32 PDT INFO remoteWorkerThread_2: SYNC 5000000118 done in 0.003 seconds 2017-06-14 23:17:36 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2 2017-06-14 23:17:36 PDT INFO remoteWorkerThread_1: SYNC 5000000166 done in 0.007 seconds 2017-06-14 23:17:47 PDT INFO remoteWorkerThread_2: SYNC 5000000119 done in 0.002 seconds 2017-06-14 23:17:48 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2 2017-06-14 23:17:48 PDT INFO remoteWorkerThread_1: SYNC 5000000167 done in 0.007 seconds 2017-06-14 23:17:52 PDT INFO remoteWorkerThread_2: SYNC 5000000120 done in 0.003 seconds 2017-06-14 23:18:03 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2 2017-06-14 23:18:03 PDT INFO remoteWorkerThread_2: SYNC 5000000121 done in 0.004 seconds 2017-06-14 23:18:03 PDT INFO remoteWorkerThread_1: SYNC 5000000168 done in 0.008 seconds 2017-06-14 23:18:14 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2 2017-06-14 23:18:14 PDT INFO remoteWorkerThread_2: SYNC 5000000122 done in 0.004 seconds 2017-06-14 23:18:14 PDT INFO remoteWorkerThread_1: SYNC 5000000169 done in 0.007 seconds 2017-06-14 23:18:18 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2 2017-06-14 23:18:18 PDT INFO remoteWorkerThread_1: SYNC 5000000170 done in 0.022 seconds 2017-06-14 23:18:26 PDT INFO remoteWorkerThread_2: SYNC 5000000123 done in 0.003 seconds 2017-06-14 23:18:30 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2 2017-06-14 23:18:30 PDT INFO remoteWorkerThread_1: SYNC 5000000171 done in 0.007 seconds 2017-06-14 23:18:31 PDT INFO remoteWorkerThread_2: SYNC 5000000124 done in 0.002 seconds 2017-06-14 23:18:44 PDT CONFIG remoteWorkerThread_2: update provider configuration 2017-06-14 23:18:44 PDT CONFIG version for "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505 2017-06-14 23:18:44 PDT INFO remoteWorkerThread_2: SYNC 5000000125 done in 0.015 seconds 2017-06-14 23:18:49 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2 2017-06-14 23:18:49 PDT INFO remoteWorkerThread_1: SYNC 5000000172 done in 0.007 seconds 2017-06-14 23:18:53 PDT INFO remoteWorkerThread_2: SYNC 5000000126 done in 0.004 seconds 2017-06-14 23:18:54 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2 2017-06-14 23:18:54 PDT INFO remoteWorkerThread_1: SYNC 5000000173 done in 0.006 seconds 2017-06-14 23:18:58 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2 2017-06-14 23:18:58 PDT INFO remoteWorkerThread_1: SYNC 5000000174 done in 0.006 seconds 2017-06-14 23:19:02 PDT INFO remoteWorkerThread_2: SYNC 5000000127 done in 0.012 seconds 2017-06-14 23:19:10 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2 2017-06-14 23:19:10 PDT INFO remoteWorkerThread_1: SYNC 5000000175 done in 0.015 seconds 2017-06-14 23:19:11 PDT INFO remoteWorkerThread_2: SYNC 5000000128 done in 0.004 seconds 2017-06-14 23:19:22 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2 2017-06-14 23:19:22 PDT INFO remoteWorkerThread_2: SYNC 5000000129 done in 0.005 seconds 2017-06-14 23:19:22 PDT INFO remoteWorkerThread_1: SYNC 5000000176 done in 0.007 seconds 2017-06-14 23:19:36 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2 2017-06-14 23:19:36 PDT INFO remoteWorkerThread_2: SYNC 5000000130 done in 0.006 seconds 2017-06-14 23:19:36 PDT INFO remoteWorkerThread_1: SYNC 5000000177 done in 0.007 seconds 2017-06-14 23:19:37 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2 2017-06-14 23:19:37 PDT INFO remoteWorkerThread_1: SYNC 5000000178 done in 0.006 seconds 2017-06-14 23:19:41 PDT INFO remoteWorkerThread_2: SYNC 5000000131 done in 0.005 seconds 2017-06-14 23:19:55 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2 2017-06-14 23:19:55 PDT INFO remoteWorkerThread_2: SYNC 5000000132 done in 0.006 seconds 2017-06-14 23:19:55 PDT INFO remoteWorkerThread_1: SYNC 5000000179 done in 0.007 seconds 2017-06-14 23:20:06 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2 2017-06-14 23:20:06 PDT INFO remoteWorkerThread_2: SYNC 5000000133 done in 0.005 seconds 2017-06-14 23:20:06 PDT INFO remoteWorkerThread_1: SYNC 5000000180 done in 0.007 seconds 2017-06-14 23:20:16 PDT INFO remoteWorkerThread_2: SYNC 5000000134 done in 0.007 seconds 2017-06-14 23:20:20 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2 2017-06-14 23:20:20 PDT INFO remoteWorkerThread_1: SYNC 5000000182 done in 0.007 seconds 2017-06-14 23:20:21 PDT INFO remoteWorkerThread_2: SYNC 5000000135 done in 0.005 seconds 2017-06-14 23:20:35 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2 2017-06-14 23:20:35 PDT INFO remoteWorkerThread_2: SYNC 5000000136 done in 0.006 seconds 2017-06-14 23:20:35 PDT INFO remoteWorkerThread_1: SYNC 5000000183 done in 0.008 seconds NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=19234 CONTEXT: SQL statement "SELECT " _region_il ".cleanupNodelock()" PL /pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=19239 CONTEXT: SQL statement "SELECT " _region_il ".cleanupNodelock()" PL /pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=19244 CONTEXT: SQL statement "SELECT " _region_il ".cleanupNodelock()" PL /pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=19348 CONTEXT: SQL statement "SELECT " _region_il ".cleanupNodelock()" PL /pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=19566 CONTEXT: SQL statement "SELECT " _region_il ".cleanupNodelock()" PL /pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=19567 CONTEXT: SQL statement "SELECT " _region_il ".cleanupNodelock()" PL /pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=19568 CONTEXT: SQL statement "SELECT " _region_il ".cleanupNodelock()" PL /pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=19663 CONTEXT: SQL statement "SELECT " _region_il ".cleanupNodelock()" PL /pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM NOTICE: Slony-I: log switch to sl_log_2 complete - truncate sl_log_1 CONTEXT: PL /pgSQL function _region_il.cleanupevent(interval) line 95 at assignment 2017-06-14 23:20:39 PDT INFO cleanupThread: 0.077 seconds for cleanupEvent() 2017-06-14 23:20:39 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2 2017-06-14 23:20:39 PDT INFO remoteWorkerThread_1: SYNC 5000000184 done in 0.008 seconds 2017-06-14 23:20:43 PDT INFO remoteWorkerThread_2: SYNC 5000000137 done in 0.005 seconds 2017-06-14 23:20:54 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2 2017-06-14 23:20:54 PDT INFO remoteWorkerThread_2: SYNC 5000000138 done in 0.005 seconds 2017-06-14 23:20:54 PDT INFO remoteWorkerThread_1: SYNC 5000000185 done in 0.007 seconds 2017-06-14 23:21:04 PDT INFO remoteWorkerThread_2: SYNC 5000000139 done in 0.009 seconds 2017-06-14 23:21:08 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2 2017-06-14 23:21:08 PDT INFO remoteWorkerThread_1: SYNC 5000000186 done in 0.006 seconds 2017-06-14 23:21:11 PDT INFO remoteWorkerThread_2: SYNC 5000000140 done in 0.005 seconds 2017-06-14 23:21:16 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2 2017-06-14 23:21:16 PDT INFO remoteWorkerThread_1: SYNC 5000000188 done in 0.007 seconds 2017-06-14 23:21:27 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2 2017-06-14 23:21:27 PDT INFO remoteWorkerThread_2: SYNC 5000000141 done in 0.006 seconds 2017-06-14 23:21:27 PDT INFO remoteWorkerThread_1: SYNC 5000000189 done in 0.007 seconds 2017-06-14 23:21:31 PDT INFO remoteWorkerThread_2: SYNC 5000000142 done in 0.005 seconds 2017-06-14 23:21:45 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2 2017-06-14 23:21:45 PDT INFO remoteWorkerThread_2: SYNC 5000000143 done in 0.005 seconds 2017-06-14 23:21:45 PDT INFO remoteWorkerThread_1: SYNC 5000000190 done in 0.007 seconds 2017-06-14 23:21:49 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2 2017-06-14 23:21:49 PDT INFO remoteWorkerThread_1: SYNC 5000000191 done in 0.008 seconds 2017-06-14 23:21:52 PDT INFO remoteWorkerThread_2: SYNC 5000000144 done in 0.005 seconds 2017-06-14 23:22:05 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2 2017-06-14 23:22:05 PDT INFO remoteWorkerThread_2: SYNC 5000000145 done in 0.006 seconds 2017-06-14 23:22:05 PDT INFO remoteWorkerThread_1: SYNC 5000000192 done in 0.008 seconds 2017-06-14 23:22:18 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2 2017-06-14 23:22:18 PDT INFO remoteWorkerThread_2: SYNC 5000000146 done in 0.005 seconds 2017-06-14 23:22:18 PDT INFO remoteWorkerThread_1: SYNC 5000000194 done in 0.007 seconds 2017-06-14 23:22:22 PDT INFO remoteWorkerThread_2: SYNC 5000000147 done in 0.005 seconds 2017-06-14 23:22:30 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2 2017-06-14 23:22:30 PDT INFO remoteWorkerThread_1: SYNC 5000000195 done in 0.016 seconds 2017-06-14 23:22:31 PDT INFO remoteWorkerThread_2: SYNC 5000000148 done in 0.005 seconds 2017-06-14 23:22:45 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2 2017-06-14 23:22:45 PDT INFO remoteWorkerThread_2: SYNC 5000000149 done in 0.006 seconds 2017-06-14 23:22:45 PDT INFO remoteWorkerThread_1: SYNC 5000000196 done in 0.008 seconds 2017-06-14 23:22:56 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2 2017-06-14 23:22:56 PDT INFO remoteWorkerThread_2: SYNC 5000000150 done in 0.006 seconds 2017-06-14 23:22:56 PDT INFO remoteWorkerThread_1: SYNC 5000000198 done in 0.008 seconds 2017-06-14 23:23:10 PDT INFO remoteWorkerThread_2: SYNC 5000000151 done in 0.005 seconds 2017-06-14 23:23:13 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2 2017-06-14 23:23:13 PDT INFO remoteWorkerThread_2: SYNC 5000000152 done in 0.017 seconds 2017-06-14 23:23:13 PDT INFO remoteWorkerThread_1: SYNC 5000000199 done in 0.019 seconds 2017-06-14 23:23:26 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2 2017-06-14 23:23:26 PDT INFO remoteWorkerThread_2: SYNC 5000000153 done in 0.005 seconds 2017-06-14 23:23:26 PDT INFO remoteWorkerThread_1: SYNC 5000000201 done in 0.007 seconds 2017-06-14 23:23:39 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2 2017-06-14 23:23:39 PDT INFO remoteWorkerThread_2: SYNC 5000000154 done in 0.006 seconds 2017-06-14 23:23:39 PDT INFO remoteWorkerThread_1: SYNC 5000000202 done in 0.007 seconds 2017-06-14 23:23:43 PDT INFO remoteWorkerThread_2: SYNC 5000000155 done in 0.004 seconds 2017-06-14 23:23:51 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2 2017-06-14 23:23:51 PDT INFO remoteWorkerThread_2: SYNC 5000000156 done in 0.005 seconds 2017-06-14 23:23:51 PDT INFO remoteWorkerThread_1: SYNC 5000000203 done in 0.007 seconds 2017-06-14 23:23:59 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2 2017-06-14 23:23:59 PDT INFO remoteWorkerThread_1: SYNC 5000000204 done in 0.007 seconds 2017-06-14 23:24:07 PDT INFO remoteWorkerThread_2: SYNC 5000000157 done in 0.005 seconds 2017-06-14 23:24:10 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2 2017-06-14 23:24:10 PDT INFO remoteWorkerThread_1: SYNC 5000000205 done in 0.008 seconds 2017-06-14 23:24:13 PDT INFO remoteWorkerThread_2: SYNC 5000000158 done in 0.005 seconds 2017-06-14 23:24:16 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2 2017-06-14 23:24:16 PDT INFO remoteWorkerThread_1: SYNC 5000000206 done in 0.006 seconds 2017-06-14 23:24:24 PDT INFO remoteWorkerThread_2: SYNC 5000000159 done in 0.005 seconds 2017-06-14 23:24:37 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2 2017-06-14 23:24:37 PDT INFO remoteWorkerThread_2: SYNC 5000000160 done in 0.006 seconds 2017-06-14 23:24:37 PDT INFO remoteWorkerThread_1: SYNC 5000000207 done in 0.008 seconds 2017-06-14 23:24:50 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2 2017-06-14 23:24:50 PDT INFO remoteWorkerThread_2: SYNC 5000000161 done in 0.005 seconds 2017-06-14 23:24:50 PDT INFO remoteWorkerThread_1: SYNC 5000000209 done in 0.007 seconds 2017-06-14 23:24:53 PDT INFO remoteWorkerThread_2: SYNC 5000000162 done in 0.005 seconds 2017-06-14 23:25:06 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2 2017-06-14 23:25:06 PDT INFO remoteWorkerThread_2: SYNC 5000000163 done in 0.013 seconds 2017-06-14 23:25:06 PDT INFO remoteWorkerThread_1: SYNC 5000000210 done in 0.014 seconds 2017-06-14 23:25:19 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2 2017-06-14 23:25:19 PDT INFO remoteWorkerThread_2: SYNC 5000000164 done in 0.005 seconds 2017-06-14 23:25:19 PDT INFO remoteWorkerThread_1: SYNC 5000000211 done in 0.007 seconds 2017-06-14 23:25:22 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2 2017-06-14 23:25:22 PDT INFO remoteWorkerThread_2: SYNC 5000000165 done in 0.006 seconds 2017-06-14 23:25:22 PDT INFO remoteWorkerThread_1: SYNC 5000000212 done in 0.008 seconds 2017-06-14 23:25:30 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2 2017-06-14 23:25:30 PDT INFO remoteWorkerThread_1: SYNC 5000000213 done in 0.006 seconds 2017-06-14 23:25:31 PDT INFO remoteWorkerThread_2: SYNC 5000000166 done in 0.004 seconds |
13. 查看数据同步状态
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 |
$ psql -h 10.189.100.226 -U slonyadmin -d region_il -c "\d il_longlive.entity" -W Password for user slonyadmin: Table "il_longlive.entity" Column | Type | Modifiers --------------+------------------------+------------------------------------------------------------------------ entity_id | integer | not null default nextval( 'il_longlive.entity_entity_id_seq' ::regclass) country_code | character varying(3) | state_code | character varying(50) | name | character varying(128) | islive | boolean | Indexes: "entity_pkey" PRIMARY KEY, btree (entity_id) Triggers: #可以看到Slony-I配置完成后原表多了触发器 _region_il_denyaccess BEFORE INSERT OR DELETE OR UPDATE ON il_longlive.entity FOR EACH ROW EXECUTE PROCEDURE _region_il.denyaccess( '_region_il' ) _region_il_truncatedeny BEFORE TRUNCATE ON il_longlive.entity FOR EACH STATEMENT EXECUTE PROCEDURE _region_il.deny_truncate() Disabled user triggers: _region_il_logtrigger AFTER INSERT OR DELETE OR UPDATE ON il_longlive.entity FOR EACH ROW EXECUTE PROCEDURE _region_il.logtrigger( '_region_il' , '101' , 'k' ) _region_il_truncatetrigger BEFORE TRUNCATE ON il_longlive.entity FOR EACH STATEMENT EXECUTE PROCEDURE _region_il.log_truncate( '101' ) $ psql -h 10.189.102.118 -U slonyadmin -d region_il -c "select count(*) from il_longlive.entity;" -W Password for user slonyadmin: count --------- (1 row) $ psql -h 10.189.100.195 -U slonyadmin -d region_il -c "select count(*) from il_longlive.entity;" -W Password for user slonyadmin: count --------- #可以看到数据已经同步到辅节点1了 (1 row) $ psql -h 10.189.100.226 -U slonyadmin -d region_il -c "select count(*) from il_longlive.entity;" -W Password for user slonyadmin: count --------- #可以看到数据已经同步到辅节点2了 (1 row) |
14. Slony-I相关表或者视图查看
sl_apply_stats: 每个节点本地的复制表的DML统计状态
sl_components: slon/slonik组件等当前的工作状态
|
1 2 3 4 5 6 7 8 9 10 11 |
region_il= # select * from sl_components; co_actor | co_pid | co_node | co_connection_pid | co_activity | co_starttime | co_event | co_eventtype ----------------------+--------+---------+-------------------+------------------+------------------------+------------+-------------- local_monitor | 2455 | 0 | 2470 | thread main loop | 2017-06-14 22:54:42-07 | | n /a local_cleanup | 2455 | 0 | 2468 | cleanupEvent | 2017-06-15 00:44:48-07 | | n /a remoteWorkerThread_2 | 2455 | 2 | 2471 | SYNC | 2017-06-15 00:49:14-07 | 5000000668 | SYNC local_listen | 2455 | 1 | 2459 | thread main loop | 2017-06-15 00:49:20-07 | | n /a remoteWorkerThread_3 | 2455 | 3 | 2467 | SYNC | 2017-06-15 00:49:17-07 | 5000000653 | SYNC remote listener | 2455 | 3 | 19059 | thread main loop | 2017-06-15 00:49:18-07 | | n /a local_sync | 2455 | 0 | 2469 | thread main loop | 2017-06-15 00:49:21-07 | | n /a (7 rows) |
sl_table: 当前集群中的表信息
|
1 2 3 4 5 6 7 8 |
region_il= # select * from sl_table ; tab_id | tab_reloid | tab_relname | tab_nspname | tab_set | tab_idxname | tab_altered | tab_comment --------+------------+----------------+-------------+---------+---------------------+-------------+--------------------------------------------------- | 19350 | entity | il_longlive | 1 | entity_pkey | f | Table il_longlive.entity with primary key | 19355 | entity_result | il_longlive | 1 | entity_result_pkey | f | Table il_longlive.entity_result with primary key | 19360 | result | il_longlive | 1 | result_pkey | f | Table il_longlive.result with primary key | 19365 | result_logbook | il_longlive | 1 | result_logbook_pkey | f | Table il_longlive.result_logbook with primary key (4 rows) |
sl_set: 当前集群中的集合信息
|
1 2 3 4 5 |
region_il= # select * from sl_set; set_id | set_origin | set_locked | set_comment --------+------------+------------+------------------------------------------ 1 | 1 | | Set 1 (il_longlive_schema) for region_il (1 row) |
sl_set: 当前集群中的节点信息
|
1 2 3 4 5 6 7 |
region_il=# select * from sl_node; no_id | no_active | no_comment | no_failed -------+-----------+-----------------------------------+----------- | t | Node 1 - region_il@10.189.102.118 | f | t | Node 2 - region_il@10.189.100.195 | f | t | Node 3 - region_il@10.189.100.226 | f (3 rows ) |
sl_status: 显示远程节点落后主节点的延迟
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 |
region_il=# select * from sl_status; -[ RECORD 1 ] -------------+------------------------------ st_origin | 1 st_received | 3 st_last_event | 5000000800 st_last_event_ts | 2017-06-15 01:03:19.585161-07 st_last_received | 5000000799 st_last_received_ts | 2017-06-15 01:03:14.281766-07 st_last_received_event_ts | 2017-06-15 01:03:09.57357-07 st_lag_num_events | 1 st_lag_time | 00:00:11.414753 -[ RECORD 2 ] -------------+------------------------------ st_origin | 1 st_received | 2 st_last_event | 5000000800 st_last_event_ts | 2017-06-15 01:03:19.585161-07 st_last_received | 5000000799 st_last_received_ts | 2017-06-15 01:03:14.193743-07 st_last_received_event_ts | 2017-06-15 01:03:09.57357-07 st_lag_num_events | 1 st_lag_time | 00:00:11.414753 |
sl_subscribe: 显示当前集群中的订阅者
|
1 2 3 4 5 6 |
region_il=# select * from sl_subscribe; sub_set | sub_provider | sub_receiver | sub_forward | sub_active ---------+--------------+--------------+-------------+------------ | 1 | 2 | t | t | 2 | 3 | t | t (2 rows ) |
Slony-I日常维护
1. 更改主表的表结构结构
1.1. 查看要修改的表结构信息
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
region_il=# \d result_logbook; Table "il_longlive.result_logbook" Column | Type | Modifiers ------------------+-----------------------------+-------------------------------------------------------------------- record_id | integer | not null default nextval( 'result_logbook_record_id_seq' ::regclass) schema | character varying (255) | not null vendor_code | character varying (255) | tag | character varying (255) | total_result_cnt | integer | exist_result_cnt | integer | new_result_cnt | integer | max_result_id | integer | islive | boolean | default true create_time | timestamp without time zone | default now() update_time | timestamp without time zone | Indexes: "result_logbook_pkey" PRIMARY KEY , btree (record_id) |
1.2. 创建修改表字段的ddl语句文件
|
1 2 |
$ cat /home/postgres/alter_table_result_logbook.sql alter table il_longlive.result_logbook add column address varchar (40); |
1.3. 通过执行slonik_execute_script命令
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 |
$ /usr/ local /pgsql/bin/slonik_execute_script 1 /home/postgres/alter_table_result_logbook.sql cluster name = region_il; node 1 admin conninfo= 'host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888' ; node 2 admin conninfo= 'host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888' ; node 3 admin conninfo= 'host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888' ; execute script ( set id = 1, filename = '/home/postgres/alter_table_result_logbook.sql' , event node = 1 ); $ cat /home/postgres/alter_table_result_logbook.sh #!/bin/sh slonik << __EOF__ cluster name = region_il; node 1 admin conninfo= 'host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888' ; node 2 admin conninfo= 'host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888' ; node 3 admin conninfo= 'host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888' ; execute script ( filename = '/home/postgres/alter_table_result_logbook.sql' , event node = 1 ); __EOF__ $ chmod +x /home/postgres/alter_table_result_logbook.sh $ /home/postgres/alter_table_result_logbook.sh |
1.4. 验证所有节点上的表结构修改是否成功
|
1 2 3 4 5 6 7 8 9 |
$ psql -h 10.189.102.118 -U slonyadmin -d region_il -c "\d il_longlive.result_logbook" | grep address Password for user slonyadmin: address | character varying (40) $ psql -h 10.189.100.195 -U slonyadmin -d region_il -c "\d il_longlive.result_logbook" | grep address Password for user slonyadmin: address | character varying (40) $ psql -h 10.189.100.226 -U slonyadmin -d region_il -c "\d il_longlive.result_logbook" | grep address Password for user slonyadmin: address | character varying (40) |
2. 向现有集群中增加一个复制表
2.1. 查看主库的的表结构信息,确认符合Slony-I的要求(主键)
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
region_il=# \d snapshot Table "il_longlive.snapshot" Column | Type | Modifiers -----------------+--------------------------------+------------------------------------------------------- id | integer | not null default nextval( 'snapshot_id_seq' ::regclass) feed_schema | character varying (255) | job | character varying (255) | table_name | character varying (255) | max_primary_key | bigint | create_time | timestamp (0) without time zone | update_time | timestamp (0) without time zone | Indexes: "snapshot_pkey" PRIMARY KEY , btree (id) |
2.2. 将主表结构导入到所有辅节点
|
1 2 |
$ pg_dump -U slonyadmin -s -t il_longlive.snapshot region_il | psql -h 10.189.100.195 -U slonyadmin region_il $ pg_dump -U slonyadmin -s -t il_longlive.snapshot region_il | psql -h 10.189.100.226 -U slonyadmin region_il |
2.3. 修改Slony-I配置文件slon_tools_conf,增加新复制集,并复制到所有节点(不是必须,为了各节点配置文件完整性)
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 |
$ cat /usr/ local /pgsql/etc/slon_tools.conf if ($ENV{ "SLONYNODES" }) { require $ENV{ "SLONYNODES" }; } else { $CLUSTER_NAME = 'region_il' ; $PIDFILE_DIR = '/usr/local/pgsql/data' ; $LOGDIR = '/usr/local/pgsql/data/pg_log' ; $MASTERNODE = 1; $DEBUGLEVEL = 0; $START_WATCHDOG = 1; $SLEEP_TIME = 60; $WATCHDOG_VERBOSE = 0;
add_node(node => 1, host => '10.189.102.118' , dbname => 'region_il' , port => 5432, user => 'slonyadmin' , password => '888888' ,);
add_node(node => 2, parent => 1, host => '10.189.100.195' , dbname => 'region_il' , port => 5432, user => 'slonyadmin' , password => '888888' ,);
add_node(node => 3, parent => 2, host => '10.189.100.226' , dbname => 'region_il' , port => 5432, user => 'slonyadmin' , password => '888888' ,); }
$SLONY_SETS = { "il_longlive_schema" => { "set_id" => 1, "table_id" => 101, "sequence_id" => 101, "pkeyedtables" => [ 'il_longlive.entity' , 'il_longlive.entity_result' , 'il_longlive.result' , 'il_longlive.result_logbook' , ],
"keyedtables" => {},
"sequences" => [ 'il_longlive.entity_entity_id_seq' , 'il_longlive.entity_result_id_seq' , 'il_longlive.result_id_seq' , 'il_longlive.result_logbook_record_id_seq' , ], },
"il_longlive_schema1" => { "set_id" => 2, "table_id" => 201, "sequence_id" => 201, "pkeyedtables" => [ 'il_longlive.snapshot' , ],
"keyedtables" => {},
"sequences" => [ 'il_longlive.snapshot_id_seq' ], },
};
if ($ENV{ "SLONYSET" }) { require $ENV{ "SLONYSET" }; }
# Please do not add or change anything below this point. 1; $ scp /usr/ local /pgsql/etc/slon_tools.conf postgres@10.189.100.195:/usr/ local /pgsql/etc/slon_tools.conf $ scp /usr/ local /pgsql/etc/slon_tools.conf postgres@10.189.100.226:/usr/ local /pgsql/etc/slon_tools.conf |
2.4. 在主节点创建新复制集
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 |
$ /usr/ local /pgsql/bin/slonik_create_set --config /usr/local/pgsql/etc/slon_tools.conf 2 cluster name = region_il; node 1 admin conninfo= 'host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888' ; node 2 admin conninfo= 'host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888' ; node 3 admin conninfo= 'host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888' ;
# TABLE ADD KEY
# CREATE SET create set (id = 2, origin = 1, comment = 'Set 2 (il_longlive_schema1) for region_il' );
# SET ADD TABLE echo 'Subscription set 2 (il_longlive_schema1) created' ; echo 'Adding tables to the subscription set' ; set add table ( set id = 2, origin = 1, id = 201, full qualified name = 'il_longlive.snapshot' , comment = 'Table il_longlive.snapshot with primary key' ); echo 'Add primary keyed table il_longlive.snapshot' ;
# SET ADD SEQUENCE echo 'Adding sequences to the subscription set' ; set add sequence ( set id = 2, origin = 1, id = 201, full qualified name = 'il_longlive.snapshot_id_seq' , comment = 'Sequence il_longlive.snapshot_id_seq' ); echo 'Add sequence il_longlive.snapshot_id_seq' ; echo 'All tables added' ; $ /usr/ local /pgsql/bin/slonik_create_set --config /usr/local/pgsql/etc/slon_tools.conf 2 | slonik <stdin>:12: Subscription set 2 (il_longlive_schema1) created <stdin>:13: Adding tables to the subscription set <stdin>:17: Add primary keyed table il_longlive.snapshot <stdin>:20: Adding sequences to the subscription set <stdin>:24: Add sequence il_longlive.snapshot_id_seq <stdin>:25: All tables added |
2.5. 在主节点创建新复制集的订阅者
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
$ /usr/ local /pgsql/bin/slonik_subscribe_set --config /usr/local/pgsql/etc/slon_tools.conf 2 2 cluster name = region_il; node 1 admin conninfo= 'host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888' ; node 2 admin conninfo= 'host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888' ; node 3 admin conninfo= 'host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888' ;
subscribe set (id = 2, provider = 1, receiver = 2, forward = yes); echo 'Subscribed nodes to set 2' ; $ /usr/ local /pgsql/bin/slonik_subscribe_set --config /usr/local/pgsql/etc/slon_tools.conf 2 3 cluster name = region_il; node 1 admin conninfo= 'host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888' ; node 2 admin conninfo= 'host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888' ; node 3 admin conninfo= 'host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888' ;
subscribe set (id = 2, provider = 2, receiver = 3, forward = yes); echo 'Subscribed nodes to set 2' ; $ /usr/ local /pgsql/bin/slonik_subscribe_set --config /usr/local/pgsql/etc/slon_tools.conf 2 2 | slonik <stdin>:7: Subscribed nodes to set 2 $ /usr/ local /pgsql/bin/slonik_subscribe_set --config /usr/local/pgsql/etc/slon_tools.conf 2 3 | slonik <stdin>:7: Subscribed nodes to set 2 |
2.6. 查看所有辅节点数据同步状态
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
$ psql -h 10.189.102.118 -U slonyadmin -d region_il -c "select count(*) from il_longlive.snapshot;" #主节点 Password for user slonyadmin: count ------- (1 row)
$ psql -h 10.189.100.196 -U slonyadmin -d region_il -c "select count(*) from il_longlive.snapshot;" #辅节点1 ^C [postgres@ec2t-pgtest-01 bin]$ psql -h 10.189.100.195 -U slonyadmin -d region_il -c "select count(*) from il_longlive.snapshot;" Password for user slonyadmin: count ------- (1 row)
$ psql -h 10.189.100.226 -U slonyadmin -d region_il -c "select count(*) from il_longlive.snapshot;" #辅节点2 Password for user slonyadmin: count ------- (1 row) |
2.7. 查看Slony-I系统表信息
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
region_il=# select * from sl_set; set_id | set_origin | set_locked | set_comment --------+------------+------------+------------------------------------------- | 1 | | Set 1 (il_longlive_schema) for region_il | 1 | | Set 2 (il_longlive_schema1) for region_il (2 rows )
region_il=# select * from sl_table; tab_id | tab_reloid | tab_relname | tab_nspname | tab_set | tab_idxname | tab_altered | tab_comment --------+------------+----------------+-------------+---------+---------------------+-------------+--------------------------------------------------- | 19350 | entity | il_longlive | 1 | entity_pkey | f | Table il_longlive.entity with primary key | 19355 | entity_result | il_longlive | 1 | entity_result_pkey | f | Table il_longlive.entity_result with primary key | 19360 | result | il_longlive | 1 | result_pkey | f | Table il_longlive.result with primary key | 19365 | result_logbook | il_longlive | 1 | result_logbook_pkey | f | Table il_longlive.result_logbook with primary key | 19380 | snapshot | il_longlive | 2 | snapshot_pkey | f | Table il_longlive.snapshot with primary key (5 rows ) |
2.8. 合并复制集
|
1 2 3 4 5 6 7 8 9 |
$ /usr/ local /pgsql/bin/slonik_merge_sets --config /usr/local/pgsql/etc/slon_tools.conf 1 1 2 cluster name = region_il; node 1 admin conninfo= 'host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888' ; node 2 admin conninfo= 'host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888' ; node 3 admin conninfo= 'host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888' ; merge set (id = 1, add id = 2, origin = 1); echo 'Replication set 2 merged into 1 on origin 1. Set 2 no longer exists.' ; $ /usr/ local /pgsql/bin/slonik_merge_sets --config /usr/local/pgsql/etc/slon_tools.conf 1 1 2 | slonik <stdin>:6: Replication set 2 merged into 1 on origin 1. Set 2 no longer exists. |
2.9. 查看Slony-I系统表信息
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
region_il=# select * from sl_set; set_id | set_origin | set_locked | set_comment --------+------------+------------+------------------------------------------ | 1 | | Set 1 (il_longlive_schema) for region_il (1 row)
region_il=# select * from sl_table; tab_id | tab_reloid | tab_relname | tab_nspname | tab_set | tab_idxname | tab_altered | tab_comment --------+------------+----------------+-------------+---------+---------------------+-------------+--------------------------------------------------- | 19350 | entity | il_longlive | 1 | entity_pkey | f | Table il_longlive.entity with primary key | 19355 | entity_result | il_longlive | 1 | entity_result_pkey | f | Table il_longlive.entity_result with primary key | 19360 | result | il_longlive | 1 | result_pkey | f | Table il_longlive.result with primary key | 19365 | result_logbook | il_longlive | 1 | result_logbook_pkey | f | Table il_longlive.result_logbook with primary key | 19380 | snapshot | il_longlive | 1 | snapshot_pkey | f | Table il_longlive.snapshot with primary key (5 rows ) |
3. 向现有集群中移除一个复制表
3.1. 查看Slony-I系统视图确认要移除的表
|
1 2 3 4 5 6 7 8 9 |
region_il=# select * from sl_table; tab_id | tab_reloid | tab_relname | tab_nspname | tab_set | tab_idxname | tab_altered | tab_comment --------+------------+----------------+-------------+---------+---------------------+-------------+--------------------------------------------------- | 19350 | entity | il_longlive | 1 | entity_pkey | f | Table il_longlive.entity with primary key | 19355 | entity_result | il_longlive | 1 | entity_result_pkey | f | Table il_longlive.entity_result with primary key | 19360 | result | il_longlive | 1 | result_pkey | f | Table il_longlive.result with primary key | 19365 | result_logbook | il_longlive | 1 | result_logbook_pkey | f | Table il_longlive.result_logbook with primary key | 19380 | snapshot | il_longlive | 1 | snapshot_pkey | f | Table il_longlive.snapshot with primary key (5 rows ) |
3.2. 移除集群中指定的表
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
$ /usr/ local /pgsql/bin/slonik_drop_table --config /usr/local/pgsql/etc/slon_tools.conf 201 1 cluster name = region_il; node 1 admin conninfo= 'host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888' ; node 2 admin conninfo= 'host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888' ; node 3 admin conninfo= 'host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888' ;
# DROP TABLE try { SET DROP TABLE (id = 201, origin = 1); } on error { echo 'Could not drop table 201 for region_il!' ; exit 1; }
$ /usr/ local /pgsql/bin/slonik_drop_table --config /usr/local/pgsql/etc/slon_tools.conf 201 1 | slonik |
3.3. 验证集群中指定的表是否移除
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 |
region_il=# select * from sl_table; tab_id | tab_reloid | tab_relname | tab_nspname | tab_set | tab_idxname | tab_altered | tab_comment --------+------------+----------------+-------------+---------+---------------------+-------------+--------------------------------------------------- | 19350 | entity | il_longlive | 1 | entity_pkey | f | Table il_longlive.entity with primary key | 19355 | entity_result | il_longlive | 1 | entity_result_pkey | f | Table il_longlive.entity_result with primary key | 19360 | result | il_longlive | 1 | result_pkey | f | Table il_longlive.result with primary key | 19365 | result_logbook | il_longlive | 1 | result_logbook_pkey | f | Table il_longlive.result_logbook with primary key (4 rows )
region_il=# \d il_longlive.snapshot #可以看到对应的表上的触发器也被移除了 Table "il_longlive.snapshot" Column | Type | Modifiers -----------------+--------------------------------+------------------------------------------------------------------- id | integer | not null default nextval( 'il_longlive.snapshot_id_seq' ::regclass) feed_schema | character varying (255) | job | character varying (255) | table_name | character varying (255) | max_primary_key | bigint | create_time | timestamp (0) without time zone | update_time | timestamp (0) without time zone | Indexes: "snapshot_pkey" PRIMARY KEY , btree (id) |
4. 向现有集群中移除一个节点
4.1. 查看Slony-I系统视图确认要移除的节点
|
1 2 3 4 5 6 7 |
region_il=# select * from sl_node; no_id | no_active | no_comment | no_failed -------+-----------+-----------------------------------+----------- | t | Node 1 - region_il@10.189.102.118 | f | t | Node 2 - region_il@10.189.100.195 | f | t | Node 3 - region_il@10.189.100.226 | f (3 rows ) |
4.2. 移除集群中指定的节点
|
1 2 3 4 5 6 7 8 9 10 11 |
$ /usr/ local /pgsql/bin/slonik_drop_node --config /usr/local/pgsql/etc/slon_tools.conf 3 1 cluster name = region_il; node 1 admin conninfo= 'host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888' ; node 2 admin conninfo= 'host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888' ; node 3 admin conninfo= 'host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888' ; drop node (id = 3, event node = 1); echo 'dropped node 3 cluster' ; $ /usr/ local /pgsql/bin/slonik_drop_node --config /usr/local/pgsql/etc/slon_tools.conf 3 1 | slonik debug: waiting for 3,5000019408 on 1 debug: waiting for 3,5000019408 on 1 <stdin>:6: dropped node 3 cluster |
4.3. 验证集群中指定的节点是否移除(此时被移除节点的Slony-I系统表等信息已经被移除)
|
1 2 3 4 5 6 |
region_il=# select * from sl_node; no_id | no_active | no_comment | no_failed -------+-----------+-----------------------------------+----------- | t | Node 1 - region_il@10.189.102.118 | f | t | Node 2 - region_il@10.189.100.195 | f (2 rows ) |
4.4. 停止被关闭节点的slon进程
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
$ ps -ef | grep slon postgres 6033 1 0 Jun02 ? 00:00:05 /usr/ local /pgsql/bin/slon -p /usr/ local /pgsql/data/region_il_node2.pid -s 1000 -d0 region_il host=ec2t-pgtest-03 dbname=region_il user =slonyadmin port=5432 password =mypna123 postgres 6050 1 0 Jun02 ? 00:00:03 /usr/bin/perl /usr/ local /pgsql/bin/slon_watchdog --config=/usr/local/pgsql/etc/slon_tools.conf node2 60 postgres 12596 6033 0 05:00 ? 0 /usr/ local /pgsql/bin/slon -p /usr/ local /pgsql/data/region_il_node2.pid -s 1000 -d0 region_il host=ec2t-pgtest-03 dbname=region_il user =slonyadmin port=5432 password =mypna123 postgres 12598 19212 0 05:00 ? 0 /usr/ local /pgsql/bin/slon -p /usr/ local /pgsql/data/region_il_node3.pid -s 1000 -d0 region_il host=10.189.100.226 dbname=region_il user =slonyadmin port=5432 password =888888 postgres 12599 15237 0 05:00 ? 0 postgres: slonyadmin region_il 10.189.100.226(29630) idle postgres 12601 5757 0 05:00 pts/3 0 grep slon postgres 19212 1 0 Jun14 ? 0 /usr/ local /pgsql/bin/slon -p /usr/ local /pgsql/data/region_il_node3.pid -s 1000 -d0 region_il host=10.189.100.226 dbname=region_il user =slonyadmin port=5432 password =888888 postgres 19232 1 0 Jun14 ? 0 /usr/bin/perl /usr/ local /pgsql/bin/slon_watchdog --config=/usr/local/pgsql/etc/slon_tools.conf node3 60 $ slon_status 3 Slon is running for the 'region_il' cluster on node3. $ slon_kill slon_kill.pl... Killing all slon and slon_watchdog instances for the cluster region_il 1. Kill slon watchdogs slon_watchdog for cluster region_il node 2 killed - PID [6050] slon_watchdog for cluster region_il node 3 killed - PID [19232] $ ps -ef | grep slon postgres 12820 5757 0 05:05 pts/3 0 grep slon |
5. 向现有集群中增加一个节点
5.1. 在新节点安装配置Slony-I(参考之前步骤)
5.2. 查看当街Slony-I系统表节点信息
|
1 2 3 4 5 6 |
region_il=# select * from sl_node; no_id | no_active | no_comment | no_failed -------+-----------+-----------------------------------+----------- | t | Node 1 - region_il@10.189.102.118 | f | t | Node 2 - region_il@10.189.100.195 | f (2 rows ) |
5.3. 向Slony-I配置文件slon_tools.conf 增加新节点信息,并将新配置文件更新到所有节点
|
1 2 3 4 5 6 7 8 9 10 |
$ cat /usr/ local /pgsql/etc/slon_tools.conf ........................................... add_node(node => 3, parent => 1, host => '10.189.100.226' , dbname => 'region_il' , port => 5432, user => 'slonyadmin' , password => '888888' ,); ............................................ |
5.4. 在主节点上增加向集群新节点信息
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 |
$ /usr/ local /pgsql/bin/slonik_add_node --config /usr/local/pgsql/etc/slon_tools.conf 3 1
# ADD NODE cluster name = region_il; node 1 admin conninfo= 'host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888' ; node 2 admin conninfo= 'host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888' ; node 3 admin conninfo= 'host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888' ; try { store node (id = 3, event node = 1, comment = 'Node 3 - region_il@10.189.100.226' ); } on error { echo 'Failed to add node to cluster' ; exit 1; }
# STORE PATHS store path (server = 3, client = 1, conninfo = 'host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888' ); store path (server = 1, client = 3, conninfo = 'host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888' ); store path (server = 3, client = 2, conninfo = 'host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888' ); store path (server = 2, client = 3, conninfo = 'host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888' ); echo 'added node 3 to cluster' ; echo 'Please start a slon replication daemon for node 3' ; $ /usr/ local /pgsql/bin/slonik_add_node --config /usr/local/pgsql/etc/slon_tools.conf 3 1 | slonik <stdin>:7: Possible unsupported PostgreSQL version (90505) 9.5, defaulting to 8.4 support <stdin>:19: added node 3 to cluster <stdin>:20: Please start a slon replication daemon for node 3 |
5.5. 在新节点启动slon进程
|
1 2 3 4 5 |
$ slon_start 3 Invoke slon for node 3 - /usr/ local /pgsql/bin/slon -p /usr/ local /pgsql/data/region_il_node3.pid -s 1000 -d0 region_il 'host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888' > /usr/ local /pgsql/data/pg_log/node3/region_il-2017-06-17.log 2>&1 & Slon successfully started for cluster region_il, node node3 PID [13136] Start the watchdog process as well... |
5.6. 在主节点添加新节点的复制订阅信息
|
1 2 3 4 5 6 7 8 9 10 11 12 |
$ /usr/ local /pgsql/bin/slonik_subscribe_set --config /usr/local/pgsql/etc/etc/slon_tools.conf 1 3 Can 't locate /usr/local/pgsql/etc/etc/slon_tools.conf in @INC (@INC contains: /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at /usr/local/pgsql/bin/slonik_subscribe_set line 29. [postgres@ec2t-pgtest-01 bin]$ /usr/local/pgsql/bin/slonik_subscribe_set --config /usr/local/pgsql/etc/slon_tools.conf 1 3 cluster name = region_il; node 1 admin conninfo=' host=10.189.102.118 dbname=region_il user =slonyadmin port=5432 password =888888 '; node 2 admin conninfo=' host=10.189.100.195 dbname=region_il user =slonyadmin port=5432 password =888888 '; node 3 admin conninfo=' host=10.189.100.226 dbname=region_il user =slonyadmin port=5432 password =888888 ';
subscribe set (id = 1, provider = 1, receiver = 3, forward = yes); echo ' Subscribed nodes to set 1'; [postgres@ec2t-pgtest-01 bin]$ /usr/ local /pgsql/bin/slonik_subscribe_set --config /usr/local/pgsql/etc/slon_tools.conf 1 3 | slonik <stdin>:7: Subscribed nodes to set 1 |
5.7. 验证数据同步(略)
总结
以上就是这篇文章的全部内容了，希望本文的内容对大家的学习或者工作具有一定的参考学习价值，如果有疑问大家可以留言交流，谢谢大家对服务器之家的支持。
原文链接：https://www.cnblogs.com/ilifeilong/p/7009322.html
查看更多关于PostgreSQL中Slony-I同步复制部署教程的详细内容...