Reorganize Partition On Partial Cluster

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Reorganize Partition On Partial Cluster

Serhat Rıfat Demircan
I have changed MAX_ROWS on partial (while a data node disconnected) cluster
then reorganized partitions. When i try to restart data node which was
disconnected i get following error:

    2015-11-13 06:09:14 [ndbd] INFO     -- Angel connected to '
192.168.60.100:1186'
    2015-11-13 06:09:14 [ndbd] INFO     -- Angel allocated nodeid: 10
    serhatd@bung:~$ tail -f /usr/local/mysql/data/ndb_10_out.log
    NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
    Started thread, index = 33, id = 15809, type = NdbfsThread
    NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
    Started thread, index = 34, id = 15810, type = NdbfsThread
    Started thread, index = 35, id = 15811, type = NdbfsThread
    Started thread, index = 36, id = 15812, type = NdbfsThread
    2015-11-13 06:09:15 [ndbd] INFO     -- Sending READ_CONFIG_REQ to index
= 2, name = DBINFO
    2015-11-13 06:09:15 [ndbd] INFO     -- Sending READ_CONFIG_REQ to index
= 3, name = DBTUP
    2015-11-13 06:09:15 [ndbd] INFO     -- Sending READ_CONFIG_REQ to index
= 4, name = DBACC
    2015-11-13 06:09:15 [ndbd] INFO     -- Sending READ_CONFIG_REQ to index
= 5, name = DBTC
    2015-11-13 06:09:16 [ndbd] INFO     -- timerHandlingLab, expected 10ms
sleep, not scheduled for: 196 (ms)
    2015-11-13 06:09:16 [ndbd] INFO     -- Sending READ_CONFIG_REQ to index
= 6, name = DBLQH
    LCPFragWatchdog : WarnElapsed : 20000(ms) MaxElapsed 60000(ms) : period
millis : 10000
    2015-11-13 06:09:16 [ndbd] INFO     -- Sending READ_CONFIG_REQ to index
= 7, name = DBTUX
    2015-11-13 06:09:16 [ndbd] INFO     -- Sending READ_CONFIG_REQ to index
= 8, name = DBDICT
    2015-11-13 06:09:16 [ndbd] INFO     -- Sending READ_CONFIG_REQ to index
= 9, name = DBDIH
    2015-11-13 06:09:16 [ndbd] INFO     -- Sending READ_CONFIG_REQ to index
= 10, name = NDBCNTR
    2015-11-13 06:09:16 [ndbd] INFO     -- Sending READ_CONFIG_REQ to index
= 11, name = QMGR
    2015-11-13 06:09:16 [ndbd] INFO     -- Sending READ_CONFIG_REQ to index
= 12, name = TRIX
    2015-11-13 06:09:16 [ndbd] INFO     -- Sending READ_CONFIG_REQ to index
= 13, name = BACKUP
    2015-11-13 06:09:16 [ndbd] INFO     -- Sending READ_CONFIG_REQ to index
= 14, name = DBUTIL
    2015-11-13 06:09:16 [ndbd] INFO     -- Sending READ_CONFIG_REQ to index
= 15, name = SUMA
    2015-11-13 06:09:16 [ndbd] INFO     -- Sending READ_CONFIG_REQ to index
= 16, name = TSMAN
    2015-11-13 06:09:16 [ndbd] INFO     -- Sending READ_CONFIG_REQ to index
= 17, name = LGMAN
    2015-11-13 06:09:16 [ndbd] INFO     -- Sending READ_CONFIG_REQ to index
= 18, name = PGMAN
    pgman: page buffer entries = 10
    pgman: page buffer entries = 10
    2015-11-13 06:09:16 [ndbd] INFO     -- Sending READ_CONFIG_REQ to index
= 19, name = RESTORE
    2015-11-13 06:09:16 [ndbd] INFO     -- Sending READ_CONFIG_REQ to index
= 20, name = DBSPJ
    2015-11-13 06:09:16 [ndbd] INFO     -- Sending READ_CONFIG_REQ to index
= 21, name = THRMAN
    2015-11-13 06:09:16 [ndbd] INFO     -- READ_CONFIG_REQ phase completed,
this phase is used to read configuration and to calculate various sizes and
allocate almost all memory needed by the data node in its lifetime
    2015-11-13 06:09:16 [ndbd] INFO     -- Start phase 0 completed
    2015-11-13 06:09:16 [ndbd] INFO     -- Phase 0 has made some file
system initialisations
    2015-11-13 06:09:16 [ndbd] INFO     -- Starting QMGR phase 1
    2015-11-13 06:09:16 [ndbd] INFO     -- DIH reported normal start, now
starting the Node Inclusion Protocol
    2015-11-13 06:09:16 [ndbd] INFO     -- findNeighbours from: 2334 old
(left: 65535 right: 65535) new (13 11)
    2015-11-13 06:09:16 [ndbd] INFO     -- Include node protocol completed,
phase 1 in QMGR completed
    2015-11-13 06:09:16 [ndbd] INFO     -- Start phase 1 completed
    2015-11-13 06:09:16 [ndbd] INFO     -- Phase 1 initialised some
variables and included node in cluster, locked memory if configured to do so
    2015-11-13 06:09:16 [ndbd] INFO     -- Asking master node to accept our
start (nodeId = 11 is master), GCI = 5622678
    2015-11-13 06:09:16 [ndbd] INFO     -- NDBCNTR master accepted us into
cluster, start NDB start phase 1
    2015-11-13 06:09:16 [ndbd] INFO     -- We are performing a node restart
    2015-11-13 06:09:16 [ndbd] INFO     -- LDM(1): Started LDM restart
phase 1 (read REDO log page headers to init REDO log data)
    2015-11-13 06:09:16 [ndbd] INFO     -- Schema file initialisation
Starting
    2015-11-13 06:09:17 [ndbd] INFO     -- Schema file initialisation
Completed
    2015-11-13 06:09:17 [ndbd] INFO     -- NDB start phase 1 completed
    2015-11-13 06:09:17 [ndbd] INFO     -- Start phase 2 completed
    2015-11-13 06:09:17 [ndbd] INFO     -- Phase 2 did more
initialisations, master accepted our start, we started REDO log
initialisations
    2015-11-13 06:09:17 [ndbd] INFO     -- Start NDB start phase 2
    2015-11-13 06:09:17 [ndbd] INFO     -- Receive arbitrator node 1
[ticket=311700088796c28f]
    2015-11-13 06:09:18 [ndbd] INFO     -- Request permission to start our
node from master Starting
    2015-11-13 06:09:18 [ndbd] INFO     -- Request permission to start our
node from master Completed
    2015-11-13 06:09:18 [ndbd] INFO     -- NDB start phase 2 completed
    2015-11-13 06:09:18 [ndbd] INFO     -- Start phase 3 completed
    2015-11-13 06:09:18 [ndbd] INFO     -- Phase 3 locked the data
dictionary, performed local connection setups, we  asked for permission to
start our node
    2015-11-13 06:09:18 [ndbd] INFO     -- Start NDB start phase 3
    2015-11-13 06:09:18 [ndbd] INFO     -- NDB start phase 3 completed
    2015-11-13 06:09:18 [ndbd] INFO     -- Start phase 4 completed
    2015-11-13 06:09:18 [ndbd] INFO     -- Phase 4 continued preparations
of the REDO log
    2015-11-13 06:09:18 [ndbd] INFO     -- Request copying of distribution
and dictionary information from master Starting
    2015-11-13 06:09:18 [ndbd] INFO     -- LDM(1):Ready to start execute
REDO log phase, prepare REDO log phase completed
    2015-11-13 06:09:21 [ndbd] INFO     -- Copying of dictionary
information from master Starting
    2015-11-13 06:09:21 [ndbd] INFO     -- Restart recreating table with id
= 10
    2015-11-13 06:09:21 [ndbd] INFO     -- Restart recreating table with id
= 12
    2015-11-13 06:09:21 [ndbd] INFO     -- Restart recreating table with id
= 11
    2015-11-13 06:09:21 [ndbd] INFO     -- Restart recreating table with id
= 13
    2015-11-13 06:09:21 [ndbd] INFO     -- Restart recreating table with id
= 1
    2015-11-13 06:09:21 [ndbd] INFO     -- Restart recreating table with id
= 43
    2015-11-13 06:09:21 [ndbd] INFO     -- Restart recreating table with id
= 44
    2015-11-13 06:09:21 [ndbd] INFO     -- Restart recreating table with id
= 76
    2015-11-13 06:09:21 [ndbd] INFO     -- Restart recreating table with id
= 93
    2015-11-13 06:09:21 [ndbd] INFO     -- Restart recreating table with id
= 102
    2015-11-13 06:09:21 [ndbd] INFO     -- Restart recreating table with id
= 141
    2015-11-13 06:09:21 [ndbd] INFO     -- Restart recreating table with id
= 149
    2015-11-13 06:09:21 [ndbd] INFO     -- Restart recreating table with id
= 152
    2015-11-13 06:09:21 [ndbd] INFO     -- Restart recreating table with id
= 156
    2015-11-13 06:09:21 [ndbd] INFO     -- Restart recreating table with id
= 170
    restartDropObj(149)
    2015-11-13 06:09:22 [ndbd] ALERT    -- Node 10: Forced node shutdown
completed. Occured during startphase 0. Initiated by signal 11.

Node can start with --initial parameter but when i restart this data after
starting (i mean after starting with --initial parameter) same situation
occurs.

Do you have any suggestions?
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Reorganize Partition On Partial Cluster

Mauritz Sundell
Hi

On 2015-11-13 05:23, Serhat Rıfat Demircan wrote:
> I have changed MAX_ROWS on partial (while a data node disconnected) cluster
> then reorganized partitions. When i try to restart data node which was
> disconnected i get following error:
Please provide the content of ndb_10_error.log.
And also the exact commands you did before restart.

Did you both change MAX_ROWS and run REORGANIZE PARTITIONS on same table?

You can also report a bug at http://bugs.mysql.com/report.php
Attach in that case the ndb*error.log files, ndb*trace.log* files, and
the ndb*out.log files.

Regards,
Mauritz

>
>      2015-11-13 06:09:14 [ndbd] INFO     -- Angel connected to '
> 192.168.60.100:1186'
>      2015-11-13 06:09:14 [ndbd] INFO     -- Angel allocated nodeid: 10
>      serhatd@bung:~$ tail -f /usr/local/mysql/data/ndb_10_out.log
>      NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
>      Started thread, index = 33, id = 15809, type = NdbfsThread
>      NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
>      Started thread, index = 34, id = 15810, type = NdbfsThread
>      Started thread, index = 35, id = 15811, type = NdbfsThread
>      Started thread, index = 36, id = 15812, type = NdbfsThread
>      2015-11-13 06:09:15 [ndbd] INFO     -- Sending READ_CONFIG_REQ to index
> = 2, name = DBINFO
>      2015-11-13 06:09:15 [ndbd] INFO     -- Sending READ_CONFIG_REQ to index
> = 3, name = DBTUP
>      2015-11-13 06:09:15 [ndbd] INFO     -- Sending READ_CONFIG_REQ to index
> = 4, name = DBACC
>      2015-11-13 06:09:15 [ndbd] INFO     -- Sending READ_CONFIG_REQ to index
> = 5, name = DBTC
>      2015-11-13 06:09:16 [ndbd] INFO     -- timerHandlingLab, expected 10ms
> sleep, not scheduled for: 196 (ms)
>      2015-11-13 06:09:16 [ndbd] INFO     -- Sending READ_CONFIG_REQ to index
> = 6, name = DBLQH
>      LCPFragWatchdog : WarnElapsed : 20000(ms) MaxElapsed 60000(ms) : period
> millis : 10000
>      2015-11-13 06:09:16 [ndbd] INFO     -- Sending READ_CONFIG_REQ to index
> = 7, name = DBTUX
>      2015-11-13 06:09:16 [ndbd] INFO     -- Sending READ_CONFIG_REQ to index
> = 8, name = DBDICT
>      2015-11-13 06:09:16 [ndbd] INFO     -- Sending READ_CONFIG_REQ to index
> = 9, name = DBDIH
>      2015-11-13 06:09:16 [ndbd] INFO     -- Sending READ_CONFIG_REQ to index
> = 10, name = NDBCNTR
>      2015-11-13 06:09:16 [ndbd] INFO     -- Sending READ_CONFIG_REQ to index
> = 11, name = QMGR
>      2015-11-13 06:09:16 [ndbd] INFO     -- Sending READ_CONFIG_REQ to index
> = 12, name = TRIX
>      2015-11-13 06:09:16 [ndbd] INFO     -- Sending READ_CONFIG_REQ to index
> = 13, name = BACKUP
>      2015-11-13 06:09:16 [ndbd] INFO     -- Sending READ_CONFIG_REQ to index
> = 14, name = DBUTIL
>      2015-11-13 06:09:16 [ndbd] INFO     -- Sending READ_CONFIG_REQ to index
> = 15, name = SUMA
>      2015-11-13 06:09:16 [ndbd] INFO     -- Sending READ_CONFIG_REQ to index
> = 16, name = TSMAN
>      2015-11-13 06:09:16 [ndbd] INFO     -- Sending READ_CONFIG_REQ to index
> = 17, name = LGMAN
>      2015-11-13 06:09:16 [ndbd] INFO     -- Sending READ_CONFIG_REQ to index
> = 18, name = PGMAN
>      pgman: page buffer entries = 10
>      pgman: page buffer entries = 10
>      2015-11-13 06:09:16 [ndbd] INFO     -- Sending READ_CONFIG_REQ to index
> = 19, name = RESTORE
>      2015-11-13 06:09:16 [ndbd] INFO     -- Sending READ_CONFIG_REQ to index
> = 20, name = DBSPJ
>      2015-11-13 06:09:16 [ndbd] INFO     -- Sending READ_CONFIG_REQ to index
> = 21, name = THRMAN
>      2015-11-13 06:09:16 [ndbd] INFO     -- READ_CONFIG_REQ phase completed,
> this phase is used to read configuration and to calculate various sizes and
> allocate almost all memory needed by the data node in its lifetime
>      2015-11-13 06:09:16 [ndbd] INFO     -- Start phase 0 completed
>      2015-11-13 06:09:16 [ndbd] INFO     -- Phase 0 has made some file
> system initialisations
>      2015-11-13 06:09:16 [ndbd] INFO     -- Starting QMGR phase 1
>      2015-11-13 06:09:16 [ndbd] INFO     -- DIH reported normal start, now
> starting the Node Inclusion Protocol
>      2015-11-13 06:09:16 [ndbd] INFO     -- findNeighbours from: 2334 old
> (left: 65535 right: 65535) new (13 11)
>      2015-11-13 06:09:16 [ndbd] INFO     -- Include node protocol completed,
> phase 1 in QMGR completed
>      2015-11-13 06:09:16 [ndbd] INFO     -- Start phase 1 completed
>      2015-11-13 06:09:16 [ndbd] INFO     -- Phase 1 initialised some
> variables and included node in cluster, locked memory if configured to do so
>      2015-11-13 06:09:16 [ndbd] INFO     -- Asking master node to accept our
> start (nodeId = 11 is master), GCI = 5622678
>      2015-11-13 06:09:16 [ndbd] INFO     -- NDBCNTR master accepted us into
> cluster, start NDB start phase 1
>      2015-11-13 06:09:16 [ndbd] INFO     -- We are performing a node restart
>      2015-11-13 06:09:16 [ndbd] INFO     -- LDM(1): Started LDM restart
> phase 1 (read REDO log page headers to init REDO log data)
>      2015-11-13 06:09:16 [ndbd] INFO     -- Schema file initialisation
> Starting
>      2015-11-13 06:09:17 [ndbd] INFO     -- Schema file initialisation
> Completed
>      2015-11-13 06:09:17 [ndbd] INFO     -- NDB start phase 1 completed
>      2015-11-13 06:09:17 [ndbd] INFO     -- Start phase 2 completed
>      2015-11-13 06:09:17 [ndbd] INFO     -- Phase 2 did more
> initialisations, master accepted our start, we started REDO log
> initialisations
>      2015-11-13 06:09:17 [ndbd] INFO     -- Start NDB start phase 2
>      2015-11-13 06:09:17 [ndbd] INFO     -- Receive arbitrator node 1
> [ticket=311700088796c28f]
>      2015-11-13 06:09:18 [ndbd] INFO     -- Request permission to start our
> node from master Starting
>      2015-11-13 06:09:18 [ndbd] INFO     -- Request permission to start our
> node from master Completed
>      2015-11-13 06:09:18 [ndbd] INFO     -- NDB start phase 2 completed
>      2015-11-13 06:09:18 [ndbd] INFO     -- Start phase 3 completed
>      2015-11-13 06:09:18 [ndbd] INFO     -- Phase 3 locked the data
> dictionary, performed local connection setups, we  asked for permission to
> start our node
>      2015-11-13 06:09:18 [ndbd] INFO     -- Start NDB start phase 3
>      2015-11-13 06:09:18 [ndbd] INFO     -- NDB start phase 3 completed
>      2015-11-13 06:09:18 [ndbd] INFO     -- Start phase 4 completed
>      2015-11-13 06:09:18 [ndbd] INFO     -- Phase 4 continued preparations
> of the REDO log
>      2015-11-13 06:09:18 [ndbd] INFO     -- Request copying of distribution
> and dictionary information from master Starting
>      2015-11-13 06:09:18 [ndbd] INFO     -- LDM(1):Ready to start execute
> REDO log phase, prepare REDO log phase completed
>      2015-11-13 06:09:21 [ndbd] INFO     -- Copying of dictionary
> information from master Starting
>      2015-11-13 06:09:21 [ndbd] INFO     -- Restart recreating table with id
> = 10
>      2015-11-13 06:09:21 [ndbd] INFO     -- Restart recreating table with id
> = 12
>      2015-11-13 06:09:21 [ndbd] INFO     -- Restart recreating table with id
> = 11
>      2015-11-13 06:09:21 [ndbd] INFO     -- Restart recreating table with id
> = 13
>      2015-11-13 06:09:21 [ndbd] INFO     -- Restart recreating table with id
> = 1
>      2015-11-13 06:09:21 [ndbd] INFO     -- Restart recreating table with id
> = 43
>      2015-11-13 06:09:21 [ndbd] INFO     -- Restart recreating table with id
> = 44
>      2015-11-13 06:09:21 [ndbd] INFO     -- Restart recreating table with id
> = 76
>      2015-11-13 06:09:21 [ndbd] INFO     -- Restart recreating table with id
> = 93
>      2015-11-13 06:09:21 [ndbd] INFO     -- Restart recreating table with id
> = 102
>      2015-11-13 06:09:21 [ndbd] INFO     -- Restart recreating table with id
> = 141
>      2015-11-13 06:09:21 [ndbd] INFO     -- Restart recreating table with id
> = 149
>      2015-11-13 06:09:21 [ndbd] INFO     -- Restart recreating table with id
> = 152
>      2015-11-13 06:09:21 [ndbd] INFO     -- Restart recreating table with id
> = 156
>      2015-11-13 06:09:21 [ndbd] INFO     -- Restart recreating table with id
> = 170
>      restartDropObj(149)
>      2015-11-13 06:09:22 [ndbd] ALERT    -- Node 10: Forced node shutdown
> completed. Occured during startphase 0. Initiated by signal 11.
>
> Node can start with --initial parameter but when i restart this data after
> starting (i mean after starting with --initial parameter) same situation
> occurs.
>
> Do you have any suggestions?
>


--
MySQL Cluster Mailing List
For list archives: http://lists.mysql.com/cluster
To unsubscribe:    http://lists.mysql.com/cluster

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Reorganize Partition On Partial Cluster

Serhat Rıfat Demircan
Hi

I both changed max_rows and run REORGANIZE PARTITIONS on same table.

Followed these step:
ndb_mgm> 10 stop
mysql> ALTER TABLE x MAX_ROWS=1000000;
mysql> ALTER TABLE x REORGANIZE PARTITIONS;
user@ndb_10$ ndbmtd


ndb_10_error.log:
------------------------
Time: Monday 9 November 2015 - 01:00:23
Status: Temporary error, restart node
Message: Another node failed during system restart, please investigate
error(s) on other node(s) (Restart error)
Error: 2308
Error data: Node 6 disconnected
Error object: QMGR (Line: 3470) 0x00000002
Program: ndbmtd
Pid: 8269 thr: 0
Version: mysql-5.6.24 ndb-7.4.6
Trace: /ndbdata/ndb_5_trace.log.25 [t1..t15]
***EOM***

On Fri, Nov 13, 2015 at 2:49 PM, Mauritz Sundell <[hidden email]
> wrote:

> Hi
>
> On 2015-11-13 05:23, Serhat Rıfat Demircan wrote:
>
>> I have changed MAX_ROWS on partial (while a data node disconnected)
>> cluster
>> then reorganized partitions. When i try to restart data node which was
>> disconnected i get following error:
>>
> Please provide the content of ndb_10_error.log.
> And also the exact commands you did before restart.
>
> Did you both change MAX_ROWS and run REORGANIZE PARTITIONS on same table?
>
> You can also report a bug at http://bugs.mysql.com/report.php
> Attach in that case the ndb*error.log files, ndb*trace.log* files, and the
> ndb*out.log files.
>
> Regards,
> Mauritz
>
>
>
>>      2015-11-13 06:09:14 [ndbd] INFO     -- Angel connected to '
>> 192.168.60.100:1186'
>>      2015-11-13 06:09:14 [ndbd] INFO     -- Angel allocated nodeid: 10
>>      serhatd@bung:~$ tail -f /usr/local/mysql/data/ndb_10_out.log
>>      NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
>>      Started thread, index = 33, id = 15809, type = NdbfsThread
>>      NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
>>      Started thread, index = 34, id = 15810, type = NdbfsThread
>>      Started thread, index = 35, id = 15811, type = NdbfsThread
>>      Started thread, index = 36, id = 15812, type = NdbfsThread
>>      2015-11-13 06:09:15 [ndbd] INFO     -- Sending READ_CONFIG_REQ to
>> index
>> = 2, name = DBINFO
>>      2015-11-13 06:09:15 [ndbd] INFO     -- Sending READ_CONFIG_REQ to
>> index
>> = 3, name = DBTUP
>>      2015-11-13 06:09:15 [ndbd] INFO     -- Sending READ_CONFIG_REQ to
>> index
>> = 4, name = DBACC
>>      2015-11-13 06:09:15 [ndbd] INFO     -- Sending READ_CONFIG_REQ to
>> index
>> = 5, name = DBTC
>>      2015-11-13 06:09:16 [ndbd] INFO     -- timerHandlingLab, expected
>> 10ms
>> sleep, not scheduled for: 196 (ms)
>>      2015-11-13 06:09:16 [ndbd] INFO     -- Sending READ_CONFIG_REQ to
>> index
>> = 6, name = DBLQH
>>      LCPFragWatchdog : WarnElapsed : 20000(ms) MaxElapsed 60000(ms) :
>> period
>> millis : 10000
>>      2015-11-13 06:09:16 [ndbd] INFO     -- Sending READ_CONFIG_REQ to
>> index
>> = 7, name = DBTUX
>>      2015-11-13 06:09:16 [ndbd] INFO     -- Sending READ_CONFIG_REQ to
>> index
>> = 8, name = DBDICT
>>      2015-11-13 06:09:16 [ndbd] INFO     -- Sending READ_CONFIG_REQ to
>> index
>> = 9, name = DBDIH
>>      2015-11-13 06:09:16 [ndbd] INFO     -- Sending READ_CONFIG_REQ to
>> index
>> = 10, name = NDBCNTR
>>      2015-11-13 06:09:16 [ndbd] INFO     -- Sending READ_CONFIG_REQ to
>> index
>> = 11, name = QMGR
>>      2015-11-13 06:09:16 [ndbd] INFO     -- Sending READ_CONFIG_REQ to
>> index
>> = 12, name = TRIX
>>      2015-11-13 06:09:16 [ndbd] INFO     -- Sending READ_CONFIG_REQ to
>> index
>> = 13, name = BACKUP
>>      2015-11-13 06:09:16 [ndbd] INFO     -- Sending READ_CONFIG_REQ to
>> index
>> = 14, name = DBUTIL
>>      2015-11-13 06:09:16 [ndbd] INFO     -- Sending READ_CONFIG_REQ to
>> index
>> = 15, name = SUMA
>>      2015-11-13 06:09:16 [ndbd] INFO     -- Sending READ_CONFIG_REQ to
>> index
>> = 16, name = TSMAN
>>      2015-11-13 06:09:16 [ndbd] INFO     -- Sending READ_CONFIG_REQ to
>> index
>> = 17, name = LGMAN
>>      2015-11-13 06:09:16 [ndbd] INFO     -- Sending READ_CONFIG_REQ to
>> index
>> = 18, name = PGMAN
>>      pgman: page buffer entries = 10
>>      pgman: page buffer entries = 10
>>      2015-11-13 06:09:16 [ndbd] INFO     -- Sending READ_CONFIG_REQ to
>> index
>> = 19, name = RESTORE
>>      2015-11-13 06:09:16 [ndbd] INFO     -- Sending READ_CONFIG_REQ to
>> index
>> = 20, name = DBSPJ
>>      2015-11-13 06:09:16 [ndbd] INFO     -- Sending READ_CONFIG_REQ to
>> index
>> = 21, name = THRMAN
>>      2015-11-13 06:09:16 [ndbd] INFO     -- READ_CONFIG_REQ phase
>> completed,
>> this phase is used to read configuration and to calculate various sizes
>> and
>> allocate almost all memory needed by the data node in its lifetime
>>      2015-11-13 06:09:16 [ndbd] INFO     -- Start phase 0 completed
>>      2015-11-13 06:09:16 [ndbd] INFO     -- Phase 0 has made some file
>> system initialisations
>>      2015-11-13 06:09:16 [ndbd] INFO     -- Starting QMGR phase 1
>>      2015-11-13 06:09:16 [ndbd] INFO     -- DIH reported normal start, now
>> starting the Node Inclusion Protocol
>>      2015-11-13 06:09:16 [ndbd] INFO     -- findNeighbours from: 2334 old
>> (left: 65535 right: 65535) new (13 11)
>>      2015-11-13 06:09:16 [ndbd] INFO     -- Include node protocol
>> completed,
>> phase 1 in QMGR completed
>>      2015-11-13 06:09:16 [ndbd] INFO     -- Start phase 1 completed
>>      2015-11-13 06:09:16 [ndbd] INFO     -- Phase 1 initialised some
>> variables and included node in cluster, locked memory if configured to do
>> so
>>      2015-11-13 06:09:16 [ndbd] INFO     -- Asking master node to accept
>> our
>> start (nodeId = 11 is master), GCI = 5622678
>>      2015-11-13 06:09:16 [ndbd] INFO     -- NDBCNTR master accepted us
>> into
>> cluster, start NDB start phase 1
>>      2015-11-13 06:09:16 [ndbd] INFO     -- We are performing a node
>> restart
>>      2015-11-13 06:09:16 [ndbd] INFO     -- LDM(1): Started LDM restart
>> phase 1 (read REDO log page headers to init REDO log data)
>>      2015-11-13 06:09:16 [ndbd] INFO     -- Schema file initialisation
>> Starting
>>      2015-11-13 06:09:17 [ndbd] INFO     -- Schema file initialisation
>> Completed
>>      2015-11-13 06:09:17 [ndbd] INFO     -- NDB start phase 1 completed
>>      2015-11-13 06:09:17 [ndbd] INFO     -- Start phase 2 completed
>>      2015-11-13 06:09:17 [ndbd] INFO     -- Phase 2 did more
>> initialisations, master accepted our start, we started REDO log
>> initialisations
>>      2015-11-13 06:09:17 [ndbd] INFO     -- Start NDB start phase 2
>>      2015-11-13 06:09:17 [ndbd] INFO     -- Receive arbitrator node 1
>> [ticket=311700088796c28f]
>>      2015-11-13 06:09:18 [ndbd] INFO     -- Request permission to start
>> our
>> node from master Starting
>>      2015-11-13 06:09:18 [ndbd] INFO     -- Request permission to start
>> our
>> node from master Completed
>>      2015-11-13 06:09:18 [ndbd] INFO     -- NDB start phase 2 completed
>>      2015-11-13 06:09:18 [ndbd] INFO     -- Start phase 3 completed
>>      2015-11-13 06:09:18 [ndbd] INFO     -- Phase 3 locked the data
>> dictionary, performed local connection setups, we  asked for permission to
>> start our node
>>      2015-11-13 06:09:18 [ndbd] INFO     -- Start NDB start phase 3
>>      2015-11-13 06:09:18 [ndbd] INFO     -- NDB start phase 3 completed
>>      2015-11-13 06:09:18 [ndbd] INFO     -- Start phase 4 completed
>>      2015-11-13 06:09:18 [ndbd] INFO     -- Phase 4 continued preparations
>> of the REDO log
>>      2015-11-13 06:09:18 [ndbd] INFO     -- Request copying of
>> distribution
>> and dictionary information from master Starting
>>      2015-11-13 06:09:18 [ndbd] INFO     -- LDM(1):Ready to start execute
>> REDO log phase, prepare REDO log phase completed
>>      2015-11-13 06:09:21 [ndbd] INFO     -- Copying of dictionary
>> information from master Starting
>>      2015-11-13 06:09:21 [ndbd] INFO     -- Restart recreating table with
>> id
>> = 10
>>      2015-11-13 06:09:21 [ndbd] INFO     -- Restart recreating table with
>> id
>> = 12
>>      2015-11-13 06:09:21 [ndbd] INFO     -- Restart recreating table with
>> id
>> = 11
>>      2015-11-13 06:09:21 [ndbd] INFO     -- Restart recreating table with
>> id
>> = 13
>>      2015-11-13 06:09:21 [ndbd] INFO     -- Restart recreating table with
>> id
>> = 1
>>      2015-11-13 06:09:21 [ndbd] INFO     -- Restart recreating table with
>> id
>> = 43
>>      2015-11-13 06:09:21 [ndbd] INFO     -- Restart recreating table with
>> id
>> = 44
>>      2015-11-13 06:09:21 [ndbd] INFO     -- Restart recreating table with
>> id
>> = 76
>>      2015-11-13 06:09:21 [ndbd] INFO     -- Restart recreating table with
>> id
>> = 93
>>      2015-11-13 06:09:21 [ndbd] INFO     -- Restart recreating table with
>> id
>> = 102
>>      2015-11-13 06:09:21 [ndbd] INFO     -- Restart recreating table with
>> id
>> = 141
>>      2015-11-13 06:09:21 [ndbd] INFO     -- Restart recreating table with
>> id
>> = 149
>>      2015-11-13 06:09:21 [ndbd] INFO     -- Restart recreating table with
>> id
>> = 152
>>      2015-11-13 06:09:21 [ndbd] INFO     -- Restart recreating table with
>> id
>> = 156
>>      2015-11-13 06:09:21 [ndbd] INFO     -- Restart recreating table with
>> id
>> = 170
>>      restartDropObj(149)
>>      2015-11-13 06:09:22 [ndbd] ALERT    -- Node 10: Forced node shutdown
>> completed. Occured during startphase 0. Initiated by signal 11.
>>
>> Node can start with --initial parameter but when i restart this data after
>> starting (i mean after starting with --initial parameter) same situation
>> occurs.
>>
>> Do you have any suggestions?
>>
>>
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Reorganize Partition On Partial Cluster

Mauritz Sundell


On 2015-11-13 13:58, Serhat Rıfat Demircan wrote:

> Hi
>
> I both changed max_rows and run REORGANIZE PARTITIONS on same table.
>
> Followed these step:
> ndb_mgm> 10 stop
> mysql> ALTER TABLE x MAX_ROWS=1000000;
> mysql> ALTER TABLE x REORGANIZE PARTITIONS;
> user@ndb_10$ ndbmtd
>
>
> ndb_10_error.log:
> ------------------------
> Time: Monday 9 November 2015 - 01:00:23
> Status: Temporary error, restart node
> Message: Another node failed during system restart, please investigate
> error(s) on other node(s) (Restart error)
This says that another node have failed during system restart, in that
case no other nodes are allowed to start either.
Check the other ndb*error.log files for the real failure, probably node 6.

Mauritz

> Error: 2308
> Error data: Node 6 disconnected
> Error object: QMGR (Line: 3470) 0x00000002
> Program: ndbmtd
> Pid: 8269 thr: 0
> Version: mysql-5.6.24 ndb-7.4.6
> Trace: /ndbdata/ndb_5_trace.log.25 [t1..t15]
> ***EOM***
>
> On Fri, Nov 13, 2015 at 2:49 PM, Mauritz Sundell
> <[hidden email] <mailto:[hidden email]>> wrote:
>
>     Hi
>
>     On 2015-11-13 05:23, Serhat Rıfat Demircan wrote:
>
>         I have changed MAX_ROWS on partial (while a data node
>         disconnected) cluster
>         then reorganized partitions. When i try to restart data node
>         which was
>         disconnected i get following error:
>
>     Please provide the content of ndb_10_error.log.
>     And also the exact commands you did before restart.
>
>     Did you both change MAX_ROWS and run REORGANIZE PARTITIONS on same
>     table?
>
>     You can also report a bug at http://bugs.mysql.com/report.php
>     Attach in that case the ndb*error.log files, ndb*trace.log* files,
>     and the ndb*out.log files.
>
>     Regards,
>     Mauritz
>
>
>
>              2015-11-13 06:09:14 [ndbd] INFO     -- Angel connected to '
>         192.168.60.100:1186 <http://192.168.60.100:1186>'
>              2015-11-13 06:09:14 [ndbd] INFO     -- Angel allocated
>         nodeid: 10
>              serhatd@bung:~$ tail -f /usr/local/mysql/data/ndb_10_out.log
>              NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
>              Started thread, index = 33, id = 15809, type = NdbfsThread
>              NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
>              Started thread, index = 34, id = 15810, type = NdbfsThread
>              Started thread, index = 35, id = 15811, type = NdbfsThread
>              Started thread, index = 36, id = 15812, type = NdbfsThread
>              2015-11-13 06:09:15 [ndbd] INFO     -- Sending
>         READ_CONFIG_REQ to index
>         = 2, name = DBINFO
>              2015-11-13 06:09:15 [ndbd] INFO     -- Sending
>         READ_CONFIG_REQ to index
>         = 3, name = DBTUP
>              2015-11-13 06:09:15 [ndbd] INFO     -- Sending
>         READ_CONFIG_REQ to index
>         = 4, name = DBACC
>              2015-11-13 06:09:15 [ndbd] INFO     -- Sending
>         READ_CONFIG_REQ to index
>         = 5, name = DBTC
>              2015-11-13 06:09:16 [ndbd] INFO     -- timerHandlingLab,
>         expected 10ms
>         sleep, not scheduled for: 196 (ms)
>              2015-11-13 06:09:16 [ndbd] INFO     -- Sending
>         READ_CONFIG_REQ to index
>         = 6, name = DBLQH
>              LCPFragWatchdog : WarnElapsed : 20000(ms) MaxElapsed
>         60000(ms) : period
>         millis : 10000
>              2015-11-13 06:09:16 [ndbd] INFO     -- Sending
>         READ_CONFIG_REQ to index
>         = 7, name = DBTUX
>              2015-11-13 06:09:16 [ndbd] INFO     -- Sending
>         READ_CONFIG_REQ to index
>         = 8, name = DBDICT
>              2015-11-13 06:09:16 [ndbd] INFO     -- Sending
>         READ_CONFIG_REQ to index
>         = 9, name = DBDIH
>              2015-11-13 06:09:16 [ndbd] INFO     -- Sending
>         READ_CONFIG_REQ to index
>         = 10, name = NDBCNTR
>              2015-11-13 06:09:16 [ndbd] INFO     -- Sending
>         READ_CONFIG_REQ to index
>         = 11, name = QMGR
>              2015-11-13 06:09:16 [ndbd] INFO     -- Sending
>         READ_CONFIG_REQ to index
>         = 12, name = TRIX
>              2015-11-13 06:09:16 [ndbd] INFO     -- Sending
>         READ_CONFIG_REQ to index
>         = 13, name = BACKUP
>              2015-11-13 06:09:16 [ndbd] INFO     -- Sending
>         READ_CONFIG_REQ to index
>         = 14, name = DBUTIL
>              2015-11-13 06:09:16 [ndbd] INFO     -- Sending
>         READ_CONFIG_REQ to index
>         = 15, name = SUMA
>              2015-11-13 06:09:16 [ndbd] INFO     -- Sending
>         READ_CONFIG_REQ to index
>         = 16, name = TSMAN
>              2015-11-13 06:09:16 [ndbd] INFO     -- Sending
>         READ_CONFIG_REQ to index
>         = 17, name = LGMAN
>              2015-11-13 06:09:16 [ndbd] INFO     -- Sending
>         READ_CONFIG_REQ to index
>         = 18, name = PGMAN
>              pgman: page buffer entries = 10
>              pgman: page buffer entries = 10
>              2015-11-13 06:09:16 [ndbd] INFO     -- Sending
>         READ_CONFIG_REQ to index
>         = 19, name = RESTORE
>              2015-11-13 06:09:16 [ndbd] INFO     -- Sending
>         READ_CONFIG_REQ to index
>         = 20, name = DBSPJ
>              2015-11-13 06:09:16 [ndbd] INFO     -- Sending
>         READ_CONFIG_REQ to index
>         = 21, name = THRMAN
>              2015-11-13 06:09:16 [ndbd] INFO     -- READ_CONFIG_REQ
>         phase completed,
>         this phase is used to read configuration and to calculate
>         various sizes and
>         allocate almost all memory needed by the data node in its lifetime
>              2015-11-13 06:09:16 [ndbd] INFO     -- Start phase 0
>         completed
>              2015-11-13 06:09:16 [ndbd] INFO     -- Phase 0 has made
>         some file
>         system initialisations
>              2015-11-13 06:09:16 [ndbd] INFO     -- Starting QMGR phase 1
>              2015-11-13 06:09:16 [ndbd] INFO     -- DIH reported
>         normal start, now
>         starting the Node Inclusion Protocol
>              2015-11-13 06:09:16 [ndbd] INFO     -- findNeighbours
>         from: 2334 old
>         (left: 65535 right: 65535) new (13 11)
>              2015-11-13 06:09:16 [ndbd] INFO     -- Include node
>         protocol completed,
>         phase 1 in QMGR completed
>              2015-11-13 06:09:16 [ndbd] INFO     -- Start phase 1
>         completed
>              2015-11-13 06:09:16 [ndbd] INFO     -- Phase 1
>         initialised some
>         variables and included node in cluster, locked memory if
>         configured to do so
>              2015-11-13 06:09:16 [ndbd] INFO     -- Asking master node
>         to accept our
>         start (nodeId = 11 is master), GCI = 5622678
>              2015-11-13 06:09:16 [ndbd] INFO     -- NDBCNTR master
>         accepted us into
>         cluster, start NDB start phase 1
>              2015-11-13 06:09:16 [ndbd] INFO     -- We are performing
>         a node restart
>              2015-11-13 06:09:16 [ndbd] INFO     -- LDM(1): Started
>         LDM restart
>         phase 1 (read REDO log page headers to init REDO log data)
>              2015-11-13 06:09:16 [ndbd] INFO     -- Schema file
>         initialisation
>         Starting
>              2015-11-13 06:09:17 [ndbd] INFO     -- Schema file
>         initialisation
>         Completed
>              2015-11-13 06:09:17 [ndbd] INFO     -- NDB start phase 1
>         completed
>              2015-11-13 06:09:17 [ndbd] INFO     -- Start phase 2
>         completed
>              2015-11-13 06:09:17 [ndbd] INFO     -- Phase 2 did more
>         initialisations, master accepted our start, we started REDO log
>         initialisations
>              2015-11-13 06:09:17 [ndbd] INFO     -- Start NDB start
>         phase 2
>              2015-11-13 06:09:17 [ndbd] INFO     -- Receive arbitrator
>         node 1
>         [ticket=311700088796c28f]
>              2015-11-13 06:09:18 [ndbd] INFO     -- Request permission
>         to start our
>         node from master Starting
>              2015-11-13 06:09:18 [ndbd] INFO     -- Request permission
>         to start our
>         node from master Completed
>              2015-11-13 06:09:18 [ndbd] INFO     -- NDB start phase 2
>         completed
>              2015-11-13 06:09:18 [ndbd] INFO     -- Start phase 3
>         completed
>              2015-11-13 06:09:18 [ndbd] INFO     -- Phase 3 locked the
>         data
>         dictionary, performed local connection setups, we asked for
>         permission to
>         start our node
>              2015-11-13 06:09:18 [ndbd] INFO     -- Start NDB start
>         phase 3
>              2015-11-13 06:09:18 [ndbd] INFO     -- NDB start phase 3
>         completed
>              2015-11-13 06:09:18 [ndbd] INFO     -- Start phase 4
>         completed
>              2015-11-13 06:09:18 [ndbd] INFO     -- Phase 4 continued
>         preparations
>         of the REDO log
>              2015-11-13 06:09:18 [ndbd] INFO     -- Request copying of
>         distribution
>         and dictionary information from master Starting
>              2015-11-13 06:09:18 [ndbd] INFO     -- LDM(1):Ready to
>         start execute
>         REDO log phase, prepare REDO log phase completed
>              2015-11-13 06:09:21 [ndbd] INFO     -- Copying of dictionary
>         information from master Starting
>              2015-11-13 06:09:21 [ndbd] INFO     -- Restart recreating
>         table with id
>         = 10
>              2015-11-13 06:09:21 [ndbd] INFO     -- Restart recreating
>         table with id
>         = 12
>              2015-11-13 06:09:21 [ndbd] INFO     -- Restart recreating
>         table with id
>         = 11
>              2015-11-13 06:09:21 [ndbd] INFO     -- Restart recreating
>         table with id
>         = 13
>              2015-11-13 06:09:21 [ndbd] INFO     -- Restart recreating
>         table with id
>         = 1
>              2015-11-13 06:09:21 [ndbd] INFO     -- Restart recreating
>         table with id
>         = 43
>              2015-11-13 06:09:21 [ndbd] INFO     -- Restart recreating
>         table with id
>         = 44
>              2015-11-13 06:09:21 [ndbd] INFO     -- Restart recreating
>         table with id
>         = 76
>              2015-11-13 06:09:21 [ndbd] INFO     -- Restart recreating
>         table with id
>         = 93
>              2015-11-13 06:09:21 [ndbd] INFO     -- Restart recreating
>         table with id
>         = 102
>              2015-11-13 06:09:21 [ndbd] INFO     -- Restart recreating
>         table with id
>         = 141
>              2015-11-13 06:09:21 [ndbd] INFO     -- Restart recreating
>         table with id
>         = 149
>              2015-11-13 06:09:21 [ndbd] INFO     -- Restart recreating
>         table with id
>         = 152
>              2015-11-13 06:09:21 [ndbd] INFO     -- Restart recreating
>         table with id
>         = 156
>              2015-11-13 06:09:21 [ndbd] INFO     -- Restart recreating
>         table with id
>         = 170
>              restartDropObj(149)
>              2015-11-13 06:09:22 [ndbd] ALERT    -- Node 10: Forced
>         node shutdown
>         completed. Occured during startphase 0. Initiated by signal 11.
>
>         Node can start with --initial parameter but when i restart
>         this data after
>         starting (i mean after starting with --initial parameter) same
>         situation
>         occurs.
>
>         Do you have any suggestions?
>
>
>

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Reorganize Partition On Partial Cluster

Serhat Rıfat Demircan
Hi,

I think wrong log content was leaded to misundertand original problem. My
problem was running REORGANIZE PARTITIONS on partial cluster. I stopped a
data node 10 gracefully for maintenance then run REORGANIZE PARTITIONS on
partial cluster. After that data node could not start.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Reorganize Partition On Partial Cluster

Mauritz Sundell


On 2015-11-13 15:04, Serhat Rıfat Demircan wrote:
>
> Hi,
>
> I think wrong log content was leaded to misundertand original problem.
> My problem was running REORGANIZE PARTITIONS on partial cluster. I
> stopped a data node 10 gracefully for maintenance then run REORGANIZE
> PARTITIONS on partial cluster. After that data node could not start.
Ok.

Just to double check, there was no entry for Nov 13 06:09 in
/usr/local/mysql/data/ndb_10_error.log
as mentioned in /usr/local/mysql/data/ndb_10_out.log:
restartDropObj(149)
      2015-11-13 06:09:22 [ndbd] ALERT    -- Node 10: Forced node shutdown
completed. Occured during startphase 0. Initiated by signal 11.


If you can reproduce the failure I would like you to collect some
information between some steps:

> Followed these step:
> ndb_mgm> 10 stop
> mysql> ALTER TABLE x MAX_ROWS=1000000;

Run at ndb_mgmd host: ndb_show_tables > tables-before-reorg.txt

 > mysql> ALTER TABLE x REORGANIZE PARTITIONS;

Run at ndb_mgmd host: ndb_show_tables > tables-after-reorg.txt

 > user@ndb_10$ ndbmtd

If data node crash, check ndb_10_error.log.
And also check if there is a restartDropObj row right before crash in
ndb_10_out.log.
If there is, check which table it tried to drop looking in
tables-before-reorg.txt.
Do mysql> show create table <table-name-of restartDropObj number> \G

Send content of both tables-*-reorg.txt, tail of ndb_10_out.log,
ndb_10_error.log, and the show create table output.

If the node did not fail in same way as before.
Try to see if it is some reorganize some specific table that is the problem.
Do you have tables using:
  foreign keys?
  blob or text columns?
  unique index (not counting primary key) ?

Mauritz
Loading...