Mysql Cluster Data Node Hangs on Phase 5 While Starting

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Mysql Cluster Data Node Hangs on Phase 5 While Starting

Serhat Rıfat Demircan
I have an mysql cluster which have 4 api nodes, 2 management nodes and 4
data nodes. Today one of data nodes has gone offline due missing heartbeat.
After i tried to restart, it hangs on phase 5.


I see many of following lines on data nodes log file:


2015-08-25 00:55:33 [ndbd] INFO     -- UPDATE_TOREF: thread: 1, state:6

2015-08-25 00:55:38 [ndbd] INFO     -- UPDATE_TOREF: thread: 1, state:6

2015-08-25 00:55:43 [ndbd] INFO     -- UPDATE_TOREF: thread: 1, state:6

2015-08-25 00:55:48 [ndbd] INFO     -- UPDATE_TOREF: thread: 1, state:6

2015-08-25 00:55:53 [ndbd] INFO     -- UPDATE_TOREF: thread: 1, state:6

2015-08-25 00:55:58 [ndbd] INFO     -- UPDATE_TOREF: thread: 1, state:6

2015-08-25 00:56:03 [ndbd] INFO     -- UPDATE_TOREF: thread: 1, state:6

2015-08-25 00:56:08 [ndbd] INFO     -- UPDATE_TOREF: thread: 1, state:6

2015-08-25 00:56:13 [ndbd] INFO     -- UPDATE_TOREF: thread: 1, state:6

2015-08-25 00:56:18 [ndbd] INFO     -- UPDATE_TOREF: thread: 1, state:6

and same time on anoter data node:


2015-08-25 00:56:58 [ndbd] INFO     -- takeOver node in progress: 5

2015-08-25 00:57:03 [ndbd] INFO     -- takeOver node in progress: 5

2015-08-25 00:57:08 [ndbd] INFO     -- takeOver node in progress: 5

2015-08-25 00:57:13 [ndbd] INFO     -- takeOver node in progress: 5

2015-08-25 00:57:18 [ndbd] INFO     -- takeOver node in progress: 5

2015-08-25 00:57:23 [ndbd] INFO     -- takeOver node in progress: 5

2015-08-25 00:57:28 [ndbd] INFO     -- takeOver node in progress: 5

2015-08-25 00:57:33 [ndbd] INFO     -- takeOver node in progress: 5

2015-08-25 00:57:38 [ndbd] INFO     -- takeOver node in progress: 5

2015-08-25 00:57:43 [ndbd] INFO     -- takeOver node in progress: 5

How can i bring this node online again?
Loading...