/canal

Alibaba mysql database binlog incremental subscriptions & consumer components

Primary LanguageJavaApache License 2.0Apache-2.0

The latest update

  1. Canal qq discussion group has been established, group number: 161559791, welcome to join the technical discussion.
  2. Canal consumer side project open source: Otter (distributed database synchronization system), address: https://github.com/alibaba/otter

Background

Early, Alibaba B2B companies because of the existence of Hangzhou and the United States dual-room deployment, there is a cross-room synchronization business needs. However, the early database synchronization business, mainly based on the trigger way to obtain incremental changes, but since 2010, the Department of the company began to gradually try to log based on the database analysis, access to incremental changes to synchronize, which derived from the increase The amount of subscription & consumer business, since then opened a new era.

Ps. The current version has support mysql and oracle part of the log analysis, the current canal open source version supports 5.7 and below (Ali internal mysql 5.7.13, 5.6.10, mysql 5.5.18 and 5.1.40 / 48)

Based on log incremental subscriptions & consumer support business:

  1. Database mirroring
  2. Database real-time backup
  3. Multi-level index (seller and buyer each sub-index)
  4. search build
  5. Business cache refresh
  6. Price changes and other important business news

Project Introduction

Name: canal [kə'næl]

Waterways / pipes / ditches

Language: pure java development

Positioning: based on database incremental log analysis, to provide incremental data subscriptions & consumption, the current main support for the mysql

Keywords: mysql binlog parser / real-time / queue & topic

working principle

Mysql master and backup copy to achieve


From the top, replication is divided into three steps:

  1. Master records changes to the binary log (these records are called binary log events, binary log events, can be viewed through show binlog events)；
  2. The slave copies the master binary log events to its relay log;
  3. The slave redo the event in the relay log, which will change its own data.

Canal's working principle:

The principle is relatively simple:：

  1. Canal simulation mysql slave interactive protocol, disguised as mysql slave, mysql master to send the dump agreement
  2. Mysql master received a request to start, to push binary log to the slave (that is, canal)
  3. Canal parse binary log object (original byte stream)

Related documentation

See the wiki page for : wiki documentation

wiki documentation

Relevant information

Related open source

  1. Alibaba distributed database synchronization system (to solve the different room): http://github.com/alibaba/otter
  2. Alibaba to Oracle data migration synchronization tool (target support MySQL / DRDS)：http://github.com/alibaba/yugong

feedback

  1. Qq exchange group: 161559791
  2. Mail exchange: jianghang115@gmail.com
  3. Sina microblogging: agapple0002
  4. Report issue: issues
[Recruitment] Alibaba middleware team recruitment JAVA senior engineer
The main job for the technical content (non-business sector), Ali middleware the whole system for the future want to have the students in the technical precipitation is still very helpful
Work Location: Hangzhou, Beijing can be. Ps. Ali has always been a good treatment, interested who can QQ, microblogging private chat. 
Specific recruitment content: https: //job.alibaba.com/zhaopin/position_detail.htm? PositionId = 32666