数据库迁移同步工具、ETL技术比较

2019年10月16日

数据库迁移同步工具比较
离线同步工具:
Sqoop、DataX

实时同步工具:
Canal、kettle、Informatica PowerCenter

DataX:

DataX使用指南
1.下载源代码

git clone https://github.com/sunshinelyz/mykit-db-sync.git

2.编译源代码

 cd {DataX_source_code_home}  #进入到源代码目录
 mvn -U clean package  assembly:assembly -Dmaven.test.skip=true  #maven编译   注意:编译过程中可能会出现编译问题:

2.1.q odps-sdk-core版本问题

Could not find artifact com.alibaba.external:bouncycastle.provider:jar:1.38-jdk15 in alimaven (https://maven.aliyun.com/nexus/content/repositories/central/) -> [Help 1]      

2.1.s 解决方法:

vim odpsreader/pom.xml
com.aliyun.odps;odps-sdk-core更换版本为0.20.7-public

2.2.q tablestore-streamclient版本问题

Could not find artifact com.aliyun.openservices:tablestore-streamclient:jar:1.0.0-SNAPSHOT

2.2.s 解决方法:
修改tablestore-streamclient的版本

<dependency>
    <groupId>com.aliyun.openservices</groupId>
    <artifactId>tablestore-streamclient</artifactId>
    <!--<version>1.0.0-SNAPSHOT</version>-->
    <version>1.0.0</version>
</dependency>

编译完成后 可用的datax包会生成在{DataX_source_code_home}/target/datax/datax目录下。

3.数据迁移同步
接下来就是进入到datax中进行数据迁移同步操作,

cd {DataX_source_code_home}/target/datax/datax  

3.1使用DataX抽取Oracle数据到MySQL
创建一个json文件oracle11mysql.json丢到job目录下:

{
"job": {
    "content": [
        {
            "reader": {
                "name": "oraclereader",
                "parameter": {
                    "column": ["zybh","xmbh","jsbh","xmmc","sfje","fylx","bz","sjc","zllx"],
                    "connection": [
                        {
                           "jdbcUrl": ["jdbc:oracle:thin:@192.168.11.91:1521:orcl"],
                            "table": ["JDZLJCXM"]
                        }
                    ],
                    "password": "jgzdwffz",
                    "username": "bjxxjgxt",
                }
            },
            "writer": {
                "name": "mysqlwriter",
                "parameter": {
                    "column": ["zybh","xmbh","jsbh","xmmc","sfje","fylx","bz","sjc","zllx"],
                    "connection": [
                        {
                            "jdbcUrl": "jdbc:mysql://192.168.11.75:3336/prison_practical_platform",
                            "table": ["JDZLJCXM"]
                        }
                    ],
                    "password": "root",
                    "username": "root",
                }
            }
        }
    ],
    "setting": {
        "speed": {
            "channel": "5"
        }
    }
}
}

启动命令: python datax.py ../job/oracle11mysql.json

参考文章:
DataX源码打包编译采坑记
通过DataX从Oracle同步数据到MySQL


相关内容已同步到justpic公众号

期待您的分享与讨论: