SpringBoot使用Sharding-JDBC實現數據分片和讀寫分離的方法
一、Sharding-JDBC簡介
Sharding-JDBC是Sharding-Sphere的一個產品,它有三個產品,分別是Sharding-JDBC、Sharding-Proxy和Sharding-Sidecar,這三個產品提供瞭標準化的數據分片、讀寫分離、柔性事務和數據治理功能。我們這裡用的是Sharding-JDBC,所以想瞭解後面兩個產品的話可以去它們官網查看。
Sharding-JDBC為輕量級Java框架,使用客戶端直連數據庫,以jar包形式提供服務,無需額外部署和依賴,可理解為增強版的JDBC驅動,兼容性特別強。適用的ORM框架有JPA, Hibernate, Mybatis, Spring JDBC Template或直接使用JDBC;第三方的數據庫連接池有DBCP, C3P0, BoneCP, Druid等;支持的數據庫有MySQL,Oracle,SQLServer和PostgreSQL;多樣化的配置文件Java,yaml,Spring Boot ,Spring命名空間。其實這裡說的都是廢話,大傢可以不看,下面我們動手開始正式配置。
二、具體的實現方式
1、maven引用
我這裡用的配置方式是Spring命名空間配置,所以隻需要引用sharding-jdbc-spring-namespace就可以瞭,還有要註意的是我用的不是當當網的sharding,註意groupId是io.shardingsphere。如果用的是其它配置方式可以去http://maven.aliyun.com/nexus/#nexus-search;quick~io.shardingsphere網站查找相應maven引用
<dependency> <groupId>io.shardingsphere</groupId> <artifactId>sharding-jdbc-spring-namespace</artifactId> <version>3.0.0.M1</version> </dependency>
2、數據庫準備
我這裡用的是mysql數據庫,根據我們項目的具體需求,我準備瞭三個主庫和對應的從庫。模擬的主庫名有master,暫時沒有做對應從庫,所以對應的從庫還是指向master;第二個主庫有master_1,對應的從庫有master_1_slaver_1,master_1_slave_2;第三個主庫有master_2,對應的從庫有master_2_slave_1,master_2_slave_2。
數據庫中的表也做瞭分表,下面是對應的mysql截圖。
這第一幅圖上的主從庫都應該在不同的服務器上的,但這裡隻是為瞭模擬所以就建在瞭本地服務器上瞭。我們讀寫分離中的寫操作隻會發生在主庫上,從庫會自動同步主庫上的數據並為讀提供數據。數據庫的主從復制在上篇博文中做瞭詳細的介紹,大傢可以去看看https://www.jb51.net/article/226077.htm
這幅圖作為我們本來的主庫,下面做的分庫和分表都是基於這個庫中的訂單表分的。所以分庫中的表隻有訂單表和訂單明細表。
第三幅圖截的是第二個主庫,裡面對訂單和訂單明細表做瞭分表操作,具體的分片策略和分片算法下面再做介紹。第三個主表和第二個主表是一樣的,所有的從表都和對應的主表是一致的。
3、Spring配置
數據庫信息配置文件db.properties配置可以配置兩份,分為開發版和測試版,如下:
# master Master.url=jdbc:mysql://localhost:3306/master?useUnicode=true&characterEncoding=utf8&autoReconnect=true&rewriteBatchedStatements=true Master.username=root Master.password=123456 Slave.url=jdbc:mysql://localhost:3306/master?useUnicode=true&characterEncoding=utf8&autoReconnect=true&rewriteBatchedStatements=true Slave.username=root Slave.password=123456 # maste_1 Master_1.url=jdbc:mysql://localhost:3306/master_1?useUnicode=true&characterEncoding=utf8&autoReconnect=true&rewriteBatchedStatements=true Master_1.username=root Master_1.password=123456 Master_1_Slave_1.url=jdbc:mysql://localhost:3306/master_1_slave_1?useUnicode=true&characterEncoding=utf8&autoReconnect=true&rewriteBatchedStatements=true Master_1_Slave_1.username=root Master_1_Slave_1.password=123456 Master_1_Slave_2.url=jdbc:mysql://localhost:3306/master_1_slave_2?useUnicode=true&characterEncoding=utf8&autoReconnect=true&rewriteBatchedStatements=true Master_1_Slave_2.username=root Master_1_Slave_2.password=123456 # master_2 Master_2.url=jdbc:mysql://localhost:3306/master_2?useUnicode=true&characterEncoding=utf8&autoReconnect=true&rewriteBatchedStatements=true Master_2.username=root Master_2.password=123456 Master_2_Slave_1.url=jdbc:mysql://localhost:3306/master_2_slave_1?useUnicode=true&characterEncoding=utf8&autoReconnect=true&rewriteBatchedStatements=true Master_2_Slave_1.username=root Master_2_Slave_1.password=123456 Master_2_Slave_2.url=jdbc:mysql://localhost:3306/master_2_slave_2?useUnicode=true&characterEncoding=utf8&autoReconnect=true&rewriteBatchedStatements=true Master_2_Slave_2.username=root Master_2_Slave_2.password=123456
Spring對應的配置:
Spring-Sphere官網中的demo裡用的都是行表達式的分片策略,但是行表達式的策略不利於數據庫和表的橫向擴展,所以我這裡用的是標準分片策略,精準分片算法和范圍分片算法。因為我們項目中暫時用的分片鍵都是user_id單一鍵,所以說不存在復合分片策略,也用不到Hint分片策略,行表達式分片策略和不分片策略。
<?xml version="1.0" encoding="UTF-8"?> <beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:context="http://www.springframework.org/schema/context" xmlns:tx="http://www.springframework.org/schema/tx" xmlns:sharding="http://shardingsphere.io/schema/shardingsphere/sharding" xmlns:master-slave="http://shardingsphere.io/schema/shardingsphere/masterslave" xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context.xsd http://www.springframework.org/schema/tx http://www.springframework.org/schema/tx/spring-tx.xsd http://shardingsphere.io/schema/shardingsphere/sharding http://shardingsphere.io/schema/shardingsphere/sharding/sharding.xsd http://shardingsphere.io/schema/shardingsphere/masterslave http://shardingsphere.io/schema/shardingsphere/masterslave/master-slave.xsd"> <context:component-scan base-package="com.jihao" /> <!-- db.properties數據庫信息配置 --> <bean id="property" class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer"> <property name="location" value="classpath:property/db_dev.properties" /> </bean> <!-- 主庫 --> <bean id="master" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close"> <property name="driverClassName" value="com.mysql.jdbc.Driver"/> <property name="url" value="${Master.url}"/> <property name="username" value="${Master.username}"/> <property name="password" value="${Master.password}"/> </bean> <!-- 主庫的從庫 --> <bean id="slave" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close"> <property name="driverClassName" value="com.mysql.jdbc.Driver"/> <property name="url" value="${Slave.url}"/> <property name="username" value="${Slave.username}"/> <property name="password" value="${Slave.password}"/> </bean> <!-- 主庫的分庫1 --> <bean id="master_1" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close"> <property name="driverClassName" value="com.mysql.jdbc.Driver"/> <property name="url" value="${Master_1.url}"/> <property name="username" value="${Master_1.username}"/> <property name="password" value="${Master_1.password}"/> </bean> <!-- 分庫1的從庫1 --> <bean id="master_1_slave_1" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close"> <property name="driverClassName" value="com.mysql.jdbc.Driver"/> <property name="url" value="${Master_1_Slave_1.url}"/> <property name="username" value="${Master_1_Slave_1.username}"/> <property name="password" value="${Master_1_Slave_1.password}"/> </bean> <!-- 分庫1的從庫2 --> <bean id="master_1_slave_2" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close"> <property name="driverClassName" value="com.mysql.jdbc.Driver"/> <property name="url" value="${Master_1_Slave_2.url}"/> <property name="username" value="${Master_1_Slave_2.username}"/> <property name="password" value="${Master_1_Slave_2.password}"/> </bean> <!-- 主庫的分庫2 --> <bean id="master_2" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close"> <property name="driverClassName" value="com.mysql.jdbc.Driver"/> <property name="url" value="${Master_2.url}"/> <property name="username" value="${Master_2.username}"/> <property name="password" value="${Master_2.password}"/> </bean> <!-- 分庫2的從庫1 --> <bean id="master_2_slave_1" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close"> <property name="driverClassName" value="com.mysql.jdbc.Driver"/> <property name="url" value="${Master_2_Slave_1.url}"/> <property name="username" value="${Master_2_Slave_1.username}"/> <property name="password" value="${Master_2_Slave_1.password}"/> </bean> <!-- 分庫2的從庫2 --> <bean id="master_2_slave_2" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close"> <property name="driverClassName" value="com.mysql.jdbc.Driver"/> <property name="url" value="${Master_2_Slave_2.url}"/> <property name="username" value="${Master_2_Slave_2.username}"/> <property name="password" value="${Master_2_Slave_2.password}"/> </bean> <!-- 主從關系配置 --> <bean id="randomStrategy" class="io.shardingsphere.core.api.algorithm.masterslave.RandomMasterSlaveLoadBalanceAlgorithm" /> <master-slave:data-source id="ms_master" master-data-source-name="master" slave-data-source-names="slave" strategy-ref="randomStrategy" /> <master-slave:data-source id="ms_master_1" master-data-source-name="master_1" slave-data-source-names="master_1_slave_1, master_1_slave_2" strategy-ref="randomStrategy" /> <master-slave:data-source id="ms_master_2" master-data-source-name="master_2" slave-data-source-names="master_2_slave_1, master_2_slave_2" strategy-ref="randomStrategy" /> <!-- 分庫策略 精確分片算法 --> <bean id="preciseDatabaseStrategy" class="com.jihao.algorithm.PreciseModuleDatabaseShardingAlgorithm" /> <!-- 分庫策略 范圍分片算法 --> <bean id="rangeDatabaseStrategy" class="com.jihao.algorithm.RangeModuleDatabaseShardingAlgorithm" /> <!-- 分表策略 精確分片算法 --> <bean id="preciseTableStrategy" class="com.jihao.algorithm.PreciseModuleTableShardingAlgorithm" /> <!-- 分表策略 范圍分片算法--> <bean id="rangeTableStrategy" class="com.jihao.algorithm.RangeModuleTableShardingAlgorithm" /> <sharding:standard-strategy id="databaseStrategy" sharding-column="user_id" precise-algorithm-ref="preciseDatabaseStrategy" range-algorithm-ref="rangeDatabaseStrategy" /> <!-- 分表策略 --> <sharding:standard-strategy id="tableStrategy" sharding-column="user_id" precise-algorithm-ref="preciseTableStrategy" range-algorithm-ref="rangeTableStrategy" /> <!-- 行表達式算法 --> <!-- <sharding:inline-strategy id="databaseStrategy" sharding-column="user_id" algorithm-expression="demo_ds_ms_$->{user_id % 2}" /> <sharding:inline-strategy id="orderTableStrategy" sharding-column="order_id" algorithm-expression="t_order_$->{order_id % 2}" /> <sharding:inline-strategy id="orderItemTableStrategy" sharding-column="order_item_id" algorithm-expression="t_order_item_$->{order_item_id % 2}" /> --> <sharding:data-source id="shardingDataSource"> <sharding:sharding-rule data-source-names="ms_master,ms_master_1,ms_master_2"> <sharding:table-rules> <sharding:table-rule logic-table="t_order" actual-data-nodes="ms_master_$->{1..2}.t_order_$->{1..3}" database-strategy-ref="databaseStrategy" table-strategy-ref="tableStrategy" generate-key-column-name="order_id"/> <sharding:table-rule logic-table="t_order_item" actual-data-nodes="ms_master_$->{1..2}.t_order_item_$->{1..3}" database-strategy-ref="databaseStrategy" table-strategy-ref="tableStrategy" generate-key-column-name="order_item_id"/> </sharding:table-rules> </sharding:sharding-rule> </sharding:data-source> <bean id="transactionManager" class="org.springframework.jdbc.datasource.DataSourceTransactionManager"> <property name="dataSource" ref="shardingDataSource" /> </bean> <tx:annotation-driven /> <bean id="sqlSessionFactory" class="org.mybatis.spring.SqlSessionFactoryBean"> <!-- 用於在控制臺打印sql(不需要的可以註釋掉這一行) --> <property name="configLocation" value="classpath:log/mybatis-config.xml"></property> <property name="dataSource" ref="shardingDataSource"/> <property name="mapperLocations" value="classpath*:com/jihao/mapper/*.xml"/> </bean> <bean class="org.mybatis.spring.mapper.MapperScannerConfigurer"> <property name="basePackage" value="com.jihao"/> <property name="sqlSessionFactoryBeanName" value="sqlSessionFactory"/> </bean> </beans>
4、精準分片算法和范圍分片算法的Java代碼
標準分片策略,精準分片算法
package com.jihao.algorithm; import io.shardingsphere.core.api.algorithm.sharding.PreciseShardingValue; import io.shardingsphere.core.api.algorithm.sharding.standard.PreciseShardingAlgorithm; import java.util.Collection; import com.alibaba.fastjson.JSON; /** * 自定義標準分片策略,使用精確分片算法(=與IN) * @author JiHao * */ public class PreciseModuleDatabaseShardingAlgorithm implements PreciseShardingAlgorithm<Long>{ @Override public String doSharding(Collection<String> availableTargetNames, PreciseShardingValue<Long> preciseShardingValue) { System.out.println("collection:" + JSON.toJSONString(availableTargetNames) + ",preciseShardingValue:" + JSON.toJSONString(preciseShardingValue)); for (String name : availableTargetNames) { // =與IN中分片鍵對應的值 String value = String.valueOf(preciseShardingValue.getValue()); // 分庫的後綴 int i = 1; // 求分庫後綴名的遞歸算法 if (name.endsWith("_" + countDatabaseNum(Long.parseLong(value), i))) { return name; } } throw new UnsupportedOperationException(); } /** * 計算該量級的數據在哪個數據庫 * @return */ private String countDatabaseNum(long columnValue, int i){ // ShardingSphereConstants每個庫中定義的數據量 long left = ShardingSphereConstants.databaseAmount * (i-1); long right = ShardingSphereConstants.databaseAmount * i; if(left < columnValue && columnValue <= right){ return String.valueOf(i); }else{ i++; return countDatabaseNum(columnValue, i); } } }
標準分片策略,范圍分片算法
package com.jihao.algorithm; import io.shardingsphere.core.api.algorithm.sharding.RangeShardingValue; import io.shardingsphere.core.api.algorithm.sharding.standard.RangeShardingAlgorithm; import java.util.ArrayList; import java.util.Collection; import java.util.List; import com.alibaba.fastjson.JSON; import com.google.common.collect.Range; /** * 自定義標準分庫策略,使用范圍分片算法(BETWEEN AND) * @author JiHao * */ public class RangeModuleDatabaseShardingAlgorithm implements RangeShardingAlgorithm<Long>{ @Override public Collection<String> doSharding( Collection<String> availableTargetNames, RangeShardingValue<Long> rangeShardingValue) { System.out.println("Range collection:" + JSON.toJSONString(availableTargetNames) + ",rangeShardingValue:" + JSON.toJSONString(rangeShardingValue)); Collection<String> collect = new ArrayList<>(); Range<Long> valueRange = rangeShardingValue.getValueRange(); // BETWEEN AND中分片鍵對應的最小值 long lowerEndpoint = Long.parseLong(String.valueOf(valueRange.lowerEndpoint())); // BETWEEN AND中分片鍵對應的最大值 long upperEndpoint = Long.parseLong(String.valueOf(valueRange.upperEndpoint())); // 分表的後綴 int i = 1; List<Integer> arrs = new ArrayList<Integer>(); // 求分表後綴名的遞歸算法 List<Integer> list = countDatabaseNum(i, lowerEndpoint, upperEndpoint, arrs); for (Integer integer : list) { for (String each : availableTargetNames) { if (each.endsWith("_" + integer)) { collect.add(each); } } } return collect; } /** * 計算該量級的數據在哪個表 * @param columnValue * @param i * @param lowerEndpoint 最小區間 * @param upperEndpoint 最大區間 * @return */ private List<Integer> countDatabaseNum(int i, long lowerEndpoint, long upperEndpoint, List<Integer> arrs){ long left = ShardingSphereConstants.databaseAmount * (i-1); long right = ShardingSphereConstants.databaseAmount * i; // 區間最大值小於分庫最大值 if(left < upperEndpoint && upperEndpoint <= right){ arrs.add(i); return arrs; }else{ if(left < lowerEndpoint && lowerEndpoint <= right){ arrs.add(i); } i++; return countDatabaseNum(i, lowerEndpoint, upperEndpoint, arrs); } } }
分庫的策略用的和分庫的代碼是一樣的,不同之處就是分庫用的是databaseAmount,分表用的是tableAmount。下面的ShardingSphereConstants的代碼。
package com.jihao.algorithm; /** * ShardingSphere中用到的常量 * @author JiHao * */ public class ShardingSphereConstants { /** * 訂單、優惠券相關的表,按用戶數量分庫,64w用戶數據為一個庫 * (0,64w] */ public static int databaseAmount = 640000; /** * 一個訂單表裡存10000的用戶訂單 * (0,1w] */ public static int tableAmount = 10000; }
到這裡所有的配置基本上都已經完成瞭,下面的測試。
5、測試
下面是測試的mybatis的測試文件,都是最基礎的就不講解瞭。
<?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE mapper PUBLIC "-//mybatis.org//DTD Mapper 3.0//EN" "http://mybatis.org/dtd/mybatis-3-mapper.dtd"> <mapper namespace="com.jihao.dao.TestShardingMapper"> <resultMap id="BaseResultMap" type="com.jihao.entity.Order"> <id column="order_id" jdbcType="INTEGER" property="orderId" /> <result column="user_id" jdbcType="INTEGER" property="userId" /> <result column="status" jdbcType="INTEGER" property="status" /> </resultMap> <insert id="insert" parameterType="com.jihao.entity.Order" useGeneratedKeys="true" keyProperty="orderId"> INSERT INTO t_order ( user_id, status ) VALUES ( #{userId,jdbcType=INTEGER}, #{status,jdbcType=VARCHAR} ) </insert> <insert id="insertItem" useGeneratedKeys="true" keyProperty="orderItemId"> INSERT INTO t_order_item ( order_id, user_id ) VALUES ( #{orderId,jdbcType=INTEGER}, #{userId,jdbcType=INTEGER} ) </insert> <select id="searchOrder" resultMap="BaseResultMap"> SELECT * from t_order </select> <select id="queryWithEqual" resultMap="BaseResultMap"> SELECT * FROM t_order WHERE user_id=51 </select> <select id="queryWithIn" resultMap="BaseResultMap"> SELECT * FROM t_order WHERE user_id IN (50, 51) </select> <select id="queryWithBetween" resultMap="BaseResultMap"> SELECT * FROM t_order WHERE user_id between 10000 and 30000 </select> <select id="queryUser" resultType="Map"> SELECT * FROM t_user </select> </mapper>
下面對應的mapper的Java代碼
package com.jihao.dao; import java.util.List; import java.util.Map; import org.apache.ibatis.annotations.Mapper; import com.jihao.entity.Order; import com.jihao.entity.OrderItem; @Mapper public interface TestShardingMapper { int insert(Order record); int insertItem(OrderItem record); List<Order> searchOrder(); List<Order> queryWithEqual(); List<Order> queryWithIn(); List<Order> queryWithBetween(); List<Map<String, Object>> queryUser(); }
下面是對應的訂單entity代碼
package com.jihao.entity; /** * 訂單 * @author JiHao */ public class Order { private Long orderId; private Integer userId; private String status; public Long getOrderId() { return orderId; } public void setOrderId(Long orderId) { this.orderId = orderId; } public Integer getUserId() { return userId; } public void setUserId(Integer userId) { this.userId = userId; } public String getStatus() { return status; } public void setStatus(String status) { this.status = status; } }
下面是對應的訂單明細entity代碼
package com.jihao.entity; /** * 測試分片 * @author JiHao */ public class OrderItem { private Long orderItemId; private Long orderId; private Integer userId; public Long getOrderId() { return orderId; } public void setOrderId(Long orderId) { this.orderId = orderId; } public Integer getUserId() { return userId; } public void setUserId(Integer userId) { this.userId = userId; } public Long getOrderItemId() { return orderItemId; } public void setOrderItemId(Long orderItemId) { this.orderItemId = orderItemId; } }
下面是測試的controller,並沒有寫Junit測試。
package com.jihao.controller.test; import java.util.List; import java.util.Map; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.stereotype.Controller; import org.springframework.web.bind.annotation.GetMapping; import org.springframework.web.bind.annotation.RequestMapping; import org.springframework.web.bind.annotation.ResponseBody; import com.jihao.dao.TestShardingMapper; import com.jihao.entity.Order; import com.jihao.entity.OrderItem; import com.jihao.result.Result; import com.jihao.result.ResultUtil; /** * 測試分片 * @author JiHao * */ @Controller @RequestMapping(value = "test") public class TestShardingController { @Autowired private TestShardingMapper testShardingMapper; /** * 測試添加 * @return */ @ResponseBody @GetMapping(value = "/testAdd") public String testAdd(){ for (int i = 0; i < 10; i++) { Order order = new Order(); // order.setUserId(50); // order.setUserId(51); // order.setUserId(10001); order.setUserId(20001); order.setStatus("INSERT_TEST"); int count = testShardingMapper.insert(order); System.out.println(count); long orderId = order.getOrderId(); System.out.println(order.getOrderId()); OrderItem item = new OrderItem(); item.setOrderId(orderId); // order.setUserId(50); // order.setUserId(51); // order.setUserId(10001); order.setUserId(20001); testShardingMapper.insertItem(item); } return "success"; } /** * 測試搜索 * @return */ @ResponseBody @GetMapping(value = "/testSearch") public Result searchData(){ List<Order> list = testShardingMapper.searchOrder(); System.out.println(list.size() + " all"); List<Order> list1 = testShardingMapper.queryWithIn(); System.out.println(list1.size() + " In"); List<Order> list2 = testShardingMapper.queryWithEqual(); System.out.println(list2.size() + " Equal"); List<Order> list3 = testShardingMapper.queryWithBetween(); System.out.println(list3.size() + " Between"); List<Map<String, Object>> list4 = testShardingMapper.queryUser(); System.out.println(list4.size() + " user"); return ResultUtil.success(null); } }
這裡要重點提出來的是做搜索測試的時候,因為主從庫都在我本地服務器上,並沒有做主從復制,大傢可以根據我上篇博文配置一下就可以順利操作瞭,如果沒有配置的話從庫裡是不會有數據的,所以在做完寫操作時把主庫中的數據手動傳輸給從庫,這樣才能讀出數據。
這裡順便給出Sharding-Sphere的官方地址http://shardingjdbc.io/index_zh.html,以及demo地址https://github.com/sharding-sphere/sharding-sphere-example(demo裡Sharding-Sphere的maven配置我在跑的時候沒跑通,需要把版本改成3.0.0.M1就ok瞭)。
到此這篇關於SpringBoot使用Sharding-JDBC實現數據分片和讀寫分離的文章就介紹到這瞭,更多相關SpringBoot使用Sharding-JDBC實現數據分片和讀寫分離內容請搜索WalkonNet以前的文章或繼續瀏覽下面的相關文章希望大傢以後多多支持WalkonNet!
推薦閱讀:
- 利用Sharding-Jdbc進行分庫分表的操作代碼
- 使用sharding-jdbc實現水平分庫+水平分表的示例代碼
- SpringBoot項目中如何實現MySQL讀寫分離詳解
- SpringBoot+MybatisPlus+Mysql+Sharding-JDBC分庫分表
- SpringBoot詳解如何實現讀寫分離