Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 26 additions & 13 deletions docs/en/structure/check.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,22 +29,35 @@ is silenced, and miss, diff, summary, and SQL records are emitted to stdout as o
part of this stdout result stream. The JSON payloads are the same objects as the file logs; SQL
payloads are the same plain statements as `sql.log`.

`miss.log` and `diff.log` use the same JSON structure (`StructCheckLog`). `src_sql` and
`dst_sql` are optional and appear only when the corresponding side has a definition:
`miss.log` and `diff.log` use the same JSON structure (`StructCheckLog`). `schema` and `tb`
identify the source object. `src_sql` and `dst_sql` are optional and appear only when the
corresponding side has a definition:

```json
{
"key": "index.db_name.tb_name.idx_name",
"key": "index.src_db.src_tb.idx_name",
"schema": "src_db",
"tb": "src_tb",
"target_schema": "dst_db",
"target_tb": "dst_tb",
"src_sql": "source definition SQL",
"dst_sql": "target definition SQL"
}
```

`key` identifies the structure object and is always present. Structure logs do not contain
`schema`, `tb`, `id_col_values`, `target_schema`, or `target_tb`. `src_sql` is included when the
source-side definition exists; `dst_sql` is included when the target-side definition exists.
Source-only missing objects usually have only `src_sql`; objects with different definitions have
both `src_sql` and `dst_sql`; target-only extra objects have only `dst_sql`.
`key` identifies the source structure object and is always present. `db` is emitted only for a
database type with a separate database dimension. `target_db`, `target_schema`, and `target_tb`
are omitted when routing does not rename the destination object. When either the schema or table
name changes, `target_schema` and `target_tb` are emitted together. Structure logs do not contain
`id_col_values`. `src_sql` is included when the source-side definition exists; `dst_sql` is
included when the target-side definition exists. Source-only missing objects usually have only
`src_sql`; objects with different definitions have both `src_sql` and `dst_sql`; target-only extra
objects have only `dst_sql`.

Internally, the checker parses `key` into a structured object location and keeps both the source
and routed target locations. The external `key` remains a string for backward compatibility;
`schema`, `tb`, and the optional target fields are generated from that structured location, so
consumers do not need to parse `key` again.

The structure key has this format:

Expand All @@ -62,15 +75,15 @@ object-specific keys such as `udt.schema.type_name`, `udf.schema.function_name(a

- `miss.log` (present in source but missing in target)
```json
{"key":"table.struct_check_test_1.not_match_miss","src_sql":"CREATE TABLE `not_match_miss` (`id` int NOT NULL, PRIMARY KEY (`id`))"}
{"key":"index.struct_check_test_1.not_match_index.i6_miss","src_sql":"CREATE INDEX `i6_miss` ON `not_match_index` (`c6`)"}
{"key":"table.struct_check_test_1.not_match_miss","schema":"struct_check_test_1","tb":"not_match_miss","src_sql":"CREATE TABLE `not_match_miss` (`id` int NOT NULL, PRIMARY KEY (`id`))"}
{"key":"index.struct_check_test_1.not_match_index.i6_miss","schema":"struct_check_test_1","tb":"not_match_index","src_sql":"CREATE INDEX `i6_miss` ON `not_match_index` (`c6`)"}
```

- `diff.log` (object definition differs, or the object exists only in the target)
```json
{"key":"index.struct_check_test_1.not_match_index.i1","src_sql":"CREATE INDEX `i1` ON `not_match_index` (`c1`)","dst_sql":"CREATE INDEX `i1` ON `not_match_index` (`c2`)"}
{"key":"table.struct_check_test_1.not_match_column","src_sql":"CREATE TABLE `not_match_column` (`id` int NOT NULL, PRIMARY KEY (`id`))","dst_sql":"CREATE TABLE `not_match_column` (`id` bigint NOT NULL, PRIMARY KEY (`id`))"}
{"key":"index.struct_check_test_1.full_index_type.index_not_match_name_dst","dst_sql":"CREATE INDEX `index_not_match_name_dst` ON `full_index_type` (`c1`)"}
{"key":"index.struct_check_test_1.not_match_index.i1","schema":"struct_check_test_1","tb":"not_match_index","src_sql":"CREATE INDEX `i1` ON `not_match_index` (`c1`)","dst_sql":"CREATE INDEX `i1` ON `not_match_index` (`c2`)"}
{"key":"table.struct_check_test_1.not_match_column","schema":"struct_check_test_1","tb":"not_match_column","src_sql":"CREATE TABLE `not_match_column` (`id` int NOT NULL, PRIMARY KEY (`id`))","dst_sql":"CREATE TABLE `not_match_column` (`id` bigint NOT NULL, PRIMARY KEY (`id`))"}
{"key":"index.struct_check_test_1.full_index_type.index_not_match_name_dst","schema":"struct_check_test_1","tb":"full_index_type","dst_sql":"CREATE INDEX `index_not_match_name_dst` ON `full_index_type` (`c1`)"}
```

- `summary.log` (overview of the check results)
Expand Down
34 changes: 22 additions & 12 deletions docs/zh/structure/check.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,21 +27,31 @@ artifact,普通运行日志不会输出到 stdout,miss、diff、summary、SQ
`diff_logger`、`sql_logger` 属于 stdout 结果流;JSON payload 与文件日志中的对象一致,SQL
payload 与 `sql.log` 中的纯 SQL 语句一致。

`miss.log` 和 `diff.log` 均采用相同的 JSON 结构(`StructCheckLog`)。`src_sql` 与
`dst_sql` 是可选字段,仅在对应侧存在结构定义时输出:
`miss.log` 和 `diff.log` 均采用相同的 JSON 结构(`StructCheckLog`)。`schema`、`tb` 用于
定位源端对象;`src_sql` 与 `dst_sql` 是可选字段,仅在对应侧存在结构定义时输出:

```json
{
"key": "index.db_name.tb_name.idx_name",
"key": "index.src_db.src_tb.idx_name",
"schema": "src_db",
"tb": "src_tb",
"target_schema": "dst_db",
"target_tb": "dst_tb",
"src_sql": "source definition SQL",
"dst_sql": "target definition SQL"
}
```

`key` 用于定位结构对象,并且始终存在。结构日志没有 `schema`、`tb`、`id_col_values`、
`target_schema`、`target_tb` 字段。源端定义存在时输出 `src_sql`,目标端定义存在时输出
`dst_sql`。源端独有的缺失对象通常只有 `src_sql`;定义不一致的对象同时包含 `src_sql`
和 `dst_sql`;目标端独有的额外对象只有 `dst_sql`。
`key` 用于定位源端结构对象,并且始终存在。`db` 仅在数据库类型具有独立 database 维度时
输出;未发生路由改名时,不输出 `target_db`、`target_schema`、`target_tb`。只要 schema 或
table 任一名称发生变化,`target_schema` 与 `target_tb` 会同时输出。结构日志不包含
`id_col_values`。源端定义存在时输出 `src_sql`,目标端定义存在时输出 `dst_sql`。源端独有的
缺失对象通常只有 `src_sql`;定义不一致的对象同时包含 `src_sql` 和 `dst_sql`;目标端独有的
额外对象只有 `dst_sql`。

校验器内部将 `key` 解析为结构化对象位置,并同时保留源端和路由后的目标端位置;对外仍输出
字符串 `key`,以兼容已有消费者。`schema`、`tb` 以及可选的 target 字段均由该结构化位置生成,
不需要消费者再次拆分 `key`。

结构 key 格式:

Expand All @@ -59,15 +69,15 @@ key,例如 `udt.schema.type_name`、`udf.schema.function_name(arguments)` 和

- `miss.log`(源端存在但目标端缺失)
```json
{"key":"table.struct_check_test_1.not_match_miss","src_sql":"CREATE TABLE `not_match_miss` (`id` int NOT NULL, PRIMARY KEY (`id`))"}
{"key":"index.struct_check_test_1.not_match_index.i6_miss","src_sql":"CREATE INDEX `i6_miss` ON `not_match_index` (`c6`)"}
{"key":"table.struct_check_test_1.not_match_miss","schema":"struct_check_test_1","tb":"not_match_miss","src_sql":"CREATE TABLE `not_match_miss` (`id` int NOT NULL, PRIMARY KEY (`id`))"}
{"key":"index.struct_check_test_1.not_match_index.i6_miss","schema":"struct_check_test_1","tb":"not_match_index","src_sql":"CREATE INDEX `i6_miss` ON `not_match_index` (`c6`)"}
```

- `diff.log`(对象定义不一致,或对象仅存在于目标端)
```json
{"key":"index.struct_check_test_1.not_match_index.i1","src_sql":"CREATE INDEX `i1` ON `not_match_index` (`c1`)","dst_sql":"CREATE INDEX `i1` ON `not_match_index` (`c2`)"}
{"key":"table.struct_check_test_1.not_match_column","src_sql":"CREATE TABLE `not_match_column` (`id` int NOT NULL, PRIMARY KEY (`id`))","dst_sql":"CREATE TABLE `not_match_column` (`id` bigint NOT NULL, PRIMARY KEY (`id`))"}
{"key":"index.struct_check_test_1.full_index_type.index_not_match_name_dst","dst_sql":"CREATE INDEX `index_not_match_name_dst` ON `full_index_type` (`c1`)"}
{"key":"index.struct_check_test_1.not_match_index.i1","schema":"struct_check_test_1","tb":"not_match_index","src_sql":"CREATE INDEX `i1` ON `not_match_index` (`c1`)","dst_sql":"CREATE INDEX `i1` ON `not_match_index` (`c2`)"}
{"key":"table.struct_check_test_1.not_match_column","schema":"struct_check_test_1","tb":"not_match_column","src_sql":"CREATE TABLE `not_match_column` (`id` int NOT NULL, PRIMARY KEY (`id`))","dst_sql":"CREATE TABLE `not_match_column` (`id` bigint NOT NULL, PRIMARY KEY (`id`))"}
{"key":"index.struct_check_test_1.full_index_type.index_not_match_name_dst","schema":"struct_check_test_1","tb":"full_index_type","dst_sql":"CREATE INDEX `index_not_match_name_dst` ON `full_index_type` (`c1`)"}
```

- `summary.log`(校验结果概览)
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
use crate::meta::struct_meta::statement::struct_statement::{StructKey, StructKeyType};
use crate::meta::struct_meta::structure::{database::Database, structure_type::StructureType};
use crate::rdb_filter::RdbFilter;

Expand All @@ -11,7 +12,7 @@ impl MysqlCreateDatabaseStatement {
self.database.name = dst_db.to_string();
}

pub fn to_sqls(&self, filter: &RdbFilter) -> anyhow::Result<Vec<(String, String)>> {
pub fn to_sqls(&self, filter: &RdbFilter) -> anyhow::Result<Vec<(StructKey, String)>> {
let mut sqls = Vec::new();
if filter.filter_structure(&StructureType::Database) {
return Ok(sqls);
Expand All @@ -31,7 +32,7 @@ impl MysqlCreateDatabaseStatement {
)
}

let key = format!("database.{}", self.database.name.clone());
let key = StructKey::new(StructKeyType::Database, [&self.database.name]);
sqls.push((key, sql));
Ok(sqls)
}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
use crate::meta::struct_meta::statement::struct_statement::{StructKey, StructKeyType};
use crate::meta::struct_meta::structure::column::ColumnDefault;
use crate::meta::struct_meta::structure::index::IndexType;
use crate::meta::struct_meta::structure::{
Expand Down Expand Up @@ -32,13 +33,13 @@ impl MysqlCreateTableStatement {
}
}

pub fn to_sqls(&mut self, filter: &RdbFilter) -> anyhow::Result<Vec<(String, String)>> {
pub fn to_sqls(&mut self, filter: &RdbFilter) -> anyhow::Result<Vec<(StructKey, String)>> {
let mut sqls = Vec::new();

if !filter.filter_structure(&StructureType::Table) {
let key = format!(
"table.{}.{}",
self.table.database_name, self.table.table_name
let key = StructKey::new(
StructKeyType::Table,
[&self.table.database_name, &self.table.table_name],
);
sqls.push((key, Self::table_to_sql(&mut self.table)));
}
Expand Down Expand Up @@ -66,18 +67,18 @@ impl MysqlCreateTableStatement {
idx_appends.push(Self::index_to_sql_appends(i));
}
_ => {
let standalone_key = format!(
"index.{}.{}.{}",
i.database_name, i.table_name, i.index_name
let standalone_key = StructKey::new(
StructKeyType::Index,
[&i.database_name, &i.table_name, &i.index_name],
);
sqls.push((standalone_key, Self::index_to_sql(i)))
}
}
}
if !idx_appends.is_empty() {
let key = format!(
"index.{}.{}",
self.indexes[0].database_name, self.indexes[0].table_name
let key = StructKey::new(
StructKeyType::Index,
[&self.indexes[0].database_name, &self.indexes[0].table_name],
);
sqls.push((
key,
Expand All @@ -93,9 +94,9 @@ impl MysqlCreateTableStatement {

if !filter.filter_structure(&StructureType::Constraint) {
for i in self.constraints.iter() {
let key = format!(
"constraint.{}.{}.{}",
i.database_name, i.table_name, i.constraint_name
let key = StructKey::new(
StructKeyType::Constraint,
[&i.database_name, &i.table_name, &i.constraint_name],
);
sqls.push((key, Self::constraint_to_sql(i)));
}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
use crate::meta::struct_meta::statement::struct_statement::{StructKey, StructKeyType};
use crate::meta::struct_meta::structure::{
rbac::PgPrivilege, rbac::PgRole, rbac::PgRoleMember, structure_type::StructureType,
};
Expand All @@ -11,7 +12,7 @@ pub struct PgCreateRbacStatement {
}

impl PgCreateRbacStatement {
pub fn to_sqls(&self, filter: &RdbFilter) -> anyhow::Result<Vec<(String, String)>> {
pub fn to_sqls(&self, filter: &RdbFilter) -> anyhow::Result<Vec<(StructKey, String)>> {
let mut sqls = Vec::new();
if filter.filter_structure(&StructureType::Rbac) {
return Ok(sqls);
Expand Down Expand Up @@ -70,7 +71,7 @@ impl PgCreateRbacStatement {
sql = format!("{} WITH {}", sql, options.join(" "));
}

sqls.push((format!("rbac.role.{}", role.name), sql));
sqls.push((StructKey::new(StructKeyType::RbacRole, [&role.name]), sql));

if !role.rol_configs.is_empty() {
for config in &role.rol_configs {
Expand All @@ -83,7 +84,10 @@ impl PgCreateRbacStatement {
let alter_sql =
format!("ALTER ROLE \"{}\" SET {} TO '{}'", role.name, param, value);
sqls.push((
format!("rbac.role_config.{}.{}", role.name, param),
StructKey::new(
StructKeyType::RbacRoleConfig,
[role.name.clone(), param.to_string()],
),
alter_sql,
));
}
Expand All @@ -98,9 +102,13 @@ impl PgCreateRbacStatement {
sql = format!("{} WITH ADMIN OPTION", sql);
}
sqls.push((
format!(
"rbac.member.{}.{}.{}",
member.role, member.member, member.admin_option
StructKey::new(
StructKeyType::RbacMember,
[
member.role.clone(),
member.member.clone(),
member.admin_option.to_string(),
],
),
sql,
));
Expand Down Expand Up @@ -394,7 +402,10 @@ mod tests {
};

let privilege = PgPrivilege {
key: "rbac.privilege.table.public.test_table.test_role.NO".to_string(),
key: StructKey::new(
StructKeyType::RbacPrivilegeTable,
["public", "test_table", "test_role", "NO"],
),
origin: "GRANT SELECT ON TABLE public.test_table TO \"test_role\"".to_string(),
};

Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
use crate::meta::struct_meta::statement::struct_statement::{StructKey, StructKeyType};
use crate::meta::struct_meta::structure::{schema::Schema, structure_type::StructureType};
use crate::rdb_filter::RdbFilter;

Expand All @@ -11,13 +12,13 @@ impl PgCreateSchemaStatement {
self.schema.name = dst_schema.to_string();
}

pub fn to_sqls(&self, filter: &RdbFilter) -> anyhow::Result<Vec<(String, String)>> {
pub fn to_sqls(&self, filter: &RdbFilter) -> anyhow::Result<Vec<(StructKey, String)>> {
let mut sqls = Vec::new();
if filter.filter_structure(&StructureType::Database) {
return Ok(sqls);
}

let key = format!("schema.{}", self.schema.name);
let key = StructKey::new(StructKeyType::Schema, [&self.schema.name]);
let sql = format!(r#"CREATE SCHEMA IF NOT EXISTS "{}""#, self.schema.name);
sqls.push((key, sql));
Ok(sqls)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ use crate::config::config_enums::DbType;
use crate::error::DtError;
use crate::meta::ddl_meta::ddl_parser::DdlParser;
use crate::meta::ddl_meta::ddl_statement::DdlStatement;
use crate::meta::struct_meta::statement::struct_statement::{StructKey, StructKeyType};
use crate::meta::struct_meta::structure::column::ColumnDefault;
use crate::meta::struct_meta::structure::{
column::Column,
Expand Down Expand Up @@ -62,36 +63,41 @@ impl PgCreateTableStatement {
}
}

pub fn to_sqls(&mut self, filter: &RdbFilter) -> anyhow::Result<Vec<(String, String)>> {
pub fn to_sqls(&mut self, filter: &RdbFilter) -> anyhow::Result<Vec<(StructKey, String)>> {
let mut sqls = Vec::new();

if !filter.filter_structure(&StructureType::Table) {
for i in self.sequences.iter() {
let key = format!("sequence.{}.{}", i.schema_name, i.sequence_name);
let key =
StructKey::new(StructKeyType::Sequence, [&i.schema_name, &i.sequence_name]);
sqls.push((key, Self::sequence_to_sql(i)));
}

let key = format!("table.{}.{}", self.table.schema_name, self.table.table_name);
let key = StructKey::new(
StructKeyType::Table,
[&self.table.schema_name, &self.table.table_name],
);
sqls.push((key, Self::table_to_sql(&mut self.table)));

for i in self.sequence_owners.iter() {
let key = format!(
"sequence_owner.{}.{}.{}",
i.schema_name, i.table_name, i.sequence_name
let key = StructKey::new(
StructKeyType::SequenceOwner,
[&i.schema_name, &i.table_name, &i.sequence_name],
);
sqls.push((key, Self::sequence_owner_to_sql(i)));
}

for i in self.column_comments.iter() {
let key = format!(
"column_comment.{}.{}.{}",
i.schema_name, i.table_name, i.column_name
let key = StructKey::new(
StructKeyType::ColumnComment,
[&i.schema_name, &i.table_name, &i.column_name],
);
sqls.push((key, Self::comment_to_sql(i)));
}

for i in self.table_comments.iter() {
let key = format!("table_comment.{}.{}", i.schema_name, i.table_name);
let key =
StructKey::new(StructKeyType::TableComment, [&i.schema_name, &i.table_name]);
sqls.push((key, Self::comment_to_sql(i)));
}
}
Expand All @@ -110,9 +116,9 @@ impl PgCreateTableStatement {
}
}

let key = format!(
"constraint.{}.{}.{}",
i.schema_name, i.table_name, i.constraint_name
let key = StructKey::new(
StructKeyType::Constraint,
[&i.schema_name, &i.table_name, &i.constraint_name],
);
sqls.push((key, Self::constraint_to_sql(i)));
}
Expand All @@ -131,7 +137,10 @@ impl PgCreateTableStatement {
}
}

let key = format!("index.{}.{}.{}", i.schema_name, i.table_name, i.index_name);
let key = StructKey::new(
StructKeyType::Index,
[&i.schema_name, &i.table_name, &i.index_name],
);
sqls.push((key, Self::index_to_sql(i)?));
}

Expand Down
Loading
Loading