From 589f5eef2ee0ba9363627c751918e551f31a89a3 Mon Sep 17 00:00:00 2001 From: leto-bbq Date: Tue, 4 Aug 2026 12:12:52 +0800 Subject: [PATCH] docs: update benchmark tool documentation --- src/.vuepress/sidebar/V2.0.x/zh-Table.ts | 1 + .../Master/Table/Tools-System/Benchmark.md | 1017 +++++++++++----- .../Master/Tree/Tools-System/Benchmark.md | 1026 +++++++++++------ .../latest-Table/Tools-System/Benchmark.md | 1017 +++++++++++----- .../latest/Tools-System/Benchmark.md | 1026 +++++++++++------ .../Master/Table/Tools-System/Benchmark.md | 954 +++++++++++---- .../Master/Tree/Tools-System/Benchmark.md | 968 +++++++++++----- .../latest-Table/Tools-System/Benchmark.md | 954 +++++++++++---- .../latest/Tools-System/Benchmark.md | 968 +++++++++++----- 9 files changed, 5515 insertions(+), 2416 deletions(-) diff --git a/src/.vuepress/sidebar/V2.0.x/zh-Table.ts b/src/.vuepress/sidebar/V2.0.x/zh-Table.ts index 3a0364a21..0388fda1b 100644 --- a/src/.vuepress/sidebar/V2.0.x/zh-Table.ts +++ b/src/.vuepress/sidebar/V2.0.x/zh-Table.ts @@ -155,6 +155,7 @@ export const zhSidebar = { children: [ { text: '命令行工具', link: 'CLI_apache' }, { text: '监控工具', link: 'Monitor-Tool_apache' }, + { text: '测试工具', link: 'Benchmark' }, { text: '集群管理工具', link: 'Maintenance-Tool_apache' }, { text: '数据导入导出', collapsible: true, children: [ diff --git a/src/UserGuide/Master/Table/Tools-System/Benchmark.md b/src/UserGuide/Master/Table/Tools-System/Benchmark.md index ea262493f..3ca07c95e 100644 --- a/src/UserGuide/Master/Table/Tools-System/Benchmark.md +++ b/src/UserGuide/Master/Table/Tools-System/Benchmark.md @@ -18,402 +18,799 @@ under the License. --> + # Benchmark Tool -## 1. **Basic Overview** +IoT Benchmark is a benchmark testing tool for time-series databases and real-time databases in Industrial Internet of Things (IIoT) scenarios. This manual introduces the tool's main capabilities, supported databases, and basic usage. The installation, configuration, and test examples primarily use IoTDB 2\.0\.x and cover both the tree model and the table model. + +![](/img/benchmark-%20English2.png) -IoT-benchmark is a time-series database benchmarking tool developed in Java for big data environments. It was developed and open-sourced by the School of Software, Tsinghua University. The tool is user-friendly, supports various write and query methods, allows storing test information and results for further queries or analysis, and integrates with Tableau for visualizing test results. +## 1\. Basic Overview -Figure 1-1 illustrates the test benchmark process and its extended functionalities, all of which can be streamlined by IoT-benchmark. It supports a variety of workloads, including write-only, read-only, and mixed write-and-read operations. Additionally, it offers software and hardware system monitoring, performance metric measurement, automated database initialization, test data analysis, and system parameter optimization. +IoT Benchmark can generate periodic time-series data based on configuration, perform writes and queries against a database, and collect metrics such as throughput, latency, and success rate. Its main capabilities include: -![](/img/benchmark-English1.png) +- Cross-platform operation: supports Linux, macOS, and Windows. -Figure 1-1 *IoT-benchmark Test Benchmark Process* +- Multiple workloads: supports write-only, query-only, and mixed read/write workloads. -IoT-benchmark adopts the modular design concept of the YCSB test tool, which separates workload generation, performance measurement, and database interface components. Its modular structure is illustrated in Figure 1-2. Unlike YCSB-based testing tools, IoT-benchmark introduces a system monitoring module that supports the persistence of both test data and system metrics. It also includes load-testing functionalities specifically designed for time-series data scenarios, such as batch writes and multiple out-of-order data insertion modes for IoT environments. +- Dataset generation: generated data can be saved to disk for repeated testing. -![](/img/benchmark-%20English2.png) +- Correctness verification: datasets can be loaded from disk to verify write and query correctness. + +- Multiple database support: tests can be performed against various time-series databases and real-time databases. IoTDB 2\.0\.x supports JDBC, REST, and multiple Session write methods. + +- Result persistence: test processes and results can be saved to files, CSV, MySQL, or IoTDB. + +- Test extensions: supports scenarios such as out-of-order writes, batch writes, cluster load testing, dual writes for comparison, and result visualization. + +### 1\.1 Supported Databases, Versions, and Access Methods + +IoT Benchmark supports the following databases and versions. During testing, use `DB_SWITCH` to select the corresponding database, version, and connection or write method. + +|Database|Supported Version|`DB_SWITCH`| +|---|---|---| +|IoTDB|v1\.x|`IoTDB-130-JDBC`, `IoTDB-130-REST`, `IoTDB-130-SESSION_BY_TABLET`, `IoTDB-130-SESSION_BY_RECORD`, `IoTDB-130-SESSION_BY_RECORDS`| +|IoTDB|v2\.x|`IoTDB-200-JDBC`, `IoTDB-200-REST`, `IoTDB-200-SESSION_BY_TABLET`, `IoTDB-200-SESSION_BY_RECORD`, `IoTDB-200-SESSION_BY_RECORDS`| +|InfluxDB|v1\.x|`InfluxDB`| +|InfluxDB|v2\.x|`InfluxDB-2.x`| +|QuestDB|v6\.0\.7|`QuestDB`| +|Microsoft SQL Server|2016 SP2|`MSSQLSERVER`| +|VictoriaMetrics|v1\.64\.0|`VictoriaMetrics`| +|SQLite|—|`SQLite`| +|OpenTSDB|2\.4\.1|`OpenTSDB`| +|KairosDB|—|`KairosDB`| +|TimescaleDB|—|`TimescaleDB`| +|TimescaleDB Cluster|Cluster|`TimescaleDB-Cluster`| +|TDengine|2\.2\.0\.2|`TDengine`| +|TDengine|3\.0\.1|`TDengine-3`| +|DolphinDB|v2\.x|`DolphinDB-2-MTW`, `DolphinDB-2-PTA`| +|DolphinDB|v3\.x|`DolphinDB-3-MTW`, `DolphinDB-3-PTA`| +|CnosDB|—|`CnosDB`| + +Notes: + +- IoTDB access methods include JDBC, REST, Session by Tablet, Session by Record, and Session by Records. + +- For DolphinDB, `MTW` means `MultithreadedTableWriter`, which buffers writes by row; `PTA` means `PartitionedTableAppender`, which appends an entire table in columnar batches. + +- Database versions, drivers, and servers must be compatible with one another. When using other databases, configure the corresponding connection and extension parameters. + +### 1\.2 IoTDB 2\.0\.x Access Methods + +The following installation, configuration, and examples primarily use IoTDB 2\.0\.x. The supported access methods are listed below. -Figure 1-2 *IoT-benchmark Modular Design* +|Access Method|`DB_SWITCH`|Description| +|---|---|---| +|JDBC|`IoTDB-200-JDBC`|Performs writes and queries through JDBC| +|REST|`IoTDB-200-REST`|Performs tests through the IoTDB REST interface| +|Session by Tablet|`IoTDB-200-SESSION_BY_TABLET`|Uses Tablet for batch writes| +|Session by Record|`IoTDB-200-SESSION_BY_RECORD`|Writes records one at a time| +|Session by Records|`IoTDB-200-SESSION_BY_RECORDS`|Writes multiple records in batches| -**Supported Databases** +`IoTDB-200-SESSION_BY_TABLET` is suitable for batch write tests. -Currently, IoT-benchmark supports the following time series databases, versions and connection methods: +## 2\. Installation and Execution -| Database | Version | Connection mmethod | -| :-------------- |:-----------| :------------------------------------------------------- | -| IoTDB | v1.x v2.x | JDBC, SessionByTablet, SessionByRecord, SessionByRecords | -| InfluxDB | v1.x v2.x | SDK | -| TimescaleDB | -- | JDBC | -| OpenTSDB | -- | HTTP Request | -| QuestDB | v6.0.7 | JDBC | -| TDengine | v2.2.0.2 | JDBC | -| VictoriaMetrics | v1.64.0 | HTTP Request | -| KairosDB | -- | HTTP Request | +This chapter uses IoTDB 2\.0\.x as the database under test. +### 2\.1 Prerequisites -## 2. **Installation and Operation** +1. Before using IoT Benchmark, prepare: -### 2.1 **Prerequisites** + - Java 17. -1. Java 8 -2. Maven 3.6+ -3. The corresponding appropriate version of the database, such as Apache IoTDB 2.0 + - Maven. -### 2.2 **How to Obtain** + - An installed and running IoTDB 2\.0\.x instance. -- **B****inary package****:** Visit https://github.com/thulab/iot-benchmark/releases to download the installation package. Extract the compressed file into a desired folder for use. + - Sufficient client CPU, memory, disk, and network resources. -- **Source Code** **Compilation (for** **Apache** **IoTDB 2.0 testing):** +2. Environment notes: - - **Compile the latest IoTDB Session package:** Download the IoTDB source code from https://github.com/apache/iotdb/tree/rc/2.0.5 and run the following command in the root directory to compile the latest IoTDB Session package: + - Linux or macOS is recommended for running tests. - ```Bash - mvn clean package install -pl session -am -DskipTests - ``` + - On Windows, use `benchmark.bat` in the installation package root directory to start a test. - - **Compile the IoT-benchmark test package:** Download the source code from https://github.com/thulab/iot-benchmark and run the following command in the root directory to compile the Apache IoTDB 2.0 test package:. + - On Linux and macOS, use `benchmark.sh` to start a test. - ```Bash - mvn clean package install -pl iotdb-2.0 -am -DskipTests - ``` + - Some system information collection capabilities in CSV recording mode are supported only on Linux. - - The compiled test package will be located at: +> Note: Do not deploy IoT Benchmark and the IoTDB instance under test in environments where they compete for resources. For formal performance testing, use separate servers and stop unrelated services. +> +> - ```Bash - ./iotdb-2.0/target/iotdb-2.0-0.0.1/iotdb-2.0-0.0.1 - ``` +### 2\.2 Obtaining IoT Benchmark -### 2.3 **Test Package Structure** +1. Download a release package -The directory structure of the test package is shown below. The test configuration file is `conf/config.properties`, and the test startup scripts are `benchmark.sh` (Linux & MacOS) and `benchmark.bat` (Windows). The detailed usage of the files is shown in the table below. +Download a release package matching the test target from [IoT Benchmark Releases](https://github.com/thulab/iot-benchmark/releases), and extract it for use. -```Shell --rw-r--r--. 1 root root 2881 Jan 10 01:36 benchmark.bat --rwxr-xr-x. 1 root root 314 Jan 10 01:36 benchmark.sh -drwxr-xr-x. 2 root root 24 Jan 10 01:36 bin --rwxr-xr-x. 1 root root 1140 Jan 10 01:36 cli-benchmark.sh -drwxr-xr-x. 2 root root 107 Jan 10 01:36 conf -drwxr-xr-x. 2 root root 4096 Jan 10 01:38 lib --rw-r--r--. 1 root root 11357 Jan 10 01:36 LICENSE --rwxr-xr-x. 1 root root 939 Jan 10 01:36 rep-benchmark.sh --rw-r--r--. 1 root root 14 Jan 10 01:36 routine +2. Build from source + +Clone the repository: + +```Bash +git clone https://github.com/thulab/iot-benchmark.git +cd iot-benchmark +``` + +Run the following command in the project root directory: + +```Bash +mvn clean package -Dmaven.test.skip=true +``` + +After the build is complete, the IoTDB 2\.0\.x package is located at: + +```Plain Text +iotdb-2.0/target/iot-benchmark-iotdb-2.0/iot-benchmark-iotdb-2.0 +``` + +Enter the installation directory: + +```Bash +cd iotdb-2.0/target/iot-benchmark-iotdb-2.0/iot-benchmark-iotdb-2.0 ``` -| Name | File | Usage | -| :--------------- | :---------------- | :-------------------------------------------------- | -| benchmark.bat | - | Startup script on Windows | -| benchmark.sh | - | Startup script on Linux/Mac | -| bin | startup.sh | Initialization script folder | -| conf | config.properties | Test scenario configuration file | -| lib | - | Dependency library | -| LICENSE | - | License file | -| cli-benchmark.sh | - | One-click startup script | -| routine | - | Automatic execution of multiple test configurations | -| rep-benchmark.sh | - | Automatic execution of multiple test scripts | +### 2\.3 Package Structure + +Common files and directories in the installation package are listed below. + +|Name|Purpose| +|---|---| +|`benchmark.sh`|Startup script for Linux and macOS| +|`benchmark.bat`|Startup script for Windows| +|`conf/config.properties`|Test scenario configuration file| +|`lib/`|Runtime dependencies| +|`logs/`|Test logs, generated after the first run| +|`data/`|Dataset or persisted result directory, generated according to the work mode and persistence configuration| +### 2\.4 Running a Test +1. Start IoTDB -### 2.4 **Execution** **of** **Tests** +First, start the target IoTDB 2\.0\.x instance and verify that the client can access its service port. The default native interface port is `6667`. -1. Modify the configuration file (conf/config.properties) according to test requirements. For example, to test Apache IoTDB 2.0, set the following parameter: +2. Modify the configuration - ```Bash - DB_SWITCH=IoTDB-200-SESSION_BY_TABLET - ``` +- Edit `conf/config.properties`. -2. Ensure the target time-series database is running. +- Minimal connection configuration example: -3. Start IoT-benchmark to execute the test. Monitor the status of both the target database and IoT-benchmark during execution. +```Properties +DB_SWITCH=IoTDB-200-SESSION_BY_TABLET +IoTDB_DIALECT_MODE=tree +HOST=127.0.0.1 +PORT=6667 +USERNAME=root +PASSWORD=root +DB_NAME=test +``` + +- To use the table model, change the setting to: + +```Properties +IoTDB_DIALECT_MODE=table +``` + +- If REST is selected: -4. Upon completion, review the results and analyze the test process. +```Properties +DB_SWITCH=IoTDB-200-REST +REST_PORT=18080 +REST_AUTHORIZATION=Basic cm9vdDpyb290 +``` + +`REST_AUTHORIZATION` configures the `Basic Authentication` information for the REST interface. The example uses the username `root` and password `root`. + +3. Check RPC compression compatibility -### 2.5 **Results Interpretation** +IoT Benchmark 2\.0 enables IoTDB RPC compression by default: -All test log files are stored in the `logs` folder, while test results are saved in the `data/csvOutput` folder. For example, the following result matrix illustrates the test outcome: +```Properties +ENABLE_IOTDB_RPC_COMPRESSION=true +``` -![](/img/bm4.png) +This feature requires IoTDB 2\.0\.6 or later. When testing an IoTDB 2\.0\.x version earlier than 2\.0\.6, set: + +```Properties +ENABLE_IOTDB_RPC_COMPRESSION=false +``` -- **Result Matrix:** - - OkOperation: Number of successful operations. - - OkPoint: Number of successfully written points (for write operations) or successfully queried points (for query operations). - - FailOperation: Number of failed operations. - - FailPoint: Number of failed write points. -- **Latency (ms) Matrix:** - - AVG: Average operation latency. - - MIN: Minimum operation latency. - - Pn: Quantile values of the overall operation distribution (e.g., P25 represents the 25th percentile, or lower quartile). +Thrift compression is a separate configuration. If it is enabled: -## 3. **Main** **Parameters** +```Properties +ENABLE_THRIFT_COMPRESSION=true +``` -### 3.1 IoTDB Service Model +Also set the following in IoTDB's `iotdb-datanode.properties`: -The `IoTDB_DIALECT_MODE` parameter supports two modes: `tree` and `table`. The default value is `tree`. +```Properties +dn_rpc_thrift_compression_enable=true +``` -- **For IoTDB 2.0 and later versions**, the `IoTDB_DIALECT_MODE` parameter must be specified, and only one mode can be set for each IoTDB instance. -- **IoTDB_DIALECT_MODE = table:** - - The number of devices must be an integer multiple of the number of tables. - - The number of tables must be an integer multiple of the number of databases. +4. Start Benchmark -Key Parameters for IoTDB Service Model - -| **Parameter name** | **Type** | **Example** | **System description** | -| :---------------------- | :------- | :---------- | :----------------------------------------------------------- | -| IoTDB_TABLE_NAME_PREFIX | String | `table_` | Prefix for table names when `IoTDB_DIALECT_MODE` is set to `table`. | -| DATA_CLIENT_NUMBER | Integer | `10` | Number of clients, must be an integer multiple of the table count. | -| SENSOR_NUMBER | Integer | `10` | Controls the number of attribute columns in the table model. | -| IoTDB_TABLE_NUMBER | Integer | `1` | Specifies the number of tables when using the table model. | - -### 3.2 **Working** **Mode** - -The `BENCHMARK_WORK_MODE` parameter supports four operational modes: - -1. **General Test Mode (****`testWithDefaultPath`****):** Configured via the `OPERATION_PROPORTION` parameter to support write-only, read-only, and mixed read-write operations. -2. **Data Generation Mode (****`generateDataMode`****):** Generates a reusable dataset, which is saved to `FILE_PATH` for subsequent use in the correctness write and correctness query modes. -3. **Single Database Correctness Write Mode (****`verificationWriteMode`****):** Verifies the correctness of dataset writing by writing the dataset generated in data generation mode. This mode supports only IoTDB v1.0+ and InfluxDB v1.x. -4. **Single Database Correctness Query Mode (****`verificationQueryMode`****):** Verifies the correctness of dataset queries after using the correctness write mode. This mode supports only IoTDB v1.0+ and InfluxDB v1.x. - -Mode configurations are shown in the following below: - -| **Mode name** | **BENCHMARK_WORK_MODE** | Description | Required Configuration | -| :------------------------------------- | :---------------------- | :------------------------------------------------------ | :------------------------- | -| General test mode | testWithDefaultPath | Supports multiple read and write mixed load operations. | `OPERATION_PROPORTION` | -| Generate data mode | generateDataMode | Generates datasets recognizable by IoT-benchmark. | `FILE_PATH` and `DATA_SET` | -| Single database correctness write mode | verificationWriteMode | Writes datasets for correctness verification. | `FILE_PATH` and `DATA_SET` | -| Single database correctness query mode | verificationQueryMode | Queries datasets to verify correctness. | `FILE_PATH` and `DATA_SET` | - -### 3.3 **Server** **Connection** **Information** - -Once the working mode is specified, the following parameters must be configured to inform IoT-benchmark of the target time-series database: - -| **Parameter** | **Type** | **Example** | D**escription** | -| :------------ | :------- | :---------------------------- | :----------------------------------------------------- | -| DB_SWITCH | String | `IoTDB-200-SESSION_BY_TABLET` | Specifies the type of time-series database under test. | -| HOST | String | `127.0.0.1` | Network address of the target time-series database. | -| PORT | Integer | `6667` | Network port of the target time-series database. | -| USERNAME | String | `root` | Login username for the time-series database. | -| PASSWORD | String | `root` | Password for the database login user. | -| DB_NAME | String | `test` | Name of the target time-series database. | -| TOKEN | String | - | Authentication token (used for InfluxDB 2.0). | - -### 3.4 **Write Scenario Parameters** - -| **Parameter** | **Type** | **Example** | D**escription** | -| :------------------------- | :-------------------- | :-------------------------- | :----------------------------------------------------------- | -| CLIENT_NUMBER | Integer | `100` | Total number of clients used for writing. | -| GROUP_NUMBER | Integer | `20` | Number of databases (only applicable for IoTDB). | -| DEVICE_NUMBER | Integer | `100` | Total number of devices. | -| SENSOR_NUMBER | Integer | `300` | Total number of sensors per device. (Control the number of attribute columns if you use the IoTDB table model) | -| INSERT_DATATYPE_PROPORTION | String | `1:1:1:1:1:1:0:0:0:0` | Ratio of data types: `BOOLEAN:INT32:INT64:FLOAT:DOUBLE:TEXT:STRING:BLOB:TIMESTAMP:DATE`. | -| POINT_STEP | Integer | `1000` | Time interval (in ms) between generated data points. | -| OP_MIN_INTERVAL | Integer | `0` | Minimum execution interval for operations (ms): if the operation takes more than the value, the next one will be executed immediately, otherwise wait (OP_MIN_INTERVAL - actual execution time) ms; if it is 0, the parameter is not effective; if it is -1, its value is consistent with POINT_STEP | -| IS_OUT_OF_ORDER | Boolean | `false` | Specifies whether to write data out of order. | -| OUT_OF_ORDER_RATIO | Floating point number | `0.3` | Proportion of out-of-order data. | -| BATCH_SIZE_PER_WRITE | Integer | `1` | Number of data rows written per batch. | -| START_TIME | Time | `2022-10-30T00:00:00+08:00` | Start timestamp for data generation. | -| LOOP | Integer | `86400` | Total number of write operations: Each type of operation will be divided according to the proportion defined by `OPERATION_PROPORTION` | -| OPERATION_PROPORTION | Character | `1:0:0:0:0:0:0:0:0:0:0` | Ratio of operation types (write:Q1:Q2:...:Q10). | - -### 3.5 **Query Scenario Parameters** - -| Parameter | Type | Example | Description | -| :------------------- | :-------- | :---------------------- | :----------------------------------------------------------- | -| QUERY_DEVICE_NUM | Integer | `2` | Number of devices involved in each query statement. | -| QUERY_SENSOR_NUM | Integer | `2` | Number of sensors involved in each query statement. | -| QUERY_AGGREGATE_FUN | Character | `count` | Aggregate functions used in queries (`COUNT`, `AVG`, `SUM`, etc.). | -| STEP_SIZE | Integer | `1` | Time interval step for time filter conditions. | -| QUERY_INTERVAL | Integer | `250000` | Time interval between query start and end times. | -| QUERY_LOWER_VALUE | Integer | `-5` | Threshold for conditional queries (`WHERE value > QUERY_LOWER_VALUE`). | -| GROUP_BY_TIME_UNIT | Integer | `20000` | The size of the group in the `GROUP BY` statement | -| LOOP | Integer | `10` | Total number of query operations: Each type of operation will be divided according to the proportion defined by `OPERATION_PROPORTION` | -| OPERATION_PROPORTION | Character | `0:0:0:0:0:0:0:0:0:0:1` | Ratio of operation types (`write:Q1:Q2:...:Q10`). | - -### 3.6 **Query Types and Example SQL** - -| Number | Query Type | IoTDB Sample SQL | -| :----- | :----------------------------- | :----------------------------------------------------------- | -| Q1 | Precise Point Query | `select v1 from root.db.d1 where time = ?` | -| Q2 | Time Range Query | `select v1 from root.db.d1 where time > ? and time < ?` | -| Q3 | Time Range with Value Filter | `select v1 from root.db.d1 where time > ? and time < ? and v1 > ?` | -| Q4 | Time Range Aggregation Query | `select count(v1) from root.db.d1 where and time > ? and time < ?` | -| Q5 | Full-Time Range with Filtering | `select count(v1) from root.db.d1 where v1 > ?` | -| Q6 | Range Aggregation with Filter | `select count(v1) from root.db.d1 where v1 > ? and time > ? and time < ?` | -| Q7 | Time Grouping Aggregation | `select count(v1) from root.db.d1 group by ([?, ?), ?, ?)` | -| Q8 | Latest Point Query | `select last v1 from root.db.d1` | -| Q9 | Descending Range Query | `select v1 from root.sg.d1 where time > ? and time < ? order by time desc` | -| Q10 | Descending Range with Filter | `select v1 from root.sg.d1 where time > ? and time < ? and v1 > ? order by time desc` | - -### 3.7 **Test process and test result persistence** - -IoT-benchmark currently supports persisting the test process and test results through configuration parameters. - -| **Parameter** | **Type** | **Example** | D**escription** | -| :-------------------- | :------- | :---------- | :----------------------------------------------------------- | -| TEST_DATA_PERSISTENCE | String | `None` | Specifies the result persistence method. Options: `None`, `IoTDB`, `MySQL`, `CSV`. | -| RECORD_SPLIT | Boolean | `true` | Whether to split results into multiple records. (Not supported by IoTDB currently.) | -| RECORD_SPLIT_MAX_LINE | Integer | `10000000` | Maximum number of rows per record (10 million rows per database table or CSV file). | -| TEST_DATA_STORE_IP | String | `127.0.0.1` | IP address of the database for result storage. | -| TEST_DATA_STORE_PORT | Integer | `6667` | Port number of the output database. | -| TEST_DATA_STORE_DB | String | `result` | Name of the output database. | -| TEST_DATA_STORE_USER | String | `root` | Username for accessing the output database. | -| TEST_DATA_STORE_PW | String | `root` | Password for accessing the output database. | - -**Result Persistence Details** - -- **CSV Mode:** If `TEST_DATA_PERSISTENCE` is set to `CSV`, a `data` folder is generated in the IoT-benchmark root directory during and after test execution. This folder contains: - - `csv` folder: Records the test process. - - `csvOutput` folder: Stores the test results. -- **MySQL Mode:** If `TEST_DATA_PERSISTENCE` is set to `MySQL`, IoT-benchmark creates the following tables in the specified MySQL database: - - **Test Process Table:** - 1. Created before the test starts. - 2. Named as: `testWithDefaultPath___`. - - **Configuration Table:** - 1. Named `CONFIG`. - 2. Stores the test configuration. - 3. Created if it does not exist. - - **Final Result Table:** - 1. Named `FINAL_RESULT`. - 2. Stores the test results after test completion. - 3. Created if it does not exist. - -### 3.8 Automation Script - -#### One-Click Script Startup - -The `cli-benchmark.sh` script allows one-click startup of IoTDB, IoTDB Benchmark monitoring, and IoTDB Benchmark testing. However, please note that this script will clear all existing data in IoTDB during startup, so use it with caution. - -**Steps to Run:** - -1. Edit the `IOTDB_HOME` parameter in `cli-benchmark.sh` to the local IoTDB directory. -2. Start the test by running the following command: +On Linux or macOS: ```Bash -> ./cli-benchmark.sh +./benchmark.sh ``` -1. After the test completes: - 1. Check test-related logs in the `logs` folder. - 2. Check monitoring-related logs in the `server-logs` folder. +On Windows: + +```Plain Text +benchmark.bat +``` -#### Automatic Execution of Multiple Tests +During the test, progress is periodically printed to the terminal. When the test completes, the main configurations, execution time, result matrix, and latency matrix are displayed. -Single tests are often insufficient without comparative results. Therefore, IoT-benchmark provides an interface for executing multiple tests in sequence. +### 2\.5 Understanding the Results -1. **Routine Configuration:** Each line in the `routine` file specifies the parameters that change for each test. For example: +Test execution information is written to the `logs` folder in the installation directory. Whether CSV files are generated or results are written to a result database depends on parameters such as `TEST_DATA_PERSISTENCE`. - ```Plain - LOOP=10 DEVICE_NUMBER=100 TEST - LOOP=20 DEVICE_NUMBER=50 TEST - LOOP=50 DEVICE_NUMBER=20 TEST - ``` +1. Result matrix -In this example, three tests will run sequentially with `LOOP` values of 10, 20, and 50. +The result matrix reports the following metrics by operation type: -Then the test process with 3 LOOP parameters of 10, 20, and 50 is executed in sequence. +|Metric|Description| +|---|---| +|`okOperation`|Number of successfully executed requests or SQL statements| +|`okPoint`|Number of successfully written data points, or data points successfully returned by queries| +|`failOperation`|Number of failed requests or SQL statements| +|`failPoint`|Number of data points that failed to be written; usually 0 for query operations| +|`throughput`|Throughput, usually equal to `okPoint / Test elapsed time`| -**Important Notes:** +The main operation names in the output include: -- Multiple parameters can be changed in each test using the format: +- `INGESTION` - ```Bash - LOOP=20 DEVICE_NUMBER=10 TEST - ``` +- `PRECISE_POINT` -- Avoid unnecessary spaces. +- `TIME_RANGE` -- The `TEST` keyword marks the start of a new test. +- `VALUE_RANGE` -- Changed parameters persist across subsequent tests unless explicitly reset. +- `AGG_RANGE` -2. **Start the Test:** After configuring the `routine` file, start multi-test execution using the following command +- `AGG_VALUE` - ```Bash - > ./rep-benchmark.sh - ``` +- `AGG_RANGE_VALUE` -Test results will be displayed in the terminal. +- `GROUP_BY` -**Important Notes:** +- `LATEST_POINT` -- Closing the terminal or losing the client connection will terminate the test process. +- `RANGE_QUERY_DESC` -- To run the test as a background daemon, execute: +- `VALUE_RANGE_QUERY_DESC` - ```Bash - > ./rep-benchmark.sh > /dev/null 2>&1 & - ``` +- `GROUP_BY_DESC` -- To monitor progress, check the logs: +- `SET_OP_QUERY` - ```Bash - > cd ./logs - > tail -f log_info.log - ``` +2. Latency matrix -## 4. Test Example +The latency matrix is measured in milliseconds. Common fields are listed below. -This example demonstrates how to configure and run an IoT-benchmark test with IoTDB 2.0 using the table model for writing and querying. +|Metric|Description| +|---|---| +|`AVG`|Average latency| +|`MIN`|Minimum latency| +|`P10`, `P25`, `MEDIAN`|Lower percentiles and median latency| +|`P75`, `P90`, `P95`|Higher-percentile latency| +|`P99`, `P999`|Tail latency| +|`MAX`|Maximum latency| +|`SLOWEST_THREAD`|Largest cumulative operation time among client threads| -```Properties +The test results also report metadata creation time and `Test elapsed time`, which excludes metadata creation. When comparing tests, ensure that the hardware, data volume, number of clients, compression configuration, and cache state are consistent across runs. + +3. Output example + +After the test completes, the terminal displays the main configurations, execution time, result matrix, and latency matrix. The following is truncated output from a write-only test: + +```Plain Text ----------------------Main Configurations---------------------- BENCHMARK_WORK_MODE=testWithDefaultPath -IoTDB_DIALECT_MODE=TABLE DB_SWITCH=IoTDB-200-SESSION_BY_TABLET +HOST=[127.0.0.1] + +GROUP_NUMBER=10 +DEVICE_NUMBER=50 +SENSOR_NUMBER=500 +SCHEMA_CLIENT_NUMBER=20 +DATA_CLIENT_NUMBER=20 + +OPERATION_PROPORTION=1:0:0:0:0:0:0:0:0:0:0:0:0 +LOOP=10000 +BATCH_SIZE_PER_WRITE=100 +--------------------------------------------------------------- + +Create schema cost 0.30 second +Test elapsed time (not include schema creation): 1238.79 second + +----------------------------------------------------------Result Matrix---------------------------------------------------------- +Operation okOperation okPoint failOperation failPoint throughput(point/s) +INGESTION 500000 25000000000 0 0 20180954.09 +--------------------------------------------------------------------------------------------------------------------------------- + +----------------------------------------------------------Latency (ms) Matrix---------------------------------------------------- +Operation AVG MIN P10 P25 MEDIAN P75 P90 P95 P99 P999 MAX SLOWEST_THREAD +INGESTION 37.78 1.67 2.02 2.29 2.86 4.14 5.62 7.43 759.69 5799.89 8309.40 1227561.44 +--------------------------------------------------------------------------------------------------------------------------------- +``` + +The output shows that: + +- Metadata creation took `0.30` seconds, and the actual test took `1238.79` seconds. + +- A total of `500000` write operations successfully wrote `25000000000` data points. + +- Both `failOperation` and `failPoint` are `0`, indicating that no write failures were recorded during this test. + +- Write throughput was `20180954.09` points per second. + +- Average latency was `37.78` ms, P95 latency was `7.43` ms, P99 latency was `759.69` ms, and maximum latency was `8309.40` ms. + +- `SLOWEST_THREAD` is the cumulative operation time of the slowest client thread, not the latency of a single request. + +The values in this example are only intended to demonstrate the output format. Actual results depend on hardware resources, network conditions, IoTDB configuration, data scale, and test parameters. + +## 3\. Main Parameters + +### 3\.1 IoTDB Data Model + +IoTDB 2\.0\.x supports the tree model and the table model. Select the model using: + +```Properties +IoTDB_DIALECT_MODE=tree +``` + +or: + +```Properties +IoTDB_DIALECT_MODE=table +``` + +The following constraints apply: + +- An IoTDB instance can use only one SQL dialect in a single test. + +- The tree model requires `DEVICE_NUMBER >= GROUP_NUMBER`. + +- In the table model, the number of devices must be a multiple of the number of tables, and the number of tables must be a multiple of the number of databases. + +- In the table model, the number of data clients must be a multiple of the number of tables. + +Common model parameters: + +|Parameter|Example|Description| +|---|---|---| +|`IoTDB_DIALECT_MODE`|`tree`|`tree` or `table`| +|`GROUP_NUMBER`|`1`|Number of databases; corresponds to the number of databases in the tree model| +|`IoTDB_TABLE_NUMBER`|`1`|Number of tables created in the table model| +|`IoTDB_TABLE_NAME_PREFIX`|`table_`|Table name prefix| +|`TABLE_TIME_COLUMN`|`time`|Name of the time column in the table model| +|`IoTDB_TABLE_WRITABLE_VIEW`|`false`|Whether to create and use writable views| + +### 3\.2 Work Modes + +Use `BENCHMARK_WORK_MODE` to select a work mode. + +|Mode|Configuration Value|Description| +|---|---|---| +|Regular test mode|`testWithDefaultPath`|Runs a write, query, or mixed workload| +|Data generation mode|`generateDataMode`|Saves the dataset generated by Benchmark to `FILE_PATH`| +|Correctness write mode|`verificationWriteMode`|Loads a dataset from `FILE_PATH` and writes it to the database| +|Correctness query mode|`verificationQueryMode`|Loads a dataset and compares it with database query results| + +Example: + +```Properties +BENCHMARK_WORK_MODE=testWithDefaultPath +``` + +Before using the correctness write and query modes, use `generateDataMode` to generate a reusable dataset. + +### 3\.3 Server Connection Information + +|Parameter|Example|Description| +|---|---|---| +|`DB_SWITCH`|`IoTDB-200-SESSION_BY_TABLET`|Database version and connection method| +|`HOST`|`127.0.0.1`|IoTDB address; separate multiple addresses with commas| +|`PORT`|`6667`|Native service port; the number of ports must match the number of `HOST` entries| +|`USERNAME`|`root`|Username| +|`PASSWORD`|`root`|Password| +|`DB_NAME`|`test`|Name of the database used for testing| +|`REST_PORT`|`18080`|REST service port| +|`REST_AUTHORIZATION`|`Basic cm9vdDpyb290`|REST authorization header| +|`ENABLE_AUTO_FETCH`|`false`|Whether Session automatically refreshes the DataNode list| + +Data cleanup parameters: + +```Properties +IS_DELETE_DATA=false +INIT_WAIT_TIME=1000 +``` + +> Warning: `IS_DELETE_DATA=true` clears test data from the target database before the test starts. Enable it only in a dedicated test environment, and verify `HOST`, `PORT`, `DB_NAME`, and account permissions before execution. +> +> + +### 3\.4 Write Scenarios + +1. Data scale and clients + +|Parameter|Example|Description| +|---|---|---| +|`DEVICE_NUMBER`|`100`|Total number of devices| +|`SENSOR_NUMBER`|`10`|Number of measurements per device; number of measurement columns in the table model| +|`GROUP_NUMBER`|`1`|Number of IoTDB databases| +|`SCHEMA_CLIENT_NUMBER`|`5`|Number of clients that register metadata| +|`DATA_CLIENT_NUMBER`|`10`|Number of clients that perform data reads and writes| +|`IS_CLIENT_BIND`|`true`|Whether devices are bound to clients| +|`REAL_INSERT_RATE`|`1.0`|Proportion of devices that actually participate in writes| +|`IS_SENSOR_TS_ALIGNMENT`|`true`|Whether measurement timestamps under the same device are aligned| + +2. Batch writes + +|Parameter|Example|Description| +|---|---|---| +|`BATCH_SIZE_PER_WRITE`|`100`|Number of data rows written per device in each batch| +|`DEVICE_NUM_PER_WRITE`|`1`|Number of devices involved in each batch write| +|`CREATE_SCHEMA`|`true`|Whether to create metadata before writing| +|`START_TIME`|`2022-01-01T00:00:00+08:00`|Start time for generated data| + +The number of data points in a single batch is: + +```Plain Text +DEVICE_NUM_PER_WRITE × SENSOR_NUMBER × BATCH_SIZE_PER_WRITE +``` + +`DEVICE_NUM_PER_WRITE` must evenly divide the number of devices assigned to a single data client. In the table model, the divisibility constraints among the number of devices, number of tables, and devices per batch must also be satisfied. + +3. Write pacing + +|Parameter|Example|Description| +|---|---|---| +|`POINT_STEP`|`5000`|Fixed interval between adjacent generated timestamps| +|`OP_MIN_INTERVAL`|`0`|Minimum interval for each loop, in ms| +|`OP_MIN_INTERVAL_RANDOM`|`false`|Whether to randomly select an interval from `[0, OP_MIN_INTERVAL)`| +|`INTERVAL_BETWEEN_WRITE_BATCH`|`0`|Minimum interval between adjacent batches in the same loop, in ms| +|`TIMESTAMP_PRECISION`|`ms`|Timestamp precision| + +Special values of `OP_MIN_INTERVAL`: + +- `0`: does not limit the loop interval. + +- `-1`: uses `POINT_STEP` as the minimum interval. + +- Positive integer: if the current loop takes less than this value, waits for the remaining time. + +4. Out-of-order writes + +```Properties +IS_OUT_OF_ORDER=false +OUT_OF_ORDER_MODE=POISSON +OUT_OF_ORDER_RATIO=0.5 +IS_REGULAR_FREQUENCY=true +``` + +Supported out-of-order modes include: + +- `POISSON`: generates out-of-order timestamps according to a Poisson distribution. + +- `BATCH`: generates out-of-order data in batches. + +5. Data types + +```Properties +INSERT_DATATYPE_PROPORTION=1:1:1:1:1:1:0:0:0:0:0 +``` + +The order of the entries is: + +```Plain Text +BOOLEAN:INT32:INT64:FLOAT:DOUBLE:TEXT:STRING:BLOB:TIMESTAMP:DATE:OBJECT +``` + +Each value represents the proportion of the corresponding data type. + +### 3\.5 Query Scenarios + +|Parameter|Example|Description| +|---|---|---| +|`QUERY_DEVICE_NUM`|`1`|Number of devices involved in each query| +|`QUERY_SENSOR_NUM`|`1`|Number of measurements involved in each query| +|`QUERY_AGGREGATE_FUN`|`count`|Aggregation function| +|`STEP_SIZE`|`0`|Step by which the query start time changes, in units of `POINT_STEP`| +|`QUERY_INTERVAL`|`250000`|Interval between query start and end times| +|`QUERY_LOWER_VALUE`|`-5`|Lower bound of the value filter| +|`GROUP_BY_TIME_UNIT`|`20000`|Group By window size| +|`QUERY_SET_OP_TYPE`|`union`|Set operation type| +|`QUERY_SET_OP_NUM`|`2`|Number of sub-sets in a set query; at least 2| +|`IS_RECENT_QUERY`|`false`|Whether to prioritize recently written data in mixed scenarios| +|`ENABLE_FIXED_QUERY`|`false`|Whether all query threads use the same device and measurement combinations| +|`RESULT_ROW_LIMIT`|`-1`|Query result row limit; `-1` means no limit| +|`ALIGN_BY_DEVICE`|`false`|Whether to use Align By Device| + +### 3\.6 Operation Proportions + +`OPERATION_PROPORTION` defines the proportions of writes and different query types. It contains 13 entries: + +```Plain Text +Write:Q1:Q2:Q3:Q4:Q5:Q6:Q7:Q8:Q9:Q10:Q11:Q12 +``` + +For example, write-only: + +```Properties +OPERATION_PROPORTION=1:0:0:0:0:0:0:0:0:0:0:0:0 +``` + +Precise point query only: + +```Properties +OPERATION_PROPORTION=0:1:0:0:0:0:0:0:0:0:0:0:0 +``` + +The operation types are listed below. + +|Number|Operation Type|Description| +|---|---|---| +|Write|Data write|Generates and writes data according to the current write configuration| +|Q1|Precise point query|Queries specified measurements by timestamp and device| +|Q2|Time range query|Range query restricted only by start and end times| +|Q3|Range query with value filter|Includes both time and value filter conditions| +|Q4|Aggregation query with time filter|Performs aggregation within a time range| +|Q5|Aggregation query with value filter|Filters by value and aggregates over the full time range| +|Q6|Aggregation query with time and value filters|Includes both time and value filter conditions| +|Q7|Time-grouped aggregation query|Group By query| +|Q8|Latest point query|Queries the latest data point of a device| +|Q9|Descending time range query|Returns range query results in descending time order| +|Q10|Descending range query with value filter|Filters by value and returns results in descending time order| +|Q11|Descending time-grouped aggregation query|Descending Group By query| +|Q12|Set operation query|Set operations such as `union`, `intersect`, or `except`| + +Q12 is supported only by the IoTDB 2\.0 table model. Each subquery in a set operation is a range query. + +### 3\.7 Test Process and Result Persistence + +```Properties +TEST_DATA_PERSISTENCE=None +``` + +Supported values include: + +- `None`: does not write the test process to an external persistence medium. + +- `CSV`: writes to CSV files. + +- `MySQL`: writes to MySQL. + +- `IoTDB`: writes to a specified IoTDB instance. + +Common parameters: + +|Parameter|Example|Description| +|---|---|---| +|`TEST_DATA_PERSISTENCE`|`None`|Persistence method| +|`RECORD_SPLIT`|`true`|Whether to split results into multiple records| +|`RECORD_SPLIT_MAX_LINE`|`10000000`|Maximum number of records in a single table or file| +|`TEST_DATA_STORE_IP`|`127.0.0.1`|Result database address| +|`TEST_DATA_STORE_PORT`|`6667`|Result database port| +|`TEST_DATA_STORE_DB`|`result`|Result database name| +|`TEST_DATA_STORE_USER`|`root`|Result database username| +|`TEST_DATA_STORE_PW`|`root`|Result database password| +|`REMARK`|`write_test`|Test note used to distinguish different tests| +|`CSV_OUTPUT`|`true`|Whether to write final results to CSV| + +When the persistence method is CSV, records are generated in the `data` directory after execution; test results are usually located in `data/csvOutput`. Test logs are always written to `logs`, regardless of whether persistence is enabled. + +The following parameters control log output frequency: + +```Properties +IS_QUIET_MODE=true +LOG_PRINT_INTERVAL=5 +RESULT_PRINT_INTERVAL=3600 +``` + +### 3\.8 Automation and Cluster Testing + +1. Limit test duration + +```Properties +TEST_MAX_TIME=3600000 +``` + +The unit is milliseconds. A value of `0` means no limit. This parameter does not include the time spent pre-registering metadata. + +2. Multi-Benchmark cluster load testing + +Use the same overall data scale configuration on multiple client machines, and set: + +```Properties +BENCHMARK_CLUSTER=true +BENCHMARK_INDEX=0 +``` + +Each Benchmark instance must use a different `BENCHMARK_INDEX`, such as `0`, `1`, and `2` in sequence. All clients should use consistent database connections, data scales, operation proportions, and other configurations. + +3. Dual-write testing + +IoT Benchmark can write the same data to two different databases for comparison: + +```Properties +IS_DOUBLE_WRITE=true +ANOTHER_DB_SWITCH= +ANOTHER_HOST=127.0.0.1 +ANOTHER_PORT=6667 +ANOTHER_USERNAME=root +ANOTHER_PASSWORD=root +ANOTHER_DB_NAME=test +``` + +Dual-write mode does not support comparisons between different versions of the same database, or direct comparisons between the IoTDB tree model and table model. + +## 4\. Examples + +This section uses small datasets to demonstrate the basic process. For formal performance testing, increase the number of devices, measurements, clients, and loops according to the target business model, and perform multiple warm-up and repeated test runs. + +### 4\.1 Write Test Example + +Test objective: use 10 data clients to simulate 100 devices, each containing 10 measurements, and perform a write-only test. + +Example configuration: + +```Properties +# Database connection +DB_SWITCH=IoTDB-200-SESSION_BY_TABLET +IoTDB_DIALECT_MODE=tree +HOST=127.0.0.1 +PORT=6667 +USERNAME=root +PASSWORD=root +DB_NAME=test + +# Safety setting: do not automatically delete existing data by default +IS_DELETE_DATA=false + +# Work mode +BENCHMARK_WORK_MODE=testWithDefaultPath +OPERATION_PROPORTION=1:0:0:0:0:0:0:0:0:0:0:0:0 + +# Data scale GROUP_NUMBER=1 -IoTDB_TABLE_NUMBER=1 -DEVICE_NUMBER=60 -REAL_INSERT_RATE=1.0 +DEVICE_NUMBER=100 SENSOR_NUMBER=10 -OPERATION_PROPORTION=1:0:0:0:0:0:0:0:0:0:0:0 -SCHEMA_CLIENT_NUMBER=10 +SCHEMA_CLIENT_NUMBER=5 DATA_CLIENT_NUMBER=10 -LOOP=10 +IS_SENSOR_TS_ALIGNMENT=true + +# Write configuration +CREATE_SCHEMA=true BATCH_SIZE_PER_WRITE=10 DEVICE_NUM_PER_WRITE=1 -START_TIME=2025-01-01T00:00:00+08:00 +LOOP=100 POINT_STEP=1000 -INSERT_DATATYPE_PROPORTION=1:1:1:1:1:1:0:0:0:0 -VECTOR=true +OP_MIN_INTERVAL=0 +START_TIME=2026-01-01T00:00:00+08:00 +INSERT_DATATYPE_PROPORTION=1:1:1:1:1:1:0:0:0:0:0 + +# Can be enabled for IoTDB 2.0.6 and later +ENABLE_IOTDB_RPC_COMPRESSION=true + +# Output +TEST_DATA_PERSISTENCE=None +CSV_OUTPUT=true +REMARK=iotdb_2_write_test +``` + +Start the test: + +```Bash +./benchmark.sh ``` -**Execution Steps:** +After completion, focus on: -1. Ensure the target database (IoTDB 2.0) is running. -2. Start IoT-benchmark using the configured parameters. -3. Upon completion, view the test results. +- `okPoint` and `failPoint` for `INGESTION`. -```Shell -Create schema cost 0.88 second -Test elapsed time (not include schema creation): 4.60 second -----------------------------------------------------------Result Matrix---------------------------------------------------------- -Operation okOperation okPoint failOperation failPoint throughput(point/s) -INGESTION 600 60000 0 0 13054.42 -PRECISE_POINT 0 0 0 0 0.00 -TIME_RANGE 0 0 0 0 0.00 -VALUE_RANGE 0 0 0 0 0.00 -AGG_RANGE 0 0 0 0 0.00 -AGG_VALUE 0 0 0 0 0.00 -AGG_RANGE_VALUE 0 0 0 0 0.00 -GROUP_BY 0 0 0 0 0.00 -LATEST_POINT 0 0 0 0 0.00 -RANGE_QUERY_DESC 0 0 0 0 0.00 -VALUE_RANGE_QUERY_DESC 0 0 0 0 0.00 -GROUP_BY_DESC 0 0 0 0 0.00 ---------------------------------------------------------------------------------------------------------------------------------- +- `throughput`. + +- `AVG`, `P95`, `P99`, and `MAX` latency. + +- Whether connection timeouts, write failures, or server exceptions are present in the logs. + +### 4\.2 Query Test Example + +Before running a query test, ensure that the target database contains data matching the query configuration. It is recommended to reuse data generated by the write test and disable automatic data deletion and metadata creation. + +The following example runs multiple query types: + +```Properties +DB_SWITCH=IoTDB-200-SESSION_BY_TABLET +IoTDB_DIALECT_MODE=tree +HOST=127.0.0.1 +PORT=6667 +USERNAME=root +PASSWORD=root +DB_NAME=test + +IS_DELETE_DATA=false +CREATE_SCHEMA=false +BENCHMARK_WORK_MODE=testWithDefaultPath + +GROUP_NUMBER=1 +DEVICE_NUMBER=100 +SENSOR_NUMBER=10 +SCHEMA_CLIENT_NUMBER=1 +DATA_CLIENT_NUMBER=10 + +# Do not perform writes; Q1-Q11 have equal proportions; the tree model does not use Q12 +OPERATION_PROPORTION=0:1:1:1:1:1:1:1:1:1:1:1:0 +LOOP=100 + +QUERY_DEVICE_NUM=2 +QUERY_SENSOR_NUM=2 +QUERY_AGGREGATE_FUN=count +STEP_SIZE=1 +QUERY_INTERVAL=250000 +QUERY_LOWER_VALUE=-5 +GROUP_BY_TIME_UNIT=20000 +``` + +To test set queries in the table model, switch the dialect to `table` and assign a proportion to Q12: + +```Properties +IoTDB_DIALECT_MODE=table +OPERATION_PROPORTION=0:0:0:0:0:0:0:0:0:0:0:0:1 +QUERY_SET_OP_TYPE=union +QUERY_SET_OP_NUM=2 +``` + +### 4\.3 Other Configuration Examples + +1. Simulate an actual write rate + +Set the minimum interval of each loop to the data timestamp interval: + +```Properties +POINT_STEP=1000 +OP_MIN_INTERVAL=-1 +``` + +To distribute write requests evenly within a loop, use: + +```Properties +INTERVAL_BETWEEN_WRITE_BATCH=100 +``` + +2. Specify test duration + +Test for one hour: + +```Properties +TEST_MAX_TIME=3600000 +``` + +Ensure that `LOOP` is sufficiently large; otherwise, the test may end when the loop count is exhausted. + +3. Control generated data patterns + +```Properties +LINE_RATIO=1 +SIN_RATIO=1 +SQUARE_RATIO=1 +RANDOM_RATIO=1 +CONSTANT_RATIO=1 +DATA_SEED=666 +STRING_LENGTH=10 +DOUBLE_LENGTH=2 +``` + +Fixing `DATA_SEED` helps generate reproducible data across multiple test runs. + +## 5\. References ---------------------------------------------------------------------------Latency (ms) Matrix-------------------------------------------------------------------------- -Operation AVG MIN P10 P25 MEDIAN P75 P90 P95 P99 P999 MAX SLOWEST_THREAD -INGESTION 41.77 0.95 1.41 2.27 6.76 24.14 63.42 127.18 1260.92 1265.72 1265.49 2581.91 -PRECISE_POINT 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 -TIME_RANGE 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 -VALUE_RANGE 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 -AGG_RANGE 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 -AGG_VALUE 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 -AGG_RANGE_VALUE 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 -GROUP_BY 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 -LATEST_POINT 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 -RANGE_QUERY_DESC 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 -VALUE_RANGE_QUERY_DESC 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 -GROUP_BY_DESC 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ -``` \ No newline at end of file +- [IoT Benchmark Documentation](https://github.com/thulab/iot-benchmark/tree/master/docs) diff --git a/src/UserGuide/Master/Tree/Tools-System/Benchmark.md b/src/UserGuide/Master/Tree/Tools-System/Benchmark.md index 4802594ed..3ca07c95e 100644 --- a/src/UserGuide/Master/Tree/Tools-System/Benchmark.md +++ b/src/UserGuide/Master/Tree/Tools-System/Benchmark.md @@ -21,470 +21,796 @@ # Benchmark Tool -## 1. **Basic Overview** +IoT Benchmark is a benchmark testing tool for time-series databases and real-time databases in Industrial Internet of Things (IIoT) scenarios. This manual introduces the tool's main capabilities, supported databases, and basic usage. The installation, configuration, and test examples primarily use IoTDB 2\.0\.x and cover both the tree model and the table model. -IoT-benchmark is a time-series database benchmarking tool developed in Java for big data environments. It was developed and open-sourced by the School of Software, Tsinghua University. The tool is user-friendly, supports various write and query methods, allows storing test information and results for further queries or analysis, and integrates with Tableau for visualizing test results. +![](/img/benchmark-%20English2.png) -Figure 1-1 illustrates the test benchmark process and its extended functionalities, all of which can be streamlined by IoT-benchmark. It supports a variety of workloads, including write-only, read-only, and mixed write-and-read operations. Additionally, it offers software and hardware system monitoring, performance metric measurement, automated database initialization, test data analysis, and system parameter optimization. +## 1\. Basic Overview -![](/img/benchmark-English1.png) +IoT Benchmark can generate periodic time-series data based on configuration, perform writes and queries against a database, and collect metrics such as throughput, latency, and success rate. Its main capabilities include: -Figure 1-1 *IoT-benchmark Test Benchmark Process* +- Cross-platform operation: supports Linux, macOS, and Windows. -IoT-benchmark adopts the modular design concept of the YCSB test tool, which separates workload generation, performance measurement, and database interface components. Its modular structure is illustrated in Figure 1-2. Unlike YCSB-based testing tools, IoT-benchmark introduces a system monitoring module that supports the persistence of both test data and system metrics. It also includes load-testing functionalities specifically designed for time-series data scenarios, such as batch writes and multiple out-of-order data insertion modes for IoT environments. +- Multiple workloads: supports write-only, query-only, and mixed read/write workloads. -![](/img/benchmark-%20English2.png) +- Dataset generation: generated data can be saved to disk for repeated testing. + +- Correctness verification: datasets can be loaded from disk to verify write and query correctness. + +- Multiple database support: tests can be performed against various time-series databases and real-time databases. IoTDB 2\.0\.x supports JDBC, REST, and multiple Session write methods. + +- Result persistence: test processes and results can be saved to files, CSV, MySQL, or IoTDB. + +- Test extensions: supports scenarios such as out-of-order writes, batch writes, cluster load testing, dual writes for comparison, and result visualization. + +### 1\.1 Supported Databases, Versions, and Access Methods + +IoT Benchmark supports the following databases and versions. During testing, use `DB_SWITCH` to select the corresponding database, version, and connection or write method. + +|Database|Supported Version|`DB_SWITCH`| +|---|---|---| +|IoTDB|v1\.x|`IoTDB-130-JDBC`, `IoTDB-130-REST`, `IoTDB-130-SESSION_BY_TABLET`, `IoTDB-130-SESSION_BY_RECORD`, `IoTDB-130-SESSION_BY_RECORDS`| +|IoTDB|v2\.x|`IoTDB-200-JDBC`, `IoTDB-200-REST`, `IoTDB-200-SESSION_BY_TABLET`, `IoTDB-200-SESSION_BY_RECORD`, `IoTDB-200-SESSION_BY_RECORDS`| +|InfluxDB|v1\.x|`InfluxDB`| +|InfluxDB|v2\.x|`InfluxDB-2.x`| +|QuestDB|v6\.0\.7|`QuestDB`| +|Microsoft SQL Server|2016 SP2|`MSSQLSERVER`| +|VictoriaMetrics|v1\.64\.0|`VictoriaMetrics`| +|SQLite|—|`SQLite`| +|OpenTSDB|2\.4\.1|`OpenTSDB`| +|KairosDB|—|`KairosDB`| +|TimescaleDB|—|`TimescaleDB`| +|TimescaleDB Cluster|Cluster|`TimescaleDB-Cluster`| +|TDengine|2\.2\.0\.2|`TDengine`| +|TDengine|3\.0\.1|`TDengine-3`| +|DolphinDB|v2\.x|`DolphinDB-2-MTW`, `DolphinDB-2-PTA`| +|DolphinDB|v3\.x|`DolphinDB-3-MTW`, `DolphinDB-3-PTA`| +|CnosDB|—|`CnosDB`| + +Notes: + +- IoTDB access methods include JDBC, REST, Session by Tablet, Session by Record, and Session by Records. + +- For DolphinDB, `MTW` means `MultithreadedTableWriter`, which buffers writes by row; `PTA` means `PartitionedTableAppender`, which appends an entire table in columnar batches. + +- Database versions, drivers, and servers must be compatible with one another. When using other databases, configure the corresponding connection and extension parameters. + +### 1\.2 IoTDB 2\.0\.x Access Methods + +The following installation, configuration, and examples primarily use IoTDB 2\.0\.x. The supported access methods are listed below. + +|Access Method|`DB_SWITCH`|Description| +|---|---|---| +|JDBC|`IoTDB-200-JDBC`|Performs writes and queries through JDBC| +|REST|`IoTDB-200-REST`|Performs tests through the IoTDB REST interface| +|Session by Tablet|`IoTDB-200-SESSION_BY_TABLET`|Uses Tablet for batch writes| +|Session by Record|`IoTDB-200-SESSION_BY_RECORD`|Writes records one at a time| +|Session by Records|`IoTDB-200-SESSION_BY_RECORDS`|Writes multiple records in batches| + +`IoTDB-200-SESSION_BY_TABLET` is suitable for batch write tests. + +## 2\. Installation and Execution + +This chapter uses IoTDB 2\.0\.x as the database under test. + +### 2\.1 Prerequisites + +1. Before using IoT Benchmark, prepare: -Figure 1-2 *IoT-benchmark Modular Design* + - Java 17. -**Supported Databases** + - Maven. -Currently, IoT-benchmark supports the following time series databases, versions and connection methods: + - An installed and running IoTDB 2\.0\.x instance. -| Database | Version | Connection mmethod | -| :-------------- |:-----------| :------------------------------------------------------- | -| IoTDB | v1.x v2.x | JDBC, SessionByTablet, SessionByRecord, SessionByRecords | -| InfluxDB | v1.x v2.x | SDK | -| TimescaleDB | -- | JDBC | -| OpenTSDB | -- | HTTP Request | -| QuestDB | v6.0.7 | JDBC | -| TDengine | v2.2.0.2 | JDBC | -| VictoriaMetrics | v1.64.0 | HTTP Request | -| KairosDB | -- | HTTP Request | + - Sufficient client CPU, memory, disk, and network resources. +2. Environment notes: -## 2. **Installation and Operation** + - Linux or macOS is recommended for running tests. -### 2.1 **Prerequisites** + - On Windows, use `benchmark.bat` in the installation package root directory to start a test. -1. Java 8 -2. Maven 3.6+ -3. The corresponding appropriate version of the database, such as Apache IoTDB 2.0 + - On Linux and macOS, use `benchmark.sh` to start a test. -### 2.2 **How to Obtain** + - Some system information collection capabilities in CSV recording mode are supported only on Linux. -- **B****inary package****:** Visit https://github.com/thulab/iot-benchmark/releases to download the installation package. Extract the compressed file into a desired folder for use. +> Note: Do not deploy IoT Benchmark and the IoTDB instance under test in environments where they compete for resources. For formal performance testing, use separate servers and stop unrelated services. +> +> -- **Source Code** **Compilation (for** **Apache** **IoTDB 2.0 testing):** +### 2\.2 Obtaining IoT Benchmark - - **Compile the latest IoTDB Session package:** Download the IoTDB source code from https://github.com/apache/iotdb/tree/rc/2.0.5 and run the following command in the root directory to compile the latest IoTDB Session package: +1. Download a release package - ```Bash - mvn clean package install -pl session -am -DskipTests - ``` +Download a release package matching the test target from [IoT Benchmark Releases](https://github.com/thulab/iot-benchmark/releases), and extract it for use. - - **Compile the IoT-benchmark test package:** Download the source code from https://github.com/thulab/iot-benchmark and run the following command in the root directory to compile the Apache IoTDB 2.0 test package:. +2. Build from source - ```Bash - mvn clean package install -pl iotdb-2.0 -am -DskipTests - ``` +Clone the repository: - - The compiled test package will be located at: +```Bash +git clone https://github.com/thulab/iot-benchmark.git +cd iot-benchmark +``` - ```Bash - ./iotdb-2.0/target/iotdb-2.0-0.0.1/iotdb-2.0-0.0.1 - ``` +Run the following command in the project root directory: -### 2.3 **Test Package Structure** +```Bash +mvn clean package -Dmaven.test.skip=true +``` -The directory structure of the test package is shown below. The test configuration file is `conf/config.properties`, and the test startup scripts are `benchmark.sh` (Linux & MacOS) and `benchmark.bat` (Windows). The detailed usage of the files is shown in the table below. +After the build is complete, the IoTDB 2\.0\.x package is located at: -```Shell --rw-r--r--. 1 root root 2881 Jan 10 01:36 benchmark.bat --rwxr-xr-x. 1 root root 314 Jan 10 01:36 benchmark.sh -drwxr-xr-x. 2 root root 24 Jan 10 01:36 bin --rwxr-xr-x. 1 root root 1140 Jan 10 01:36 cli-benchmark.sh -drwxr-xr-x. 2 root root 107 Jan 10 01:36 conf -drwxr-xr-x. 2 root root 4096 Jan 10 01:38 lib --rw-r--r--. 1 root root 11357 Jan 10 01:36 LICENSE --rwxr-xr-x. 1 root root 939 Jan 10 01:36 rep-benchmark.sh --rw-r--r--. 1 root root 14 Jan 10 01:36 routine +```Plain Text +iotdb-2.0/target/iot-benchmark-iotdb-2.0/iot-benchmark-iotdb-2.0 ``` -| Name | File | Usage | -| :--------------- | :---------------- | :-------------------------------------------------- | -| benchmark.bat | - | Startup script on Windows | -| benchmark.sh | - | Startup script on Linux/Mac | -| bin | startup.sh | Initialization script folder | -| conf | config.properties | Test scenario configuration file | -| lib | - | Dependency library | -| LICENSE | - | License file | -| cli-benchmark.sh | - | One-click startup script | -| routine | - | Automatic execution of multiple test configurations | -| rep-benchmark.sh | - | Automatic execution of multiple test scripts | +Enter the installation directory: +```Bash +cd iotdb-2.0/target/iot-benchmark-iotdb-2.0/iot-benchmark-iotdb-2.0 +``` +### 2\.3 Package Structure -### 2.4 **Execution** **of** **Tests** +Common files and directories in the installation package are listed below. -1. Modify the configuration file (conf/config.properties) according to test requirements. For example, to test Apache IoTDB 2.0, set the following parameter: +|Name|Purpose| +|---|---| +|`benchmark.sh`|Startup script for Linux and macOS| +|`benchmark.bat`|Startup script for Windows| +|`conf/config.properties`|Test scenario configuration file| +|`lib/`|Runtime dependencies| +|`logs/`|Test logs, generated after the first run| +|`data/`|Dataset or persisted result directory, generated according to the work mode and persistence configuration| - ```Bash - DB_SWITCH=IoTDB-200-SESSION_BY_TABLET - ``` +### 2\.4 Running a Test -2. Ensure the target time-series database is running. +1. Start IoTDB -3. Start IoT-benchmark to execute the test. Monitor the status of both the target database and IoT-benchmark during execution. +First, start the target IoTDB 2\.0\.x instance and verify that the client can access its service port. The default native interface port is `6667`. -4. Upon completion, review the results and analyze the test process. +2. Modify the configuration -### 2.5 **Results Interpretation** +- Edit `conf/config.properties`. -All test log files are stored in the `logs` folder, while test results are saved in the `data/csvOutput` folder. For example, the following result matrix illustrates the test outcome: +- Minimal connection configuration example: -![](/img/bm4.png) +```Properties +DB_SWITCH=IoTDB-200-SESSION_BY_TABLET +IoTDB_DIALECT_MODE=tree +HOST=127.0.0.1 +PORT=6667 +USERNAME=root +PASSWORD=root +DB_NAME=test +``` + +- To use the table model, change the setting to: -- **Result Matrix:** - - OkOperation: Number of successful operations. - - OkPoint: Number of successfully written points (for write operations) or successfully queried points (for query operations). - - FailOperation: Number of failed operations. - - FailPoint: Number of failed write points. -- **Latency (ms) Matrix:** - - AVG: Average operation latency. - - MIN: Minimum operation latency. - - Pn: Quantile values of the overall operation distribution (e.g., P25 represents the 25th percentile, or lower quartile). +```Properties +IoTDB_DIALECT_MODE=table +``` -## 3. **Main** **Parameters** +- If REST is selected: + +```Properties +DB_SWITCH=IoTDB-200-REST +REST_PORT=18080 +REST_AUTHORIZATION=Basic cm9vdDpyb290 +``` -### 3.1 IoTDB Service Model +`REST_AUTHORIZATION` configures the `Basic Authentication` information for the REST interface. The example uses the username `root` and password `root`. + +3. Check RPC compression compatibility + +IoT Benchmark 2\.0 enables IoTDB RPC compression by default: + +```Properties +ENABLE_IOTDB_RPC_COMPRESSION=true +``` + +This feature requires IoTDB 2\.0\.6 or later. When testing an IoTDB 2\.0\.x version earlier than 2\.0\.6, set: + +```Properties +ENABLE_IOTDB_RPC_COMPRESSION=false +``` + +Thrift compression is a separate configuration. If it is enabled: + +```Properties +ENABLE_THRIFT_COMPRESSION=true +``` -The `IoTDB_DIALECT_MODE` parameter supports two modes: `tree` and `table`. The default value is `tree`. +Also set the following in IoTDB's `iotdb-datanode.properties`: -- **For IoTDB 2.0 and later versions**, the `IoTDB_DIALECT_MODE` parameter must be specified, and only one mode can be set for each IoTDB instance. -- **IoTDB_DIALECT_MODE = tree:** - - The number of devices must be greater than or equal to the number of databases. +```Properties +dn_rpc_thrift_compression_enable=true +``` -### 3.2 **Working** **Mode** +4. Start Benchmark -The `BENCHMARK_WORK_MODE` parameter supports four operational modes: - -1. **General Test Mode (****`testWithDefaultPath`****):** Configured via the `OPERATION_PROPORTION` parameter to support write-only, read-only, and mixed read-write operations. -2. **Data Generation Mode (****`generateDataMode`****):** Generates a reusable dataset, which is saved to `FILE_PATH` for subsequent use in the correctness write and correctness query modes. -3. **Single Database Correctness Write Mode (****`verificationWriteMode`****):** Verifies the correctness of dataset writing by writing the dataset generated in data generation mode. This mode supports only IoTDB v1.0+ and InfluxDB v1.x. -4. **Single Database Correctness Query Mode (****`verificationQueryMode`****):** Verifies the correctness of dataset queries after using the correctness write mode. This mode supports only IoTDB v1.0+ and InfluxDB v1.x. - -Mode configurations are shown in the following below: - -| **Mode name** | **BENCHMARK_WORK_MODE** | Description | Required Configuration | -| :------------------------------------- | :---------------------- | :------------------------------------------------------ | :------------------------- | -| General test mode | testWithDefaultPath | Supports multiple read and write mixed load operations. | `OPERATION_PROPORTION` | -| Generate data mode | generateDataMode | Generates datasets recognizable by IoT-benchmark. | `FILE_PATH` and `DATA_SET` | -| Single database correctness write mode | verificationWriteMode | Writes datasets for correctness verification. | `FILE_PATH` and `DATA_SET` | -| Single database correctness query mode | verificationQueryMode | Queries datasets to verify correctness. | `FILE_PATH` and `DATA_SET` | - -### 3.3 **Server** **Connection** **Information** - -Once the working mode is specified, the following parameters must be configured to inform IoT-benchmark of the target time-series database: - -| **Parameter** | **Type** | **Example** | D**escription** | -| :------------ | :------- | :---------------------------- | :----------------------------------------------------- | -| DB_SWITCH | String | `IoTDB-200-SESSION_BY_TABLET` | Specifies the type of time-series database under test. | -| HOST | String | `127.0.0.1` | Network address of the target time-series database. | -| PORT | Integer | `6667` | Network port of the target time-series database. | -| USERNAME | String | `root` | Login username for the time-series database. | -| PASSWORD | String | `root` | Password for the database login user. | -| DB_NAME | String | `test` | Name of the target time-series database. | -| TOKEN | String | - | Authentication token (used for InfluxDB 2.0). | - -### 3.4 **Write Scenario Parameters** - -| **Parameter** | **Type** | **Example** | D**escription** | -| :------------------------- | :-------------------- | :-------------------------- | :----------------------------------------------------------- | -| CLIENT_NUMBER | Integer | `100` | Total number of clients used for writing. | -| GROUP_NUMBER | Integer | `20` | Number of databases (only applicable for IoTDB). | -| DEVICE_NUMBER | Integer | `100` | Total number of devices. | -| SENSOR_NUMBER | Integer | `300` | Total number of sensors per device. (Control the number of attribute columns if you use the IoTDB table model) | -| INSERT_DATATYPE_PROPORTION | String | `1:1:1:1:1:1:0:0:0:0` | Ratio of data types: `BOOLEAN:INT32:INT64:FLOAT:DOUBLE:TEXT:STRING:BLOB:TIMESTAMP:DATE`. | -| POINT_STEP | Integer | `1000` | Time interval (in ms) between generated data points. | -| OP_MIN_INTERVAL | Integer | `0` | Minimum execution interval for operations (ms): if the operation takes more than the value, the next one will be executed immediately, otherwise wait (OP_MIN_INTERVAL - actual execution time) ms; if it is 0, the parameter is not effective; if it is -1, its value is consistent with POINT_STEP | -| IS_OUT_OF_ORDER | Boolean | `false` | Specifies whether to write data out of order. | -| OUT_OF_ORDER_RATIO | Floating point number | `0.3` | Proportion of out-of-order data. | -| BATCH_SIZE_PER_WRITE | Integer | `1` | Number of data rows written per batch. | -| START_TIME | Time | `2022-10-30T00:00:00+08:00` | Start timestamp for data generation. | -| LOOP | Integer | `86400` | Total number of write operations: Each type of operation will be divided according to the proportion defined by `OPERATION_PROPORTION` | -| OPERATION_PROPORTION | Character | `1:0:0:0:0:0:0:0:0:0:0` | Ratio of operation types (write:Q1:Q2:...:Q10). | - -### 3.5 **Query Scenario Parameters** - -| Parameter | Type | Example | Description | -| :------------------- | :-------- | :---------------------- | :----------------------------------------------------------- | -| QUERY_DEVICE_NUM | Integer | `2` | Number of devices involved in each query statement. | -| QUERY_SENSOR_NUM | Integer | `2` | Number of sensors involved in each query statement. | -| QUERY_AGGREGATE_FUN | Character | `count` | Aggregate functions used in queries (`COUNT`, `AVG`, `SUM`, etc.). | -| STEP_SIZE | Integer | `1` | Time interval step for time filter conditions. | -| QUERY_INTERVAL | Integer | `250000` | Time interval between query start and end times. | -| QUERY_LOWER_VALUE | Integer | `-5` | Threshold for conditional queries (`WHERE value > QUERY_LOWER_VALUE`). | -| GROUP_BY_TIME_UNIT | Integer | `20000` | The size of the group in the `GROUP BY` statement | -| LOOP | Integer | `10` | Total number of query operations: Each type of operation will be divided according to the proportion defined by `OPERATION_PROPORTION` | -| OPERATION_PROPORTION | Character | `0:0:0:0:0:0:0:0:0:0:1` | Ratio of operation types (`write:Q1:Q2:...:Q10`). | - -### 3.6 **Query Types and Example SQL** - -| Number | Query Type | IoTDB Sample SQL | -| :----- | :----------------------------- | :----------------------------------------------------------- | -| Q1 | Precise Point Query | `select v1 from root.db.d1 where time = ?` | -| Q2 | Time Range Query | `select v1 from root.db.d1 where time > ? and time < ?` | -| Q3 | Time Range with Value Filter | `select v1 from root.db.d1 where time > ? and time < ? and v1 > ?` | -| Q4 | Time Range Aggregation Query | `select count(v1) from root.db.d1 where and time > ? and time < ?` | -| Q5 | Full-Time Range with Filtering | `select count(v1) from root.db.d1 where v1 > ?` | -| Q6 | Range Aggregation with Filter | `select count(v1) from root.db.d1 where v1 > ? and time > ? and time < ?` | -| Q7 | Time Grouping Aggregation | `select count(v1) from root.db.d1 group by ([?, ?), ?, ?)` | -| Q8 | Latest Point Query | `select last v1 from root.db.d1` | -| Q9 | Descending Range Query | `select v1 from root.sg.d1 where time > ? and time < ? order by time desc` | -| Q10 | Descending Range with Filter | `select v1 from root.sg.d1 where time > ? and time < ? and v1 > ? order by time desc` | - -### 3.7 **Test process and test result persistence** - -IoT-benchmark currently supports persisting the test process and test results through configuration parameters. - -| **Parameter** | **Type** | **Example** | D**escription** | -| :-------------------- | :------- | :---------- | :----------------------------------------------------------- | -| TEST_DATA_PERSISTENCE | String | `None` | Specifies the result persistence method. Options: `None`, `IoTDB`, `MySQL`, `CSV`. | -| RECORD_SPLIT | Boolean | `true` | Whether to split results into multiple records. (Not supported by IoTDB currently.) | -| RECORD_SPLIT_MAX_LINE | Integer | `10000000` | Maximum number of rows per record (10 million rows per database table or CSV file). | -| TEST_DATA_STORE_IP | String | `127.0.0.1` | IP address of the database for result storage. | -| TEST_DATA_STORE_PORT | Integer | `6667` | Port number of the output database. | -| TEST_DATA_STORE_DB | String | `result` | Name of the output database. | -| TEST_DATA_STORE_USER | String | `root` | Username for accessing the output database. | -| TEST_DATA_STORE_PW | String | `root` | Password for accessing the output database. | - -**Result Persistence Details** - -- **CSV Mode:** If `TEST_DATA_PERSISTENCE` is set to `CSV`, a `data` folder is generated in the IoT-benchmark root directory during and after test execution. This folder contains: - - `csv` folder: Records the test process. - - `csvOutput` folder: Stores the test results. -- **MySQL Mode:** If `TEST_DATA_PERSISTENCE` is set to `MySQL`, IoT-benchmark creates the following tables in the specified MySQL database: - - **Test Process Table:** - 1. Created before the test starts. - 2. Named as: `testWithDefaultPath___`. - - **Configuration Table:** - 1. Named `CONFIG`. - 2. Stores the test configuration. - 3. Created if it does not exist. - - **Final Result Table:** - 1. Named `FINAL_RESULT`. - 2. Stores the test results after test completion. - 3. Created if it does not exist. - -### 3.8 Automation Script - -#### One-Click Script Startup - -The `cli-benchmark.sh` script allows one-click startup of IoTDB, IoTDB Benchmark monitoring, and IoTDB Benchmark testing. However, please note that this script will clear all existing data in IoTDB during startup, so use it with caution. - -**Steps to Run:** - -1. Edit the `IOTDB_HOME` parameter in `cli-benchmark.sh` to the local IoTDB directory. -2. Start the test by running the following command: +On Linux or macOS: ```Bash -> ./cli-benchmark.sh +./benchmark.sh +``` + +On Windows: + +```Plain Text +benchmark.bat +``` + +During the test, progress is periodically printed to the terminal. When the test completes, the main configurations, execution time, result matrix, and latency matrix are displayed. + +### 2\.5 Understanding the Results + +Test execution information is written to the `logs` folder in the installation directory. Whether CSV files are generated or results are written to a result database depends on parameters such as `TEST_DATA_PERSISTENCE`. + +1. Result matrix + +The result matrix reports the following metrics by operation type: + +|Metric|Description| +|---|---| +|`okOperation`|Number of successfully executed requests or SQL statements| +|`okPoint`|Number of successfully written data points, or data points successfully returned by queries| +|`failOperation`|Number of failed requests or SQL statements| +|`failPoint`|Number of data points that failed to be written; usually 0 for query operations| +|`throughput`|Throughput, usually equal to `okPoint / Test elapsed time`| + +The main operation names in the output include: + +- `INGESTION` + +- `PRECISE_POINT` + +- `TIME_RANGE` + +- `VALUE_RANGE` + +- `AGG_RANGE` + +- `AGG_VALUE` + +- `AGG_RANGE_VALUE` + +- `GROUP_BY` + +- `LATEST_POINT` + +- `RANGE_QUERY_DESC` + +- `VALUE_RANGE_QUERY_DESC` + +- `GROUP_BY_DESC` + +- `SET_OP_QUERY` + +2. Latency matrix + +The latency matrix is measured in milliseconds. Common fields are listed below. + +|Metric|Description| +|---|---| +|`AVG`|Average latency| +|`MIN`|Minimum latency| +|`P10`, `P25`, `MEDIAN`|Lower percentiles and median latency| +|`P75`, `P90`, `P95`|Higher-percentile latency| +|`P99`, `P999`|Tail latency| +|`MAX`|Maximum latency| +|`SLOWEST_THREAD`|Largest cumulative operation time among client threads| + +The test results also report metadata creation time and `Test elapsed time`, which excludes metadata creation. When comparing tests, ensure that the hardware, data volume, number of clients, compression configuration, and cache state are consistent across runs. + +3. Output example + +After the test completes, the terminal displays the main configurations, execution time, result matrix, and latency matrix. The following is truncated output from a write-only test: + +```Plain Text +----------------------Main Configurations---------------------- +BENCHMARK_WORK_MODE=testWithDefaultPath +DB_SWITCH=IoTDB-200-SESSION_BY_TABLET +HOST=[127.0.0.1] + +GROUP_NUMBER=10 +DEVICE_NUMBER=50 +SENSOR_NUMBER=500 +SCHEMA_CLIENT_NUMBER=20 +DATA_CLIENT_NUMBER=20 + +OPERATION_PROPORTION=1:0:0:0:0:0:0:0:0:0:0:0:0 +LOOP=10000 +BATCH_SIZE_PER_WRITE=100 +--------------------------------------------------------------- + +Create schema cost 0.30 second +Test elapsed time (not include schema creation): 1238.79 second + +----------------------------------------------------------Result Matrix---------------------------------------------------------- +Operation okOperation okPoint failOperation failPoint throughput(point/s) +INGESTION 500000 25000000000 0 0 20180954.09 +--------------------------------------------------------------------------------------------------------------------------------- + +----------------------------------------------------------Latency (ms) Matrix---------------------------------------------------- +Operation AVG MIN P10 P25 MEDIAN P75 P90 P95 P99 P999 MAX SLOWEST_THREAD +INGESTION 37.78 1.67 2.02 2.29 2.86 4.14 5.62 7.43 759.69 5799.89 8309.40 1227561.44 +--------------------------------------------------------------------------------------------------------------------------------- +``` + +The output shows that: + +- Metadata creation took `0.30` seconds, and the actual test took `1238.79` seconds. + +- A total of `500000` write operations successfully wrote `25000000000` data points. + +- Both `failOperation` and `failPoint` are `0`, indicating that no write failures were recorded during this test. + +- Write throughput was `20180954.09` points per second. + +- Average latency was `37.78` ms, P95 latency was `7.43` ms, P99 latency was `759.69` ms, and maximum latency was `8309.40` ms. + +- `SLOWEST_THREAD` is the cumulative operation time of the slowest client thread, not the latency of a single request. + +The values in this example are only intended to demonstrate the output format. Actual results depend on hardware resources, network conditions, IoTDB configuration, data scale, and test parameters. + +## 3\. Main Parameters + +### 3\.1 IoTDB Data Model + +IoTDB 2\.0\.x supports the tree model and the table model. Select the model using: + +```Properties +IoTDB_DIALECT_MODE=tree +``` + +or: + +```Properties +IoTDB_DIALECT_MODE=table +``` + +The following constraints apply: + +- An IoTDB instance can use only one SQL dialect in a single test. + +- The tree model requires `DEVICE_NUMBER >= GROUP_NUMBER`. + +- In the table model, the number of devices must be a multiple of the number of tables, and the number of tables must be a multiple of the number of databases. + +- In the table model, the number of data clients must be a multiple of the number of tables. + +Common model parameters: + +|Parameter|Example|Description| +|---|---|---| +|`IoTDB_DIALECT_MODE`|`tree`|`tree` or `table`| +|`GROUP_NUMBER`|`1`|Number of databases; corresponds to the number of databases in the tree model| +|`IoTDB_TABLE_NUMBER`|`1`|Number of tables created in the table model| +|`IoTDB_TABLE_NAME_PREFIX`|`table_`|Table name prefix| +|`TABLE_TIME_COLUMN`|`time`|Name of the time column in the table model| +|`IoTDB_TABLE_WRITABLE_VIEW`|`false`|Whether to create and use writable views| + +### 3\.2 Work Modes + +Use `BENCHMARK_WORK_MODE` to select a work mode. + +|Mode|Configuration Value|Description| +|---|---|---| +|Regular test mode|`testWithDefaultPath`|Runs a write, query, or mixed workload| +|Data generation mode|`generateDataMode`|Saves the dataset generated by Benchmark to `FILE_PATH`| +|Correctness write mode|`verificationWriteMode`|Loads a dataset from `FILE_PATH` and writes it to the database| +|Correctness query mode|`verificationQueryMode`|Loads a dataset and compares it with database query results| + +Example: + +```Properties +BENCHMARK_WORK_MODE=testWithDefaultPath +``` + +Before using the correctness write and query modes, use `generateDataMode` to generate a reusable dataset. + +### 3\.3 Server Connection Information + +|Parameter|Example|Description| +|---|---|---| +|`DB_SWITCH`|`IoTDB-200-SESSION_BY_TABLET`|Database version and connection method| +|`HOST`|`127.0.0.1`|IoTDB address; separate multiple addresses with commas| +|`PORT`|`6667`|Native service port; the number of ports must match the number of `HOST` entries| +|`USERNAME`|`root`|Username| +|`PASSWORD`|`root`|Password| +|`DB_NAME`|`test`|Name of the database used for testing| +|`REST_PORT`|`18080`|REST service port| +|`REST_AUTHORIZATION`|`Basic cm9vdDpyb290`|REST authorization header| +|`ENABLE_AUTO_FETCH`|`false`|Whether Session automatically refreshes the DataNode list| + +Data cleanup parameters: + +```Properties +IS_DELETE_DATA=false +INIT_WAIT_TIME=1000 +``` + +> Warning: `IS_DELETE_DATA=true` clears test data from the target database before the test starts. Enable it only in a dedicated test environment, and verify `HOST`, `PORT`, `DB_NAME`, and account permissions before execution. +> +> + +### 3\.4 Write Scenarios + +1. Data scale and clients + +|Parameter|Example|Description| +|---|---|---| +|`DEVICE_NUMBER`|`100`|Total number of devices| +|`SENSOR_NUMBER`|`10`|Number of measurements per device; number of measurement columns in the table model| +|`GROUP_NUMBER`|`1`|Number of IoTDB databases| +|`SCHEMA_CLIENT_NUMBER`|`5`|Number of clients that register metadata| +|`DATA_CLIENT_NUMBER`|`10`|Number of clients that perform data reads and writes| +|`IS_CLIENT_BIND`|`true`|Whether devices are bound to clients| +|`REAL_INSERT_RATE`|`1.0`|Proportion of devices that actually participate in writes| +|`IS_SENSOR_TS_ALIGNMENT`|`true`|Whether measurement timestamps under the same device are aligned| + +2. Batch writes + +|Parameter|Example|Description| +|---|---|---| +|`BATCH_SIZE_PER_WRITE`|`100`|Number of data rows written per device in each batch| +|`DEVICE_NUM_PER_WRITE`|`1`|Number of devices involved in each batch write| +|`CREATE_SCHEMA`|`true`|Whether to create metadata before writing| +|`START_TIME`|`2022-01-01T00:00:00+08:00`|Start time for generated data| + +The number of data points in a single batch is: + +```Plain Text +DEVICE_NUM_PER_WRITE × SENSOR_NUMBER × BATCH_SIZE_PER_WRITE +``` + +`DEVICE_NUM_PER_WRITE` must evenly divide the number of devices assigned to a single data client. In the table model, the divisibility constraints among the number of devices, number of tables, and devices per batch must also be satisfied. + +3. Write pacing + +|Parameter|Example|Description| +|---|---|---| +|`POINT_STEP`|`5000`|Fixed interval between adjacent generated timestamps| +|`OP_MIN_INTERVAL`|`0`|Minimum interval for each loop, in ms| +|`OP_MIN_INTERVAL_RANDOM`|`false`|Whether to randomly select an interval from `[0, OP_MIN_INTERVAL)`| +|`INTERVAL_BETWEEN_WRITE_BATCH`|`0`|Minimum interval between adjacent batches in the same loop, in ms| +|`TIMESTAMP_PRECISION`|`ms`|Timestamp precision| + +Special values of `OP_MIN_INTERVAL`: + +- `0`: does not limit the loop interval. + +- `-1`: uses `POINT_STEP` as the minimum interval. + +- Positive integer: if the current loop takes less than this value, waits for the remaining time. + +4. Out-of-order writes + +```Properties +IS_OUT_OF_ORDER=false +OUT_OF_ORDER_MODE=POISSON +OUT_OF_ORDER_RATIO=0.5 +IS_REGULAR_FREQUENCY=true +``` + +Supported out-of-order modes include: + +- `POISSON`: generates out-of-order timestamps according to a Poisson distribution. + +- `BATCH`: generates out-of-order data in batches. + +5. Data types + +```Properties +INSERT_DATATYPE_PROPORTION=1:1:1:1:1:1:0:0:0:0:0 +``` + +The order of the entries is: + +```Plain Text +BOOLEAN:INT32:INT64:FLOAT:DOUBLE:TEXT:STRING:BLOB:TIMESTAMP:DATE:OBJECT +``` + +Each value represents the proportion of the corresponding data type. + +### 3\.5 Query Scenarios + +|Parameter|Example|Description| +|---|---|---| +|`QUERY_DEVICE_NUM`|`1`|Number of devices involved in each query| +|`QUERY_SENSOR_NUM`|`1`|Number of measurements involved in each query| +|`QUERY_AGGREGATE_FUN`|`count`|Aggregation function| +|`STEP_SIZE`|`0`|Step by which the query start time changes, in units of `POINT_STEP`| +|`QUERY_INTERVAL`|`250000`|Interval between query start and end times| +|`QUERY_LOWER_VALUE`|`-5`|Lower bound of the value filter| +|`GROUP_BY_TIME_UNIT`|`20000`|Group By window size| +|`QUERY_SET_OP_TYPE`|`union`|Set operation type| +|`QUERY_SET_OP_NUM`|`2`|Number of sub-sets in a set query; at least 2| +|`IS_RECENT_QUERY`|`false`|Whether to prioritize recently written data in mixed scenarios| +|`ENABLE_FIXED_QUERY`|`false`|Whether all query threads use the same device and measurement combinations| +|`RESULT_ROW_LIMIT`|`-1`|Query result row limit; `-1` means no limit| +|`ALIGN_BY_DEVICE`|`false`|Whether to use Align By Device| + +### 3\.6 Operation Proportions + +`OPERATION_PROPORTION` defines the proportions of writes and different query types. It contains 13 entries: + +```Plain Text +Write:Q1:Q2:Q3:Q4:Q5:Q6:Q7:Q8:Q9:Q10:Q11:Q12 ``` -1. After the test completes: -1. Check test-related logs in the `logs` folder. -2. Check monitoring-related logs in the `server-logs` folder. +For example, write-only: -#### Automatic Execution of Multiple Tests +```Properties +OPERATION_PROPORTION=1:0:0:0:0:0:0:0:0:0:0:0:0 +``` -Single tests are often insufficient without comparative results. Therefore, IoT-benchmark provides an interface for executing multiple tests in sequence. +Precise point query only: -1. **Routine Configuration:** Each line in the `routine` file specifies the parameters that change for each test. For example: +```Properties +OPERATION_PROPORTION=0:1:0:0:0:0:0:0:0:0:0:0:0 +``` - ```Plain - LOOP=10 DEVICE_NUMBER=100 TEST - LOOP=20 DEVICE_NUMBER=50 TEST - LOOP=50 DEVICE_NUMBER=20 TEST - ``` +The operation types are listed below. + +|Number|Operation Type|Description| +|---|---|---| +|Write|Data write|Generates and writes data according to the current write configuration| +|Q1|Precise point query|Queries specified measurements by timestamp and device| +|Q2|Time range query|Range query restricted only by start and end times| +|Q3|Range query with value filter|Includes both time and value filter conditions| +|Q4|Aggregation query with time filter|Performs aggregation within a time range| +|Q5|Aggregation query with value filter|Filters by value and aggregates over the full time range| +|Q6|Aggregation query with time and value filters|Includes both time and value filter conditions| +|Q7|Time-grouped aggregation query|Group By query| +|Q8|Latest point query|Queries the latest data point of a device| +|Q9|Descending time range query|Returns range query results in descending time order| +|Q10|Descending range query with value filter|Filters by value and returns results in descending time order| +|Q11|Descending time-grouped aggregation query|Descending Group By query| +|Q12|Set operation query|Set operations such as `union`, `intersect`, or `except`| + +Q12 is supported only by the IoTDB 2\.0 table model. Each subquery in a set operation is a range query. + +### 3\.7 Test Process and Result Persistence + +```Properties +TEST_DATA_PERSISTENCE=None +``` -In this example, three tests will run sequentially with `LOOP` values of 10, 20, and 50. +Supported values include: -Then the test process with 3 LOOP parameters of 10, 20, and 50 is executed in sequence. +- `None`: does not write the test process to an external persistence medium. -**Important Notes:** +- `CSV`: writes to CSV files. -- Multiple parameters can be changed in each test using the format: +- `MySQL`: writes to MySQL. - ```Bash - LOOP=20 DEVICE_NUMBER=10 TEST - ``` +- `IoTDB`: writes to a specified IoTDB instance. -- Avoid unnecessary spaces. +Common parameters: -- The `TEST` keyword marks the start of a new test. +|Parameter|Example|Description| +|---|---|---| +|`TEST_DATA_PERSISTENCE`|`None`|Persistence method| +|`RECORD_SPLIT`|`true`|Whether to split results into multiple records| +|`RECORD_SPLIT_MAX_LINE`|`10000000`|Maximum number of records in a single table or file| +|`TEST_DATA_STORE_IP`|`127.0.0.1`|Result database address| +|`TEST_DATA_STORE_PORT`|`6667`|Result database port| +|`TEST_DATA_STORE_DB`|`result`|Result database name| +|`TEST_DATA_STORE_USER`|`root`|Result database username| +|`TEST_DATA_STORE_PW`|`root`|Result database password| +|`REMARK`|`write_test`|Test note used to distinguish different tests| +|`CSV_OUTPUT`|`true`|Whether to write final results to CSV| -- Changed parameters persist across subsequent tests unless explicitly reset. +When the persistence method is CSV, records are generated in the `data` directory after execution; test results are usually located in `data/csvOutput`. Test logs are always written to `logs`, regardless of whether persistence is enabled. -2. **Start the Test:** After configuring the `routine` file, start multi-test execution using the following command +The following parameters control log output frequency: - ```Bash - > ./rep-benchmark.sh - ``` +```Properties +IS_QUIET_MODE=true +LOG_PRINT_INTERVAL=5 +RESULT_PRINT_INTERVAL=3600 +``` -Test results will be displayed in the terminal. +### 3\.8 Automation and Cluster Testing -**Important Notes:** +1. Limit test duration -- Closing the terminal or losing the client connection will terminate the test process. +```Properties +TEST_MAX_TIME=3600000 +``` -- To run the test as a background daemon, execute: +The unit is milliseconds. A value of `0` means no limit. This parameter does not include the time spent pre-registering metadata. - ```Bash - > ./rep-benchmark.sh > /dev/null 2>&1 & - ``` +2. Multi-Benchmark cluster load testing -- To monitor progress, check the logs: +Use the same overall data scale configuration on multiple client machines, and set: - ```Bash - > cd ./logs - > tail -f log_info.log - ``` +```Properties +BENCHMARK_CLUSTER=true +BENCHMARK_INDEX=0 +``` -## 4. Use Case +Each Benchmark instance must use a different `BENCHMARK_INDEX`, such as `0`, `1`, and `2` in sequence. All clients should use consistent database connections, data scales, operation proportions, and other configurations. -We take the application of CRRC Qingdao Sifang Vehicle Research Institute Co., Ltd. as an example, and refer to the scene described in "Apache IoTDB in Intelligent Operation and Maintenance Platform Storage" for practical operation instructions. +3. Dual-write testing -Test objective: Simulate the actual needs of switching time series databases in the scene of CRRC Qingdao Sifang Institute, and compare the performance of the expected IoTDB and KairosDB used by the original system. +IoT Benchmark can write the same data to two different databases for comparison: -Test environment: In order to ensure that the impact of other irrelevant services and processes on database performance and the mutual influence between different databases are eliminated during the experiment, the local databases in this experiment are deployed and run on multiple independent virtual servers with the same resource configuration. Therefore, this experiment set up 4 Linux (CentOS7 /x86) virtual machines, and deployed IoT-benchmark, IoTDB database, KairosDB database, and MySQL database on them respectively. The specific resource configuration of each virtual machine is shown in Table 4-1. The specific usage of each virtual machine is shown in Table 4-2. +```Properties +IS_DOUBLE_WRITE=true +ANOTHER_DB_SWITCH= +ANOTHER_HOST=127.0.0.1 +ANOTHER_PORT=6667 +ANOTHER_USERNAME=root +ANOTHER_PASSWORD=root +ANOTHER_DB_NAME=test +``` -Table 4-1 Virtual machine configuration information +Dual-write mode does not support comparisons between different versions of the same database, or direct comparisons between the IoTDB tree model and table model. -| Hardware Configuration Information | Value | -| ---------------------------------- | ------- | -| OS system | CentOS7 | -| number of CPU cores | 16 | -| memory | 32G | -| hard disk | 200G | -| network | Gigabit | +## 4\. Examples -Table 4-2 Virtual machine usage +This section uses small datasets to demonstrate the basic process. For formal performance testing, increase the number of devices, measurements, clients, and loops according to the target business model, and perform multiple warm-up and repeated test runs. -| IP | Usage | -| ---------- | ------------- | -| 172.21.4.2 | IoT-benchmark | -| 172.21.4.3 | Apache-iotdb | -| 172.21.4.4 | KaiosDB | -| 172.21.4.5 | MySQL | +### 4\.1 Write Test Example -### 4.1 Write Test +Test objective: use 10 data clients to simulate 100 devices, each containing 10 measurements, and perform a write-only test. -Scenario description: Create 100 clients to simulate 100 trains, each train has 3000 sensors, the data type is DOUBLE, the data time interval is 500ms (2Hz), and they are sent sequentially. Referring to the above requirements, we need to modify the IoT-benchmark configuration parameters as listed in Table 4-3. +Example configuration: -Table 4-3 Configuration parameter information +```Properties +# Database connection +DB_SWITCH=IoTDB-200-SESSION_BY_TABLET +IoTDB_DIALECT_MODE=tree +HOST=127.0.0.1 +PORT=6667 +USERNAME=root +PASSWORD=root +DB_NAME=test -| Parameter Name | IoTDB Value | KairosDB Value | -| -------------------------- | --------------------------- | -------------- | -| DB_SWITCH | IoTDB-013-SESSION_BY_TABLET | KairosDB | -| HOST | 172.21.4.3 | 172.21.4.4 | -| PORT | 6667 | 8080 | -| BENCHMARK_WORK_MODE | testWithDefaultPath | | -| OPERATION_PROPORTION | 1:0:0:0:0:0:0:0:0:0:0 | | -| CLIENT_NUMBER | 100 | | -| GROUP_NUMBER | 10 | | -| DEVICE_NUMBER | 100 | | -| SENSOR_NUMBER | 3000 | | -| INSERT_DATATYPE_PROPORTION | 0:0:0:0:1:0 | | -| POINT_STEP | 500 | | -| OP_MIN_INTERVAL | 0 | | -| IS_OUT_OF_ORDER | false | | -| BATCH_SIZE_PER_WRITE | 1 | | -| LOOP | 10000 | | -| TEST_DATA_PERSISTENCE | MySQL | | -| TEST_DATA_STORE_IP | 172.21.4.5 | | -| TEST_DATA_STORE_PORT | 3306 | | -| TEST_DATA_STORE_DB | demo | | -| TEST_DATA_STORE_USER | root | | -| TEST_DATA_STORE_PW | admin | | -| REMARK | demo | | +# Safety setting: do not automatically delete existing data by default +IS_DELETE_DATA=false -First, start the tested time series databases Apache-IoTDB and KairosDB on 172.21.4.3 and 172.21.4.4 respectively, and then start server resource monitoring through the ser-benchamrk\.sh script on 172.21.4.2, 172.21.4.3 and 172.21.4.4 (Figure 4-1). Then modify the conf/config.properties files in the iotdb-0.13-0.0.1 and kairosdb-0.0.1 folders in 172.21.4.2 according to Table 4-3 to meet the test requirements. Use benchmark\.sh to start the writing test of Apache-IoTDB and KairosDB successively. +# Work mode +BENCHMARK_WORK_MODE=testWithDefaultPath +OPERATION_PROPORTION=1:0:0:0:0:0:0:0:0:0:0:0:0 +# Data scale +GROUP_NUMBER=1 +DEVICE_NUMBER=100 +SENSOR_NUMBER=10 +SCHEMA_CLIENT_NUMBER=5 +DATA_CLIENT_NUMBER=10 +IS_SENSOR_TS_ALIGNMENT=true -![img](/img/bm5.png) +# Write configuration +CREATE_SCHEMA=true +BATCH_SIZE_PER_WRITE=10 +DEVICE_NUM_PER_WRITE=1 +LOOP=100 +POINT_STEP=1000 +OP_MIN_INTERVAL=0 +START_TIME=2026-01-01T00:00:00+08:00 +INSERT_DATATYPE_PROPORTION=1:1:1:1:1:1:0:0:0:0:0 -Figure 4-1 Server monitoring tasks +# Can be enabled for IoTDB 2.0.6 and later +ENABLE_IOTDB_RPC_COMPRESSION=true -For example, if we first start the test on KairosDB, IoT-benchmark will create a CONFIG data table in the MySQL database to store the configuration information of this test (Figure 4-2), and there will be a log output of the current test progress during the test execution (Figure 4-3) . When the test is completed, the test result will be output (Figure 4-3), and the result will be written into the FINAL_RESULT data table (Figure 4-4). +# Output +TEST_DATA_PERSISTENCE=None +CSV_OUTPUT=true +REMARK=iotdb_2_write_test +``` -![img](/img/bm6.png) +Start the test: -Figure 4-2 Test configuration information table +```Bash +./benchmark.sh +``` + +After completion, focus on: + +- `okPoint` and `failPoint` for `INGESTION`. -![img](/img/bm7.png) -![img](/img/bm8.png) -![img](/img/bm9.png) -![img](/img/bm10.png) +- `throughput`. -Figure 4-3 Test progress and results +- `AVG`, `P95`, `P99`, and `MAX` latency. -![img](/img/bm11.png) +- Whether connection timeouts, write failures, or server exceptions are present in the logs. +### 4\.2 Query Test Example +Before running a query test, ensure that the target database contains data matching the query configuration. It is recommended to reuse data generated by the write test and disable automatic data deletion and metadata creation. -Figure 4-4 Test result table +The following example runs multiple query types: -Afterwards, we will start the test on Apache-IoTDB. The same IoT-benchmark will write the test configuration information in the MySQL database CONFIG data table. During the test execution, there will be a log to output the current test progress. When the test is completed, the test result will be output, and the result will be written into the FINAL_RESULT data table. +```Properties +DB_SWITCH=IoTDB-200-SESSION_BY_TABLET +IoTDB_DIALECT_MODE=tree +HOST=127.0.0.1 +PORT=6667 +USERNAME=root +PASSWORD=root +DB_NAME=test -According to the test result information, we know that under the same configuration the write delay times of Apache-IoTDB and KairosDB are 55.98ms and 1324.45ms respectively; the write throughputs are 5,125,600.86 points/second and 224,819.01 points/second respectively; the tests were executed respectively 585.30 seconds and 11777.99 seconds. And KairosDB has a write failure. After investigation, it is found that the data disk usage has reached 100%, and there is no disk space to continue receiving data. However, Apache-IoTDB has no write failure, and the disk space occupied after all data is written is only 4.7G (as shown in Figure 4-5); Apache-IoTDB is better than KairosDB in terms of write throughput and disk occupation. Of course, there will be other tests in the follow-up to observe and compare from various aspects, such as query performance, file compression ratio, data security, etc. +IS_DELETE_DATA=false +CREATE_SCHEMA=false +BENCHMARK_WORK_MODE=testWithDefaultPath -![img](/img/bm12.png) +GROUP_NUMBER=1 +DEVICE_NUMBER=100 +SENSOR_NUMBER=10 +SCHEMA_CLIENT_NUMBER=1 +DATA_CLIENT_NUMBER=10 +# Do not perform writes; Q1-Q11 have equal proportions; the tree model does not use Q12 +OPERATION_PROPORTION=0:1:1:1:1:1:1:1:1:1:1:1:0 +LOOP=100 + +QUERY_DEVICE_NUM=2 +QUERY_SENSOR_NUM=2 +QUERY_AGGREGATE_FUN=count +STEP_SIZE=1 +QUERY_INTERVAL=250000 +QUERY_LOWER_VALUE=-5 +GROUP_BY_TIME_UNIT=20000 +``` -Figure 4-5 Disk usage +To test set queries in the table model, switch the dialect to `table` and assign a proportion to Q12: -So what is the resource usage of each server during the test? What is the specific performance of each write operation? At this time, we can visualize the data in the server monitoring table and test process recording table by installing and using Tableau. The use of Tableau will not be introduced in this article. After connecting to the data table for test data persistence, the specific results are as follows (taking Apache-IoTDB as an example): +```Properties +IoTDB_DIALECT_MODE=table +OPERATION_PROPORTION=0:0:0:0:0:0:0:0:0:0:0:0:1 +QUERY_SET_OP_TYPE=union +QUERY_SET_OP_NUM=2 +``` +### 4\.3 Other Configuration Examples -![img](/img/bm13.png) -![img](/img/bm14.png) +1. Simulate an actual write rate +Set the minimum interval of each loop to the data timestamp interval: -Figure 4-6 Visualization of testing process in Tableau +```Properties +POINT_STEP=1000 +OP_MIN_INTERVAL=-1 +``` -### 4.2 Query Test +To distribute write requests evenly within a loop, use: -Scenario description: In the writing test scenario, 10 clients are simulated to perform all types of query tasks on the data stored in the time series database Apache-IoTDB. The configuration is as follows. +```Properties +INTERVAL_BETWEEN_WRITE_BATCH=100 +``` -Table 4-4 Configuration parameter information +2. Specify test duration -| Parameter Name | Example | -| -------------------- | --------------------- | -| CLIENT_NUMBER | 10 | -| QUERY_DEVICE_NUM | 2 | -| QUERY_SENSOR_NUM | 2 | -| QUERY_AGGREGATE_FUN | count | -| STEP_SIZE | 1 | -| QUERY_INTERVAL | 250000 | -| QUERY_LOWER_VALUE | -5 | -| GROUP_BY_TIME_UNIT | 20000 | -| LOOP | 30 | -| OPERATION_PROPORTION | 0:1:1:1:1:1:1:1:1:1:1 | +Test for one hour: -Results: +```Properties +TEST_MAX_TIME=3600000 +``` -![img](/img/bm15.png) +Ensure that `LOOP` is sufficiently large; otherwise, the test may end when the loop count is exhausted. -Figure 4-7 Query test results +3. Control generated data patterns -### 4.3 Description of Other Parameters +```Properties +LINE_RATIO=1 +SIN_RATIO=1 +SQUARE_RATIO=1 +RANDOM_RATIO=1 +CONSTANT_RATIO=1 +DATA_SEED=666 +STRING_LENGTH=10 +DOUBLE_LENGTH=2 +``` -In the previous chapters, the write performance comparison between Apache-IoTDB and KairosDB was performed, but if the user wants to perform a simulated real write rate test, how to configure it? How to control if the test time is too long? Are there any regularities in the generated simulated data? If the IoT-Benchmark server configuration is low, can multiple machines be used to simulate pressure output? +Fixing `DATA_SEED` helps generate reproducible data across multiple test runs. -Table 4-5 Configuration parameter information +## 5\. References -| Scenario | Parameter | Value | Notes | -| ------------------------------------------------------------ | -------------------------- | ------------------------------------------------------------ | ------------------------------------------------------------ | -| Simulate real write rate | OP_INTERVAL | -1 | You can also enter an integer to control the operation interval. | -| Specify test duration (1 hour) | TEST_MAX_TIME | 3600000 | The unit is ms; the LOOP execution time needs to be greater than this value. | -| Define the law of simulated data: support all data types, and the number is evenly classified; support five data distributions, and the number is evenly distributed; the length of the string is 10; the number of decimal places is 2. | INSERT_DATATYPE_PROPORTION | 1:1:1:1:1:1 | Data type distribution proportion | -| LINE_RATIO | 1 | linear | | -| SIN_RATIO | 1 | Fourier function | | -| SQUARE_RATIO | 1 | Square wave | | -| RANDOM_RATIO | 1 | Random number | | -| CONSTANT_RATIO | 1 | Constant | | -| STRING_LENGTH | 10 | String length | | -| DOUBLE_LENGTH | 2 | Decimal places | | -| Three machines simulate data writing of 300 devices | BENCHMARK_CLUSTER | true | Enable multi-benchmark mode | -| BENCHMARK_INDEX | 0, 1, 3 | Take the writing parameters in the [write test](./Benchmark.md#_4-1-write-test) as an example: No. 0 is responsible for writing data of device numbers 0-99; No. 1 is responsible for writing data of device numbers 100-199; No. 2 is responsible for writing data of device numbers 200-299. | | \ No newline at end of file +- [IoT Benchmark Documentation](https://github.com/thulab/iot-benchmark/tree/master/docs) diff --git a/src/UserGuide/latest-Table/Tools-System/Benchmark.md b/src/UserGuide/latest-Table/Tools-System/Benchmark.md index c2fd1939c..3ca07c95e 100644 --- a/src/UserGuide/latest-Table/Tools-System/Benchmark.md +++ b/src/UserGuide/latest-Table/Tools-System/Benchmark.md @@ -18,402 +18,799 @@ under the License. --> + # Benchmark Tool -## 1. **Basic Overview** +IoT Benchmark is a benchmark testing tool for time-series databases and real-time databases in Industrial Internet of Things (IIoT) scenarios. This manual introduces the tool's main capabilities, supported databases, and basic usage. The installation, configuration, and test examples primarily use IoTDB 2\.0\.x and cover both the tree model and the table model. + +![](/img/benchmark-%20English2.png) -IoT-benchmark is a time-series database benchmarking tool developed in Java for big data environments. It was developed and open-sourced by the School of Software, Tsinghua University. The tool is user-friendly, supports various write and query methods, allows storing test information and results for further queries or analysis, and integrates with Tableau for visualizing test results. +## 1\. Basic Overview -Figure 1-1 illustrates the test benchmark process and its extended functionalities, all of which can be streamlined by IoT-benchmark. It supports a variety of workloads, including write-only, read-only, and mixed write-and-read operations. Additionally, it offers software and hardware system monitoring, performance metric measurement, automated database initialization, test data analysis, and system parameter optimization. +IoT Benchmark can generate periodic time-series data based on configuration, perform writes and queries against a database, and collect metrics such as throughput, latency, and success rate. Its main capabilities include: -![](/img/benchmark-English1.png) +- Cross-platform operation: supports Linux, macOS, and Windows. -Figure 1-1 *IoT-benchmark Test Benchmark Process* +- Multiple workloads: supports write-only, query-only, and mixed read/write workloads. -IoT-benchmark adopts the modular design concept of the YCSB test tool, which separates workload generation, performance measurement, and database interface components. Its modular structure is illustrated in Figure 1-2. Unlike YCSB-based testing tools, IoT-benchmark introduces a system monitoring module that supports the persistence of both test data and system metrics. It also includes load-testing functionalities specifically designed for time-series data scenarios, such as batch writes and multiple out-of-order data insertion modes for IoT environments. +- Dataset generation: generated data can be saved to disk for repeated testing. -![](/img/benchmark-%20English2.png) +- Correctness verification: datasets can be loaded from disk to verify write and query correctness. + +- Multiple database support: tests can be performed against various time-series databases and real-time databases. IoTDB 2\.0\.x supports JDBC, REST, and multiple Session write methods. + +- Result persistence: test processes and results can be saved to files, CSV, MySQL, or IoTDB. + +- Test extensions: supports scenarios such as out-of-order writes, batch writes, cluster load testing, dual writes for comparison, and result visualization. + +### 1\.1 Supported Databases, Versions, and Access Methods + +IoT Benchmark supports the following databases and versions. During testing, use `DB_SWITCH` to select the corresponding database, version, and connection or write method. + +|Database|Supported Version|`DB_SWITCH`| +|---|---|---| +|IoTDB|v1\.x|`IoTDB-130-JDBC`, `IoTDB-130-REST`, `IoTDB-130-SESSION_BY_TABLET`, `IoTDB-130-SESSION_BY_RECORD`, `IoTDB-130-SESSION_BY_RECORDS`| +|IoTDB|v2\.x|`IoTDB-200-JDBC`, `IoTDB-200-REST`, `IoTDB-200-SESSION_BY_TABLET`, `IoTDB-200-SESSION_BY_RECORD`, `IoTDB-200-SESSION_BY_RECORDS`| +|InfluxDB|v1\.x|`InfluxDB`| +|InfluxDB|v2\.x|`InfluxDB-2.x`| +|QuestDB|v6\.0\.7|`QuestDB`| +|Microsoft SQL Server|2016 SP2|`MSSQLSERVER`| +|VictoriaMetrics|v1\.64\.0|`VictoriaMetrics`| +|SQLite|—|`SQLite`| +|OpenTSDB|2\.4\.1|`OpenTSDB`| +|KairosDB|—|`KairosDB`| +|TimescaleDB|—|`TimescaleDB`| +|TimescaleDB Cluster|Cluster|`TimescaleDB-Cluster`| +|TDengine|2\.2\.0\.2|`TDengine`| +|TDengine|3\.0\.1|`TDengine-3`| +|DolphinDB|v2\.x|`DolphinDB-2-MTW`, `DolphinDB-2-PTA`| +|DolphinDB|v3\.x|`DolphinDB-3-MTW`, `DolphinDB-3-PTA`| +|CnosDB|—|`CnosDB`| + +Notes: + +- IoTDB access methods include JDBC, REST, Session by Tablet, Session by Record, and Session by Records. + +- For DolphinDB, `MTW` means `MultithreadedTableWriter`, which buffers writes by row; `PTA` means `PartitionedTableAppender`, which appends an entire table in columnar batches. + +- Database versions, drivers, and servers must be compatible with one another. When using other databases, configure the corresponding connection and extension parameters. + +### 1\.2 IoTDB 2\.0\.x Access Methods + +The following installation, configuration, and examples primarily use IoTDB 2\.0\.x. The supported access methods are listed below. -Figure 1-2 *IoT-benchmark Modular Design* +|Access Method|`DB_SWITCH`|Description| +|---|---|---| +|JDBC|`IoTDB-200-JDBC`|Performs writes and queries through JDBC| +|REST|`IoTDB-200-REST`|Performs tests through the IoTDB REST interface| +|Session by Tablet|`IoTDB-200-SESSION_BY_TABLET`|Uses Tablet for batch writes| +|Session by Record|`IoTDB-200-SESSION_BY_RECORD`|Writes records one at a time| +|Session by Records|`IoTDB-200-SESSION_BY_RECORDS`|Writes multiple records in batches| -**Supported Databases** +`IoTDB-200-SESSION_BY_TABLET` is suitable for batch write tests. -Currently, IoT-benchmark supports the following time series databases, versions and connection methods: +## 2\. Installation and Execution -| Database | Version | Connection mmethod | -| :-------------- |:-----------| :------------------------------------------------------- | -| IoTDB | v1.x v2.x | JDBC, SessionByTablet, SessionByRecord, SessionByRecords | -| InfluxDB | v1.x v2.x | SDK | -| TimescaleDB | -- | JDBC | -| OpenTSDB | -- | HTTP Request | -| QuestDB | v6.0.7 | JDBC | -| TDengine | v2.2.0.2 | JDBC | -| VictoriaMetrics | v1.64.0 | HTTP Request | -| KairosDB | -- | HTTP Request | +This chapter uses IoTDB 2\.0\.x as the database under test. +### 2\.1 Prerequisites -## 2. **Installation and Operation** +1. Before using IoT Benchmark, prepare: -### 2.1 **Prerequisites** + - Java 17. -1. Java 8 -2. Maven 3.6+ -3. The corresponding appropriate version of the database, such as Apache IoTDB 2.0 + - Maven. -### 2.2 **How to Obtain** + - An installed and running IoTDB 2\.0\.x instance. -- **B****inary package****:** Visit https://github.com/thulab/iot-benchmark/releases to download the installation package. Extract the compressed file into a desired folder for use. + - Sufficient client CPU, memory, disk, and network resources. -- **Source Code** **Compilation (for** **Apache** **IoTDB 2.0 testing):** +2. Environment notes: - - **Compile the latest IoTDB Session package:** Download the IoTDB source code from https://github.com/apache/iotdb/tree/rc/2.0.5 and run the following command in the root directory to compile the latest IoTDB Session package: + - Linux or macOS is recommended for running tests. - ```Bash - mvn clean package install -pl session -am -DskipTests - ``` + - On Windows, use `benchmark.bat` in the installation package root directory to start a test. - - **Compile the IoT-benchmark test package:** Download the source code from https://github.com/thulab/iot-benchmark and run the following command in the root directory to compile the Apache IoTDB 2.0 test package:. + - On Linux and macOS, use `benchmark.sh` to start a test. - ```Bash - mvn clean package install -pl iotdb-2.0 -am -DskipTests - ``` + - Some system information collection capabilities in CSV recording mode are supported only on Linux. - - The compiled test package will be located at: +> Note: Do not deploy IoT Benchmark and the IoTDB instance under test in environments where they compete for resources. For formal performance testing, use separate servers and stop unrelated services. +> +> - ```Bash - ./iotdb-2.0/target/iotdb-2.0-0.0.1/iotdb-2.0-0.0.1 - ``` +### 2\.2 Obtaining IoT Benchmark -### 2.3 **Test Package Structure** +1. Download a release package -The directory structure of the test package is shown below. The test configuration file is `conf/config.properties`, and the test startup scripts are `benchmark.sh` (Linux & MacOS) and `benchmark.bat` (Windows). The detailed usage of the files is shown in the table below. +Download a release package matching the test target from [IoT Benchmark Releases](https://github.com/thulab/iot-benchmark/releases), and extract it for use. -```Shell --rw-r--r--. 1 root root 2881 Jan 10 01:36 benchmark.bat --rwxr-xr-x. 1 root root 314 Jan 10 01:36 benchmark.sh -drwxr-xr-x. 2 root root 24 Jan 10 01:36 bin --rwxr-xr-x. 1 root root 1140 Jan 10 01:36 cli-benchmark.sh -drwxr-xr-x. 2 root root 107 Jan 10 01:36 conf -drwxr-xr-x. 2 root root 4096 Jan 10 01:38 lib --rw-r--r--. 1 root root 11357 Jan 10 01:36 LICENSE --rwxr-xr-x. 1 root root 939 Jan 10 01:36 rep-benchmark.sh --rw-r--r--. 1 root root 14 Jan 10 01:36 routine +2. Build from source + +Clone the repository: + +```Bash +git clone https://github.com/thulab/iot-benchmark.git +cd iot-benchmark +``` + +Run the following command in the project root directory: + +```Bash +mvn clean package -Dmaven.test.skip=true +``` + +After the build is complete, the IoTDB 2\.0\.x package is located at: + +```Plain Text +iotdb-2.0/target/iot-benchmark-iotdb-2.0/iot-benchmark-iotdb-2.0 +``` + +Enter the installation directory: + +```Bash +cd iotdb-2.0/target/iot-benchmark-iotdb-2.0/iot-benchmark-iotdb-2.0 ``` -| Name | File | Usage | -| :--------------- | :---------------- | :-------------------------------------------------- | -| benchmark.bat | - | Startup script on Windows | -| benchmark.sh | - | Startup script on Linux/Mac | -| bin | startup.sh | Initialization script folder | -| conf | config.properties | Test scenario configuration file | -| lib | - | Dependency library | -| LICENSE | - | License file | -| cli-benchmark.sh | - | One-click startup script | -| routine | - | Automatic execution of multiple test configurations | -| rep-benchmark.sh | - | Automatic execution of multiple test scripts | +### 2\.3 Package Structure + +Common files and directories in the installation package are listed below. + +|Name|Purpose| +|---|---| +|`benchmark.sh`|Startup script for Linux and macOS| +|`benchmark.bat`|Startup script for Windows| +|`conf/config.properties`|Test scenario configuration file| +|`lib/`|Runtime dependencies| +|`logs/`|Test logs, generated after the first run| +|`data/`|Dataset or persisted result directory, generated according to the work mode and persistence configuration| +### 2\.4 Running a Test +1. Start IoTDB -### 2.4 **Execution** **of** **Tests** +First, start the target IoTDB 2\.0\.x instance and verify that the client can access its service port. The default native interface port is `6667`. -1. Modify the configuration file (conf/config.properties) according to test requirements. For example, to test Apache IoTDB 2.0, set the following parameter: +2. Modify the configuration - ```Bash - DB_SWITCH=IoTDB-200-SESSION_BY_TABLET - ``` +- Edit `conf/config.properties`. -2. Ensure the target time-series database is running. +- Minimal connection configuration example: -3. Start IoT-benchmark to execute the test. Monitor the status of both the target database and IoT-benchmark during execution. +```Properties +DB_SWITCH=IoTDB-200-SESSION_BY_TABLET +IoTDB_DIALECT_MODE=tree +HOST=127.0.0.1 +PORT=6667 +USERNAME=root +PASSWORD=root +DB_NAME=test +``` + +- To use the table model, change the setting to: + +```Properties +IoTDB_DIALECT_MODE=table +``` + +- If REST is selected: -4. Upon completion, review the results and analyze the test process. +```Properties +DB_SWITCH=IoTDB-200-REST +REST_PORT=18080 +REST_AUTHORIZATION=Basic cm9vdDpyb290 +``` + +`REST_AUTHORIZATION` configures the `Basic Authentication` information for the REST interface. The example uses the username `root` and password `root`. + +3. Check RPC compression compatibility -### 2.5 **Results Interpretation** +IoT Benchmark 2\.0 enables IoTDB RPC compression by default: -All test log files are stored in the `logs` folder, while test results are saved in the `data/csvOutput` folder. For example, the following result matrix illustrates the test outcome: +```Properties +ENABLE_IOTDB_RPC_COMPRESSION=true +``` -![](/img/bm4.png) +This feature requires IoTDB 2\.0\.6 or later. When testing an IoTDB 2\.0\.x version earlier than 2\.0\.6, set: + +```Properties +ENABLE_IOTDB_RPC_COMPRESSION=false +``` -- **Result Matrix:** - - OkOperation: Number of successful operations. - - OkPoint: Number of successfully written points (for write operations) or successfully queried points (for query operations). - - FailOperation: Number of failed operations. - - FailPoint: Number of failed write points. -- **Latency (ms) Matrix:** - - AVG: Average operation latency. - - MIN: Minimum operation latency. - - Pn: Quantile values of the overall operation distribution (e.g., P25 represents the 25th percentile, or lower quartile). +Thrift compression is a separate configuration. If it is enabled: -## 3. **Main** **Parameters** +```Properties +ENABLE_THRIFT_COMPRESSION=true +``` -### 3.1 IoTDB Service Model +Also set the following in IoTDB's `iotdb-datanode.properties`: -The `IoTDB_DIALECT_MODE` parameter supports two modes: `tree` and `table`. The default value is `tree`. +```Properties +dn_rpc_thrift_compression_enable=true +``` -- **For IoTDB 2.0 and later versions**, the `IoTDB_DIALECT_MODE` parameter must be specified, and only one mode can be set for each IoTDB instance. -- **IoTDB_DIALECT_MODE = table:** - - The number of devices must be an integer multiple of the number of tables. - - The number of tables must be an integer multiple of the number of databases. +4. Start Benchmark -Key Parameters for IoTDB Service Model - -| **Parameter name** | **Type** | **Example** | **System description** | -| :---------------------- | :------- | :---------- |:--------------------------------------------------------------------| -| IoTDB_TABLE_NAME_PREFIX | String | `table_` | Prefix for table names when `IoTDB_DIALECT_MODE` is set to `table`. | -| DATA_CLIENT_NUMBER | Integer | `10` | Number of clients, must be an integer multiple of the table count. | -| SENSOR_NUMBER | Integer | `10` | Controls the number of attribute columns in the table mode. | -| IoTDB_TABLE_NUMBER | Integer | `1` | Specifies the number of tables when using the table mode. | - -### 3.2 **Working** **Mode** - -The `BENCHMARK_WORK_MODE` parameter supports four operational modes: - -1. **General Test Mode (****`testWithDefaultPath`****):** Configured via the `OPERATION_PROPORTION` parameter to support write-only, read-only, and mixed read-write operations. -2. **Data Generation Mode (****`generateDataMode`****):** Generates a reusable dataset, which is saved to `FILE_PATH` for subsequent use in the correctness write and correctness query modes. -3. **Single Database Correctness Write Mode (****`verificationWriteMode`****):** Verifies the correctness of dataset writing by writing the dataset generated in data generation mode. This mode supports only IoTDB v1.0+ and InfluxDB v1.x. -4. **Single Database Correctness Query Mode (****`verificationQueryMode`****):** Verifies the correctness of dataset queries after using the correctness write mode. This mode supports only IoTDB v1.0+ and InfluxDB v1.x. - -Mode configurations are shown in the following below: - -| **Mode name** | **BENCHMARK_WORK_MODE** | Description | Required Configuration | -| :------------------------------------- | :---------------------- | :------------------------------------------------------ | :------------------------- | -| General test mode | testWithDefaultPath | Supports multiple read and write mixed load operations. | `OPERATION_PROPORTION` | -| Generate data mode | generateDataMode | Generates datasets recognizable by IoT-benchmark. | `FILE_PATH` and `DATA_SET` | -| Single database correctness write mode | verificationWriteMode | Writes datasets for correctness verification. | `FILE_PATH` and `DATA_SET` | -| Single database correctness query mode | verificationQueryMode | Queries datasets to verify correctness. | `FILE_PATH` and `DATA_SET` | - -### 3.3 **Server** **Connection** **Information** - -Once the working mode is specified, the following parameters must be configured to inform IoT-benchmark of the target time-series database: - -| **Parameter** | **Type** | **Example** | D**escription** | -| :------------ | :------- | :---------------------------- | :----------------------------------------------------- | -| DB_SWITCH | String | `IoTDB-200-SESSION_BY_TABLET` | Specifies the type of time-series database under test. | -| HOST | String | `127.0.0.1` | Network address of the target time-series database. | -| PORT | Integer | `6667` | Network port of the target time-series database. | -| USERNAME | String | `root` | Login username for the time-series database. | -| PASSWORD | String | `root` | Password for the database login user. | -| DB_NAME | String | `test` | Name of the target time-series database. | -| TOKEN | String | - | Authentication token (used for InfluxDB 2.0). | - -### 3.4 **Write Scenario Parameters** - -| **Parameter** | **Type** | **Example** | D**escription** | -| :------------------------- | :-------------------- | :-------------------------- |:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| CLIENT_NUMBER | Integer | `100` | Total number of clients used for writing. | -| GROUP_NUMBER | Integer | `20` | Number of databases (only applicable for IoTDB). | -| DEVICE_NUMBER | Integer | `100` | Total number of devices. | -| SENSOR_NUMBER | Integer | `300` | Total number of sensors per device. (Control the number of attribute columns if you use the IoTDB table mode) | -| INSERT_DATATYPE_PROPORTION | String | `1:1:1:1:1:1:0:0:0:0` | Ratio of data types: `BOOLEAN:INT32:INT64:FLOAT:DOUBLE:TEXT:STRING:BLOB:TIMESTAMP:DATE`. | -| POINT_STEP | Integer | `1000` | Time interval (in ms) between generated data points. | -| OP_MIN_INTERVAL | Integer | `0` | Minimum execution interval for operations (ms): if the operation takes more than the value, the next one will be executed immediately, otherwise wait (OP_MIN_INTERVAL - actual execution time) ms; if it is 0, the parameter is not effective; if it is -1, its value is consistent with POINT_STEP | -| IS_OUT_OF_ORDER | Boolean | `false` | Specifies whether to write data out of order. | -| OUT_OF_ORDER_RATIO | Floating point number | `0.3` | Proportion of out-of-order data. | -| BATCH_SIZE_PER_WRITE | Integer | `1` | Number of data rows written per batch. | -| START_TIME | Time | `2022-10-30T00:00:00+08:00` | Start timestamp for data generation. | -| LOOP | Integer | `86400` | Total number of write operations: Each type of operation will be divided according to the proportion defined by `OPERATION_PROPORTION` | -| OPERATION_PROPORTION | Character | `1:0:0:0:0:0:0:0:0:0:0` | Ratio of operation types (write:Q1:Q2:...:Q10). | - -### 3.5 **Query Scenario Parameters** - -| Parameter | Type | Example | Description | -| :------------------- | :-------- | :---------------------- | :----------------------------------------------------------- | -| QUERY_DEVICE_NUM | Integer | `2` | Number of devices involved in each query statement. | -| QUERY_SENSOR_NUM | Integer | `2` | Number of sensors involved in each query statement. | -| QUERY_AGGREGATE_FUN | Character | `count` | Aggregate functions used in queries (`COUNT`, `AVG`, `SUM`, etc.). | -| STEP_SIZE | Integer | `1` | Time interval step for time filter conditions. | -| QUERY_INTERVAL | Integer | `250000` | Time interval between query start and end times. | -| QUERY_LOWER_VALUE | Integer | `-5` | Threshold for conditional queries (`WHERE value > QUERY_LOWER_VALUE`). | -| GROUP_BY_TIME_UNIT | Integer | `20000` | The size of the group in the `GROUP BY` statement | -| LOOP | Integer | `10` | Total number of query operations: Each type of operation will be divided according to the proportion defined by `OPERATION_PROPORTION` | -| OPERATION_PROPORTION | Character | `0:0:0:0:0:0:0:0:0:0:1` | Ratio of operation types (`write:Q1:Q2:...:Q10`). | - -### 3.6 **Query Types and Example SQL** - -| Number | Query Type | IoTDB Sample SQL | -| :----- | :----------------------------- | :----------------------------------------------------------- | -| Q1 | Precise Point Query | `select v1 from root.db.d1 where time = ?` | -| Q2 | Time Range Query | `select v1 from root.db.d1 where time > ? and time < ?` | -| Q3 | Time Range with Value Filter | `select v1 from root.db.d1 where time > ? and time < ? and v1 > ?` | -| Q4 | Time Range Aggregation Query | `select count(v1) from root.db.d1 where and time > ? and time < ?` | -| Q5 | Full-Time Range with Filtering | `select count(v1) from root.db.d1 where v1 > ?` | -| Q6 | Range Aggregation with Filter | `select count(v1) from root.db.d1 where v1 > ? and time > ? and time < ?` | -| Q7 | Time Grouping Aggregation | `select count(v1) from root.db.d1 group by ([?, ?), ?, ?)` | -| Q8 | Latest Point Query | `select last v1 from root.db.d1` | -| Q9 | Descending Range Query | `select v1 from root.sg.d1 where time > ? and time < ? order by time desc` | -| Q10 | Descending Range with Filter | `select v1 from root.sg.d1 where time > ? and time < ? and v1 > ? order by time desc` | - -### 3.7 **Test process and test result persistence** - -IoT-benchmark currently supports persisting the test process and test results through configuration parameters. - -| **Parameter** | **Type** | **Example** | D**escription** | -| :-------------------- | :------- | :---------- | :----------------------------------------------------------- | -| TEST_DATA_PERSISTENCE | String | `None` | Specifies the result persistence method. Options: `None`, `IoTDB`, `MySQL`, `CSV`. | -| RECORD_SPLIT | Boolean | `true` | Whether to split results into multiple records. (Not supported by IoTDB currently.) | -| RECORD_SPLIT_MAX_LINE | Integer | `10000000` | Maximum number of rows per record (10 million rows per database table or CSV file). | -| TEST_DATA_STORE_IP | String | `127.0.0.1` | IP address of the database for result storage. | -| TEST_DATA_STORE_PORT | Integer | `6667` | Port number of the output database. | -| TEST_DATA_STORE_DB | String | `result` | Name of the output database. | -| TEST_DATA_STORE_USER | String | `root` | Username for accessing the output database. | -| TEST_DATA_STORE_PW | String | `root` | Password for accessing the output database. | - -**Result Persistence Details** - -- **CSV Mode:** If `TEST_DATA_PERSISTENCE` is set to `CSV`, a `data` folder is generated in the IoT-benchmark root directory during and after test execution. This folder contains: - - `csv` folder: Records the test process. - - `csvOutput` folder: Stores the test results. -- **MySQL Mode:** If `TEST_DATA_PERSISTENCE` is set to `MySQL`, IoT-benchmark creates the following tables in the specified MySQL database: - - **Test Process Table:** - 1. Created before the test starts. - 2. Named as: `testWithDefaultPath___`. - - **Configuration Table:** - 1. Named `CONFIG`. - 2. Stores the test configuration. - 3. Created if it does not exist. - - **Final Result Table:** - 1. Named `FINAL_RESULT`. - 2. Stores the test results after test completion. - 3. Created if it does not exist. - -### 3.8 Automation Script - -#### One-Click Script Startup - -The `cli-benchmark.sh` script allows one-click startup of IoTDB, IoTDB Benchmark monitoring, and IoTDB Benchmark testing. However, please note that this script will clear all existing data in IoTDB during startup, so use it with caution. - -**Steps to Run:** - -1. Edit the `IOTDB_HOME` parameter in `cli-benchmark.sh` to the local IoTDB directory. -2. Start the test by running the following command: +On Linux or macOS: ```Bash -> ./cli-benchmark.sh +./benchmark.sh ``` -1. After the test completes: - 1. Check test-related logs in the `logs` folder. - 2. Check monitoring-related logs in the `server-logs` folder. +On Windows: + +```Plain Text +benchmark.bat +``` -#### Automatic Execution of Multiple Tests +During the test, progress is periodically printed to the terminal. When the test completes, the main configurations, execution time, result matrix, and latency matrix are displayed. -Single tests are often insufficient without comparative results. Therefore, IoT-benchmark provides an interface for executing multiple tests in sequence. +### 2\.5 Understanding the Results -1. **Routine Configuration:** Each line in the `routine` file specifies the parameters that change for each test. For example: +Test execution information is written to the `logs` folder in the installation directory. Whether CSV files are generated or results are written to a result database depends on parameters such as `TEST_DATA_PERSISTENCE`. - ```Plain - LOOP=10 DEVICE_NUMBER=100 TEST - LOOP=20 DEVICE_NUMBER=50 TEST - LOOP=50 DEVICE_NUMBER=20 TEST - ``` +1. Result matrix -In this example, three tests will run sequentially with `LOOP` values of 10, 20, and 50. +The result matrix reports the following metrics by operation type: -Then the test process with 3 LOOP parameters of 10, 20, and 50 is executed in sequence. +|Metric|Description| +|---|---| +|`okOperation`|Number of successfully executed requests or SQL statements| +|`okPoint`|Number of successfully written data points, or data points successfully returned by queries| +|`failOperation`|Number of failed requests or SQL statements| +|`failPoint`|Number of data points that failed to be written; usually 0 for query operations| +|`throughput`|Throughput, usually equal to `okPoint / Test elapsed time`| -**Important Notes:** +The main operation names in the output include: -- Multiple parameters can be changed in each test using the format: +- `INGESTION` - ```Bash - LOOP=20 DEVICE_NUMBER=10 TEST - ``` +- `PRECISE_POINT` -- Avoid unnecessary spaces. +- `TIME_RANGE` -- The `TEST` keyword marks the start of a new test. +- `VALUE_RANGE` -- Changed parameters persist across subsequent tests unless explicitly reset. +- `AGG_RANGE` -2. **Start the Test:** After configuring the `routine` file, start multi-test execution using the following command +- `AGG_VALUE` - ```Bash - > ./rep-benchmark.sh - ``` +- `AGG_RANGE_VALUE` -Test results will be displayed in the terminal. +- `GROUP_BY` -**Important Notes:** +- `LATEST_POINT` -- Closing the terminal or losing the client connection will terminate the test process. +- `RANGE_QUERY_DESC` -- To run the test as a background daemon, execute: +- `VALUE_RANGE_QUERY_DESC` - ```Bash - > ./rep-benchmark.sh > /dev/null 2>&1 & - ``` +- `GROUP_BY_DESC` -- To monitor progress, check the logs: +- `SET_OP_QUERY` - ```Bash - > cd ./logs - > tail -f log_info.log - ``` +2. Latency matrix -## 4. Test Example +The latency matrix is measured in milliseconds. Common fields are listed below. -This example demonstrates how to configure and run an IoT-benchmark test with IoTDB 2.0 using the table mode for writing and querying. +|Metric|Description| +|---|---| +|`AVG`|Average latency| +|`MIN`|Minimum latency| +|`P10`, `P25`, `MEDIAN`|Lower percentiles and median latency| +|`P75`, `P90`, `P95`|Higher-percentile latency| +|`P99`, `P999`|Tail latency| +|`MAX`|Maximum latency| +|`SLOWEST_THREAD`|Largest cumulative operation time among client threads| -```Properties +The test results also report metadata creation time and `Test elapsed time`, which excludes metadata creation. When comparing tests, ensure that the hardware, data volume, number of clients, compression configuration, and cache state are consistent across runs. + +3. Output example + +After the test completes, the terminal displays the main configurations, execution time, result matrix, and latency matrix. The following is truncated output from a write-only test: + +```Plain Text ----------------------Main Configurations---------------------- BENCHMARK_WORK_MODE=testWithDefaultPath -IoTDB_DIALECT_MODE=TABLE DB_SWITCH=IoTDB-200-SESSION_BY_TABLET +HOST=[127.0.0.1] + +GROUP_NUMBER=10 +DEVICE_NUMBER=50 +SENSOR_NUMBER=500 +SCHEMA_CLIENT_NUMBER=20 +DATA_CLIENT_NUMBER=20 + +OPERATION_PROPORTION=1:0:0:0:0:0:0:0:0:0:0:0:0 +LOOP=10000 +BATCH_SIZE_PER_WRITE=100 +--------------------------------------------------------------- + +Create schema cost 0.30 second +Test elapsed time (not include schema creation): 1238.79 second + +----------------------------------------------------------Result Matrix---------------------------------------------------------- +Operation okOperation okPoint failOperation failPoint throughput(point/s) +INGESTION 500000 25000000000 0 0 20180954.09 +--------------------------------------------------------------------------------------------------------------------------------- + +----------------------------------------------------------Latency (ms) Matrix---------------------------------------------------- +Operation AVG MIN P10 P25 MEDIAN P75 P90 P95 P99 P999 MAX SLOWEST_THREAD +INGESTION 37.78 1.67 2.02 2.29 2.86 4.14 5.62 7.43 759.69 5799.89 8309.40 1227561.44 +--------------------------------------------------------------------------------------------------------------------------------- +``` + +The output shows that: + +- Metadata creation took `0.30` seconds, and the actual test took `1238.79` seconds. + +- A total of `500000` write operations successfully wrote `25000000000` data points. + +- Both `failOperation` and `failPoint` are `0`, indicating that no write failures were recorded during this test. + +- Write throughput was `20180954.09` points per second. + +- Average latency was `37.78` ms, P95 latency was `7.43` ms, P99 latency was `759.69` ms, and maximum latency was `8309.40` ms. + +- `SLOWEST_THREAD` is the cumulative operation time of the slowest client thread, not the latency of a single request. + +The values in this example are only intended to demonstrate the output format. Actual results depend on hardware resources, network conditions, IoTDB configuration, data scale, and test parameters. + +## 3\. Main Parameters + +### 3\.1 IoTDB Data Model + +IoTDB 2\.0\.x supports the tree model and the table model. Select the model using: + +```Properties +IoTDB_DIALECT_MODE=tree +``` + +or: + +```Properties +IoTDB_DIALECT_MODE=table +``` + +The following constraints apply: + +- An IoTDB instance can use only one SQL dialect in a single test. + +- The tree model requires `DEVICE_NUMBER >= GROUP_NUMBER`. + +- In the table model, the number of devices must be a multiple of the number of tables, and the number of tables must be a multiple of the number of databases. + +- In the table model, the number of data clients must be a multiple of the number of tables. + +Common model parameters: + +|Parameter|Example|Description| +|---|---|---| +|`IoTDB_DIALECT_MODE`|`tree`|`tree` or `table`| +|`GROUP_NUMBER`|`1`|Number of databases; corresponds to the number of databases in the tree model| +|`IoTDB_TABLE_NUMBER`|`1`|Number of tables created in the table model| +|`IoTDB_TABLE_NAME_PREFIX`|`table_`|Table name prefix| +|`TABLE_TIME_COLUMN`|`time`|Name of the time column in the table model| +|`IoTDB_TABLE_WRITABLE_VIEW`|`false`|Whether to create and use writable views| + +### 3\.2 Work Modes + +Use `BENCHMARK_WORK_MODE` to select a work mode. + +|Mode|Configuration Value|Description| +|---|---|---| +|Regular test mode|`testWithDefaultPath`|Runs a write, query, or mixed workload| +|Data generation mode|`generateDataMode`|Saves the dataset generated by Benchmark to `FILE_PATH`| +|Correctness write mode|`verificationWriteMode`|Loads a dataset from `FILE_PATH` and writes it to the database| +|Correctness query mode|`verificationQueryMode`|Loads a dataset and compares it with database query results| + +Example: + +```Properties +BENCHMARK_WORK_MODE=testWithDefaultPath +``` + +Before using the correctness write and query modes, use `generateDataMode` to generate a reusable dataset. + +### 3\.3 Server Connection Information + +|Parameter|Example|Description| +|---|---|---| +|`DB_SWITCH`|`IoTDB-200-SESSION_BY_TABLET`|Database version and connection method| +|`HOST`|`127.0.0.1`|IoTDB address; separate multiple addresses with commas| +|`PORT`|`6667`|Native service port; the number of ports must match the number of `HOST` entries| +|`USERNAME`|`root`|Username| +|`PASSWORD`|`root`|Password| +|`DB_NAME`|`test`|Name of the database used for testing| +|`REST_PORT`|`18080`|REST service port| +|`REST_AUTHORIZATION`|`Basic cm9vdDpyb290`|REST authorization header| +|`ENABLE_AUTO_FETCH`|`false`|Whether Session automatically refreshes the DataNode list| + +Data cleanup parameters: + +```Properties +IS_DELETE_DATA=false +INIT_WAIT_TIME=1000 +``` + +> Warning: `IS_DELETE_DATA=true` clears test data from the target database before the test starts. Enable it only in a dedicated test environment, and verify `HOST`, `PORT`, `DB_NAME`, and account permissions before execution. +> +> + +### 3\.4 Write Scenarios + +1. Data scale and clients + +|Parameter|Example|Description| +|---|---|---| +|`DEVICE_NUMBER`|`100`|Total number of devices| +|`SENSOR_NUMBER`|`10`|Number of measurements per device; number of measurement columns in the table model| +|`GROUP_NUMBER`|`1`|Number of IoTDB databases| +|`SCHEMA_CLIENT_NUMBER`|`5`|Number of clients that register metadata| +|`DATA_CLIENT_NUMBER`|`10`|Number of clients that perform data reads and writes| +|`IS_CLIENT_BIND`|`true`|Whether devices are bound to clients| +|`REAL_INSERT_RATE`|`1.0`|Proportion of devices that actually participate in writes| +|`IS_SENSOR_TS_ALIGNMENT`|`true`|Whether measurement timestamps under the same device are aligned| + +2. Batch writes + +|Parameter|Example|Description| +|---|---|---| +|`BATCH_SIZE_PER_WRITE`|`100`|Number of data rows written per device in each batch| +|`DEVICE_NUM_PER_WRITE`|`1`|Number of devices involved in each batch write| +|`CREATE_SCHEMA`|`true`|Whether to create metadata before writing| +|`START_TIME`|`2022-01-01T00:00:00+08:00`|Start time for generated data| + +The number of data points in a single batch is: + +```Plain Text +DEVICE_NUM_PER_WRITE × SENSOR_NUMBER × BATCH_SIZE_PER_WRITE +``` + +`DEVICE_NUM_PER_WRITE` must evenly divide the number of devices assigned to a single data client. In the table model, the divisibility constraints among the number of devices, number of tables, and devices per batch must also be satisfied. + +3. Write pacing + +|Parameter|Example|Description| +|---|---|---| +|`POINT_STEP`|`5000`|Fixed interval between adjacent generated timestamps| +|`OP_MIN_INTERVAL`|`0`|Minimum interval for each loop, in ms| +|`OP_MIN_INTERVAL_RANDOM`|`false`|Whether to randomly select an interval from `[0, OP_MIN_INTERVAL)`| +|`INTERVAL_BETWEEN_WRITE_BATCH`|`0`|Minimum interval between adjacent batches in the same loop, in ms| +|`TIMESTAMP_PRECISION`|`ms`|Timestamp precision| + +Special values of `OP_MIN_INTERVAL`: + +- `0`: does not limit the loop interval. + +- `-1`: uses `POINT_STEP` as the minimum interval. + +- Positive integer: if the current loop takes less than this value, waits for the remaining time. + +4. Out-of-order writes + +```Properties +IS_OUT_OF_ORDER=false +OUT_OF_ORDER_MODE=POISSON +OUT_OF_ORDER_RATIO=0.5 +IS_REGULAR_FREQUENCY=true +``` + +Supported out-of-order modes include: + +- `POISSON`: generates out-of-order timestamps according to a Poisson distribution. + +- `BATCH`: generates out-of-order data in batches. + +5. Data types + +```Properties +INSERT_DATATYPE_PROPORTION=1:1:1:1:1:1:0:0:0:0:0 +``` + +The order of the entries is: + +```Plain Text +BOOLEAN:INT32:INT64:FLOAT:DOUBLE:TEXT:STRING:BLOB:TIMESTAMP:DATE:OBJECT +``` + +Each value represents the proportion of the corresponding data type. + +### 3\.5 Query Scenarios + +|Parameter|Example|Description| +|---|---|---| +|`QUERY_DEVICE_NUM`|`1`|Number of devices involved in each query| +|`QUERY_SENSOR_NUM`|`1`|Number of measurements involved in each query| +|`QUERY_AGGREGATE_FUN`|`count`|Aggregation function| +|`STEP_SIZE`|`0`|Step by which the query start time changes, in units of `POINT_STEP`| +|`QUERY_INTERVAL`|`250000`|Interval between query start and end times| +|`QUERY_LOWER_VALUE`|`-5`|Lower bound of the value filter| +|`GROUP_BY_TIME_UNIT`|`20000`|Group By window size| +|`QUERY_SET_OP_TYPE`|`union`|Set operation type| +|`QUERY_SET_OP_NUM`|`2`|Number of sub-sets in a set query; at least 2| +|`IS_RECENT_QUERY`|`false`|Whether to prioritize recently written data in mixed scenarios| +|`ENABLE_FIXED_QUERY`|`false`|Whether all query threads use the same device and measurement combinations| +|`RESULT_ROW_LIMIT`|`-1`|Query result row limit; `-1` means no limit| +|`ALIGN_BY_DEVICE`|`false`|Whether to use Align By Device| + +### 3\.6 Operation Proportions + +`OPERATION_PROPORTION` defines the proportions of writes and different query types. It contains 13 entries: + +```Plain Text +Write:Q1:Q2:Q3:Q4:Q5:Q6:Q7:Q8:Q9:Q10:Q11:Q12 +``` + +For example, write-only: + +```Properties +OPERATION_PROPORTION=1:0:0:0:0:0:0:0:0:0:0:0:0 +``` + +Precise point query only: + +```Properties +OPERATION_PROPORTION=0:1:0:0:0:0:0:0:0:0:0:0:0 +``` + +The operation types are listed below. + +|Number|Operation Type|Description| +|---|---|---| +|Write|Data write|Generates and writes data according to the current write configuration| +|Q1|Precise point query|Queries specified measurements by timestamp and device| +|Q2|Time range query|Range query restricted only by start and end times| +|Q3|Range query with value filter|Includes both time and value filter conditions| +|Q4|Aggregation query with time filter|Performs aggregation within a time range| +|Q5|Aggregation query with value filter|Filters by value and aggregates over the full time range| +|Q6|Aggregation query with time and value filters|Includes both time and value filter conditions| +|Q7|Time-grouped aggregation query|Group By query| +|Q8|Latest point query|Queries the latest data point of a device| +|Q9|Descending time range query|Returns range query results in descending time order| +|Q10|Descending range query with value filter|Filters by value and returns results in descending time order| +|Q11|Descending time-grouped aggregation query|Descending Group By query| +|Q12|Set operation query|Set operations such as `union`, `intersect`, or `except`| + +Q12 is supported only by the IoTDB 2\.0 table model. Each subquery in a set operation is a range query. + +### 3\.7 Test Process and Result Persistence + +```Properties +TEST_DATA_PERSISTENCE=None +``` + +Supported values include: + +- `None`: does not write the test process to an external persistence medium. + +- `CSV`: writes to CSV files. + +- `MySQL`: writes to MySQL. + +- `IoTDB`: writes to a specified IoTDB instance. + +Common parameters: + +|Parameter|Example|Description| +|---|---|---| +|`TEST_DATA_PERSISTENCE`|`None`|Persistence method| +|`RECORD_SPLIT`|`true`|Whether to split results into multiple records| +|`RECORD_SPLIT_MAX_LINE`|`10000000`|Maximum number of records in a single table or file| +|`TEST_DATA_STORE_IP`|`127.0.0.1`|Result database address| +|`TEST_DATA_STORE_PORT`|`6667`|Result database port| +|`TEST_DATA_STORE_DB`|`result`|Result database name| +|`TEST_DATA_STORE_USER`|`root`|Result database username| +|`TEST_DATA_STORE_PW`|`root`|Result database password| +|`REMARK`|`write_test`|Test note used to distinguish different tests| +|`CSV_OUTPUT`|`true`|Whether to write final results to CSV| + +When the persistence method is CSV, records are generated in the `data` directory after execution; test results are usually located in `data/csvOutput`. Test logs are always written to `logs`, regardless of whether persistence is enabled. + +The following parameters control log output frequency: + +```Properties +IS_QUIET_MODE=true +LOG_PRINT_INTERVAL=5 +RESULT_PRINT_INTERVAL=3600 +``` + +### 3\.8 Automation and Cluster Testing + +1. Limit test duration + +```Properties +TEST_MAX_TIME=3600000 +``` + +The unit is milliseconds. A value of `0` means no limit. This parameter does not include the time spent pre-registering metadata. + +2. Multi-Benchmark cluster load testing + +Use the same overall data scale configuration on multiple client machines, and set: + +```Properties +BENCHMARK_CLUSTER=true +BENCHMARK_INDEX=0 +``` + +Each Benchmark instance must use a different `BENCHMARK_INDEX`, such as `0`, `1`, and `2` in sequence. All clients should use consistent database connections, data scales, operation proportions, and other configurations. + +3. Dual-write testing + +IoT Benchmark can write the same data to two different databases for comparison: + +```Properties +IS_DOUBLE_WRITE=true +ANOTHER_DB_SWITCH= +ANOTHER_HOST=127.0.0.1 +ANOTHER_PORT=6667 +ANOTHER_USERNAME=root +ANOTHER_PASSWORD=root +ANOTHER_DB_NAME=test +``` + +Dual-write mode does not support comparisons between different versions of the same database, or direct comparisons between the IoTDB tree model and table model. + +## 4\. Examples + +This section uses small datasets to demonstrate the basic process. For formal performance testing, increase the number of devices, measurements, clients, and loops according to the target business model, and perform multiple warm-up and repeated test runs. + +### 4\.1 Write Test Example + +Test objective: use 10 data clients to simulate 100 devices, each containing 10 measurements, and perform a write-only test. + +Example configuration: + +```Properties +# Database connection +DB_SWITCH=IoTDB-200-SESSION_BY_TABLET +IoTDB_DIALECT_MODE=tree +HOST=127.0.0.1 +PORT=6667 +USERNAME=root +PASSWORD=root +DB_NAME=test + +# Safety setting: do not automatically delete existing data by default +IS_DELETE_DATA=false + +# Work mode +BENCHMARK_WORK_MODE=testWithDefaultPath +OPERATION_PROPORTION=1:0:0:0:0:0:0:0:0:0:0:0:0 + +# Data scale GROUP_NUMBER=1 -IoTDB_TABLE_NUMBER=1 -DEVICE_NUMBER=60 -REAL_INSERT_RATE=1.0 +DEVICE_NUMBER=100 SENSOR_NUMBER=10 -OPERATION_PROPORTION=1:0:0:0:0:0:0:0:0:0:0:0 -SCHEMA_CLIENT_NUMBER=10 +SCHEMA_CLIENT_NUMBER=5 DATA_CLIENT_NUMBER=10 -LOOP=10 +IS_SENSOR_TS_ALIGNMENT=true + +# Write configuration +CREATE_SCHEMA=true BATCH_SIZE_PER_WRITE=10 DEVICE_NUM_PER_WRITE=1 -START_TIME=2025-01-01T00:00:00+08:00 +LOOP=100 POINT_STEP=1000 -INSERT_DATATYPE_PROPORTION=1:1:1:1:1:1:0:0:0:0 -VECTOR=true +OP_MIN_INTERVAL=0 +START_TIME=2026-01-01T00:00:00+08:00 +INSERT_DATATYPE_PROPORTION=1:1:1:1:1:1:0:0:0:0:0 + +# Can be enabled for IoTDB 2.0.6 and later +ENABLE_IOTDB_RPC_COMPRESSION=true + +# Output +TEST_DATA_PERSISTENCE=None +CSV_OUTPUT=true +REMARK=iotdb_2_write_test +``` + +Start the test: + +```Bash +./benchmark.sh ``` -**Execution Steps:** +After completion, focus on: -1. Ensure the target database (IoTDB 2.0) is running. -2. Start IoT-benchmark using the configured parameters. -3. Upon completion, view the test results. +- `okPoint` and `failPoint` for `INGESTION`. -```Shell -Create schema cost 0.88 second -Test elapsed time (not include schema creation): 4.60 second -----------------------------------------------------------Result Matrix---------------------------------------------------------- -Operation okOperation okPoint failOperation failPoint throughput(point/s) -INGESTION 600 60000 0 0 13054.42 -PRECISE_POINT 0 0 0 0 0.00 -TIME_RANGE 0 0 0 0 0.00 -VALUE_RANGE 0 0 0 0 0.00 -AGG_RANGE 0 0 0 0 0.00 -AGG_VALUE 0 0 0 0 0.00 -AGG_RANGE_VALUE 0 0 0 0 0.00 -GROUP_BY 0 0 0 0 0.00 -LATEST_POINT 0 0 0 0 0.00 -RANGE_QUERY_DESC 0 0 0 0 0.00 -VALUE_RANGE_QUERY_DESC 0 0 0 0 0.00 -GROUP_BY_DESC 0 0 0 0 0.00 ---------------------------------------------------------------------------------------------------------------------------------- +- `throughput`. + +- `AVG`, `P95`, `P99`, and `MAX` latency. + +- Whether connection timeouts, write failures, or server exceptions are present in the logs. + +### 4\.2 Query Test Example + +Before running a query test, ensure that the target database contains data matching the query configuration. It is recommended to reuse data generated by the write test and disable automatic data deletion and metadata creation. + +The following example runs multiple query types: + +```Properties +DB_SWITCH=IoTDB-200-SESSION_BY_TABLET +IoTDB_DIALECT_MODE=tree +HOST=127.0.0.1 +PORT=6667 +USERNAME=root +PASSWORD=root +DB_NAME=test + +IS_DELETE_DATA=false +CREATE_SCHEMA=false +BENCHMARK_WORK_MODE=testWithDefaultPath + +GROUP_NUMBER=1 +DEVICE_NUMBER=100 +SENSOR_NUMBER=10 +SCHEMA_CLIENT_NUMBER=1 +DATA_CLIENT_NUMBER=10 + +# Do not perform writes; Q1-Q11 have equal proportions; the tree model does not use Q12 +OPERATION_PROPORTION=0:1:1:1:1:1:1:1:1:1:1:1:0 +LOOP=100 + +QUERY_DEVICE_NUM=2 +QUERY_SENSOR_NUM=2 +QUERY_AGGREGATE_FUN=count +STEP_SIZE=1 +QUERY_INTERVAL=250000 +QUERY_LOWER_VALUE=-5 +GROUP_BY_TIME_UNIT=20000 +``` + +To test set queries in the table model, switch the dialect to `table` and assign a proportion to Q12: + +```Properties +IoTDB_DIALECT_MODE=table +OPERATION_PROPORTION=0:0:0:0:0:0:0:0:0:0:0:0:1 +QUERY_SET_OP_TYPE=union +QUERY_SET_OP_NUM=2 +``` + +### 4\.3 Other Configuration Examples + +1. Simulate an actual write rate + +Set the minimum interval of each loop to the data timestamp interval: + +```Properties +POINT_STEP=1000 +OP_MIN_INTERVAL=-1 +``` + +To distribute write requests evenly within a loop, use: + +```Properties +INTERVAL_BETWEEN_WRITE_BATCH=100 +``` + +2. Specify test duration + +Test for one hour: + +```Properties +TEST_MAX_TIME=3600000 +``` + +Ensure that `LOOP` is sufficiently large; otherwise, the test may end when the loop count is exhausted. + +3. Control generated data patterns + +```Properties +LINE_RATIO=1 +SIN_RATIO=1 +SQUARE_RATIO=1 +RANDOM_RATIO=1 +CONSTANT_RATIO=1 +DATA_SEED=666 +STRING_LENGTH=10 +DOUBLE_LENGTH=2 +``` + +Fixing `DATA_SEED` helps generate reproducible data across multiple test runs. + +## 5\. References ---------------------------------------------------------------------------Latency (ms) Matrix-------------------------------------------------------------------------- -Operation AVG MIN P10 P25 MEDIAN P75 P90 P95 P99 P999 MAX SLOWEST_THREAD -INGESTION 41.77 0.95 1.41 2.27 6.76 24.14 63.42 127.18 1260.92 1265.72 1265.49 2581.91 -PRECISE_POINT 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 -TIME_RANGE 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 -VALUE_RANGE 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 -AGG_RANGE 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 -AGG_VALUE 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 -AGG_RANGE_VALUE 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 -GROUP_BY 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 -LATEST_POINT 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 -RANGE_QUERY_DESC 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 -VALUE_RANGE_QUERY_DESC 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 -GROUP_BY_DESC 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ -``` \ No newline at end of file +- [IoT Benchmark Documentation](https://github.com/thulab/iot-benchmark/tree/master/docs) diff --git a/src/UserGuide/latest/Tools-System/Benchmark.md b/src/UserGuide/latest/Tools-System/Benchmark.md index 19fbdf1bd..3ca07c95e 100644 --- a/src/UserGuide/latest/Tools-System/Benchmark.md +++ b/src/UserGuide/latest/Tools-System/Benchmark.md @@ -21,470 +21,796 @@ # Benchmark Tool -## 1. **Basic Overview** +IoT Benchmark is a benchmark testing tool for time-series databases and real-time databases in Industrial Internet of Things (IIoT) scenarios. This manual introduces the tool's main capabilities, supported databases, and basic usage. The installation, configuration, and test examples primarily use IoTDB 2\.0\.x and cover both the tree model and the table model. -IoT-benchmark is a time-series database benchmarking tool developed in Java for big data environments. It was developed and open-sourced by the School of Software, Tsinghua University. The tool is user-friendly, supports various write and query methods, allows storing test information and results for further queries or analysis, and integrates with Tableau for visualizing test results. +![](/img/benchmark-%20English2.png) -Figure 1-1 illustrates the test benchmark process and its extended functionalities, all of which can be streamlined by IoT-benchmark. It supports a variety of workloads, including write-only, read-only, and mixed write-and-read operations. Additionally, it offers software and hardware system monitoring, performance metric measurement, automated database initialization, test data analysis, and system parameter optimization. +## 1\. Basic Overview -![](/img/benchmark-English1.png) +IoT Benchmark can generate periodic time-series data based on configuration, perform writes and queries against a database, and collect metrics such as throughput, latency, and success rate. Its main capabilities include: -Figure 1-1 *IoT-benchmark Test Benchmark Process* +- Cross-platform operation: supports Linux, macOS, and Windows. -IoT-benchmark adopts the modular design concept of the YCSB test tool, which separates workload generation, performance measurement, and database interface components. Its modular structure is illustrated in Figure 1-2. Unlike YCSB-based testing tools, IoT-benchmark introduces a system monitoring module that supports the persistence of both test data and system metrics. It also includes load-testing functionalities specifically designed for time-series data scenarios, such as batch writes and multiple out-of-order data insertion modes for IoT environments. +- Multiple workloads: supports write-only, query-only, and mixed read/write workloads. -![](/img/benchmark-%20English2.png) +- Dataset generation: generated data can be saved to disk for repeated testing. + +- Correctness verification: datasets can be loaded from disk to verify write and query correctness. + +- Multiple database support: tests can be performed against various time-series databases and real-time databases. IoTDB 2\.0\.x supports JDBC, REST, and multiple Session write methods. + +- Result persistence: test processes and results can be saved to files, CSV, MySQL, or IoTDB. + +- Test extensions: supports scenarios such as out-of-order writes, batch writes, cluster load testing, dual writes for comparison, and result visualization. + +### 1\.1 Supported Databases, Versions, and Access Methods + +IoT Benchmark supports the following databases and versions. During testing, use `DB_SWITCH` to select the corresponding database, version, and connection or write method. + +|Database|Supported Version|`DB_SWITCH`| +|---|---|---| +|IoTDB|v1\.x|`IoTDB-130-JDBC`, `IoTDB-130-REST`, `IoTDB-130-SESSION_BY_TABLET`, `IoTDB-130-SESSION_BY_RECORD`, `IoTDB-130-SESSION_BY_RECORDS`| +|IoTDB|v2\.x|`IoTDB-200-JDBC`, `IoTDB-200-REST`, `IoTDB-200-SESSION_BY_TABLET`, `IoTDB-200-SESSION_BY_RECORD`, `IoTDB-200-SESSION_BY_RECORDS`| +|InfluxDB|v1\.x|`InfluxDB`| +|InfluxDB|v2\.x|`InfluxDB-2.x`| +|QuestDB|v6\.0\.7|`QuestDB`| +|Microsoft SQL Server|2016 SP2|`MSSQLSERVER`| +|VictoriaMetrics|v1\.64\.0|`VictoriaMetrics`| +|SQLite|—|`SQLite`| +|OpenTSDB|2\.4\.1|`OpenTSDB`| +|KairosDB|—|`KairosDB`| +|TimescaleDB|—|`TimescaleDB`| +|TimescaleDB Cluster|Cluster|`TimescaleDB-Cluster`| +|TDengine|2\.2\.0\.2|`TDengine`| +|TDengine|3\.0\.1|`TDengine-3`| +|DolphinDB|v2\.x|`DolphinDB-2-MTW`, `DolphinDB-2-PTA`| +|DolphinDB|v3\.x|`DolphinDB-3-MTW`, `DolphinDB-3-PTA`| +|CnosDB|—|`CnosDB`| + +Notes: + +- IoTDB access methods include JDBC, REST, Session by Tablet, Session by Record, and Session by Records. + +- For DolphinDB, `MTW` means `MultithreadedTableWriter`, which buffers writes by row; `PTA` means `PartitionedTableAppender`, which appends an entire table in columnar batches. + +- Database versions, drivers, and servers must be compatible with one another. When using other databases, configure the corresponding connection and extension parameters. + +### 1\.2 IoTDB 2\.0\.x Access Methods + +The following installation, configuration, and examples primarily use IoTDB 2\.0\.x. The supported access methods are listed below. + +|Access Method|`DB_SWITCH`|Description| +|---|---|---| +|JDBC|`IoTDB-200-JDBC`|Performs writes and queries through JDBC| +|REST|`IoTDB-200-REST`|Performs tests through the IoTDB REST interface| +|Session by Tablet|`IoTDB-200-SESSION_BY_TABLET`|Uses Tablet for batch writes| +|Session by Record|`IoTDB-200-SESSION_BY_RECORD`|Writes records one at a time| +|Session by Records|`IoTDB-200-SESSION_BY_RECORDS`|Writes multiple records in batches| + +`IoTDB-200-SESSION_BY_TABLET` is suitable for batch write tests. + +## 2\. Installation and Execution + +This chapter uses IoTDB 2\.0\.x as the database under test. + +### 2\.1 Prerequisites + +1. Before using IoT Benchmark, prepare: -Figure 1-2 *IoT-benchmark Modular Design* + - Java 17. -**Supported Databases** + - Maven. -Currently, IoT-benchmark supports the following time series databases, versions and connection methods: + - An installed and running IoTDB 2\.0\.x instance. -| Database | Version | Connection mmethod | -| :-------------- |:-----------| :------------------------------------------------------- | -| IoTDB | v1.x v2.x | JDBC, SessionByTablet, SessionByRecord, SessionByRecords | -| InfluxDB | v1.x v2.x | SDK | -| TimescaleDB | -- | JDBC | -| OpenTSDB | -- | HTTP Request | -| QuestDB | v6.0.7 | JDBC | -| TDengine | v2.2.0.2 | JDBC | -| VictoriaMetrics | v1.64.0 | HTTP Request | -| KairosDB | -- | HTTP Request | + - Sufficient client CPU, memory, disk, and network resources. +2. Environment notes: -## 2. **Installation and Operation** + - Linux or macOS is recommended for running tests. -### 2.1 **Prerequisites** + - On Windows, use `benchmark.bat` in the installation package root directory to start a test. -1. Java 8 -2. Maven 3.6+ -3. The corresponding appropriate version of the database, such as Apache IoTDB 2.0 + - On Linux and macOS, use `benchmark.sh` to start a test. -### 2.2 **How to Obtain** + - Some system information collection capabilities in CSV recording mode are supported only on Linux. -- **B****inary package****:** Visit https://github.com/thulab/iot-benchmark/releases to download the installation package. Extract the compressed file into a desired folder for use. +> Note: Do not deploy IoT Benchmark and the IoTDB instance under test in environments where they compete for resources. For formal performance testing, use separate servers and stop unrelated services. +> +> -- **Source Code** **Compilation (for** **Apache** **IoTDB 2.0 testing):** +### 2\.2 Obtaining IoT Benchmark - - **Compile the latest IoTDB Session package:** Download the IoTDB source code from https://github.com/apache/iotdb/tree/rc/2.0.5 and run the following command in the root directory to compile the latest IoTDB Session package: +1. Download a release package - ```Bash - mvn clean package install -pl session -am -DskipTests - ``` +Download a release package matching the test target from [IoT Benchmark Releases](https://github.com/thulab/iot-benchmark/releases), and extract it for use. - - **Compile the IoT-benchmark test package:** Download the source code from https://github.com/thulab/iot-benchmark and run the following command in the root directory to compile the Apache IoTDB 2.0 test package:. +2. Build from source - ```Bash - mvn clean package install -pl iotdb-2.0 -am -DskipTests - ``` +Clone the repository: - - The compiled test package will be located at: +```Bash +git clone https://github.com/thulab/iot-benchmark.git +cd iot-benchmark +``` - ```Bash - ./iotdb-2.0/target/iotdb-2.0-0.0.1/iotdb-2.0-0.0.1 - ``` +Run the following command in the project root directory: -### 2.3 **Test Package Structure** +```Bash +mvn clean package -Dmaven.test.skip=true +``` -The directory structure of the test package is shown below. The test configuration file is `conf/config.properties`, and the test startup scripts are `benchmark.sh` (Linux & MacOS) and `benchmark.bat` (Windows). The detailed usage of the files is shown in the table below. +After the build is complete, the IoTDB 2\.0\.x package is located at: -```Shell --rw-r--r--. 1 root root 2881 Jan 10 01:36 benchmark.bat --rwxr-xr-x. 1 root root 314 Jan 10 01:36 benchmark.sh -drwxr-xr-x. 2 root root 24 Jan 10 01:36 bin --rwxr-xr-x. 1 root root 1140 Jan 10 01:36 cli-benchmark.sh -drwxr-xr-x. 2 root root 107 Jan 10 01:36 conf -drwxr-xr-x. 2 root root 4096 Jan 10 01:38 lib --rw-r--r--. 1 root root 11357 Jan 10 01:36 LICENSE --rwxr-xr-x. 1 root root 939 Jan 10 01:36 rep-benchmark.sh --rw-r--r--. 1 root root 14 Jan 10 01:36 routine +```Plain Text +iotdb-2.0/target/iot-benchmark-iotdb-2.0/iot-benchmark-iotdb-2.0 ``` -| Name | File | Usage | -| :--------------- | :---------------- | :-------------------------------------------------- | -| benchmark.bat | - | Startup script on Windows | -| benchmark.sh | - | Startup script on Linux/Mac | -| bin | startup.sh | Initialization script folder | -| conf | config.properties | Test scenario configuration file | -| lib | - | Dependency library | -| LICENSE | - | License file | -| cli-benchmark.sh | - | One-click startup script | -| routine | - | Automatic execution of multiple test configurations | -| rep-benchmark.sh | - | Automatic execution of multiple test scripts | +Enter the installation directory: +```Bash +cd iotdb-2.0/target/iot-benchmark-iotdb-2.0/iot-benchmark-iotdb-2.0 +``` +### 2\.3 Package Structure -### 2.4 **Execution** **of** **Tests** +Common files and directories in the installation package are listed below. -1. Modify the configuration file (conf/config.properties) according to test requirements. For example, to test Apache IoTDB 2.0, set the following parameter: +|Name|Purpose| +|---|---| +|`benchmark.sh`|Startup script for Linux and macOS| +|`benchmark.bat`|Startup script for Windows| +|`conf/config.properties`|Test scenario configuration file| +|`lib/`|Runtime dependencies| +|`logs/`|Test logs, generated after the first run| +|`data/`|Dataset or persisted result directory, generated according to the work mode and persistence configuration| - ```Bash - DB_SWITCH=IoTDB-200-SESSION_BY_TABLET - ``` +### 2\.4 Running a Test -2. Ensure the target time-series database is running. +1. Start IoTDB -3. Start IoT-benchmark to execute the test. Monitor the status of both the target database and IoT-benchmark during execution. +First, start the target IoTDB 2\.0\.x instance and verify that the client can access its service port. The default native interface port is `6667`. -4. Upon completion, review the results and analyze the test process. +2. Modify the configuration -### 2.5 **Results Interpretation** +- Edit `conf/config.properties`. -All test log files are stored in the `logs` folder, while test results are saved in the `data/csvOutput` folder. For example, the following result matrix illustrates the test outcome: +- Minimal connection configuration example: -![](/img/bm4.png) +```Properties +DB_SWITCH=IoTDB-200-SESSION_BY_TABLET +IoTDB_DIALECT_MODE=tree +HOST=127.0.0.1 +PORT=6667 +USERNAME=root +PASSWORD=root +DB_NAME=test +``` + +- To use the table model, change the setting to: -- **Result Matrix:** - - OkOperation: Number of successful operations. - - OkPoint: Number of successfully written points (for write operations) or successfully queried points (for query operations). - - FailOperation: Number of failed operations. - - FailPoint: Number of failed write points. -- **Latency (ms) Matrix:** - - AVG: Average operation latency. - - MIN: Minimum operation latency. - - Pn: Quantile values of the overall operation distribution (e.g., P25 represents the 25th percentile, or lower quartile). +```Properties +IoTDB_DIALECT_MODE=table +``` -## 3. **Main** **Parameters** +- If REST is selected: + +```Properties +DB_SWITCH=IoTDB-200-REST +REST_PORT=18080 +REST_AUTHORIZATION=Basic cm9vdDpyb290 +``` -### 3.1 IoTDB Service Model +`REST_AUTHORIZATION` configures the `Basic Authentication` information for the REST interface. The example uses the username `root` and password `root`. + +3. Check RPC compression compatibility + +IoT Benchmark 2\.0 enables IoTDB RPC compression by default: + +```Properties +ENABLE_IOTDB_RPC_COMPRESSION=true +``` + +This feature requires IoTDB 2\.0\.6 or later. When testing an IoTDB 2\.0\.x version earlier than 2\.0\.6, set: + +```Properties +ENABLE_IOTDB_RPC_COMPRESSION=false +``` + +Thrift compression is a separate configuration. If it is enabled: + +```Properties +ENABLE_THRIFT_COMPRESSION=true +``` -The `IoTDB_DIALECT_MODE` parameter supports two modes: `tree` and `table`. The default value is `tree`. +Also set the following in IoTDB's `iotdb-datanode.properties`: -- **For IoTDB 2.0 and later versions**, the `IoTDB_DIALECT_MODE` parameter must be specified, and only one mode can be set for each IoTDB instance. -- **IoTDB_DIALECT_MODE = tree:** - - The number of devices must be greater than or equal to the number of databases. +```Properties +dn_rpc_thrift_compression_enable=true +``` -### 3.2 **Working** **Mode** +4. Start Benchmark -The `BENCHMARK_WORK_MODE` parameter supports four operational modes: - -1. **General Test Mode (****`testWithDefaultPath`****):** Configured via the `OPERATION_PROPORTION` parameter to support write-only, read-only, and mixed read-write operations. -2. **Data Generation Mode (****`generateDataMode`****):** Generates a reusable dataset, which is saved to `FILE_PATH` for subsequent use in the correctness write and correctness query modes. -3. **Single Database Correctness Write Mode (****`verificationWriteMode`****):** Verifies the correctness of dataset writing by writing the dataset generated in data generation mode. This mode supports only IoTDB v1.0+ and InfluxDB v1.x. -4. **Single Database Correctness Query Mode (****`verificationQueryMode`****):** Verifies the correctness of dataset queries after using the correctness write mode. This mode supports only IoTDB v1.0+ and InfluxDB v1.x. - -Mode configurations are shown in the following below: - -| **Mode name** | **BENCHMARK_WORK_MODE** | Description | Required Configuration | -| :------------------------------------- | :---------------------- | :------------------------------------------------------ | :------------------------- | -| General test mode | testWithDefaultPath | Supports multiple read and write mixed load operations. | `OPERATION_PROPORTION` | -| Generate data mode | generateDataMode | Generates datasets recognizable by IoT-benchmark. | `FILE_PATH` and `DATA_SET` | -| Single database correctness write mode | verificationWriteMode | Writes datasets for correctness verification. | `FILE_PATH` and `DATA_SET` | -| Single database correctness query mode | verificationQueryMode | Queries datasets to verify correctness. | `FILE_PATH` and `DATA_SET` | - -### 3.3 **Server** **Connection** **Information** - -Once the working mode is specified, the following parameters must be configured to inform IoT-benchmark of the target time-series database: - -| **Parameter** | **Type** | **Example** | D**escription** | -| :------------ | :------- | :---------------------------- | :----------------------------------------------------- | -| DB_SWITCH | String | `IoTDB-200-SESSION_BY_TABLET` | Specifies the type of time-series database under test. | -| HOST | String | `127.0.0.1` | Network address of the target time-series database. | -| PORT | Integer | `6667` | Network port of the target time-series database. | -| USERNAME | String | `root` | Login username for the time-series database. | -| PASSWORD | String | `root` | Password for the database login user. | -| DB_NAME | String | `test` | Name of the target time-series database. | -| TOKEN | String | - | Authentication token (used for InfluxDB 2.0). | - -### 3.4 **Write Scenario Parameters** - -| **Parameter** | **Type** | **Example** | D**escription** | -| :------------------------- | :-------------------- | :-------------------------- |:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| CLIENT_NUMBER | Integer | `100` | Total number of clients used for writing. | -| GROUP_NUMBER | Integer | `20` | Number of databases (only applicable for IoTDB). | -| DEVICE_NUMBER | Integer | `100` | Total number of devices. | -| SENSOR_NUMBER | Integer | `300` | Total number of sensors per device. (Control the number of attribute columns if you use the IoTDB table mode) | -| INSERT_DATATYPE_PROPORTION | String | `1:1:1:1:1:1:0:0:0:0` | Ratio of data types: `BOOLEAN:INT32:INT64:FLOAT:DOUBLE:TEXT:STRING:BLOB:TIMESTAMP:DATE`. | -| POINT_STEP | Integer | `1000` | Time interval (in ms) between generated data points. | -| OP_MIN_INTERVAL | Integer | `0` | Minimum execution interval for operations (ms): if the operation takes more than the value, the next one will be executed immediately, otherwise wait (OP_MIN_INTERVAL - actual execution time) ms; if it is 0, the parameter is not effective; if it is -1, its value is consistent with POINT_STEP | -| IS_OUT_OF_ORDER | Boolean | `false` | Specifies whether to write data out of order. | -| OUT_OF_ORDER_RATIO | Floating point number | `0.3` | Proportion of out-of-order data. | -| BATCH_SIZE_PER_WRITE | Integer | `1` | Number of data rows written per batch. | -| START_TIME | Time | `2022-10-30T00:00:00+08:00` | Start timestamp for data generation. | -| LOOP | Integer | `86400` | Total number of write operations: Each type of operation will be divided according to the proportion defined by `OPERATION_PROPORTION` | -| OPERATION_PROPORTION | Character | `1:0:0:0:0:0:0:0:0:0:0` | Ratio of operation types (write:Q1:Q2:...:Q10). | - -### 3.5 **Query Scenario Parameters** - -| Parameter | Type | Example | Description | -| :------------------- | :-------- | :---------------------- | :----------------------------------------------------------- | -| QUERY_DEVICE_NUM | Integer | `2` | Number of devices involved in each query statement. | -| QUERY_SENSOR_NUM | Integer | `2` | Number of sensors involved in each query statement. | -| QUERY_AGGREGATE_FUN | Character | `count` | Aggregate functions used in queries (`COUNT`, `AVG`, `SUM`, etc.). | -| STEP_SIZE | Integer | `1` | Time interval step for time filter conditions. | -| QUERY_INTERVAL | Integer | `250000` | Time interval between query start and end times. | -| QUERY_LOWER_VALUE | Integer | `-5` | Threshold for conditional queries (`WHERE value > QUERY_LOWER_VALUE`). | -| GROUP_BY_TIME_UNIT | Integer | `20000` | The size of the group in the `GROUP BY` statement | -| LOOP | Integer | `10` | Total number of query operations: Each type of operation will be divided according to the proportion defined by `OPERATION_PROPORTION` | -| OPERATION_PROPORTION | Character | `0:0:0:0:0:0:0:0:0:0:1` | Ratio of operation types (`write:Q1:Q2:...:Q10`). | - -### 3.6 **Query Types and Example SQL** - -| Number | Query Type | IoTDB Sample SQL | -| :----- | :----------------------------- | :----------------------------------------------------------- | -| Q1 | Precise Point Query | `select v1 from root.db.d1 where time = ?` | -| Q2 | Time Range Query | `select v1 from root.db.d1 where time > ? and time < ?` | -| Q3 | Time Range with Value Filter | `select v1 from root.db.d1 where time > ? and time < ? and v1 > ?` | -| Q4 | Time Range Aggregation Query | `select count(v1) from root.db.d1 where and time > ? and time < ?` | -| Q5 | Full-Time Range with Filtering | `select count(v1) from root.db.d1 where v1 > ?` | -| Q6 | Range Aggregation with Filter | `select count(v1) from root.db.d1 where v1 > ? and time > ? and time < ?` | -| Q7 | Time Grouping Aggregation | `select count(v1) from root.db.d1 group by ([?, ?), ?, ?)` | -| Q8 | Latest Point Query | `select last v1 from root.db.d1` | -| Q9 | Descending Range Query | `select v1 from root.sg.d1 where time > ? and time < ? order by time desc` | -| Q10 | Descending Range with Filter | `select v1 from root.sg.d1 where time > ? and time < ? and v1 > ? order by time desc` | - -### 3.7 **Test process and test result persistence** - -IoT-benchmark currently supports persisting the test process and test results through configuration parameters. - -| **Parameter** | **Type** | **Example** | D**escription** | -| :-------------------- | :------- | :---------- | :----------------------------------------------------------- | -| TEST_DATA_PERSISTENCE | String | `None` | Specifies the result persistence method. Options: `None`, `IoTDB`, `MySQL`, `CSV`. | -| RECORD_SPLIT | Boolean | `true` | Whether to split results into multiple records. (Not supported by IoTDB currently.) | -| RECORD_SPLIT_MAX_LINE | Integer | `10000000` | Maximum number of rows per record (10 million rows per database table or CSV file). | -| TEST_DATA_STORE_IP | String | `127.0.0.1` | IP address of the database for result storage. | -| TEST_DATA_STORE_PORT | Integer | `6667` | Port number of the output database. | -| TEST_DATA_STORE_DB | String | `result` | Name of the output database. | -| TEST_DATA_STORE_USER | String | `root` | Username for accessing the output database. | -| TEST_DATA_STORE_PW | String | `root` | Password for accessing the output database. | - -**Result Persistence Details** - -- **CSV Mode:** If `TEST_DATA_PERSISTENCE` is set to `CSV`, a `data` folder is generated in the IoT-benchmark root directory during and after test execution. This folder contains: - - `csv` folder: Records the test process. - - `csvOutput` folder: Stores the test results. -- **MySQL Mode:** If `TEST_DATA_PERSISTENCE` is set to `MySQL`, IoT-benchmark creates the following tables in the specified MySQL database: - - **Test Process Table:** - 1. Created before the test starts. - 2. Named as: `testWithDefaultPath___`. - - **Configuration Table:** - 1. Named `CONFIG`. - 2. Stores the test configuration. - 3. Created if it does not exist. - - **Final Result Table:** - 1. Named `FINAL_RESULT`. - 2. Stores the test results after test completion. - 3. Created if it does not exist. - -### 3.8 Automation Script - -#### One-Click Script Startup - -The `cli-benchmark.sh` script allows one-click startup of IoTDB, IoTDB Benchmark monitoring, and IoTDB Benchmark testing. However, please note that this script will clear all existing data in IoTDB during startup, so use it with caution. - -**Steps to Run:** - -1. Edit the `IOTDB_HOME` parameter in `cli-benchmark.sh` to the local IoTDB directory. -2. Start the test by running the following command: +On Linux or macOS: ```Bash -> ./cli-benchmark.sh +./benchmark.sh +``` + +On Windows: + +```Plain Text +benchmark.bat +``` + +During the test, progress is periodically printed to the terminal. When the test completes, the main configurations, execution time, result matrix, and latency matrix are displayed. + +### 2\.5 Understanding the Results + +Test execution information is written to the `logs` folder in the installation directory. Whether CSV files are generated or results are written to a result database depends on parameters such as `TEST_DATA_PERSISTENCE`. + +1. Result matrix + +The result matrix reports the following metrics by operation type: + +|Metric|Description| +|---|---| +|`okOperation`|Number of successfully executed requests or SQL statements| +|`okPoint`|Number of successfully written data points, or data points successfully returned by queries| +|`failOperation`|Number of failed requests or SQL statements| +|`failPoint`|Number of data points that failed to be written; usually 0 for query operations| +|`throughput`|Throughput, usually equal to `okPoint / Test elapsed time`| + +The main operation names in the output include: + +- `INGESTION` + +- `PRECISE_POINT` + +- `TIME_RANGE` + +- `VALUE_RANGE` + +- `AGG_RANGE` + +- `AGG_VALUE` + +- `AGG_RANGE_VALUE` + +- `GROUP_BY` + +- `LATEST_POINT` + +- `RANGE_QUERY_DESC` + +- `VALUE_RANGE_QUERY_DESC` + +- `GROUP_BY_DESC` + +- `SET_OP_QUERY` + +2. Latency matrix + +The latency matrix is measured in milliseconds. Common fields are listed below. + +|Metric|Description| +|---|---| +|`AVG`|Average latency| +|`MIN`|Minimum latency| +|`P10`, `P25`, `MEDIAN`|Lower percentiles and median latency| +|`P75`, `P90`, `P95`|Higher-percentile latency| +|`P99`, `P999`|Tail latency| +|`MAX`|Maximum latency| +|`SLOWEST_THREAD`|Largest cumulative operation time among client threads| + +The test results also report metadata creation time and `Test elapsed time`, which excludes metadata creation. When comparing tests, ensure that the hardware, data volume, number of clients, compression configuration, and cache state are consistent across runs. + +3. Output example + +After the test completes, the terminal displays the main configurations, execution time, result matrix, and latency matrix. The following is truncated output from a write-only test: + +```Plain Text +----------------------Main Configurations---------------------- +BENCHMARK_WORK_MODE=testWithDefaultPath +DB_SWITCH=IoTDB-200-SESSION_BY_TABLET +HOST=[127.0.0.1] + +GROUP_NUMBER=10 +DEVICE_NUMBER=50 +SENSOR_NUMBER=500 +SCHEMA_CLIENT_NUMBER=20 +DATA_CLIENT_NUMBER=20 + +OPERATION_PROPORTION=1:0:0:0:0:0:0:0:0:0:0:0:0 +LOOP=10000 +BATCH_SIZE_PER_WRITE=100 +--------------------------------------------------------------- + +Create schema cost 0.30 second +Test elapsed time (not include schema creation): 1238.79 second + +----------------------------------------------------------Result Matrix---------------------------------------------------------- +Operation okOperation okPoint failOperation failPoint throughput(point/s) +INGESTION 500000 25000000000 0 0 20180954.09 +--------------------------------------------------------------------------------------------------------------------------------- + +----------------------------------------------------------Latency (ms) Matrix---------------------------------------------------- +Operation AVG MIN P10 P25 MEDIAN P75 P90 P95 P99 P999 MAX SLOWEST_THREAD +INGESTION 37.78 1.67 2.02 2.29 2.86 4.14 5.62 7.43 759.69 5799.89 8309.40 1227561.44 +--------------------------------------------------------------------------------------------------------------------------------- +``` + +The output shows that: + +- Metadata creation took `0.30` seconds, and the actual test took `1238.79` seconds. + +- A total of `500000` write operations successfully wrote `25000000000` data points. + +- Both `failOperation` and `failPoint` are `0`, indicating that no write failures were recorded during this test. + +- Write throughput was `20180954.09` points per second. + +- Average latency was `37.78` ms, P95 latency was `7.43` ms, P99 latency was `759.69` ms, and maximum latency was `8309.40` ms. + +- `SLOWEST_THREAD` is the cumulative operation time of the slowest client thread, not the latency of a single request. + +The values in this example are only intended to demonstrate the output format. Actual results depend on hardware resources, network conditions, IoTDB configuration, data scale, and test parameters. + +## 3\. Main Parameters + +### 3\.1 IoTDB Data Model + +IoTDB 2\.0\.x supports the tree model and the table model. Select the model using: + +```Properties +IoTDB_DIALECT_MODE=tree +``` + +or: + +```Properties +IoTDB_DIALECT_MODE=table +``` + +The following constraints apply: + +- An IoTDB instance can use only one SQL dialect in a single test. + +- The tree model requires `DEVICE_NUMBER >= GROUP_NUMBER`. + +- In the table model, the number of devices must be a multiple of the number of tables, and the number of tables must be a multiple of the number of databases. + +- In the table model, the number of data clients must be a multiple of the number of tables. + +Common model parameters: + +|Parameter|Example|Description| +|---|---|---| +|`IoTDB_DIALECT_MODE`|`tree`|`tree` or `table`| +|`GROUP_NUMBER`|`1`|Number of databases; corresponds to the number of databases in the tree model| +|`IoTDB_TABLE_NUMBER`|`1`|Number of tables created in the table model| +|`IoTDB_TABLE_NAME_PREFIX`|`table_`|Table name prefix| +|`TABLE_TIME_COLUMN`|`time`|Name of the time column in the table model| +|`IoTDB_TABLE_WRITABLE_VIEW`|`false`|Whether to create and use writable views| + +### 3\.2 Work Modes + +Use `BENCHMARK_WORK_MODE` to select a work mode. + +|Mode|Configuration Value|Description| +|---|---|---| +|Regular test mode|`testWithDefaultPath`|Runs a write, query, or mixed workload| +|Data generation mode|`generateDataMode`|Saves the dataset generated by Benchmark to `FILE_PATH`| +|Correctness write mode|`verificationWriteMode`|Loads a dataset from `FILE_PATH` and writes it to the database| +|Correctness query mode|`verificationQueryMode`|Loads a dataset and compares it with database query results| + +Example: + +```Properties +BENCHMARK_WORK_MODE=testWithDefaultPath +``` + +Before using the correctness write and query modes, use `generateDataMode` to generate a reusable dataset. + +### 3\.3 Server Connection Information + +|Parameter|Example|Description| +|---|---|---| +|`DB_SWITCH`|`IoTDB-200-SESSION_BY_TABLET`|Database version and connection method| +|`HOST`|`127.0.0.1`|IoTDB address; separate multiple addresses with commas| +|`PORT`|`6667`|Native service port; the number of ports must match the number of `HOST` entries| +|`USERNAME`|`root`|Username| +|`PASSWORD`|`root`|Password| +|`DB_NAME`|`test`|Name of the database used for testing| +|`REST_PORT`|`18080`|REST service port| +|`REST_AUTHORIZATION`|`Basic cm9vdDpyb290`|REST authorization header| +|`ENABLE_AUTO_FETCH`|`false`|Whether Session automatically refreshes the DataNode list| + +Data cleanup parameters: + +```Properties +IS_DELETE_DATA=false +INIT_WAIT_TIME=1000 +``` + +> Warning: `IS_DELETE_DATA=true` clears test data from the target database before the test starts. Enable it only in a dedicated test environment, and verify `HOST`, `PORT`, `DB_NAME`, and account permissions before execution. +> +> + +### 3\.4 Write Scenarios + +1. Data scale and clients + +|Parameter|Example|Description| +|---|---|---| +|`DEVICE_NUMBER`|`100`|Total number of devices| +|`SENSOR_NUMBER`|`10`|Number of measurements per device; number of measurement columns in the table model| +|`GROUP_NUMBER`|`1`|Number of IoTDB databases| +|`SCHEMA_CLIENT_NUMBER`|`5`|Number of clients that register metadata| +|`DATA_CLIENT_NUMBER`|`10`|Number of clients that perform data reads and writes| +|`IS_CLIENT_BIND`|`true`|Whether devices are bound to clients| +|`REAL_INSERT_RATE`|`1.0`|Proportion of devices that actually participate in writes| +|`IS_SENSOR_TS_ALIGNMENT`|`true`|Whether measurement timestamps under the same device are aligned| + +2. Batch writes + +|Parameter|Example|Description| +|---|---|---| +|`BATCH_SIZE_PER_WRITE`|`100`|Number of data rows written per device in each batch| +|`DEVICE_NUM_PER_WRITE`|`1`|Number of devices involved in each batch write| +|`CREATE_SCHEMA`|`true`|Whether to create metadata before writing| +|`START_TIME`|`2022-01-01T00:00:00+08:00`|Start time for generated data| + +The number of data points in a single batch is: + +```Plain Text +DEVICE_NUM_PER_WRITE × SENSOR_NUMBER × BATCH_SIZE_PER_WRITE +``` + +`DEVICE_NUM_PER_WRITE` must evenly divide the number of devices assigned to a single data client. In the table model, the divisibility constraints among the number of devices, number of tables, and devices per batch must also be satisfied. + +3. Write pacing + +|Parameter|Example|Description| +|---|---|---| +|`POINT_STEP`|`5000`|Fixed interval between adjacent generated timestamps| +|`OP_MIN_INTERVAL`|`0`|Minimum interval for each loop, in ms| +|`OP_MIN_INTERVAL_RANDOM`|`false`|Whether to randomly select an interval from `[0, OP_MIN_INTERVAL)`| +|`INTERVAL_BETWEEN_WRITE_BATCH`|`0`|Minimum interval between adjacent batches in the same loop, in ms| +|`TIMESTAMP_PRECISION`|`ms`|Timestamp precision| + +Special values of `OP_MIN_INTERVAL`: + +- `0`: does not limit the loop interval. + +- `-1`: uses `POINT_STEP` as the minimum interval. + +- Positive integer: if the current loop takes less than this value, waits for the remaining time. + +4. Out-of-order writes + +```Properties +IS_OUT_OF_ORDER=false +OUT_OF_ORDER_MODE=POISSON +OUT_OF_ORDER_RATIO=0.5 +IS_REGULAR_FREQUENCY=true +``` + +Supported out-of-order modes include: + +- `POISSON`: generates out-of-order timestamps according to a Poisson distribution. + +- `BATCH`: generates out-of-order data in batches. + +5. Data types + +```Properties +INSERT_DATATYPE_PROPORTION=1:1:1:1:1:1:0:0:0:0:0 +``` + +The order of the entries is: + +```Plain Text +BOOLEAN:INT32:INT64:FLOAT:DOUBLE:TEXT:STRING:BLOB:TIMESTAMP:DATE:OBJECT +``` + +Each value represents the proportion of the corresponding data type. + +### 3\.5 Query Scenarios + +|Parameter|Example|Description| +|---|---|---| +|`QUERY_DEVICE_NUM`|`1`|Number of devices involved in each query| +|`QUERY_SENSOR_NUM`|`1`|Number of measurements involved in each query| +|`QUERY_AGGREGATE_FUN`|`count`|Aggregation function| +|`STEP_SIZE`|`0`|Step by which the query start time changes, in units of `POINT_STEP`| +|`QUERY_INTERVAL`|`250000`|Interval between query start and end times| +|`QUERY_LOWER_VALUE`|`-5`|Lower bound of the value filter| +|`GROUP_BY_TIME_UNIT`|`20000`|Group By window size| +|`QUERY_SET_OP_TYPE`|`union`|Set operation type| +|`QUERY_SET_OP_NUM`|`2`|Number of sub-sets in a set query; at least 2| +|`IS_RECENT_QUERY`|`false`|Whether to prioritize recently written data in mixed scenarios| +|`ENABLE_FIXED_QUERY`|`false`|Whether all query threads use the same device and measurement combinations| +|`RESULT_ROW_LIMIT`|`-1`|Query result row limit; `-1` means no limit| +|`ALIGN_BY_DEVICE`|`false`|Whether to use Align By Device| + +### 3\.6 Operation Proportions + +`OPERATION_PROPORTION` defines the proportions of writes and different query types. It contains 13 entries: + +```Plain Text +Write:Q1:Q2:Q3:Q4:Q5:Q6:Q7:Q8:Q9:Q10:Q11:Q12 ``` -1. After the test completes: - 1. Check test-related logs in the `logs` folder. - 2. Check monitoring-related logs in the `server-logs` folder. +For example, write-only: -#### Automatic Execution of Multiple Tests +```Properties +OPERATION_PROPORTION=1:0:0:0:0:0:0:0:0:0:0:0:0 +``` -Single tests are often insufficient without comparative results. Therefore, IoT-benchmark provides an interface for executing multiple tests in sequence. +Precise point query only: -1. **Routine Configuration:** Each line in the `routine` file specifies the parameters that change for each test. For example: +```Properties +OPERATION_PROPORTION=0:1:0:0:0:0:0:0:0:0:0:0:0 +``` - ```Plain - LOOP=10 DEVICE_NUMBER=100 TEST - LOOP=20 DEVICE_NUMBER=50 TEST - LOOP=50 DEVICE_NUMBER=20 TEST - ``` +The operation types are listed below. + +|Number|Operation Type|Description| +|---|---|---| +|Write|Data write|Generates and writes data according to the current write configuration| +|Q1|Precise point query|Queries specified measurements by timestamp and device| +|Q2|Time range query|Range query restricted only by start and end times| +|Q3|Range query with value filter|Includes both time and value filter conditions| +|Q4|Aggregation query with time filter|Performs aggregation within a time range| +|Q5|Aggregation query with value filter|Filters by value and aggregates over the full time range| +|Q6|Aggregation query with time and value filters|Includes both time and value filter conditions| +|Q7|Time-grouped aggregation query|Group By query| +|Q8|Latest point query|Queries the latest data point of a device| +|Q9|Descending time range query|Returns range query results in descending time order| +|Q10|Descending range query with value filter|Filters by value and returns results in descending time order| +|Q11|Descending time-grouped aggregation query|Descending Group By query| +|Q12|Set operation query|Set operations such as `union`, `intersect`, or `except`| + +Q12 is supported only by the IoTDB 2\.0 table model. Each subquery in a set operation is a range query. + +### 3\.7 Test Process and Result Persistence + +```Properties +TEST_DATA_PERSISTENCE=None +``` -In this example, three tests will run sequentially with `LOOP` values of 10, 20, and 50. +Supported values include: -Then the test process with 3 LOOP parameters of 10, 20, and 50 is executed in sequence. +- `None`: does not write the test process to an external persistence medium. -**Important Notes:** +- `CSV`: writes to CSV files. -- Multiple parameters can be changed in each test using the format: +- `MySQL`: writes to MySQL. - ```Bash - LOOP=20 DEVICE_NUMBER=10 TEST - ``` +- `IoTDB`: writes to a specified IoTDB instance. -- Avoid unnecessary spaces. +Common parameters: -- The `TEST` keyword marks the start of a new test. +|Parameter|Example|Description| +|---|---|---| +|`TEST_DATA_PERSISTENCE`|`None`|Persistence method| +|`RECORD_SPLIT`|`true`|Whether to split results into multiple records| +|`RECORD_SPLIT_MAX_LINE`|`10000000`|Maximum number of records in a single table or file| +|`TEST_DATA_STORE_IP`|`127.0.0.1`|Result database address| +|`TEST_DATA_STORE_PORT`|`6667`|Result database port| +|`TEST_DATA_STORE_DB`|`result`|Result database name| +|`TEST_DATA_STORE_USER`|`root`|Result database username| +|`TEST_DATA_STORE_PW`|`root`|Result database password| +|`REMARK`|`write_test`|Test note used to distinguish different tests| +|`CSV_OUTPUT`|`true`|Whether to write final results to CSV| -- Changed parameters persist across subsequent tests unless explicitly reset. +When the persistence method is CSV, records are generated in the `data` directory after execution; test results are usually located in `data/csvOutput`. Test logs are always written to `logs`, regardless of whether persistence is enabled. -2. **Start the Test:** After configuring the `routine` file, start multi-test execution using the following command +The following parameters control log output frequency: - ```Bash - > ./rep-benchmark.sh - ``` +```Properties +IS_QUIET_MODE=true +LOG_PRINT_INTERVAL=5 +RESULT_PRINT_INTERVAL=3600 +``` -Test results will be displayed in the terminal. +### 3\.8 Automation and Cluster Testing -**Important Notes:** +1. Limit test duration -- Closing the terminal or losing the client connection will terminate the test process. +```Properties +TEST_MAX_TIME=3600000 +``` -- To run the test as a background daemon, execute: +The unit is milliseconds. A value of `0` means no limit. This parameter does not include the time spent pre-registering metadata. - ```Bash - > ./rep-benchmark.sh > /dev/null 2>&1 & - ``` +2. Multi-Benchmark cluster load testing -- To monitor progress, check the logs: +Use the same overall data scale configuration on multiple client machines, and set: - ```Bash - > cd ./logs - > tail -f log_info.log - ``` +```Properties +BENCHMARK_CLUSTER=true +BENCHMARK_INDEX=0 +``` -## 4. Use Case +Each Benchmark instance must use a different `BENCHMARK_INDEX`, such as `0`, `1`, and `2` in sequence. All clients should use consistent database connections, data scales, operation proportions, and other configurations. -We take the application of CRRC Qingdao Sifang Vehicle Research Institute Co., Ltd. as an example, and refer to the scene described in "Apache IoTDB in Intelligent Operation and Maintenance Platform Storage" for practical operation instructions. +3. Dual-write testing -Test objective: Simulate the actual needs of switching time series databases in the scene of CRRC Qingdao Sifang Institute, and compare the performance of the expected IoTDB and KairosDB used by the original system. +IoT Benchmark can write the same data to two different databases for comparison: -Test environment: In order to ensure that the impact of other irrelevant services and processes on database performance and the mutual influence between different databases are eliminated during the experiment, the local databases in this experiment are deployed and run on multiple independent virtual servers with the same resource configuration. Therefore, this experiment set up 4 Linux (CentOS7 /x86) virtual machines, and deployed IoT-benchmark, IoTDB database, KairosDB database, and MySQL database on them respectively. The specific resource configuration of each virtual machine is shown in Table 4-1. The specific usage of each virtual machine is shown in Table 4-2. +```Properties +IS_DOUBLE_WRITE=true +ANOTHER_DB_SWITCH= +ANOTHER_HOST=127.0.0.1 +ANOTHER_PORT=6667 +ANOTHER_USERNAME=root +ANOTHER_PASSWORD=root +ANOTHER_DB_NAME=test +``` -Table 4-1 Virtual machine configuration information +Dual-write mode does not support comparisons between different versions of the same database, or direct comparisons between the IoTDB tree model and table model. -| Hardware Configuration Information | Value | -| ---------------------------------- | ------- | -| OS system | CentOS7 | -| number of CPU cores | 16 | -| memory | 32G | -| hard disk | 200G | -| network | Gigabit | +## 4\. Examples -Table 4-2 Virtual machine usage +This section uses small datasets to demonstrate the basic process. For formal performance testing, increase the number of devices, measurements, clients, and loops according to the target business model, and perform multiple warm-up and repeated test runs. -| IP | Usage | -| ---------- | ------------- | -| 172.21.4.2 | IoT-benchmark | -| 172.21.4.3 | Apache-iotdb | -| 172.21.4.4 | KaiosDB | -| 172.21.4.5 | MySQL | +### 4\.1 Write Test Example -### 4.1 Write Test +Test objective: use 10 data clients to simulate 100 devices, each containing 10 measurements, and perform a write-only test. -Scenario description: Create 100 clients to simulate 100 trains, each train has 3000 sensors, the data type is DOUBLE, the data time interval is 500ms (2Hz), and they are sent sequentially. Referring to the above requirements, we need to modify the IoT-benchmark configuration parameters as listed in Table 4-3. +Example configuration: -Table 4-3 Configuration parameter information +```Properties +# Database connection +DB_SWITCH=IoTDB-200-SESSION_BY_TABLET +IoTDB_DIALECT_MODE=tree +HOST=127.0.0.1 +PORT=6667 +USERNAME=root +PASSWORD=root +DB_NAME=test -| Parameter Name | IoTDB Value | KairosDB Value | -| -------------------------- | --------------------------- | -------------- | -| DB_SWITCH | IoTDB-013-SESSION_BY_TABLET | KairosDB | -| HOST | 172.21.4.3 | 172.21.4.4 | -| PORT | 6667 | 8080 | -| BENCHMARK_WORK_MODE | testWithDefaultPath | | -| OPERATION_PROPORTION | 1:0:0:0:0:0:0:0:0:0:0 | | -| CLIENT_NUMBER | 100 | | -| GROUP_NUMBER | 10 | | -| DEVICE_NUMBER | 100 | | -| SENSOR_NUMBER | 3000 | | -| INSERT_DATATYPE_PROPORTION | 0:0:0:0:1:0 | | -| POINT_STEP | 500 | | -| OP_MIN_INTERVAL | 0 | | -| IS_OUT_OF_ORDER | false | | -| BATCH_SIZE_PER_WRITE | 1 | | -| LOOP | 10000 | | -| TEST_DATA_PERSISTENCE | MySQL | | -| TEST_DATA_STORE_IP | 172.21.4.5 | | -| TEST_DATA_STORE_PORT | 3306 | | -| TEST_DATA_STORE_DB | demo | | -| TEST_DATA_STORE_USER | root | | -| TEST_DATA_STORE_PW | admin | | -| REMARK | demo | | +# Safety setting: do not automatically delete existing data by default +IS_DELETE_DATA=false -First, start the tested time series databases Apache-IoTDB and KairosDB on 172.21.4.3 and 172.21.4.4 respectively, and then start server resource monitoring through the ser-benchamrk\.sh script on 172.21.4.2, 172.21.4.3 and 172.21.4.4 (Figure 4-1). Then modify the conf/config.properties files in the iotdb-0.13-0.0.1 and kairosdb-0.0.1 folders in 172.21.4.2 according to Table 4-3 to meet the test requirements. Use benchmark\.sh to start the writing test of Apache-IoTDB and KairosDB successively. +# Work mode +BENCHMARK_WORK_MODE=testWithDefaultPath +OPERATION_PROPORTION=1:0:0:0:0:0:0:0:0:0:0:0:0 +# Data scale +GROUP_NUMBER=1 +DEVICE_NUMBER=100 +SENSOR_NUMBER=10 +SCHEMA_CLIENT_NUMBER=5 +DATA_CLIENT_NUMBER=10 +IS_SENSOR_TS_ALIGNMENT=true -![img](/img/bm5.png) +# Write configuration +CREATE_SCHEMA=true +BATCH_SIZE_PER_WRITE=10 +DEVICE_NUM_PER_WRITE=1 +LOOP=100 +POINT_STEP=1000 +OP_MIN_INTERVAL=0 +START_TIME=2026-01-01T00:00:00+08:00 +INSERT_DATATYPE_PROPORTION=1:1:1:1:1:1:0:0:0:0:0 -Figure 4-1 Server monitoring tasks +# Can be enabled for IoTDB 2.0.6 and later +ENABLE_IOTDB_RPC_COMPRESSION=true -For example, if we first start the test on KairosDB, IoT-benchmark will create a CONFIG data table in the MySQL database to store the configuration information of this test (Figure 4-2), and there will be a log output of the current test progress during the test execution (Figure 4-3) . When the test is completed, the test result will be output (Figure 4-3), and the result will be written into the FINAL_RESULT data table (Figure 4-4). +# Output +TEST_DATA_PERSISTENCE=None +CSV_OUTPUT=true +REMARK=iotdb_2_write_test +``` -![img](/img/bm6.png) +Start the test: -Figure 4-2 Test configuration information table +```Bash +./benchmark.sh +``` + +After completion, focus on: + +- `okPoint` and `failPoint` for `INGESTION`. -![img](/img/bm7.png) -![img](/img/bm8.png) -![img](/img/bm9.png) -![img](/img/bm10.png) +- `throughput`. -Figure 4-3 Test progress and results +- `AVG`, `P95`, `P99`, and `MAX` latency. -![img](/img/bm11.png) +- Whether connection timeouts, write failures, or server exceptions are present in the logs. +### 4\.2 Query Test Example +Before running a query test, ensure that the target database contains data matching the query configuration. It is recommended to reuse data generated by the write test and disable automatic data deletion and metadata creation. -Figure 4-4 Test result table +The following example runs multiple query types: -Afterwards, we will start the test on Apache-IoTDB. The same IoT-benchmark will write the test configuration information in the MySQL database CONFIG data table. During the test execution, there will be a log to output the current test progress. When the test is completed, the test result will be output, and the result will be written into the FINAL_RESULT data table. +```Properties +DB_SWITCH=IoTDB-200-SESSION_BY_TABLET +IoTDB_DIALECT_MODE=tree +HOST=127.0.0.1 +PORT=6667 +USERNAME=root +PASSWORD=root +DB_NAME=test -According to the test result information, we know that under the same configuration the write delay times of Apache-IoTDB and KairosDB are 55.98ms and 1324.45ms respectively; the write throughputs are 5,125,600.86 points/second and 224,819.01 points/second respectively; the tests were executed respectively 585.30 seconds and 11777.99 seconds. And KairosDB has a write failure. After investigation, it is found that the data disk usage has reached 100%, and there is no disk space to continue receiving data. However, Apache-IoTDB has no write failure, and the disk space occupied after all data is written is only 4.7G (as shown in Figure 4-5); Apache-IoTDB is better than KairosDB in terms of write throughput and disk occupation. Of course, there will be other tests in the follow-up to observe and compare from various aspects, such as query performance, file compression ratio, data security, etc. +IS_DELETE_DATA=false +CREATE_SCHEMA=false +BENCHMARK_WORK_MODE=testWithDefaultPath -![img](/img/bm12.png) +GROUP_NUMBER=1 +DEVICE_NUMBER=100 +SENSOR_NUMBER=10 +SCHEMA_CLIENT_NUMBER=1 +DATA_CLIENT_NUMBER=10 +# Do not perform writes; Q1-Q11 have equal proportions; the tree model does not use Q12 +OPERATION_PROPORTION=0:1:1:1:1:1:1:1:1:1:1:1:0 +LOOP=100 + +QUERY_DEVICE_NUM=2 +QUERY_SENSOR_NUM=2 +QUERY_AGGREGATE_FUN=count +STEP_SIZE=1 +QUERY_INTERVAL=250000 +QUERY_LOWER_VALUE=-5 +GROUP_BY_TIME_UNIT=20000 +``` -Figure 4-5 Disk usage +To test set queries in the table model, switch the dialect to `table` and assign a proportion to Q12: -So what is the resource usage of each server during the test? What is the specific performance of each write operation? At this time, we can visualize the data in the server monitoring table and test process recording table by installing and using Tableau. The use of Tableau will not be introduced in this article. After connecting to the data table for test data persistence, the specific results are as follows (taking Apache-IoTDB as an example): +```Properties +IoTDB_DIALECT_MODE=table +OPERATION_PROPORTION=0:0:0:0:0:0:0:0:0:0:0:0:1 +QUERY_SET_OP_TYPE=union +QUERY_SET_OP_NUM=2 +``` +### 4\.3 Other Configuration Examples -![img](/img/bm13.png) -![img](/img/bm14.png) +1. Simulate an actual write rate +Set the minimum interval of each loop to the data timestamp interval: -Figure 4-6 Visualization of testing process in Tableau +```Properties +POINT_STEP=1000 +OP_MIN_INTERVAL=-1 +``` -### 4.2 Query Test +To distribute write requests evenly within a loop, use: -Scenario description: In the writing test scenario, 10 clients are simulated to perform all types of query tasks on the data stored in the time series database Apache-IoTDB. The configuration is as follows. +```Properties +INTERVAL_BETWEEN_WRITE_BATCH=100 +``` -Table 4-4 Configuration parameter information +2. Specify test duration -| Parameter Name | Example | -| -------------------- | --------------------- | -| CLIENT_NUMBER | 10 | -| QUERY_DEVICE_NUM | 2 | -| QUERY_SENSOR_NUM | 2 | -| QUERY_AGGREGATE_FUN | count | -| STEP_SIZE | 1 | -| QUERY_INTERVAL | 250000 | -| QUERY_LOWER_VALUE | -5 | -| GROUP_BY_TIME_UNIT | 20000 | -| LOOP | 30 | -| OPERATION_PROPORTION | 0:1:1:1:1:1:1:1:1:1:1 | +Test for one hour: -Results: +```Properties +TEST_MAX_TIME=3600000 +``` -![img](/img/bm15.png) +Ensure that `LOOP` is sufficiently large; otherwise, the test may end when the loop count is exhausted. -Figure 4-7 Query test results +3. Control generated data patterns -### 4.3 Description of Other Parameters +```Properties +LINE_RATIO=1 +SIN_RATIO=1 +SQUARE_RATIO=1 +RANDOM_RATIO=1 +CONSTANT_RATIO=1 +DATA_SEED=666 +STRING_LENGTH=10 +DOUBLE_LENGTH=2 +``` -In the previous chapters, the write performance comparison between Apache-IoTDB and KairosDB was performed, but if the user wants to perform a simulated real write rate test, how to configure it? How to control if the test time is too long? Are there any regularities in the generated simulated data? If the IoT-Benchmark server configuration is low, can multiple machines be used to simulate pressure output? +Fixing `DATA_SEED` helps generate reproducible data across multiple test runs. -Table 4-5 Configuration parameter information +## 5\. References -| Scenario | Parameter | Value | Notes | -| ------------------------------------------------------------ | -------------------------- | ------------------------------------------------------------ | ------------------------------------------------------------ | -| Simulate real write rate | OP_INTERVAL | -1 | You can also enter an integer to control the operation interval. | -| Specify test duration (1 hour) | TEST_MAX_TIME | 3600000 | The unit is ms; the LOOP execution time needs to be greater than this value. | -| Define the law of simulated data: support all data types, and the number is evenly classified; support five data distributions, and the number is evenly distributed; the length of the string is 10; the number of decimal places is 2. | INSERT_DATATYPE_PROPORTION | 1:1:1:1:1:1 | Data type distribution proportion | -| LINE_RATIO | 1 | linear | | -| SIN_RATIO | 1 | Fourier function | | -| SQUARE_RATIO | 1 | Square wave | | -| RANDOM_RATIO | 1 | Random number | | -| CONSTANT_RATIO | 1 | Constant | | -| STRING_LENGTH | 10 | String length | | -| DOUBLE_LENGTH | 2 | Decimal places | | -| Three machines simulate data writing of 300 devices | BENCHMARK_CLUSTER | true | Enable multi-benchmark mode | -| BENCHMARK_INDEX | 0, 1, 3 | Take the writing parameters in the [write test](./Benchmark.md#_4-1-write-test) as an example: No. 0 is responsible for writing data of device numbers 0-99; No. 1 is responsible for writing data of device numbers 100-199; No. 2 is responsible for writing data of device numbers 200-299. | | \ No newline at end of file +- [IoT Benchmark Documentation](https://github.com/thulab/iot-benchmark/tree/master/docs) diff --git a/src/zh/UserGuide/Master/Table/Tools-System/Benchmark.md b/src/zh/UserGuide/Master/Table/Tools-System/Benchmark.md index 8d7d578ef..78818a0d3 100644 --- a/src/zh/UserGuide/Master/Table/Tools-System/Benchmark.md +++ b/src/zh/UserGuide/Master/Table/Tools-System/Benchmark.md @@ -21,350 +21,800 @@ # 测试工具 -## 1. 基本概述 +IoT Benchmark 是面向工业物联网（IIoT）场景的时序数据库、实时数据库基准测试工具。本手册介绍工具的主要能力、支持的数据库及基本使用方法；安装、配置和测试案例主要以 IoTDB 2\.0\.x 为例，并同时覆盖树模型和表模型。 -IoT-benchmark 是基于 Java 和大数据环境开发的时序数据库基准测试工具，由清华大学软件学院研发并开源。它使用方便，支持多种写入以及查询方式，支持存储测试信息和结果以供进一步查询或分析，支持与 Tableau 集成以可视化测试结果。 +![](/img/bm2.png) -下图1-1囊括了测试基准流程及其他扩展功能。这些流程可以由IoT-benchmark 统一来完成。IoT Benchmark 支持多种工作负载，包括**纯写入、纯查询、写入查询混合**等，支持**软硬件系统监控、测试指标度量**等监控功能，还实现了**初始化数据库自动化、测试数据分析及系统参数优化**等功能。 +## 1\. 基本概述 -![](/img/bm1.png) +IoT Benchmark 可以按照配置生成周期性时间序列数据，对数据库执行写入和查询，并统计吞吐、延迟及成功率等指标。主要能力包括： -图1-1 +- 跨平台运行：支持 Linux、macOS 和 Windows。 -借鉴 YCSB 测试工具将工作负载生成、性能指标测量和数据库接口三个组件分离的设计思想，IoT-benchmark 的模块化设计如图1-2所示。与基于 YCSB 的测试工具系统不同的是，IoT-benchmark 增加了系统监控模块，支持测试数据和系统指标监控数据的持久化。此外也增加了一些特别针对时序数据场景的特殊负载测试功能，如支持物联网场景的批量写入和多种乱序数据写入模式。 +- 多种负载：支持纯写入、纯查询及读写混合负载。 -![](/img/bm2.png) +- 数据集生成：可将生成的数据保存到磁盘，供重复测试使用。 + +- 正确性验证：可从磁盘加载数据集，执行写入和查询正确性验证。 + +- 多数据库支持：可对多种时序数据库和实时数据库执行测试；IoTDB 2\.0\.x 支持 JDBC、REST 和多种 Session 写入方式。 + +- 结果持久化：支持将测试过程和结果保存到文件、CSV、MySQL 或 IoTDB。 + +- 测试扩展：支持乱序写入、批量写入、集群压测、双写对比和结果可视化等场景。 + +### 1\.1 支持的数据库、版本和接入方式 + +IoT Benchmark 支持如下数据库和版本。测试时通过 `DB_SWITCH` 选择对应的数据库、版本及连接或写入方式。 + +|数据库|支持版本|`DB_SWITCH`| +|---|---|---| +|IoTDB|v1\.x|`IoTDB-130-JDBC`、`IoTDB-130-REST`、`IoTDB-130-SESSION_BY_TABLET`、`IoTDB-130-SESSION_BY_RECORD`、`IoTDB-130-SESSION_BY_RECORDS`| +|IoTDB|v2\.x|`IoTDB-200-JDBC`、`IoTDB-200-REST`、`IoTDB-200-SESSION_BY_TABLET`、`IoTDB-200-SESSION_BY_RECORD`、`IoTDB-200-SESSION_BY_RECORDS`| +|InfluxDB|v1\.x|`InfluxDB`| +|InfluxDB|v2\.x|`InfluxDB-2.x`| +|QuestDB|v6\.0\.7|`QuestDB`| +|Microsoft SQL Server|2016 SP2|`MSSQLSERVER`| +|VictoriaMetrics|v1\.64\.0|`VictoriaMetrics`| +|SQLite|—|`SQLite`| +|OpenTSDB|2\.4\.1|`OpenTSDB`| +|KairosDB|—|`KairosDB`| +|TimescaleDB|—|`TimescaleDB`| +|TimescaleDB Cluster|Cluster|`TimescaleDB-Cluster`| +|TDengine|2\.2\.0\.2|`TDengine`| +|TDengine|3\.0\.1|`TDengine-3`| +|DolphinDB|v2\.x|`DolphinDB-2-MTW`、`DolphinDB-2-PTA`| +|DolphinDB|v3\.x|`DolphinDB-3-MTW`、`DolphinDB-3-PTA`| +|CnosDB|—|`CnosDB`| -图1-2 +其中： +- IoTDB 的接入方式包括 JDBC、REST、Session by Tablet、Session by Record 和 Session by Records。 -目前 IoT-benchmark 支持如下时间序列数据库、版本和连接方式： +- DolphinDB 的 `MTW` 表示 `MultithreadedTableWriter`，按行缓冲写入；`PTA` 表示 `PartitionedTableAppender`，按批次进行列式整表追加。 -| 数据库 | 版本 | 连接方式 | -| :-------------- |:-----------| :------------------------------------------------------- | -| IoTDB | v1.x v2.x | jdbc、sessionByTablet、sessionByRecord、sessionByRecords | -| InfluxDB | v1.x v2.x | SDK | -| TimescaleDB | -- | jdbc | -| OpenTSDB | -- | Http Request | -| QuestDB | v6.0.7 | jdbc | -| TDengine | v2.2.0.2 | jdbc | -| VictoriaMetrics | v1.64.0 | Http Request | -| KairosDB | -- | Http Request | +- 数据库版本、驱动和服务端必须相互匹配。使用其他数据库时，还需要配置相应的连接及扩展参数。 +### 1\.2 IoTDB 2\.0\.x 接入方式 -表1-1大数据测试基准对比 +后续安装、配置和案例以 IoTDB 2\.0\.x 为主。支持的接入方式如下。 -## 2. 安装运行 +|接入方式|`DB_SWITCH`|说明| +|---|---|---| +|JDBC|`IoTDB-200-JDBC`|通过 JDBC 执行写入和查询| +|REST|`IoTDB-200-REST`|通过 IoTDB REST 接口执行测试| +|Session by Tablet|`IoTDB-200-SESSION_BY_TABLET`|使用 Tablet 批量写入| +|Session by Record|`IoTDB-200-SESSION_BY_RECORD`|逐条写入记录| +|Session by Records|`IoTDB-200-SESSION_BY_RECORDS`|批量写入多条记录| -### 2.1 前置条件 +`IoTDB-200-SESSION_BY_TABLET` 适用于批量写入测试。 -1. Java 8 -2. Maven 3.6+ -3. 对应的合适版本的数据库，如 Apache IoTDB 2.0 +## 2\. 安装运行 +本章以 IoTDB 2\.0\.x 作为被测数据库。 +### 2\.1 前置条件 -### 2.2 获取方式 +1. 使用 IoT Benchmark 前需要准备： -- 获取二进制包：进入[这里](https://github.com/thulab/iot-benchmark/releases) 下载需要的安装包。下载下来为一个压缩文件，选择文件夹解压即可使用。 -- 源代码编译（可用 Apache IoTDB 2.0 的测试）： - - 第一步（编译 IoTDB Session 最新包）：进入[官网](https://github.com/apache/iotdb/tree/rc/2.0.5)下载 IoTDB 源码，在根目录下运行命令 mvn clean package install -pl session -am -DskipTests 编译 IoTDB Session 的最新包。 - - 第二步（编译 IoTDB Benchmark 测试包）：进入[官网](https://github.com/thulab/iot-benchmark)下载源码，在根目录下运行 mvn clean package install -pl iotdb-2.0 -am -DskipTests 编译测试 Apache IoTDB 2.0版本的测试包，测试包位置与根目录的相对路径为 ./iotdb-2.0/target/iotdb-2.0-0.0.1/iotdb-2.0-0.0.1 + - Java 17。 + - Maven。 -### 2.3 测试包结构 + - 已安装并能够正常运行的 IoTDB 2\.0\.x。 -测试包的目录结构如下所示。其中测试配置文件为conf/config.properties，测试启动脚本为benchmark\.sh (Linux & MacOS) 和 benchmark.bat (Windows)，详细文件用途见下表所示。 + - 足够的客户端 CPU、内存、磁盘和网络资源。 + +2. 环境相关说明： + + - 推荐使用 Linux 或 macOS 执行测试。 + + - Windows 使用安装包根目录下的 `benchmark.bat` 启动测试。 + + - Linux 和 macOS 使用 `benchmark.sh` 启动测试。 + + - CSV 记录模式中的部分系统信息采集能力仅支持 Linux。 + +> 注意：不要将 IoT Benchmark 与被测 IoTDB 部署在资源相互竞争的环境中。正式性能测试建议使用独立服务器，并关闭无关服务。 +> +> -```Shell --rw-r--r--. 1 root root 2881 1月 10 01:36 benchmark.bat --rwxr-xr-x. 1 root root 314 1月 10 01:36 benchmark.sh -drwxr-xr-x. 2 root root 24 1月 10 01:36 bin --rwxr-xr-x. 1 root root 1140 1月 10 01:36 cli-benchmark.sh -drwxr-xr-x. 2 root root 107 1月 10 01:36 conf -drwxr-xr-x. 2 root root 4096 1月 10 01:38 lib --rw-r--r--. 1 root root 11357 1月 10 01:36 LICENSE --rwxr-xr-x. 1 root root 939 1月 10 01:36 rep-benchmark.sh --rw-r--r--. 1 root root 14 1月 10 01:36 routine +### 2\.2 获取方式 + +1. 获取发行包 + +可以从 [IoT Benchmark Releases](https://github.com/thulab/iot-benchmark/releases) 下载与测试目标匹配的发行包，解压后使用。 + +2. 从源码构建 + +克隆仓库： + +```Bash +git clone https://github.com/thulab/iot-benchmark.git +cd iot-benchmark ``` -| 名称 | 子文件 | 用途 | -| :--------------- | :---------------- | :------------------------ | -| benchmark.bat | - | Windows环境运行启动脚本 | -| benchmark.sh | - | Linux/Mac环境运行启动脚本 | -| bin | startup.sh | 初始化脚本文件夹 | -| conf | config.properties | 测试场景配置文件 | -| lib | - | 依赖库文件 | -| LICENSE | - | 许可文件 | -| cli-benchmark.sh | - | 一键化启动脚本 | -| routine | | 多项测试配置文件 | -| rep-benchmark.sh | | 多项测试启动脚本 | +在项目根目录执行： -表2-1文件和文件夹列表用途 +```Bash +mvn clean package -Dmaven.test.skip=true +``` -### 2.4 执行测试 +构建完成后，IoTDB 2\.0\.x 安装包位于： -1. 按照测试需求修改配置文件，主要参数介绍见第3节，对应配置文件为conf/config.properties，**比如测试Apache IoTDB 2.0，则需要修改 DB_SWITCH=IoTDB-200-SESSION_BY_TABLET** -2. 启动被测时间序列数据库 -3. 通过运行 -4. 启动IoT-benchmark执行测试。执行中观测被测时间序列数据库和IoT-benchmark状态，执行完毕后查看结果和分析测试过程。 +```Plain Text +iotdb-2.0/target/iot-benchmark-iotdb-2.0/iot-benchmark-iotdb-2.0 +``` -### 2.5 结果说明 +进入安装目录： -测试的所有日志文件被存放于 logs 文件夹下，测试的结果在测试完成后被存放到 data/csvOutput 文件夹下，例如测试后我们得到了如下的结果矩阵： +```Bash +cd iotdb-2.0/target/iot-benchmark-iotdb-2.0/iot-benchmark-iotdb-2.0 +``` -![](/img/bm4.png) +### 2\.3 测试包结构 -- Result Matrix - - OkOperation：成功的对应操作次数 - - OkPoint：对于写入操作，是成功写入的点数；对于查询操作，是成功查询到的点数。 - - FailOperation：失败的对应操作次数 - - FailPoint：对于写入操作是写入失败的点数 -- Latency(mx) Matrix - - AVG：操作平均耗时 - - MIN：操作最小耗时 - - Pn：操作整体分布的对应分位值，比如P25是下四分位数 +安装包中的常用文件和目录如下。 +|名称|用途| +|---|---| +|`benchmark.sh`|Linux、macOS 启动脚本| +|`benchmark.bat`|Windows 启动脚本| +|`conf/config.properties`|测试场景配置文件| +|`lib/`|运行依赖库| +|`logs/`|测试日志，首次运行后生成| +|`data/`|数据集或持久化结果目录，按工作模式和持久化配置生成| +### 2\.4 执行测试 -## 3. 主要参数 +1. 启动 IoTDB +首先启动目标 IoTDB 2\.0\.x，并确认客户端能够访问其服务端口。默认原生接口端口为 `6667`。 -### 3.1 IoTDB服务模型 +2. 修改配置 -参数`IoTDB_DIALECT_MODE`支持tree、table,默认值为tree。 +- 编辑 `conf/config.properties` -- 当被测数据库为IoTDB-2.0及以上版本时需指定sql_dialect, 并且一个IoTDB只能指定一种。 -- sql_dialect等于table时，要满足：device数为table数的整数倍，table数为database数的整数倍 -- 表模型模式下，调整参数如下： +- 最小连接配置示例： -| **参数名称** | **类型** | **示例** | **系统描述** | -| :---------------------- | :------- | :------- | :----------------------------------------------------------- | -| IoTDB_TABLE_NAME_PREFIX | 字符串 | table_ | IoTDB 表模型下(IoTDB_DIALECT_MODE=table)，表名称前缀 | -| DATA_CLIENT_NUMBER | 整数 | 10 | IoTDB 表模型下(IoTDB_DIALECT_MODE=table)，client数为table数的整数倍 | -| SENSOR_NUMBER | 整数 | 10 | IoTDB 表模型下(IoTDB_DIALECT_MODE=table)，则控制属性列数量 | -| IoTDB_TABLE_NUMBER | 整数 | 1 | IoTDB 表模型下(IoTDB_DIALECT_MODE=table)，表的数量 | +```Properties +DB_SWITCH=IoTDB-200-SESSION_BY_TABLET +IoTDB_DIALECT_MODE=tree +HOST=127.0.0.1 +PORT=6667 +USERNAME=root +PASSWORD=root +DB_NAME=test +``` -### 3.2 工作模式 +- 使用表模型时改为： -工作模式参数“`BENCHMARK_WORK_MODE`”可选项有如下四种模式： +```Properties +IoTDB_DIALECT_MODE=table +``` -- 常用测试模式：结合配置`OPERATION_PROPORTION`参数达到“纯写入”、“纯查询”和“读写混合”的测试操作。 -- 生成数据模式：为了生成可以重复使用的数据集，iot-benchmark提供生成数据集的模式，生成数据集到FILE_PATH，以供后续使用正确性写入模式和正确性查询模式使用。 -- 单数据库正确性写入模式：为了验证数据集写入的正确性，您可以使用该模式写入生成数据模式中生成的数据集，目前该模式仅支持IoTDB v1.0 及更新的版本和InfluxDB v1.x。 -- 单数据库正确性查询模式：在运行这个模式之前需要先使用正确性写入模式写入数据到数据库。为了验证数据集写入的正确性，您可以使用该模式查询写入到数据库中的数据集，目前该模式仅支持IoTDB v1.0 和 InfluxDB v1.x。 +- 如果选择 REST： -| **模式名称** | **BENCHMARK_WORK_MODE** | **模式内容** | -| :--------------------- | :---------------------- | :------------------------------- | -| 常规测试模式 | testWithDefaultPath | 支持多种读和写操作的混合负载 | -| 生成数据模式 | generateDataMode | 生成Benchmark本身识别的数据 | -| 单数据库正确性写入模式 | verificationWriteMode | 需要配置 FILE_PATH 以及 DATA_SET | -| 单数据库正确性查询模式 | verificationQueryMode | 需要配置 FILE_PATH 以及 DATA_SET | +```Properties +DB_SWITCH=IoTDB-200-REST +REST_PORT=18080 +REST_AUTHORIZATION=Basic cm9vdDpyb290 +``` -### 3.3 服务器连接信息 +`REST_AUTHORIZATION `用于配置 REST 接口的 `Basic Authentication` 信息。示例使用用户名 root、密码 root。 -工作模式指定后，被测时序数据库的信息会通过如下参数告知IoT-benchmark +3. 检查 RPC 压缩兼容性 -| **参数名称** | **类型** | **示例** | **系统描述** | -| :----------- | :------- | :-------------------------- | :---------------------------------------------- | -| DB_SWITCH | 字符串 | IoTDB-200-SESSION_BY_TABLET | 被测时序数据库类型 | -| HOST | 字符串 | 127.0.0.1 | 被测时序数据库网络地址 | -| PORT | 整数 | 6667 | 被测时序数据库网络端口 | -| USERNAME | 字符串 | root | 被测时序数据库登录用户名 | -| PASSWORD | 字符串 | root | 被测时序数据库登录用户的密码 | -| DB_NAME | 字符串 | test | 被测时序数据库名称 | -| TOKEN | 字符串 | | 被测时序数据库连接认证Token（InfluxDB 2.0使用） | - -### 3.4 写入场景 - -| **参数名称** | **类型** | **示例** | **系统描述** | -| :------------------------- | :------- | :------------------------ | :----------------------------------------------------------- | -| CLIENT_NUMBER | 整数 | 100 | 客户端总数 | -| GROUP_NUMBER | 整数 | 20 | 数据库的数量；仅针对IoTDB。 | -| DEVICE_NUMBER | 整数 | 100 | 设备总数 | -| SENSOR_NUMBER | 整数 | 300 | 每个设备的传感器总数； **如果使用 IoTDB 表模型，则控制属性列数量** | -| INSERT_DATATYPE_PROPORTION | 字符串 | 1:1:1:1:1:1 | 设备的数据类型比例，BOOLEAN:INT32:INT64:FLOAT:DOUBLE:TEXT | -| POINT_STEP | 整数 | 1000 | 数据间时间戳间隔，即生成的数据两个时间戳之间的固定长度。 | -| OP_MIN_INTERVAL | 整数 | 0 | 操作最小执行间隔：若操作耗时大于该值则立即执行下一个，否则等待 (OP_MIN_INTERVAL-实际执行时间) ms；如果为0，则参数不生效；如果为-1，则其值和POINT_STEP一致 | -| IS_OUT_OF_ORDER | 布尔 | false | 是否乱序写入 | -| OUT_OF_ORDER_RATIO | 浮点数 | 0.3 | 乱序写入的数据比例 | -| BATCH_SIZE_PER_WRITE | 整数 | 1 | 批写入数据行数（一次写入多少行数据） | -| START_TIME | 时间 | 2022-10-30T00:00:00+08:00 | 写入数据的开始时间戳；以该时间戳为起点开始模拟创建数据时间戳。 | -| LOOP | 整数 | 86400 | 总操作次数：具体每种类型操作会按OPERATION_PROPORTION定义的比例划分 | -| OPERATION_PROPORTION | 字符 | 1:0:0:0:0:0:0:0:0:0:0 | # 各操作的比例,按照顺序为 写入:Q1:Q2:Q3:Q4:Q5:Q6:Q7:Q8:Q9:Q10, 请注意使用英文冒号。比例中的每一项是整数。 | +IoT Benchmark 2\.0 默认开启 IoTDB RPC 压缩： + +```Properties +ENABLE_IOTDB_RPC_COMPRESSION=true +``` +该功能要求被测 IoTDB 版本为 2\.0\.6 或更高版本。如果测试 2\.0\.6 之前的 IoTDB 2\.0\.x，请设置： -### 3.5 查询场景 +```Properties +ENABLE_IOTDB_RPC_COMPRESSION=false +``` -| 参数名称 | 类型 | 示例 | 系统描述 | -| :------------------- | :--- | :-------------------- | :----------------------------------------------------------- | -| QUERY_DEVICE_NUM | 整数 | 2 | 每条查询语句中查询涉及到的设备数量 | -| QUERY_SENSOR_NUM | 整数 | 2 | 每条查询语句中查询涉及到的传感器数量 | -| QUERY_AGGREGATE_FUN | 字符 | count | 在聚集查询中使用的聚集函数，比如count、avg、sum、max_time等 | -| STEP_SIZE | 整数 | 1 | 时间过滤条件的时间起点变化步长，若设为0则每个查询的时间过滤条件是一样的，单位：POINT_STEP | -| QUERY_INTERVAL | 整数 | 250000 | 起止时间的查询中开始时间与结束时间之间的时间间隔，和Group By中的时间间隔 | -| QUERY_LOWER_VALUE | 整数 | -5 | 条件查询子句时的参数，where xxx > QUERY_LOWER_VALUE | -| GROUP_BY_TIME_UNIT | 整数 | 20000 | Group by语句中的组的大小 | -| LOOP | 整数 | 10 | 总操作次数：具体每种类型操作会按OPERATION_PROPORTION定义的比例划分 | -| OPERATION_PROPORTION | 字符 | 0:0:0:0:0:0:0:0:0:0:1 | 写入:Q1:Q2:Q3:Q4:Q5:Q6:Q7:Q8:Q9:Q10 | +Thrift 压缩是另一项独立配置。如果启用： +```Properties +ENABLE_THRIFT_COMPRESSION=true +``` -### 3.6 操作比例 +还需要在 IoTDB 的 `iotdb-datanode.properties` 中设置： -| 编号 | 查询类型 | IoTDB 示例 SQL | -| :--- | :--------------------------- | :----------------------------------------------------------- | -| Q1 | 精确点查询 | select v1 from root.db.d1 where time = ? | -| Q2 | 时间范围查询 | select v1 from root.db.d1 where time > ? and time < ? | -| Q3 | 带值过滤的时间范围查询 | select v1 from root.db.d1 where time > ? and time < ? and v1 > ? | -| Q4 | 时间范围聚合查询 | select count(v1) from root.db.d1 where and time > ? and time < ? | -| Q5 | 带值过滤的全时间范围聚合查询 | select count(v1) from root.db.d1 where v1 > ? | -| Q6 | 带值过滤的时间范围聚合查询 | select count(v1) from root.db.d1 where v1 > ? and time > ? and time < ? | -| Q7 | 时间分组聚合查询 | select count(v1) from root.db.d1 group by ([?, ?), ?, ?) | -| Q8 | 最新点查询 | select last v1 from root.db.d1 | -| Q9 | 倒序范围查询 | select v1 from root.sg.d1 where time > ? and time < ? order by time desc | -| Q10 | 倒序带值过滤的范围查询 | select v1 from root.sg.d1 where time > ? and time < ? and v1 > ? order by time desc | +```Properties +dn_rpc_thrift_compression_enable=true +``` -### 3.7 测试过程和测试结果持久化 +4. 启动 Benchmark -IoT-benchmark目前支持通过配置参数将测试过程和测试结果持久化: - -| **参数名称** | **类型** | **示例** | **系统描述** | -| :-------------------- | :------- | :-------- | :----------------------------------------------------------- | -| TEST_DATA_PERSISTENCE | 字符串 | None | 结果持久化选择，支持None，IoTDB，MySQL和CSV | -| RECORD_SPLIT | 布尔 | true | 是否将结果划分后输出到多个记录, IoTDB 暂时不支持 | -| RECORD_SPLIT_MAX_LINE | 整数 | 10000000 | 记录行数的上限（每个数据库表或CSV文件按照总行数为1千万切分存放） | -| TEST_DATA_STORE_IP | 字符串 | 127.0.0.1 | 输出数据库的IP地址 | -| TEST_DATA_STORE_PORT | 整数 | 6667 | 输出数据库的端口号 | -| TEST_DATA_STORE_DB | 字符串 | result | 输出数据库的名称 | -| TEST_DATA_STORE_USER | 字符串 | root | 输出数据库的用户名 | -| TEST_DATA_STORE_PW | 字符串 | root | 输出数据库的用户密码 | - -- 如果我们设置“`TEST_DATA_PERSISTENCE=CSV`”，测试执行时和执行完毕后我们可以在IoT-benchmark根目录下看到新生成的`data`文件夹，其下包含`csv`文件夹记录测试过程；`csvOutput`文件夹记录测试结果。 -- 如果我们设置“`TEST_DATA_PERSISTENCE=MySQL`”，它会在测试开始前在指定的MySQL数据库中创建命名如“testWithDefaultPath_被测数据库名称_备注_测试启动时间”的数据表记录测试过程；会在名为“CONFIG”的数据表（如果不存在则创建该表），写入本次测试的配置信息；当测试完成时会在名为“FINAL_RESULT”的数据表（如果不存在则创建该表）中写入本次测试结果。 - -### 3.8 自动化脚本 - -#### 一键化启动脚本 - -您可以通过`cli-benchmark.sh`脚本一键化启动IoTDB、监控的IoTDB Benchmark和测试的IoTDB Benchmark，但需要注意该脚本启动时会清理IoTDB中的**所有数据**，请谨慎使用。 - -首先，您需要修改`cli-benchmark.sh`中的`IOTDB_HOME`参数为您本地的IoTDB所在的文件夹。 - -然后您可以使用脚本启动测试 +Linux 或 macOS： ```Bash -> ./cli-benchmark.sh +./benchmark.sh +``` + +Windows： + +```Plain Text +benchmark.bat ``` -测试完成后您可以在`logs`文件夹中查看测试相关日志，在`server-logs`文件夹中查看监控相关日志。 +测试过程中终端会周期性输出进度。完成后会输出主要配置、执行时间、结果矩阵和延迟矩阵。 + +### 2\.5 结果说明 + +测试执行信息会写入安装目录下的 `logs` 文件夹。是否生成 CSV 或写入结果数据库由 `TEST_DATA_PERSISTENCE` 等参数决定。 + +1. 结果矩阵 + +结果矩阵按操作类型统计以下指标： + +|指标|说明| +|---|---| +|`okOperation`|成功执行的请求或 SQL 数量| +|`okPoint`|写入成功的数据点数，或查询成功返回的数据点数| +|`failOperation`|执行失败的请求或 SQL 数量| +|`failPoint`|写入失败的数据点数；查询操作通常为 0| +|`throughput`|吞吐量，通常等于 `okPoint / Test elapsed time`| + +输出中的主要操作名称包括： + +- `INGESTION` + +- `PRECISE_POINT` + +- `TIME_RANGE` + +- `VALUE_RANGE` + +- `AGG_RANGE` + +- `AGG_VALUE` + +- `AGG_RANGE_VALUE` + +- `GROUP_BY` + +- `LATEST_POINT` + +- `RANGE_QUERY_DESC` + +- `VALUE_RANGE_QUERY_DESC` + +- `GROUP_BY_DESC` -#### 自动执行多项测试 +- `SET_OP_QUERY` -通常，除非与其他测试结果进行比较，否则单个测试是没有意义的。因此，我们提供了一个接口来通过一次启动执行多个测试。 +2. 延迟矩阵 -- 配置 routine +延迟矩阵以毫秒为单位，常见字段如下： -这个文件的每一行应该是每个测试过程会改变的参数（否则就变成复制测试）。例如，"例程"文件是： +|指标|说明| +|---|---| +|`AVG`|平均延迟| +|`MIN`|最小延迟| +|`P10`、`P25`、`MEDIAN`|低分位和中位延迟| +|`P75`、`P90`、`P95`|较高分位延迟| +|`P99`、`P999`|尾延迟| +|`MAX`|最大延迟| +|`SLOWEST_THREAD`|客户端线程中最大的累计操作时间| -```Plain -LOOP=10 DEVICE_NUMBER=100 TEST -LOOP=20 DEVICE_NUMBER=50 TEST -LOOP=50 DEVICE_NUMBER=20 TEST +测试结果还会给出元数据创建耗时和不包含元数据创建的 `Test elapsed time`。对比测试时，应确保各轮测试的硬件、数据量、客户端数、压缩配置和缓存状态一致。 + +3. 输出示例 + +测试完成后，终端会输出本次测试的主要配置、执行时间、结果矩阵和延迟矩阵。以下为一次纯写入测试的截断输出： + +```Plain Text +----------------------Main Configurations---------------------- +BENCHMARK_WORK_MODE=testWithDefaultPath +DB_SWITCH=IoTDB-200-SESSION_BY_TABLET +HOST=[127.0.0.1] + +GROUP_NUMBER=10 +DEVICE_NUMBER=50 +SENSOR_NUMBER=500 +SCHEMA_CLIENT_NUMBER=20 +DATA_CLIENT_NUMBER=20 + +OPERATION_PROPORTION=1:0:0:0:0:0:0:0:0:0:0:0:0 +LOOP=10000 +BATCH_SIZE_PER_WRITE=100 +--------------------------------------------------------------- + +Create schema cost 0.30 second +Test elapsed time (not include schema creation): 1238.79 second + +----------------------------------------------------------Result Matrix---------------------------------------------------------- +Operation okOperation okPoint failOperation failPoint throughput(point/s) +INGESTION 500000 25000000000 0 0 20180954.09 +--------------------------------------------------------------------------------------------------------------------------------- + +----------------------------------------------------------Latency (ms) Matrix---------------------------------------------------- +Operation AVG MIN P10 P25 MEDIAN P75 P90 P95 P99 P999 MAX SLOWEST_THREAD +INGESTION 37.78 1.67 2.02 2.29 2.86 4.14 5.62 7.43 759.69 5799.89 8309.40 1227561.44 +--------------------------------------------------------------------------------------------------------------------------------- ``` -然后依次执行3个LOOP参数分别为10、20、50的测试过程。 +从该输出可以得到： -> 注意： -> -> 您可以使用“LOOP=20 DEVICE_NUMBER=10 TEST”等格式更改每个测试中的多个参数，不允许使用不必要的空间。 关键字"TEST"意味着新的测试开始。如果您更改不同的参数，更改后的参数将保留在下一次测试中。 +- 元数据创建耗时为 `0.30` 秒，正式测试耗时为 `1238.79` 秒。 -- 开始测试 +- 共成功执行 `500000` 次写入操作，写入 `25000000000` 个数据点。 -配置文件routine后，您可以通过启动脚本启动多测试任务： +- `failOperation` 和 `failPoint` 均为 `0`，表示该轮测试没有记录到写入失败。 -```Bash -> ./rep-benchmark.sh +- 写入吞吐为 `20180954.09` 点/秒。 + +- 平均延迟为 `37.78` ms，P95 延迟为 `7.43` ms，P99 延迟为 `759.69` ms，最大延迟为 `8309.40` ms。 + +- `SLOWEST_THREAD` 是最慢客户端线程的累计操作时间，不是单次请求延迟。 + +示例中的数值仅用于展示输出格式。实际结果取决于硬件资源、网络环境、IoTDB 配置、数据规模和测试参数。 + +## 3\. 主要参数 + +### 3\.1 IoTDB 服务模型 + +IoTDB 2\.0\.x 支持树模型和表模型，通过以下参数选择： + +```Properties +IoTDB_DIALECT_MODE=tree +``` + +或： + +```Properties +IoTDB_DIALECT_MODE=table +``` + +相关约束如下： + +- 一个 IoTDB 实例在一次测试中只能选择一种 SQL 方言。 + +- 树模型要求 `DEVICE_NUMBER >= GROUP_NUMBER`。 + +- 表模型要求设备数是表数的整数倍，表数是数据库数的整数倍。 + +- 表模型下，数据客户端数应为表数的整数倍。 + + + +常用模型参数： + +|参数|示例|说明| +|---|---|---| +|`IoTDB_DIALECT_MODE`|`tree`|`tree` 或 `table`| +|`GROUP_NUMBER`|`1`|数据库数量；树模型下对应 database 数量| +|`IoTDB_TABLE_NUMBER`|`1`|表模型下创建的表数量| +|`IoTDB_TABLE_NAME_PREFIX`|`table_`|表名称前缀| +|`TABLE_TIME_COLUMN`|`time`|表模型时间列名称| +|`IoTDB_TABLE_WRITABLE_VIEW`|`false`|是否创建并使用可写视图| + +### 3\.2 工作模式 + +通过 `BENCHMARK_WORK_MODE` 选择工作模式。 + +|模式|配置值|说明| +|---|---|---| +|常规测试模式|`testWithDefaultPath`|执行写入、查询或混合负载| +|生成数据模式|`generateDataMode`|将 Benchmark 生成的数据集保存到 `FILE_PATH`| +|正确性写入模式|`verificationWriteMode`|从 `FILE_PATH` 加载数据集并写入数据库| +|正确性查询模式|`verificationQueryMode`|加载数据集并与数据库查询结果进行比对| + +示例： + +```Properties +BENCHMARK_WORK_MODE=testWithDefaultPath ``` -然后测试信息将显示在终端中。 +使用正确性写入和查询模式前，应先通过 `generateDataMode` 生成可复用数据集。 + +### 3\.3 服务器连接信息 + +|参数|示例|说明| +|---|---|---| +|`DB_SWITCH`|`IoTDB-200-SESSION_BY_TABLET`|数据库版本和连接方式| +|`HOST`|`127.0.0.1`|IoTDB 地址；多个地址使用英文逗号分隔| +|`PORT`|`6667`|原生服务端口，数量应与 `HOST` 一致| +|`USERNAME`|`root`|用户名| +|`PASSWORD`|`root`|密码| +|`DB_NAME`|`test`|测试使用的数据库名称| +|`REST_PORT`|`18080`|REST 服务端口| +|`REST_AUTHORIZATION`|`Basic cm9vdDpyb290`|REST 授权头| +|`ENABLE_AUTO_FETCH`|`false`|Session 是否自动刷新 DataNode 列表| + +清理数据相关参数： -> 注意： +```Properties +IS_DELETE_DATA=false +INIT_WAIT_TIME=1000 +``` + +> 警告：`IS_DELETE_DATA=true` 会在测试开始前清理目标数据库中的测试数据。仅可在专用测试环境中启用，并在执行前确认 `HOST`、`PORT`、`DB_NAME` 和账号权限。 +> > -> 如果您关闭终端或失去与客户端机器的连接，测试过程将终止。 如果输出传输到终端，则与任何其他情况相同。 -使用此接口通常需要很长时间，您可能希望将测试过程作为守护程序执行。这样，您可以通过启动脚本将测试任务作为守护程序启动： +### 3\.4 写入场景 -```Bash -> ./rep-benchmark.sh > /dev/null 2>&1 & +1. 数据规模和客户端 + +|参数|示例|说明| +|---|---|---| +|`DEVICE_NUMBER`|`100`|设备总数| +|`SENSOR_NUMBER`|`10`|每个设备的测点数；表模型下为测点列数量| +|`GROUP_NUMBER`|`1`|IoTDB database 数量| +|`SCHEMA_CLIENT_NUMBER`|`5`|注册元数据的客户端数| +|`DATA_CLIENT_NUMBER`|`10`|执行数据读写的客户端数| +|`IS_CLIENT_BIND`|`true`|是否将设备绑定到客户端| +|`REAL_INSERT_RATE`|`1.0`|实际参与写入的设备比例| +|`IS_SENSOR_TS_ALIGNMENT`|`true`|同一设备下测点时间戳是否对齐| + +2. 批量写入 + +|参数|示例|说明| +|---|---|---| +|`BATCH_SIZE_PER_WRITE`|`100`|每批每个设备写入的数据行数| +|`DEVICE_NUM_PER_WRITE`|`1`|每批写入涉及的设备数| +|`CREATE_SCHEMA`|`true`|写入前是否创建元数据| +|`START_TIME`|`2022-01-01T00:00:00+08:00`|生成数据的起始时间| + +单个批次的数据点数为： + +```Plain Text +DEVICE_NUM_PER_WRITE × SENSOR_NUMBER × BATCH_SIZE_PER_WRITE ``` -在这种情况下，如果您想知道发生了什么，可以通过以下命令查看日志信息： +`DEVICE_NUM_PER_WRITE` 必须能够整除单个数据客户端所负责的设备数。表模型下还应满足设备数、表数和单批设备数之间的整除约束。 -```Bash -> cd ./logs -> tail -f log_info.log +3. 写入节奏 + +|参数|示例|说明| +|---|---|---| +|`POINT_STEP`|`5000`|相邻生成时间戳的固定间隔| +|`OP_MIN_INTERVAL`|`0`|每个 loop 的最小执行间隔，单位 ms| +|`OP_MIN_INTERVAL_RANDOM`|`false`|是否在 `[0, OP_MIN_INTERVAL)` 中随机选择间隔| +|`INTERVAL_BETWEEN_WRITE_BATCH`|`0`|同一 loop 内相邻 batch 的最小间隔，单位 ms| +|`TIMESTAMP_PRECISION`|`ms`|时间戳精度| + +`OP_MIN_INTERVAL` 的特殊值： + +- `0`：不限制 loop 间隔。 + +- `-1`：使用 `POINT_STEP` 作为最小间隔。 + +- 正整数：如果当前 loop 耗时不足该值，则等待剩余时间。 + +4. 乱序写入 + +```Properties +IS_OUT_OF_ORDER=false +OUT_OF_ORDER_MODE=POISSON +OUT_OF_ORDER_RATIO=0.5 +IS_REGULAR_FREQUENCY=true ``` -## 4. 测试示例 +支持的乱序模式包括： + +- `POISSON`：按照泊松分布生成乱序时间戳。 -以IoTDB 2.0为例，使用IoT-Benchemark测试表模型写入及查询，修改IoT-Benchemark配置文件如下： +- `BATCH`：按批次生成乱序数据。 + +5. 数据类型 ```Properties -----------------------Main Configurations---------------------- -BENCHMARK_WORK_MODE=testWithDefaultPath -IoTDB_DIALECT_MODE=TABLE +INSERT_DATATYPE_PROPORTION=1:1:1:1:1:1:0:0:0:0:0 +``` + +各项顺序为： + +```Plain Text +BOOLEAN:INT32:INT64:FLOAT:DOUBLE:TEXT:STRING:BLOB:TIMESTAMP:DATE:OBJECT +``` + +各项数值表示相应数据类型的比例。 + +### 3\.5 查询场景 + +|参数|示例|说明| +|---|---|---| +|`QUERY_DEVICE_NUM`|`1`|每条查询涉及的设备数| +|`QUERY_SENSOR_NUM`|`1`|每条查询涉及的测点数| +|`QUERY_AGGREGATE_FUN`|`count`|聚合函数| +|`STEP_SIZE`|`0`|查询起始时间的变化步长，单位为 `POINT_STEP`| +|`QUERY_INTERVAL`|`250000`|查询起止时间间隔| +|`QUERY_LOWER_VALUE`|`-5`|值过滤条件下限| +|`GROUP_BY_TIME_UNIT`|`20000`|Group By 窗口大小| +|`QUERY_SET_OP_TYPE`|`union`|集合操作类型| +|`QUERY_SET_OP_NUM`|`2`|集合查询的子集合数量，至少为 2| +|`IS_RECENT_QUERY`|`false`|混合场景下是否优先查询最近写入的数据| +|`ENABLE_FIXED_QUERY`|`false`|是否让所有查询线程使用相同设备和测点组合| +|`RESULT_ROW_LIMIT`|`-1`|查询结果行数限制；`-1` 表示不使用限制| +|`ALIGN_BY_DEVICE`|`false`|是否使用 Align By Device| + +### 3\.6 操作比例 + +`OPERATION_PROPORTION` 用于定义写入和各类查询的比例，共包含 13 项： + +```Plain Text +写入:Q1:Q2:Q3:Q4:Q5:Q6:Q7:Q8:Q9:Q10:Q11:Q12 +``` + +例如，纯写入： + +```Properties +OPERATION_PROPORTION=1:0:0:0:0:0:0:0:0:0:0:0:0 +``` + +纯精确点查询： + +```Properties +OPERATION_PROPORTION=0:1:0:0:0:0:0:0:0:0:0:0:0 +``` + +操作类型如下。 + +|编号|操作类型|说明| +|---|---|---| +|写入|数据写入|按当前写入配置生成并写入数据| +|Q1|精确点查询|按时间戳和设备查询指定测点| +|Q2|时间范围查询|只限制起止时间的范围查询| +|Q3|带值过滤的范围查询|同时包含时间和值过滤条件| +|Q4|带时间过滤的聚合查询|在时间范围内执行聚合| +|Q5|带值过滤的聚合查询|在全时间范围内按值过滤并聚合| +|Q6|带时间和值过滤的聚合查询|同时包含时间和值过滤条件| +|Q7|时间分组聚合查询|Group By 查询| +|Q8|最近点查询|查询设备的最新数据点| +|Q9|倒序时间范围查询|按时间倒序返回范围查询结果| +|Q10|倒序带值过滤的范围查询|带值过滤并按时间倒序返回| +|Q11|倒序时间分组聚合查询|倒序 Group By 查询| +|Q12|集合操作查询|`union`、`intersect` 或 `except` 等集合操作| + +Q12 仅支持 IoTDB 2\.0 表模型。集合操作中的各子查询为范围查询。 + +### 3\.7 测试过程和结果持久化 + +```Properties +TEST_DATA_PERSISTENCE=None +``` + +支持的取值包括： + +- `None`：不将测试过程写入外部持久化介质。 + +- `CSV`：写入 CSV 文件。 + +- `MySQL`：写入 MySQL。 + +- `IoTDB`：写入指定的 IoTDB。 + +常用参数： + +|参数|示例|说明| +|---|---|---| +|`TEST_DATA_PERSISTENCE`|`None`|持久化方式| +|`RECORD_SPLIT`|`true`|是否将结果拆分为多条记录| +|`RECORD_SPLIT_MAX_LINE`|`10000000`|单表或单文件的最大记录数| +|`TEST_DATA_STORE_IP`|`127.0.0.1`|结果数据库地址| +|`TEST_DATA_STORE_PORT`|`6667`|结果数据库端口| +|`TEST_DATA_STORE_DB`|`result`|结果数据库名称| +|`TEST_DATA_STORE_USER`|`root`|结果数据库用户名| +|`TEST_DATA_STORE_PW`|`root`|结果数据库密码| +|`REMARK`|`write_test`|测试备注，可用于区分不同测试| +|`CSV_OUTPUT`|`true`|是否将最终结果写入 CSV| + +设置为 CSV 时，运行后会在 `data` 目录下生成相关记录；其中测试结果通常位于 `data/csvOutput`。无论是否开启持久化，测试日志均写入 `logs`。 + + + +日志输出频率可通过以下参数控制： + +```Properties +IS_QUIET_MODE=true +LOG_PRINT_INTERVAL=5 +RESULT_PRINT_INTERVAL=3600 +``` + +### 3\.8 自动化和集群测试 + +1. 限制测试时长 + +```Properties +TEST_MAX_TIME=3600000 +``` + +单位为毫秒。设置为 `0` 表示不限制。该参数不包含预注册元数据所消耗的时间。 + +2. 多 Benchmark 集群压测 + +在多台客户端机器上使用相同的总体数据规模配置，并设置： + +```Properties +BENCHMARK_CLUSTER=true +BENCHMARK_INDEX=0 +``` + +每个 Benchmark 实例必须使用不同的 `BENCHMARK_INDEX`，例如依次使用 `0`、`1`、`2`。所有客户端应保持数据库连接、数据规模和操作比例等配置一致。 + +3. 双写测试 + +IoT Benchmark 支持将同一份数据写入两个不同的数据库进行对比： + +```Properties +IS_DOUBLE_WRITE=true +ANOTHER_DB_SWITCH=<另一数据库类型> +ANOTHER_HOST=127.0.0.1 +ANOTHER_PORT=6667 +ANOTHER_USERNAME=root +ANOTHER_PASSWORD=root +ANOTHER_DB_NAME=test +``` + +双写模式不支持同一数据库不同版本之间的比较，也不支持 IoTDB 树模型与表模型之间的直接比较。 + +## 4\. 使用示例 + +本节使用小规模数据演示基本流程。正式性能测试应根据目标业务模型扩大设备数、测点数、客户端数和循环次数，并进行多轮预热和重复测试。 + +### 4\.1 写入测试示例 + +测试目标：使用 10 个数据客户端，模拟 100 台设备，每台设备包含 10 个测点，执行纯写入测试。 + +配置示例： + +```Properties +# 数据库连接 DB_SWITCH=IoTDB-200-SESSION_BY_TABLET +IoTDB_DIALECT_MODE=tree +HOST=127.0.0.1 +PORT=6667 +USERNAME=root +PASSWORD=root +DB_NAME=test + +# 安全设置：默认不自动删除已有数据 +IS_DELETE_DATA=false + +# 工作模式 +BENCHMARK_WORK_MODE=testWithDefaultPath +OPERATION_PROPORTION=1:0:0:0:0:0:0:0:0:0:0:0:0 + +# 数据规模 GROUP_NUMBER=1 -IoTDB_TABLE_NUMBER=1 -DEVICE_NUMBER=60 -REAL_INSERT_RATE=1.0 +DEVICE_NUMBER=100 SENSOR_NUMBER=10 -OPERATION_PROPORTION=1:0:0:0:0:0:0:0:0:0:0:0 -SCHEMA_CLIENT_NUMBER=10 +SCHEMA_CLIENT_NUMBER=5 DATA_CLIENT_NUMBER=10 -LOOP=10 +IS_SENSOR_TS_ALIGNMENT=true + +# 写入配置 +CREATE_SCHEMA=true BATCH_SIZE_PER_WRITE=10 DEVICE_NUM_PER_WRITE=1 -START_TIME=2025-01-01T00:00:00+08:00 +LOOP=100 POINT_STEP=1000 -INSERT_DATATYPE_PROPORTION=1:1:1:1:1:1:0:0:0:0 -VECTOR=true +OP_MIN_INTERVAL=0 +START_TIME=2026-01-01T00:00:00+08:00 +INSERT_DATATYPE_PROPORTION=1:1:1:1:1:1:0:0:0:0:0 + +# IoTDB 2.0.6 及以上版本可开启 +ENABLE_IOTDB_RPC_COMPRESSION=true + +# 输出 +TEST_DATA_PERSISTENCE=None +CSV_OUTPUT=true +REMARK=iotdb_2_write_test ``` -在被测数据库正常运行下，启动IoT-Benchmark服务，等待执行结束后，可查看打印结果如下： +启动测试： -```Shell -Create schema cost 0.88 second -Test elapsed time (not include schema creation): 4.60 second -----------------------------------------------------------Result Matrix---------------------------------------------------------- -Operation okOperation okPoint failOperation failPoint throughput(point/s) -INGESTION 600 60000 0 0 13054.42 -PRECISE_POINT 0 0 0 0 0.00 -TIME_RANGE 0 0 0 0 0.00 -VALUE_RANGE 0 0 0 0 0.00 -AGG_RANGE 0 0 0 0 0.00 -AGG_VALUE 0 0 0 0 0.00 -AGG_RANGE_VALUE 0 0 0 0 0.00 -GROUP_BY 0 0 0 0 0.00 -LATEST_POINT 0 0 0 0 0.00 -RANGE_QUERY_DESC 0 0 0 0 0.00 -VALUE_RANGE_QUERY_DESC 0 0 0 0 0.00 -GROUP_BY_DESC 0 0 0 0 0.00 ---------------------------------------------------------------------------------------------------------------------------------- +```Bash +./benchmark.sh +``` + +完成后重点查看： + +- `INGESTION` 的 `okPoint` 和 `failPoint`。 + +- 吞吐量 `throughput`。 + +- `AVG`、`P95`、`P99`、`MAX` 延迟。 + +- 是否存在连接超时、写入失败或服务端异常日志。 + +### 4\.2 查询测试示例 + +执行查询测试前，应确保目标数据库中已经存在与查询配置匹配的数据。建议复用写入测试生成的数据，并关闭自动删数和元数据创建。 + +以下示例同时执行多种查询： + +```Properties +DB_SWITCH=IoTDB-200-SESSION_BY_TABLET +IoTDB_DIALECT_MODE=tree +HOST=127.0.0.1 +PORT=6667 +USERNAME=root +PASSWORD=root +DB_NAME=test + +IS_DELETE_DATA=false +CREATE_SCHEMA=false +BENCHMARK_WORK_MODE=testWithDefaultPath + +GROUP_NUMBER=1 +DEVICE_NUMBER=100 +SENSOR_NUMBER=10 +SCHEMA_CLIENT_NUMBER=1 +DATA_CLIENT_NUMBER=10 + +# 不执行写入，Q1～Q11 各占相同比例；树模型不使用 Q12 +OPERATION_PROPORTION=0:1:1:1:1:1:1:1:1:1:1:1:0 +LOOP=100 + +QUERY_DEVICE_NUM=2 +QUERY_SENSOR_NUM=2 +QUERY_AGGREGATE_FUN=count +STEP_SIZE=1 +QUERY_INTERVAL=250000 +QUERY_LOWER_VALUE=-5 +GROUP_BY_TIME_UNIT=20000 +``` + +如果测试表模型集合查询，可将方言切换为 `table`，并为 Q12 设置比例： + +```Properties +IoTDB_DIALECT_MODE=table +OPERATION_PROPORTION=0:0:0:0:0:0:0:0:0:0:0:0:1 +QUERY_SET_OP_TYPE=union +QUERY_SET_OP_NUM=2 +``` + +### 4\.3 其他配置示例 + +1. 模拟真实写入速率 + +让每个 loop 的最小间隔与数据时间戳间隔一致： + +```Properties +POINT_STEP=1000 +OP_MIN_INTERVAL=-1 +``` + +如果希望写入请求在一个 loop 内均匀分布，可使用： + +```Properties +INTERVAL_BETWEEN_WRITE_BATCH=100 +``` + +2. 指定测试时长 + +测试 1 小时： + +```Properties +TEST_MAX_TIME=3600000 +``` + +应确保 `LOOP` 足够大，否则测试可能先因循环次数耗尽而结束。 + +3. 控制生成数据规律 + +```Properties +LINE_RATIO=1 +SIN_RATIO=1 +SQUARE_RATIO=1 +RANDOM_RATIO=1 +CONSTANT_RATIO=1 +DATA_SEED=666 +STRING_LENGTH=10 +DOUBLE_LENGTH=2 +``` + +固定 `DATA_SEED` 有助于在多轮测试中生成可重复的数据。 + +## 5\. 参考资料 ---------------------------------------------------------------------------Latency (ms) Matrix-------------------------------------------------------------------------- -Operation AVG MIN P10 P25 MEDIAN P75 P90 P95 P99 P999 MAX SLOWEST_THREAD -INGESTION 41.77 0.95 1.41 2.27 6.76 24.14 63.42 127.18 1260.92 1265.72 1265.49 2581.91 -PRECISE_POINT 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 -TIME_RANGE 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 -VALUE_RANGE 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 -AGG_RANGE 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 -AGG_VALUE 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 -AGG_RANGE_VALUE 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 -GROUP_BY 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 -LATEST_POINT 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 -RANGE_QUERY_DESC 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 -VALUE_RANGE_QUERY_DESC 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 -GROUP_BY_DESC 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ -``` \ No newline at end of file +- [IoT Benchmark 文档](https://github.com/thulab/iot-benchmark/tree/master/docs) diff --git a/src/zh/UserGuide/Master/Tree/Tools-System/Benchmark.md b/src/zh/UserGuide/Master/Tree/Tools-System/Benchmark.md index 6a4e9095f..78818a0d3 100644 --- a/src/zh/UserGuide/Master/Tree/Tools-System/Benchmark.md +++ b/src/zh/UserGuide/Master/Tree/Tools-System/Benchmark.md @@ -21,424 +21,800 @@ # 测试工具 -## 1. 基本概述 +IoT Benchmark 是面向工业物联网（IIoT）场景的时序数据库、实时数据库基准测试工具。本手册介绍工具的主要能力、支持的数据库及基本使用方法；安装、配置和测试案例主要以 IoTDB 2\.0\.x 为例，并同时覆盖树模型和表模型。 -IoT-benchmark 是基于 Java 和大数据环境开发的时序数据库基准测试工具，由清华大学软件学院研发并开源。它使用方便，支持多种写入以及查询方式，支持存储测试信息和结果以供进一步查询或分析，支持与 Tableau 集成以可视化测试结果。 +![](/img/bm2.png) -下图1-1囊括了测试基准流程及其他扩展功能。这些流程可以由IoT-benchmark 统一来完成。IoT Benchmark 支持多种工作负载，包括**纯写入、纯查询、写入查询混合**等，支持**软硬件系统监控、测试指标度量**等监控功能，还实现了**初始化数据库自动化、测试数据分析及系统参数优化**等功能。 +## 1\. 基本概述 -![](/img/bm1.png) +IoT Benchmark 可以按照配置生成周期性时间序列数据，对数据库执行写入和查询，并统计吞吐、延迟及成功率等指标。主要能力包括： -图1-1 +- 跨平台运行：支持 Linux、macOS 和 Windows。 -借鉴 YCSB 测试工具将工作负载生成、性能指标测量和数据库接口三个组件分离的设计思想，IoT-benchmark 的模块化设计如图1-2所示。与基于 YCSB 的测试工具系统不同的是，IoT-benchmark 增加了系统监控模块，支持测试数据和系统指标监控数据的持久化。此外也增加了一些特别针对时序数据场景的特殊负载测试功能，如支持物联网场景的批量写入和多种乱序数据写入模式。 +- 多种负载：支持纯写入、纯查询及读写混合负载。 -![](/img/bm2.png) +- 数据集生成：可将生成的数据保存到磁盘，供重复测试使用。 -图1-2 +- 正确性验证：可从磁盘加载数据集，执行写入和查询正确性验证。 +- 多数据库支持：可对多种时序数据库和实时数据库执行测试；IoTDB 2\.0\.x 支持 JDBC、REST 和多种 Session 写入方式。 -目前 IoT-benchmark 支持如下时间序列数据库、版本和连接方式： +- 结果持久化：支持将测试过程和结果保存到文件、CSV、MySQL 或 IoTDB。 -| 数据库 | 版本 | 连接方式 | -| :-------------- |:-----------| :------------------------------------------------------- | -| IoTDB | v1.x v2.x | jdbc、sessionByTablet、sessionByRecord、sessionByRecords | -| InfluxDB | v1.x v2.x | SDK | -| TimescaleDB | -- | jdbc | -| OpenTSDB | -- | Http Request | -| QuestDB | v6.0.7 | jdbc | -| TDengine | v2.2.0.2 | jdbc | -| VictoriaMetrics | v1.64.0 | Http Request | -| KairosDB | -- | Http Request | +- 测试扩展：支持乱序写入、批量写入、集群压测、双写对比和结果可视化等场景。 +### 1\.1 支持的数据库、版本和接入方式 -表1-1大数据测试基准对比 +IoT Benchmark 支持如下数据库和版本。测试时通过 `DB_SWITCH` 选择对应的数据库、版本及连接或写入方式。 -## 2. 安装运行 +|数据库|支持版本|`DB_SWITCH`| +|---|---|---| +|IoTDB|v1\.x|`IoTDB-130-JDBC`、`IoTDB-130-REST`、`IoTDB-130-SESSION_BY_TABLET`、`IoTDB-130-SESSION_BY_RECORD`、`IoTDB-130-SESSION_BY_RECORDS`| +|IoTDB|v2\.x|`IoTDB-200-JDBC`、`IoTDB-200-REST`、`IoTDB-200-SESSION_BY_TABLET`、`IoTDB-200-SESSION_BY_RECORD`、`IoTDB-200-SESSION_BY_RECORDS`| +|InfluxDB|v1\.x|`InfluxDB`| +|InfluxDB|v2\.x|`InfluxDB-2.x`| +|QuestDB|v6\.0\.7|`QuestDB`| +|Microsoft SQL Server|2016 SP2|`MSSQLSERVER`| +|VictoriaMetrics|v1\.64\.0|`VictoriaMetrics`| +|SQLite|—|`SQLite`| +|OpenTSDB|2\.4\.1|`OpenTSDB`| +|KairosDB|—|`KairosDB`| +|TimescaleDB|—|`TimescaleDB`| +|TimescaleDB Cluster|Cluster|`TimescaleDB-Cluster`| +|TDengine|2\.2\.0\.2|`TDengine`| +|TDengine|3\.0\.1|`TDengine-3`| +|DolphinDB|v2\.x|`DolphinDB-2-MTW`、`DolphinDB-2-PTA`| +|DolphinDB|v3\.x|`DolphinDB-3-MTW`、`DolphinDB-3-PTA`| +|CnosDB|—|`CnosDB`| -### 2.1 前置条件 +其中： -1. Java 8 -2. Maven 3.6+ -3. 对应的合适版本的数据库，如 Apache IoTDB 2.0 +- IoTDB 的接入方式包括 JDBC、REST、Session by Tablet、Session by Record 和 Session by Records。 +- DolphinDB 的 `MTW` 表示 `MultithreadedTableWriter`，按行缓冲写入；`PTA` 表示 `PartitionedTableAppender`，按批次进行列式整表追加。 +- 数据库版本、驱动和服务端必须相互匹配。使用其他数据库时，还需要配置相应的连接及扩展参数。 -### 2.2 获取方式 +### 1\.2 IoTDB 2\.0\.x 接入方式 -- 获取二进制包：进入[这里](https://github.com/thulab/iot-benchmark/releases) 下载需要的安装包。下载下来为一个压缩文件，选择文件夹解压即可使用。 -- 源代码编译（可用 Apache IoTDB 2.0 的测试）： - - 第一步（编译 IoTDB Session 最新包）：进入[官网](https://github.com/apache/iotdb/tree/rc/2.0.5)下载 IoTDB 源码，在根目录下运行命令 mvn clean package install -pl session -am -DskipTests 编译 IoTDB Session 的最新包。 - - 第二步（编译 IoTDB Benchmark 测试包）：进入[官网](https://github.com/thulab/iot-benchmark)下载源码，在根目录下运行 mvn clean package install -pl iotdb-2.0 -am -DskipTests 编译测试 Apache IoTDB 2.0版本的测试包，测试包位置与根目录的相对路径为 ./iotdb-2.0/target/iotdb-2.0-0.0.1/iotdb-2.0-0.0.1 +后续安装、配置和案例以 IoTDB 2\.0\.x 为主。支持的接入方式如下。 +|接入方式|`DB_SWITCH`|说明| +|---|---|---| +|JDBC|`IoTDB-200-JDBC`|通过 JDBC 执行写入和查询| +|REST|`IoTDB-200-REST`|通过 IoTDB REST 接口执行测试| +|Session by Tablet|`IoTDB-200-SESSION_BY_TABLET`|使用 Tablet 批量写入| +|Session by Record|`IoTDB-200-SESSION_BY_RECORD`|逐条写入记录| +|Session by Records|`IoTDB-200-SESSION_BY_RECORDS`|批量写入多条记录| -### 2.3 测试包结构 +`IoTDB-200-SESSION_BY_TABLET` 适用于批量写入测试。 -测试包的目录结构如下所示。其中测试配置文件为conf/config.properties，测试启动脚本为benchmark\.sh (Linux & MacOS) 和 benchmark.bat (Windows)，详细文件用途见下表所示。 +## 2\. 安装运行 -```Shell --rw-r--r--. 1 root root 2881 1月 10 01:36 benchmark.bat --rwxr-xr-x. 1 root root 314 1月 10 01:36 benchmark.sh -drwxr-xr-x. 2 root root 24 1月 10 01:36 bin --rwxr-xr-x. 1 root root 1140 1月 10 01:36 cli-benchmark.sh -drwxr-xr-x. 2 root root 107 1月 10 01:36 conf -drwxr-xr-x. 2 root root 4096 1月 10 01:38 lib --rw-r--r--. 1 root root 11357 1月 10 01:36 LICENSE --rwxr-xr-x. 1 root root 939 1月 10 01:36 rep-benchmark.sh --rw-r--r--. 1 root root 14 1月 10 01:36 routine -``` +本章以 IoTDB 2\.0\.x 作为被测数据库。 -| 名称 | 子文件 | 用途 | -| :--------------- | :---------------- | :------------------------ | -| benchmark.bat | - | Windows环境运行启动脚本 | -| benchmark.sh | - | Linux/Mac环境运行启动脚本 | -| bin | startup.sh | 初始化脚本文件夹 | -| conf | config.properties | 测试场景配置文件 | -| lib | - | 依赖库文件 | -| LICENSE | - | 许可文件 | -| cli-benchmark.sh | - | 一键化启动脚本 | -| routine | | 多项测试配置文件 | -| rep-benchmark.sh | | 多项测试启动脚本 | +### 2\.1 前置条件 -表2-1文件和文件夹列表用途 +1. 使用 IoT Benchmark 前需要准备： -### 2.4 执行测试 + - Java 17。 -1. 按照测试需求修改配置文件，主要参数介绍见第3节，对应配置文件为conf/config.properties，**比如测试Apache IoTDB 2.0，则需要修改 DB_SWITCH=IoTDB-200-SESSION_BY_TABLET** -2. 启动被测时间序列数据库 -3. 通过运行 -4. 启动IoT-benchmark执行测试。执行中观测被测时间序列数据库和IoT-benchmark状态，执行完毕后查看结果和分析测试过程。 + - Maven。 -### 2.5 结果说明 + - 已安装并能够正常运行的 IoTDB 2\.0\.x。 -测试的所有日志文件被存放于 logs 文件夹下，测试的结果在测试完成后被存放到 data/csvOutput 文件夹下，例如测试后我们得到了如下的结果矩阵： + - 足够的客户端 CPU、内存、磁盘和网络资源。 -![](/img/bm4.png) +2. 环境相关说明： -- Result Matrix - - OkOperation：成功的对应操作次数 - - OkPoint：对于写入操作，是成功写入的点数；对于查询操作，是成功查询到的点数。 - - FailOperation：失败的对应操作次数 - - FailPoint：对于写入操作是写入失败的点数 -- Latency(mx) Matrix - - AVG：操作平均耗时 - - MIN：操作最小耗时 - - Pn：操作整体分布的对应分位值，比如P25是下四分位数 + - 推荐使用 Linux 或 macOS 执行测试。 + - Windows 使用安装包根目录下的 `benchmark.bat` 启动测试。 + - Linux 和 macOS 使用 `benchmark.sh` 启动测试。 -## 3. 主要参数 + - CSV 记录模式中的部分系统信息采集能力仅支持 Linux。 + +> 注意：不要将 IoT Benchmark 与被测 IoTDB 部署在资源相互竞争的环境中。正式性能测试建议使用独立服务器，并关闭无关服务。 +> +> +### 2\.2 获取方式 -### 3.1 IoTDB服务模型 +1. 获取发行包 -参数`IoTDB_DIALECT_MODE`支持tree、table,默认值为tree。 +可以从 [IoT Benchmark Releases](https://github.com/thulab/iot-benchmark/releases) 下载与测试目标匹配的发行包，解压后使用。 + +2. 从源码构建 + +克隆仓库： + +```Bash +git clone https://github.com/thulab/iot-benchmark.git +cd iot-benchmark +``` -- 当被测数据库为IoTDB-2.0及以上版本时需指定sql_dialect, 并且一个IoTDB只能指定一种。 -- sql_dialect等于tree时，要满足：device数量 >= database数量 +在项目根目录执行： -### 3.2 工作模式 +```Bash +mvn clean package -Dmaven.test.skip=true +``` -工作模式参数“`BENCHMARK_WORK_MODE`”可选项有如下四种模式： +构建完成后，IoTDB 2\.0\.x 安装包位于： -- 常用测试模式：结合配置`OPERATION_PROPORTION`参数达到“纯写入”、“纯查询”和“读写混合”的测试操作。 -- 生成数据模式：为了生成可以重复使用的数据集，iot-benchmark提供生成数据集的模式，生成数据集到FILE_PATH，以供后续使用正确性写入模式和正确性查询模式使用。 -- 单数据库正确性写入模式：为了验证数据集写入的正确性，您可以使用该模式写入生成数据模式中生成的数据集，目前该模式仅支持IoTDB v1.0 及更新的版本和InfluxDB v1.x。 -- 单数据库正确性查询模式：在运行这个模式之前需要先使用正确性写入模式写入数据到数据库。为了验证数据集写入的正确性，您可以使用该模式查询写入到数据库中的数据集，目前该模式仅支持IoTDB v1.0 和 InfluxDB v1.x。 +```Plain Text +iotdb-2.0/target/iot-benchmark-iotdb-2.0/iot-benchmark-iotdb-2.0 +``` -| **模式名称** | **BENCHMARK_WORK_MODE** | **模式内容** | -| :--------------------- | :---------------------- | :------------------------------- | -| 常规测试模式 | testWithDefaultPath | 支持多种读和写操作的混合负载 | -| 生成数据模式 | generateDataMode | 生成Benchmark本身识别的数据 | -| 单数据库正确性写入模式 | verificationWriteMode | 需要配置 FILE_PATH 以及 DATA_SET | -| 单数据库正确性查询模式 | verificationQueryMode | 需要配置 FILE_PATH 以及 DATA_SET | +进入安装目录： -### 3.3 服务器连接信息 +```Bash +cd iotdb-2.0/target/iot-benchmark-iotdb-2.0/iot-benchmark-iotdb-2.0 +``` -工作模式指定后，被测时序数据库的信息会通过如下参数告知IoT-benchmark +### 2\.3 测试包结构 -| **参数名称** | **类型** | **示例** | **系统描述** | -| :----------- | :------- | :-------------------------- | :---------------------------------------------- | -| DB_SWITCH | 字符串 | IoTDB-200-SESSION_BY_TABLET | 被测时序数据库类型 | -| HOST | 字符串 | 127.0.0.1 | 被测时序数据库网络地址 | -| PORT | 整数 | 6667 | 被测时序数据库网络端口 | -| USERNAME | 字符串 | root | 被测时序数据库登录用户名 | -| PASSWORD | 字符串 | root | 被测时序数据库登录用户的密码 | -| DB_NAME | 字符串 | test | 被测时序数据库名称 | -| TOKEN | 字符串 | | 被测时序数据库连接认证Token（InfluxDB 2.0使用） | +安装包中的常用文件和目录如下。 -### 3.4 写入场景 +|名称|用途| +|---|---| +|`benchmark.sh`|Linux、macOS 启动脚本| +|`benchmark.bat`|Windows 启动脚本| +|`conf/config.properties`|测试场景配置文件| +|`lib/`|运行依赖库| +|`logs/`|测试日志，首次运行后生成| +|`data/`|数据集或持久化结果目录，按工作模式和持久化配置生成| -| **参数名称** | **类型** | **示例** | **系统描述** | -| :------------------------- | :------- | :------------------------ | :----------------------------------------------------------- | -| CLIENT_NUMBER | 整数 | 100 | 客户端总数 | -| GROUP_NUMBER | 整数 | 20 | 数据库的数量；仅针对IoTDB。 | -| DEVICE_NUMBER | 整数 | 100 | 设备总数 | -| SENSOR_NUMBER | 整数 | 300 | 每个设备的传感器总数； **如果使用 IoTDB 表模型，则控制属性列数量** | -| INSERT_DATATYPE_PROPORTION | 字符串 | 1:1:1:1:1:1 | 设备的数据类型比例，BOOLEAN:INT32:INT64:FLOAT:DOUBLE:TEXT | -| POINT_STEP | 整数 | 1000 | 数据间时间戳间隔，即生成的数据两个时间戳之间的固定长度。 | -| OP_MIN_INTERVAL | 整数 | 0 | 操作最小执行间隔：若操作耗时大于该值则立即执行下一个，否则等待 (OP_MIN_INTERVAL-实际执行时间) ms；如果为0，则参数不生效；如果为-1，则其值和POINT_STEP一致 | -| IS_OUT_OF_ORDER | 布尔 | false | 是否乱序写入 | -| OUT_OF_ORDER_RATIO | 浮点数 | 0.3 | 乱序写入的数据比例 | -| BATCH_SIZE_PER_WRITE | 整数 | 1 | 批写入数据行数（一次写入多少行数据） | -| START_TIME | 时间 | 2022-10-30T00:00:00+08:00 | 写入数据的开始时间戳；以该时间戳为起点开始模拟创建数据时间戳。 | -| LOOP | 整数 | 86400 | 总操作次数：具体每种类型操作会按OPERATION_PROPORTION定义的比例划分 | -| OPERATION_PROPORTION | 字符 | 1:0:0:0:0:0:0:0:0:0:0 | # 各操作的比例,按照顺序为 写入:Q1:Q2:Q3:Q4:Q5:Q6:Q7:Q8:Q9:Q10, 请注意使用英文冒号。比例中的每一项是整数。 | +### 2\.4 执行测试 +1. 启动 IoTDB -### 3.5 查询场景 +首先启动目标 IoTDB 2\.0\.x，并确认客户端能够访问其服务端口。默认原生接口端口为 `6667`。 -| 参数名称 | 类型 | 示例 | 系统描述 | -| :------------------- | :--- | :-------------------- | :----------------------------------------------------------- | -| QUERY_DEVICE_NUM | 整数 | 2 | 每条查询语句中查询涉及到的设备数量 | -| QUERY_SENSOR_NUM | 整数 | 2 | 每条查询语句中查询涉及到的传感器数量 | -| QUERY_AGGREGATE_FUN | 字符 | count | 在聚集查询中使用的聚集函数，比如count、avg、sum、max_time等 | -| STEP_SIZE | 整数 | 1 | 时间过滤条件的时间起点变化步长，若设为0则每个查询的时间过滤条件是一样的，单位：POINT_STEP | -| QUERY_INTERVAL | 整数 | 250000 | 起止时间的查询中开始时间与结束时间之间的时间间隔，和Group By中的时间间隔 | -| QUERY_LOWER_VALUE | 整数 | -5 | 条件查询子句时的参数，where xxx > QUERY_LOWER_VALUE | -| GROUP_BY_TIME_UNIT | 整数 | 20000 | Group by语句中的组的大小 | -| LOOP | 整数 | 10 | 总操作次数：具体每种类型操作会按OPERATION_PROPORTION定义的比例划分 | -| OPERATION_PROPORTION | 字符 | 0:0:0:0:0:0:0:0:0:0:1 | 写入:Q1:Q2:Q3:Q4:Q5:Q6:Q7:Q8:Q9:Q10 | +2. 修改配置 +- 编辑 `conf/config.properties` -### 3.6 操作比例 +- 最小连接配置示例： -| 编号 | 查询类型 | IoTDB 示例 SQL | -| :--- | :--------------------------- | :----------------------------------------------------------- | -| Q1 | 精确点查询 | select v1 from root.db.d1 where time = ? | -| Q2 | 时间范围查询 | select v1 from root.db.d1 where time > ? and time < ? | -| Q3 | 带值过滤的时间范围查询 | select v1 from root.db.d1 where time > ? and time < ? and v1 > ? | -| Q4 | 时间范围聚合查询 | select count(v1) from root.db.d1 where and time > ? and time < ? | -| Q5 | 带值过滤的全时间范围聚合查询 | select count(v1) from root.db.d1 where v1 > ? | -| Q6 | 带值过滤的时间范围聚合查询 | select count(v1) from root.db.d1 where v1 > ? and time > ? and time < ? | -| Q7 | 时间分组聚合查询 | select count(v1) from root.db.d1 group by ([?, ?), ?, ?) | -| Q8 | 最新点查询 | select last v1 from root.db.d1 | -| Q9 | 倒序范围查询 | select v1 from root.sg.d1 where time > ? and time < ? order by time desc | -| Q10 | 倒序带值过滤的范围查询 | select v1 from root.sg.d1 where time > ? and time < ? and v1 > ? order by time desc | +```Properties +DB_SWITCH=IoTDB-200-SESSION_BY_TABLET +IoTDB_DIALECT_MODE=tree +HOST=127.0.0.1 +PORT=6667 +USERNAME=root +PASSWORD=root +DB_NAME=test +``` -### 3.7 测试过程和测试结果持久化 +- 使用表模型时改为： -IoT-benchmark目前支持通过配置参数将测试过程和测试结果持久化: +```Properties +IoTDB_DIALECT_MODE=table +``` -| **参数名称** | **类型** | **示例** | **系统描述** | -| :-------------------- | :------- | :-------- | :----------------------------------------------------------- | -| TEST_DATA_PERSISTENCE | 字符串 | None | 结果持久化选择，支持None，IoTDB，MySQL和CSV | -| RECORD_SPLIT | 布尔 | true | 是否将结果划分后输出到多个记录, IoTDB 暂时不支持 | -| RECORD_SPLIT_MAX_LINE | 整数 | 10000000 | 记录行数的上限（每个数据库表或CSV文件按照总行数为1千万切分存放） | -| TEST_DATA_STORE_IP | 字符串 | 127.0.0.1 | 输出数据库的IP地址 | -| TEST_DATA_STORE_PORT | 整数 | 6667 | 输出数据库的端口号 | -| TEST_DATA_STORE_DB | 字符串 | result | 输出数据库的名称 | -| TEST_DATA_STORE_USER | 字符串 | root | 输出数据库的用户名 | -| TEST_DATA_STORE_PW | 字符串 | root | 输出数据库的用户密码 | - -- 如果我们设置“`TEST_DATA_PERSISTENCE=CSV`”，测试执行时和执行完毕后我们可以在IoT-benchmark根目录下看到新生成的`data`文件夹，其下包含`csv`文件夹记录测试过程；`csvOutput`文件夹记录测试结果。 -- 如果我们设置“`TEST_DATA_PERSISTENCE=MySQL`”，它会在测试开始前在指定的MySQL数据库中创建命名如“testWithDefaultPath_被测数据库名称_备注_测试启动时间”的数据表记录测试过程；会在名为“CONFIG”的数据表（如果不存在则创建该表），写入本次测试的配置信息；当测试完成时会在名为“FINAL_RESULT”的数据表（如果不存在则创建该表）中写入本次测试结果。 - -### 3.8 自动化脚本 - -#### 一键化启动脚本 - -您可以通过`cli-benchmark.sh`脚本一键化启动IoTDB、监控的IoTDB Benchmark和测试的IoTDB Benchmark，但需要注意该脚本启动时会清理IoTDB中的**所有数据**，请谨慎使用。 - -首先，您需要修改`cli-benchmark.sh`中的`IOTDB_HOME`参数为您本地的IoTDB所在的文件夹。 - -然后您可以使用脚本启动测试 +- 如果选择 REST： -```Bash -> ./cli-benchmark.sh +```Properties +DB_SWITCH=IoTDB-200-REST +REST_PORT=18080 +REST_AUTHORIZATION=Basic cm9vdDpyb290 ``` -测试完成后您可以在`logs`文件夹中查看测试相关日志，在`server-logs`文件夹中查看监控相关日志。 +`REST_AUTHORIZATION `用于配置 REST 接口的 `Basic Authentication` 信息。示例使用用户名 root、密码 root。 -#### 自动执行多项测试 +3. 检查 RPC 压缩兼容性 -通常，除非与其他测试结果进行比较，否则单个测试是没有意义的。因此，我们提供了一个接口来通过一次启动执行多个测试。 +IoT Benchmark 2\.0 默认开启 IoTDB RPC 压缩： -- 配置 routine +```Properties +ENABLE_IOTDB_RPC_COMPRESSION=true +``` -这个文件的每一行应该是每个测试过程会改变的参数（否则就变成复制测试）。例如，"例程"文件是： +该功能要求被测 IoTDB 版本为 2\.0\.6 或更高版本。如果测试 2\.0\.6 之前的 IoTDB 2\.0\.x，请设置： -```Plain -LOOP=10 DEVICE_NUMBER=100 TEST -LOOP=20 DEVICE_NUMBER=50 TEST -LOOP=50 DEVICE_NUMBER=20 TEST +```Properties +ENABLE_IOTDB_RPC_COMPRESSION=false ``` -然后依次执行3个LOOP参数分别为10、20、50的测试过程。 +Thrift 压缩是另一项独立配置。如果启用： -> 注意： -> -> 您可以使用“LOOP=20 DEVICE_NUMBER=10 TEST”等格式更改每个测试中的多个参数，不允许使用不必要的空间。 关键字"TEST"意味着新的测试开始。如果您更改不同的参数，更改后的参数将保留在下一次测试中。 +```Properties +ENABLE_THRIFT_COMPRESSION=true +``` -- 开始测试 +还需要在 IoTDB 的 `iotdb-datanode.properties` 中设置： -配置文件routine后，您可以通过启动脚本启动多测试任务： +```Properties +dn_rpc_thrift_compression_enable=true +``` + +4. 启动 Benchmark + +Linux 或 macOS： ```Bash -> ./rep-benchmark.sh +./benchmark.sh +``` + +Windows： + +```Plain Text +benchmark.bat ``` -然后测试信息将显示在终端中。 +测试过程中终端会周期性输出进度。完成后会输出主要配置、执行时间、结果矩阵和延迟矩阵。 + +### 2\.5 结果说明 + +测试执行信息会写入安装目录下的 `logs` 文件夹。是否生成 CSV 或写入结果数据库由 `TEST_DATA_PERSISTENCE` 等参数决定。 + +1. 结果矩阵 + +结果矩阵按操作类型统计以下指标： + +|指标|说明| +|---|---| +|`okOperation`|成功执行的请求或 SQL 数量| +|`okPoint`|写入成功的数据点数，或查询成功返回的数据点数| +|`failOperation`|执行失败的请求或 SQL 数量| +|`failPoint`|写入失败的数据点数；查询操作通常为 0| +|`throughput`|吞吐量，通常等于 `okPoint / Test elapsed time`| + +输出中的主要操作名称包括： + +- `INGESTION` + +- `PRECISE_POINT` + +- `TIME_RANGE` + +- `VALUE_RANGE` + +- `AGG_RANGE` + +- `AGG_VALUE` + +- `AGG_RANGE_VALUE` + +- `GROUP_BY` + +- `LATEST_POINT` + +- `RANGE_QUERY_DESC` + +- `VALUE_RANGE_QUERY_DESC` + +- `GROUP_BY_DESC` + +- `SET_OP_QUERY` + +2. 延迟矩阵 + +延迟矩阵以毫秒为单位，常见字段如下： + +|指标|说明| +|---|---| +|`AVG`|平均延迟| +|`MIN`|最小延迟| +|`P10`、`P25`、`MEDIAN`|低分位和中位延迟| +|`P75`、`P90`、`P95`|较高分位延迟| +|`P99`、`P999`|尾延迟| +|`MAX`|最大延迟| +|`SLOWEST_THREAD`|客户端线程中最大的累计操作时间| + +测试结果还会给出元数据创建耗时和不包含元数据创建的 `Test elapsed time`。对比测试时，应确保各轮测试的硬件、数据量、客户端数、压缩配置和缓存状态一致。 + +3. 输出示例 + +测试完成后，终端会输出本次测试的主要配置、执行时间、结果矩阵和延迟矩阵。以下为一次纯写入测试的截断输出： + +```Plain Text +----------------------Main Configurations---------------------- +BENCHMARK_WORK_MODE=testWithDefaultPath +DB_SWITCH=IoTDB-200-SESSION_BY_TABLET +HOST=[127.0.0.1] + +GROUP_NUMBER=10 +DEVICE_NUMBER=50 +SENSOR_NUMBER=500 +SCHEMA_CLIENT_NUMBER=20 +DATA_CLIENT_NUMBER=20 + +OPERATION_PROPORTION=1:0:0:0:0:0:0:0:0:0:0:0:0 +LOOP=10000 +BATCH_SIZE_PER_WRITE=100 +--------------------------------------------------------------- + +Create schema cost 0.30 second +Test elapsed time (not include schema creation): 1238.79 second + +----------------------------------------------------------Result Matrix---------------------------------------------------------- +Operation okOperation okPoint failOperation failPoint throughput(point/s) +INGESTION 500000 25000000000 0 0 20180954.09 +--------------------------------------------------------------------------------------------------------------------------------- + +----------------------------------------------------------Latency (ms) Matrix---------------------------------------------------- +Operation AVG MIN P10 P25 MEDIAN P75 P90 P95 P99 P999 MAX SLOWEST_THREAD +INGESTION 37.78 1.67 2.02 2.29 2.86 4.14 5.62 7.43 759.69 5799.89 8309.40 1227561.44 +--------------------------------------------------------------------------------------------------------------------------------- +``` + +从该输出可以得到： + +- 元数据创建耗时为 `0.30` 秒，正式测试耗时为 `1238.79` 秒。 + +- 共成功执行 `500000` 次写入操作，写入 `25000000000` 个数据点。 + +- `failOperation` 和 `failPoint` 均为 `0`，表示该轮测试没有记录到写入失败。 + +- 写入吞吐为 `20180954.09` 点/秒。 + +- 平均延迟为 `37.78` ms，P95 延迟为 `7.43` ms，P99 延迟为 `759.69` ms，最大延迟为 `8309.40` ms。 + +- `SLOWEST_THREAD` 是最慢客户端线程的累计操作时间，不是单次请求延迟。 + +示例中的数值仅用于展示输出格式。实际结果取决于硬件资源、网络环境、IoTDB 配置、数据规模和测试参数。 + +## 3\. 主要参数 + +### 3\.1 IoTDB 服务模型 + +IoTDB 2\.0\.x 支持树模型和表模型，通过以下参数选择： + +```Properties +IoTDB_DIALECT_MODE=tree +``` + +或： + +```Properties +IoTDB_DIALECT_MODE=table +``` + +相关约束如下： + +- 一个 IoTDB 实例在一次测试中只能选择一种 SQL 方言。 + +- 树模型要求 `DEVICE_NUMBER >= GROUP_NUMBER`。 + +- 表模型要求设备数是表数的整数倍，表数是数据库数的整数倍。 + +- 表模型下，数据客户端数应为表数的整数倍。 + + + +常用模型参数： + +|参数|示例|说明| +|---|---|---| +|`IoTDB_DIALECT_MODE`|`tree`|`tree` 或 `table`| +|`GROUP_NUMBER`|`1`|数据库数量；树模型下对应 database 数量| +|`IoTDB_TABLE_NUMBER`|`1`|表模型下创建的表数量| +|`IoTDB_TABLE_NAME_PREFIX`|`table_`|表名称前缀| +|`TABLE_TIME_COLUMN`|`time`|表模型时间列名称| +|`IoTDB_TABLE_WRITABLE_VIEW`|`false`|是否创建并使用可写视图| + +### 3\.2 工作模式 + +通过 `BENCHMARK_WORK_MODE` 选择工作模式。 -> 注意： +|模式|配置值|说明| +|---|---|---| +|常规测试模式|`testWithDefaultPath`|执行写入、查询或混合负载| +|生成数据模式|`generateDataMode`|将 Benchmark 生成的数据集保存到 `FILE_PATH`| +|正确性写入模式|`verificationWriteMode`|从 `FILE_PATH` 加载数据集并写入数据库| +|正确性查询模式|`verificationQueryMode`|加载数据集并与数据库查询结果进行比对| + +示例： + +```Properties +BENCHMARK_WORK_MODE=testWithDefaultPath +``` + +使用正确性写入和查询模式前，应先通过 `generateDataMode` 生成可复用数据集。 + +### 3\.3 服务器连接信息 + +|参数|示例|说明| +|---|---|---| +|`DB_SWITCH`|`IoTDB-200-SESSION_BY_TABLET`|数据库版本和连接方式| +|`HOST`|`127.0.0.1`|IoTDB 地址；多个地址使用英文逗号分隔| +|`PORT`|`6667`|原生服务端口，数量应与 `HOST` 一致| +|`USERNAME`|`root`|用户名| +|`PASSWORD`|`root`|密码| +|`DB_NAME`|`test`|测试使用的数据库名称| +|`REST_PORT`|`18080`|REST 服务端口| +|`REST_AUTHORIZATION`|`Basic cm9vdDpyb290`|REST 授权头| +|`ENABLE_AUTO_FETCH`|`false`|Session 是否自动刷新 DataNode 列表| + +清理数据相关参数： + +```Properties +IS_DELETE_DATA=false +INIT_WAIT_TIME=1000 +``` + +> 警告：`IS_DELETE_DATA=true` 会在测试开始前清理目标数据库中的测试数据。仅可在专用测试环境中启用，并在执行前确认 `HOST`、`PORT`、`DB_NAME` 和账号权限。 +> > -> 如果您关闭终端或失去与客户端机器的连接，测试过程将终止。 如果输出传输到终端，则与任何其他情况相同。 -使用此接口通常需要很长时间，您可能希望将测试过程作为守护程序执行。这样，您可以通过启动脚本将测试任务作为守护程序启动： +### 3\.4 写入场景 -```Bash -> ./rep-benchmark.sh > /dev/null 2>&1 & +1. 数据规模和客户端 + +|参数|示例|说明| +|---|---|---| +|`DEVICE_NUMBER`|`100`|设备总数| +|`SENSOR_NUMBER`|`10`|每个设备的测点数；表模型下为测点列数量| +|`GROUP_NUMBER`|`1`|IoTDB database 数量| +|`SCHEMA_CLIENT_NUMBER`|`5`|注册元数据的客户端数| +|`DATA_CLIENT_NUMBER`|`10`|执行数据读写的客户端数| +|`IS_CLIENT_BIND`|`true`|是否将设备绑定到客户端| +|`REAL_INSERT_RATE`|`1.0`|实际参与写入的设备比例| +|`IS_SENSOR_TS_ALIGNMENT`|`true`|同一设备下测点时间戳是否对齐| + +2. 批量写入 + +|参数|示例|说明| +|---|---|---| +|`BATCH_SIZE_PER_WRITE`|`100`|每批每个设备写入的数据行数| +|`DEVICE_NUM_PER_WRITE`|`1`|每批写入涉及的设备数| +|`CREATE_SCHEMA`|`true`|写入前是否创建元数据| +|`START_TIME`|`2022-01-01T00:00:00+08:00`|生成数据的起始时间| + +单个批次的数据点数为： + +```Plain Text +DEVICE_NUM_PER_WRITE × SENSOR_NUMBER × BATCH_SIZE_PER_WRITE ``` -在这种情况下，如果您想知道发生了什么，可以通过以下命令查看日志信息： +`DEVICE_NUM_PER_WRITE` 必须能够整除单个数据客户端所负责的设备数。表模型下还应满足设备数、表数和单批设备数之间的整除约束。 -```Bash -> cd ./logs -> tail -f log_info.log +3. 写入节奏 + +|参数|示例|说明| +|---|---|---| +|`POINT_STEP`|`5000`|相邻生成时间戳的固定间隔| +|`OP_MIN_INTERVAL`|`0`|每个 loop 的最小执行间隔，单位 ms| +|`OP_MIN_INTERVAL_RANDOM`|`false`|是否在 `[0, OP_MIN_INTERVAL)` 中随机选择间隔| +|`INTERVAL_BETWEEN_WRITE_BATCH`|`0`|同一 loop 内相邻 batch 的最小间隔，单位 ms| +|`TIMESTAMP_PRECISION`|`ms`|时间戳精度| + +`OP_MIN_INTERVAL` 的特殊值： + +- `0`：不限制 loop 间隔。 + +- `-1`：使用 `POINT_STEP` 作为最小间隔。 + +- 正整数：如果当前 loop 耗时不足该值，则等待剩余时间。 + +4. 乱序写入 + +```Properties +IS_OUT_OF_ORDER=false +OUT_OF_ORDER_MODE=POISSON +OUT_OF_ORDER_RATIO=0.5 +IS_REGULAR_FREQUENCY=true +``` + +支持的乱序模式包括： + +- `POISSON`：按照泊松分布生成乱序时间戳。 + +- `BATCH`：按批次生成乱序数据。 + +5. 数据类型 + +```Properties +INSERT_DATATYPE_PROPORTION=1:1:1:1:1:1:0:0:0:0:0 +``` + +各项顺序为： + +```Plain Text +BOOLEAN:INT32:INT64:FLOAT:DOUBLE:TEXT:STRING:BLOB:TIMESTAMP:DATE:OBJECT +``` + +各项数值表示相应数据类型的比例。 + +### 3\.5 查询场景 + +|参数|示例|说明| +|---|---|---| +|`QUERY_DEVICE_NUM`|`1`|每条查询涉及的设备数| +|`QUERY_SENSOR_NUM`|`1`|每条查询涉及的测点数| +|`QUERY_AGGREGATE_FUN`|`count`|聚合函数| +|`STEP_SIZE`|`0`|查询起始时间的变化步长，单位为 `POINT_STEP`| +|`QUERY_INTERVAL`|`250000`|查询起止时间间隔| +|`QUERY_LOWER_VALUE`|`-5`|值过滤条件下限| +|`GROUP_BY_TIME_UNIT`|`20000`|Group By 窗口大小| +|`QUERY_SET_OP_TYPE`|`union`|集合操作类型| +|`QUERY_SET_OP_NUM`|`2`|集合查询的子集合数量，至少为 2| +|`IS_RECENT_QUERY`|`false`|混合场景下是否优先查询最近写入的数据| +|`ENABLE_FIXED_QUERY`|`false`|是否让所有查询线程使用相同设备和测点组合| +|`RESULT_ROW_LIMIT`|`-1`|查询结果行数限制；`-1` 表示不使用限制| +|`ALIGN_BY_DEVICE`|`false`|是否使用 Align By Device| + +### 3\.6 操作比例 + +`OPERATION_PROPORTION` 用于定义写入和各类查询的比例，共包含 13 项： + +```Plain Text +写入:Q1:Q2:Q3:Q4:Q5:Q6:Q7:Q8:Q9:Q10:Q11:Q12 +``` + +例如，纯写入： + +```Properties +OPERATION_PROPORTION=1:0:0:0:0:0:0:0:0:0:0:0:0 +``` + +纯精确点查询： + +```Properties +OPERATION_PROPORTION=0:1:0:0:0:0:0:0:0:0:0:0:0 +``` + +操作类型如下。 + +|编号|操作类型|说明| +|---|---|---| +|写入|数据写入|按当前写入配置生成并写入数据| +|Q1|精确点查询|按时间戳和设备查询指定测点| +|Q2|时间范围查询|只限制起止时间的范围查询| +|Q3|带值过滤的范围查询|同时包含时间和值过滤条件| +|Q4|带时间过滤的聚合查询|在时间范围内执行聚合| +|Q5|带值过滤的聚合查询|在全时间范围内按值过滤并聚合| +|Q6|带时间和值过滤的聚合查询|同时包含时间和值过滤条件| +|Q7|时间分组聚合查询|Group By 查询| +|Q8|最近点查询|查询设备的最新数据点| +|Q9|倒序时间范围查询|按时间倒序返回范围查询结果| +|Q10|倒序带值过滤的范围查询|带值过滤并按时间倒序返回| +|Q11|倒序时间分组聚合查询|倒序 Group By 查询| +|Q12|集合操作查询|`union`、`intersect` 或 `except` 等集合操作| + +Q12 仅支持 IoTDB 2\.0 表模型。集合操作中的各子查询为范围查询。 + +### 3\.7 测试过程和结果持久化 + +```Properties +TEST_DATA_PERSISTENCE=None ``` -## 4. 实际案例 +支持的取值包括： + +- `None`：不将测试过程写入外部持久化介质。 + +- `CSV`：写入 CSV 文件。 + +- `MySQL`：写入 MySQL。 + +- `IoTDB`：写入指定的 IoTDB。 + +常用参数： -我们以中车青岛四方车辆研究所有限公司应用为例，参考《ＡｐａｃｈｅＩｏＴＤＢ在智能运维平台存储中的应用》中描述的场景进行实际操作说明。 +|参数|示例|说明| +|---|---|---| +|`TEST_DATA_PERSISTENCE`|`None`|持久化方式| +|`RECORD_SPLIT`|`true`|是否将结果拆分为多条记录| +|`RECORD_SPLIT_MAX_LINE`|`10000000`|单表或单文件的最大记录数| +|`TEST_DATA_STORE_IP`|`127.0.0.1`|结果数据库地址| +|`TEST_DATA_STORE_PORT`|`6667`|结果数据库端口| +|`TEST_DATA_STORE_DB`|`result`|结果数据库名称| +|`TEST_DATA_STORE_USER`|`root`|结果数据库用户名| +|`TEST_DATA_STORE_PW`|`root`|结果数据库密码| +|`REMARK`|`write_test`|测试备注，可用于区分不同测试| +|`CSV_OUTPUT`|`true`|是否将最终结果写入 CSV| -测试目标：模拟中车青岛四方所场景因切换时间序列数据库实际需求，对比预期使用的IoTDB和原有系统使用的KairosDB性能。 +设置为 CSV 时，运行后会在 `data` 目录下生成相关记录；其中测试结果通常位于 `data/csvOutput`。无论是否开启持久化，测试日志均写入 `logs`。 -测试环境：为了保证在实验过程中消除其他无关服务与进程对数据库性能的影响，以及不同数据库之间的相互影响，本实验中的本地数据库均部署并运行在资源配置相同的多个独立的虚拟机上。因此，本实验搭建了 4 台 Linux( CentOS7 /x86) 虚拟机，并分别在上面部署了IoT-benchmark、 IoTDB数据库、KairosDB数据库、MySQL数据库。每一台虚拟机的具体资源配置如表4-1所示。每一台虚拟机的具体用途如表4-2所示。 -表4-1虚拟机配置信息 -| 硬件配置信息 | 系统描述 | -| ------------ | -------- | -| OS System | CentOS7 | -| CPU核数 | 16 | -| 内存 | 32G | -| 硬盘 | 200G | -| 网卡 | 千兆 | +日志输出频率可通过以下参数控制： +```Properties +IS_QUIET_MODE=true +LOG_PRINT_INTERVAL=5 +RESULT_PRINT_INTERVAL=3600 +``` + +### 3\.8 自动化和集群测试 + +1. 限制测试时长 +```Properties +TEST_MAX_TIME=3600000 +``` -表4-2虚拟机用途 +单位为毫秒。设置为 `0` 表示不限制。该参数不包含预注册元数据所消耗的时间。 -| IP | 用途 | -| ---------- | ------------- | -| 172.21.4.2 | IoT-benchmark | -| 172.21.4.3 | Apache-iotdb | -| 172.21.4.4 | KaiosDB | -| 172.21.4.5 | MySQL | +2. 多 Benchmark 集群压测 -### 4.1 写入测试 +在多台客户端机器上使用相同的总体数据规模配置，并设置： -场景描述：创建100个客户端来模拟100列车、每列车3000个传感器、数据类型为DOUBLE类型、数据时间间隔为500ms（2Hz）、顺序发送。参考以上需求我们需要修改IoT-benchmark配置参数如表4-3中所列。 +```Properties +BENCHMARK_CLUSTER=true +BENCHMARK_INDEX=0 +``` -表4-3配置参数信息 +每个 Benchmark 实例必须使用不同的 `BENCHMARK_INDEX`，例如依次使用 `0`、`1`、`2`。所有客户端应保持数据库连接、数据规模和操作比例等配置一致。 -| 参数名称 | IoTDB值 | KairosDB值 | -| -------------------------- | --------------------------- | ---------- | -| DB_SWITCH | IoTDB-013-SESSION_BY_TABLET | KairosDB | -| HOST | 172.21.4.3 | 172.21.4.4 | -| PORT | 6667 | 8080 | -| BENCHMARK_WORK_MODE | testWithDefaultPath | | -| OPERATION_PROPORTION | 1:0:0:0:0:0:0:0:0:0:0 | | -| CLIENT_NUMBER | 100 | | -| GROUP_NUMBER | 10 | | -| DEVICE_NUMBER | 100 | | -| SENSOR_NUMBER | 3000 | | -| INSERT_DATATYPE_PROPORTION | 0:0:0:0:1:0 | | -| POINT_STEP | 500 | | -| OP_MIN_INTERVAL | 0 | | -| IS_OUT_OF_ORDER | false | | -| BATCH_SIZE_PER_WRITE | 1 | | -| LOOP | 10000 | | -| TEST_DATA_PERSISTENCE | MySQL | | -| TEST_DATA_STORE_IP | 172.21.4.5 | | -| TEST_DATA_STORE_PORT | 3306 | | -| TEST_DATA_STORE_DB | demo | | -| TEST_DATA_STORE_USER | root | | -| TEST_DATA_STORE_PW | admin | | -| REMARK | demo | | +3. 双写测试 -首先在172.21.4.3和172.21.4.4上分别启动被测时间序列数据库Apache-IoTDB和KairosDB，之后在172.21.4.2、172.21.4.3和172.21.4.4上通过ser-benchamrk.sh脚本启动服务器资源监控（图4-1）。然后按照表4-3在172.21.4.2分别修改iotdb-0.13-0.0.1和kairosdb-0.0.1文件夹内的conf/config.properties文件满足测试需求。先后使用benchmark.sh启动对Apache-IoTDB和KairosDB的写入测试。 +IoT Benchmark 支持将同一份数据写入两个不同的数据库进行对比： -![img](/img/bm5.png) +```Properties +IS_DOUBLE_WRITE=true +ANOTHER_DB_SWITCH=<另一数据库类型> +ANOTHER_HOST=127.0.0.1 +ANOTHER_PORT=6667 +ANOTHER_USERNAME=root +ANOTHER_PASSWORD=root +ANOTHER_DB_NAME=test +``` -图4-1服务器监控任务 +双写模式不支持同一数据库不同版本之间的比较，也不支持 IoTDB 树模型与表模型之间的直接比较。 -​ 例如我们首先启动对KairosDB的测试，IoT-benchmark会在MySQL数据库中创建CONFIG数据表存放本次测试配置信息（图4-2），测试执行中会有日志输出当前测试进度（图4-3）。测试完成时会输出本次测试结果（图4-3），同时将结果写入FINAL_RESULT数据表中（图4-4）。 +## 4\. 使用示例 -![](/img/bm6.png) +本节使用小规模数据演示基本流程。正式性能测试应根据目标业务模型扩大设备数、测点数、客户端数和循环次数，并进行多轮预热和重复测试。 -图4-2测试配置信息表 +### 4\.1 写入测试示例 -![](/img/bm7.png) -![](/img/bm8.png) -![](/img/bm9.png) -![](/img/bm10.png) +测试目标：使用 10 个数据客户端，模拟 100 台设备，每台设备包含 10 个测点，执行纯写入测试。 -图4-3测试进度和结果 +配置示例： -![](/img/bm11.png) +```Properties +# 数据库连接 +DB_SWITCH=IoTDB-200-SESSION_BY_TABLET +IoTDB_DIALECT_MODE=tree +HOST=127.0.0.1 +PORT=6667 +USERNAME=root +PASSWORD=root +DB_NAME=test -图4-4测试结果表 +# 安全设置：默认不自动删除已有数据 +IS_DELETE_DATA=false -之后我们再启动对Apache-IoTDB的测试，同样的IoT-benchmark会在MySQL数据库CONFIG数据表中写入本次测试配置信息，测试执行中会有日志输出当前测试进度。测试完成时会输出本次测试结果，同时将结果写入FINAL_RESULT数据表中。 +# 工作模式 +BENCHMARK_WORK_MODE=testWithDefaultPath +OPERATION_PROPORTION=1:0:0:0:0:0:0:0:0:0:0:0:0 -依照测试结果信息我们知道同样的配置写入Apache-IoTDB和KairosDB写入延时时间分别为：55.98ms和1324.45ms；写入吞吐分别为：5,125,600.86点/秒和224,819.01点/秒；测试分别执行了585.30秒和11777.99秒。并且KairosDB有写入失败出现，排查后发现是数据磁盘使用率已达到100%，无磁盘空间继续接收数据。而Apache-IoTDB无写入失败现象，全部数据写入完毕后占用磁盘空间仅为4.7G（如图4-5所示）；从写入吞吐和磁盘占用情况上看Apache-IoTDB均优于KairosDB。当然后续还有其他测试来从多方面观察和对比，比如查询性能、文件压缩比、数据安全性等。 +# 数据规模 +GROUP_NUMBER=1 +DEVICE_NUMBER=100 +SENSOR_NUMBER=10 +SCHEMA_CLIENT_NUMBER=5 +DATA_CLIENT_NUMBER=10 +IS_SENSOR_TS_ALIGNMENT=true -![](/img/bm12.png) +# 写入配置 +CREATE_SCHEMA=true +BATCH_SIZE_PER_WRITE=10 +DEVICE_NUM_PER_WRITE=1 +LOOP=100 +POINT_STEP=1000 +OP_MIN_INTERVAL=0 +START_TIME=2026-01-01T00:00:00+08:00 +INSERT_DATATYPE_PROPORTION=1:1:1:1:1:1:0:0:0:0:0 -图4-5磁盘使用情况 +# IoTDB 2.0.6 及以上版本可开启 +ENABLE_IOTDB_RPC_COMPRESSION=true -那么测试过程中各个服务器资源使用情况如何呢？每个写操作具体的表现如何呢？这个时候我们就可以通过安装和使用Tableau来可视化服务器监控表和测试过程记录表内的数据了。Tableau的使用本文不展开介绍，通过它连接测试数据持久化的数据表后具体结果下如图（以Apache-IoTDB为例）： +# 输出 +TEST_DATA_PERSISTENCE=None +CSV_OUTPUT=true +REMARK=iotdb_2_write_test +``` + +启动测试： + +```Bash +./benchmark.sh +``` -![](/img/bm13.png) -![](/img/bm14.png) +完成后重点查看： -图4-6Tableau可视化测试过程 +- `INGESTION` 的 `okPoint` 和 `failPoint`。 +- 吞吐量 `throughput`。 +- `AVG`、`P95`、`P99`、`MAX` 延迟。 -### 4.2 查询测试 +- 是否存在连接超时、写入失败或服务端异常日志。 -场景描述：在写入测试场景下模拟10个客户端对时序数据库Apache-IoTDB内存放的数据进行全类型查询任务。配置如下： +### 4\.2 查询测试示例 -表4-4配置参数信息 +执行查询测试前，应确保目标数据库中已经存在与查询配置匹配的数据。建议复用写入测试生成的数据，并关闭自动删数和元数据创建。 -| 参数名称 | 示例 | -| -------------------- | --------------------- | -| CLIENT_NUMBER | 10 | -| QUERY_DEVICE_NUM | 2 | -| QUERY_SENSOR_NUM | 2 | -| QUERY_AGGREGATE_FUN | count | -| STEP_SIZE | 1 | -| QUERY_INTERVAL | 250000 | -| QUERY_LOWER_VALUE | -5 | -| GROUP_BY_TIME_UNIT | 20000 | -| LOOP | 30 | -| OPERATION_PROPORTION | 0:1:1:1:1:1:1:1:1:1:1 | +以下示例同时执行多种查询： -执行结果： +```Properties +DB_SWITCH=IoTDB-200-SESSION_BY_TABLET +IoTDB_DIALECT_MODE=tree +HOST=127.0.0.1 +PORT=6667 +USERNAME=root +PASSWORD=root +DB_NAME=test -![](/img/bm15.png) +IS_DELETE_DATA=false +CREATE_SCHEMA=false +BENCHMARK_WORK_MODE=testWithDefaultPath -图4-7查询测试结果 +GROUP_NUMBER=1 +DEVICE_NUMBER=100 +SENSOR_NUMBER=10 +SCHEMA_CLIENT_NUMBER=1 +DATA_CLIENT_NUMBER=10 -### 4.3 其他参数说明 +# 不执行写入，Q1～Q11 各占相同比例；树模型不使用 Q12 +OPERATION_PROPORTION=0:1:1:1:1:1:1:1:1:1:1:1:0 +LOOP=100 + +QUERY_DEVICE_NUM=2 +QUERY_SENSOR_NUM=2 +QUERY_AGGREGATE_FUN=count +STEP_SIZE=1 +QUERY_INTERVAL=250000 +QUERY_LOWER_VALUE=-5 +GROUP_BY_TIME_UNIT=20000 +``` + +如果测试表模型集合查询，可将方言切换为 `table`，并为 Q12 设置比例： + +```Properties +IoTDB_DIALECT_MODE=table +OPERATION_PROPORTION=0:0:0:0:0:0:0:0:0:0:0:0:1 +QUERY_SET_OP_TYPE=union +QUERY_SET_OP_NUM=2 +``` + +### 4\.3 其他配置示例 + +1. 模拟真实写入速率 + +让每个 loop 的最小间隔与数据时间戳间隔一致： + +```Properties +POINT_STEP=1000 +OP_MIN_INTERVAL=-1 +``` + +如果希望写入请求在一个 loop 内均匀分布，可使用： + +```Properties +INTERVAL_BETWEEN_WRITE_BATCH=100 +``` + +2. 指定测试时长 + +测试 1 小时： + +```Properties +TEST_MAX_TIME=3600000 +``` + +应确保 `LOOP` 足够大，否则测试可能先因循环次数耗尽而结束。 + +3. 控制生成数据规律 + +```Properties +LINE_RATIO=1 +SIN_RATIO=1 +SQUARE_RATIO=1 +RANDOM_RATIO=1 +CONSTANT_RATIO=1 +DATA_SEED=666 +STRING_LENGTH=10 +DOUBLE_LENGTH=2 +``` -之前章节中针对Apache-IoTDB和KairosDB进行写入性能对比，但是用户如果要执行模拟真实写入速率测试该如何配置？测试时间过长该如何控制呢？生成的模拟数据有哪些规律吗？如果IoT-Benchmark服务器配置较低，可以使用多台机器模拟压力输出吗？ +固定 `DATA_SEED` 有助于在多轮测试中生成可重复的数据。 -表4-5配置参数信息 +## 5\. 参考资料 -| 场景 | 参数 | 值 | 说明 | -| ------------------------------------------------------------ | -------------------------- |---------------------------------------------------------------------------------------------------| --------------------------------- | -| 模拟真实写入速率 | OP_INTERVAL | -1 | 也可输入整数控制操作间隔 | -| 指定测试时长（1小时） | TEST_MAX_TIME | 3600000 | 单位 ms；需要LOOP执行时间大于该值 | -| 定义模拟数据规律：支持全部数据类型，数量平均分类；支持五种数据分布，数量平均分布；字符串长度为10；小数位数为2 | INSERT_DATATYPE_PROPORTION | 1:1:1:1:1:1 | 数据类型分布比率 | -| LINE_RATIO | 1 | 线性 | | -| SIN_RATIO | 1 | 傅里叶函数 | | -| SQUARE_RATIO | 1 | 方波 | | -| RANDOM_RATIO | 1 | 随机数 | | -| CONSTANT_RATIO | 1 | 常数 | | -| STRING_LENGTH | 10 | 字符串长度 | | -| DOUBLE_LENGTH | 2 | 小数位数 | | -| 三台机器模拟300台设备数据写入 | BENCHMARK_CLUSTER | true | 开启多benchmark模式 | -| BENCHMARK_INDEX | 0、1、3 | 以[写入测试](./Benchmark.md#_4-1-写入测试)写入参数为例：0号负责设备编号0-99数据写入；1号负责设备编号100-199数据写入；2号负责设备编号200-299数据写入； | | \ No newline at end of file +- [IoT Benchmark 文档](https://github.com/thulab/iot-benchmark/tree/master/docs) diff --git a/src/zh/UserGuide/latest-Table/Tools-System/Benchmark.md b/src/zh/UserGuide/latest-Table/Tools-System/Benchmark.md index 8d7d578ef..78818a0d3 100644 --- a/src/zh/UserGuide/latest-Table/Tools-System/Benchmark.md +++ b/src/zh/UserGuide/latest-Table/Tools-System/Benchmark.md @@ -21,350 +21,800 @@ # 测试工具 -## 1. 基本概述 +IoT Benchmark 是面向工业物联网（IIoT）场景的时序数据库、实时数据库基准测试工具。本手册介绍工具的主要能力、支持的数据库及基本使用方法；安装、配置和测试案例主要以 IoTDB 2\.0\.x 为例，并同时覆盖树模型和表模型。 -IoT-benchmark 是基于 Java 和大数据环境开发的时序数据库基准测试工具，由清华大学软件学院研发并开源。它使用方便，支持多种写入以及查询方式，支持存储测试信息和结果以供进一步查询或分析，支持与 Tableau 集成以可视化测试结果。 +![](/img/bm2.png) -下图1-1囊括了测试基准流程及其他扩展功能。这些流程可以由IoT-benchmark 统一来完成。IoT Benchmark 支持多种工作负载，包括**纯写入、纯查询、写入查询混合**等，支持**软硬件系统监控、测试指标度量**等监控功能，还实现了**初始化数据库自动化、测试数据分析及系统参数优化**等功能。 +## 1\. 基本概述 -![](/img/bm1.png) +IoT Benchmark 可以按照配置生成周期性时间序列数据，对数据库执行写入和查询，并统计吞吐、延迟及成功率等指标。主要能力包括： -图1-1 +- 跨平台运行：支持 Linux、macOS 和 Windows。 -借鉴 YCSB 测试工具将工作负载生成、性能指标测量和数据库接口三个组件分离的设计思想，IoT-benchmark 的模块化设计如图1-2所示。与基于 YCSB 的测试工具系统不同的是，IoT-benchmark 增加了系统监控模块，支持测试数据和系统指标监控数据的持久化。此外也增加了一些特别针对时序数据场景的特殊负载测试功能，如支持物联网场景的批量写入和多种乱序数据写入模式。 +- 多种负载：支持纯写入、纯查询及读写混合负载。 -![](/img/bm2.png) +- 数据集生成：可将生成的数据保存到磁盘，供重复测试使用。 + +- 正确性验证：可从磁盘加载数据集，执行写入和查询正确性验证。 + +- 多数据库支持：可对多种时序数据库和实时数据库执行测试；IoTDB 2\.0\.x 支持 JDBC、REST 和多种 Session 写入方式。 + +- 结果持久化：支持将测试过程和结果保存到文件、CSV、MySQL 或 IoTDB。 + +- 测试扩展：支持乱序写入、批量写入、集群压测、双写对比和结果可视化等场景。 + +### 1\.1 支持的数据库、版本和接入方式 + +IoT Benchmark 支持如下数据库和版本。测试时通过 `DB_SWITCH` 选择对应的数据库、版本及连接或写入方式。 + +|数据库|支持版本|`DB_SWITCH`| +|---|---|---| +|IoTDB|v1\.x|`IoTDB-130-JDBC`、`IoTDB-130-REST`、`IoTDB-130-SESSION_BY_TABLET`、`IoTDB-130-SESSION_BY_RECORD`、`IoTDB-130-SESSION_BY_RECORDS`| +|IoTDB|v2\.x|`IoTDB-200-JDBC`、`IoTDB-200-REST`、`IoTDB-200-SESSION_BY_TABLET`、`IoTDB-200-SESSION_BY_RECORD`、`IoTDB-200-SESSION_BY_RECORDS`| +|InfluxDB|v1\.x|`InfluxDB`| +|InfluxDB|v2\.x|`InfluxDB-2.x`| +|QuestDB|v6\.0\.7|`QuestDB`| +|Microsoft SQL Server|2016 SP2|`MSSQLSERVER`| +|VictoriaMetrics|v1\.64\.0|`VictoriaMetrics`| +|SQLite|—|`SQLite`| +|OpenTSDB|2\.4\.1|`OpenTSDB`| +|KairosDB|—|`KairosDB`| +|TimescaleDB|—|`TimescaleDB`| +|TimescaleDB Cluster|Cluster|`TimescaleDB-Cluster`| +|TDengine|2\.2\.0\.2|`TDengine`| +|TDengine|3\.0\.1|`TDengine-3`| +|DolphinDB|v2\.x|`DolphinDB-2-MTW`、`DolphinDB-2-PTA`| +|DolphinDB|v3\.x|`DolphinDB-3-MTW`、`DolphinDB-3-PTA`| +|CnosDB|—|`CnosDB`| -图1-2 +其中： +- IoTDB 的接入方式包括 JDBC、REST、Session by Tablet、Session by Record 和 Session by Records。 -目前 IoT-benchmark 支持如下时间序列数据库、版本和连接方式： +- DolphinDB 的 `MTW` 表示 `MultithreadedTableWriter`，按行缓冲写入；`PTA` 表示 `PartitionedTableAppender`，按批次进行列式整表追加。 -| 数据库 | 版本 | 连接方式 | -| :-------------- |:-----------| :------------------------------------------------------- | -| IoTDB | v1.x v2.x | jdbc、sessionByTablet、sessionByRecord、sessionByRecords | -| InfluxDB | v1.x v2.x | SDK | -| TimescaleDB | -- | jdbc | -| OpenTSDB | -- | Http Request | -| QuestDB | v6.0.7 | jdbc | -| TDengine | v2.2.0.2 | jdbc | -| VictoriaMetrics | v1.64.0 | Http Request | -| KairosDB | -- | Http Request | +- 数据库版本、驱动和服务端必须相互匹配。使用其他数据库时，还需要配置相应的连接及扩展参数。 +### 1\.2 IoTDB 2\.0\.x 接入方式 -表1-1大数据测试基准对比 +后续安装、配置和案例以 IoTDB 2\.0\.x 为主。支持的接入方式如下。 -## 2. 安装运行 +|接入方式|`DB_SWITCH`|说明| +|---|---|---| +|JDBC|`IoTDB-200-JDBC`|通过 JDBC 执行写入和查询| +|REST|`IoTDB-200-REST`|通过 IoTDB REST 接口执行测试| +|Session by Tablet|`IoTDB-200-SESSION_BY_TABLET`|使用 Tablet 批量写入| +|Session by Record|`IoTDB-200-SESSION_BY_RECORD`|逐条写入记录| +|Session by Records|`IoTDB-200-SESSION_BY_RECORDS`|批量写入多条记录| -### 2.1 前置条件 +`IoTDB-200-SESSION_BY_TABLET` 适用于批量写入测试。 -1. Java 8 -2. Maven 3.6+ -3. 对应的合适版本的数据库，如 Apache IoTDB 2.0 +## 2\. 安装运行 +本章以 IoTDB 2\.0\.x 作为被测数据库。 +### 2\.1 前置条件 -### 2.2 获取方式 +1. 使用 IoT Benchmark 前需要准备： -- 获取二进制包：进入[这里](https://github.com/thulab/iot-benchmark/releases) 下载需要的安装包。下载下来为一个压缩文件，选择文件夹解压即可使用。 -- 源代码编译（可用 Apache IoTDB 2.0 的测试）： - - 第一步（编译 IoTDB Session 最新包）：进入[官网](https://github.com/apache/iotdb/tree/rc/2.0.5)下载 IoTDB 源码，在根目录下运行命令 mvn clean package install -pl session -am -DskipTests 编译 IoTDB Session 的最新包。 - - 第二步（编译 IoTDB Benchmark 测试包）：进入[官网](https://github.com/thulab/iot-benchmark)下载源码，在根目录下运行 mvn clean package install -pl iotdb-2.0 -am -DskipTests 编译测试 Apache IoTDB 2.0版本的测试包，测试包位置与根目录的相对路径为 ./iotdb-2.0/target/iotdb-2.0-0.0.1/iotdb-2.0-0.0.1 + - Java 17。 + - Maven。 -### 2.3 测试包结构 + - 已安装并能够正常运行的 IoTDB 2\.0\.x。 -测试包的目录结构如下所示。其中测试配置文件为conf/config.properties，测试启动脚本为benchmark\.sh (Linux & MacOS) 和 benchmark.bat (Windows)，详细文件用途见下表所示。 + - 足够的客户端 CPU、内存、磁盘和网络资源。 + +2. 环境相关说明： + + - 推荐使用 Linux 或 macOS 执行测试。 + + - Windows 使用安装包根目录下的 `benchmark.bat` 启动测试。 + + - Linux 和 macOS 使用 `benchmark.sh` 启动测试。 + + - CSV 记录模式中的部分系统信息采集能力仅支持 Linux。 + +> 注意：不要将 IoT Benchmark 与被测 IoTDB 部署在资源相互竞争的环境中。正式性能测试建议使用独立服务器，并关闭无关服务。 +> +> -```Shell --rw-r--r--. 1 root root 2881 1月 10 01:36 benchmark.bat --rwxr-xr-x. 1 root root 314 1月 10 01:36 benchmark.sh -drwxr-xr-x. 2 root root 24 1月 10 01:36 bin --rwxr-xr-x. 1 root root 1140 1月 10 01:36 cli-benchmark.sh -drwxr-xr-x. 2 root root 107 1月 10 01:36 conf -drwxr-xr-x. 2 root root 4096 1月 10 01:38 lib --rw-r--r--. 1 root root 11357 1月 10 01:36 LICENSE --rwxr-xr-x. 1 root root 939 1月 10 01:36 rep-benchmark.sh --rw-r--r--. 1 root root 14 1月 10 01:36 routine +### 2\.2 获取方式 + +1. 获取发行包 + +可以从 [IoT Benchmark Releases](https://github.com/thulab/iot-benchmark/releases) 下载与测试目标匹配的发行包，解压后使用。 + +2. 从源码构建 + +克隆仓库： + +```Bash +git clone https://github.com/thulab/iot-benchmark.git +cd iot-benchmark ``` -| 名称 | 子文件 | 用途 | -| :--------------- | :---------------- | :------------------------ | -| benchmark.bat | - | Windows环境运行启动脚本 | -| benchmark.sh | - | Linux/Mac环境运行启动脚本 | -| bin | startup.sh | 初始化脚本文件夹 | -| conf | config.properties | 测试场景配置文件 | -| lib | - | 依赖库文件 | -| LICENSE | - | 许可文件 | -| cli-benchmark.sh | - | 一键化启动脚本 | -| routine | | 多项测试配置文件 | -| rep-benchmark.sh | | 多项测试启动脚本 | +在项目根目录执行： -表2-1文件和文件夹列表用途 +```Bash +mvn clean package -Dmaven.test.skip=true +``` -### 2.4 执行测试 +构建完成后，IoTDB 2\.0\.x 安装包位于： -1. 按照测试需求修改配置文件，主要参数介绍见第3节，对应配置文件为conf/config.properties，**比如测试Apache IoTDB 2.0，则需要修改 DB_SWITCH=IoTDB-200-SESSION_BY_TABLET** -2. 启动被测时间序列数据库 -3. 通过运行 -4. 启动IoT-benchmark执行测试。执行中观测被测时间序列数据库和IoT-benchmark状态，执行完毕后查看结果和分析测试过程。 +```Plain Text +iotdb-2.0/target/iot-benchmark-iotdb-2.0/iot-benchmark-iotdb-2.0 +``` -### 2.5 结果说明 +进入安装目录： -测试的所有日志文件被存放于 logs 文件夹下，测试的结果在测试完成后被存放到 data/csvOutput 文件夹下，例如测试后我们得到了如下的结果矩阵： +```Bash +cd iotdb-2.0/target/iot-benchmark-iotdb-2.0/iot-benchmark-iotdb-2.0 +``` -![](/img/bm4.png) +### 2\.3 测试包结构 -- Result Matrix - - OkOperation：成功的对应操作次数 - - OkPoint：对于写入操作，是成功写入的点数；对于查询操作，是成功查询到的点数。 - - FailOperation：失败的对应操作次数 - - FailPoint：对于写入操作是写入失败的点数 -- Latency(mx) Matrix - - AVG：操作平均耗时 - - MIN：操作最小耗时 - - Pn：操作整体分布的对应分位值，比如P25是下四分位数 +安装包中的常用文件和目录如下。 +|名称|用途| +|---|---| +|`benchmark.sh`|Linux、macOS 启动脚本| +|`benchmark.bat`|Windows 启动脚本| +|`conf/config.properties`|测试场景配置文件| +|`lib/`|运行依赖库| +|`logs/`|测试日志，首次运行后生成| +|`data/`|数据集或持久化结果目录，按工作模式和持久化配置生成| +### 2\.4 执行测试 -## 3. 主要参数 +1. 启动 IoTDB +首先启动目标 IoTDB 2\.0\.x，并确认客户端能够访问其服务端口。默认原生接口端口为 `6667`。 -### 3.1 IoTDB服务模型 +2. 修改配置 -参数`IoTDB_DIALECT_MODE`支持tree、table,默认值为tree。 +- 编辑 `conf/config.properties` -- 当被测数据库为IoTDB-2.0及以上版本时需指定sql_dialect, 并且一个IoTDB只能指定一种。 -- sql_dialect等于table时，要满足：device数为table数的整数倍，table数为database数的整数倍 -- 表模型模式下，调整参数如下： +- 最小连接配置示例： -| **参数名称** | **类型** | **示例** | **系统描述** | -| :---------------------- | :------- | :------- | :----------------------------------------------------------- | -| IoTDB_TABLE_NAME_PREFIX | 字符串 | table_ | IoTDB 表模型下(IoTDB_DIALECT_MODE=table)，表名称前缀 | -| DATA_CLIENT_NUMBER | 整数 | 10 | IoTDB 表模型下(IoTDB_DIALECT_MODE=table)，client数为table数的整数倍 | -| SENSOR_NUMBER | 整数 | 10 | IoTDB 表模型下(IoTDB_DIALECT_MODE=table)，则控制属性列数量 | -| IoTDB_TABLE_NUMBER | 整数 | 1 | IoTDB 表模型下(IoTDB_DIALECT_MODE=table)，表的数量 | +```Properties +DB_SWITCH=IoTDB-200-SESSION_BY_TABLET +IoTDB_DIALECT_MODE=tree +HOST=127.0.0.1 +PORT=6667 +USERNAME=root +PASSWORD=root +DB_NAME=test +``` -### 3.2 工作模式 +- 使用表模型时改为： -工作模式参数“`BENCHMARK_WORK_MODE`”可选项有如下四种模式： +```Properties +IoTDB_DIALECT_MODE=table +``` -- 常用测试模式：结合配置`OPERATION_PROPORTION`参数达到“纯写入”、“纯查询”和“读写混合”的测试操作。 -- 生成数据模式：为了生成可以重复使用的数据集，iot-benchmark提供生成数据集的模式，生成数据集到FILE_PATH，以供后续使用正确性写入模式和正确性查询模式使用。 -- 单数据库正确性写入模式：为了验证数据集写入的正确性，您可以使用该模式写入生成数据模式中生成的数据集，目前该模式仅支持IoTDB v1.0 及更新的版本和InfluxDB v1.x。 -- 单数据库正确性查询模式：在运行这个模式之前需要先使用正确性写入模式写入数据到数据库。为了验证数据集写入的正确性，您可以使用该模式查询写入到数据库中的数据集，目前该模式仅支持IoTDB v1.0 和 InfluxDB v1.x。 +- 如果选择 REST： -| **模式名称** | **BENCHMARK_WORK_MODE** | **模式内容** | -| :--------------------- | :---------------------- | :------------------------------- | -| 常规测试模式 | testWithDefaultPath | 支持多种读和写操作的混合负载 | -| 生成数据模式 | generateDataMode | 生成Benchmark本身识别的数据 | -| 单数据库正确性写入模式 | verificationWriteMode | 需要配置 FILE_PATH 以及 DATA_SET | -| 单数据库正确性查询模式 | verificationQueryMode | 需要配置 FILE_PATH 以及 DATA_SET | +```Properties +DB_SWITCH=IoTDB-200-REST +REST_PORT=18080 +REST_AUTHORIZATION=Basic cm9vdDpyb290 +``` -### 3.3 服务器连接信息 +`REST_AUTHORIZATION `用于配置 REST 接口的 `Basic Authentication` 信息。示例使用用户名 root、密码 root。 -工作模式指定后，被测时序数据库的信息会通过如下参数告知IoT-benchmark +3. 检查 RPC 压缩兼容性 -| **参数名称** | **类型** | **示例** | **系统描述** | -| :----------- | :------- | :-------------------------- | :---------------------------------------------- | -| DB_SWITCH | 字符串 | IoTDB-200-SESSION_BY_TABLET | 被测时序数据库类型 | -| HOST | 字符串 | 127.0.0.1 | 被测时序数据库网络地址 | -| PORT | 整数 | 6667 | 被测时序数据库网络端口 | -| USERNAME | 字符串 | root | 被测时序数据库登录用户名 | -| PASSWORD | 字符串 | root | 被测时序数据库登录用户的密码 | -| DB_NAME | 字符串 | test | 被测时序数据库名称 | -| TOKEN | 字符串 | | 被测时序数据库连接认证Token（InfluxDB 2.0使用） | - -### 3.4 写入场景 - -| **参数名称** | **类型** | **示例** | **系统描述** | -| :------------------------- | :------- | :------------------------ | :----------------------------------------------------------- | -| CLIENT_NUMBER | 整数 | 100 | 客户端总数 | -| GROUP_NUMBER | 整数 | 20 | 数据库的数量；仅针对IoTDB。 | -| DEVICE_NUMBER | 整数 | 100 | 设备总数 | -| SENSOR_NUMBER | 整数 | 300 | 每个设备的传感器总数； **如果使用 IoTDB 表模型，则控制属性列数量** | -| INSERT_DATATYPE_PROPORTION | 字符串 | 1:1:1:1:1:1 | 设备的数据类型比例，BOOLEAN:INT32:INT64:FLOAT:DOUBLE:TEXT | -| POINT_STEP | 整数 | 1000 | 数据间时间戳间隔，即生成的数据两个时间戳之间的固定长度。 | -| OP_MIN_INTERVAL | 整数 | 0 | 操作最小执行间隔：若操作耗时大于该值则立即执行下一个，否则等待 (OP_MIN_INTERVAL-实际执行时间) ms；如果为0，则参数不生效；如果为-1，则其值和POINT_STEP一致 | -| IS_OUT_OF_ORDER | 布尔 | false | 是否乱序写入 | -| OUT_OF_ORDER_RATIO | 浮点数 | 0.3 | 乱序写入的数据比例 | -| BATCH_SIZE_PER_WRITE | 整数 | 1 | 批写入数据行数（一次写入多少行数据） | -| START_TIME | 时间 | 2022-10-30T00:00:00+08:00 | 写入数据的开始时间戳；以该时间戳为起点开始模拟创建数据时间戳。 | -| LOOP | 整数 | 86400 | 总操作次数：具体每种类型操作会按OPERATION_PROPORTION定义的比例划分 | -| OPERATION_PROPORTION | 字符 | 1:0:0:0:0:0:0:0:0:0:0 | # 各操作的比例,按照顺序为 写入:Q1:Q2:Q3:Q4:Q5:Q6:Q7:Q8:Q9:Q10, 请注意使用英文冒号。比例中的每一项是整数。 | +IoT Benchmark 2\.0 默认开启 IoTDB RPC 压缩： + +```Properties +ENABLE_IOTDB_RPC_COMPRESSION=true +``` +该功能要求被测 IoTDB 版本为 2\.0\.6 或更高版本。如果测试 2\.0\.6 之前的 IoTDB 2\.0\.x，请设置： -### 3.5 查询场景 +```Properties +ENABLE_IOTDB_RPC_COMPRESSION=false +``` -| 参数名称 | 类型 | 示例 | 系统描述 | -| :------------------- | :--- | :-------------------- | :----------------------------------------------------------- | -| QUERY_DEVICE_NUM | 整数 | 2 | 每条查询语句中查询涉及到的设备数量 | -| QUERY_SENSOR_NUM | 整数 | 2 | 每条查询语句中查询涉及到的传感器数量 | -| QUERY_AGGREGATE_FUN | 字符 | count | 在聚集查询中使用的聚集函数，比如count、avg、sum、max_time等 | -| STEP_SIZE | 整数 | 1 | 时间过滤条件的时间起点变化步长，若设为0则每个查询的时间过滤条件是一样的，单位：POINT_STEP | -| QUERY_INTERVAL | 整数 | 250000 | 起止时间的查询中开始时间与结束时间之间的时间间隔，和Group By中的时间间隔 | -| QUERY_LOWER_VALUE | 整数 | -5 | 条件查询子句时的参数，where xxx > QUERY_LOWER_VALUE | -| GROUP_BY_TIME_UNIT | 整数 | 20000 | Group by语句中的组的大小 | -| LOOP | 整数 | 10 | 总操作次数：具体每种类型操作会按OPERATION_PROPORTION定义的比例划分 | -| OPERATION_PROPORTION | 字符 | 0:0:0:0:0:0:0:0:0:0:1 | 写入:Q1:Q2:Q3:Q4:Q5:Q6:Q7:Q8:Q9:Q10 | +Thrift 压缩是另一项独立配置。如果启用： +```Properties +ENABLE_THRIFT_COMPRESSION=true +``` -### 3.6 操作比例 +还需要在 IoTDB 的 `iotdb-datanode.properties` 中设置： -| 编号 | 查询类型 | IoTDB 示例 SQL | -| :--- | :--------------------------- | :----------------------------------------------------------- | -| Q1 | 精确点查询 | select v1 from root.db.d1 where time = ? | -| Q2 | 时间范围查询 | select v1 from root.db.d1 where time > ? and time < ? | -| Q3 | 带值过滤的时间范围查询 | select v1 from root.db.d1 where time > ? and time < ? and v1 > ? | -| Q4 | 时间范围聚合查询 | select count(v1) from root.db.d1 where and time > ? and time < ? | -| Q5 | 带值过滤的全时间范围聚合查询 | select count(v1) from root.db.d1 where v1 > ? | -| Q6 | 带值过滤的时间范围聚合查询 | select count(v1) from root.db.d1 where v1 > ? and time > ? and time < ? | -| Q7 | 时间分组聚合查询 | select count(v1) from root.db.d1 group by ([?, ?), ?, ?) | -| Q8 | 最新点查询 | select last v1 from root.db.d1 | -| Q9 | 倒序范围查询 | select v1 from root.sg.d1 where time > ? and time < ? order by time desc | -| Q10 | 倒序带值过滤的范围查询 | select v1 from root.sg.d1 where time > ? and time < ? and v1 > ? order by time desc | +```Properties +dn_rpc_thrift_compression_enable=true +``` -### 3.7 测试过程和测试结果持久化 +4. 启动 Benchmark -IoT-benchmark目前支持通过配置参数将测试过程和测试结果持久化: - -| **参数名称** | **类型** | **示例** | **系统描述** | -| :-------------------- | :------- | :-------- | :----------------------------------------------------------- | -| TEST_DATA_PERSISTENCE | 字符串 | None | 结果持久化选择，支持None，IoTDB，MySQL和CSV | -| RECORD_SPLIT | 布尔 | true | 是否将结果划分后输出到多个记录, IoTDB 暂时不支持 | -| RECORD_SPLIT_MAX_LINE | 整数 | 10000000 | 记录行数的上限（每个数据库表或CSV文件按照总行数为1千万切分存放） | -| TEST_DATA_STORE_IP | 字符串 | 127.0.0.1 | 输出数据库的IP地址 | -| TEST_DATA_STORE_PORT | 整数 | 6667 | 输出数据库的端口号 | -| TEST_DATA_STORE_DB | 字符串 | result | 输出数据库的名称 | -| TEST_DATA_STORE_USER | 字符串 | root | 输出数据库的用户名 | -| TEST_DATA_STORE_PW | 字符串 | root | 输出数据库的用户密码 | - -- 如果我们设置“`TEST_DATA_PERSISTENCE=CSV`”，测试执行时和执行完毕后我们可以在IoT-benchmark根目录下看到新生成的`data`文件夹，其下包含`csv`文件夹记录测试过程；`csvOutput`文件夹记录测试结果。 -- 如果我们设置“`TEST_DATA_PERSISTENCE=MySQL`”，它会在测试开始前在指定的MySQL数据库中创建命名如“testWithDefaultPath_被测数据库名称_备注_测试启动时间”的数据表记录测试过程；会在名为“CONFIG”的数据表（如果不存在则创建该表），写入本次测试的配置信息；当测试完成时会在名为“FINAL_RESULT”的数据表（如果不存在则创建该表）中写入本次测试结果。 - -### 3.8 自动化脚本 - -#### 一键化启动脚本 - -您可以通过`cli-benchmark.sh`脚本一键化启动IoTDB、监控的IoTDB Benchmark和测试的IoTDB Benchmark，但需要注意该脚本启动时会清理IoTDB中的**所有数据**，请谨慎使用。 - -首先，您需要修改`cli-benchmark.sh`中的`IOTDB_HOME`参数为您本地的IoTDB所在的文件夹。 - -然后您可以使用脚本启动测试 +Linux 或 macOS： ```Bash -> ./cli-benchmark.sh +./benchmark.sh +``` + +Windows： + +```Plain Text +benchmark.bat ``` -测试完成后您可以在`logs`文件夹中查看测试相关日志，在`server-logs`文件夹中查看监控相关日志。 +测试过程中终端会周期性输出进度。完成后会输出主要配置、执行时间、结果矩阵和延迟矩阵。 + +### 2\.5 结果说明 + +测试执行信息会写入安装目录下的 `logs` 文件夹。是否生成 CSV 或写入结果数据库由 `TEST_DATA_PERSISTENCE` 等参数决定。 + +1. 结果矩阵 + +结果矩阵按操作类型统计以下指标： + +|指标|说明| +|---|---| +|`okOperation`|成功执行的请求或 SQL 数量| +|`okPoint`|写入成功的数据点数，或查询成功返回的数据点数| +|`failOperation`|执行失败的请求或 SQL 数量| +|`failPoint`|写入失败的数据点数；查询操作通常为 0| +|`throughput`|吞吐量，通常等于 `okPoint / Test elapsed time`| + +输出中的主要操作名称包括： + +- `INGESTION` + +- `PRECISE_POINT` + +- `TIME_RANGE` + +- `VALUE_RANGE` + +- `AGG_RANGE` + +- `AGG_VALUE` + +- `AGG_RANGE_VALUE` + +- `GROUP_BY` + +- `LATEST_POINT` + +- `RANGE_QUERY_DESC` + +- `VALUE_RANGE_QUERY_DESC` + +- `GROUP_BY_DESC` -#### 自动执行多项测试 +- `SET_OP_QUERY` -通常，除非与其他测试结果进行比较，否则单个测试是没有意义的。因此，我们提供了一个接口来通过一次启动执行多个测试。 +2. 延迟矩阵 -- 配置 routine +延迟矩阵以毫秒为单位，常见字段如下： -这个文件的每一行应该是每个测试过程会改变的参数（否则就变成复制测试）。例如，"例程"文件是： +|指标|说明| +|---|---| +|`AVG`|平均延迟| +|`MIN`|最小延迟| +|`P10`、`P25`、`MEDIAN`|低分位和中位延迟| +|`P75`、`P90`、`P95`|较高分位延迟| +|`P99`、`P999`|尾延迟| +|`MAX`|最大延迟| +|`SLOWEST_THREAD`|客户端线程中最大的累计操作时间| -```Plain -LOOP=10 DEVICE_NUMBER=100 TEST -LOOP=20 DEVICE_NUMBER=50 TEST -LOOP=50 DEVICE_NUMBER=20 TEST +测试结果还会给出元数据创建耗时和不包含元数据创建的 `Test elapsed time`。对比测试时，应确保各轮测试的硬件、数据量、客户端数、压缩配置和缓存状态一致。 + +3. 输出示例 + +测试完成后，终端会输出本次测试的主要配置、执行时间、结果矩阵和延迟矩阵。以下为一次纯写入测试的截断输出： + +```Plain Text +----------------------Main Configurations---------------------- +BENCHMARK_WORK_MODE=testWithDefaultPath +DB_SWITCH=IoTDB-200-SESSION_BY_TABLET +HOST=[127.0.0.1] + +GROUP_NUMBER=10 +DEVICE_NUMBER=50 +SENSOR_NUMBER=500 +SCHEMA_CLIENT_NUMBER=20 +DATA_CLIENT_NUMBER=20 + +OPERATION_PROPORTION=1:0:0:0:0:0:0:0:0:0:0:0:0 +LOOP=10000 +BATCH_SIZE_PER_WRITE=100 +--------------------------------------------------------------- + +Create schema cost 0.30 second +Test elapsed time (not include schema creation): 1238.79 second + +----------------------------------------------------------Result Matrix---------------------------------------------------------- +Operation okOperation okPoint failOperation failPoint throughput(point/s) +INGESTION 500000 25000000000 0 0 20180954.09 +--------------------------------------------------------------------------------------------------------------------------------- + +----------------------------------------------------------Latency (ms) Matrix---------------------------------------------------- +Operation AVG MIN P10 P25 MEDIAN P75 P90 P95 P99 P999 MAX SLOWEST_THREAD +INGESTION 37.78 1.67 2.02 2.29 2.86 4.14 5.62 7.43 759.69 5799.89 8309.40 1227561.44 +--------------------------------------------------------------------------------------------------------------------------------- ``` -然后依次执行3个LOOP参数分别为10、20、50的测试过程。 +从该输出可以得到： -> 注意： -> -> 您可以使用“LOOP=20 DEVICE_NUMBER=10 TEST”等格式更改每个测试中的多个参数，不允许使用不必要的空间。 关键字"TEST"意味着新的测试开始。如果您更改不同的参数，更改后的参数将保留在下一次测试中。 +- 元数据创建耗时为 `0.30` 秒，正式测试耗时为 `1238.79` 秒。 -- 开始测试 +- 共成功执行 `500000` 次写入操作，写入 `25000000000` 个数据点。 -配置文件routine后，您可以通过启动脚本启动多测试任务： +- `failOperation` 和 `failPoint` 均为 `0`，表示该轮测试没有记录到写入失败。 -```Bash -> ./rep-benchmark.sh +- 写入吞吐为 `20180954.09` 点/秒。 + +- 平均延迟为 `37.78` ms，P95 延迟为 `7.43` ms，P99 延迟为 `759.69` ms，最大延迟为 `8309.40` ms。 + +- `SLOWEST_THREAD` 是最慢客户端线程的累计操作时间，不是单次请求延迟。 + +示例中的数值仅用于展示输出格式。实际结果取决于硬件资源、网络环境、IoTDB 配置、数据规模和测试参数。 + +## 3\. 主要参数 + +### 3\.1 IoTDB 服务模型 + +IoTDB 2\.0\.x 支持树模型和表模型，通过以下参数选择： + +```Properties +IoTDB_DIALECT_MODE=tree +``` + +或： + +```Properties +IoTDB_DIALECT_MODE=table +``` + +相关约束如下： + +- 一个 IoTDB 实例在一次测试中只能选择一种 SQL 方言。 + +- 树模型要求 `DEVICE_NUMBER >= GROUP_NUMBER`。 + +- 表模型要求设备数是表数的整数倍，表数是数据库数的整数倍。 + +- 表模型下，数据客户端数应为表数的整数倍。 + + + +常用模型参数： + +|参数|示例|说明| +|---|---|---| +|`IoTDB_DIALECT_MODE`|`tree`|`tree` 或 `table`| +|`GROUP_NUMBER`|`1`|数据库数量；树模型下对应 database 数量| +|`IoTDB_TABLE_NUMBER`|`1`|表模型下创建的表数量| +|`IoTDB_TABLE_NAME_PREFIX`|`table_`|表名称前缀| +|`TABLE_TIME_COLUMN`|`time`|表模型时间列名称| +|`IoTDB_TABLE_WRITABLE_VIEW`|`false`|是否创建并使用可写视图| + +### 3\.2 工作模式 + +通过 `BENCHMARK_WORK_MODE` 选择工作模式。 + +|模式|配置值|说明| +|---|---|---| +|常规测试模式|`testWithDefaultPath`|执行写入、查询或混合负载| +|生成数据模式|`generateDataMode`|将 Benchmark 生成的数据集保存到 `FILE_PATH`| +|正确性写入模式|`verificationWriteMode`|从 `FILE_PATH` 加载数据集并写入数据库| +|正确性查询模式|`verificationQueryMode`|加载数据集并与数据库查询结果进行比对| + +示例： + +```Properties +BENCHMARK_WORK_MODE=testWithDefaultPath ``` -然后测试信息将显示在终端中。 +使用正确性写入和查询模式前，应先通过 `generateDataMode` 生成可复用数据集。 + +### 3\.3 服务器连接信息 + +|参数|示例|说明| +|---|---|---| +|`DB_SWITCH`|`IoTDB-200-SESSION_BY_TABLET`|数据库版本和连接方式| +|`HOST`|`127.0.0.1`|IoTDB 地址；多个地址使用英文逗号分隔| +|`PORT`|`6667`|原生服务端口，数量应与 `HOST` 一致| +|`USERNAME`|`root`|用户名| +|`PASSWORD`|`root`|密码| +|`DB_NAME`|`test`|测试使用的数据库名称| +|`REST_PORT`|`18080`|REST 服务端口| +|`REST_AUTHORIZATION`|`Basic cm9vdDpyb290`|REST 授权头| +|`ENABLE_AUTO_FETCH`|`false`|Session 是否自动刷新 DataNode 列表| + +清理数据相关参数： -> 注意： +```Properties +IS_DELETE_DATA=false +INIT_WAIT_TIME=1000 +``` + +> 警告：`IS_DELETE_DATA=true` 会在测试开始前清理目标数据库中的测试数据。仅可在专用测试环境中启用，并在执行前确认 `HOST`、`PORT`、`DB_NAME` 和账号权限。 +> > -> 如果您关闭终端或失去与客户端机器的连接，测试过程将终止。 如果输出传输到终端，则与任何其他情况相同。 -使用此接口通常需要很长时间，您可能希望将测试过程作为守护程序执行。这样，您可以通过启动脚本将测试任务作为守护程序启动： +### 3\.4 写入场景 -```Bash -> ./rep-benchmark.sh > /dev/null 2>&1 & +1. 数据规模和客户端 + +|参数|示例|说明| +|---|---|---| +|`DEVICE_NUMBER`|`100`|设备总数| +|`SENSOR_NUMBER`|`10`|每个设备的测点数；表模型下为测点列数量| +|`GROUP_NUMBER`|`1`|IoTDB database 数量| +|`SCHEMA_CLIENT_NUMBER`|`5`|注册元数据的客户端数| +|`DATA_CLIENT_NUMBER`|`10`|执行数据读写的客户端数| +|`IS_CLIENT_BIND`|`true`|是否将设备绑定到客户端| +|`REAL_INSERT_RATE`|`1.0`|实际参与写入的设备比例| +|`IS_SENSOR_TS_ALIGNMENT`|`true`|同一设备下测点时间戳是否对齐| + +2. 批量写入 + +|参数|示例|说明| +|---|---|---| +|`BATCH_SIZE_PER_WRITE`|`100`|每批每个设备写入的数据行数| +|`DEVICE_NUM_PER_WRITE`|`1`|每批写入涉及的设备数| +|`CREATE_SCHEMA`|`true`|写入前是否创建元数据| +|`START_TIME`|`2022-01-01T00:00:00+08:00`|生成数据的起始时间| + +单个批次的数据点数为： + +```Plain Text +DEVICE_NUM_PER_WRITE × SENSOR_NUMBER × BATCH_SIZE_PER_WRITE ``` -在这种情况下，如果您想知道发生了什么，可以通过以下命令查看日志信息： +`DEVICE_NUM_PER_WRITE` 必须能够整除单个数据客户端所负责的设备数。表模型下还应满足设备数、表数和单批设备数之间的整除约束。 -```Bash -> cd ./logs -> tail -f log_info.log +3. 写入节奏 + +|参数|示例|说明| +|---|---|---| +|`POINT_STEP`|`5000`|相邻生成时间戳的固定间隔| +|`OP_MIN_INTERVAL`|`0`|每个 loop 的最小执行间隔，单位 ms| +|`OP_MIN_INTERVAL_RANDOM`|`false`|是否在 `[0, OP_MIN_INTERVAL)` 中随机选择间隔| +|`INTERVAL_BETWEEN_WRITE_BATCH`|`0`|同一 loop 内相邻 batch 的最小间隔，单位 ms| +|`TIMESTAMP_PRECISION`|`ms`|时间戳精度| + +`OP_MIN_INTERVAL` 的特殊值： + +- `0`：不限制 loop 间隔。 + +- `-1`：使用 `POINT_STEP` 作为最小间隔。 + +- 正整数：如果当前 loop 耗时不足该值，则等待剩余时间。 + +4. 乱序写入 + +```Properties +IS_OUT_OF_ORDER=false +OUT_OF_ORDER_MODE=POISSON +OUT_OF_ORDER_RATIO=0.5 +IS_REGULAR_FREQUENCY=true ``` -## 4. 测试示例 +支持的乱序模式包括： + +- `POISSON`：按照泊松分布生成乱序时间戳。 -以IoTDB 2.0为例，使用IoT-Benchemark测试表模型写入及查询，修改IoT-Benchemark配置文件如下： +- `BATCH`：按批次生成乱序数据。 + +5. 数据类型 ```Properties -----------------------Main Configurations---------------------- -BENCHMARK_WORK_MODE=testWithDefaultPath -IoTDB_DIALECT_MODE=TABLE +INSERT_DATATYPE_PROPORTION=1:1:1:1:1:1:0:0:0:0:0 +``` + +各项顺序为： + +```Plain Text +BOOLEAN:INT32:INT64:FLOAT:DOUBLE:TEXT:STRING:BLOB:TIMESTAMP:DATE:OBJECT +``` + +各项数值表示相应数据类型的比例。 + +### 3\.5 查询场景 + +|参数|示例|说明| +|---|---|---| +|`QUERY_DEVICE_NUM`|`1`|每条查询涉及的设备数| +|`QUERY_SENSOR_NUM`|`1`|每条查询涉及的测点数| +|`QUERY_AGGREGATE_FUN`|`count`|聚合函数| +|`STEP_SIZE`|`0`|查询起始时间的变化步长，单位为 `POINT_STEP`| +|`QUERY_INTERVAL`|`250000`|查询起止时间间隔| +|`QUERY_LOWER_VALUE`|`-5`|值过滤条件下限| +|`GROUP_BY_TIME_UNIT`|`20000`|Group By 窗口大小| +|`QUERY_SET_OP_TYPE`|`union`|集合操作类型| +|`QUERY_SET_OP_NUM`|`2`|集合查询的子集合数量，至少为 2| +|`IS_RECENT_QUERY`|`false`|混合场景下是否优先查询最近写入的数据| +|`ENABLE_FIXED_QUERY`|`false`|是否让所有查询线程使用相同设备和测点组合| +|`RESULT_ROW_LIMIT`|`-1`|查询结果行数限制；`-1` 表示不使用限制| +|`ALIGN_BY_DEVICE`|`false`|是否使用 Align By Device| + +### 3\.6 操作比例 + +`OPERATION_PROPORTION` 用于定义写入和各类查询的比例，共包含 13 项： + +```Plain Text +写入:Q1:Q2:Q3:Q4:Q5:Q6:Q7:Q8:Q9:Q10:Q11:Q12 +``` + +例如，纯写入： + +```Properties +OPERATION_PROPORTION=1:0:0:0:0:0:0:0:0:0:0:0:0 +``` + +纯精确点查询： + +```Properties +OPERATION_PROPORTION=0:1:0:0:0:0:0:0:0:0:0:0:0 +``` + +操作类型如下。 + +|编号|操作类型|说明| +|---|---|---| +|写入|数据写入|按当前写入配置生成并写入数据| +|Q1|精确点查询|按时间戳和设备查询指定测点| +|Q2|时间范围查询|只限制起止时间的范围查询| +|Q3|带值过滤的范围查询|同时包含时间和值过滤条件| +|Q4|带时间过滤的聚合查询|在时间范围内执行聚合| +|Q5|带值过滤的聚合查询|在全时间范围内按值过滤并聚合| +|Q6|带时间和值过滤的聚合查询|同时包含时间和值过滤条件| +|Q7|时间分组聚合查询|Group By 查询| +|Q8|最近点查询|查询设备的最新数据点| +|Q9|倒序时间范围查询|按时间倒序返回范围查询结果| +|Q10|倒序带值过滤的范围查询|带值过滤并按时间倒序返回| +|Q11|倒序时间分组聚合查询|倒序 Group By 查询| +|Q12|集合操作查询|`union`、`intersect` 或 `except` 等集合操作| + +Q12 仅支持 IoTDB 2\.0 表模型。集合操作中的各子查询为范围查询。 + +### 3\.7 测试过程和结果持久化 + +```Properties +TEST_DATA_PERSISTENCE=None +``` + +支持的取值包括： + +- `None`：不将测试过程写入外部持久化介质。 + +- `CSV`：写入 CSV 文件。 + +- `MySQL`：写入 MySQL。 + +- `IoTDB`：写入指定的 IoTDB。 + +常用参数： + +|参数|示例|说明| +|---|---|---| +|`TEST_DATA_PERSISTENCE`|`None`|持久化方式| +|`RECORD_SPLIT`|`true`|是否将结果拆分为多条记录| +|`RECORD_SPLIT_MAX_LINE`|`10000000`|单表或单文件的最大记录数| +|`TEST_DATA_STORE_IP`|`127.0.0.1`|结果数据库地址| +|`TEST_DATA_STORE_PORT`|`6667`|结果数据库端口| +|`TEST_DATA_STORE_DB`|`result`|结果数据库名称| +|`TEST_DATA_STORE_USER`|`root`|结果数据库用户名| +|`TEST_DATA_STORE_PW`|`root`|结果数据库密码| +|`REMARK`|`write_test`|测试备注，可用于区分不同测试| +|`CSV_OUTPUT`|`true`|是否将最终结果写入 CSV| + +设置为 CSV 时，运行后会在 `data` 目录下生成相关记录；其中测试结果通常位于 `data/csvOutput`。无论是否开启持久化，测试日志均写入 `logs`。 + + + +日志输出频率可通过以下参数控制： + +```Properties +IS_QUIET_MODE=true +LOG_PRINT_INTERVAL=5 +RESULT_PRINT_INTERVAL=3600 +``` + +### 3\.8 自动化和集群测试 + +1. 限制测试时长 + +```Properties +TEST_MAX_TIME=3600000 +``` + +单位为毫秒。设置为 `0` 表示不限制。该参数不包含预注册元数据所消耗的时间。 + +2. 多 Benchmark 集群压测 + +在多台客户端机器上使用相同的总体数据规模配置，并设置： + +```Properties +BENCHMARK_CLUSTER=true +BENCHMARK_INDEX=0 +``` + +每个 Benchmark 实例必须使用不同的 `BENCHMARK_INDEX`，例如依次使用 `0`、`1`、`2`。所有客户端应保持数据库连接、数据规模和操作比例等配置一致。 + +3. 双写测试 + +IoT Benchmark 支持将同一份数据写入两个不同的数据库进行对比： + +```Properties +IS_DOUBLE_WRITE=true +ANOTHER_DB_SWITCH=<另一数据库类型> +ANOTHER_HOST=127.0.0.1 +ANOTHER_PORT=6667 +ANOTHER_USERNAME=root +ANOTHER_PASSWORD=root +ANOTHER_DB_NAME=test +``` + +双写模式不支持同一数据库不同版本之间的比较，也不支持 IoTDB 树模型与表模型之间的直接比较。 + +## 4\. 使用示例 + +本节使用小规模数据演示基本流程。正式性能测试应根据目标业务模型扩大设备数、测点数、客户端数和循环次数，并进行多轮预热和重复测试。 + +### 4\.1 写入测试示例 + +测试目标：使用 10 个数据客户端，模拟 100 台设备，每台设备包含 10 个测点，执行纯写入测试。 + +配置示例： + +```Properties +# 数据库连接 DB_SWITCH=IoTDB-200-SESSION_BY_TABLET +IoTDB_DIALECT_MODE=tree +HOST=127.0.0.1 +PORT=6667 +USERNAME=root +PASSWORD=root +DB_NAME=test + +# 安全设置：默认不自动删除已有数据 +IS_DELETE_DATA=false + +# 工作模式 +BENCHMARK_WORK_MODE=testWithDefaultPath +OPERATION_PROPORTION=1:0:0:0:0:0:0:0:0:0:0:0:0 + +# 数据规模 GROUP_NUMBER=1 -IoTDB_TABLE_NUMBER=1 -DEVICE_NUMBER=60 -REAL_INSERT_RATE=1.0 +DEVICE_NUMBER=100 SENSOR_NUMBER=10 -OPERATION_PROPORTION=1:0:0:0:0:0:0:0:0:0:0:0 -SCHEMA_CLIENT_NUMBER=10 +SCHEMA_CLIENT_NUMBER=5 DATA_CLIENT_NUMBER=10 -LOOP=10 +IS_SENSOR_TS_ALIGNMENT=true + +# 写入配置 +CREATE_SCHEMA=true BATCH_SIZE_PER_WRITE=10 DEVICE_NUM_PER_WRITE=1 -START_TIME=2025-01-01T00:00:00+08:00 +LOOP=100 POINT_STEP=1000 -INSERT_DATATYPE_PROPORTION=1:1:1:1:1:1:0:0:0:0 -VECTOR=true +OP_MIN_INTERVAL=0 +START_TIME=2026-01-01T00:00:00+08:00 +INSERT_DATATYPE_PROPORTION=1:1:1:1:1:1:0:0:0:0:0 + +# IoTDB 2.0.6 及以上版本可开启 +ENABLE_IOTDB_RPC_COMPRESSION=true + +# 输出 +TEST_DATA_PERSISTENCE=None +CSV_OUTPUT=true +REMARK=iotdb_2_write_test ``` -在被测数据库正常运行下，启动IoT-Benchmark服务，等待执行结束后，可查看打印结果如下： +启动测试： -```Shell -Create schema cost 0.88 second -Test elapsed time (not include schema creation): 4.60 second -----------------------------------------------------------Result Matrix---------------------------------------------------------- -Operation okOperation okPoint failOperation failPoint throughput(point/s) -INGESTION 600 60000 0 0 13054.42 -PRECISE_POINT 0 0 0 0 0.00 -TIME_RANGE 0 0 0 0 0.00 -VALUE_RANGE 0 0 0 0 0.00 -AGG_RANGE 0 0 0 0 0.00 -AGG_VALUE 0 0 0 0 0.00 -AGG_RANGE_VALUE 0 0 0 0 0.00 -GROUP_BY 0 0 0 0 0.00 -LATEST_POINT 0 0 0 0 0.00 -RANGE_QUERY_DESC 0 0 0 0 0.00 -VALUE_RANGE_QUERY_DESC 0 0 0 0 0.00 -GROUP_BY_DESC 0 0 0 0 0.00 ---------------------------------------------------------------------------------------------------------------------------------- +```Bash +./benchmark.sh +``` + +完成后重点查看： + +- `INGESTION` 的 `okPoint` 和 `failPoint`。 + +- 吞吐量 `throughput`。 + +- `AVG`、`P95`、`P99`、`MAX` 延迟。 + +- 是否存在连接超时、写入失败或服务端异常日志。 + +### 4\.2 查询测试示例 + +执行查询测试前，应确保目标数据库中已经存在与查询配置匹配的数据。建议复用写入测试生成的数据，并关闭自动删数和元数据创建。 + +以下示例同时执行多种查询： + +```Properties +DB_SWITCH=IoTDB-200-SESSION_BY_TABLET +IoTDB_DIALECT_MODE=tree +HOST=127.0.0.1 +PORT=6667 +USERNAME=root +PASSWORD=root +DB_NAME=test + +IS_DELETE_DATA=false +CREATE_SCHEMA=false +BENCHMARK_WORK_MODE=testWithDefaultPath + +GROUP_NUMBER=1 +DEVICE_NUMBER=100 +SENSOR_NUMBER=10 +SCHEMA_CLIENT_NUMBER=1 +DATA_CLIENT_NUMBER=10 + +# 不执行写入，Q1～Q11 各占相同比例；树模型不使用 Q12 +OPERATION_PROPORTION=0:1:1:1:1:1:1:1:1:1:1:1:0 +LOOP=100 + +QUERY_DEVICE_NUM=2 +QUERY_SENSOR_NUM=2 +QUERY_AGGREGATE_FUN=count +STEP_SIZE=1 +QUERY_INTERVAL=250000 +QUERY_LOWER_VALUE=-5 +GROUP_BY_TIME_UNIT=20000 +``` + +如果测试表模型集合查询，可将方言切换为 `table`，并为 Q12 设置比例： + +```Properties +IoTDB_DIALECT_MODE=table +OPERATION_PROPORTION=0:0:0:0:0:0:0:0:0:0:0:0:1 +QUERY_SET_OP_TYPE=union +QUERY_SET_OP_NUM=2 +``` + +### 4\.3 其他配置示例 + +1. 模拟真实写入速率 + +让每个 loop 的最小间隔与数据时间戳间隔一致： + +```Properties +POINT_STEP=1000 +OP_MIN_INTERVAL=-1 +``` + +如果希望写入请求在一个 loop 内均匀分布，可使用： + +```Properties +INTERVAL_BETWEEN_WRITE_BATCH=100 +``` + +2. 指定测试时长 + +测试 1 小时： + +```Properties +TEST_MAX_TIME=3600000 +``` + +应确保 `LOOP` 足够大，否则测试可能先因循环次数耗尽而结束。 + +3. 控制生成数据规律 + +```Properties +LINE_RATIO=1 +SIN_RATIO=1 +SQUARE_RATIO=1 +RANDOM_RATIO=1 +CONSTANT_RATIO=1 +DATA_SEED=666 +STRING_LENGTH=10 +DOUBLE_LENGTH=2 +``` + +固定 `DATA_SEED` 有助于在多轮测试中生成可重复的数据。 + +## 5\. 参考资料 ---------------------------------------------------------------------------Latency (ms) Matrix-------------------------------------------------------------------------- -Operation AVG MIN P10 P25 MEDIAN P75 P90 P95 P99 P999 MAX SLOWEST_THREAD -INGESTION 41.77 0.95 1.41 2.27 6.76 24.14 63.42 127.18 1260.92 1265.72 1265.49 2581.91 -PRECISE_POINT 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 -TIME_RANGE 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 -VALUE_RANGE 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 -AGG_RANGE 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 -AGG_VALUE 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 -AGG_RANGE_VALUE 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 -GROUP_BY 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 -LATEST_POINT 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 -RANGE_QUERY_DESC 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 -VALUE_RANGE_QUERY_DESC 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 -GROUP_BY_DESC 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ -``` \ No newline at end of file +- [IoT Benchmark 文档](https://github.com/thulab/iot-benchmark/tree/master/docs) diff --git a/src/zh/UserGuide/latest/Tools-System/Benchmark.md b/src/zh/UserGuide/latest/Tools-System/Benchmark.md index 6a4e9095f..78818a0d3 100644 --- a/src/zh/UserGuide/latest/Tools-System/Benchmark.md +++ b/src/zh/UserGuide/latest/Tools-System/Benchmark.md @@ -21,424 +21,800 @@ # 测试工具 -## 1. 基本概述 +IoT Benchmark 是面向工业物联网（IIoT）场景的时序数据库、实时数据库基准测试工具。本手册介绍工具的主要能力、支持的数据库及基本使用方法；安装、配置和测试案例主要以 IoTDB 2\.0\.x 为例，并同时覆盖树模型和表模型。 -IoT-benchmark 是基于 Java 和大数据环境开发的时序数据库基准测试工具，由清华大学软件学院研发并开源。它使用方便，支持多种写入以及查询方式，支持存储测试信息和结果以供进一步查询或分析，支持与 Tableau 集成以可视化测试结果。 +![](/img/bm2.png) -下图1-1囊括了测试基准流程及其他扩展功能。这些流程可以由IoT-benchmark 统一来完成。IoT Benchmark 支持多种工作负载，包括**纯写入、纯查询、写入查询混合**等，支持**软硬件系统监控、测试指标度量**等监控功能，还实现了**初始化数据库自动化、测试数据分析及系统参数优化**等功能。 +## 1\. 基本概述 -![](/img/bm1.png) +IoT Benchmark 可以按照配置生成周期性时间序列数据，对数据库执行写入和查询，并统计吞吐、延迟及成功率等指标。主要能力包括： -图1-1 +- 跨平台运行：支持 Linux、macOS 和 Windows。 -借鉴 YCSB 测试工具将工作负载生成、性能指标测量和数据库接口三个组件分离的设计思想，IoT-benchmark 的模块化设计如图1-2所示。与基于 YCSB 的测试工具系统不同的是，IoT-benchmark 增加了系统监控模块，支持测试数据和系统指标监控数据的持久化。此外也增加了一些特别针对时序数据场景的特殊负载测试功能，如支持物联网场景的批量写入和多种乱序数据写入模式。 +- 多种负载：支持纯写入、纯查询及读写混合负载。 -![](/img/bm2.png) +- 数据集生成：可将生成的数据保存到磁盘，供重复测试使用。 -图1-2 +- 正确性验证：可从磁盘加载数据集，执行写入和查询正确性验证。 +- 多数据库支持：可对多种时序数据库和实时数据库执行测试；IoTDB 2\.0\.x 支持 JDBC、REST 和多种 Session 写入方式。 -目前 IoT-benchmark 支持如下时间序列数据库、版本和连接方式： +- 结果持久化：支持将测试过程和结果保存到文件、CSV、MySQL 或 IoTDB。 -| 数据库 | 版本 | 连接方式 | -| :-------------- |:-----------| :------------------------------------------------------- | -| IoTDB | v1.x v2.x | jdbc、sessionByTablet、sessionByRecord、sessionByRecords | -| InfluxDB | v1.x v2.x | SDK | -| TimescaleDB | -- | jdbc | -| OpenTSDB | -- | Http Request | -| QuestDB | v6.0.7 | jdbc | -| TDengine | v2.2.0.2 | jdbc | -| VictoriaMetrics | v1.64.0 | Http Request | -| KairosDB | -- | Http Request | +- 测试扩展：支持乱序写入、批量写入、集群压测、双写对比和结果可视化等场景。 +### 1\.1 支持的数据库、版本和接入方式 -表1-1大数据测试基准对比 +IoT Benchmark 支持如下数据库和版本。测试时通过 `DB_SWITCH` 选择对应的数据库、版本及连接或写入方式。 -## 2. 安装运行 +|数据库|支持版本|`DB_SWITCH`| +|---|---|---| +|IoTDB|v1\.x|`IoTDB-130-JDBC`、`IoTDB-130-REST`、`IoTDB-130-SESSION_BY_TABLET`、`IoTDB-130-SESSION_BY_RECORD`、`IoTDB-130-SESSION_BY_RECORDS`| +|IoTDB|v2\.x|`IoTDB-200-JDBC`、`IoTDB-200-REST`、`IoTDB-200-SESSION_BY_TABLET`、`IoTDB-200-SESSION_BY_RECORD`、`IoTDB-200-SESSION_BY_RECORDS`| +|InfluxDB|v1\.x|`InfluxDB`| +|InfluxDB|v2\.x|`InfluxDB-2.x`| +|QuestDB|v6\.0\.7|`QuestDB`| +|Microsoft SQL Server|2016 SP2|`MSSQLSERVER`| +|VictoriaMetrics|v1\.64\.0|`VictoriaMetrics`| +|SQLite|—|`SQLite`| +|OpenTSDB|2\.4\.1|`OpenTSDB`| +|KairosDB|—|`KairosDB`| +|TimescaleDB|—|`TimescaleDB`| +|TimescaleDB Cluster|Cluster|`TimescaleDB-Cluster`| +|TDengine|2\.2\.0\.2|`TDengine`| +|TDengine|3\.0\.1|`TDengine-3`| +|DolphinDB|v2\.x|`DolphinDB-2-MTW`、`DolphinDB-2-PTA`| +|DolphinDB|v3\.x|`DolphinDB-3-MTW`、`DolphinDB-3-PTA`| +|CnosDB|—|`CnosDB`| -### 2.1 前置条件 +其中： -1. Java 8 -2. Maven 3.6+ -3. 对应的合适版本的数据库，如 Apache IoTDB 2.0 +- IoTDB 的接入方式包括 JDBC、REST、Session by Tablet、Session by Record 和 Session by Records。 +- DolphinDB 的 `MTW` 表示 `MultithreadedTableWriter`，按行缓冲写入；`PTA` 表示 `PartitionedTableAppender`，按批次进行列式整表追加。 +- 数据库版本、驱动和服务端必须相互匹配。使用其他数据库时，还需要配置相应的连接及扩展参数。 -### 2.2 获取方式 +### 1\.2 IoTDB 2\.0\.x 接入方式 -- 获取二进制包：进入[这里](https://github.com/thulab/iot-benchmark/releases) 下载需要的安装包。下载下来为一个压缩文件，选择文件夹解压即可使用。 -- 源代码编译（可用 Apache IoTDB 2.0 的测试）： - - 第一步（编译 IoTDB Session 最新包）：进入[官网](https://github.com/apache/iotdb/tree/rc/2.0.5)下载 IoTDB 源码，在根目录下运行命令 mvn clean package install -pl session -am -DskipTests 编译 IoTDB Session 的最新包。 - - 第二步（编译 IoTDB Benchmark 测试包）：进入[官网](https://github.com/thulab/iot-benchmark)下载源码，在根目录下运行 mvn clean package install -pl iotdb-2.0 -am -DskipTests 编译测试 Apache IoTDB 2.0版本的测试包，测试包位置与根目录的相对路径为 ./iotdb-2.0/target/iotdb-2.0-0.0.1/iotdb-2.0-0.0.1 +后续安装、配置和案例以 IoTDB 2\.0\.x 为主。支持的接入方式如下。 +|接入方式|`DB_SWITCH`|说明| +|---|---|---| +|JDBC|`IoTDB-200-JDBC`|通过 JDBC 执行写入和查询| +|REST|`IoTDB-200-REST`|通过 IoTDB REST 接口执行测试| +|Session by Tablet|`IoTDB-200-SESSION_BY_TABLET`|使用 Tablet 批量写入| +|Session by Record|`IoTDB-200-SESSION_BY_RECORD`|逐条写入记录| +|Session by Records|`IoTDB-200-SESSION_BY_RECORDS`|批量写入多条记录| -### 2.3 测试包结构 +`IoTDB-200-SESSION_BY_TABLET` 适用于批量写入测试。 -测试包的目录结构如下所示。其中测试配置文件为conf/config.properties，测试启动脚本为benchmark\.sh (Linux & MacOS) 和 benchmark.bat (Windows)，详细文件用途见下表所示。 +## 2\. 安装运行 -```Shell --rw-r--r--. 1 root root 2881 1月 10 01:36 benchmark.bat --rwxr-xr-x. 1 root root 314 1月 10 01:36 benchmark.sh -drwxr-xr-x. 2 root root 24 1月 10 01:36 bin --rwxr-xr-x. 1 root root 1140 1月 10 01:36 cli-benchmark.sh -drwxr-xr-x. 2 root root 107 1月 10 01:36 conf -drwxr-xr-x. 2 root root 4096 1月 10 01:38 lib --rw-r--r--. 1 root root 11357 1月 10 01:36 LICENSE --rwxr-xr-x. 1 root root 939 1月 10 01:36 rep-benchmark.sh --rw-r--r--. 1 root root 14 1月 10 01:36 routine -``` +本章以 IoTDB 2\.0\.x 作为被测数据库。 -| 名称 | 子文件 | 用途 | -| :--------------- | :---------------- | :------------------------ | -| benchmark.bat | - | Windows环境运行启动脚本 | -| benchmark.sh | - | Linux/Mac环境运行启动脚本 | -| bin | startup.sh | 初始化脚本文件夹 | -| conf | config.properties | 测试场景配置文件 | -| lib | - | 依赖库文件 | -| LICENSE | - | 许可文件 | -| cli-benchmark.sh | - | 一键化启动脚本 | -| routine | | 多项测试配置文件 | -| rep-benchmark.sh | | 多项测试启动脚本 | +### 2\.1 前置条件 -表2-1文件和文件夹列表用途 +1. 使用 IoT Benchmark 前需要准备： -### 2.4 执行测试 + - Java 17。 -1. 按照测试需求修改配置文件，主要参数介绍见第3节，对应配置文件为conf/config.properties，**比如测试Apache IoTDB 2.0，则需要修改 DB_SWITCH=IoTDB-200-SESSION_BY_TABLET** -2. 启动被测时间序列数据库 -3. 通过运行 -4. 启动IoT-benchmark执行测试。执行中观测被测时间序列数据库和IoT-benchmark状态，执行完毕后查看结果和分析测试过程。 + - Maven。 -### 2.5 结果说明 + - 已安装并能够正常运行的 IoTDB 2\.0\.x。 -测试的所有日志文件被存放于 logs 文件夹下，测试的结果在测试完成后被存放到 data/csvOutput 文件夹下，例如测试后我们得到了如下的结果矩阵： + - 足够的客户端 CPU、内存、磁盘和网络资源。 -![](/img/bm4.png) +2. 环境相关说明： -- Result Matrix - - OkOperation：成功的对应操作次数 - - OkPoint：对于写入操作，是成功写入的点数；对于查询操作，是成功查询到的点数。 - - FailOperation：失败的对应操作次数 - - FailPoint：对于写入操作是写入失败的点数 -- Latency(mx) Matrix - - AVG：操作平均耗时 - - MIN：操作最小耗时 - - Pn：操作整体分布的对应分位值，比如P25是下四分位数 + - 推荐使用 Linux 或 macOS 执行测试。 + - Windows 使用安装包根目录下的 `benchmark.bat` 启动测试。 + - Linux 和 macOS 使用 `benchmark.sh` 启动测试。 -## 3. 主要参数 + - CSV 记录模式中的部分系统信息采集能力仅支持 Linux。 + +> 注意：不要将 IoT Benchmark 与被测 IoTDB 部署在资源相互竞争的环境中。正式性能测试建议使用独立服务器，并关闭无关服务。 +> +> +### 2\.2 获取方式 -### 3.1 IoTDB服务模型 +1. 获取发行包 -参数`IoTDB_DIALECT_MODE`支持tree、table,默认值为tree。 +可以从 [IoT Benchmark Releases](https://github.com/thulab/iot-benchmark/releases) 下载与测试目标匹配的发行包，解压后使用。 + +2. 从源码构建 + +克隆仓库： + +```Bash +git clone https://github.com/thulab/iot-benchmark.git +cd iot-benchmark +``` -- 当被测数据库为IoTDB-2.0及以上版本时需指定sql_dialect, 并且一个IoTDB只能指定一种。 -- sql_dialect等于tree时，要满足：device数量 >= database数量 +在项目根目录执行： -### 3.2 工作模式 +```Bash +mvn clean package -Dmaven.test.skip=true +``` -工作模式参数“`BENCHMARK_WORK_MODE`”可选项有如下四种模式： +构建完成后，IoTDB 2\.0\.x 安装包位于： -- 常用测试模式：结合配置`OPERATION_PROPORTION`参数达到“纯写入”、“纯查询”和“读写混合”的测试操作。 -- 生成数据模式：为了生成可以重复使用的数据集，iot-benchmark提供生成数据集的模式，生成数据集到FILE_PATH，以供后续使用正确性写入模式和正确性查询模式使用。 -- 单数据库正确性写入模式：为了验证数据集写入的正确性，您可以使用该模式写入生成数据模式中生成的数据集，目前该模式仅支持IoTDB v1.0 及更新的版本和InfluxDB v1.x。 -- 单数据库正确性查询模式：在运行这个模式之前需要先使用正确性写入模式写入数据到数据库。为了验证数据集写入的正确性，您可以使用该模式查询写入到数据库中的数据集，目前该模式仅支持IoTDB v1.0 和 InfluxDB v1.x。 +```Plain Text +iotdb-2.0/target/iot-benchmark-iotdb-2.0/iot-benchmark-iotdb-2.0 +``` -| **模式名称** | **BENCHMARK_WORK_MODE** | **模式内容** | -| :--------------------- | :---------------------- | :------------------------------- | -| 常规测试模式 | testWithDefaultPath | 支持多种读和写操作的混合负载 | -| 生成数据模式 | generateDataMode | 生成Benchmark本身识别的数据 | -| 单数据库正确性写入模式 | verificationWriteMode | 需要配置 FILE_PATH 以及 DATA_SET | -| 单数据库正确性查询模式 | verificationQueryMode | 需要配置 FILE_PATH 以及 DATA_SET | +进入安装目录： -### 3.3 服务器连接信息 +```Bash +cd iotdb-2.0/target/iot-benchmark-iotdb-2.0/iot-benchmark-iotdb-2.0 +``` -工作模式指定后，被测时序数据库的信息会通过如下参数告知IoT-benchmark +### 2\.3 测试包结构 -| **参数名称** | **类型** | **示例** | **系统描述** | -| :----------- | :------- | :-------------------------- | :---------------------------------------------- | -| DB_SWITCH | 字符串 | IoTDB-200-SESSION_BY_TABLET | 被测时序数据库类型 | -| HOST | 字符串 | 127.0.0.1 | 被测时序数据库网络地址 | -| PORT | 整数 | 6667 | 被测时序数据库网络端口 | -| USERNAME | 字符串 | root | 被测时序数据库登录用户名 | -| PASSWORD | 字符串 | root | 被测时序数据库登录用户的密码 | -| DB_NAME | 字符串 | test | 被测时序数据库名称 | -| TOKEN | 字符串 | | 被测时序数据库连接认证Token（InfluxDB 2.0使用） | +安装包中的常用文件和目录如下。 -### 3.4 写入场景 +|名称|用途| +|---|---| +|`benchmark.sh`|Linux、macOS 启动脚本| +|`benchmark.bat`|Windows 启动脚本| +|`conf/config.properties`|测试场景配置文件| +|`lib/`|运行依赖库| +|`logs/`|测试日志，首次运行后生成| +|`data/`|数据集或持久化结果目录，按工作模式和持久化配置生成| -| **参数名称** | **类型** | **示例** | **系统描述** | -| :------------------------- | :------- | :------------------------ | :----------------------------------------------------------- | -| CLIENT_NUMBER | 整数 | 100 | 客户端总数 | -| GROUP_NUMBER | 整数 | 20 | 数据库的数量；仅针对IoTDB。 | -| DEVICE_NUMBER | 整数 | 100 | 设备总数 | -| SENSOR_NUMBER | 整数 | 300 | 每个设备的传感器总数； **如果使用 IoTDB 表模型，则控制属性列数量** | -| INSERT_DATATYPE_PROPORTION | 字符串 | 1:1:1:1:1:1 | 设备的数据类型比例，BOOLEAN:INT32:INT64:FLOAT:DOUBLE:TEXT | -| POINT_STEP | 整数 | 1000 | 数据间时间戳间隔，即生成的数据两个时间戳之间的固定长度。 | -| OP_MIN_INTERVAL | 整数 | 0 | 操作最小执行间隔：若操作耗时大于该值则立即执行下一个，否则等待 (OP_MIN_INTERVAL-实际执行时间) ms；如果为0，则参数不生效；如果为-1，则其值和POINT_STEP一致 | -| IS_OUT_OF_ORDER | 布尔 | false | 是否乱序写入 | -| OUT_OF_ORDER_RATIO | 浮点数 | 0.3 | 乱序写入的数据比例 | -| BATCH_SIZE_PER_WRITE | 整数 | 1 | 批写入数据行数（一次写入多少行数据） | -| START_TIME | 时间 | 2022-10-30T00:00:00+08:00 | 写入数据的开始时间戳；以该时间戳为起点开始模拟创建数据时间戳。 | -| LOOP | 整数 | 86400 | 总操作次数：具体每种类型操作会按OPERATION_PROPORTION定义的比例划分 | -| OPERATION_PROPORTION | 字符 | 1:0:0:0:0:0:0:0:0:0:0 | # 各操作的比例,按照顺序为 写入:Q1:Q2:Q3:Q4:Q5:Q6:Q7:Q8:Q9:Q10, 请注意使用英文冒号。比例中的每一项是整数。 | +### 2\.4 执行测试 +1. 启动 IoTDB -### 3.5 查询场景 +首先启动目标 IoTDB 2\.0\.x，并确认客户端能够访问其服务端口。默认原生接口端口为 `6667`。 -| 参数名称 | 类型 | 示例 | 系统描述 | -| :------------------- | :--- | :-------------------- | :----------------------------------------------------------- | -| QUERY_DEVICE_NUM | 整数 | 2 | 每条查询语句中查询涉及到的设备数量 | -| QUERY_SENSOR_NUM | 整数 | 2 | 每条查询语句中查询涉及到的传感器数量 | -| QUERY_AGGREGATE_FUN | 字符 | count | 在聚集查询中使用的聚集函数，比如count、avg、sum、max_time等 | -| STEP_SIZE | 整数 | 1 | 时间过滤条件的时间起点变化步长，若设为0则每个查询的时间过滤条件是一样的，单位：POINT_STEP | -| QUERY_INTERVAL | 整数 | 250000 | 起止时间的查询中开始时间与结束时间之间的时间间隔，和Group By中的时间间隔 | -| QUERY_LOWER_VALUE | 整数 | -5 | 条件查询子句时的参数，where xxx > QUERY_LOWER_VALUE | -| GROUP_BY_TIME_UNIT | 整数 | 20000 | Group by语句中的组的大小 | -| LOOP | 整数 | 10 | 总操作次数：具体每种类型操作会按OPERATION_PROPORTION定义的比例划分 | -| OPERATION_PROPORTION | 字符 | 0:0:0:0:0:0:0:0:0:0:1 | 写入:Q1:Q2:Q3:Q4:Q5:Q6:Q7:Q8:Q9:Q10 | +2. 修改配置 +- 编辑 `conf/config.properties` -### 3.6 操作比例 +- 最小连接配置示例： -| 编号 | 查询类型 | IoTDB 示例 SQL | -| :--- | :--------------------------- | :----------------------------------------------------------- | -| Q1 | 精确点查询 | select v1 from root.db.d1 where time = ? | -| Q2 | 时间范围查询 | select v1 from root.db.d1 where time > ? and time < ? | -| Q3 | 带值过滤的时间范围查询 | select v1 from root.db.d1 where time > ? and time < ? and v1 > ? | -| Q4 | 时间范围聚合查询 | select count(v1) from root.db.d1 where and time > ? and time < ? | -| Q5 | 带值过滤的全时间范围聚合查询 | select count(v1) from root.db.d1 where v1 > ? | -| Q6 | 带值过滤的时间范围聚合查询 | select count(v1) from root.db.d1 where v1 > ? and time > ? and time < ? | -| Q7 | 时间分组聚合查询 | select count(v1) from root.db.d1 group by ([?, ?), ?, ?) | -| Q8 | 最新点查询 | select last v1 from root.db.d1 | -| Q9 | 倒序范围查询 | select v1 from root.sg.d1 where time > ? and time < ? order by time desc | -| Q10 | 倒序带值过滤的范围查询 | select v1 from root.sg.d1 where time > ? and time < ? and v1 > ? order by time desc | +```Properties +DB_SWITCH=IoTDB-200-SESSION_BY_TABLET +IoTDB_DIALECT_MODE=tree +HOST=127.0.0.1 +PORT=6667 +USERNAME=root +PASSWORD=root +DB_NAME=test +``` -### 3.7 测试过程和测试结果持久化 +- 使用表模型时改为： -IoT-benchmark目前支持通过配置参数将测试过程和测试结果持久化: +```Properties +IoTDB_DIALECT_MODE=table +``` -| **参数名称** | **类型** | **示例** | **系统描述** | -| :-------------------- | :------- | :-------- | :----------------------------------------------------------- | -| TEST_DATA_PERSISTENCE | 字符串 | None | 结果持久化选择，支持None，IoTDB，MySQL和CSV | -| RECORD_SPLIT | 布尔 | true | 是否将结果划分后输出到多个记录, IoTDB 暂时不支持 | -| RECORD_SPLIT_MAX_LINE | 整数 | 10000000 | 记录行数的上限（每个数据库表或CSV文件按照总行数为1千万切分存放） | -| TEST_DATA_STORE_IP | 字符串 | 127.0.0.1 | 输出数据库的IP地址 | -| TEST_DATA_STORE_PORT | 整数 | 6667 | 输出数据库的端口号 | -| TEST_DATA_STORE_DB | 字符串 | result | 输出数据库的名称 | -| TEST_DATA_STORE_USER | 字符串 | root | 输出数据库的用户名 | -| TEST_DATA_STORE_PW | 字符串 | root | 输出数据库的用户密码 | - -- 如果我们设置“`TEST_DATA_PERSISTENCE=CSV`”，测试执行时和执行完毕后我们可以在IoT-benchmark根目录下看到新生成的`data`文件夹，其下包含`csv`文件夹记录测试过程；`csvOutput`文件夹记录测试结果。 -- 如果我们设置“`TEST_DATA_PERSISTENCE=MySQL`”，它会在测试开始前在指定的MySQL数据库中创建命名如“testWithDefaultPath_被测数据库名称_备注_测试启动时间”的数据表记录测试过程；会在名为“CONFIG”的数据表（如果不存在则创建该表），写入本次测试的配置信息；当测试完成时会在名为“FINAL_RESULT”的数据表（如果不存在则创建该表）中写入本次测试结果。 - -### 3.8 自动化脚本 - -#### 一键化启动脚本 - -您可以通过`cli-benchmark.sh`脚本一键化启动IoTDB、监控的IoTDB Benchmark和测试的IoTDB Benchmark，但需要注意该脚本启动时会清理IoTDB中的**所有数据**，请谨慎使用。 - -首先，您需要修改`cli-benchmark.sh`中的`IOTDB_HOME`参数为您本地的IoTDB所在的文件夹。 - -然后您可以使用脚本启动测试 +- 如果选择 REST： -```Bash -> ./cli-benchmark.sh +```Properties +DB_SWITCH=IoTDB-200-REST +REST_PORT=18080 +REST_AUTHORIZATION=Basic cm9vdDpyb290 ``` -测试完成后您可以在`logs`文件夹中查看测试相关日志，在`server-logs`文件夹中查看监控相关日志。 +`REST_AUTHORIZATION `用于配置 REST 接口的 `Basic Authentication` 信息。示例使用用户名 root、密码 root。 -#### 自动执行多项测试 +3. 检查 RPC 压缩兼容性 -通常，除非与其他测试结果进行比较，否则单个测试是没有意义的。因此，我们提供了一个接口来通过一次启动执行多个测试。 +IoT Benchmark 2\.0 默认开启 IoTDB RPC 压缩： -- 配置 routine +```Properties +ENABLE_IOTDB_RPC_COMPRESSION=true +``` -这个文件的每一行应该是每个测试过程会改变的参数（否则就变成复制测试）。例如，"例程"文件是： +该功能要求被测 IoTDB 版本为 2\.0\.6 或更高版本。如果测试 2\.0\.6 之前的 IoTDB 2\.0\.x，请设置： -```Plain -LOOP=10 DEVICE_NUMBER=100 TEST -LOOP=20 DEVICE_NUMBER=50 TEST -LOOP=50 DEVICE_NUMBER=20 TEST +```Properties +ENABLE_IOTDB_RPC_COMPRESSION=false ``` -然后依次执行3个LOOP参数分别为10、20、50的测试过程。 +Thrift 压缩是另一项独立配置。如果启用： -> 注意： -> -> 您可以使用“LOOP=20 DEVICE_NUMBER=10 TEST”等格式更改每个测试中的多个参数，不允许使用不必要的空间。 关键字"TEST"意味着新的测试开始。如果您更改不同的参数，更改后的参数将保留在下一次测试中。 +```Properties +ENABLE_THRIFT_COMPRESSION=true +``` -- 开始测试 +还需要在 IoTDB 的 `iotdb-datanode.properties` 中设置： -配置文件routine后，您可以通过启动脚本启动多测试任务： +```Properties +dn_rpc_thrift_compression_enable=true +``` + +4. 启动 Benchmark + +Linux 或 macOS： ```Bash -> ./rep-benchmark.sh +./benchmark.sh +``` + +Windows： + +```Plain Text +benchmark.bat ``` -然后测试信息将显示在终端中。 +测试过程中终端会周期性输出进度。完成后会输出主要配置、执行时间、结果矩阵和延迟矩阵。 + +### 2\.5 结果说明 + +测试执行信息会写入安装目录下的 `logs` 文件夹。是否生成 CSV 或写入结果数据库由 `TEST_DATA_PERSISTENCE` 等参数决定。 + +1. 结果矩阵 + +结果矩阵按操作类型统计以下指标： + +|指标|说明| +|---|---| +|`okOperation`|成功执行的请求或 SQL 数量| +|`okPoint`|写入成功的数据点数，或查询成功返回的数据点数| +|`failOperation`|执行失败的请求或 SQL 数量| +|`failPoint`|写入失败的数据点数；查询操作通常为 0| +|`throughput`|吞吐量，通常等于 `okPoint / Test elapsed time`| + +输出中的主要操作名称包括： + +- `INGESTION` + +- `PRECISE_POINT` + +- `TIME_RANGE` + +- `VALUE_RANGE` + +- `AGG_RANGE` + +- `AGG_VALUE` + +- `AGG_RANGE_VALUE` + +- `GROUP_BY` + +- `LATEST_POINT` + +- `RANGE_QUERY_DESC` + +- `VALUE_RANGE_QUERY_DESC` + +- `GROUP_BY_DESC` + +- `SET_OP_QUERY` + +2. 延迟矩阵 + +延迟矩阵以毫秒为单位，常见字段如下： + +|指标|说明| +|---|---| +|`AVG`|平均延迟| +|`MIN`|最小延迟| +|`P10`、`P25`、`MEDIAN`|低分位和中位延迟| +|`P75`、`P90`、`P95`|较高分位延迟| +|`P99`、`P999`|尾延迟| +|`MAX`|最大延迟| +|`SLOWEST_THREAD`|客户端线程中最大的累计操作时间| + +测试结果还会给出元数据创建耗时和不包含元数据创建的 `Test elapsed time`。对比测试时，应确保各轮测试的硬件、数据量、客户端数、压缩配置和缓存状态一致。 + +3. 输出示例 + +测试完成后，终端会输出本次测试的主要配置、执行时间、结果矩阵和延迟矩阵。以下为一次纯写入测试的截断输出： + +```Plain Text +----------------------Main Configurations---------------------- +BENCHMARK_WORK_MODE=testWithDefaultPath +DB_SWITCH=IoTDB-200-SESSION_BY_TABLET +HOST=[127.0.0.1] + +GROUP_NUMBER=10 +DEVICE_NUMBER=50 +SENSOR_NUMBER=500 +SCHEMA_CLIENT_NUMBER=20 +DATA_CLIENT_NUMBER=20 + +OPERATION_PROPORTION=1:0:0:0:0:0:0:0:0:0:0:0:0 +LOOP=10000 +BATCH_SIZE_PER_WRITE=100 +--------------------------------------------------------------- + +Create schema cost 0.30 second +Test elapsed time (not include schema creation): 1238.79 second + +----------------------------------------------------------Result Matrix---------------------------------------------------------- +Operation okOperation okPoint failOperation failPoint throughput(point/s) +INGESTION 500000 25000000000 0 0 20180954.09 +--------------------------------------------------------------------------------------------------------------------------------- + +----------------------------------------------------------Latency (ms) Matrix---------------------------------------------------- +Operation AVG MIN P10 P25 MEDIAN P75 P90 P95 P99 P999 MAX SLOWEST_THREAD +INGESTION 37.78 1.67 2.02 2.29 2.86 4.14 5.62 7.43 759.69 5799.89 8309.40 1227561.44 +--------------------------------------------------------------------------------------------------------------------------------- +``` + +从该输出可以得到： + +- 元数据创建耗时为 `0.30` 秒，正式测试耗时为 `1238.79` 秒。 + +- 共成功执行 `500000` 次写入操作，写入 `25000000000` 个数据点。 + +- `failOperation` 和 `failPoint` 均为 `0`，表示该轮测试没有记录到写入失败。 + +- 写入吞吐为 `20180954.09` 点/秒。 + +- 平均延迟为 `37.78` ms，P95 延迟为 `7.43` ms，P99 延迟为 `759.69` ms，最大延迟为 `8309.40` ms。 + +- `SLOWEST_THREAD` 是最慢客户端线程的累计操作时间，不是单次请求延迟。 + +示例中的数值仅用于展示输出格式。实际结果取决于硬件资源、网络环境、IoTDB 配置、数据规模和测试参数。 + +## 3\. 主要参数 + +### 3\.1 IoTDB 服务模型 + +IoTDB 2\.0\.x 支持树模型和表模型，通过以下参数选择： + +```Properties +IoTDB_DIALECT_MODE=tree +``` + +或： + +```Properties +IoTDB_DIALECT_MODE=table +``` + +相关约束如下： + +- 一个 IoTDB 实例在一次测试中只能选择一种 SQL 方言。 + +- 树模型要求 `DEVICE_NUMBER >= GROUP_NUMBER`。 + +- 表模型要求设备数是表数的整数倍，表数是数据库数的整数倍。 + +- 表模型下，数据客户端数应为表数的整数倍。 + + + +常用模型参数： + +|参数|示例|说明| +|---|---|---| +|`IoTDB_DIALECT_MODE`|`tree`|`tree` 或 `table`| +|`GROUP_NUMBER`|`1`|数据库数量；树模型下对应 database 数量| +|`IoTDB_TABLE_NUMBER`|`1`|表模型下创建的表数量| +|`IoTDB_TABLE_NAME_PREFIX`|`table_`|表名称前缀| +|`TABLE_TIME_COLUMN`|`time`|表模型时间列名称| +|`IoTDB_TABLE_WRITABLE_VIEW`|`false`|是否创建并使用可写视图| + +### 3\.2 工作模式 + +通过 `BENCHMARK_WORK_MODE` 选择工作模式。 -> 注意： +|模式|配置值|说明| +|---|---|---| +|常规测试模式|`testWithDefaultPath`|执行写入、查询或混合负载| +|生成数据模式|`generateDataMode`|将 Benchmark 生成的数据集保存到 `FILE_PATH`| +|正确性写入模式|`verificationWriteMode`|从 `FILE_PATH` 加载数据集并写入数据库| +|正确性查询模式|`verificationQueryMode`|加载数据集并与数据库查询结果进行比对| + +示例： + +```Properties +BENCHMARK_WORK_MODE=testWithDefaultPath +``` + +使用正确性写入和查询模式前，应先通过 `generateDataMode` 生成可复用数据集。 + +### 3\.3 服务器连接信息 + +|参数|示例|说明| +|---|---|---| +|`DB_SWITCH`|`IoTDB-200-SESSION_BY_TABLET`|数据库版本和连接方式| +|`HOST`|`127.0.0.1`|IoTDB 地址；多个地址使用英文逗号分隔| +|`PORT`|`6667`|原生服务端口，数量应与 `HOST` 一致| +|`USERNAME`|`root`|用户名| +|`PASSWORD`|`root`|密码| +|`DB_NAME`|`test`|测试使用的数据库名称| +|`REST_PORT`|`18080`|REST 服务端口| +|`REST_AUTHORIZATION`|`Basic cm9vdDpyb290`|REST 授权头| +|`ENABLE_AUTO_FETCH`|`false`|Session 是否自动刷新 DataNode 列表| + +清理数据相关参数： + +```Properties +IS_DELETE_DATA=false +INIT_WAIT_TIME=1000 +``` + +> 警告：`IS_DELETE_DATA=true` 会在测试开始前清理目标数据库中的测试数据。仅可在专用测试环境中启用，并在执行前确认 `HOST`、`PORT`、`DB_NAME` 和账号权限。 +> > -> 如果您关闭终端或失去与客户端机器的连接，测试过程将终止。 如果输出传输到终端，则与任何其他情况相同。 -使用此接口通常需要很长时间，您可能希望将测试过程作为守护程序执行。这样，您可以通过启动脚本将测试任务作为守护程序启动： +### 3\.4 写入场景 -```Bash -> ./rep-benchmark.sh > /dev/null 2>&1 & +1. 数据规模和客户端 + +|参数|示例|说明| +|---|---|---| +|`DEVICE_NUMBER`|`100`|设备总数| +|`SENSOR_NUMBER`|`10`|每个设备的测点数；表模型下为测点列数量| +|`GROUP_NUMBER`|`1`|IoTDB database 数量| +|`SCHEMA_CLIENT_NUMBER`|`5`|注册元数据的客户端数| +|`DATA_CLIENT_NUMBER`|`10`|执行数据读写的客户端数| +|`IS_CLIENT_BIND`|`true`|是否将设备绑定到客户端| +|`REAL_INSERT_RATE`|`1.0`|实际参与写入的设备比例| +|`IS_SENSOR_TS_ALIGNMENT`|`true`|同一设备下测点时间戳是否对齐| + +2. 批量写入 + +|参数|示例|说明| +|---|---|---| +|`BATCH_SIZE_PER_WRITE`|`100`|每批每个设备写入的数据行数| +|`DEVICE_NUM_PER_WRITE`|`1`|每批写入涉及的设备数| +|`CREATE_SCHEMA`|`true`|写入前是否创建元数据| +|`START_TIME`|`2022-01-01T00:00:00+08:00`|生成数据的起始时间| + +单个批次的数据点数为： + +```Plain Text +DEVICE_NUM_PER_WRITE × SENSOR_NUMBER × BATCH_SIZE_PER_WRITE ``` -在这种情况下，如果您想知道发生了什么，可以通过以下命令查看日志信息： +`DEVICE_NUM_PER_WRITE` 必须能够整除单个数据客户端所负责的设备数。表模型下还应满足设备数、表数和单批设备数之间的整除约束。 -```Bash -> cd ./logs -> tail -f log_info.log +3. 写入节奏 + +|参数|示例|说明| +|---|---|---| +|`POINT_STEP`|`5000`|相邻生成时间戳的固定间隔| +|`OP_MIN_INTERVAL`|`0`|每个 loop 的最小执行间隔，单位 ms| +|`OP_MIN_INTERVAL_RANDOM`|`false`|是否在 `[0, OP_MIN_INTERVAL)` 中随机选择间隔| +|`INTERVAL_BETWEEN_WRITE_BATCH`|`0`|同一 loop 内相邻 batch 的最小间隔，单位 ms| +|`TIMESTAMP_PRECISION`|`ms`|时间戳精度| + +`OP_MIN_INTERVAL` 的特殊值： + +- `0`：不限制 loop 间隔。 + +- `-1`：使用 `POINT_STEP` 作为最小间隔。 + +- 正整数：如果当前 loop 耗时不足该值，则等待剩余时间。 + +4. 乱序写入 + +```Properties +IS_OUT_OF_ORDER=false +OUT_OF_ORDER_MODE=POISSON +OUT_OF_ORDER_RATIO=0.5 +IS_REGULAR_FREQUENCY=true +``` + +支持的乱序模式包括： + +- `POISSON`：按照泊松分布生成乱序时间戳。 + +- `BATCH`：按批次生成乱序数据。 + +5. 数据类型 + +```Properties +INSERT_DATATYPE_PROPORTION=1:1:1:1:1:1:0:0:0:0:0 +``` + +各项顺序为： + +```Plain Text +BOOLEAN:INT32:INT64:FLOAT:DOUBLE:TEXT:STRING:BLOB:TIMESTAMP:DATE:OBJECT +``` + +各项数值表示相应数据类型的比例。 + +### 3\.5 查询场景 + +|参数|示例|说明| +|---|---|---| +|`QUERY_DEVICE_NUM`|`1`|每条查询涉及的设备数| +|`QUERY_SENSOR_NUM`|`1`|每条查询涉及的测点数| +|`QUERY_AGGREGATE_FUN`|`count`|聚合函数| +|`STEP_SIZE`|`0`|查询起始时间的变化步长，单位为 `POINT_STEP`| +|`QUERY_INTERVAL`|`250000`|查询起止时间间隔| +|`QUERY_LOWER_VALUE`|`-5`|值过滤条件下限| +|`GROUP_BY_TIME_UNIT`|`20000`|Group By 窗口大小| +|`QUERY_SET_OP_TYPE`|`union`|集合操作类型| +|`QUERY_SET_OP_NUM`|`2`|集合查询的子集合数量，至少为 2| +|`IS_RECENT_QUERY`|`false`|混合场景下是否优先查询最近写入的数据| +|`ENABLE_FIXED_QUERY`|`false`|是否让所有查询线程使用相同设备和测点组合| +|`RESULT_ROW_LIMIT`|`-1`|查询结果行数限制；`-1` 表示不使用限制| +|`ALIGN_BY_DEVICE`|`false`|是否使用 Align By Device| + +### 3\.6 操作比例 + +`OPERATION_PROPORTION` 用于定义写入和各类查询的比例，共包含 13 项： + +```Plain Text +写入:Q1:Q2:Q3:Q4:Q5:Q6:Q7:Q8:Q9:Q10:Q11:Q12 +``` + +例如，纯写入： + +```Properties +OPERATION_PROPORTION=1:0:0:0:0:0:0:0:0:0:0:0:0 +``` + +纯精确点查询： + +```Properties +OPERATION_PROPORTION=0:1:0:0:0:0:0:0:0:0:0:0:0 +``` + +操作类型如下。 + +|编号|操作类型|说明| +|---|---|---| +|写入|数据写入|按当前写入配置生成并写入数据| +|Q1|精确点查询|按时间戳和设备查询指定测点| +|Q2|时间范围查询|只限制起止时间的范围查询| +|Q3|带值过滤的范围查询|同时包含时间和值过滤条件| +|Q4|带时间过滤的聚合查询|在时间范围内执行聚合| +|Q5|带值过滤的聚合查询|在全时间范围内按值过滤并聚合| +|Q6|带时间和值过滤的聚合查询|同时包含时间和值过滤条件| +|Q7|时间分组聚合查询|Group By 查询| +|Q8|最近点查询|查询设备的最新数据点| +|Q9|倒序时间范围查询|按时间倒序返回范围查询结果| +|Q10|倒序带值过滤的范围查询|带值过滤并按时间倒序返回| +|Q11|倒序时间分组聚合查询|倒序 Group By 查询| +|Q12|集合操作查询|`union`、`intersect` 或 `except` 等集合操作| + +Q12 仅支持 IoTDB 2\.0 表模型。集合操作中的各子查询为范围查询。 + +### 3\.7 测试过程和结果持久化 + +```Properties +TEST_DATA_PERSISTENCE=None ``` -## 4. 实际案例 +支持的取值包括： + +- `None`：不将测试过程写入外部持久化介质。 + +- `CSV`：写入 CSV 文件。 + +- `MySQL`：写入 MySQL。 + +- `IoTDB`：写入指定的 IoTDB。 + +常用参数： -我们以中车青岛四方车辆研究所有限公司应用为例，参考《ＡｐａｃｈｅＩｏＴＤＢ在智能运维平台存储中的应用》中描述的场景进行实际操作说明。 +|参数|示例|说明| +|---|---|---| +|`TEST_DATA_PERSISTENCE`|`None`|持久化方式| +|`RECORD_SPLIT`|`true`|是否将结果拆分为多条记录| +|`RECORD_SPLIT_MAX_LINE`|`10000000`|单表或单文件的最大记录数| +|`TEST_DATA_STORE_IP`|`127.0.0.1`|结果数据库地址| +|`TEST_DATA_STORE_PORT`|`6667`|结果数据库端口| +|`TEST_DATA_STORE_DB`|`result`|结果数据库名称| +|`TEST_DATA_STORE_USER`|`root`|结果数据库用户名| +|`TEST_DATA_STORE_PW`|`root`|结果数据库密码| +|`REMARK`|`write_test`|测试备注，可用于区分不同测试| +|`CSV_OUTPUT`|`true`|是否将最终结果写入 CSV| -测试目标：模拟中车青岛四方所场景因切换时间序列数据库实际需求，对比预期使用的IoTDB和原有系统使用的KairosDB性能。 +设置为 CSV 时，运行后会在 `data` 目录下生成相关记录；其中测试结果通常位于 `data/csvOutput`。无论是否开启持久化，测试日志均写入 `logs`。 -测试环境：为了保证在实验过程中消除其他无关服务与进程对数据库性能的影响，以及不同数据库之间的相互影响，本实验中的本地数据库均部署并运行在资源配置相同的多个独立的虚拟机上。因此，本实验搭建了 4 台 Linux( CentOS7 /x86) 虚拟机，并分别在上面部署了IoT-benchmark、 IoTDB数据库、KairosDB数据库、MySQL数据库。每一台虚拟机的具体资源配置如表4-1所示。每一台虚拟机的具体用途如表4-2所示。 -表4-1虚拟机配置信息 -| 硬件配置信息 | 系统描述 | -| ------------ | -------- | -| OS System | CentOS7 | -| CPU核数 | 16 | -| 内存 | 32G | -| 硬盘 | 200G | -| 网卡 | 千兆 | +日志输出频率可通过以下参数控制： +```Properties +IS_QUIET_MODE=true +LOG_PRINT_INTERVAL=5 +RESULT_PRINT_INTERVAL=3600 +``` + +### 3\.8 自动化和集群测试 + +1. 限制测试时长 +```Properties +TEST_MAX_TIME=3600000 +``` -表4-2虚拟机用途 +单位为毫秒。设置为 `0` 表示不限制。该参数不包含预注册元数据所消耗的时间。 -| IP | 用途 | -| ---------- | ------------- | -| 172.21.4.2 | IoT-benchmark | -| 172.21.4.3 | Apache-iotdb | -| 172.21.4.4 | KaiosDB | -| 172.21.4.5 | MySQL | +2. 多 Benchmark 集群压测 -### 4.1 写入测试 +在多台客户端机器上使用相同的总体数据规模配置，并设置： -场景描述：创建100个客户端来模拟100列车、每列车3000个传感器、数据类型为DOUBLE类型、数据时间间隔为500ms（2Hz）、顺序发送。参考以上需求我们需要修改IoT-benchmark配置参数如表4-3中所列。 +```Properties +BENCHMARK_CLUSTER=true +BENCHMARK_INDEX=0 +``` -表4-3配置参数信息 +每个 Benchmark 实例必须使用不同的 `BENCHMARK_INDEX`，例如依次使用 `0`、`1`、`2`。所有客户端应保持数据库连接、数据规模和操作比例等配置一致。 -| 参数名称 | IoTDB值 | KairosDB值 | -| -------------------------- | --------------------------- | ---------- | -| DB_SWITCH | IoTDB-013-SESSION_BY_TABLET | KairosDB | -| HOST | 172.21.4.3 | 172.21.4.4 | -| PORT | 6667 | 8080 | -| BENCHMARK_WORK_MODE | testWithDefaultPath | | -| OPERATION_PROPORTION | 1:0:0:0:0:0:0:0:0:0:0 | | -| CLIENT_NUMBER | 100 | | -| GROUP_NUMBER | 10 | | -| DEVICE_NUMBER | 100 | | -| SENSOR_NUMBER | 3000 | | -| INSERT_DATATYPE_PROPORTION | 0:0:0:0:1:0 | | -| POINT_STEP | 500 | | -| OP_MIN_INTERVAL | 0 | | -| IS_OUT_OF_ORDER | false | | -| BATCH_SIZE_PER_WRITE | 1 | | -| LOOP | 10000 | | -| TEST_DATA_PERSISTENCE | MySQL | | -| TEST_DATA_STORE_IP | 172.21.4.5 | | -| TEST_DATA_STORE_PORT | 3306 | | -| TEST_DATA_STORE_DB | demo | | -| TEST_DATA_STORE_USER | root | | -| TEST_DATA_STORE_PW | admin | | -| REMARK | demo | | +3. 双写测试 -首先在172.21.4.3和172.21.4.4上分别启动被测时间序列数据库Apache-IoTDB和KairosDB，之后在172.21.4.2、172.21.4.3和172.21.4.4上通过ser-benchamrk.sh脚本启动服务器资源监控（图4-1）。然后按照表4-3在172.21.4.2分别修改iotdb-0.13-0.0.1和kairosdb-0.0.1文件夹内的conf/config.properties文件满足测试需求。先后使用benchmark.sh启动对Apache-IoTDB和KairosDB的写入测试。 +IoT Benchmark 支持将同一份数据写入两个不同的数据库进行对比： -![img](/img/bm5.png) +```Properties +IS_DOUBLE_WRITE=true +ANOTHER_DB_SWITCH=<另一数据库类型> +ANOTHER_HOST=127.0.0.1 +ANOTHER_PORT=6667 +ANOTHER_USERNAME=root +ANOTHER_PASSWORD=root +ANOTHER_DB_NAME=test +``` -图4-1服务器监控任务 +双写模式不支持同一数据库不同版本之间的比较，也不支持 IoTDB 树模型与表模型之间的直接比较。 -​ 例如我们首先启动对KairosDB的测试，IoT-benchmark会在MySQL数据库中创建CONFIG数据表存放本次测试配置信息（图4-2），测试执行中会有日志输出当前测试进度（图4-3）。测试完成时会输出本次测试结果（图4-3），同时将结果写入FINAL_RESULT数据表中（图4-4）。 +## 4\. 使用示例 -![](/img/bm6.png) +本节使用小规模数据演示基本流程。正式性能测试应根据目标业务模型扩大设备数、测点数、客户端数和循环次数，并进行多轮预热和重复测试。 -图4-2测试配置信息表 +### 4\.1 写入测试示例 -![](/img/bm7.png) -![](/img/bm8.png) -![](/img/bm9.png) -![](/img/bm10.png) +测试目标：使用 10 个数据客户端，模拟 100 台设备，每台设备包含 10 个测点，执行纯写入测试。 -图4-3测试进度和结果 +配置示例： -![](/img/bm11.png) +```Properties +# 数据库连接 +DB_SWITCH=IoTDB-200-SESSION_BY_TABLET +IoTDB_DIALECT_MODE=tree +HOST=127.0.0.1 +PORT=6667 +USERNAME=root +PASSWORD=root +DB_NAME=test -图4-4测试结果表 +# 安全设置：默认不自动删除已有数据 +IS_DELETE_DATA=false -之后我们再启动对Apache-IoTDB的测试，同样的IoT-benchmark会在MySQL数据库CONFIG数据表中写入本次测试配置信息，测试执行中会有日志输出当前测试进度。测试完成时会输出本次测试结果，同时将结果写入FINAL_RESULT数据表中。 +# 工作模式 +BENCHMARK_WORK_MODE=testWithDefaultPath +OPERATION_PROPORTION=1:0:0:0:0:0:0:0:0:0:0:0:0 -依照测试结果信息我们知道同样的配置写入Apache-IoTDB和KairosDB写入延时时间分别为：55.98ms和1324.45ms；写入吞吐分别为：5,125,600.86点/秒和224,819.01点/秒；测试分别执行了585.30秒和11777.99秒。并且KairosDB有写入失败出现，排查后发现是数据磁盘使用率已达到100%，无磁盘空间继续接收数据。而Apache-IoTDB无写入失败现象，全部数据写入完毕后占用磁盘空间仅为4.7G（如图4-5所示）；从写入吞吐和磁盘占用情况上看Apache-IoTDB均优于KairosDB。当然后续还有其他测试来从多方面观察和对比，比如查询性能、文件压缩比、数据安全性等。 +# 数据规模 +GROUP_NUMBER=1 +DEVICE_NUMBER=100 +SENSOR_NUMBER=10 +SCHEMA_CLIENT_NUMBER=5 +DATA_CLIENT_NUMBER=10 +IS_SENSOR_TS_ALIGNMENT=true -![](/img/bm12.png) +# 写入配置 +CREATE_SCHEMA=true +BATCH_SIZE_PER_WRITE=10 +DEVICE_NUM_PER_WRITE=1 +LOOP=100 +POINT_STEP=1000 +OP_MIN_INTERVAL=0 +START_TIME=2026-01-01T00:00:00+08:00 +INSERT_DATATYPE_PROPORTION=1:1:1:1:1:1:0:0:0:0:0 -图4-5磁盘使用情况 +# IoTDB 2.0.6 及以上版本可开启 +ENABLE_IOTDB_RPC_COMPRESSION=true -那么测试过程中各个服务器资源使用情况如何呢？每个写操作具体的表现如何呢？这个时候我们就可以通过安装和使用Tableau来可视化服务器监控表和测试过程记录表内的数据了。Tableau的使用本文不展开介绍，通过它连接测试数据持久化的数据表后具体结果下如图（以Apache-IoTDB为例）： +# 输出 +TEST_DATA_PERSISTENCE=None +CSV_OUTPUT=true +REMARK=iotdb_2_write_test +``` + +启动测试： + +```Bash +./benchmark.sh +``` -![](/img/bm13.png) -![](/img/bm14.png) +完成后重点查看： -图4-6Tableau可视化测试过程 +- `INGESTION` 的 `okPoint` 和 `failPoint`。 +- 吞吐量 `throughput`。 +- `AVG`、`P95`、`P99`、`MAX` 延迟。 -### 4.2 查询测试 +- 是否存在连接超时、写入失败或服务端异常日志。 -场景描述：在写入测试场景下模拟10个客户端对时序数据库Apache-IoTDB内存放的数据进行全类型查询任务。配置如下： +### 4\.2 查询测试示例 -表4-4配置参数信息 +执行查询测试前，应确保目标数据库中已经存在与查询配置匹配的数据。建议复用写入测试生成的数据，并关闭自动删数和元数据创建。 -| 参数名称 | 示例 | -| -------------------- | --------------------- | -| CLIENT_NUMBER | 10 | -| QUERY_DEVICE_NUM | 2 | -| QUERY_SENSOR_NUM | 2 | -| QUERY_AGGREGATE_FUN | count | -| STEP_SIZE | 1 | -| QUERY_INTERVAL | 250000 | -| QUERY_LOWER_VALUE | -5 | -| GROUP_BY_TIME_UNIT | 20000 | -| LOOP | 30 | -| OPERATION_PROPORTION | 0:1:1:1:1:1:1:1:1:1:1 | +以下示例同时执行多种查询： -执行结果： +```Properties +DB_SWITCH=IoTDB-200-SESSION_BY_TABLET +IoTDB_DIALECT_MODE=tree +HOST=127.0.0.1 +PORT=6667 +USERNAME=root +PASSWORD=root +DB_NAME=test -![](/img/bm15.png) +IS_DELETE_DATA=false +CREATE_SCHEMA=false +BENCHMARK_WORK_MODE=testWithDefaultPath -图4-7查询测试结果 +GROUP_NUMBER=1 +DEVICE_NUMBER=100 +SENSOR_NUMBER=10 +SCHEMA_CLIENT_NUMBER=1 +DATA_CLIENT_NUMBER=10 -### 4.3 其他参数说明 +# 不执行写入，Q1～Q11 各占相同比例；树模型不使用 Q12 +OPERATION_PROPORTION=0:1:1:1:1:1:1:1:1:1:1:1:0 +LOOP=100 + +QUERY_DEVICE_NUM=2 +QUERY_SENSOR_NUM=2 +QUERY_AGGREGATE_FUN=count +STEP_SIZE=1 +QUERY_INTERVAL=250000 +QUERY_LOWER_VALUE=-5 +GROUP_BY_TIME_UNIT=20000 +``` + +如果测试表模型集合查询，可将方言切换为 `table`，并为 Q12 设置比例： + +```Properties +IoTDB_DIALECT_MODE=table +OPERATION_PROPORTION=0:0:0:0:0:0:0:0:0:0:0:0:1 +QUERY_SET_OP_TYPE=union +QUERY_SET_OP_NUM=2 +``` + +### 4\.3 其他配置示例 + +1. 模拟真实写入速率 + +让每个 loop 的最小间隔与数据时间戳间隔一致： + +```Properties +POINT_STEP=1000 +OP_MIN_INTERVAL=-1 +``` + +如果希望写入请求在一个 loop 内均匀分布，可使用： + +```Properties +INTERVAL_BETWEEN_WRITE_BATCH=100 +``` + +2. 指定测试时长 + +测试 1 小时： + +```Properties +TEST_MAX_TIME=3600000 +``` + +应确保 `LOOP` 足够大，否则测试可能先因循环次数耗尽而结束。 + +3. 控制生成数据规律 + +```Properties +LINE_RATIO=1 +SIN_RATIO=1 +SQUARE_RATIO=1 +RANDOM_RATIO=1 +CONSTANT_RATIO=1 +DATA_SEED=666 +STRING_LENGTH=10 +DOUBLE_LENGTH=2 +``` -之前章节中针对Apache-IoTDB和KairosDB进行写入性能对比，但是用户如果要执行模拟真实写入速率测试该如何配置？测试时间过长该如何控制呢？生成的模拟数据有哪些规律吗？如果IoT-Benchmark服务器配置较低，可以使用多台机器模拟压力输出吗？ +固定 `DATA_SEED` 有助于在多轮测试中生成可重复的数据。 -表4-5配置参数信息 +## 5\. 参考资料 -| 场景 | 参数 | 值 | 说明 | -| ------------------------------------------------------------ | -------------------------- |---------------------------------------------------------------------------------------------------| --------------------------------- | -| 模拟真实写入速率 | OP_INTERVAL | -1 | 也可输入整数控制操作间隔 | -| 指定测试时长（1小时） | TEST_MAX_TIME | 3600000 | 单位 ms；需要LOOP执行时间大于该值 | -| 定义模拟数据规律：支持全部数据类型，数量平均分类；支持五种数据分布，数量平均分布；字符串长度为10；小数位数为2 | INSERT_DATATYPE_PROPORTION | 1:1:1:1:1:1 | 数据类型分布比率 | -| LINE_RATIO | 1 | 线性 | | -| SIN_RATIO | 1 | 傅里叶函数 | | -| SQUARE_RATIO | 1 | 方波 | | -| RANDOM_RATIO | 1 | 随机数 | | -| CONSTANT_RATIO | 1 | 常数 | | -| STRING_LENGTH | 10 | 字符串长度 | | -| DOUBLE_LENGTH | 2 | 小数位数 | | -| 三台机器模拟300台设备数据写入 | BENCHMARK_CLUSTER | true | 开启多benchmark模式 | -| BENCHMARK_INDEX | 0、1、3 | 以[写入测试](./Benchmark.md#_4-1-写入测试)写入参数为例：0号负责设备编号0-99数据写入；1号负责设备编号100-199数据写入；2号负责设备编号200-299数据写入； | | \ No newline at end of file +- [IoT Benchmark 文档](https://github.com/thulab/iot-benchmark/tree/master/docs)