Statistics

Sync Load and Async Load In my previous article, I introduced the statistics in TiDB and how it initializes them. However, there is an issue: even with comprehensive initialization, statistics may still be missing for columns that are not indexed. Additionally, after initialization, statistics may be evicted from memory due to memory pressure. In this article, I will introduce two methods to load statistics in TiDB on the fly: Sync Load and Async Load....

Statistics Statistics collection is a crucial process of modern database systems, forming the backbone of query optimization. In TiDB, statistics are indispensable, serving as the sole source of information for estimating query costs and selecting the most efficient execution plan. TiDB collects several types of statistics for each table, including: TopN values (most frequent values to reflect data skewness) Histograms (data distribution) Number of Distinct Values (NDV) Other statistical metrics These statistics will be stored in some system tables, such as mysql....

Background Recently, we have been tackling the challenge of supporting 3 million tables within a single TiDB cluster. One of the most significant hurdles we’ve faced is optimizing the performance of statistics collection. In its current implementation, TiDB gathers basic table information from all servers and consolidates it into a single system table. While functional, this approach becomes highly inefficient when managing millions of tables, consuming excessive CPU and taking a considerable amount of time....

Background TiDB provides the analyze table table_name command to generate statistics for tables. When analyzing partitioned tables, TiDB processes each partition independently and in parallel. Once analysis completes, TiDB aggregates the individual partition statistics into a single global statistics object for the entire table. The analyze table command has two concurrency-related parameters: tidb_build_stats_concurrency: Determines how many partitions can be processed simultaneously during statistics collection tidb_analyze_partition_concurrency: Controls parallel workers for saving partition statistics However, these parameters have several drawbacks:...