[Opt](warmup) Limit the download rate of warmup task #60180

bobhan1 · 2026-01-23T08:49:54Z

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

Thearas · 2026-01-23T08:50:00Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

gavinchou · 2026-01-23T08:56:31Z

be/src/cloud/config.cpp


+// Rate limit for warmup download in bytes per second, default 100MB/s
+// 0 means no limit
+DEFINE_mInt64(warmup_download_rate_limit_bytes_per_second, "104857600");


suggest naming file_cache_warmup_download_rate_limit_bytes_per_second

gavinchou · 2026-01-23T09:00:03Z

be/src/runtime/exec_env_init.cpp

    _index_policy_mgr = new IndexPolicyMgr();

+    // Initialize warmup download rate limiter for cloud mode
+    if (config::is_cloud_mode() && config::warmup_download_rate_limit_bytes_per_second > 0) {


better dynamically configurable

dataroaring · 2026-01-23T18:04:09Z

run buildall

doris-robot · 2026-01-23T20:00:24Z

TPC-H: Total hot run time: 31379 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit eb945cc91a85690791d0cae80ec37b398827aa34, data reload: false

------ Round 1 ----------------------------------
q1	17631	4781	4569	4569
q2	2036	305	195	195
q3	10226	1311	732	732
q4	10212	835	306	306
q5	7540	2156	1875	1875
q6	191	171	137	137
q7	869	728	586	586
q8	9266	1406	1107	1107
q9	4905	4570	4550	4550
q10	6856	1697	1244	1244
q11	512	305	274	274
q12	396	390	226	226
q13	17773	3961	3107	3107
q14	249	236	216	216
q15	613	528	526	526
q16	645	670	576	576
q17	666	830	475	475
q18	6621	6492	6612	6492
q19	1109	1093	703	703
q20	430	410	265	265
q21	3241	2573	2163	2163
q22	1151	1104	1055	1055
Total cold run time: 103138 ms
Total hot run time: 31379 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5786	5005	4912	4912
q2	329	400	301	301
q3	2386	2882	2496	2496
q4	1533	1861	1454	1454
q5	4370	4485	4307	4307
q6	235	177	129	129
q7	2111	2002	1828	1828
q8	2672	2417	2391	2391
q9	7123	7340	7090	7090
q10	2549	2729	2350	2350
q11	545	467	469	467
q12	708	733	643	643
q13	3717	4091	3104	3104
q14	266	285	269	269
q15	533	505	494	494
q16	607	645	624	624
q17	1096	1285	1299	1285
q18	7313	7270	7270	7270
q19	847	823	811	811
q20	1885	2032	1864	1864
q21	4552	4448	4266	4266
q22	1031	1038	961	961
Total cold run time: 52194 ms
Total hot run time: 49316 ms

doris-robot · 2026-01-23T20:11:05Z

TPC-DS: Total hot run time: 172627 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit eb945cc91a85690791d0cae80ec37b398827aa34, data reload: false

query5	4347	661	486	486
query6	324	223	201	201
query7	4206	454	259	259
query8	342	260	240	240
query9	8690	2858	2891	2858
query10	450	316	275	275
query11	15314	15246	14841	14841
query12	179	124	121	121
query13	1247	468	369	369
query14	5578	3017	2888	2888
query14_1	2686	2708	2684	2684
query15	196	190	177	177
query16	1000	404	497	404
query17	1122	682	584	584
query18	2437	443	327	327
query19	197	178	154	154
query20	124	119	117	117
query21	212	138	117	117
query22	3879	4178	3866	3866
query23	16282	15572	15331	15331
query23_1	15421	15351	15310	15310
query24	7127	1539	1170	1170
query24_1	1178	1169	1190	1169
query25	545	477	423	423
query26	1240	267	154	154
query27	2757	464	281	281
query28	4546	2168	2167	2167
query29	809	553	469	469
query30	316	250	205	205
query31	775	642	555	555
query32	83	82	78	78
query33	563	367	323	323
query34	904	884	530	530
query35	725	773	690	690
query36	887	889	840	840
query37	141	107	91	91
query38	2755	2723	2672	2672
query39	782	751	742	742
query39_1	713	720	714	714
query40	234	135	120	120
query41	77	71	68	68
query42	101	99	97	97
query43	452	424	418	418
query44	1365	754	785	754
query45	186	186	177	177
query46	824	936	579	579
query47	1456	1467	1409	1409
query48	314	322	245	245
query49	599	424	368	368
query50	674	261	231	231
query51	3783	3752	3752	3752
query52	90	92	88	88
query53	217	224	169	169
query54	284	258	247	247
query55	83	78	76	76
query56	304	300	298	298
query57	1012	955	947	947
query58	266	255	260	255
query59	1998	2153	2062	2062
query60	345	341	318	318
query61	147	148	148	148
query62	379	353	314	314
query63	192	164	163	163
query64	4979	1129	884	884
query65	3764	3697	3739	3697
query66	1439	410	316	316
query67	15590	15562	15494	15494
query68	2391	1078	726	726
query69	396	315	281	281
query70	1045	970	915	915
query71	305	291	264	264
query72	5323	3181	3270	3181
query73	600	714	312	312
query74	8687	8707	8478	8478
query75	2265	2338	1867	1867
query76	2290	1060	651	651
query77	359	371	304	304
query78	9943	9945	9147	9147
query79	1164	909	574	574
query80	1315	522	437	437
query81	545	265	231	231
query82	1021	149	117	117
query83	366	252	245	245
query84	259	116	93	93
query85	883	499	412	412
query86	411	295	284	284
query87	2905	2840	2716	2716
query88	3534	2588	2562	2562
query89	302	271	253	253
query90	1948	173	159	159
query91	167	155	135	135
query92	74	72	70	70
query93	1173	1034	655	655
query94	654	313	290	290
query95	583	327	364	327
query96	654	491	225	225
query97	2350	2362	2346	2346
query98	206	203	201	201
query99	592	588	485	485
Total cold run time: 245719 ms
Total hot run time: 172627 ms

doris-robot · 2026-01-23T20:16:05Z

ClickBench: Total hot run time: 26.76 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit eb945cc91a85690791d0cae80ec37b398827aa34, data reload: false

query1	0.06	0.05	0.05
query2	0.10	0.04	0.04
query3	0.26	0.09	0.09
query4	1.61	0.12	0.11
query5	0.28	0.25	0.25
query6	1.15	0.65	0.65
query7	0.04	0.03	0.02
query8	0.06	0.04	0.04
query9	0.56	0.52	0.50
query10	0.58	0.55	0.55
query11	0.14	0.10	0.09
query12	0.14	0.10	0.12
query13	0.61	0.58	0.58
query14	0.96	0.94	0.95
query15	0.80	0.77	0.77
query16	0.39	0.38	0.40
query17	1.00	0.97	1.04
query18	0.23	0.21	0.21
query19	1.90	1.79	1.87
query20	0.02	0.01	0.01
query21	15.46	0.27	0.15
query22	5.24	0.05	0.04
query23	16.10	0.28	0.10
query24	0.93	0.51	0.48
query25	0.12	0.08	0.05
query26	0.13	0.13	0.14
query27	0.06	0.05	0.05
query28	3.88	1.08	0.88
query29	12.53	3.88	3.10
query30	0.29	0.14	0.11
query31	2.84	0.64	0.39
query32	3.28	0.56	0.46
query33	2.96	2.96	3.08
query34	16.11	5.06	4.40
query35	4.46	4.47	4.49
query36	0.64	0.50	0.50
query37	0.12	0.07	0.06
query38	0.07	0.04	0.04
query39	0.05	0.03	0.03
query40	0.17	0.13	0.13
query41	0.09	0.04	0.03
query42	0.05	0.03	0.03
query43	0.04	0.04	0.03
Total cold run time: 96.51 s
Total hot run time: 26.76 s

doris-robot · 2026-01-23T21:35:44Z

BE UT Coverage Report

Increment line coverage 12.50% (2/16) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	52.46% (19116/36442)
Line Coverage	35.83% (177583/495689)
Region Coverage	32.30% (137297/425014)
Branch Coverage	33.23% (59427/178826)

update

71694fd

bobhan1 requested review from dataroaring and gavinchou as code owners January 23, 2026 08:49

gavinchou reviewed Jan 23, 2026

View reviewed changes

change config name

eb945cc

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Opt](warmup) Limit the download rate of warmup task #60180

[Opt](warmup) Limit the download rate of warmup task #60180

bobhan1 commented Jan 23, 2026

Uh oh!

Thearas commented Jan 23, 2026

Uh oh!

gavinchou Jan 23, 2026

Uh oh!

gavinchou Jan 23, 2026 •

edited

Loading

Uh oh!

dataroaring commented Jan 23, 2026

Uh oh!

doris-robot commented Jan 23, 2026

Uh oh!

doris-robot commented Jan 23, 2026

Uh oh!

doris-robot commented Jan 23, 2026

Uh oh!

doris-robot commented Jan 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

[Opt](warmup) Limit the download rate of warmup task #60180

Are you sure you want to change the base?

[Opt](warmup) Limit the download rate of warmup task #60180

Conversation

bobhan1 commented Jan 23, 2026

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

Thearas commented Jan 23, 2026

Uh oh!

gavinchou Jan 23, 2026

Choose a reason for hiding this comment

Uh oh!

gavinchou Jan 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dataroaring commented Jan 23, 2026

Uh oh!

doris-robot commented Jan 23, 2026

Uh oh!

doris-robot commented Jan 23, 2026

Uh oh!

doris-robot commented Jan 23, 2026

Uh oh!

doris-robot commented Jan 23, 2026

BE UT Coverage Report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

gavinchou Jan 23, 2026 •

edited

Loading