Namespace(subparser='bench', bench_type='serve', dispatch_function=<function BenchmarkServingSubcommand.cmd at 0x73d54c4328e0>, trust_remote_code=False, seed=0, num_prompts=20, dataset_name='random', no_stream=False, dataset_path=None, no_oversample=False, skip_chat_template=False, enable_multimodal_chat=False, disable_shuffle=False, custom_output_len=256, custom_ensure_client_side_data=False, spec_bench_output_len=256, spec_bench_category=None, sonnet_input_len=550, sonnet_output_len=150, sonnet_prefix_len=200, sharegpt_output_len=None, timed_trace_chunk_hash_size=16, timed_trace_sec_multiplier=1, timed_trace_label_timestamp='timestamp', timed_trace_label_input_length='input_length', timed_trace_label_output_length='output_length', timed_trace_label_hash_ids='hash_ids', blazedit_min_distance=0.0, blazedit_max_distance=1.0, asr_max_audio_len_sec=inf, asr_min_audio_len_sec=0.0, random_input_len=1024, random_output_len=256, random_range_ratio='0.0', random_prefix_len=0, random_batch_size=1, no_reranker=False, random_mm_base_items_per_request=1, random_mm_num_mm_items_range_ratio=0.0, random_mm_limit_mm_per_prompt={'image': 255, 'video': 1}, random_mm_bucket_config={(256, 256, 1): 0.5, (720, 1280, 1): 0.5, (720, 1280, 16): 0.0}, hf_subset=None, hf_split=None, hf_name=None, hf_output_len=None, bfcl_categories=None, prefix_repetition_prefix_len=256, prefix_repetition_suffix_len=256, prefix_repetition_num_prefixes=10, prefix_repetition_output_len=128, speed_bench_dataset_subset='qualitative', speed_bench_output_len=4096, speed_bench_category=None, label=None, backend='openai-chat', base_url='http://localhost:8000', host='127.0.0.1', port=8000, endpoint='/v1/chat/completions', header=None, max_concurrency=1, model='q', input_len=None, output_len=None, tokenizer='/workspace/models/awq', tokenizer_mode='auto', use_beam_search=False, logprobs=None, request_rate=inf, burstiness=1.0, probe_request_rate=0.0, disable_tqdm=False, num_warmups=0, profile=False, save_result=False, save_detailed=False, append_result=False, metadata=None, result_dir=None, result_filename=None, ignore_eos=False, self_timed=None, percentile_metrics=None, metric_percentiles='99', goodput=None, request_id_prefix='bench-ed2f276a-', top_p=None, top_k=None, min_p=None, temperature=None, frequency_penalty=None, presence_penalty=None, repetition_penalty=None, served_model_name=None, lora_modules=None, lora_assignment='random', ramp_up_strategy=None, ramp_up_start_rps=None, ramp_up_end_rps=None, ready_check_timeout_sec=0, chat_template_kwargs=None, extra_body=None, skip_tokenizer_init=False, insecure=False, plot_timeline=False, timeline_itl_thresholds='25,50', plot_dataset_stats=False)
INFO 09-27 09:46:43 [utils.py:90] Sampling input_len from [1024, 1024] and output_len from [256, 256]
WARNING: vllm bench serve no longer sets temperature==0 (greedy) in requests by default. The default will be determined on the server side and can be model/API specific. For the old behavior, include --temperature=0.
Starting initial single prompt test run...
Skipping endpoint ready check.
Starting main benchmark run...
Traffic request rate: inf
Burstiness factor: 1.0 (Poisson process)
Maximum request concurrency: 1

  0%|          | 0/20 [00:00<?, ?it/s]
  5%|▌         | 1/20 [00:03<01:06,  3.52s/it]
 10%|█         | 2/20 [00:07<01:03,  3.53s/it]
 15%|█▌        | 3/20 [00:10<00:59,  3.53s/it]
 20%|██        | 4/20 [00:14<00:56,  3.53s/it]
 25%|██▌       | 5/20 [00:17<00:52,  3.53s/it]
 30%|███       | 6/20 [00:21<00:49,  3.53s/it]
 35%|███▌      | 7/20 [00:24<00:45,  3.53s/it]
 40%|████      | 8/20 [00:28<00:42,  3.53s/it]
 45%|████▌     | 9/20 [00:31<00:38,  3.53s/it]
 50%|█████     | 10/20 [00:35<00:35,  3.53s/it]
 55%|█████▌    | 11/20 [00:38<00:31,  3.53s/it]
 60%|██████    | 12/20 [00:42<00:28,  3.53s/it]
 65%|██████▌   | 13/20 [00:45<00:24,  3.53s/it]
 70%|███████   | 14/20 [00:49<00:21,  3.53s/it]
 75%|███████▌  | 15/20 [00:52<00:17,  3.53s/it]
 80%|████████  | 16/20 [00:56<00:14,  3.53s/it]
 85%|████████▌ | 17/20 [01:00<00:10,  3.53s/it]
 90%|█████████ | 18/20 [01:03<00:07,  3.53s/it]
 95%|█████████▌| 19/20 [01:07<00:03,  3.53s/it]
100%|██████████| 20/20 [01:10<00:00,  3.53s/it]
100%|██████████| 20/20 [01:10<00:00,  3.53s/it]
tip: install termplotlib and gnuplot to plot the metrics
============ Serving Benchmark Result ============
Successful requests:                     20        
Failed requests:                         0         
Maximum request concurrency:             1         
Benchmark duration (s):                  70.61     
Total input tokens:                      21529     
Total generated tokens:                  5120      
Request throughput (req/s):              0.28      
Output token throughput (tok/s):         72.52     
Peak output token throughput (tok/s):    79.00     
Peak concurrent requests:                2.00      
Total token throughput (tok/s):          377.43    
---------------Time to First Token----------------
Mean TTFT (ms):                          278.19    
Median TTFT (ms):                        278.09    
P99 TTFT (ms):                           280.63    
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          12.75     
Median TPOT (ms):                        12.75     
P99 TPOT (ms):                           12.77     
---------------Inter-token Latency----------------
Mean ITL (ms):                           12.70     
Median ITL (ms):                         12.76     
P99 ITL (ms):                            14.18     
==================================================