microsoft/MInference

Issues

• [Bug]: vllm executor.driver_worker. 'RayWorkerWrapper' object has no attribute 'model_runner'

  #67 opened a month ago by TPLink32
  8

• [Question]: How can I generate topk_dims_file_path with self-trained model when in searching pattern?

  #36 opened 2 months ago by Amanda-Barbara
  5

• Performance Degradation when Using MInference with Qwen2-7B-Instruct Model

  #71 opened 18 days ago by yumingfan-0219
  1

• [Question]: Memory measurement

  #68 opened a month ago by HayeonLee
  2

• [Bug]: Normalization of QK product to estimate top-k tokens and slashes

  #69 opened a month ago by PiotrNawrot
  3

• [Question]: OOM occurs when reproducing end-to-end latency tests with a single A100

  #65 opened a month ago by HayeonLee
  2

• [Question]: Confusion about Optimal Search Pattern Configuration

  #64 opened a month ago by Dianaia
  2

• [Question]: It seems that minference does not currently support tensor parallelism under vllm, right? Because in a multi-card environment, the head_id here is incorrect compared to a single card

  #62 opened a month ago by zh2333
  2

• [Question]: AssertionError: The model /workspace/model/llm/Qwen/Qwen2-7B-Instruct you specified is not supported.

  #60 opened a month ago by LIUKAI0815
  2

• [Feature Request]: Support Mistral Model

  #39 opened 2 months ago by PatchouliTIS
  10

• Does MInference supports CUDA11.8?

  #56 opened a month ago by hensiesp32
  4

• [Bug]: When I'm doing distributed inference using vllm and minference, the example starts with an error when I set tensor_parallel_size to a value greater than 1

  #63 opened a month ago by zh2333
  1

• [Question]: RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method

  #58 opened a month ago by IvanDeng0
  3

• [Question]: Errors when I reproduce results in Table 5 (MInference + SnapKV) & poor results with attn_type=minference_with_dense

  #55 opened a month ago by HayeonLee
  2

• [Question]: Why is every head config saved with "vertical_and_slash"?

  #57 opened a month ago by fmmoret
  1

• [Question]: How does VLLM use MInference through OpenAI Compatible Server?

  #40 opened 2 months ago by jueming0312
  2

• [ToDo]: V0.1.5 Iteration Plan

  #27 opened 2 months ago by iofu728
  0

• [ToDo]: V0.1.6 Iteration Plan

  #50 opened 2 months ago by iofu728
  0

• [Question]: attn_type="minference" and attn_type= "hf" got different result

  #52 opened 2 months ago by qiling1345
  1

• Shape of slash mismatch when input batchsize > 1

  #53 opened 2 months ago by polarispw
  0

• [Question]: FlashAttention only supports Ampere GPUs or newer

  #51 opened 2 months ago by qiling1345
  3

• [Question]: Question about the settings of vertical_size and slash_size in vertical_and_slash pattern

  #47 opened 2 months ago by ALUKErnel
  4

• [Question]: MInference Pre filling is slower than the vllm original version

  #18 opened 2 months ago by junior-zsy
  6

• [Question]: Hi，it seems that Qwen2-7B-Instruct is not support in my case

  #48 opened 2 months ago by lwj2001
  1

• [Question]: run with local models

  #41 opened 2 months ago by qiling1345
  4

• [Question]: I got this error "No CUDA GPUs are available", how can I run with CPU ?

  #38 opened 2 months ago by qiling1345
  2

• [Question]: Does vertical_slash_sparse_attention supported to concatenate all batches into a single row for operation like flash_attn_2_cuda.varlen_fwd?

  #46 opened 2 months ago by Amanda-Barbara
  1

• [Question]: ModuleNotFoundError: No module named 'minference.cuda'

  #45 opened 2 months ago by lai-serena
  1

• [Bug]: NameError: name 'cache_ops' is not defined

  #42 opened 2 months ago by Zoro528
  2

• [Question]: Why is running MInference/examples/run_vllm.py not as fast as running vllm alone?

  #43 opened 2 months ago by zjjznw123
  1

• [Question]: Run Local Qwen2-7B-Instruct Model Failed, show not supported。

  #34 opened 2 months ago by wgimperial
  5

• [Question]: "Building wheel for flash_attn (setup.py) " for a long time without any notification

  #37 opened 2 months ago by qiling1345
  1

• [Question]: Is A6000 supported?

  #23 opened 2 months ago by yawzhe
  2

• [Question]: The speed of minference in short context

  #33 opened 2 months ago by maxin9966
  1

• [Question]: RuntimeError: indices should be either on cpu or on the same device as the indexed tensor (cuda:1)

  #25 opened 2 months ago by LiweiPE
  1

• [Bug]: missing warnings import in `setup.py`

  #28 opened 2 months ago by devangvin
  2

• [Question]: Very low quality outputs for long sequences

  #29 opened 2 months ago by cckao
  1

• [Question]: Question about KV-cache storage

  #20 opened 2 months ago by DerrickYLJ
  5

• [Feature Request]: Could you please support microsoft/Phi-3-medium-128k-instruct? Thank you!

  #26 opened 2 months ago by cckao
  1

• [Question]: python run_vllm.py TypeError: 'type' object is not subscriptable

  #13 opened 2 months ago by junior-zsy
  6

• [Bug]: MInference必须使用fla-attention吗？加速推理，A6000服务器不支持flas-attention

  #24 opened 2 months ago by yawzhe
  1

• [Question]: vertical slash pattern

  #21 opened 2 months ago by SimJeg
  1

• [Question]: Some questions on the code

  #17 opened 2 months ago by cyLi-Tiger
  4

• [Question]: For the tests such as RULER and InfiniteBench mentioned in the paper, what datasets are used to search for patterns?

  #16 opened 2 months ago by hijkzzz
  5

• [Question]: pip install minference error: cannot import name 'packaging' from 'pkg_resources'

  #12 opened 2 months ago by junior-zsy
  1

• [Question]: Hope to supplement the situation of increasing HBM usage with the context.

  #7 opened 2 months ago by Arcmoon-Hu
  2

• Action required: migrate or opt-out of migration to GitHub inside Microsoft

  #1 opened 3 months ago by microsoft-github-policy-service
  3