dvmazur/mixtral-offloading

Issues

• Having issue loading my HQQ quantized model

  #35 opened 2 months ago by BeichenHuang
  0

• How to split the model parameter safetensors file into multiple small files

  #34 opened 2 months ago by YLSnowy
  5

• Implementation of benchmarks (C4 perplexity, Wikitext perplexity)

  #33 opened 2 months ago by ChengSashankh
  0

• runtimeerror when nbit = 4 and group_size =64

  #32 opened 2 months ago by Eutenacity
  0

• (Colab) Clear GPU RAM usage after running the generation code without restarting instance

  #28 opened 2 months ago by ScottLinnn
  1

• Can this be used for Jambo inference

  #30 opened 2 months ago by freQuensy23-coder
  1

• Trition Issues in Running the Code Locally

  #31 opened 2 months ago by amangupt01
  1

• Can it run on multi-GPU?

  #15 opened 5 months ago by drdh
  10

• a strange issue with default parameters " RuntimeError about memory"

  #26 opened 3 months ago by a1564740774
  0

• Run on second GPU (torch.device("cuda:1"))

  #24 opened 4 months ago by imabot2
  1

• Update Requirements.txt

  #25 opened 4 months ago by Soumadip-Saha
  0

• 4bit-3bit model produces gibberish when plugged into demo

  #23 opened 4 months ago by jjblum
  0

• Run without quantization

  #22 opened 5 months ago by freQuensy23-coder
  9

• Is it possible to finetune this on a custom dataset?

  #17 opened 5 months ago by asmith26
  7

• hqq_aten package not installed.

  #21 opened 5 months ago by LeMoussel
  1

• need mixtral offload for NousResearch/Nous-Hermes-2-Mixtral-8x7B-DPO

  #19 opened 5 months ago by githubpradeep
  0

• CUDA OOM errors in wsl2

  #18 opened 5 months ago by MrNova111
  0

• Can it run with LlamaIndex?

  #16 opened 5 months ago by LeMoussel
  0

• How to use the offloading in my MoE model?

  #13 opened 5 months ago by WangRongsheng
  4

• Doesn't work

  #14 opened 5 months ago by SanskarX10
  10

• Session crashed on colab

  #7 opened 5 months ago by bitsnaps
  4

• Mixtral OffLoading/GGUF/ExLlamaV2, which approach to use?

  #11 opened 5 months ago by LeMoussel
  1

• exl2

  #4 opened 5 months ago by eramax
  2

• Enhancing the Efficacy of MoE Offloading with Speculative Prefetching Strategies

  #10 opened 5 months ago by yihong1120
  0