Keeping up with the latest LLMs !
History
- 2024.7 🔥東工大からLlama3の日本語継続学習モデルが発表！ - 2024.6 🔥ELYZAからLlama3の日本語継続学習モデルが発表！ - 2024.6 🔥Googleから27BのGemma2が公開！何が強みか教えて！ - 2024.6 🔥NVIDIAが340Bの巨大モデルを公開！publicにしては最大級 - 2024.6 🔥QWen2シリーズが登場！日本語も優秀！ - 2024.5 🔥MicrosoftからPhi-3シリーズが登場！ - 2024.5 🔥Stockmarkから100Bの日本語モデルがリリース!さすがGENIAC - 2024.4 🔥MetaからLlama3がリリース!まずは8Bと70B! - 2024.4 🔥CohereからCommand-R+がリリース!研究用に重みも公開. - 2024.4 🔥Databricksより132BのMoEモデルが公開されました！大きい！ - 2024.3 Cohereからプロダクション向けCommand-Rがリリース!研究用に重みも公開. - 2024.3 ELYZAからLlama2の追加学習日本語モデルのデモがリリースされました！ - 2024.3 東工大からMixtralの追加学習日本語モデル[Swallow-MX](), [Swallow-MS]()がリリースされました！👏 - 2024.2 GoogleからGeminiで用いられているLLM [Gemma](https://blog.google/technology/developers/gemma-open-models/)をオープンにするとのお達しが出ました! - 2024.2 Kotoba Technologyと東工大から[日本語Mamba 2.8B](https://huggingface.co/kotoba-tech/kotomamba-2.8B-v1.0)が公開されました! - 2024.2 Alibabaの[QWen](https://qwenlm.github.io/blog/qwen1.5/)が1.5にアップグレードされました！！ - 2024.2 Reka AIから21BでGemini Pro, GPT-3.5超えと発表されました. - 2024.2 LLM-jpのモデルが更新されました！v1.1 - 2024.2 カラクリから70B日本語LLMが公開されました！ - 2024.1 [リコー](https://www.nikkei.com/article/DGXZRSP667803_R30C24A1000000/)が13B日本語LLMを発表しました！ - 2024.1 Phi-2のMoE, Phixtralが公開されました！ - 2023.12 Phi-2のライセンスがMITに変更されました！ - 2023.12 ELYZAから日本語[13Bモデル](https://huggingface.co/elyza/ELYZA-japanese-Llama-2-13b)がリリースされました. - 2023.12 東工大から[Swallow](https://tokyotech-llm.github.io)がリリースされました. - 2023.12 MistralAIから[Mixtral-8x7B](https://github.com/open-compass/MixtralKit)がリリースされました. - 2023.12 [日本語LLMの学習データを問題視する記事](https://github.com/AUGMXNT/shisa/wiki/A-Review-of-Public-Japanese-Training-Sets#analysis)が公開されました.| When? | Name | HF? | Size(max) | License | pretraining/base | finetuning | misc. |
|---|---|---|---|---|---|---|---|
| 2024.7 | Reflection | HF | 70B | Llama3.1 | Llama 3.1 | synthetic data (Glaive) | |
| 2024.7 | Llama3.1(Meta) | HF | 70B, 405B | Llama3.1 | |||
| 2024.6 | Gemma2(Google) | HF | 2B, 9B, 27B | gemma | |||
| 2024.6 | Nemotron(NVIDIA) | HF | 340B | - | - | ||
| 2024.6 | Qwen2(Alibaba) | HF | 7~72B | tongyi-qianwen | - | - | |
| 2024.4 | Phi-3(Microsoft) | HF | 3.8B, 13B | MIT | Phi-3 datasets | - | |
| 2024.4 | Llama 3(Meta) | HF | 70B | META LLAMA3 | extended to 120B | ||
| 2024.4 | Wizart-8x22B(Microsoft) | HF | 8x22B | apache-2.0 | Mixtral-8x22B(Mistral) | MoE, closed now | |
| 2024.4 | Mixtral-8x22B(Mistral) | HF | 8x22B | apache-2.0 | MoE | ||
| 2024.4 | Command-R+(Cohere) | HF | 104B | non commercial | RAG capability | ||
| 2024.4 | DBRX(Databricks) | HF | 132B | databricks | MoE | ||
| 2024.3 | Grok-1 | 314B | MoE | ||||
| 2024.3 | BTX(Meta) | MoE | |||||
| 2024.3 | Command-R(Cohere) | HF | 35B | non commercial | RAG capability | ||
| 2024.2 | Aya(Cohere) | HF | 13B | apache-2.0 | multilingual | ||
| 2024.2 | Gemma(Google) | 8.5B | application open for reseachers | ||||
| 2024.2 | Miqu | HF | 70B | none | leaked from Mistral | ||
| 2024.2 | Reka Flash | 21B | not public | ||||
| 2024.1 | Self-Rewarding(Meta) | arxiv | 70B | Llama2 | Llama2 | - | DPO |
| 2024.1 | Phixtral | HF | 2.7Bx4 | MIT | MoE | ||
| 2023.12 | LongNet(Microsoft) | arXiv | - | apache-2.0 | MAGNETO | input 1B token | |
| 2023.12 | Phi-2(Microsoft) | HF | 2.7B | MIT | |||
| 2023.12 | gigaGPT(Cerebras) | 70B, 175B | apache-2.0 | ||||
| 2023.12 | Mixtral-8x7B | HF | 8x7B | apache-2.0 | MoE, offloading | ||
| 2023.12 | Mamba | HF | 2.8B | apache-2.0 | based on state space model | ||
| 2023.11 | QWen(Alibaba) | HF | 72B | license | 3T tokens | beats Llama2 | |
| 2023.10 | Self-RAG | HF | apache-2.0 | 13B | critic model | ||
| 2023.9 | TinyLlama | HF | apache-2.0 | 1.1B | based on Llama, 3T token | ||
| 2023.9 | Xwin-LM | HF | 70B | Llama2 | based on Llama2 | also codes and math | |
| 2023.7 | Llama2(Meta) | HF | 70B | Llama2 | 2T tokens | chat-hf seems the best | |
| name | HF |
- PaLM(540B), PaLM2(340B) and GPT-4 are not open.
- MoE : mixture of experts
| When? | Name | HF? | Size | License | pretraining | finetuning | misc. |
|---|---|---|---|---|---|---|---|
| 2024.7 | Llama-3.1-70B-Japanese-Instruct-2407 | HF | 70B | Llama3.1 | Llama3.1 | ||
| 2024.7 | LLama3-Swallow | HF | 70B | Llama3 | Llama3 | ||
| 2024.6 | LLama3ELYZA-JP-8B | HF | 8B | Llama3 | Llama3 | 70B not open | |
| 2024.6 | KARAKURI LM 8x7B | HF | 8x7B | Apache-2.0 | MoE | ||
| 2024.5 | Stockmark-100B | HF | 100B | MIT | |||
| 2024.3 | youko(rinna) | HF | 8B | Llama3 | Llama3 | ||
| 2024.3 | EvoLLM-JP | HF | 7B | MSR(non-commercial) | |||
| 2024.3 | Swallow-MX(東工大) | HF | 8x7B | Mixtralベース | |||
| 2024.2 | KARAKURI 70B | HF | 70B | cc-by-sa-4.0 | Llama2-70Bベース | note | |
| 2023.12 | ELYZA-japanese-Llama-2-13b | HF | 13B | Llama-2-13b-chatベース | |||
| 2023.12 | Swallow(東工大) | HF | 70B | Llama2-70Bベース | |||
| 2023.11 | StableLM(StabilityAI) | HF | 70B | Llama2-70Bベース | |||
| 2023.10 | LLM-jp | HF | 13B | DPO追加あり | |||
| name | HF |
See more on awesome-japanese-llm and 日本語LLM評価
| When? | Name | HF? | Size | License | pretraining | finetuning/continual | test | misc. |
|---|---|---|---|---|---|---|---|---|
| 2024.8 | LLaVA-Med++ | 8B | ? | MedTrinity-25M | VQA-RAD etc. | |||
| 2024.7 | MedLlama3-JP(EQUES) | HF | 8B | Llama3 | Llama3 | japanese, merge model | ||
| 2024.7 | Llama3-Preferred-MedSwallow | HF | 70B | Llama3 | Llama3 | japanese | ||
| 2024.7 | Med42-v2 | HF | 8,70B | Llama3 | llama3 | ~1B tokens, including medical flashcards, exam questions, and open-domain dialogues. | ||
| 2024.7 | JMedLLM-v1 | HF | 7B | qwen | Qwen2 | japanese | ||
| 2024.6 | MedSwallow | HF | 70B | cc-by-nc-sa | Swallow | japanese | ||
| 2024.5 | MMed-LLama3-8B(上海交通大学) | HF | 8B | cc-by-sa | Llama3 | |||
| 2024.5 | medX(JiviAI) | HF | 8B | Apache-2.0 | Llama3 | 100,000+ data, ORPO | ||
| 2024.4 | UltraMedical(TsinghuaC3I) | HF | 8B | - | Llama3 | |||
| 2024.4 | Meditron(EPFL) | - | 8B | - | Llama3 | MedQA, MedMCQA, PubmedQA | SOTA | |
| 2024.4 | OpenBioLLM-70B | HF | ? | - | SOTA | |||
| 2024.4 | Med-Gemini(Google) | closed | ? | - | Gemini | multimodal | ||
| 2024.4 | Hippocrates | HF | 7B | |||||
| 2024.3 | AdaptLLM(Microsoft Research) | HF | 7B, 13B | reading comprehensive corpora | ||||
| 2024.3 | Apollo | HF | ~7B | |||||
| 2024.2 | BiMediX | HF | non-commercial | 8x7B | mixtral8x7B | MoE | ||
| 2024.2 | Health-LLM(Rutgersなど) | RAG | ||||||
| 2024.2 | BioMistral | HF | 7B | - | ||||
| 2024.1 | AMIE(Google) | not open | - | - | based on PaLM 2 | EHR | ||
| 2023.12 | Medprompt(Microsoft) | not open | - | - | GPT-4 | none | multi-modal | |
| 2023.12 | JMedLoRA(UTokyo) | HF | 70B | none | none | QLoRA | IgakuQA | Japanese, insufficient quality |
| 2023.11 | Meditron(EPFL) | HF | 70B | Llama2 | Llama2 | GAP-Replay(48.1B) | dataset,score | |
| 2023.8 | BioMedGPT(Luo et al.) | HF | 10B | |||||
| 2023.8 | PMC-LLaMa | HF | 13B | |||||
| 2023.7 | Med-Flamingo | HF | 8.3B | ? | OpenFlamingo | MTB | Visual USMLE | based on Flamingo |
| 2023.7 | LLaVa-Med(Microsoft) | HF | 13B | - | LLaVa | medical dataset | VAQ-RAD, SLAKE, PathVQA | multi-modal |
| 2023.7 | Med-PaLM M(Google) | not open | - | PaLM2 | multi-modal | |||
| 2023.5 | Almanac(Stanford), journal | ? | ? | text-davinci-003 | RAG | |||
| 2023.5 | Med-PaLM2(Google) | not open | 340B | - | PaLM2 | |||
| 2022.12 | Med-PaLM(Google) | not open | 540B | - | PaLM | |||
| name | HF |
See also
For Japanese medical dataset, see JMedData4LLM.
- MedQA (USMLE)
- MedMCQA
- PubMedQA
- PubHealth
- MMLU : includes medicine and other related fields(clinical topics covering clinical knowledge, college biology, college medicine, medical genetics, professional medicine and anatomy)
- HeadQA : Spanish healthcare system
- K-Q&A
- Clincal Case Challenges : NEHM dataset and JAMA dataset
- MeDiSumQA : discharge summaries from the MIMIC-IV
- MeDiSumCode : ICD-10 codes
- MedNLI : MIMIC-III dataset, logical relationship between a premise and a hypothesis
- MeQSum : summarizing health queries
- LongHealth : 20 patient records, answer questions about them from a long document.
- MTB: chopped cleaned text and images collected from 4721 textbooks.
- PMC-15M : the largest biomedical image-text dataset
- PMC-OA : 1.6M image-caption pairs
- MedICaT: image, caption, textual reference
- VQA-RAD : 3515 question–answer pairs on 315 radiology images.
- SLAKE : bilingual dataset (English&Chinese) consisting of 642 images and 14,028 question-answer pairs
- PathVQA : pathology image + caption
- Visual USMLE : 618 USMLE-style QA
- MedVTE: numeric understanding
- MedAlign(Stanford)
- MIMIC-ECG-IV : ECG-caption dataset
- ECG-QA
- MedEval
- MedTrinity
- Clinical NLP 2023
See more on He et al.(2023)