beomi / llama-2-ko-7b

huggingface.co
Total runs: 5.8K
24-hour runs: 0
7-day runs: 0
30-day runs: 0
Model's Last Updated: 2023年12月27日
text-generation

Introduction of llama-2-ko-7b

Model Details of llama-2-ko-7b

Update Log

Llama-2-Ko 🦙🇰🇷

Llama-2-Ko serves as an advanced iteration of Llama 2, benefiting from an expanded vocabulary and the inclusion of a Korean corpus in its further pretraining. Just like its predecessor, Llama-2-Ko operates within the broad range of generative text models that stretch from 7 billion to 70 billion parameters. This repository focuses on the 7B pretrained version, which is tailored to fit the Hugging Face Transformers format. For access to the other models, feel free to consult the index provided below.

Model Details

Model Developers Junbum Lee (Beomi)

Variations Llama-2-Ko will come in a range of parameter sizes — 7B, 13B, and 70B — as well as pretrained and fine-tuned variations.

Input Models input text only.

Output Models generate text only.

Model Architecture

Llama-2-Ko is an auto-regressive language model that uses an optimized transformer architecture based on Llama-2.

Training Data Params Content Length GQA Tokens LR
Llama 2 A new mix of Korean online data 7B 4k >40B* 1e -5
*Plan to train upto 200B tokens

Vocab Expansion

Model Name Vocabulary Size Description
Original Llama-2 32000 Sentencepiece BPE
Expanded Llama-2-Ko 46336 Sentencepiece BPE. Added Korean vocab and merges

Tokenizing "안녕하세요, 오늘은 날씨가 좋네요."

Model Tokens
Llama-2 ['▁', '안', '<0xEB>', '<0x85>', '<0x95>', '하', '세', '요', ',', '▁', '오', '<0xEB>', '<0x8A>', '<0x98>', '은', '▁', '<0xEB>', '<0x82>', '<0xA0>', '씨', '가', '▁', '<0xEC>', '<0xA2>', '<0x8B>', '<0xEB>', '<0x84>', '<0xA4>', '요']
Llama-2-Ko ['▁안녕', '하세요', ',', '▁오늘은', '▁날', '씨가', '▁좋네요']

Tokenizing "Llama 2: Open Foundation and Fine-Tuned Chat Models"

Model Tokens
Llama-2 ['▁L', 'l', 'ama', '▁', '2', ':', '▁Open', '▁Foundation', '▁and', '▁Fine', '-', 'T', 'un', 'ed', '▁Ch', 'at', '▁Mod', 'els']
Llama-2-Ko ['▁L', 'l', 'ama', '▁', '2', ':', '▁Open', '▁Foundation', '▁and', '▁Fine', '-', 'T', 'un', 'ed', '▁Ch', 'at', '▁Mod', 'els']

Model Benchmark

LM Eval Harness - Korean (polyglot branch)
NSMC (Acc) - 50000 full test

TBD

COPA (F1)
Model 0-shot 5-shot 10-shot 50-shot
https://huggingface.co/skt/ko-gpt-trinity-1.2B-v0.5 0.6696 0.6477 0.6419 0.6514
https://huggingface.co/kakaobrain/kogpt 0.7345 0.7287 0.7277 0.7479
https://huggingface.co/facebook/xglm-7.5B 0.6723 0.6731 0.6769 0.7119
https://huggingface.co/EleutherAI/polyglot-ko-1.3b 0.7196 0.7193 0.7204 0.7206
https://huggingface.co/EleutherAI/polyglot-ko-3.8b 0.7595 0.7608 0.7638 0.7788
https://huggingface.co/EleutherAI/polyglot-ko-5.8b 0.7745 0.7676 0.7775 0.7887
https://huggingface.co/EleutherAI/polyglot-ko-12.8b 0.7937 0.8108 0.8037 0.8369
Llama-2 Original 7B* 0.562033 0.575982 0.576216 0.595532
Llama-2-Ko-7b 20B (10k) 0.738780 0.762639 0.780761 0.797863
Llama-2-Ko-7b 40B (20k) 0.743630 0.792716 0.803746 0.825944
*Llama-2 Original 7B used https://huggingface.co/meta-llama/Llama-2-7b-hf (w/o tokenizer updated)
HellaSwag (F1)
Model 0-shot 5-shot 10-shot 50-shot
https://huggingface.co/skt/ko-gpt-trinity-1.2B-v0.5 0.5243 0.5272 0.5166 0.5352
https://huggingface.co/kakaobrain/kogpt 0.5590 0.5833 0.5828 0.5907
https://huggingface.co/facebook/xglm-7.5B 0.5665 0.5689 0.5565 0.5622
https://huggingface.co/EleutherAI/polyglot-ko-1.3b 0.5247 0.5260 0.5278 0.5427
https://huggingface.co/EleutherAI/polyglot-ko-3.8b 0.5707 0.5830 0.5670 0.5787
https://huggingface.co/EleutherAI/polyglot-ko-5.8b 0.5976 0.5998 0.5979 0.6208
https://huggingface.co/EleutherAI/polyglot-ko-12.8b 0.5954 0.6306 0.6098 0.6118
Llama-2 Original 7B* 0.415390 0.431382 0.421342 0.442003
Llama-2-Ko-7b 20B (10k) 0.451757 0.466751 0.472607 0.482776
Llama-2-Ko-7b 40B (20k) 0.456246 0.465665 0.469810 0.477374
*Llama-2 Original 7B used https://huggingface.co/meta-llama/Llama-2-7b-hf (w/o tokenizer updated)
BoolQ (F1)
Model 0-shot 5-shot 10-shot 50-shot
https://huggingface.co/skt/ko-gpt-trinity-1.2B-v0.5 0.3356 0.4014 0.3640 0.3560
https://huggingface.co/kakaobrain/kogpt 0.4514 0.5981 0.5499 0.5202
https://huggingface.co/facebook/xglm-7.5B 0.4464 0.3324 0.3324 0.3324
https://huggingface.co/EleutherAI/polyglot-ko-1.3b 0.3552 0.4751 0.4109 0.4038
https://huggingface.co/EleutherAI/polyglot-ko-3.8b 0.4320 0.5263 0.4930 0.4038
https://huggingface.co/EleutherAI/polyglot-ko-5.8b 0.4356 0.5698 0.5187 0.5236
https://huggingface.co/EleutherAI/polyglot-ko-12.8b 0.4818 0.6041 0.6289 0.6448
Llama-2 Original 7B* 0.352050 0.563238 0.474788 0.419222
Llama-2-Ko-7b 20B (10k) 0.360656 0.679743 0.680109 0.662152
Llama-2-Ko-7b 40B (20k) 0.578640 0.697747 0.708358 0.714423
*Llama-2 Original 7B used https://huggingface.co/meta-llama/Llama-2-7b-hf (w/o tokenizer updated)
SentiNeg (F1)
Model 0-shot 5-shot 10-shot 50-shot
https://huggingface.co/skt/ko-gpt-trinity-1.2B-v0.5 0.6065 0.6878 0.7280 0.8413
https://huggingface.co/kakaobrain/kogpt 0.3747 0.8942 0.9294 0.9698
https://huggingface.co/facebook/xglm-7.5B 0.3578 0.4471 0.3964 0.5271
https://huggingface.co/EleutherAI/polyglot-ko-1.3b 0.6790 0.6257 0.5514 0.7851
https://huggingface.co/EleutherAI/polyglot-ko-3.8b 0.4858 0.7950 0.7320 0.7851
https://huggingface.co/EleutherAI/polyglot-ko-5.8b 0.3394 0.8841 0.8808 0.9521
https://huggingface.co/EleutherAI/polyglot-ko-12.8b 0.9117 0.9015 0.9345 0.9723
Llama-2 Original 7B* 0.347502 0.529124 0.480641 0.788457
Llama-2-Ko-7b 20B (10k) 0.485546 0.829503 0.871141 0.851253
Llama-2-Ko-7b 40B (20k) 0.459447 0.761079 0.727611 0.936988
*Llama-2 Original 7B used https://huggingface.co/meta-llama/Llama-2-7b-hf (w/o tokenizer updated)
Note for oobabooga/text-generation-webui

Remove ValueError at load_tokenizer function(line 109 or near), in modules/models.py .

diff --git a/modules/models.py b/modules/models.py
index 232d5fa..de5b7a0 100644
--- a/modules/models.py
+++ b/modules/models.py
@@ -106,7 +106,7 @@ def load_tokenizer(model_name, model):
                 trust_remote_code=shared.args.trust_remote_code,
                 use_fast=False
             )
-        except ValueError:
+        except:
             tokenizer = AutoTokenizer.from_pretrained(
                 path_to_model,
                 trust_remote_code=shared.args.trust_remote_code,

Since Llama-2-Ko uses FastTokenizer provided by HF tokenizers NOT sentencepiece package, it is required to use use_fast=True option when initialize tokenizer.

Apple Sillicon does not support BF16 computing, use CPU instead. (BF16 is supported when using NVIDIA GPU)

Citation
@misc {l._junbum_2023,
    author       = { {L. Junbum} },
    title        = { llama-2-ko-7b (Revision 4a9993e) },
    year         = 2023,
    url          = { https://huggingface.co/beomi/llama-2-ko-7b },
    doi          = { 10.57967/hf/1098 },
    publisher    = { Hugging Face }
}
Acknowledgement

The training is supported by TPU Research Cloud program.

Open LLM Leaderboard Evaluation Results

Detailed results can be found here

Metric Value
Avg. 39.43
ARC (25-shot) 48.46
HellaSwag (10-shot) 75.28
MMLU (5-shot) 39.56
TruthfulQA (0-shot) 34.49
Winogrande (5-shot) 72.14
GSM8K (5-shot) 1.97
DROP (3-shot) 4.1

Runs of beomi llama-2-ko-7b on huggingface.co

5.8K
Total runs
0
24-hour runs
0
3-day runs
0
7-day runs
0
30-day runs

More Information About llama-2-ko-7b huggingface.co Model

llama-2-ko-7b huggingface.co

llama-2-ko-7b huggingface.co is an AI model on huggingface.co that provides llama-2-ko-7b's model effect (), which can be used instantly with this beomi llama-2-ko-7b model. huggingface.co supports a free trial of the llama-2-ko-7b model, and also provides paid use of the llama-2-ko-7b. Support call llama-2-ko-7b model through api, including Node.js, Python, http.

llama-2-ko-7b huggingface.co Url

https://huggingface.co/beomi/llama-2-ko-7b

beomi llama-2-ko-7b online free

llama-2-ko-7b huggingface.co is an online trial and call api platform, which integrates llama-2-ko-7b's modeling effects, including api services, and provides a free online trial of llama-2-ko-7b, you can try llama-2-ko-7b online for free by clicking the link below.

beomi llama-2-ko-7b online free url in huggingface.co:

https://huggingface.co/beomi/llama-2-ko-7b

llama-2-ko-7b install

llama-2-ko-7b is an open source model from GitHub that offers a free installation service, and any user can find llama-2-ko-7b on GitHub to install. At the same time, huggingface.co provides the effect of llama-2-ko-7b install, users can directly use llama-2-ko-7b installed effect in huggingface.co for debugging and trial. It also supports api for free installation.

llama-2-ko-7b install url in huggingface.co:

https://huggingface.co/beomi/llama-2-ko-7b

Url of llama-2-ko-7b

llama-2-ko-7b huggingface.co Url

Provider of llama-2-ko-7b huggingface.co

beomi
ORGANIZATIONS

Other API from beomi

huggingface.co

Total runs: 9.1K
Run Growth: -330
Growth Rate: -3.87%
Updated: 2023年3月30日
huggingface.co

Total runs: 6.0K
Run Growth: 3.9K
Growth Rate: 67.70%
Updated: 2024年3月26日
huggingface.co

Total runs: 5.7K
Run Growth: 2.9K
Growth Rate: 51.06%
Updated: 2024年5月23日
huggingface.co

Total runs: 5.0K
Run Growth: 360
Growth Rate: 7.31%
Updated: 2024年7月8日
huggingface.co

Total runs: 4.0K
Run Growth: 2.6K
Growth Rate: 63.38%
Updated: 2024年3月26日
huggingface.co

Total runs: 3.2K
Run Growth: 1.1K
Growth Rate: 35.79%
Updated: 2023年7月20日
huggingface.co

Total runs: 2.2K
Run Growth: -1.1K
Growth Rate: -47.70%
Updated: 2024年5月27日
huggingface.co

Total runs: 2.0K
Run Growth: -993
Growth Rate: -48.65%
Updated: 2023年6月28日
huggingface.co

Total runs: 730
Run Growth: -2.5K
Growth Rate: -376.26%
Updated: 2023年3月30日
huggingface.co

Total runs: 177
Run Growth: 163
Growth Rate: 93.14%
Updated: 2021年11月22日
huggingface.co

Total runs: 164
Run Growth: -664
Growth Rate: -439.74%
Updated: 2024年8月21日
huggingface.co

Total runs: 160
Run Growth: -26
Growth Rate: -20.00%
Updated: 2023年9月15日
huggingface.co

Total runs: 120
Run Growth: -733
Growth Rate: -595.93%
Updated: 2023年11月13日
huggingface.co

Total runs: 111
Run Growth: 93
Growth Rate: 83.78%
Updated: 2023年5月7日
huggingface.co

Total runs: 60
Run Growth: -778
Growth Rate: -1275.41%
Updated: 2024年2月20日
huggingface.co

Total runs: 55
Run Growth: -335
Growth Rate: -587.72%
Updated: 2023年7月11日
huggingface.co

Total runs: 42
Run Growth: 25
Growth Rate: 64.10%
Updated: 2024年7月8日
huggingface.co

Total runs: 27
Run Growth: 17
Growth Rate: 58.62%
Updated: 2023年6月29日
huggingface.co

Total runs: 25
Run Growth: -35
Growth Rate: -145.83%
Updated: 2024年7月8日
huggingface.co

Total runs: 13
Run Growth: 2
Growth Rate: 15.38%
Updated: 2023年3月8日
huggingface.co

Total runs: 9
Run Growth: 5
Growth Rate: 55.56%
Updated: 2022年2月28日
huggingface.co

Total runs: 7
Run Growth: -1
Growth Rate: -14.29%
Updated: 2021年6月10日
huggingface.co

Total runs: 6
Run Growth: -6
Growth Rate: -100.00%
Updated: 2021年5月19日
huggingface.co

Total runs: 6
Run Growth: -9
Growth Rate: -150.00%
Updated: 2021年5月21日
huggingface.co

Total runs: 6
Run Growth: 3
Growth Rate: 50.00%
Updated: 2023年11月3日
huggingface.co

Total runs: 0
Run Growth: 0
Growth Rate: 0.00%
Updated: 2022年2月10日
huggingface.co

Total runs: 0
Run Growth: 0
Growth Rate: 0.00%
Updated: 2021年6月10日