neuralmagic / Meta-Llama-3.1-8B-Instruct-FP8-dynamic

huggingface.co
Total runs: 4.8K
24-hour runs: 176
7-day runs: 748
30-day runs: 2.0K
Model's Last Updated: 10月 19 2024
text-generation

Introduction of Meta-Llama-3.1-8B-Instruct-FP8-dynamic

Model Details of Meta-Llama-3.1-8B-Instruct-FP8-dynamic

Meta-Llama-3.1-8B-Instruct-FP8-dynamic

Model Overview
  • Model Architecture: Meta-Llama-3.1
    • Input: Text
    • Output: Text
  • Model Optimizations:
    • Weight quantization: FP8
    • Activation quantization: FP8
  • Intended Use Cases: Intended for commercial and research use in multiple languages. Similarly to Meta-Llama-3.1-8B-Instruct , this models is intended for assistant-like chat.
  • Out-of-scope: Use in any manner that violates applicable laws or regulations (including trade compliance laws). Use in languages other than English.
  • Release Date: 7/23/2024
  • Version: 1.0
  • License(s): llama3.1
  • Model Developers: Neural Magic

Quantized version of Meta-Llama-3.1-8B-Instruct . It achieves an average score of 73.81 on the OpenLLM benchmark (version 1), whereas the unquantized model achieves 74.17.

Model Optimizations

This model was obtained by quantizing the weights and activations of Meta-Llama-3.1-8B-Instruct to FP8 data type, ready for inference with vLLM built from source. This optimization reduces the number of bits per parameter from 16 to 8, reducing the disk size and GPU memory requirements by approximately 50%.

Only the weights and activations of the linear operators within transformers blocks are quantized. Symmetric per-channel quantization is applied, in which a linear scaling per output dimension maps the FP8 representations of the quantized weights and activations. Activations are also quantized on a per-token dynamic basis. LLM Compressor is used for quantization with 512 sequences of UltraChat.

Deployment
Use with vLLM

This model can be deployed efficiently using the vLLM backend, as shown in the example below.

from vllm import LLM, SamplingParams
from transformers import AutoTokenizer

model_id = "neuralmagic/Meta-Llama-3.1-8B-Instruct-FP8-dynamic"

sampling_params = SamplingParams(temperature=0.6, top_p=0.9, max_tokens=256)

tokenizer = AutoTokenizer.from_pretrained(model_id)

messages = [
    {"role": "system", "content": "You are a pirate chatbot who always responds in pirate speak!"},
    {"role": "user", "content": "Who are you?"},
]

prompts = tokenizer.apply_chat_template(messages, tokenize=False)

llm = LLM(model=model_id)

outputs = llm.generate(prompts, sampling_params)

generated_text = outputs[0].outputs[0].text
print(generated_text)

vLLM aslo supports OpenAI-compatible serving. See the documentation for more details.

Creation

This model was created by applying LLM Compressor with calibration samples from UltraChat , as presented in the code snipet below.

import torch

from transformers import AutoTokenizer

from llmcompressor.transformers import SparseAutoModelForCausalLM, oneshot
from llmcompressor.transformers.compression.helpers import (  # noqa
    calculate_offload_device_map,
    custom_offload_device_map,
)

recipe = """
quant_stage:
    quant_modifiers:
        QuantizationModifier:
            ignore: ["lm_head"]
            config_groups:
                group_0:
                    weights:
                        num_bits: 8
                        type: float
                        strategy: channel
                        dynamic: false
                        symmetric: true
                    input_activations:
                        num_bits: 8
                        type: float
                        strategy: token
                        dynamic: true
                        symmetric: true
                    targets: ["Linear"]
"""

model_stub = "meta-llama/Meta-Llama-3.1-8B-Instruct"
model_name = model_stub.split("/")[-1]

device_map = calculate_offload_device_map(
    model_stub, reserve_for_hessians=False, num_gpus=1, torch_dtype=torch.float16
)

model = SparseAutoModelForCausalLM.from_pretrained(
    model_stub, torch_dtype=torch.float16, device_map=device_map
)

output_dir = f"./{model_name}-FP8-dynamic"

oneshot(
    model=model,
    recipe=recipe,
    output_dir=output_dir,
    save_compressed=True,
    tokenizer=AutoTokenizer.from_pretrained(model_stub),
)
Evaluation

The model was evaluated on MMLU, ARC-Challenge, GSM-8K, Hellaswag, Winogrande and TruthfulQA. Evaluation was conducted using the Neural Magic fork of lm-evaluation-harness (branch llama_3.1_instruct) and the vLLM engine. This version of the lm-evaluation-harness includes versions of ARC-Challenge and GSM-8K that match the prompting style of Meta-Llama-3.1-Instruct-evals .

Accuracy
Open LLM Leaderboard evaluation scores
Benchmark Meta-Llama-3.1-8B-Instruct Meta-Llama-3.1-8B-Instruct-FP8-dynamic(this model) Recovery
MMLU (5-shot) 67.94 68.09 100.2%
ARC Challenge (0-shot) 83.11 82.34 99.07%
GSM-8K (CoT, 8-shot, strict-match) 82.03 82.34 100.3%
Hellaswag (10-shot) 80.01 79.68 99.59%
Winogrande (5-shot) 77.90 77.03 98.88%
TruthfulQA (0-shot, mc2) 54.04 53.37 98.76%
Average 74.17 73.81 99.48%
Reproduction

The results were obtained using the following commands:

MMLU
lm_eval \
  --model vllm \
  --model_args pretrained="neuralmagic/Meta-Llama-3.1-8B-Instruct-FP8-dynamic",dtype=auto,add_bos_token=True,max_model_len=4096,tensor_parallel_size=1 \
  --tasks mmlu \
  --num_fewshot 5 \
  --batch_size auto
ARC-Challenge
lm_eval \
  --model vllm \
  --model_args pretrained="neuralmagic/Meta-Llama-3.1-8B-Instruct-FP8-dynamic",dtype=auto,add_bos_token=True,max_model_len=4096,tensor_parallel_size=1 \
  --tasks arc_challenge_llama_3.1_instruct \
  --apply_chat_template \
  --num_fewshot 0 \
  --batch_size auto
GSM-8K
lm_eval \
  --model vllm \
  --model_args pretrained="neuralmagic/Meta-Llama-3.1-8B-Instruct-FP8-dynamic",dtype=auto,add_bos_token=True,max_model_len=4096,tensor_parallel_size=1 \
  --tasks gsm8k_cot_llama_3.1_instruct \
  --apply_chat_template \
  --fewshot_as_multiturn \
  --num_fewshot 8 \
  --batch_size auto
Hellaswag
lm_eval \
  --model vllm \
  --model_args pretrained="neuralmagic/Meta-Llama-3.1-8B-Instruct-FP8-dynamic",dtype=auto,add_bos_token=True,max_model_len=4096,tensor_parallel_size=1 \
  --tasks hellaswag \
  --num_fewshot 10 \
  --batch_size auto
Winogrande
lm_eval \
  --model vllm \
  --model_args pretrained="neuralmagic/Meta-Llama-3.1-8B-Instruct-FP8-dynamic",dtype=auto,add_bos_token=True,max_model_len=4096,tensor_parallel_size=1 \
  --tasks winogrande \
  --num_fewshot 5 \
  --batch_size auto
TruthfulQA
lm_eval \
  --model vllm \
  --model_args pretrained="neuralmagic/Meta-Llama-3.1-8B-Instruct-FP8-dynamic",dtype=auto,add_bos_token=True,max_model_len=4096,tensor_parallel_size=1 \
  --tasks truthfulqa \
  --num_fewshot 0 \
  --batch_size auto

Runs of neuralmagic Meta-Llama-3.1-8B-Instruct-FP8-dynamic on huggingface.co

4.8K
Total runs
176
24-hour runs
-86
3-day runs
748
7-day runs
2.0K
30-day runs

More Information About Meta-Llama-3.1-8B-Instruct-FP8-dynamic huggingface.co Model

More Meta-Llama-3.1-8B-Instruct-FP8-dynamic license Visit here:

https://choosealicense.com/licenses/llama3.1

Meta-Llama-3.1-8B-Instruct-FP8-dynamic huggingface.co

Meta-Llama-3.1-8B-Instruct-FP8-dynamic huggingface.co is an AI model on huggingface.co that provides Meta-Llama-3.1-8B-Instruct-FP8-dynamic's model effect (), which can be used instantly with this neuralmagic Meta-Llama-3.1-8B-Instruct-FP8-dynamic model. huggingface.co supports a free trial of the Meta-Llama-3.1-8B-Instruct-FP8-dynamic model, and also provides paid use of the Meta-Llama-3.1-8B-Instruct-FP8-dynamic. Support call Meta-Llama-3.1-8B-Instruct-FP8-dynamic model through api, including Node.js, Python, http.

Meta-Llama-3.1-8B-Instruct-FP8-dynamic huggingface.co Url

https://huggingface.co/neuralmagic/Meta-Llama-3.1-8B-Instruct-FP8-dynamic

neuralmagic Meta-Llama-3.1-8B-Instruct-FP8-dynamic online free

Meta-Llama-3.1-8B-Instruct-FP8-dynamic huggingface.co is an online trial and call api platform, which integrates Meta-Llama-3.1-8B-Instruct-FP8-dynamic's modeling effects, including api services, and provides a free online trial of Meta-Llama-3.1-8B-Instruct-FP8-dynamic, you can try Meta-Llama-3.1-8B-Instruct-FP8-dynamic online for free by clicking the link below.

neuralmagic Meta-Llama-3.1-8B-Instruct-FP8-dynamic online free url in huggingface.co:

https://huggingface.co/neuralmagic/Meta-Llama-3.1-8B-Instruct-FP8-dynamic

Meta-Llama-3.1-8B-Instruct-FP8-dynamic install

Meta-Llama-3.1-8B-Instruct-FP8-dynamic is an open source model from GitHub that offers a free installation service, and any user can find Meta-Llama-3.1-8B-Instruct-FP8-dynamic on GitHub to install. At the same time, huggingface.co provides the effect of Meta-Llama-3.1-8B-Instruct-FP8-dynamic install, users can directly use Meta-Llama-3.1-8B-Instruct-FP8-dynamic installed effect in huggingface.co for debugging and trial. It also supports api for free installation.

Meta-Llama-3.1-8B-Instruct-FP8-dynamic install url in huggingface.co:

https://huggingface.co/neuralmagic/Meta-Llama-3.1-8B-Instruct-FP8-dynamic

Url of Meta-Llama-3.1-8B-Instruct-FP8-dynamic

Meta-Llama-3.1-8B-Instruct-FP8-dynamic huggingface.co Url

Provider of Meta-Llama-3.1-8B-Instruct-FP8-dynamic huggingface.co

neuralmagic
ORGANIZATIONS

Other API from neuralmagic