stabilityai / stablelm-base-alpha-7b-v2

huggingface.co
Total runs: 2.0K
24-hour runs: 0
7-day runs: 539
30-day runs: 475
Model's Last Updated: September 11 2023
text-generation

Introduction of stablelm-base-alpha-7b-v2

Model Details of stablelm-base-alpha-7b-v2

StableLM-Base-Alpha-7B-v2

Model Description

StableLM-Base-Alpha-7B-v2 is a 7 billion parameter decoder-only language model pre-trained on diverse English datasets. This model is the successor to the first StableLM-Base-Alpha-7B model, addressing previous shortcomings through the use of improved data sources and mixture ratios.

Usage

Get started generating text with StableLM-Base-Alpha-7B-v2 by using the following code snippet:

from transformers import AutoModelForCausalLM, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("stabilityai/stablelm-base-alpha-7b-v2")
model = AutoModelForCausalLM.from_pretrained(
  "stabilityai/stablelm-base-alpha-7b-v2",
  trust_remote_code=True,
  torch_dtype="auto",
)
model.cuda()
inputs = tokenizer("The weather is always wonderful", return_tensors="pt").to("cuda")
tokens = model.generate(
  **inputs,
  max_new_tokens=64,
  temperature=0.75,
  top_p=0.95,
  do_sample=True,
)
print(tokenizer.decode(tokens[0], skip_special_tokens=True))
Model Details
  • Developed by : Stability AI
  • Model type : StableLM-Base-Alpha-v2 models are auto-regressive language models based on the transformer decoder architecture.
  • Language(s) : English
  • Library : GPT-NeoX
  • License : Model checkpoints are licensed under the Creative Commons license ( CC BY-SA-4.0 ). Under this license, you must give credit to Stability AI, provide a link to the license, and indicate if changes were made . You may do so in any reasonable manner, but not in any way that suggests the Stability AI endorses you or your use.
  • Contact : For questions and comments about the model, please email lm@stability.ai
Model Architecture
Parameters Hidden Size Layers Heads Sequence Length
6,890,209,280 4096 32 32 4096

The model is a decoder-only transformer similar to the StableLM-Base-Alpha (v1) with the following configurations:

Training

StableLM-Base-Alpha-7B-v2 is pre-trained using a multi-stage context length extension schedule following similar work ( Nijkamp et al. 2023 ); first pre-training at a context length of 2048 for 1 trillion tokens, then fine-tuning at a context length of 4096 for another 100B tokens.

Training Dataset

The first pre-training stage relies on 1 trillion tokens sourced from a mix of the public Falcon RefinedWeb extract ( Penedo et al., 2023 ), RedPajama-Data ( Together Computer 2023 , The Pile ( Gao et al., 2020 ), and internal datasets with web text sampled at a rate of 71%.

In the second stage, we include the StarCoder ( Li et al., 2023 ) dataset and down sample web text to 55% while increasing sampling proportions of naturally long text examples in the aforementioned sources.

Training Procedure

The model is pre-trained on the dataset mixes mentioned above in mixed-precision (FP16), optimized with AdamW, and trained using the NeoX tokenizer with a vocabulary size of 50,257. We outline the complete hyperparameters choices in the project's GitHub repository - config .

Training Infrastructure
  • Hardware : StableLM-Base-Alpha-7B-v2 was trained on the Stability AI cluster - occupying 384 NVIDIA A100 40GB GPUs across AWS P4d instances. Training took approximately 16.33 days to complete across both stages.

  • Software : We use a fork of gpt-neox ( EleutherAI, 2021 ) and train under 2D parallelism (Data and Tensor Parallel) with ZeRO-1 ( Rajbhandari et al., 2019 ) and rely on flash-attention as well as rotary embedding kernels from FlashAttention-2 ( Dao et al., 2023 )

Use and Limitations
Intended Use

These models are intended to be used by all individuals as foundational models for application-specific fine-tuning without strict limitations on commercial use.

Limitations and bias

The pre-training dataset may have contained offensive or inappropriate content even after applying data cleansing filters which can be reflected in the model-generated text. We recommend that users exercise caution when using these models in production systems. Do not use the models for any applications that may cause harm or distress to individuals or groups.

How to cite
@misc{StableLMAlphaV2Models, 
      url={[https://huggingface.co/stabilityai/stablelm-base-alpha-7b-v2](https://huggingface.co/stabilityai/stablelm-base-alpha-7b-v2)},
      title={StableLM Alpha v2 Models},
      author={Tow, Jonathan}
}

Runs of stabilityai stablelm-base-alpha-7b-v2 on huggingface.co

2.0K
Total runs
0
24-hour runs
230
3-day runs
539
7-day runs
475
30-day runs

More Information About stablelm-base-alpha-7b-v2 huggingface.co Model

More stablelm-base-alpha-7b-v2 license Visit here:

https://choosealicense.com/licenses/cc-by-sa-4.0

stablelm-base-alpha-7b-v2 huggingface.co

stablelm-base-alpha-7b-v2 huggingface.co is an AI model on huggingface.co that provides stablelm-base-alpha-7b-v2's model effect (), which can be used instantly with this stabilityai stablelm-base-alpha-7b-v2 model. huggingface.co supports a free trial of the stablelm-base-alpha-7b-v2 model, and also provides paid use of the stablelm-base-alpha-7b-v2. Support call stablelm-base-alpha-7b-v2 model through api, including Node.js, Python, http.

stablelm-base-alpha-7b-v2 huggingface.co Url

https://huggingface.co/stabilityai/stablelm-base-alpha-7b-v2

stabilityai stablelm-base-alpha-7b-v2 online free

stablelm-base-alpha-7b-v2 huggingface.co is an online trial and call api platform, which integrates stablelm-base-alpha-7b-v2's modeling effects, including api services, and provides a free online trial of stablelm-base-alpha-7b-v2, you can try stablelm-base-alpha-7b-v2 online for free by clicking the link below.

stabilityai stablelm-base-alpha-7b-v2 online free url in huggingface.co:

https://huggingface.co/stabilityai/stablelm-base-alpha-7b-v2

stablelm-base-alpha-7b-v2 install

stablelm-base-alpha-7b-v2 is an open source model from GitHub that offers a free installation service, and any user can find stablelm-base-alpha-7b-v2 on GitHub to install. At the same time, huggingface.co provides the effect of stablelm-base-alpha-7b-v2 install, users can directly use stablelm-base-alpha-7b-v2 installed effect in huggingface.co for debugging and trial. It also supports api for free installation.

stablelm-base-alpha-7b-v2 install url in huggingface.co:

https://huggingface.co/stabilityai/stablelm-base-alpha-7b-v2

Url of stablelm-base-alpha-7b-v2

stablelm-base-alpha-7b-v2 huggingface.co Url

Provider of stablelm-base-alpha-7b-v2 huggingface.co

stabilityai
ORGANIZATIONS

Other API from stabilityai

huggingface.co

Total runs: 143.2K
Run Growth: 8.3K
Growth Rate: 5.81%
Updated: August 04 2023
huggingface.co

Total runs: 137.2K
Run Growth: 16.4K
Growth Rate: 11.93%
Updated: July 10 2024
huggingface.co

Total runs: 34.3K
Run Growth: 3.4K
Growth Rate: 10.04%
Updated: August 09 2024
huggingface.co

Total runs: 378
Run Growth: -97.9K
Growth Rate: -25898.41%
Updated: August 03 2024
huggingface.co

Total runs: 0
Run Growth: 0
Growth Rate: 0.00%
Updated: July 10 2024
huggingface.co

Total runs: 0
Run Growth: 0
Growth Rate: 0.00%
Updated: April 13 2024