GGUF is a new format introduced by the llama.cpp team on August 21st 2023. It is a replacement for GGML, which is no longer supported by llama.cpp.
Here is an incomplete list of clients and libraries that are known to support GGUF:
llama.cpp
. The source project for GGUF. Offers a CLI and a server option.
llama-cpp-python
, a Python library with GPU accel, LangChain support, and OpenAI-compatible API server.
LM Studio
, an easy-to-use and powerful local GUI for Windows and macOS (Silicon), with GPU acceleration. Linux available, in beta as of 27/11/2023.
text-generation-webui
, the most widely used web UI, with many features and powerful extensions. Supports GPU acceleration.
KoboldCpp
, a fully featured web UI, with GPU accel across all platforms and GPU architectures. Especially good for story telling.
GPT4All
, a free and open source local running GUI, supporting Windows, Linux and macOS with full GPU accel.
LoLLMS Web UI
, a great web UI with many interesting and unique features, including a full model library for easy model selection.
Faraday.dev
, an attractive and easy to use character-based chat GUI for Windows and macOS (both Silicon and Intel), with GPU acceleration.
candle
, a Rust ML framework with a focus on performance, including GPU support, and ease of use.
ctransformers
, a Python library with GPU accel, LangChain support, and OpenAI-compatible AI server. Note, as of time of writing (November 27th 2023), ctransformers has not been updated in a long time and does not support many recent models.
You can then start chatting with the model,
e.g.
prompt it with something like:
"Albert likes to surf every week. Each surfing session lasts for 4 hours and costs $20 per hour. How much would Albert spend in 5 weeks?"
Evaluation
We evaluate Mathstral 7B and open-weight models of the similar size on industry-standard benchmarks.
Benchmarks
MATH
GSM8K (8-shot)
Odyssey Math maj@16
GRE Math maj@16
AMC 2023 maj@16
AIME 2024 maj@16
Mathstral 7B
56.6
77.1
37.2
56.9
42.4
2/30
DeepSeek Math 7B
44.4
80.6
27.6
44.6
28.0
0/30
Llama3 8B
28.4
75.4
24.0
26.2
34.4
0/30
GLM4 9B
50.2
48.8
18.9
46.2
36.0
1/30
QWen2 7B
56.8
32.7
24.8
58.5
35.2
2/30
Gemma2 9B
48.3
69.5
18.6
52.3
31.2
1/30
The Mistral AI Team
Albert Jiang, Alexandre Sablayrolles, Alexis Tacnet, Alok Kothari, Antoine Roux, Arthur Mensch, Audrey Herblin-Stoop, Augustin Garreau, Austin Birky, Bam4d, Baptiste Bout, Baudouin de Monicault, Blanche Savary, Carole Rambaud, Caroline Feldman, Devendra Singh Chaplot, Diego de las Casas, Eleonore Arcelin, Emma Bou Hanna, Etienne Metzger, Gaspard Blanchet, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Harizo Rajaona, Henri Roussez, Hichem Sattouf, Ian Mack, Jean-Malo Delignon, Jessica Chudnovsky, Justus Murke, Kartik Khandelwal, Lawrence Stewart, Louis Martin, Louis Ternon, Lucile Saulnier, Lélio Renard Lavaud, Margaret Jennings, Marie Pellat, Marie Torelli, Marie-Anne Lachaux, Marjorie Janiewicz, Mickaël Seznec, Nicolas Schuhl, Niklas Muhs, Olivier de Garrigues, Patrick von Platen, Paul Jacob, Pauline Buche, Pavan Kumar Reddy, Perry Savas, Pierre Stock, Romain Sauvestre, Sagar Vaze, Sandeep Subramanian, Saurabh Garg, Sophia Yang, Szymon Antoniak, Teven Le Scao, Thibault Schueller, Thibaut Lavril, Thomas Wang, Théophile Gervet, Timothée Lacroix, Valera Nemychnikova, Wendy Shang, William El Sayed, William Marshall
Runs of MaziyarPanahi mathstral-7B-v0.1-GGUF on huggingface.co
848.1K
Total runs
-98.5K
24-hour runs
-345.5K
3-day runs
-483.6K
7-day runs
-1.4M
30-day runs
More Information About mathstral-7B-v0.1-GGUF huggingface.co Model
mathstral-7B-v0.1-GGUF huggingface.co
mathstral-7B-v0.1-GGUF huggingface.co is an AI model on huggingface.co that provides mathstral-7B-v0.1-GGUF's model effect (), which can be used instantly with this MaziyarPanahi mathstral-7B-v0.1-GGUF model. huggingface.co supports a free trial of the mathstral-7B-v0.1-GGUF model, and also provides paid use of the mathstral-7B-v0.1-GGUF. Support call mathstral-7B-v0.1-GGUF model through api, including Node.js, Python, http.
mathstral-7B-v0.1-GGUF huggingface.co is an online trial and call api platform, which integrates mathstral-7B-v0.1-GGUF's modeling effects, including api services, and provides a free online trial of mathstral-7B-v0.1-GGUF, you can try mathstral-7B-v0.1-GGUF online for free by clicking the link below.
MaziyarPanahi mathstral-7B-v0.1-GGUF online free url in huggingface.co:
mathstral-7B-v0.1-GGUF is an open source model from GitHub that offers a free installation service, and any user can find mathstral-7B-v0.1-GGUF on GitHub to install. At the same time, huggingface.co provides the effect of mathstral-7B-v0.1-GGUF install, users can directly use mathstral-7B-v0.1-GGUF installed effect in huggingface.co for debugging and trial. It also supports api for free installation.
mathstral-7B-v0.1-GGUF install url in huggingface.co: