ThursdAI - Special Episode, interview with Nous Research and Enrico Shippole, fine-tuning LLaMa 2...
Hey there, welcome to this special edition of ThursdAI. This episode is featuring an interview with Nous Research, a group of folks who fine-tune open source large language models to make them better. If you are interested to hear how finetuning an open source model works, dataset preparation, context scaling and more, tune in!
You will hear from Karan (https://twitter.com/karan4d) , Teknium (https://twitter.com/Teknium1) , LBJ (https://twitter.com/Dogesator) from Nous Research (https://twitter.com/NousResearch) and Enrico (https://twitter.com/EnricoShippole) who worked along side them.
To clarify, Enrico is going in depth into the method called Rope Scaling, which is a clever hack, that extends the context length of LLaMa models significantly and his project LLongMa which is an extended version of LLaMa with 8000 token context window.
The first voice you will hear is Alex Volkov the host of ThursdAI who doesn’t usually have a lisp, but for some reason, during the recording, twitter spaces decided to mute all the S sounds.
Links and acknowledgments:
* Nous Research - https://nousresearch.com/ (@nousresearch (https://twitter.com/nousresearch) )
* Redmond Puffin 13b (https://huggingface.co/NousResearch/Redmond-Puffin-13B) - First LLaMa Finetune
* LLongMa (https://huggingface.co/conceptofmind/LLongMA-2-7b) - LLaMa finetune with 8K context (by Encrico, emozilla (https://twitter.com/theemozilla/status/1676597615750701057) and KaioKenDev (https://twitter.com/kaiokendev1) )
* Nous-Hermes-Llama2-13b-GPTQ (https://huggingface.co/NousResearch/Nous-Hermes-Llama2-13b-GPTQ) - Hermes Finetune was released after the recording 🎊
Psst, if you like this, why don’t you subscribe? Or if you are subscribed, consider a paid subscription to support #ThursdAI
Show transcription with timestamps:
Alex Volkov - targum.video (@altryne (https://twitter.com/altryne) )[00:00:55] Yeah. That's awesome. So I guess with this, maybe, Karan, if you if you are able to, can you you talk about Nous research and how kind of how it started and what the what are you guys doing, and then we'll dive into the kind of, you know, Hermes and and Puffin and the methods and and all of it.
karan (@karan4d (https://twitter.com/karan4d) )[00:01:16] Absolutely. Nous research. I mean, I I myself and many other of us are just, like, enthusiasts that we're fine tuning models like, you know, GPTJ or GPT 2. And, you know, we all are on Twitter. We're all on Discord, and kind of just found each other and had this same mentality of we wanna we wanna make these models. We wanna kinda take the power back from people like OpenAI and anthropic. We want stuff to be able to run easy for everyone. And a lot of like minds started to show up.
karan (@karan4d (https://twitter.com/karan4d) )[00:01:50] I think that Technium's addition initially to Nous research, Jim, kinda showing up. And himself, I and human working on compiling the Hermes dataset was really what came to attract people when Hermes came out. I think we just have a really strong and robust, like, data curation thesis in terms of that. And I think that have just some of the most talented people who have come to join us and just volunteer and work with us on stuff. And I absolutely must say, I can see in the in the listeners is our compute provider, Redmond AI.
karan (@karan4d (https://twitter.com/karan4d) )[00:02:30] And, you know, none of this none of these models would be possible without Redmond's generous sponsorship for us to be able to deliver these things lightning fast, you know, without making us through a bunch of hoops just a a total total pleasure to work with. So I would I have to shell and say, you know, I highly recommend everyone check out Redmond as because they really make our project possible.
Alex Volkov - targum.video (@altryne (https://twitter.com/altryne) )[00:02:52] Absolutely. So shout out to Redmond AI and folks give them a follow. They're the the only square avatar in the audience. Go take them out. And, Karan, thanks for that. I wanna just do a mic check for teknium. Teknium. Can you speak now? Can you? Can I hear you?
Teknium (e/λ) (@Teknium1 (https://twitter.com/Teknium1) )[00:03:08] Yeah. My phone died right when you were introducing me earlier.
Alex Volkov - targum.video (@altryne (https://twitter.com/altryne) )[00:03:10] Yep. What's up, Eric? -- sometimes on Twitter basis. Welcome, Technium. So briefly, going back to question. I don't know if you heard it. What besides the commercial and kind of the the contact window, what kind of caught your eye in the llama, at least the base until you guys started, or have you also, like, the other guys not had a second to play with the base model and dove into fine tuning directly?
Teknium (e/λ) (@Teknium1 (https://twitter.com/Teknium1) )[00:03:35] Yeah. The only thing that really caught my eye was the chat model and how horribly RLHF it was.
Alex Volkov - targum.video (@altryne (http...
Escucha en redes sociales
I translated a Oleina ad to English, because Vinny from vinesauce saw one of the ads.
Translated using targum.video with some minor fixes by me! (like the grandpa saying “I'm not going to let you do this.”, Instead of “Oh, please!” For example.)
خدمة تستخدم الذكاء الاصطناعي لترجمة مقاطع الفيديو ووضع الترجمة بدقة متميزة أسفل الفيديو بدون أي جهد
إذا عندك مقطع وودك تضع له ترجمة، سواء عربي أو انجليزي، فهذه خدمة تستخدم الذكاء الاصطناعي لترجمة مقاطع الفيديو ووضع الترجمة بدقة متميزة أسفل الفيديو، بدون أي جهد إضافي منك! جربتها على عدة مقاطع عندي وكانت دقتها ممتازة مقارنة بالأدوات الثانية! رابط الخدمة: targum.video
ThursdAI - Special Episode, interview with Nous Research and Enrico Shippole, fine-tuning LLaMa 2...
Hey there, welcome to this special edition of ThursdAI. This episode is featuring an interview with Nous Research, a group of folks who fine-tune open source large language models to make them better. If you are interested to hear how finetuning an open source model works, dataset preparation, context scaling and more, tune in! You will hear from Karan (https://twitter.com/karan4d) , Teknium (https://twitter.com/Teknium1) , LBJ (https://twitter.com/Dogesator) from Nous Research (https://twitter.com/NousResearch) and Enrico (https://twitter.com/EnricoShippole) who worked along side them. To clarify, Enrico is going in depth into the method called Rope Scaling, which is a clever hack, that extends the context length of LLaMa models significantly and his project LLongMa which is an extended version of LLaMa with 8000 token context window. The first voice you will hear is Alex Volkov the host of ThursdAI who doesn’t usually have a lisp, but for some reason, during the recording, twitter spaces decided to mute all the S sounds. Links and acknowledgments: * Nous Research - https://nousresearch.com/ (@nousresearch (https://twitter.com/nousresearch) ) * Redmond Puffin 13b (https://huggingface.co/NousResearch/Redmond-Puffin-13B) - First LLaMa Finetune * LLongMa (https://huggingface.co/conceptofmind/LLongMA-2-7b) - LLaMa finetune with 8K context (by Encrico, emozilla (https://twitter.com/theemozilla/status/1676597615750701057) and KaioKenDev (https://twitter.com/kaiokendev1) ) * Nous-Hermes-Llama2-13b-GPTQ (https://huggingface.co/NousResearch/Nous-Hermes-Llama2-13b-GPTQ) - Hermes Finetune was released after the recording 🎊 Psst, if you like this, why don’t you subscribe? Or if you are subscribed, consider a paid subscription to support #ThursdAI Show transcription with timestamps: Alex Volkov - targum.video (@altryne (https://twitter.com/altryne) )[00:00:55] Yeah. That's awesome. So I guess with this, maybe, Karan, if you if you are able to, can you you talk about Nous research and how kind of how it started and what the what are you guys doing, and then we'll dive into the kind of, you know, Hermes and and Puffin and the methods and and all of it. karan (@karan4d (https://twitter.com/karan4d) )[00:01:16] Absolutely. Nous research. I mean, I I myself and many other of us are just, like, enthusiasts that we're fine tuning models like, you know, GPTJ or GPT 2. And, you know, we all are on Twitter. We're all on Discord, and kind of just found each other and had this same mentality of we wanna we wanna make these models. We wanna kinda take the power back from people like OpenAI and anthropic. We want stuff to be able to run easy for everyone. And a lot of like minds started to show up. karan (@karan4d (https://twitter.com/karan4d) )[00:01:50] I think that Technium's addition initially to Nous research, Jim, kinda showing up. And himself, I and human working on compiling the Hermes dataset was really what came to attract people when Hermes came out. I think we just have a really strong and robust, like, data curation thesis in terms of that. And I think that have just some of the most talented people who have come to join us and just volunteer and work with us on stuff. And I absolutely must say, I can see in the in the listeners is our compute provider, Redmond AI. karan (@karan4d (https://twitter.com/karan4d) )[00:02:30] And, you know, none of this none of these models would be possible without Redmond's generous sponsorship for us to be able to deliver these things lightning fast, you know, without making us through a bunch of hoops just a a total total pleasure to work with. So I would I have to shell and say, you know, I highly recommend everyone check out Redmond as because they really make our project possible. Alex Volkov - targum.video (@altryne (https://twitter.com/altryne) )[00:02:52] Absolutely. So shout out to Redmond AI and folks give them a follow. They're the the only square avatar in the audience. Go take them out. And, Karan, thanks for that. I wanna just do a mic check for teknium. Teknium. Can you speak now? Can you? Can I hear you? Teknium (e/λ) (@Teknium1 (https://twitter.com/Teknium1) )[00:03:08] Yeah. My phone died right when you were introducing me earlier. Alex Volkov - targum.video (@altryne (https://twitter.com/altryne) )[00:03:10] Yep. What's up, Eric? -- sometimes on Twitter basis. Welcome, Technium. So briefly, going back to question. I don't know if you heard it. What besides the commercial and kind of the the contact window, what kind of caught your eye in the llama, at least the base until you guys started, or have you also, like, the other guys not had a second to play with the base model and dove into fine tuning directly? Teknium (e/λ) (@Teknium1 (https://twitter.com/Teknium1) )[00:03:35] Yeah. The only thing that really caught my eye was the chat model and how horribly RLHF it was. Alex Volkov - targum.video (@altryne (http...