Post
3342
Wow i can post on HF now!
Love HF so much 🤗❤️
Love HF so much 🤗❤️
- 6 replies
Koki Tanaka (田中 康紀) PRO
gojiteji
AI & ML interests
NLP, efficient AI, Speech Language Model, LLM, AI hardware
CEO @mocomoco_inc
Recent Activity
liked a model about 10 hours ago
Qwen/Qwen3.8-27B updated a Space 13 days ago
gojiteji/Fable5GeneratedThisSheet published a Space 13 days ago
gojiteji/Fable5GeneratedThisSheetOrganizations
reacted to sosoai's post with 🤗 over 2 years ago
reacted to akhaliq's post with ➕🤗 over 2 years ago
Post
5004
Layer Skip
Enabling Early Exit Inference and Self-Speculative Decoding
LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding (2404.16710)
We present LayerSkip, an end-to-end solution to speed-up inference of large language models (LLMs). First, during training we apply layer dropout, with low dropout rates for earlier layers and higher dropout rates for later layers, and an early exit loss where all transformer layers share the same exit. Second, during inference, we show that this training recipe increases the accuracy of early exit at earlier layers, without adding any auxiliary layers or modules to the model. Third, we present a novel self-speculative decoding solution where we exit at early layers and verify and correct with remaining layers of the model. Our proposed self-speculative decoding approach has less memory footprint than other speculative decoding approaches and benefits from shared compute and activations of the draft and verification stages. We run experiments on different Llama model sizes on different types of training: pretraining from scratch, continual pretraining, finetuning on specific data domain, and finetuning on specific task. We implement our inference solution and show speedups of up to 2.16x on summarization for CNN/DM documents, 1.82x on coding, and 2.0x on TOPv2 semantic parsing task.
Enabling Early Exit Inference and Self-Speculative Decoding
LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding (2404.16710)
We present LayerSkip, an end-to-end solution to speed-up inference of large language models (LLMs). First, during training we apply layer dropout, with low dropout rates for earlier layers and higher dropout rates for later layers, and an early exit loss where all transformer layers share the same exit. Second, during inference, we show that this training recipe increases the accuracy of early exit at earlier layers, without adding any auxiliary layers or modules to the model. Third, we present a novel self-speculative decoding solution where we exit at early layers and verify and correct with remaining layers of the model. Our proposed self-speculative decoding approach has less memory footprint than other speculative decoding approaches and benefits from shared compute and activations of the draft and verification stages. We run experiments on different Llama model sizes on different types of training: pretraining from scratch, continual pretraining, finetuning on specific data domain, and finetuning on specific task. We implement our inference solution and show speedups of up to 2.16x on summarization for CNN/DM documents, 1.82x on coding, and 2.0x on TOPv2 semantic parsing task.
posted an update over 2 years ago
Post
6644
VTuber Logo Generator❤️🪄⭐️ by @gojiteji
gojiteji/VTuberLogoGenerator
How this works:
- mistralai/Mixtral-8x7B-Instruct-v0.1 for Japanese transliteration.
- Stable Diffusion 3 for logo generation.
- simple k-means for color selection.
gojiteji/VTuberLogoGenerator
How this works:
- mistralai/Mixtral-8x7B-Instruct-v0.1 for Japanese transliteration.
- Stable Diffusion 3 for logo generation.
- simple k-means for color selection.
- 8 replies