knowledge-distillation

Malicious tool definition detected

Tool: SKILL.md [1/2] Description: --- name: knowledge-distillation description: Compress large language models using knowledge distillation from teacher to student models.

Tool: SKILL.md [2/2] Description: logits = teacher(**batch).logits teacher_logits_list.append(logits) # Average teacher predictions avg_teacher_logits = torch.stack(teacher_logits_list).mean(dim=0) # Student learns from ensemble student_logits = student(**batch).logits loss = F.kl_div( F.log_softmax(student_logits, dim=-1), F.softmax(avg_teacher_logits, dim=-1), reduction='batchmean' ) return loss ``` ## Production Deployment ### Complete Training Script ```python from transformers import Traine

Malicious tool definition detected

Tool: references/minillm.md Description: # MiniLLM: Reverse KL Divergence for LLM Distillation Based on arXiv 2306.08543 (2024) - MiniLLM: Knowledge Distillation of Large Language Models ## Overview **Source**: https://arxiv.org/abs/2306.08543 **GitHub**: https://github.com/microsoft/LMOps/tree/main/minillm MiniLLM replaces forward KLD with reverse KLD for knowledge distillation, achieving better performance on generative language models.