cuda-kernels

Malicious tool definition detected

Tool: SKILL.md [1/3] Description: --- name: cuda-kernels description: "Provides guidance for writing and benchmarking optimized CUDA kernels for NVIDIA GPUs (H100, A100, T4) targeting HuggingFace diffusers and transformers libraries.

Tool: SKILL.md [2/3] Description: aim for multiples of 132 | | Threads/SM | 2048 | Max 16 blocks of 128 threads per SM | | Shared Memory | 192 KB/SM | Large tiles possible | | L2 Cache | 50 MB | Reuse across blocks | | Memory BW | 3.35 TB/s | Coalesced access critical | | Warp Size | 32 | All reductions use warp shuffles | ### Quick Comparison (H100 vs A100 vs T4) | Spec | H100 | A100 | T4 | |------|------|------|-----| | SMs | 132 | 108 | 40 | | Memory BW | 3.35 TB/s | 2.0 TB/s | 320 GB/s | | S

Tool: SKILL.md [3/3] Description: - for video - LTX-Video computes its own RoPE via `LTXVideoRotaryPosEmbed` ### GEGLU vs GELU - **GEGLU**: Input `[batch, seq, 2*hidden]` -> Output `[batch, seq, hidden]` - **GELU**: Standard activation - **LTX-Video uses GELU, NOT GEGLU** ### AdaLN - Formula: `norm(x) * weight * (1 + scale) + shift` - Used in DiT blocks for conditioning ## Performance Profiling ```bash # NVIDIA Nsight Systems nsys profile -o profile python your_script.py # NVIDIA Nsight Compute

Malicious tool definition detected

Tool: manifest.txt Description: # Files for kernels skills add SKILL.md references/a100-optimization-guide.md references/diffusers-h100.md references/diffusers-integration.md references/h100-optimization-guide.md references/huggingface-kernels-integration.md

Malicious tool definition detected

Tool: references/a100-optimization-guide.md [1/2] Description: # A100 GPU Optimization Guide for Diffusers/Transformers Kernels Deep dive into A100-specific optimizations for diffusion model and LLM CUDA kernels.

Tool: references/a100-optimization-guide.md [2/2] Description: ## Performance Profiling ### Expected Performance (A100 vs H100) | Kernel | A100 (ms) | H100 (ms) | H100 Speedup | |--------|-----------|-----------|--------------| | RMSNorm [2, 1024, 2048] | ~0.08 | 0.054 | 1.5x | | GEGLU [2, 1024, 4096] | ~0.05 | 0.030 | 1.7x | ### Nsight Profiling ```bash # Same commands work on A100 nsys profile -o a100_profile python your_script.py ncu --set full -o a100_metrics.ncu-rep python your_script.py #

Malicious tool definition detected

Tool: references/diffusers-h100.md [1/2] Description: # H100 CUDA Kernels for Diffusers This skill provides patterns and guidance for developing optimized CUDA kernels targeting NVIDIA H100 GPUs (compute capability 9.0) for use with the HuggingFace diffusers library.

Tool: references/diffusers-h100.md [2/2] Description: 2, MAX_THREADS); threads = max(threads, WARP_SIZE); threads = (threads + 32 - 1) / 32 * 32; // Round to warp boundary ``` ## Supported Data Types All kernels support three precision modes: - `__half` (FP16) - Default for inference - `__nv_bfloat16` (BF16) - Preferred for training - `float` (FP32) - Reference/debugging ## Building Kernels ### With Nix (Recommended) ```bash nix run .#build-and-copy --max-jobs 2 --cores 8 -L ``` ### With pip/uv

Malicious tool definition detected

Tool: references/diffusers-integration.md [1/2] Description: # Diffusers Pipeline Integration Guide Complete guide for integrating custom CUDA kernels into HuggingFace diffusers pipelines.

Tool: references/diffusers-integration.md [2/2] Description: 2.

Malicious tool definition detected

Tool: references/h100-optimization-guide.md [1/2] Description: # H100 GPU Optimization Guide for Diffusers Kernels Deep dive into H100-specific optimizations for diffusion model CUDA kernels.

Tool: references/h100-optimization-guide.md [2/2] Description: my_kernel.occupancy(max_block_size) ``` ## Precision and Numerical Stability ### BF16 vs FP16 For diffusion models: ``` FP16: 1 sign + 5 exponent + 10 mantissa - Better precision (10 bits) - Smaller range (±65504) - Risk of overflow in attention scores BF16: 1 sign + 8 exponent + 7 mantissa - Same range as FP32 - Less precision (7 bits) - Safer for attention (no overflow) - Preferred for training ``` ### Online Softmax for Attention

Malicious tool definition detected

Tool: references/huggingface-kernels-integration.md [1/2] Description: # HuggingFace Kernels Integration Guide Complete guide for using and publishing CUDA kernels with the HuggingFace Kernels library (`get_kernel`).

Tool: references/huggingface-kernels-integration.md [2/2]

Malicious tool definition detected

Tool: references/kernel-templates.md [1/3] Description: # CUDA Kernel Templates for H100 Diffusers Complete, copy-paste ready templates for implementing new kernels.

Tool: references/kernel-templates.md [2/3]

Tool: references/kernel-templates.md [3/3] Description: static_cast<const float*>(input.data_ptr()), static_cast<float*>(output.data_ptr()), total_elements, stream ); } else { TORCH_CHECK(false, "Unsupported dtype"); } } // In TORCH_LIBRARY_EXPAND: // ops.def("your_kernel_forward(Tensor!

Malicious tool definition detected

Tool: references/t4-optimization-guide.md [1/2] Description: # T4 GPU Optimization Guide for Diffusers/Transformers Kernels Deep dive into T4-specific optimizations for diffusion model and LLM CUDA kernels.

Tool: references/t4-optimization-guide.md [2/2] Description: overflow float scale = 1.0f / sqrtf((float)head_dim); // For T4 FP16: May need additional scaling // scale *= 0.125f; // Extra scaling if overflow occurs ``` ### Mixed Precision Pattern Always accumulate in FP32: ```cuda // Input in FP16 (T4) float sum = 0.0f; // Accumulate in FP32 for (int i = tid; i < hidden_size; i += blockDim.x) { float val = __half2float(input[i]); // Convert to FP32 sum += val * val; } // Reduction in FP32 sum =

Malicious tool definition detected

Tool: references/transformers-integration.md [1/2] Description: # Transformers Library Integration Guide Complete guide for integrating custom CUDA kernels into HuggingFace transformers models.

Tool: references/transformers-integration.md [2/2] Description: = tokenizer("Hello, my name is", return_tensors="pt").to("cuda") with torch.inference_mode(): outputs = model.generate(**inputs, max_new_tokens=20) print(tokenizer.decode(outputs[0])) ``` ## Performance Optimization ### Enable Flash Attention 2 ```python model = AutoModelForCausalLM.from_pretrained( model_id, torch_dtype=torch.bfloat16, attn_implementation="flash_attention_2", device_map="cuda" ) ``` ### Use torch.compile ```python

Malicious tool definition detected

Tool: references/troubleshooting.md Description: # Troubleshooting Guide Common issues and solutions when working with H100 CUDA kernels for diffusers. ## Build Issues ### 1.

Malicious tool definition detected

Tool: scripts/benchmark_example.py [1/2] Description: #!/usr/bin/env python3 """ Benchmarking script for LTX-Video with/without custom H100 CUDA kernels.

Tool: scripts/benchmark_example.py [2/2] Description: stability print(f" Device: {torch.cuda.get_device_name(0)}") print(f"Dtype: {dtype}") # Check kernel availability and configuration if use_optimized_kernels: if not KERNELS_AVAILABLE: print(" ERROR: Optimized kernels requested but not available!") print("Please build the kernels first or use --no-optimized-kernels") sys.exit(1) print("Custom kernels: ENABLED") else: print("Custom kernels: DISABLED (baseline)") # Modify output path to include

Malicious tool definition detected

Tool: scripts/benchmark_rmsnorm.py Description: #!/usr/bin/env python3 """ Micro-benchmark for RMSNorm kernel to verify vectorized optimization.

Malicious tool definition detected

Tool: scripts/huggingface_kernels_example.py Description: #!/usr/bin/env python3 """ Example: Using HuggingFace Kernels library to load and use optimized CUDA kernels.

Malicious tool definition detected

Tool: scripts/ltx_kernel_injection_example.py Description: #!/usr/bin/env python3 """ Minimal example: Inject custom CUDA kernels into LTX-Video pipeline.

Malicious tool definition detected

Tool: scripts/transformers_injection_example.py Description: #!/usr/bin/env python3 """ Minimal example: Inject custom CUDA kernels into HuggingFace Transformers models.