Hugging Face released "Beyond LoRA," a comprehensive benchmark that evaluates 40+ parameter-efficient fine-tuning (PEFT) methods on a standardized set of tasks. The standout finding: LoRA is not the best PEFT method for image tasks — OFT (Orthogonal Fine-Tuning) beats LoRA on 7 out of 9 image benchmarks.
The benchmark structure: 40+ PEFT methods evaluated on 18 tasks spanning language, code, image generation, image classification, audio, and multimodal. Each method is tested at multiple "parameter budgets" (0.1%, 1%, 5% of full model parameters) to evaluate the efficiency-quality trade-off.
The findings:
- On language tasks, LoRA remains competitive — it wins on 6 out of 9 language benchmarks.
- On code tasks, LoRA and DoRA tie for the lead.
- On image generation tasks, OFT (Orthogonal Fine-Tuning) wins by 3-7 points over LoRA.
- On multimodal tasks, QLoRA + IA³ is the best combination.
The "Beyond LoRA" message: the paper is a clear message that LoRA is not the "one true method" — different tasks favor different PEFT methods. The benchmark provides a "PEFT selection guide" for practitioners: choose OFT for image generation, LoRA for language, QLoRA for low-memory deployment, etc.
The bigger takeaway: "PEFT method selection" is becoming a real engineering discipline. Most practitioners default to LoRA because it's well-known, but the Beyond LoRA benchmark shows that significant quality gains are possible by choosing the right method. For the industry, this means PEFT vendors will need to provide "method selection" tools, not just a single method.