Matrix Decomposition

In the mathematical discipline of linear algebra, a matrix decomposition or matrix factorization is a factorization of a matrix into a product of matrices.

• 참고 https://en.wikipedia.org/wiki/Matrix_decomposition

개요

• 참고 https://ettrends.etri.re.kr/ettrends/176/0905176005/34-2_40-50.pdf

경량 딥러닝 기술은

1. 경량 딥러닝 알고리즘 연구: 알고리즘 자체를 적은 연산과 효율적인 구조로 설계하여, 기존 모델 대비 효율을 극대화
2. 알고리즘 경량화 기술: 만들어진 모델의 파라미터들을 줄이는 모델 압축(Model Compression) 등의 기법이 적용

• • 이진화(Binarization) 딥러닝에 사용되는 가중치의 부동 소수점 수를 0과 1로 표현하여, 그 크기를 줄이는 방법도 고려

Application(model compression and acceleration)

• Matrix factorization was widely studied in the deep learning domain for model compression and acceleration

LadaBERT

• 참고 https://arxiv.org/pdf/2004.04124.pdf
• LadaBERT is based on an iterative hybrid model compression framework consisting of weighting pruning, matrix factorization and knowledge distillation
• The architecture and weights of student model are inherited from the BERT teacher
  • In each iteration, the student model is first compressed by a small ratio based on weight pruning and matrix factorization
    • weight pruning and matrix factorization help to generate better initial and intermediate status in the knowledge distillation iterations,
    • the accuracy and efficiency of model compression can be greatly improved.