Korea Advanced Institute of Science and Technology (KAIST, President Shin Sung-chul) announced on Tuesday that a research team led by Professor Lee Jae-gil of KAIST School of Computing has developed a new learning technology that drastically improves level of accurate estimation and training speed of deep learning model.
Learning process takes place in a deep learning model through a method that repeatedly optimizes the model’s parameters. The model will select few samples from training data during each repetition and use them for optimization. Selected samples are called a batch. Because highest level of accuracy is not guaranteed at all times if batches are selected randomly, there have been many active studies trying to figure out better ways for models to select batches.
Methodology developed by the research team constructs the best batch according to the progress of learning process. It utilizes results from previous deductions in order to select data that will be most helpful for learning process.
If results from each deduction stage are very consistent, the methodology believes that corresponding data are not helpful learning due to them being too easy or difficult. On the other hand, it deems data necessary if results from deductions are not consistent because it sees that there may be some mix-ups in deductions.
The research team named its methodology as “Recency Bias” and applied to various convolutional neural networks (CNN). It found that its methodology was able to minimize errors up to 21% compared to current methodologies regarding image categorization prediction. Training duration was reduced by 59% at the maximum.
“Our methodology will be a core technology when it comes to deep learning.” said Song Hwan-joon who is going through a doctoral program and is the lead author of the study. “It will definitely improve overall performance of deep learning as it can be applied widely to various deep neural networks.”
Professor Lee said that the methodology can have significant ripple effects in machine learning and deep learning industries if it is added to the current library of deep learning that includes TensorFlow and PyTorch.
Staff Reporter Kim, Youngjoon | kyj85@etnews.com