This is a very straightforward practical approach:
https://github.com/NVIDIA/DIGITS/tree/master/examples/fine-tuning
One trick I’ve learned from somewhere (can’t find the link, unfortunately), which is a break from the above tutorial, is to simply reduce the base learning rate by an order of magnitude when transferring, while simultaneously seting the “new” layers to have their lr_mult an order of magnitude higher than the rest of the network
so, to initially train:
- learning rate =.01
- each layer's lr_rate = 1
to transfer learn:
- learning rate = .001
- each previous layer's lr_rate = 1
- new layers lr_rate = 10
This saves a lot of editing of the files and allows for small amounts of adjustment to existing layers, while focusing the bulk of the learning on the newer layers, yet still resisting to overfitting small datasets.
This is pretty informative:
https://cs231n.github.io/transfer-learning/
And I think this is the original TL paper as far as I remember: