This page contains a list of papers on multi-task learning for computer vision. Please create a pull request if you wish to add anything. If you are interested, consider reading our recent survey paper.
Multi-Task Learning for Dense Prediction Tasks: A Survey
Simon Vandenhende, Stamatios Georgoulis, Wouter Van Gansbeke, Marc Proesmans, Dengxin Dai and Luc Van Gool.
- Survey papers
- Datasets
- Architectures
- Neural Architecture Search
- Optimization strategies
- Transfer learning
-
Vandenhende, S., Georgoulis, S., Van Gansbeke, W., Proesmans, M., Dai, D., & Van Gool, L. Multi-Task Learning for Dense Prediction Tasks: A Survey, ArXiv, 2020. [PyTorch]
-
Ruder, S. An overview of multi-task learning in deep neural networks, ArXiv, 2017.
-
Zhang, Y. A survey on multi-task learning, ArXiv, 2017.
-
Gong, T., Lee, T., Stephenson, C., Renduchintala, V., Padhy, S., Ndirango, A., ... & Elibol, O. H. A comparison of loss weighting strategies for multi task learning in deep neural networks, IEEE Access, 2019.
The following datasets have been regularly used in the context of multi-task learning:
-
Misra, I., Shrivastava, A., Gupta, A., & Hebert, M. Cross-stitch networks for multi-task learning, CVPR, 2016. [PyTorch]
-
Gao, Y., Ma, J., Zhao, M., Liu, W., & Yuille, A. L. Nddr-cnn: Layerwise feature fusing in multi-task cnns by neural discriminative dimensionality reduction, CVPR, 2019. [Tensorflow] [PyTorch]
-
Liu, S., Johns, E., & Davison, A. J. End-to-end multi-task learning with attention, CVPR, 2019. [PyTorch]
-
Xu, D., Ouyang, W., Wang, X., & Sebe, N. Pad-net: Multi-tasks guided prediction-and-distillation network for simultaneous depth estimation and scene parsing, CVPR, 2018.
-
Zhang, Z., Cui, Z., Xu, C., Jie, Z., Li, X., & Yang, J. Joint task-recursive learning for semantic segmentation and depth estimation, ECCV, 2018.
-
Ruder, S., Bingel, J., Augenstein, I., & Søgaard, A. Latent multi-task architecture learning, AAAI, 2019.
-
Zhang, Z., Cui, Z., Xu, C., Yan, Y., Sebe, N., & Yang, J. Pattern-affinitive propagation across depth, surface normal and semantic segmentation, CVPR, 2019.
-
Zhou, L., Cui, Z., Xu, C., Zhang, Z., Wang, C., Zhang, T., & Yang, J. Pattern-Structure Diffusion for Multi-Task Learning, CVPR, 2020.
-
Vandenhende, S., Georgoulis, S., & Van Gool, L. MTI-Net: Multi-Scale Task Interaction Networks for Multi-Task Learning, ECCV, 2020. [PyTorch]
-
Yang, Y., & Hospedales, T. Deep multi-task representation learning: A tensor factorisation approach, ICLR, 2017.
-
Kokkinos, Iasonas. Ubernet: Training a universal convolutional neural network for low-, mid-, and high-level vision using diverse datasets and limited memory, CVPR, 2017.
-
Rebuffi, S. A., Bilen, H., & Vedaldi, A. Learning multiple visual domains with residual adapters, NIPS, 2017.
-
Long, M., Cao, Z., Wang, J., & Philip, S. Y. Learning multiple tasks with multilinear relationship networks, NIPS, 2017.
-
Meyerson, E., & Miikkulainen, R. Beyond shared hierarchies: Deep multitask learning through soft layer ordering, ICLR, 2018.
-
Rosenbaum, C., Klinger, T., & Riemer, M. Routing networks: Adaptive selection of non-linear functions for multi-task learning, ICLR, 2018.
-
Mallya, A., Davis, D., & Lazebnik, S. Piggyback: Adapting a single network to multiple tasks by learning to mask weights, ECCV, 2018.
-
Rebuffi, S. A., Bilen, H., & Vedaldi, A. Efficient parametrization of multi-domain deep neural networks, CVPR, 2018.
-
Maninis, K. K., Radosavovic, I., & Kokkinos, I. Attentive single-tasking of multiple tasks, CVPR, 2019. [PyTorch]
-
Kanakis, M., Bruggemann, D., Saha, S., Georgoulis, S., Obukhov, A., & Van Gool, L. Reparameterizing Convolutions for Incremental Multi-Task Learning without Task Interference, ECCV, 2020.
-
Wang, Q., Ke, J., Greaves, J., Chu, G., Bender, G., Sbaiz, L., Go, A., Howard, A., Yang, F., Yang, M.H. & Gilbert, J. Multi-path Neural Networks for On-device Multi-domain Visual Classification, ArXiv, 2020.
-
Lu, Y., Kumar, A., Zhai, S., Cheng, Y., Javidi, T., & Feris, R. Fully-adaptive feature sharing in multi-task networks with applications in person attribute classification, CVPR, 2017.
-
Bragman, F. J., Tanno, R., Ourselin, S., Alexander, D. C., & Cardoso, J. Stochastic filter groups for multi-task cnns: Learning specialist and generalist convolution kernels, ICCV, 2019.
-
Newell, A., Jiang, L., Wang, C., Li, L. J., & Deng, J. Feature partitioning for efficient multi-task architectures, ArXiv, 2019.
-
Guo, P., Lee, C. Y., & Ulbricht, D. Learning to Branch for Multi-Task Learning, ICML, 2020.
-
Standley, T., Zamir, A. R., Chen, D., Guibas, L., Malik, J., & Savarese, S. Which Tasks Should Be Learned Together in Multi-task Learning?, ICML, 2020.
-
Vandenhende, S., Georgoulis, S., De Brabandere, B., & Van Gool, L. Branched multi-task networks: deciding what layers to share, BMVC, 2020.
-
Bruggemann, D., Kanakis, M., Georgoulis, S., & Van Gool, L. Automated Search for Resource-Efficient Branched Multi-Task Networks, BMVC, 2020.
-
Sun, X., Panda, R., & Feris, R. AdaShare: Learning What To Share For Efficient Deep Multi-Task Learning, NIPS, 2020.
-
Kendall, A., Gal, Y., & Cipolla, R. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics, CVPR, 2018.
-
Zhao, X., Li, H., Shen, X., Liang, X., & Wu, Y. A modulation module for multi-task learning with applications in image retrieval, ECCV, 2018.
-
Chen, Z., Badrinarayanan, V., Lee, C. Y., & Rabinovich, A. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks, ICML, 2018.
-
Sener, O., & Koltun, V. Multi-task learning as multi-objective optimization, NIPS, 2018. [PyTorch]
-
Liu, P., Qiu, X., & Huang, X. Adversarial multi-task learning for text classification, ACL, 2018.
-
Guo, M., Haque, A., Huang, D. A., Yeung, S., & Fei-Fei, L. Dynamic task prioritization for multitask learning, ECCV, 2018.
-
Lin, X., Zhen, H. L., Li, Z., Zhang, Q. F., & Kwong, S. Pareto multi-task learning, NIPS, 2019.
-
Suteu, M., & Guo, Y. Regularizing Deep Multi-Task Networks using Orthogonal Gradients, ArXiv, 2019.
-
Yu, T., Kumar, S., Gupta, A., Levine, S., Hausman, K., & Finn, C. Gradient surgery for multi-task learning, NIPS, 2020. [Tensorflow]
-
Chen, Z., Ngiam, J., Huang, Y., Luong, T., Kretzschmar, H., Chai, Y., & Anguelov, D. Just Pick a Sign: Optimizing Deep Multitask Models with Gradient Sign Dropout, NIPS, 2020.
-
Li, W. H., & Bilen, H. Knowledge Distillation for Multi-task Learning, ECCV-Workshop, 2020. [PyTorch]
-
Cui, Y., Song, Y., Sun, C., Howard, A., & Belongie, S. Large scale fine-grained categorization and domain-specific transfer learning, CVPR, 2018.
-
Zamir, A. R., Sax, A., Shen, W., Guibas, L. J., Malik, J., & Savarese, S. Taskonomy: Disentangling task transfer learning, CVPR, 2018. [PyTorch]
-
Achille, A., Lam, M., Tewari, R., Ravichandran, A., Maji, S., Fowlkes, C. C., ... & Perona, P. Task2vec: Task embedding for meta-learning, ICCV, 2019. [PyTorch]
-
Dwivedi, K., & Roig, G. Representation similarity analysis for efficient task taxonomy & transfer learning, CVPR, 2019. [PyTorch]
-
Mao, C., Gupta, A., Nitin, V., Ray, B., Song, S., Yang, J., & Vondrick, C. Multitask Learning Strengthens Adversarial Robustness, ECCV, 2020. [PyTorch]
-
Zamir, A. R., Sax, A., Cheerla, N., Suri, R., Cao, Z., Malik, J., & Guibas, L. J. Robust Learning Through Cross-Task Consistency, CVPR, 2020.