Abstract
We developed a complete holistic medical multimedia system for gastrointestinal (GI) tract disease detection and localization based on Computer Vision (CV) and Artificial Intelligence (AI). Our DeepEIR system is designed as flexible, generalizable, adaptable, efficient and accurate solution that supports various endoscopic devices including wireless capsular endoscopes, and it can be easily expanded with new diseases and objects. The system can both process a vast amount of data off-line and perform real-time support during live medical procedures. We also contributed to the problem of medical data availability for research community. We collected, annotated, and published several datasets and data annotation tools as open source. Our datasets (Kvasir, Nerthus and Medico) immediately got a lot of attention and are used by many research teams. This work connects Multimedia and Medicine and uses Image Analysis, Machine Learning (ML), Convolutional Neural Networks (CNN), Deep Learning (DL) and Generative Adversarial Networks (GAN) to support doctors in their daily routine, reduce lesion overlooking and, therefore, have a societal impact by helping people to survive lethal diseases.
List of papers
| Paper I: LIRE - Open Source Visual Information Retrieval. Mathias Lux, Michael Riegler, Pål Halvorsen, Konstantin Pogorelov, Nektarios Anagnostopoulos. MMSys’16 May 10-13, 2016. DOI: 10.1145/2910017.2910630. The article is not available in DUO due to publisher restrictions. The published version is available at: https://doi.org/10.1145/2910017.2910630 |
| Paper II: Konstantin Pogorelov, Zeno Albisser, Olga Ostroukhova, Mathias Lux, Dag Johansen, Pål Halvorsen, and Michael Riegler. 2018. OpenSea - Open Search Based Classification Tool. In Proceedings of 9th ACM Multimedia Systems Conference, Amsterdam, Netherlands, June 12–15, 2018 (MMSys’18), 6 pages. DOI: 10.1145/3204949.3208128. The article is not available in DUO due to publisher restrictions. The published version is available at: https://doi.org/10.1145/3204949.3208128 |
| Paper III: Explorative Hyperbolic-Tree-Based Clustering Tool for Unsupervised Knowledge Discovery. Michael Riegler, Konstantin Pogorelov, Mathias Lux, Pål Halvorsen, Carsten Griwodz, Thomas de Lange, Sigrun Losada Eskeland. 2016, 14th International Workshop on Content-Based Multimedia Indexing (CBMI), Bucharest, 2016, pp. 1-4. DOI: 10.1109/CBMI.2016.7500271. The article is not available in DUO due to publisher restrictions. The published version is available at: https://doi.org/10.1109/CBMI.2016.7500271 |
| Paper IV: Konstantin Pogorelov, Michael Riegler, Pål Halvorsen, and Carsten Griwodz. 2017. ClusterTag: Interactive Visualization, Clustering and Tagging Tool for Big Image Collections. In Proceedings of ICMR ’17, Bucharest, Romania, June 6–9, 2017, 5 pages. DOI: 10.1145/3078971.3079018. The article is not available in DUO due to publisher restrictions. The published version is available at: https://doi.org/10.1145/3078971.3079018 |
| Paper V: EIR - Efficient Computer Aided Diagnosis Framework for Gastrointestinal Endoscopies. Michael Riegler, Konstantin Pogorelov, Pål Halvorsen, Thomas de Lange, Carsten Griwodz, Peter Thelin Schmidt, Sigrun Losada Eskeland, Dag Johansen. 2016 14th International Workshop on Content-Based Multimedia Indexing (CBMI), Bucharest, 2016, pp. 1-6. 2016 IEEE. DOI: 10.1109/CBMI.2016.7500257. The article is not available in DUO due to publisher restrictions. The published version is available at: https://doi.org/10.1109/CBMI.2016.7500257 |
| Paper VI: Michael Riegler, Konstantin Pogorelov, Sigrun Losada Eskeland, Peter Thelin Schmidt, Zeno Albisser, Dag Johansen, Carsten Griwodz, Pål Halvorsen, and Thomas de Lange. 2017. From annotation to computer-aided diagnosis: Detailed evaluation of a medical multimedia system. ACM Trans. Multimedia Comput. Commun. Appl. 13, 3, Article 26 (May 2017), 26 pages. DOI: 10.1145/3079765. The article is not available in DUO due to publisher restrictions. The published version is available at: https://doi.org/10.1145/3079765 |
| Paper VII: Multimedia and Medicine: Teammates for Better Disease Detection and Survival. Michael Riegler, Mathias Lux, Carsten Griwodz, Concetto Spampinato , Thomas de Lange, Sigrun L. Eskeland, Konstantin Pogorelov, Wallapak Tavanapong, Peter T. Schmidt, Cathal Gurrin, Dag Johansen, Håvard Johansen, Pål Halvorsen. MM ’16, October 15 - 19, 2016, Amsterdam, Netherlands. DOI: 10.1145/2964284.2976760. The article is not available in DUO due to publisher restrictions. The published version is available at: https://doi.org/10.1145/2964284.2976760 |
| Paper VIII: Konstantin Pogorelov, Sigrun Losada Eskeland, Thomas de Lange, Carsten Griwodz, Kristin Ranheim Randel, Håkon Kvale Stensland, Duc-Tien Dang-Nguyen, Concetto Spampinato, Dag Johansen, Michael Riegler, and Pål Halvorsen. 2017. A Holistic Multimedia System for Gastrointestinal Tract Disease Detection. In Proceedings of MMSys ’17, Taipei, Taiwan, June 20–23, 2017, 12 pages. DOI: 10.1145/3083187.3083189. The article is not available in DUO due to publisher restrictions. The published version is available at: https://doi.org/10.1145/3083187.3083189 |
| Paper IX: GPU-accelerated Real-time Gastrointestinal Diseases Detection. Konstantin Pogorelov, Michael Riegler, Pål Halvorsen, Peter Thelin Schmidt, Carsten Griwodz, Dag Johansen, Sigrun Losada Eskeland, Thomas de Lange. 2016 IEEE 29th International Symposium on Computer-Based Medical Systems (CBMS), Dublin, 2016, pp. 185-190. DOI: 10.1109/CBMS.2016.63. The article is not available in DUO due to publisher restrictions. The published version is available at: https://doi.org/10.1109/CBMS.2016.63 |
| Paper X: Efficient Processing of Videos in a Multi-Auditory Environment Using Device Lending of GPUs. Konstantin Pogorelov, Michael Riegler, Jonas Markussen, Håkon Kvale Stensland, Pål Halvorsen, Carsten Griwodz, Sigrun Losada Eskeland, Thomas de Lange. MMSys’16 May 10-13, 2016, Klagenfurt, Austria. DOI: 10.1145/2910017.2910636. The article is not available in DUO due to publisher restrictions. The published version is available at: https://doi.org/10.1145/2910017.2910636 |
| Paper XI: Efficient disease detection in gastrointestinal videos - global features versus neural networks. Konstantin Pogorelov, Michael Riegler, Sigrun Losada Eskeland, Thomas de Lange, Dag Johansen, Carsten Griwodz, Peter Thelin Schmidt, Pål Halvorsen. Multimed Tools Appl (2017) 76:22493–22525. DOI: 10.1007/s11042-017-4989-y. The article is included in the thesis. Also available at: https://doi.org/10.1007/s11042-017-4989-y |
| Paper XII: Konstantin Pogorelov, Kristin Ranheim Randel, Carsten Griwodz, Sigrun Losada Eskeland, Thomas de Lange, Dag Johansen, Concetto Spampinato, Duc-Tien Dang-Nguyen, Mathias Lux, Peter Thelin Schmidt, Michael Riegler, and Pål Halvorsen. 2017. Kvasir: A Multi-Class Image Dataset for Computer Aided Gastrointestinal Disease Detection. In Proceedings of MMSys ’17, Taipei, Taiwan, June 20–23, 2017, 6 pages. DOI: 10.1145/3083187.3083212 The article is not available in DUO due to publisher restrictions. The published version is available at: https://doi.org/10.1145/3083187.3083212 |
| Paper XIII: Nerthus: A Bowel Preparation Quality Video Dataset. Konstantin Pogorelov, Kristin Ranheim Randel, Thomas de Lange, Sigrun Losada Eskeland, Carsten Griwodz, Dag Johansen, Concetto Spampinato, Mario Taschwer, Mathias Lux, Peter Thelin Schmidt, Michael Riegler, and Pål Halvorsen. 2017. Nerthus: A Bowel Preparation Quality Video Dataset. In Proceedings of MMSys ’17, Taipei, Taiwan, June 20–23, 2017, 5 pages. DOI: 10.1145/3083187.3083216. The article is not available in DUO due to publisher restrictions. The published version is available at: https://doi.org/10.1145/3083187.3083216 |
| Paper XIV: Deep Learning and Handcrafted Feature Based Approaches for Automatic Detection of Angiectasia. Konstantin Pogorelov, Olga Ostroukhova, Andreas Petlund, Pål Halvorsen, Thomas de Lange, Håvard Nygaard Espeland, Tomas Kupka, Carsten Griwodz and Michael Riegler. 2018 IEEE EMBS International Conference on Biomedical & Health Informatics (BHI), Las Vegas, NV, 2018, pp. 365-368. DOI: 10.1109/BHI.2018.8333444. The article is not available in DUO due to publisher restrictions. The published version is available at: https://doi.org/10.1109/BHI.2018.8333444 |
| Paper XV: Deep Learning and Hand-crafted Feature Based Approaches for Polyp Detection in Medical Videos. Konstantin Pogorelov, Olga Ostroukhova, Mattis Jeppsson, Håvard Espeland, Carsten Griwodz, Thomas de Lange, Dag Johansen, Michael Riegler and Pål Halvorsen.2018 IEEE 31st International Symposium on Computer-Based Medical Systems (CBMS), Karlstad, 2018, pp. 381-386. DOI: 10.1109/CBMS.2018.00073. The article is not available in DUO due to publisher restrictions. The published version is available at: https://doi.org/10.1109/CBMS.2018.00073 |