Basic Information
Sotf-Teacher:
- Title: End-to-End Semi-Supervised Object Detection with Soft Teacher
- Conference: ICCV2021
- Implementation: Soft-Teacher
- Citation:
1
2
3
4
5
6@article{2021sotf,
title={End-to-End Semi-Supervised Object Detection with Soft Teacher},
author={Xu, Mengde and Zhang, Zheng and Hu, Han and Wang, Jianfeng and Wang, Lijuan and Wei, Fangyun and Bai, Xiang and Liu, Zicheng},
journal={Proceedings of the {IEEE/CVF} International Conference on Computer Vision ({ICCV})},
year={2021}
}
Consistent-Teacher:
- Title: Consistent-Teacher: Towards Reducing Inconsistent Pseudo-targets in Semi-supervised Object Detection
- Conference: CVPR 2023 Spotlight Paper
- Implementation: Consistent-Teacher
- Citation:
1
2
3
4
5
6@article{2023consistent,
author = {Xinjiang Wang, Xingyi Yang, Shilong Zhang, Yijiang Li, Litong Feng, Shijie Fang, Chengqi Lyu, Kai Chen, Wayne Zhang },
title = {Consistent-Teacher: Towards Reducing Inconsistent Pseudo-targets in Semi-supervised Object Detection},
journal = {The {IEEE/CVF} Computer Vision and Pattern Recognition Conference ({CVPR})},
year = {2023},
}
Related Works - Milestone on the way to SSL
Mean-Teacher Model
- Title: Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
- Conference: NIPS 2017
- Implementation: Mean-Teacher
半监督学习是一种利用少量有标签数据和大量无标签数据进行训练以提高模型性能的机器学习方法。其中,最广泛的一种模式是Teacher-Student
模式。模型训练分为三个步骤:
- 利用少量已标注数据对初始模型进行训练,
- 使用教师模型对大量未标注数据进行推断,并将推断结果作为伪标签加入训练集,
- 期望学生模型能够准确检测这些伪标签,并对增强后的输入样本做出一致的预测。
Mean-Teacher
利用有标签数据和无标签数据进行训练,其中Teacher
模型生成伪标签,并给Student
模型作为监督信号。Teacher
模型的参数由是Student
模型参数的指数滑动平均Exponential Moving Average
得到。相对应,Student
模型在进行过增强的未标注样本上进行训练,利用Teacher
模型推断得到的伪标签进行监督。通过这种方式,Mean-teacher
可以实现多视角一致的自监督训练。
The ‘Mean-Teacher’ model can be affected by the following three issues:
- Pseudo-labels are inconsistent,
- Pseudo-label with noise,
- Pseudo-label could lead to overfitting.
These issues can lead to degraded model performance. Based on Mean-Teacher, the existing methods improve the accuracy of pseudo-labels through various methods, but due to the lack of sufficient labeling data, the problem of poor quality of pseudo-label bounding boxes often occurs in the training process in SSOD problems, which leads to the instability and performance degradation of the model.
Mean-Teacher
模型的可能会受到以下三个问题的影响:
- 伪标签不一致,
- 伪标签含噪声,
- 伪标签过拟合。
这些问题都可能会导致模型的性能下降。在 Mean-Teacher 的基础上,现有半监督目标检测方法通过各种方法提高伪标签的准确性,但由于缺乏足够的标注数据,在SSOD问题中,训练过程中常常出现伪标签边界框质量较差的问题,并导致模型的不稳定性和性能的下降。
Motivation
在SSOD背景下,伪标签的不稳定性带来的问题:
- 分配不一致: 当前主流的两阶段(
Two-stage
)或者单阶段(Single-stage
)目标检测网络都使用基于IoU
阈值的静态ancho
r分配方法,这种方法对于伪标签框中的噪声非常敏感。即使伪标签框中只有微小的噪声,伪标签的不稳定性也会导致anchor
分配的不同。- 任务不一致:在主流的半监督目标检测方法中,分类与回归任务的不一致也是导致不稳定性的一个重要原因。为了筛选高质量的伪标签,通常会使用分类置信度作为指标,并设置阈值来筛除低置信度的伪标签框。然而,一个伪标签框的分类置信度好坏并不一定能反映其定位准确度的高低。因此,利用分类置信度进行伪标签筛选的方法会进一步加剧伪标签在训练过程中的不稳定性。
- 时序不一致: 固定阈值筛选伪标签的方法同样会导致不一致性。在半监督目标检测中,为了筛选高质量的伪标签进行训练,常常采用一个固定的阈值对分类的置信度进行筛选。然而,这种方法会导致在训练不同阶段的不一致性。在训练初期,由于模型对预测结果不够自信,固定的阈值会导致过少的伪标签框被筛选,而随着模型的不断训练,每张图的伪标签框数量会逐渐增多,直到训练后期过多。
Introduction of Soft-Teacher
[1, 2] conduct a multi-stage training schema, with the first stage training an initial detector using labeled data, followed by a pseudo-labeling process for unlabeled data and a re-training step based on the pseudo labeled unannotated data. These multi-stage approaches achieve reasonably good accuracy, however, the final performance is limited by the quality of pseudo labels generated by an initial and probably inaccurate detector trained using a small amount of labeled data.
- [1] A simple semi-supervised learning framework for object detection. arXiv:2005.04757, 2020.
- [2] Rethinking pre-training and self-training. NIPS, 2020.
- 教师模型和学生模型是两个完全相同的结构,因为要进行
EMA
更新,两者都是带有预训练的随机初始化。- 有标签图片采用常规的
pipeline
流程,利用学生模型进行预测,计算得到有标签的loss
,包括分类和回归分支loss
。- 参考
FixMatch
做法,无标签数据会经过强和弱两种不同的augmentation pipeline
,其中弱增强线输入到教师模型,而强增强线用于学生模型- 对于弱增强线的图片,经过教师模型推理预测,NMS 后处理可以得到检测结果。前面说过伪框的质量对最终性能影响非常大,需要小心处理,作者采用了高阈值来过滤教师模型的检测结果将其作为强增强线学生模型预测值中分类分支的标签,然而这可能导致许多学习模型预测值中真正的候选框被错误地分配为背景样本。 为了解决这个问题,作者建议使用可靠性度量来加权每个“背景”候选框的损失,而实测发现教师模型产生的背景检测分数可以很好地作为可靠性度量。
- 由于分类分支和检测分支预测的不一致性以及任务的不一致性,我们也需要找到一个可靠性指标来反应伪框的可信度,但是观察发现定位精度和前景分值没有很大联系,所以作者采用了另一种方法即通过框抖动
box jittering
选择可靠的边界框来训练学生模型的定位分支,这种方法首先多次抖动伪前景框候选; 然后在利用教师模型对这些抖动框进行回归(实际上是rcnn
分支进一步refine
),并将这些回归框的方差用作可靠性度量;最后将具有足够高可靠性的box
候选用于学生定位分支的训练。- 可以看出强增强线学生模型的无标签分类和回归分支的伪标签是不一样的。教师模型采用
Mean teachers
方法进行更新。学生模型的分类和回归分支的loss
由教师模型产生的分类分数加权。这种监督方式实测效果远好于hard
标签训练方式,所以它才称为soft teacher
。
以上,它主要是提出一个E2E
联合学习算法,并提出了两个新的改进:
- soft teacher 机制,其中每个无标记分支预测值的分类损失由教师网络产生的分类分数加权。
- box jittering 机制,用于选择可靠的伪框以进行无标记分支预测值的回归分支框学习。
Reference of this section
- 超实用半监督目标检测 Soft Teacher 及 MMDetection 最强代码实践 ZhiHu_01
Note on Experiment Settings:
Fully Labeled Data: In this setting, the entire train2017 is used as the labeled data and unlabeled2017 is used as the additional unlabeled data. This setting is more challenging. Its goal is to use the additional unlabeled data to improve a well-trained detector on large-scale labeled data.
Introduction of Consistent-Teacher
This study delves into the inconsistency of semi-supervised object detection (SSOD
) using pseudo-targets. The study found that the oscillating pseudo-targets can make it hard to train an accurate detector. This is because it introduces noise into the student’s training, which can lead to severe overfitting problems. To address this issue, the study proposes a solution called Consistent-Teacher.
Consistent-Teacher reduces the inconsistency by:
- Adopting an adaptive anchor assignment (
ASA
) strategy that substitutes the static Intersection over Union (IoU
)-based strategy. This makes the student network resistant to noisy pseudo-bounding boxes. - Using a 3D feature alignment module (
FAM-3D
) to calibrate the subtask predictions. The module allows each classification feature to adaptively query the optimal feature vector for the regression task at arbitrary scales and locations. - A Gaussian Mixture Model (
GMM
) dynamically revises the score threshold of pseudo-bboxes, stabilizing the number of ground truths early and remedies the unreliable supervision signal during training.
Consistent-Teacher 通过以下方式减少不一致:
- 采用自适应锚点分配 (ASA) 策略,取代基于并集 (IoU) 的静态交叉点策略。
- 使用 3D 特征对齐模块 (FAM-3D) 校准子任务预测。
- 高斯混合模型(GMM)动态修正伪bbox的得分阈值,及早稳定地面实况数量,并修复训练过程中不可靠的监督信号。