This is an interesting paper combining machine learning and psychology, and brings interesting insights about what can be learned from naturalistic, egocentric, real-world datasets, and how the learned representations can be used on downstream tasks. It's well-written and clearly presented, and likely of interest to the general NeurIPS audience. Reviewers 3 and 4 initially had concerns about the model's performance, and suggestions about using other datasets. After discussion and the rebuttal, they were able to be convinced that these run counter to the main motivation of the study. They subsequently raised their scores and all reviewers - and myself - are in agreement to accept.

Weaknesses: - Despite the extensive empirical evaluations, the three multimodal variants as proposed by the paper are direct extensions of the DeepCluster algorithm [4]. The main contributions appear to be (1) a working pipeline which demonstrates that variants of DeepCluster works with video and audio encoders; (2) scaling up the training to extremely large datasets. While both contributions are interesting, they appear to me to be less relevant to the audience of NeurIPS. It would also be great if such conjectures are accompanied with empirical evaluations on more diverse tasks than the three classification datasets. That would help the audience understand when to apply the XDC variant of DeepCluster (e.g. is it specific to audio and visual in videos, or is it more general?),

The reviewers generally agree this paper has great execution, a great idea, and great results. The reviewers noted the impact that self-supervised learning on video can have, which has been less explored than the image counterpart. The reviewers also praised the strong empirical results, which will be of high interest to the community.

Post-feedback update: Thanks for your update. Your additional explanations and results will help improve the paper, and I definitely think this work is strong and should be accepted. The framework itself is new, and the authors make it very clear how prior work fits into the framework as special cases. At the same time, a good case is made for why this framework is useful to have and how it can be better to use than prior losses. Quality: this paper makes a compelling case for the framework it introduces.

All reviewers agreed that this paper make a nice contribution to NeurIPS by providing a novel general framework for generating calibrated surrogate loss functions for structured prediction problems. On the other hand, in discussion, they also stressed that including some baselines (e.g., SSVM/CRF approximation/SPEN) in the experiments and reporting runtimes could make this paper much stronger. The authors should implement their promised changes in the camera-ready version.

The paper presents three GNN architectural guidelines for combating this, which can lead to improved predictions, particularly on networks exhibiting heterophilous structure (i.e., non-homophilous labels). The design choices are motivated theoretically and intuitively and then combined into a single model that can provide better predictions on networks with heterophilous structure, as demonstrated by synthetic and real-world data experiments. The paper provides a number of interesting insights into why certain GNN architectural choices can help predictions in the case of low network homophily. Although not mentioned in their paper, a similar idea to higher-order neighborhoods (Section 3.1.2) I believe that these ideas provide further motivation for the design choices appearing in this paper and including them will strengthen some of the intuition.

As the topic of the this work is very interesting, I'm not frustrated by the presentation, which has a large space for improvement. First, the motivation of the work is not so clear. Why is that an important concern to be able to poison a G-SSL model? The three references [1-3] provided seem at least a decade ago. Are they widely applied in modern security/privacy sensitive applications?

I recommend acceptance of this article subject to some corrections. The novelty of considering data poisoning in GSSL can be interesting for the NeurIPS community. The results presented here are founded by theoretical results and experiments are conducted. It opens new lines of research and can trigger new results. Because the paper has several weak points, we insist on the importance to handle the following comments. However, the overall structure of the presentation is acceptable.

The integration of convnets with the conditional random fields to model the structural dependencies of facial landmarks during face alignment is nice contribution. Previously proposed methods in this direction were hybrid systems (eg. OpenFace versions) and not fully integrated. The authors evaluate on multiple datasets (300W, 300W-Video, Menpo & COFW-68) and compare results with other methods. Both inter- and cross-dataset performance are provided.

The authors did a good job of addressing reviewer concerns in their rebuttal and the reviewers converged towards an accept recommendation.