Neural Information Processing Systems http://nips.cc/

Neural Information Processing Systems http://nips.cc/

Disentangling methods aim to make explicit compositional or factorial structure.

Code data has been shown to enhance the reasoning capabilities of large language models (LLMs), but it remains unclear which aspects of code are most responsible. We investigate this question with a systematic, data-centric framework. We construct parallel instruction datasets in ten programming languages and apply controlled perturbations that selectively disrupt structural or semantic properties of code. We then finetune LLMs from five model families and eight scales on each variant and evaluate their performance on natural language, math, and code tasks. Across 3,331 experiments, our results show that LLMs are more vulnerable to structural perturbations than semantic ones, particularly on math and code tasks. Appropriate abstractions like pseudocode and flowcharts can be as effective as code, while encoding the same information with fewer tokens without adhering to original syntax can often retain or even improve performance. Remarkably, even corrupted code with misleading signals remains competitive when surface-level regularities persist. Finally, syntactic styles also shape task-specific gains with Python favoring natural language reasoning and lower-level languages such as Java and Rust favoring math. Through our systematic framework, we aim to provide insight into how different properties of code influence reasoning and inform the design of training data for enhancing LLM reasoning capabilities.

Department of Mathematics, The Hong Kong University of Science and Technology Clear Water Bay, Hong Kong, China yyangdc@connect.ust.hk In this supplementary material, we first prove Theorem 3.2, and then Theorems 3.1 and 3.3. We use σ to denote the ReLU activation function in neural networks, which is σ (x) = max {x, 0}. We use notation O () and O () to express the order of function slightly differently, where O () omits the universal constant not relying on d while O () omits the constant related to d . So far, most of the related works assume that the target distribution µ is supported on a compact set, for example Chen et al. (2020) and Liang (2020).

This study examines the impact of tokenized Java code length on the accuracy and explicitness of ten major LLMs in vulnerability detection. Using chi-square tests and known ground truth, we found inconsistencies across models: some, like GPT-4, Mistral, and Mixtral, showed robustness, while others exhibited a significant link between tokenized length and performance. We recommend future LLM development focus on minimizing the influence of input length for better vulnerability detection. Additionally, preprocessing techniques that reduce token count while preserving code structure could enhance LLM accuracy and explicitness in these tasks.

This approach tries to connect together the literature of learning deep embeddings and the literature on disentangled representation learning. In particular, it leverages the weakly supervised training process of the deep embeddings literature, which relies on the availability of data that either belongs to a class (shares a particular generative factor) or not. The authors encourage disentangling by separating the examples of the different classes based on independent per-embedding-dimension hypothesis testing using the F-statistic loss. Two examples are considered to belong to different classes based on their dissimilarity on a subset of d dimensions of the embedding, which allows the approach to be more flexible. The authors also show that this approach works when the separation is done not on class labels, but on attribute features.

As defined by Bengio et al. [1], representation recent scholarly reviews on the topic [8, 7]. Accordingly, learning transforms observations into a format that captures a metric designed to quantify modularity and compactness the essence of data's inherent patterns and structures. An should also assess informativeness i.e., the extent to which ideal representation should exhibit five key characteristics: (a) latent codes encapsulate information about generative factors. Disentanglement, ensuring separate encoding of interpretable When the ground truth factors of variation are identifiable, factors; (b) Informativeness, capturing the diversity of data; (c) this informativeness transforms into explicitness, denoting the Invariance, maintaining stability across changes in unrelated comprehensive representation of all recognised factors [9].