In the first case, the non-standard representation prevents benefiting from latest network architectures for neural representations; while, in the latter case, therasterized representation, when encoded vianetworks, results inlossof data fidelity, as font-specific discontinuities like edges and corners are difficult torepresent using neural networks.

Unfortunately, many real-world problems of practical interest have these characteristics.

Therearetwoupdatingstrategies: 1) mimicking strategy to generate similar samples based on original data, preserving stylistic and contextual essence, and 2) extending strategy that further expands existing samples at varying cognitive levels by adapting Bloom's taxonomy ofeducational objectives. Extensiveexperiments onupdated MMLU andBIG-Bench demonstrate thestability oftheproposed strategiesandfindthat the mimicking strategy can effectively alleviate issues of overestimation from benchmark leakage. In cases where the efficient mimicking strategy fails, our extending strategystill showspromising results.

D2C uses a learned diffusion-based prior over the latent representations to improve generation and contrastive selfsupervised learning to improve representation quality.

A promising direction istouse linear recurrent architectures (LRUs), where dense recurrent weights are replaced with a complex-valued diagonal, making RTRL efficient.