Automated reverse engineering of HTML/CSS code from UI screenshots is an important yet challenging problem with broad applications in website development and design. In this paper, we propose a novel vision-code transformer (ViCT) composed of a vision encoder processing the screenshots and a language decoder to generate the code. They are initialized by pre-trained models such as ViT/DiT and GPT-2/LLaMA but aligning the two modalities requires end-to-end finetuning, which aims to minimize the visual discrepancy between the code-rendered webpage and the original screenshot. However, the rendering is non-differentiable and causes costly overhead. We address this problem by actor-critic fine-tuning where a visual critic without rendering (ViCR) is developed to predict visual discrepancy given the original and generated code. To train and evaluate our models, we created two synthetic datasets of varying complexity, with over 75,000 unique (code, screenshot) pairs. We evaluate the UI-to-Code performance using a combination of automated metrics such as MSE, BLEU, IoU, and a novel htmlBLEU score. ViCT outperforms a strong baseline model DiT-GPT2, improving IoU from 0.64 to 0.79 and lowering MSE from 12.25 to 9.02. With much lower computational cost, it can achieve comparable performance as when using a larger decoder such as LLaMA.

Large language models often necessitate grounding on external knowledge to generate faithful and reliable answers. Yet even with the correct groundings in the reference, they can ignore them and rely on wrong groundings or their inherent biases to hallucinate when users, being largely unaware of the specifics of the stored information, pose questions that might not directly correlate with the retrieved groundings. In this work, we formulate this knowledge alignment problem and introduce MixAlign, a framework that interacts with both the human user and the knowledge base to obtain and integrate clarifications on how the user question relates to the stored information. MixAlign employs a language model to achieve automatic knowledge alignment and, if necessary, further enhances this alignment through human user clarifications. Experimental results highlight the crucial role of knowledge alignment in boosting model performance and mitigating hallucination, with improvements noted up to 22.2% and 27.1% respectively. We also demonstrate the effectiveness of MixAlign in improving knowledge alignment by producing high-quality, user-centered clarifications.

Explaining the behaviour of intelligent agents learned by reinforcement learning (RL) to humans is challenging yet crucial due to their incomprehensible proprioceptive states, variational intermediate goals, and resultant unpredictability. Moreover, one-step explanations for RL agents can be ambiguous as they fail to account for the agent's future behaviour at each transition, adding to the complexity of explaining robot actions. By leveraging abstracted actions that map to task-specific primitives, we avoid explanations on the movement level. To further improve the transparency and explainability of robotic systems, we propose an explainable Q-Map learning framework that combines reward decomposition (RD) with abstracted action spaces, allowing for non-ambiguous and high-level explanations based on object properties in the task. We demonstrate the effectiveness of our framework through quantitative and qualitative analysis of two robotic scenarios, showcasing visual and textual explanations, from output artefacts of RD explanations, that are easy for humans to comprehend. Additionally, we demonstrate the versatility of integrating these artefacts with large language models (LLMs) for reasoning and interactive querying.

Human Intelligence (HI) excels at combining basic skills to solve complex tasks. This capability is vital for Artificial Intelligence (AI) and should be embedded in comprehensive AI Agents, enabling them to harness expert models for complex task-solving towards Artificial General Intelligence (AGI). Large Language Models (LLMs) show promising learning and reasoning abilities, and can effectively use external models, tools, plugins, or APIs to tackle complex problems. In this work, we introduce OpenAGI, an open-source AGI research and development platform designed for solving multi-step, real-world tasks. Specifically, OpenAGI uses a dual strategy, integrating standard benchmark tasks for benchmarking and evaluation, and open-ended tasks including more expandable models, tools, plugins, or APIs for creative problem-solving. Tasks are presented as natural language queries to the LLM, which then selects and executes appropriate models. We also propose a Reinforcement Learning from Task Feedback (RLTF) mechanism that uses task results to improve the LLM's task-solving ability, which creates a self-improving AI feedback loop. While we acknowledge that AGI is a broad and multifaceted research challenge with no singularly defined solution path, the integration of LLMs with domain-specific expert models, inspired by mirroring the blend of general and specialized intelligence in humans, offers a promising approach towards AGI.

On an ordinary weekday in November, Bletchley Park plays host to a mixture of elderly pensioners and bands of unruly schoolchildren, visiting to learn about the codebreakers--including computing pioneer Alan Turing--who were based here during World War II, and helped the Allied Forces defeat the Nazis. But this is no ordinary week, and these are no ordinary visitors. On Wednesday and Thursday, delegates from 27 governments around the world, as well as the heads of top artificial intelligence companies, gathered for the world's first AI Safety Summit at this former stately home near London, now a museum. The high-profile event, hosted by the Rishi Sunak-led U.K. government, caps a year of intense escalation in global discussions about AI safety, following the launch of ChatGPT nearly a year ago. The chatbot displayed for the first time--to many users at least--the powerful general capabilities of the latest generation of AI systems.

Brave joins the growing list of browsers that come with built-in generative AI assistants. The open source browser developer has started rolling out an update for Brave on desktop, which gives users access to its AI assistant Leo. Brave introduced Leo through its Nightly experimental channel back in August and has been testing it ever since. The assistant is based on the Llama 2 large language model, which Microsoft and Meta had developed together for commercial and research purposes. Like other AI assistants, users can ask Leo to do various tasks, such as creating summaries of web pages and videos, translating and/or rewriting pages and even generating new content.

Yesterday, 28 countries including the US, members of the EU, and China signed a declaration warning that artificial intelligence is advancing with such speed and uncertainty that it could cause "serious, even catastrophic, harm." The declaration, announced at the AI Safety Summit organized by the British government and held at the historic World War II code-breaking site, Bletchley Park, also calls for international collaboration to define and explore the risks from the development of more powerful AI models, including large language models such as those powering chatbots like ChatGPT. "This is a landmark achievement that sees the world's greatest AI powers agree on the urgency behind understanding the risks of AI--helping ensure the long-term future of our children and grandchildren," the UK prime minister, Rishi Sunak, said in a statement. The venue for the Summit paid homage to Alan Turing, the British mathematician who did foundational work on both computing and AI, and who helped the Allies break Nazi codes during the Second World War by developing early computing devices. The AI hype-train has a knack for turning even close allies into competitors, though.

Public chatbots powered by large language models, or LLMs, emerged just in the last year, and the field of LLM cybersecurity is in its early stages. But researchers have already found these models vulnerable to a type of attack called "prompt injection," where bad actors sneakily present the model with commands. In some examples, attackers hide prompts inside webpages the chatbot later reads, tricking the chatbot into downloading malware, helping with financial fraud or repeating dangerous misinformation.

Alice Globus, head of Nanotronics, said AI could minimize the damage done by recent malware attacks on hospitals and the Colonial Pipeline shutdown in 2021. The CEO of one of the most popular artificial intelligence platforms is warning that AI systems could eventually be capable of "superhuman persuasion." "I expect AI to be capable of superhuman persuasion well before it is superhuman at general intelligence," Sam Altman, CEO of OpenAI, the company behind the popular ChatGPT platform, said on social media earlier this month. He added that such capabilities could "lead to some very strange outcomes." Altman's comments come as fears over what rapidly developing AI technology might eventually be capable of have continued to grow, with some speculating that the technology might surpass the cognitive functions of humans.

China has agreed to work with the United States, European Union and other countries to collectively manage the risk from artificial intelligence at a British summit on Wednesday aimed at charting a safe way forward for the rapidly evolving technology. Some tech executives and political leaders have warned that the rapid development of AI poses an existential threat to the world if not controlled, sparking a race by governments and international institutions to design safeguards and regulations. In a first for Western efforts to manage its safe development, a Chinese vice minister joined U.S. and EU leaders and tech bosses such as Elon Musk and ChatGPT's Sam Altman at Bletchley Park, home of Britain's World War Two code-breakers.