The performance of Machine Learning (ML) and Deep Learning (DL)-based Intrusion Detection and Prevention Systems (IDS/IPS) is critically dependent on the relevance and quality of the datasets used for training and evaluation. However, current AI model evaluation practices for developing IDS/IPS focus predominantly on accuracy metrics, often overlooking whether datasets represent industry-specific threats. To address this gap, we introduce a novel multi-dimensional framework that integrates the MITRE ATT&CK knowledge base for threat intelligence and employs five complementary metrics that together provide a comprehensive assessment of dataset suitability. Methodologically, this framework combines threat intelligence, natural language processing, and quantitative analysis to assess the suitability of datasets for specific industry contexts. Applying this framework to nine publicly available IDS/IPS datasets reveals significant gaps in threat coverage, particularly in the healthcare, energy, and financial sectors. In particular, recent datasets (e.g., CIC-IoMT, CIC-UNSW-NB15) align better with sector-specific threats, whereas others, like CICIoV-24, underperform despite their recency. Our findings provide a standardized, interpretable approach for selecting datasets aligned with sector-specific operational requirements, ultimately enhancing the real-world effectiveness of AI-driven IDS/IPS deployments. The efficiency and practicality of the framework are validated through deployment in a real-world case study, underscoring its capacity to inform dataset selection and enhance the effectiveness of AI-driven IDS/IPS in operational environments.

We identify ten pitfalls as don'ts of machine learning in security and propose dos as actionable recommendations to support researchers in avoiding the pitfalls where possible. Furthermore, we identify open problems that cannot be mitigated easily and require further research effort (§2).

During an interview in 2017, Andrew Ng--one of the most renowned computer scientists in the field of artificial intelligence (AI)--was reported to say: "Just as electricity transformed almost everything 100 years ago, today I actually have a hard time thinking of an industry that I don't think AI will transform in the next several years." Indeed, over the last decade, we have observed a rebirth of interest in AI and, more specifically, in its machine learning (ML) subfield, which is aimed at designing algorithms that learn from examples. This has been fueled by the availability of large volumes of data over the Internet, the increased computing power of today's hardware and cloud infrastructures, and the algorithmic improvements in deep learning and neural networks, which have shown tremendous progress in dealing with text, audio, image, and video data. Their success has been even reinforced more recently by the advent of foundational and generative AI models that can generate realistic text, images, and videos with impressive quality. For these reasons, AI and ML have been fostering important advancements in healthcare, automotive, robotics, recommendation systems, chatbots, and many other applications.

In this paper, we introduce SecQA, a novel dataset tailored for evaluating the performance of Large Language Models (LLMs) in the domain of computer security. Utilizing multiple-choice questions generated by GPT-4 based on the "Computer Systems Security: Planning for Success" textbook, SecQA aims to assess LLMs' understanding and application of security principles. We detail the structure and intent of SecQA, which includes two versions of increasing complexity, to provide a concise evaluation across various difficulty levels. Additionally, we present an extensive evaluation of prominent LLMs, including GPT-3.5-Turbo, GPT-4, Llama-2, Vicuna, Mistral, and Zephyr models, using both 0-shot and 5-shot learning settings. Our results, encapsulated in the SecQA v1 and v2 datasets, highlight the varying capabilities and limitations of these models in the computer security context. This study not only offers insights into the current state of LLMs in understanding security-related content but also establishes SecQA as a benchmark for future advancements in this critical research area.

Will artificial intelligence become clever enough to upend computer security? AI is already surprising the world of art by producing masterpieces in any style on demand. If AIs can act like a bard while delivering the comprehensive power of the best search engines, why can't they shatter security protocols, too? The answers are complex, rapidly evolving, and still murky. AI makes some parts of defending computers against attack easier.

Artificial intelligence (AI) is on everyone’s mind: autopilot for Teslas, deepfakes for fun or fake news, DALL·E 2 for the untalented artist, big data and machine learning for data analysis. And CERN is no exception. While all those AI applications come with pros and cons, with benefits and risks, AI systems deserve their own consideration when it comes to computer security. So we took the opportunity to ask a renowned expert in AI, ChatGPT, how best to secure AI systems. Here’s the answer we got: “Artificial intelligence (AI) is a rapidly advancing technology that has the potential to greatly benefit society in a variety of ways. However, it also poses new security risks that must be addressed. Here are some best practices for ensuring the security of AI systems: Secure data inputs and outputs. AI systems rely on large amounts of data to learn and make decisions. This data must be protected from unauthorised access and tampering to ensure the integrity and accuracy of the system. Use secure algorithms and protocols. AI systems use complex algorithms and protocols to process data and make decisions. It is important to use secure algorithms that have been thoroughly vetted and tested, and to ensure that all communication between the AI system and other systems is encrypted. Monitor and audit AI systems. Regular monitoring and auditing of AI systems can help detect and prevent security breaches. This includes monitoring for unusual or suspicious behavior, as well as auditing the system’s log files for any signs of unauthorised access or tampering. Implement access controls. Access controls can help prevent unauthorised access to AI systems and the data they process. This includes setting up user accounts and assigning roles and permissions, as well as implementing authentication mechanisms such as biometrics or multi-factor authentication. Stay informed and educated. As AI technology continues to evolve, it is important to stay informed about the latest security risks and best practices. This includes attending relevant conferences and workshops, reading industry publications and staying up-to-date with software updates and patches. Be prepared for the worst. AI systems can be vulnerable to cyberattacks, and it is important to have a plan in place to respond to a security incident. This includes having a trained incident response team, and regularly testing incident response plans to ensure they are effective. By following these best practices, organisations can help ensure the security of their AI systems and protect themselves from potential cyber threats. As AI is permeating all industries, it is crucial for everyone to be aware of the security risks and best practices to mitigate them.” Thanks a lot, ChatGPT bot. So far, nothing special here for AI – it’s a matter of applying the same general security paradigms that every computer user and every service manager should apply to any well-managed computing service (like those run by CERN IT) and individual computing device (like your smartphone or laptop): Make sure that you only input valid and trusted data, URLs, etc. Use encryption to protect your privacy Have intrusion detection and/or an anti-malware program running in the background Deploy access control and opt into 2FA security Train yourself and have auto-update enabled Have an incident response plan. Six simple steps to make your computing service more secure, and to protect your personal assets stored on your laptop and smartphone. Six simple steps, common sense for security-aware humans. Six simple steps, apparently also rather logical for an AI bot. So, tell us: how human is AI; how AI are you? ______ Do you want to learn more about computer security incidents and issues at CERN? Follow our Monthly Report. For further information, questions or help, check our website or contact us at Computer.Security@cern.ch.

AI systems rely on large amounts of data to learn and make decisions. This data must be protected from unauthorised access and tampering to ensure the integrity and accuracy of the system. Use secure algorithms and protocols. AI systems use complex algorithms and protocols to process data and make decisions. It is important to use secure algorithms that have been thoroughly vetted and tested, and to ensure that all communication between the AI system and other systems is encrypted.

We are seeking outstanding candidates with strong analytical and problem solving skills, who are strong in written and oral communication (in English), and have documented experience in the development of complex compute systems. The applicant should have provable skills in the state-of-the-art web-development frameworks, virtualization techniques as well as database technologies. Expertise in clinical data science and machine learning, as well as computer security and data privacy are welcome. A large roadblock of medical research is the difficult access to sensitive data which therefore hinders the training of complex and powerful machine learning concepts. This issue is amplified when considering rare diseases with low incidence numbers per hospital.

Public concern about computer ethics and worry about the social impacts of computing has fomented the "techlash." Newspaper headlines describe company data scandals and breaches; the ways that communication platforms promote social division and radicalization; government surveillance using systems developed by private industry; machine learning algorithms that reify entrenched racism, sexism, cisnormativity, ablism, and homophobia; and mounting concerns about the environmental impact of computing resources. How can we change the field of computing so that ethics is as central a concern as growth, efficiency, and innovation? There is no one intervention to change an entire field: instead, broad change will take a combination of guidelines, governance, and advocacy. None is easy and each raises complex questions, but each approach represents a tool for building an ethical culture of computing.

Helping companies make sense of their data. From real-time cybercrime mapping to penetration testing, machine learning has become a crucial part of cybersecurity. Fortunately, machine learning can help solve the most common tasks, including pattern detection, prediction, regression, and classification. In an era of large amounts of data and a shortage of network security talents, machine learning seems to be an alternative to solve many problems. Indeed, through machine learning, when applied to computer security, we can sort through millions of files to discover threats.