Workshop Agenda
Tuesday, December 10, 2024 - all times are in HST (UTC-10)
8:15 am – 8:30 am: Gathering
8:30 am – 8:45 am: Welcome, Goals, and Agenda
8:45 am – 10:15 am: Session 1: Paper Discussion
VaktBLE: Evaluating a Wireless Man-in-the-Middle Bridge against Malevolent BLE Connections
Matheus Eduardo Garbelini and Sudipta Chattopadhyay (Singapore University of Technology and Design)
Abstract & Speakers, LASER Abstract & Bios, ACSAC Paper, Slides
Experimental Approaches for Robust Authentication Based on Hardware Impairments
Valeria Loscri and Ildi Alla (Inria Lille- Nord Europe Lille)
Abstract & Speakers, LASER Abstract & Bios, ACSAC Paper, Slides
Electromagnetic Interference Experiments - Design and Best Practices
Marcell Szakály and Sebastian Köhler (University of Oxford)
Abstract & Speakers, LASER Abstract & Bios, ACSAC Paper, Slides
10:15 am – 10:45 am: Break
10:45 am – 12:15 pm: Session 2: Paper Discussion
Evaluating Machine Learning-Based Binary Vulnerability Detectors: Lessons From the Trenches
Sima Arasteh (University of Southern California)
Abstract & Speakers, LASER Abstract & Bios, ACSAC Paper, Slides
Evaluating Approaches for Identifying Cryptographic Functions in Binaries
Yongming Fan (Purdue University)
Abstract & Speakers, LASER Abstract & Bios, ACSAC Paper, Slides
Cybersecurity Research in 3D Bioprinting: Experiments and Evaluation Challenges and Future Directions
Muhammad Ahsan (Virginia Commonwealth University)
Abstract & Speakers, LASER Abstract & Bios, ACSAC Paper, Slides
12:15 pm – 1:45 pm: Lunch
1:45 pm – 3:00 pm: Session 3: Keynote Talk
Keynote: Well, It Worked on My Computer: Reproducibility in Computer Security Research
Patrick Traynor (University of Florida)
3:00 pm – 3:30 pm: Break
3:30 pm – 5:00 pm: Session 4: Paper Discussion
From Groundhog Day to Ground Truth Data: We Need Curated Datasets to Spur Innovation
Anurag Swarnim Yadav and Joseph N. Wilson (University of Florida)
Abstract & Speakers, LASER Abstract & Bios, ACSAC Paper, Slides
Experimental Aspects of Investigating the Failures and Shortcomings of Security Countermeasures
Md Rayhanur Rahman (The University of Alabama)
Abstract & Speakers, LASER Abstract & Bios, ACSAC Paper, Slides
Benchmarking Large Language Models: From General Purpose to Cybersecurity Applications
Dipkamal Bhusal (Rochester Institute of Technology)
Abstract & Speakers, LASER Abstract & Bios, ACSAC Paper, Slides
5:00 pm – 5:15 pm: Wrap-Up
5:15 pm: Adjourn
Abstracts and Speakers
VaktBLE: Evaluating a Wireless Man-in-the-Middle Bridge against Malevolent BLE Connections
Presenters: Matheus Eduardo Garbelini and Sudipta Chattopadhyay (Singapore University of Technology and Design)
Abstract: We conceptualize, design, and evaluate VaktBLE, a novel framework to defend Bluetooth Low Energy (BLE) peripherals against over-the-air attacks in real-time. To this end, we implement an extensible framework that can systematically track, analyze, and block BLE communication down to the link layer, thus allowing us to defend commercial off-the-shelf devices against complex BLE attacks without requiring the use of binary patching frameworks. During the evaluation of VaktBLE, we measured its performance, capability, and robustness against several over-the-air threats such as denial-of-service, security bypass, and adaptative attacks. Our results highlight the extensibility of VaktBLE to counteract protocol-level attacks and rogue peripherals while not incurring false positives when attacks are randomly mixed with benign connection attempts. This talk discussed the experiments conducted to evaluate our work against many state-of-the-art BLE offensive tools, as well as the challenges faced during evaluation setup and comparison with related works.
Bios: Born in Brazil, Matheus Eduardo Garbelini is currently a Research Fellow at Singapore University of Technology and Design (SUTD) and member of ASSET Research Group. He has a background in Electronics Engineering and loves to mess with low-level wireless software and hardware. Matheus is known for the discovery of a collection of wireless vulnerabilities such as Sweyntooth, Braktooth, and 5Ghoul.
Sudipta Chattopadhyay is an Associate Professor with the Information Systems Technology and Design Pillar, Singapore University of Technology and Design, Singapore. His general research interests lie in the broad area of cyber security including but not limited to safety & security for AI, Wireless Technologies, and Internet of Things (IoTs). Together with his student, he discovered SweynTooth, BrakTooth and 5Ghoul, families of Bluetooth and 5G NR vulnerabilities that affect billions of devices worldwide. His research has been featured in Channel News Asia, WIRED, PC Magazine, Hacker News, among others. His discovery has also generated cyber security alerts from government regulatory agencies including CSA (Singapore), DHS, and FDA. He is an Associate Editor of ACM Computing Surveys. He aims to target research topics that not only hold significant academic value, but also shows solid pathway for technology translation or high-impact (open-source) software tools.
Experimental Approaches for Robust Authentication Based on Hardware Impairments
Presenters: Valeria Loscri and Ildi Alla (Inria Lille- Nord Europe Lille)
Abstract: Physical Layer Authentication is arising a lot of interest in the context of IoT devices, to detect and identify intruders with low overhead approaches. Specific hardware impairments such as Carrier Frequency Offset (CFO), Direct Current Offset (DCO) and Phase Offset (PO) can be considered to create unique signatures for device identification. Anyway, such impairments can be subject to variability due to external environmental interactions, or different setting. In order to characterize these aspects, an experimental approach is paramount and also permits to assess the robustness of each impairment alone. This analysis and evaluation are preliminary, to design hybrid approaches, based on combination of different impairments. Reprogrammable devices have been used to build in as much as possible close characteristics, that are more challenging for authentication approaches. Statistical authentication approaches were also considered before diving into Machine Learning (ML) Models. Another set of experiments was also devoted to the features extraction and characterization of ML in different scenarios, in order to characterise the most representative working conditions and potential identification usurpation attacks. The controlled experiments were paramount to optimize and fine-tuning the different proposed approaches. Key points in our evaluations were not only the performance in terms of accuracy but also the detection time and the energy efficiency, that we also evaluated.
Bios: Valeria Loscri is a research director in the FUN team at Inria Lille (France), which she joined in 2013. From December 2006 to September 2013, she was a researcher at the TITAN laboratory at the University of Calabria, Italy. She received her MSc and PhD degrees in Computer Science in 2003 and 2007, respectively, from the University of Calabria and her HDR (Habilitation à diriger des recherches) in 2018 from the University of Lille (France). Her research interests include cybersecurity in wireless networks, emerging technologies for new communication paradigms such as Visible Light Communication (VLC), millimeter waves, cooperation and coexistence of heterogeneous wireless devices. She has been involved in several European projects (Horizon Europe MLSysOps, H2020 CyberSANE, European FP7 VITAL project, etc.). She was nominated for the Women Stars in Computer Networking and Communications 2021 by the IEEE Communication Society. She is part of the editorial board of IEEE COMST, Transactions on Information Forensics and Security, Elsevier ComNet, ComCom. She is a member of the TPC in several leading international conferences, such as IEEE ESORICS, IEEE CNS, IEEE INFOCOM, IEEE PerCom. She is the chair and scientific holder of the European BEiNG-WISE COST Action (since 2023). Since 2019, she has been an international scientific delegate of Inria Lille.
Ildi Alla received his Bachelor’s and master’s degrees in electronic engineering from the Polytechnic University of Tirana, Albania. Additionally, he earned a Master of Science from the University of Technology of Compiègne (UTC) | Sorbonne University Association, France, specializing in Automation and Robotics for Intelligent Systems. He is currently pursuing a Ph.D. at Inria Centre at the University of Lille, focusing on the detection and localization of cyber-attacks in wireless networks. His research interests lie at the intersection of wireless networking, signal processing, machine learning, and data fusion to enhance security against malicious activities. His work includes multi-modal sensor fusion for drone detection and device authentication in multi-node networks via RF fingerprinting.
Electromagnetic Interference Experiments - Design and Best Practices
Presenters: Marcell Szakály and Sebastian Köhler (University of Oxford)
Abstract: Electromagnetic Interference (EMI) attacks are an active area of research in the computer security community. By nature, this is a very experimental field, which borders on electrical engineering topics. In this talk, we present details of the experiments we conducted during our research into EMI attacks against power converters, highlighting important takeaways for researchers and reviewers alike in the field. In particular, we discuss hardware best practices to avoid cross talk and unwanted signal propagation, two common pitfalls which can impact measurements. We discuss the experimental design and measurement techniques we used to ensure meaningful results. We also draw attention to negative results, which often go unpublished, and discuss the difficulty of replication studies due to exact hardware requirements. Finally, we invite discussion into these issues.
Bios: Marcell Szakály is a 3rd year PhD student in the Systems Security Lab, Department of Computer Science at the University of Oxford, with a background and Masters degree in Physics. During his PhD, he is working on the physical-layer aspects of cybersecurity, including Electromagnetic Interference against sensors, physical layer attacks against satellite communications, and electric vehicle charging security.
Sebastian Köhler is a Senior Research Associate and Royal Academy of Engineering UK IC Fellow in the Systems Security Lab, Department of Computer Science at the University of Oxford, where he works on and coordinates research activities in wireless and physical-layer security. His work focuses on the security of various systems, ranging from space and satellite systems to autonomous and electric vehicles. For his work during his PhD, which revealed a serious vulnerability in a widely used electric vehicle charging protocol, he was awarded the EPSRC Doctoral Prize and MPLS Early Career Research Impact Award.
Evaluating Machine Learning-Based Binary Vulnerability Detectors: Lessons From the Trenches
Presenter: Sima Arasteh (University of Southern California)
Abstract: In recent years, we have conducted two research studies to evaluate the effectiveness of neural networks in binary vulnerability discovery. Our first approach, Bin2vec, utilizes the control flow graph of a function, whereas the more recent BinHunter captures both control and data flow from code segments associated with vulnerabilities within functions. The primary challenge in building, evaluating, and writing about such machine learning-based techniques is acquiring a sufficiently large and relevant dataset for training and testing. The Juliet test suite, a widely recognized dataset covering over 118 types of vulnerabilities, provides a well-labeled set of examples of buggy and patched code. However, for a vulnerability detection tool to be practical, it must be able to identify vulnerabilities in real-world applications. Here, we used the CVE database to identify historical vulnerabilities in popular open-source programs, and assembled a paired repository of buggy and non-buggy real-world code. Assembling these datasets was surprisingly tricky, and required us to crawl the Debian archive to identify precise points at which specific CVEs had been patched, and use tools like Quilt to rebuild buggy and non-buggy versions of programs in our dataset. Comparing our systems to the baselines, each of which came with their own elaborate setups, added to these complications. I will talk about some of these challenges, and the lessons learned along the way.
Bio: Sima Arasteh is a PhD student at USC with research interests in software security, binary analysis, and hardware security. She focuses on uncovering software vulnerabilities through both static and dynamic analysis techniques.
Evaluating Approaches for Identifying Cryptographic Functions in Binaries
Presenter: Yongming Fan (Purdue University)
Abstract: Cryptographic functions are essential for securing electronic communications, but unfortunately, they are often misused, poorly implemented, or developed hastily. They can also be exploited for malicious purposes, such as hiding payloads in malware, particularly in closed-source binaries, which complicates independent analysis. Detecting these functions in binaries can indicate malicious activity and serve as a focus for cryptographic analysis and vulnerability assessment. A variety of work across industry and academia has focused on developing techniques and tools that are specifically tailored to identifying different cryptographic primitives in binary applications. However, these efforts often lack a comprehensive, standardized testing framework. Re-executing these tools and reproducing their experiments also present significant challenges. Therefore, we conducted reproduction and replication studies, comparing each tool’s design and approach against their original experiments, while developing new benchmarks and metrics to more effectively evaluate these tools through new experiments.
Bio: Yongming Fan is a PhD student in the Department of Computer Science at Purdue University, specializing in applied cryptography. His research centers on bridging the gap between theoretical cryptography and practical security solutions, with a focus on zero-knowledge proofs, software security, and privacy-enhancing technologies. He is dedicated to developing automated security evaluation tools to assess the security of cryptographic protocol implementations, ensuring their resilience against emerging threats and vulnerabilities.
Cybersecurity Research in 3D Bioprinting: Experiments and Evaluation Challenges and Future Directions
Presenter: Muhammad Ahsan (Virginia Commonwealth University)
Abstract: 3D bioprinting, as an application of additive manufacturing, is becoming increasingly established and has a transformative impact on biomanufacturing. Given its use in critical applications such as tissue engineering and regenerative medicine, securing these processes from adversarial threats is essential. However, as a relatively new technology, 3D bioprinting offers unique challenges, including the lack of universally accepted standards for bioink composition and printing optimization. While best practices are emerging, including printability ratios, parameter optimization indexes, and cell viability metrics, no formalized guidelines exist. Likewise, security standards for the manufacturing and operation of 3D bioprinters are also lacking, potentially creating vulnerabilities. An adversary with access to a bioprinter could manipulate printing parameters, potentially compromising process integrity. Utilizing in-situ monitoring systems that gather physical process data offers a promising research direction to detect such manipulations in real time. This talk will delve into the unique challenges of working with bioprinters such as handling live cells, the impact of parameter modifications on cell viability and print structural integrity, and the intricacies of gathering and analyzing sensor data to identify potential sabotage attempts in 3D bioprinting.
Bio: Muhammad Ahsan is a Ph.D. candidate in the Security and Forensics Engineering (SAFE) Lab at the Department of Computer Science, Virginia Commonwealth University. His research focuses on enhancing the security of additive manufacturing processes through side-channel data analysis. His broader research interests encompass Cyber-Physical Systems (CPS) and Industrial IoT security. Ahsan holds both Master’s and Bachelor’s degrees in Electrical Engineering from the University of Engineering and Technology (UET), Lahore, Pakistan.
Keynote: Well, It Worked on My Computer: Reproducibility in Computer Security Research
Presenter: Patrick Traynor (University of Florida)
Abstract: Has your research ever been frustrated by failed attempts to reproduce related work? You’re not alone. In this talk, I’ll discuss the largest ever study of reproducibility in Computer Security Research, investigate the impact of AECs, and point to concrete steps that can help to alleviate this problem.
Bio: Patrick Traynor is the John and Mary Lou Dasburg Preeminent Chair in Engineering at the University of Florida. He earned his Ph.D. from the Pennsylvania State University in 2008. His work covers a wide array of topics, ranging from cellular and mobile security and deepfakes to systems security and reproducibility. He’s helped to co-found three companies based on his research, and is an avid runner.
From Groundhog Day to Ground Truth Data: We Need Curated Datasets to Spur Innovation
Presenters: Anurag Swarnim Yadav and Joseph N. Wilson (University of Florida)
Abstract: A number of recent works have explored the use of Large Language Models (LLMs) to detect and/or repair programs with security vulnerabilities. We have shown that some of these efforts have employed datasets that exhibit defects that lead to overestimates of their capabilities. Over and over (and over) again, we have seen instances of poor datasets being used to produce dubious results in different settings. Despite the defective datasets used in these recent works, some of the publications were assigned data quality certifications under the ACM artifact and review badging process. This points out an important factor not considered in the seminal NAS report Reproducibility and Replicability in Science, namely, the quality of datasets employed in research and experimentation. If one employs the same flawed dataset as described in a publication that accurately reports a machine learning process and its output, one will inevitably produce the same results even if dataset errors render their conclusions inaccurate. The fact that results can be reproduced does not in itself mean they demonstrate a valid hypothesis. We review the experiments we performed that led to our understanding of the critical problems we have encountered in published experiments and discuss the scope of that problem. We explore the experimental procedures needed to develop machine learning systems for vulnerability detection and repair including careful attention to dataset requirements. We point out both the difficulties and benefits of creating valid, high-quality datasets for such systems. Finally, we propose that either a government- or industry-sponsored effort be funded to generate and curate datasets to support innovation in program security vulnerability detection and repair.
Bios: Anurag Swarnim Yadav is a Ph.D. candidate in Computer Science at the University of Florida, where he is supervised by Dr. Joseph N. Wilson. His research specializes in developing innovative techniques for detecting and repairing software vulnerabilities.
Joseph N. Wilson is an Associate Professor and Graduate Program Director at the University of Florida CISE Department. He received his PhD from the Department of Applied Mathematics and Computer Science at the University of Virginia in 1984. Over the many years since then, his research has spanned a number of disciplines including programming languages, image processing and computer vision, machine learning (especially as applied to the detection and remediation of landmines), and information security.
Experimental Aspects of Investigating the Failures and Shortcomings of Security Countermeasures
Presenter: Md Rayhanur Rahman (The University of Alabama)
Abstract: Organizations increasingly recognize cybersecurity as a significant challenge and are investing substantial resources in countermeasures to mitigate cyberattacks. Yet, high-impact incidents, such as the Colonial Pipeline attack that led to gasoline shortages across multiple states, are becoming more frequent. This study hypothesizes that organizations encounter persistent challenges in implementing effective security countermeasures. Using generative AI, we analyze academic and grey literature to identify these challenges. In this workshop, we discuss our methodology, including prompting techniques, qualitative coding, and thematic analysis, as well as the obstacles encountered in our research. Through the workshop discussion, we look forward to identifying future research directions for large-scale user studies on security countermeasure effectiveness, the development of actionable security metrics, and collaborative opportunities.
Bio: Dr. Md Rayhanur Rahman is an Assistant Professor in the Department of Computer Science at The University of Alabama. He earned his Ph.D. in Computer Science from North Carolina State University in Raleigh, NC, USA. His research interests include software engineering and cybersecurity, with recent work focused on extracting actionable intelligence from open-source knowledge bases to help software and security practitioners design proactive defenses. His research utilizes a variety of methods, including natural language processing, machine learning, static analysis, and grounded theory-based approaches.
Benchmarking Large Language Models: From General Purpose to Cybersecurity Applications
Presenter: Dipkamal Bhusal (Rochester Institute of Technology)
Abstract: Large Language Models (LLMs) have demonstrated promising potential in various domains, yet their real-world applications are hindered by challenges such as hallucinations and lack of reliability. This talk will explore how to effectively benchmark LLMs, beginning with the motivations for benchmarking and the insights gained from general-purpose evaluations. We will then shift focus to the design of domain-specific benchmarks tailored to cybersecurity applications. Drawing on our work from SECURE (accepted at ACSAC’24) and CTIBench (accepted at NeurIPS’24), which assess LLMs' performance in Industrial Control Systems (ICS) and Cyber Threat Intelligence (CTI), we will discuss the limitations of existing benchmarks and emphasize the need for specialized evaluations in security contexts. SECURE and CTIBench offer comprehensive frameworks for evaluating LLMs on practical cybersecurity tasks, including knowledge extraction, reasoning, and understanding. We will discuss how to craft such tasks for specific domains and assess state-of-the-art models on these tasks to gain critical insights into their strengths and weaknesses. The talk will also explore how to fine-tune LLMs for specific security tasks, and challenges associated with them.
Bio: Dipkamal Bhusal is a fourth-year PhD student at the Rochester Institute of Technology with primary research in reliable machine learning and explainable machine learning. He obtained a bachelor's in Electronics and Communication Engineering (2016) and a master's in Information Engineering (2021) from Tribhuvan University, Nepal. Before pursuing his PhD, Dipkamal cofounded AI startups in Kathmandu (Nepal) and was engaged in the research, and development of various machine learning products.