Workshop Agenda
Tuesday, December 9, 2025 - all times are in HST (UTC-10)
8:15 am – 8:30 am: Gathering
8:30 am – 8:45 am: Welcome and Introduction
Overview of workshop goals, format, and expectations for interactive discussion
8:45 am – 10:15 am: Session 1: Experiment Design and Reproducibility Foundations
Taming the Chaos: Managing Reproducibility in Experiments and Results
Sebastian Neef (TU-Berlin)
Abstract & Speakers, LASER Abstract & Bio, ACSAC Paper, Slides
TS;DR (too spammy; didn’t read) - Challenges when disclosing vulnerabilities to small- and medium-sized businesses
Anne Hennig (KIT)
Abstract & Speakers, LASER Abstract & Bios, ACSAC Paper, Slides
EM-Flow: Advanced Electromagnetic Control Flow Verification for Embedded Systems
Carson Stillman (U. Florida)
Abstract & Speakers, LASER Abstract & Bios, ACSAC Paper, Slides
10:15 am – 10:30 am: Break
10:30 am – 12:00 pm: Session 2: System-Level Experimentation and Validation
Zeus-IoT: Experimental Evaluation of Integrity Enforcement on OpenWrt
Alireza Roshandel (Boston University)
Abstract & Speakers, LASER Abstract & Bio, ACSAC Paper, Slides
Experimental Aspects of Evaluation of LLM Responses to End-user Security Questions
Vijay Prakash (NYU)
Abstract & Speakers, LASER Abstract & Bio, ACSAC Paper, Slides
12:00 pm – 1:30 pm: Lunch
1:30 pm – 3:00 pm: Invited Talks
Invited Talk: Supporting Artifact Evaluation – LLMs to the Rescue?
Jan Pennekamp (RWTH Aachen University)
Invited Talk: The SPHERE Research Infrastructure and its Support for Artifact Evaluation
David Balenson (USC-ISI)
3:00 pm – 3:15 pm: Break
3:15 pm – 4:45 pm: Session 3: Experimental Frameworks for AI and Security Tools
Securing LLM-Generated Code: Experimental Insights from APILOT
Weiheng Bai (University of Minnesota)
Abstract & Speakers, LASER Abstract & Bio, ACSAC Paper, Slides
Siren: A Learning-Based Multi-Turn Attack Framework for Simulating Real-World Human Jailbreak Behaviors
Yi Zhao (The Hong Kong Polytechnic University)
Abstract & Speakers, LASER Abstract & Bio, ACSAC Paper, Slides
4:45 pm – 5:00 pm: Closing Remarks
5:00 pm: Adjourn
Abstracts and Speakers
Taming the Chaos: Managing Reproducibility in Experiments and Results
Presenter: Sebastian Neef (TU-Berlin)
Abstract: Empirical research often involves complex experiments and data analysis pipelines: preprocessing datasets, running experiments, applying analysis & evaluation scripts. Naturally, it is hard to cover all edge-cases on first try, so iterative refinement and execution will lead to multiple script versions and intermediate datasets, potentially creating chaos and threatening reproducibility. In this talk, we will first present our approach of managing this chaos with a special naming convention and dockerization, followed by a discussion about alternative approaches (i.e. virtualization, Jupyter-Notebooks, etc.). The aim is to collaboratively come up with practical recommendations for future work to improve the understanding and reproducibility of datasets, analysis scripts and results (by others).
Bio: Sebastian Neef (Technische University Berlin) is a PhD candidate at the chair for Security in Telecommunications (SecT) led by Prof. Dr. Jean-Pierre Seifert at the Technische Universität Berlin, Germany. He holds a Master's degree in Computer Science and his research interests are internet and software security with a strong focus on web security and large-scale vulnerability analysis.
TS;DR (too spammy; didn’t read) - Challenges when disclosing vulnerabilities to small- and medium-sized businesses
Presenter: Anne Hennig (Karlsruhe Institute of Technology)
Abstract: Vulnerabilities in IT systems are known to cause considerable damage. Attacks can be unspecific, but most of them exploit existing vulnerabilities in a targeted manner. While it is relatively easy - not only for attackers but also for security researchers - to locate vulnerabilities in IT systems, reporting vulnerabilities has proven to be no easy task. In our paper „Fix it - If you can! Towards Understanding the Impact of Tool Support and Domain Owners’ Reactions to SSHFP Misconfigurations“ we sent notifications to n=518 domain owners with misconfigured SSHFP records to make domain owners aware about the potential security risk. Within this experiment, we investigated the effect of providing additional tool support, finding that tool support might increase remediation when the sender of the notification is different than the institution providing the tool. Furthermore, it also became clear that simply measuring remediation rates cannot be an indicator of the success of a notification campaign. While analyzing responses we received from domain owners, we found that individual challenges faced by domain owners should be taken into account when determining the success of a notification campaign.
As part of another research project, where we notified website owners about unauthorized third-party redirects on their websites, we investigated the effects of different senders (researcher, Federal CERT Germany, hosting provider) and framings, finding that both had no effect on the remediation behavior. In follow-up interviews with 42 website owners who did not remediate their websites, we found that while our notifications were delivered, many interviewees admitted they either overlooked or dismissed them as spam. This pattern persisted across different senders and framings, highlighting a critical challenge for future notification campaigns.
Both notification experiments revealed significant challenges in terms of establishing contact. When considering related work, we further found that there are no one-size fits all recommendations for vulnerability notification campaigns. It seems that each campaign needs individual considerations. However, some recommendations for notification campaigns can be deduced from our own and related work. By presenting these recommendations, we hope to provide security researchers and practitioners with effective strategies for their notification campaigns, and encourage the security community to report vulnerabilities they identified.
Bios: Anne Hennig is a research assistant at the Karlsruhe Institute of Technology (KIT) in the SECUSO Research Group of Prof. Dr. Melanie Volkamer. She holds a Master’s degree in Science Communication and focused her research on how to communicate security and privacy issues to the public.
Sebastian Neef is a PhD candidate at the chair for Security in Telecommunications (SecT) led by Prof. Dr. Jean-Pierre Seifert at the Technische Universität Berlin, Germany. He holds a Master's degree in Computer Science and his research interests are internet and software security with a strong focus on web security and large-scale vulnerability analysis.
Peter Mayer is Associate Professor at the Department of Mathematics and Computer Science at the University of Southern Denmark (SDU) and a KASTEL Security Research Labs Fellow at Karlsruhe Institute of Technology (KIT). His research focuses on "End-user Viable Information Security & Privacy Solutions". Thereby, independently of whether the end-user of a security solution is a lay-person, an admin, or a developer, the focus always lies on making security & privacy solutions viable for the target audience, i.e., taking into consideration their specific needs and skill sets.
EM-Flow: Advanced Electromagnetic Control Flow Verification for Embedded Systems
Presenter: Carson Stillman (University of Florida)
Abstract: Embedded devices play a major role in supporting critical infrastructure, but lack many of the security protections of sophisticated systems. Determining whether these devices are compromised is therefore a challenge. In this work, we describe a novel control flow verification methodology via electromagnetic (EM) emanations, by combining side channel disassembly solutions with control flow verification. Current disassembler solutions are unable to accurately track the control flow of our target programs because the single-run inference accuracy is too low. In our methodology, we overcome this limitation by introducing basic block detection. We track the execution of six target programs using an EM probe, amplifier, and oscilloscope for sampling, employing a feature selection methodology using the continuous wavelet transform and multilayer perceptron networks to identify expected basic blocks in succession. By combining the decisions of multiple basic block classifiers, we can reliably spot deviations in program behavior, with a 96.5% detection accuracy across all tested programs, and attacks. Finally, we show the transferability of our methodology to different instances of our evaluated processors, using closed-set domain adaptation, reaching up to 98.7% convergence with our baseline models. Through this process we learned how combining disassembly techniques with control flow verification can be used to detect subtle attacks, including three and five clock cycle insertions, where others fail.
Bios: Carson Stillman (University of Florida) is currently pursuing his Ph.D. under the supervision of Dr. Kevin Butler, and Dr. Sara Rampazzi at the University of Florida. His work focuses on the security of embedded and cyber-physical systems through side channel analysis.
Kevin Butler (University of Florida) is a professor of Computer and Information Science and Engineering and director of the Florida Institute for Cybersecurity Research at the University of Florida, Gainesville, FL. Butler’s research interests include the security and trustworthiness of systems and the security of users accessing them.
Zeus-IoT: Experimental Evaluation of Integrity Enforcement on OpenWrt
Presenter: Alireza Roshandel (Boston University)
Abstract: IoT devices have static software packages and do not serve as general-purpose machines, which makes Integrity Measurement Architecture (IMA)-style integrity checks feasible. However, IMA measures only native binaries and fails to detect interpreter-loaded scripts that bypass its mmap hooks. Zeus-IoT closes this gap by extending allowlist verification to interpreter-loaded scripts. We integrated Zeus-IoT into OpenWrt (a widely used aftermarket IoT OS) and evaluated it in QEMU and on commodity routers. In this talk, we focus on the design of our evaluation, describing how each experiment was structured, which metrics were chosen, and why they matter. Our measurements capture both security coverage and practical deployability, including malware prevention effectiveness, runtime overhead, and device-level performance under realistic workloads. We also discuss the challenges of deploying Zeus-IoT on real hardware and share key lessons learned throughout the development and evaluation process.
Bio: Alireza Roshandel is a second-year PhD student in the Department of Electrical and Computer Engineering at Boston University and a member of SecLaBU. His research focuses on system and software security.
Experimental Aspects of Evaluation of LLM Responses to End-user Security Questions
Presenter: Vijay Prakash (NYU)
Abstract: Large language models (LLMs) demonstrate broad applicability across domains, yet a systematic understanding of their knowledge and limitations in specialized fields such as computer security remains limited. This work presents a qualitative evaluation of LLM responses to open-ended, end-user security questions. We describe our methodology for collecting and categorizing questions across diverse security topics, organizing them by knowledge category and thematic focus, and generating corresponding model outputs. We further detail the development of evaluation criteria and the procedures employed in the assessment. The results of this study provide an analysis of LLM performance, with particular attention to recurring errors and limitations, thereby offering insights into the reliability and boundaries of current models in the context of computer security.
Bio: Vijay Prakash is a fifth-year PhD candidate at NYU, advised by Prof. Danny Yuxing Huang and affiliated with the Center for Cybersecurity. His research focuses on responsible AI, particularly on identifying and reducing end-user security, privacy, and safety risks in large language models through expert-in-the-loop empirical evaluations.
Invited Talk: Supporting Artifact Evaluation – LLMs to the Rescue?
Speaker: Jan Pennekamp (RWTH Aachen University)
Abstract: Peer reviews and artifact evaluation are essential for ensuring the transparency and reliability of research, closing the gap between exploratory work and real-world deployment. However, checking for reproducibility is a time-consuming task. The growing number of paper and artifact submissions further complicates the situation. In this talk, we will explore whether large language models (LLMs) can take over tedious and repetitive tasks of artifact evaluation to unburden reviewers. Specifically, we will consider (i) text-based reproducibility rating, (ii) autonomous sandboxed execution environment preparation, and (iii) assessment of methodological pitfalls. These aspects could help in jointly preparing submissions for human expert reviews, thereby streamlining the peer review.
About the Speaker: Dr. Jan Pennekamp is a postdoctoral researcher with a strong interdisciplinary background at RWTH Aachen University, Germany. His primary research interests include security and privacy aspects in the IIoT, healthcare, and supply chains as well as privacy-enhancing technologies, the design of privacy-preserving protocols, secure computations, and their applications. Jan is further interested in best practices in research.
Invited Talk: The SPHERE Research Infrastructure and its Support for Artifact Evaluation
Speaker: David Balenson (USC-ISI)
Abstract: The Security and Privacy Heterogeneous Environment for Reproducible Experimentation (SPHERE) is a global research infrastructure funded by the NSF Mid-Scale RI program to advance reproducible cybersecurity and privacy research. Developed by USC Information Sciences Institute, Northeastern University, and the University of Utah, SPHERE provides diverse enclaves and user portals for rigorous, transparent experimentation across computing, machine learning, IoT, cyber-physical, embedded, and programmable networking domains. Its unified architecture supports researchers, educators, and artifact evaluation committees through reproducibility services, artifact packaging and verification tools, and libraries of representative experimentation environments (REEs). A dedicated Artifact Evaluation Committee (AEC) portal enables authors to share and reviewers to assess artifacts in controlled, repeatable settings. Now in its third year, SPHERE is expanding beta access, integrating new enclaves, and engaging the community to port and host mature artifacts—laying the foundation for sustainable, open, and verifiable cybersecurity and privacy research.
About the Speaker: David Balenson is Interim Director of the Networking and Cybersecurity Division at the USC Information Sciences Institute and Community Outreach Director for the NSF-funded SPHERE Research Infrastructure. He leads efforts to promote SPHERE and advance reproducible cybersecurity and privacy research. His interests include cybersecurity for critical infrastructure and cyber-physical systems such as automotive and autonomous vehicles, as well as experimentation, technology transition, and multidisciplinary collaboration. He holds B.S. and M.S. degrees in Computer Science from the University of Maryland.
Securing LLM-Generated Code: Experimental Insights from APILOT
Presenter: Weiheng Bai (University of Minnesota)
Abstract: We conduct a large-scale empirical evaluation of APILOT, a lightweight system that mitigates outdated API recommendations in large language model (LLM) generated code. Our experiments span 17 major LLMs, including GPT-4-Turbo, CodeLlama, and Llama-3.1-70B and tested on a benchmark of 723 outdated APIs drawn from 15 high-impact Python packages. Using metrics that jointly assess security (FAPI reduction), usability (ExecRate), and functionality (ICE-Score), we show that APILOT reduces outdated API recommendations by 75% on average, with some models achieving complete mitigation, while improving executability by 35–38% without degrading functionality. Further ablation studies reveal that a similarity threshold of 0.4 and five iterative refinements strike the best balance between detection accuracy and runtime cost. Together, these results demonstrate that APILOT provides a scalable, model-agnostic, and version-aware solution to enhance the security and usability of LLM-assisted programming.
Bio: Weiheng Bai is a Ph.D. candidate in Computer Science at the University of Minnesota, advised by Prof. Kangjie Lu, and an Applied Scientist at Amazon. His research focuses on AI security, LLM safety, and system security, with an emphasis on building practical defenses for real-world AI systems. He is the lead author of APILOT, accepted at ACSAC 2025, which introduces a scalable framework for mitigating outdated API risks in LLM-generated code. His broader work spans dataset compliance, program analysis, and secure code generation, with publications at top venues such as USENIX Security, NDSS, and ACSAC.
Siren: A Learning-Based Multi-Turn Attack Framework for Simulating Real-World Human Jailbreak Behaviors
Presenter: Yi Zhao (The Hong Kong Polytechnic University)
Abstract: Our presentation will discuss the experimental design of our learning paradigm, which addresses the fact that effective multi-turn red-teaming has largely been a manual process. We will also highlight a critical challenge we discovered during our experiments: highly safety-aligned LLMs refuse to "play the role" of an attacker, which complicates automation. We will detail how our specific learning-based approach overcomes this obstacle.
Bio: Yi Zhao is a PhD student in the Department of Computing at The Hong Kong Polytechnic University, focusing on natural language processing, LLM safety, and visual question answering.