Workshop Agenda
Tuesday, December 8, 2026 - all times are in PST (UTC-8)
8:30 am – 9:00 am: Welcome, Introductions, Workshop Goals and Format
9:00 am – 10:15 am: Session 1: Experimental Challenges in Cyber-Physical Systems
9:00 am – 9:30 am
How Much Can We Learn Without Testing to Failure? Experimental Lessons from the FLIR identiFINDER2
Roshane Taylor (Augusta University)
ACSAC Paper Abstract & Speakers, LASER Abstract & Bio, ACSAC Paper, Slides
9:30 am – 10:00 am
From NXP to STM32: Experimental Challenges and Lessons Learned in Building VeCoDI for Secure Edge AI
Mohamed Khalil Kiri (EURECOM, France)
ACSAC Paper Abstract & Speakers, LASER Abstract & Bio, ACSAC Paper, Slides
10:00 am – 10:15 am
Session Discussion
Theme: Hardware and safety constraints, experimental limitations, and lessons learned.
10:15 am – 10:45 am: Morning Break
10:45 am – 12:00 pm: Session 2: Reproducibility and Benchmarking
10:45 am – 11:15 am
Lessons Learned from Benchmarking Automated YARA Rule Generation for In-the-Wild Malware Families
Mabon Ninan (Texas A&M University)
ACSAC Paper Abstract & Speakers, LASER Abstract & Bios, ACSAC Paper, Slides
11:15 am – 11:45 am
Reproduction and Benchmarking Lessons from TTP Extraction Research
Sean Frankum (University of Georgia)
ACSAC Paper Abstract & Speakers, LASER Abstract & Bio, ACSAC Paper, Slides
11:45 am – 12:00 pm
Session Discussion
Theme: Reproducibility, ground truth, dataset quality, and fair benchmarking.
12:00 pm – 1:00 pm: Lunch
1:00 pm – 2:00 pm: Keynote
Jelena Mirkovic
USC Information Sciences Institute
2:00 pm – 3:15 pm: Session 3: Wireless Security Experimentation
2:00 pm – 2:30 pm
Devilray: A Reconfigurable Adversarial Baseline for Revealing Blind Spots in Fake Base Station Detection
Duckwoo Kim (KAIST)
ACSAC Paper Abstract & Speakers, LASER Abstract & Bios, ACSAC Paper, Slides
2:30 pm – 3:00 pm
Behind SfS: Experimental Design and Lessons Learned from a Wireless Replay Protection Testbed
Ala Darabseh (New York University Abu Dhabi)
ACSAC Paper Abstract & Speakers, LASER Abstract & Bios, ACSAC Paper, Slides
3:00 pm – 3:15 pm
Session Discussion
Theme: Wireless testbeds, realistic attacker models, and experimental validation.
3:15 pm – 3:45 pm: Afternoon Break
3:45 pm – 5:00 pm: Session 4: Real-World Network Measurement and Evaluation
3:45 pm – 4:15 pm
From Benchmark to Deployment: How Should We Evaluate Learning-Based IDS?
Bilel Saghrouchni (INSA Lyon / SPIE ICS, France)
ACSAC Paper Abstract & Speakers, LASER Abstract & Bio, ACSAC Paper, Slides
4:15 pm – 4:45 pm
Measurements and Emulation of a Cross-layer Attack on Bitcoin and the Lightning Network
Charmaine Ndolo (Dresden University of Technology)
ACSAC Paper Abstract & Speakers, LASER Abstract & Bio, ACSAC Paper, Slides
4:45 pm – 5:00 pm
Session Discussion
Theme: Network measurement, emulation, and operational realism.
5:00 pm – 5:25 pm: Workshop Wrap-Up
Closing Remarks and Round-Robin Discussion
5:25 pm – 5:30 pm: Group Photo
5:30 pm: Adjourn
Abstracts and Speakers
How Much Can We Learn Without Testing to Failure? Experimental Lessons from the FLIR identiFINDER2
Presenter: Roshane Taylor, School of Computer and Cyber Sciences, Augusta University
Abstract: Assessing safety-relevant embedded systems requires balancing meaningful security validation against the possibility that testing itself may disrupt the device. This presentation revisits our controlled security assessment of the FLIR identiFINDER2, focusing on the experimental decisions, challenges, negative results, and lessons learned rather than repeating the technical findings of our ACSAC paper.
We discuss our isolated cybersecurity testbed, separate authorized Cs-137 check-source validation, and controlled experiments demonstrating configuration-dependent changes in operator-facing classification and alarm behavior. We also examine the residual visual warning that remained active, observed firmware-update signature checks, unexpected device instability, and limitations in experimental evidence collection and reproducibility.
These experiences raise broader methodological questions: How much evidence is sufficient to support a security finding? When does additional controlled validation introduce more risk than scientific value? How can researchers enable meaningful independent verification while protecting sensitive experimental artifacts?
Through an interactive discussion, we invite participants to examine evidence sufficiency, experimental stopping criteria, reproducibility, and safety-aware security assessment practices for cyber-physical systems.
Bio: Roshane Taylor is a Ph.D. student and Graduate Researcher in Computer and Cyber Sciences at Augusta University and an Adjunct Assistant Professor of Cyber Operations at the University of Maryland Global Campus. A retired U.S. Army Cyber Operations Officer, his research focuses on cybersecurity assessment of safety-relevant cyber-physical systems, medical, industrial, and operational technologies. His work emphasizes evidence-driven vulnerability validation, configuration integrity, operational resilience, and safety-aware experimentation. He is the presenting author of an accepted ACSAC 2026 paper examining the security of the FLIR identiFINDER2.
From NXP to STM32: Experimental Challenges and Lessons Learned in Building VeCoDI for Secure Edge AI
Presenter: Mohamed Khalil Kiri, EURECOM, Sophia Antipolis, France
Abstract: Deploying confidential and verifiable deep neural network (DNN) inference on resource-constrained microcontrollers presents practical challenges that go beyond security design. This talk shares the experimental experience of developing VeCoDI, a framework for secure edge AI based on ARM TrustZone-M.
The initial implementation plan targeted an NXP platform combining a neural processing unit (NPU) with TrustZone. However, limitations in software maturity and driver support made this approach difficult to pursue within the project timeline. The implementation therefore moved to an STM32 NUCLEO-L552ZE-Q development board featuring an ARM Cortex-M33 processor and TrustZone-M, running Zephyr RTOS and Trusted Firmware-M. This transition required reconsidering the implementation strategy under strict memory and hardware-access constraints.
The talk examines the practical trade-offs encountered when considering alternative designs, including executing the entire inference pipeline in the Secure World and partitioning the model across Secure and Non-Secure Worlds. These considerations helped shape SHANGRI-LA, a third execution environment designed to isolate inference code from untrusted software without placing the entire inference pipeline inside the Secure World.
The presentation also discusses the challenges of integrating the framework with the embedded software stack, evaluating it on real hardware, and measuring its security and performance costs. The evaluation reported a reduction in the Secure-World trusted computing base from 34,865 to 1,523 lines of code, while inference latency increased from 46.62 ms to 46.73 ms in the reported benchmark.
Beyond the results themselves, the talk will focus on lessons learned from platform selection, implementation constraints, experimental design, and evaluation on resource-constrained devices. It will conclude with an interactive discussion on reproducibility, the reporting of unsuccessful approaches, and methodological practices that can improve experimental cybersecurity research.
Bio: Mohamed Khalil Kiri is a cybersecurity engineering student at EURECOM, France. His research interests include embedded systems security, trusted execution environments, and secure machine learning on resource-constrained devices. He contributed to the design, implementation, and experimental evaluation of VeCoDI, focusing on TrustZone-M-based isolation and secure DNN inference on microcontrollers.
Lessons Learned from Benchmarking Automated YARA Rule Generation for In-the-Wild Malware Families
Presenter: Mabon Ninan, Texas A&M University
Abstract: Automating YARA rule generation is crucial for scaling malware incident response, yet evaluating these systems on real-world datasets is challenging, and strategies to do so reveal serious methodological pitfalls. In this talk, we share key experimental insights and lessons learned from developing and benchmarking AutoPYara, an open-source framework designed to overcome the structural instabilities automatic YARA rule generators may face, on 71,000 malware binaries collected by AV companies from infected machines. First, we show the challenges of deriving approximate ground truths for in-the-wild samples with no real ground-truth labels to use as an experimental reference. Specifically, we show that labeling strategies that produce more homogeneous clusters (e.g., code-similarity-based) outperform more heterogeneous ones (e.g., AV-label-based) in controlled settings, but they also limit the coverage of in-the-wild samples. Second, we show that algorithmic strategies for handling homogeneous and heterogeneous malware sets also differ significantly. Specifically, we show that an automatic method designed for relatively homogeneous cases, like a Gaussian mixture model (GMM), consistently underestimates the number of clusters that ideally partition the heterogeneous malware set compared to more heterogeneous strategies, like density-based (e.g., DBSCAN of augmented K-Means). Finally, we show that heterogeneity can benefit rule generators when applied in the right context. Specifically, we show that although AV labels are not good approximations of ground truth for experiments with individual malware families, they better estimate cluster partitioning for heterogeneous malware datasets than homogeneous approaches, recovering performance in challenging cases. These results show that methodological choices can lead to completely different conclusions, underestimating how reliable the rule generation process currently is and how much development our community still needs in this direction.
Bios: Mabon Ninan is currently pursuing his Ph.D. under the supervision of Dr. Marcus Botacin at the Department of Computer Science and Engineering (CSE) at Texas A&M University (TAMU). His research interests include algorithms and machine learning for malware detection.
Dr. Marcus Botacin is an Assistant Professor at the Department of Computer Science and Engineering (CSE) at Texas A&M University (TAMU). His research interests span all domains of malware detection, from hardware to software approaches for dynamic and static analysis.
Reproduction and Benchmarking Lessons from TTP Extraction Research
Presenter: Sean Frankum, University of Georgia
Abstract: Cyber threat reports contain natural language descriptions of threat actors’ tactics, techniques, and procedures (TTPs), with MITRE ATT&CK serving as a primary reference vocabulary. Automatically labeling these texts with the correct TTPs is the goal of many prior works that we examined in our reproduction and benchmarking study. In that study, we attempted to reproduce published TTP extraction experiments using original implementations and developed a common framework and benchmark dataset to evaluate those implementations on data not included in their previous evaluations. In this presentation, we explain our study design choices, including using original implementations instead of reimplementing core methods, focusing the evaluation on a common task, and reducing data annotation effort by using labeled data from previous studies and threat reports whose authors include TTP labels. These choices helped us preserve working implementations and expand the available evaluation data, but directly comparing tools that support different TTP label sets. We use these experiences to anchor a broader discussion of how researchers reproduce and directly compare existing tools.
Bio: Sean Frankum is a Ph.D. student in Computer Science at the University of Georgia. His research interests include cybersecurity, artificial intelligence, and cyber threat intelligence. His work examines the extraction of structured information from cyber threat reports and the use of AI to support security analysts.
Devilray: A Reconfigurable Adversarial Baseline for Revealing Blind Spots in Fake Base Station Detection
Presenter: Duckwoo Kim, KAIST
Abstract: Fake Base Station (FBS) detection has been studied for over two decades, yet financial and regulatory barriers to commercial FBS (C-FBS) devices have left most detectors designed and evaluated against self-built prototypes. Devilray addresses this gap with a reconfigurable adversarial baseline. It is anchored in the analysis of a C-FBS and extended with 3GPP-derived operational variants, covering 2,592 FBS instances. This talk discusses the design and execution of the experiments behind Devilray, along with the challenges encountered and the lessons learned.
We first describe how we characterize FBS operations using a single C-FBS as an empirical anchor. We then discuss how we design the generation of 2,592 FBS instances and validate them end-to-end with commercial UEs, while detector evaluation relies on a targeted sample of 10 instances. Exhaustive validation requires addressing the time cost of over-the-air experiments and defining success criteria across heterogeneous attack configurations, which we handle through automation. We also share further lessons learned from building and evaluating Devilray.
Bios: Duckwoo Kim is a Ph.D. student in the School of Electrical Engineering at KAIST, advised by Prof. Yongdae Kim. He received his M.S. in Electrical Engineering from KAIST in 2021. His research focuses on cellular network security, with an emphasis on fake base station attacks and their detection. He is a co-first author of Devilray and the presenter of this LASER talk.
Taekkyung Oh is a Postdoctoral Researcher in the School of Electrical Engineering at KAIST. He received his Ph.D. in Information Security from KAIST in 2026, advised by Prof. Yongdae Kim. His research interests include cellular and satellite network security, wireless system security, and cyber-physical system security. His work has appeared in USENIX Security, ACM MobiCom, and ACM WiSec, and his vulnerability disclosures have been acknowledged by Samsung, Qualcomm, MediaTek, Google, and the GSMA. He is a co-first author of Devilray.
Behind SfS: Experimental Design and Lessons Learned from a Wireless Replay Protection Testbed
Presenter: Ala Darabseh, New York University Abu Dhabi
Abstract: Evaluating replay protection in wireless broadcast systems requires careful consideration of timing uncertainty, adversarial capabilities, and the practical constraints of real-world communication. In this talk, we present the experimental methodology behind Shuffling for Safeguarding (SfS), focusing on the design and implementation of a three-node Raspberry Pi testbed consisting of a sender, receiver, and replay attacker communicating over an IEEE 802.11 ad-hoc wireless network.
We discuss the experimental design choices, including time synchronization, replay-attack implementation, selection of attacker delays, and timing-validation windows. Our evaluation examines conventional and permutation-aware replay attacks across second- and millisecond-scale delays, highlighting the trade-offs between replay detection, timing tolerance, and communication performance. Beyond the reported results, we discuss practical implementation challenges, experimental limitations, lessons learned from conducting over-the-air experiments, and our approach to supporting reproducibility through publicly available artifacts.
The talk aims to encourage discussion on the challenges of experimentally evaluating timing-dependent security mechanisms and the broader applicability of these methodologies to wireless broadcast systems.
Bios: Ala Darabseh is a Research Engineer at the Center for Cyber Security at New York University Abu Dhabi. She received her B.Sc. in Computer Information Systems and M.Sc. in Computer Science from Jordan University of Science and Technology. Her research focuses on wireless communication security, aviation and satellite systems, and the design and experimental evaluation of security mechanisms. Her work includes replay protection in wireless broadcast systems, ADS-B security, and secure aircraft localization.
Christina Pöpper is an Associate Professor of Computer Science at New York University Abu Dhabi, where she leads the Cyber Security and Privacy Lab. Her research focuses on cybersecurity and privacy, particularly wireless and communication security, cellular networks, secure localization, and aviation security. She received her Ph.D. in Computer Science from ETH Zurich and has served in leadership roles within the international security research community, including ACM WiSec and NDSS.
From Benchmark to Deployment: How Should We Evaluate Learning-Based IDS?
Presenter: Bilel Saghrouchni, INSA Lyon - SPIE ICS, France
Abstract: Machine Learning (ML)-based Intrusion Detection Systems (IDS) are widely studied in the literature but evaluating them remains a major challenge: the perfect dataset does not exist. Datasets are imbalanced or artificially balanced, contain too many attacks type, assume abundant labels, or contain irrelevant features. As a result, IDS evaluation is often treated as a classic ML/ data-science benchmark, overlooking operational constraints and needs. In this talk, we present how we tried to get as close as possible to an operational evaluation of TRACER, a semi-supervised reinforcement-learning IDS, using four literature datasets (NSL-KDD, CICIDS17, UNSW-NB15, ToN-IoT). Our methodology combines (i) metrics that consider both threats and benign traffic, (ii) per-attack evaluation rather than aggregate scores, and (iii) operationally motivated scenarios: label scarcity, and zero-day detection simulated through a leave-one-attack-out protocol. We also discuss what this approach revealed and where it fell short, including the dependence of zero-day detection on the separability of attacks in the learned representation, and the gap between dataset-replay measurements and live traffic.
Bio: Bilel Saghrouchni received his M.Eng. degree in Computer Sciences and Telecommunications from INSA Lyon, France, in 2022, having completed part of his studies at the National University of Singapore. He is currently a Ph.D. Candidate at INSA Lyon in collaboration with SPIE ICS, where he has conducted his research for 3 years. His main research interests encompass continuous intrusion and anomalies detection using cutting-edge Machine and Deep Learning techniques and network traffic features engineering. Before beginning his Ph.D., he gained valuable industry experience as a Software Engineer specializing in ML, data science and Natural Language Processing. His research has been published in international conferences, including ACSAC26, KES26, KES25, and CSNet24 where he received the best poster award. email : bilel.saghrouchni@insa-lyon.fr This work was supervised by Frédéric Le Mouël, full professor at the University of Lyon (France), and Bogdan Szanto, cybersecurity engineer at SPIE ICS (France). They will not be attending the workshop.
Measurements and Emulation of a Cross-layer Attack on Bitcoin and the Lightning Network
Presenter: Charmaine Ndolo, Dresden University of Technology
Abstract: Bitcoin and the Lightning network (LN) have each been studied extensively at the network level, but their relation has so far been overlooked. Our paper presents the first cross-layer, network-level study of the two networks and shows how co-location of a Bitcoin and an LN node can be exploited by a passive network adversary to link an IP address to a Bitcoin address and, in some cases, a real-world entity. This talk gives an overview of the attack before turning to the experimental side of the work. It will cover the measurements we ran on the live networks as well as the emulation testbed we built to evaluate the attack end to end. We will then open the floor for discussion on some of the challenges we encountered and on future work.
Bio: Charmaine Ndolo is a PhD candidate at the Chair for Privacy and Security at the Dresden University of Technology, where her work focuses on the security and resilience of Bitcoin and its broader ecosystem. Her research spans both the base layer and the Lightning Network, investigating how real-world adversaries can exploit weaknesses at the network level to undermine the security, privacy and reliability of Bitcoin payments.