The workshop will take place between 9 a.m. and 5 p.m. (SGT) on December 12 in the Ballroom Melati Junior (Room No. 4111) of Marina Bay Sands, Singapore (CDC Conference Venue).

Paper on the Impact of Releasing Code Recording of the Workshop

Overview

Over the past years, the scientific community has grown more cognizant of the importance and challenges of transparent and reproducible research. This topic has become increasingly important given the rise of complex algorithms (e.g., machine learning models or optimization-based algorithms), which cannot be adequately documented in standard publications alone. Benchmarking and code sharing are two key instruments that researchers use to improve reproducibility. Benchmarks have played a critical role in advancing the state of the art in machine learning research. Analogously, well-established benchmarks in controls could enable researchers to compare the effectiveness of different control algorithms. There are currently only a few benchmarks available for comparing control algorithms (e.g., the Autonomie simulation model of a Toyota Prius or the shared experimental testbed Robotarium). Limited comparisons are also due to the modest number of open-source implementations of control algorithms. Over a six-year period (2016-2021), we found that the percentage of papers with code at CDC has more than doubled. However, we also found that at CDC 2021 only 2.6% of publications had code (compared to around 5% at the robotics conference ICRA and over 60% at the machine learning conference NeurIPS). These trends are encouraging, but there is still much work to be done to promote and increase efforts toward reproducible research that accelerates innovation. Benchmarking and releasing code alongside papers can serve as a critical first step in this direction. By organizing this workshop at the Conference on Decision and Control (CDC 2023), we aim to increase awareness of these challenges and inspire attendees to contribute to benchmarking efforts and share open-source code through publication in the future.

Discussion Themes | Call for Contributions | Program | Speakers | Organizers

Discussion Themes

Theme A: Reproducibility and Open-Source Code

Reproducibility of algorithms and experiments is a fundamental requirement of scientific progress. In recent years, we have seen the proliferation of tools to make algorithms, experiments, and results open-source and reproducible; however, their adoption in the controls community did not grow at the same pace. This has led to challenges in reproducing results within the controls community. Through 30-minute lecture-style talks, the speakers will talk about their research, how they have made their research open-source and reproducible, what the challenges are, and the benefits they have experienced from doing so.

Theme B: Benchmarking

Benchmarks allow researchers to evaluate the effectiveness of their approaches against the state-of-the-art, and have played a critical role in advancing machine learning research. Similarly, well-established benchmarks in controls could enable quantitative comparisons between different control approaches and help direct the progress of controls research. However, there are only a few benchmarks currently available for comparing control algorithms. We additionally plan to coordinate with the organizers of the “Autonomous Driving Control Benchmark Challenge” to gain their perspective on benchmarking. Through 30-minute lecture-style talks, the speakers will talk about their research, how they have benchmarked their research, and the benefits they have experienced from doing so.

Program

The workshop location is Ballroom Melati Junior (Room No. 4111) of Marina Bay Sands, Singapore (CDC Conference Venue). For virtual attendance check our YouTube channel: http://tiny.cc/cdc-ws-23-youtube

Below is a tentative program for the workshop. Times are in SGT.

Theme A: Reproducibility and Open-Source Code

09:00 – 09:10: Opening Remarks
09:10 – 09:40: Theme A Introduction
09:40 – 10:20: Lightning Talks I
10:20 – 10:30: Coffee Break
10:30 – 11:00: Lightning Talks II
11:00 – 11:30: Theme A Invited Talk 1 by Magnus Egerstedt on “The Robotarium: A Remotely Accessible Robotics Lab”
11:30 – 12:00: Theme A Invited Talk 2 by Sawyer Fuller on “Open Source Controls Software in Robot Flies and Education”
12:00 – 12:30: Theme A Focused Panel Discussion with Stefan Sosnowski and Sawyer Fuller
12:30 – 13:30: Lunch Social

Theme B: Benchmarking

13:30 – 13:40: Theme B Introduction
13:40 – 14:20: Lightning Talks III
14:20 – 15:00: Lightning Talks IV
15:00 – 15:20: Coffee Break
15:20 – 15:50: Theme B Invited Talk 1 by Antonin Raffin on “Practical Tips for Reliable Reinforcement Learning”
15:50 – 16:20: Theme B Invited Talk 2 by Changliu Liu
16:20 – 16:50: Theme B Focused Panel Discussion with Antonin Raffin and Changliu Liu
16:50 – 17:00: Concluding Remarks
17:00 – 18:00: Social Gathering

Invited Speakers

Organizers