Search |  Contact |  SRI Home Do not follow this link, or your host will be blocked from this site. This is a spider trap. Do not follow this link, or your host will be blocked from this site. This is a spider trap. Do not follow this link, or your host will be blocked from this site. This is a spider trap.A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A ASRI International.  333 Ravenswood Avenue.  Menlo Park, CA 94025-3493. SRI International is a nonprofit corporation.

AIC Seminar Series

CAMeLeon: Standardizing Reinforcement Learning Competency Assessment for Custom Agents and Environments

Sam ShowalterSRI AIC / UC Irvine[Home Page]

Date:  Friday, August 27th 2021 at 4:00pm

Location:  Zoom:  (Directions)

Join Remotely: 

Join ZoomGov Meeting

Today we increasingly rely on machine learning models to autonomously reason about the world. However, existing research indicates that even well-trained systems are often unreliable when exposed to unfamiliar or out-of-distribution (OOD) inputs. In turn, research on machine learning robustness continues to grow in order to address this need. Under DARPA’s Competency Awareness Machine Learning (CAML) project, SRI has been working on this problem for some time, specifically exploring methods to analyze the competency of a reinforcement learning agent as it interacts with an environment. Though competency is an abstract concept, we have built a suite of tools to quantify and characterize competency along many different dimensions. For my internship, I designed and built CAMeLeon, a research package that allows scientists to easily leverage this toolkit and assess the competency of custom agents and environments through a single standard API. Moreover, CAMeLeon can assist with rapid environment development and model training by leveraging a set of highly optimized distributed computing tools. In this talk, we will discuss the need for tools like CAMeLeon, details about its design, and close with a demonstration of its capabilities on Canniballs, a custom game built to examine competency in RL agents.`

   Bio for Sam Showalter

Sam Showalter is a computer science PhD student at the University of California, Irvine exploring machine learning robustness, novelty detection, and domain adaptation. His primary interests include examining autonomous perception and decision making under uncertainty as well as building competent, trustworthy ML systems. Previously, he has worked as a technology consultant for West Monroe Partners and as a quantum computing researcher for the National Institute of Standards and Technology (NIST). To learn more about or connect with Sam, you can visit his website here:

   Note for Visitors to SRI

Photography or broadcast of the event is prohibited unless specifically authorized by SRI. Reporters must coordinate with SRI 24 hours in advance before attending.
Please arrive at least 10 minutes early as you will need to sign in by following instructions by the lobby phone at Building E (or call Wilma Lenz at 650 859 4904, or Eunice Tseng at 650 859 2799). SRI is located at 333 Ravenswood Avenue in Menlo Park. Visitors may park in the parking lots off Fourth Street. Detailed directions to SRI, as well as maps, are available from the Visiting AIC web page. There are two entrances to SRI International located on Ravenswood Ave. Please check the Building E entrance signage.

SRI International
©2022 SRI International 333 Ravenswood Avenue, Menlo Park, CA 94025-3493
SRI International is an independent, nonprofit corporation. Privacy policy