NeurIPS 2023 Citylearn Challenge - Starter Kit
This repository is the NeurIPS 2023 Citylearn Challenge Submission template and Starter kit! Clone the repository to compete now!
This repository contains:
- Documentation on how to submit your models to the leaderboard
- The procedure for best practices and information on how we evaluate your agent, etc.
- Starter code for you to get started!
- Competition Overview
- Getting Started
- How to write your own agent?
- How to start participating?
- Other Concepts
- Important links
Buildings are responsible for 30% of greenhouse gas emissions. At the same time buildings are taking a more active role in the power system by providing benefits to the electrical grid. As such, buildings are an unexplored opportunity to address climate change.
The CityLearn Challenge makes use of the CityLearn Gym environment as an opportunity to compete in investigating the potential of artificial intelligence (AI) and distributed control systems to tackle multiple problems within the built-environment domain. It is designed to attract a multidisciplinary participation audience including researchers, industry experts, sustainability enthusiasts and AI hobbyists as a means of crowd-sourcing solutions to these problems.
The CityLearn Challenge 2023 is a two-track challenge where either track is independent of, but may inform design choices in the other track. Both tracks make use of the same dataset of a synthetic single-family neighborhood and are run in parallel.
This phase provides participants with an opportunity to familiarize themselves with the competition, CityLearn environment and raise issues bordering on the problem statement, source code, dataset quality and documentation to be addressed by the organizers. A solution example will also be provided so that participants can test the submission process and see their submissions show up on the leaderboard. The submissions and leaderboard in this phase are not taken into account during Phase II and subsequent selection of winners in Phase III.
This is the competition round. Participants will be able to see how each of their submissions rank against each other and how their latest submission ranks against other participantsβ submissions in a public leaderboard. There are also changes made to the environment and online evaluation dataset in this phase. At the end of this phase, new submissions will be halted and existing submissions are evaluated against another different dataset but the scores and rankings are kept private and visible to only the challenge organizers.
During this phase, winners will be selected and announced. Also, the organizers will develop an executive summary of the competition that documents the preparation, winning solutions, challenges faced and lessons learned.
- Sign up to join the competition on the AIcrowd website.
- Fork this starter kit repository. You can use this link to create a fork.
- Clone your forked repo and start developing your agent.
- Develop your agent(s) following the template in how to write your own agent section.
- Develop your reward function following the template in how to write your own reward function section.
- Submit your trained models to AIcrowd Gitlab for evaluation (full instructions below). The automated evaluation setup will evaluate the submissions on the citylearn simulator and report the metrics on the leaderboard of the competition.
We recommend that you place the code for all your agents in the agents
directory (though it is not mandatory). You should implement the
register_reset
- The function is used to get the prediction of the first observation afterenv.reset
predict
- This function is called to get actions all observations, except the first.
Add your agent name in user_agent.py
, this is what will be used for the evaluations.
An example are provided in agents/rbc_agent.py
We recommend that you place the code for all your reward functions in the rewards
directory (though it is not mandatory). You should implement the
calculate
- The function is used to get the reward for all observations.
Add your reward class name in user_reward.py
, this is what will be used for the evaluations.
An example are provided in rewards/comfort_reward.py
- Add your SSH key to AIcrowd GitLab
You can add your SSH Keys to your GitLab account by going to your profile settings here. If you do not have SSH Keys, you will first need to generate one.
-
Fork the repository. You can use this link to create a fork.
-
Clone the repository
git clone git@gitlab.aicrowd.com:aicrowd/challenges/citylearn-challenge/citylearn-2023-starter-kit.git
-
Install competition specific dependencies!
cd citylearn-2023-starter-kit pip install -r requirements.txt
-
Write your own agent as described in How to write your own agent section.
-
Write your own reward function as described in How to write your own reward function section.
-
Test your agent locally using
python local_evaluation.py
-
Make a submission as described in How to make a submission section.
We accept submissions with custom runtime, so you don't need to worry about which libraries or framework to pick from.
The configuration files typically include requirements.txt
(pypi packages), apt.txt
(apt packages) or even your own Dockerfile
.
You can check detailed information about the same in the π runtime.md file.
Please follow the example structure as it is in the starter kit for the code structure. The different files and directories have following meaning:
.
βββ aicrowd.json # Submission meta information - like your username
βββ apt.txt # Linux packages to be installed inside docker image
βββ requirements.txt # Python packages to be installed
βββ local_evaluation.py # Use this to check your agent evaluation flow locally
βββ data/ # Contains schema files for citylearn simulator (for local testing)
βββ agents # Place your agents related code here
βββ rbc_agent.py # Simple rule based agent
βββ user_agent.py # IMPORTANT: Add your agent name here
βββ rewards # Place your reward related code here
βββ comfort_reward.py # Example reward function
βββ user_reward.py # IMPORTANT: Add your reward class here
Finally, you must specify an AIcrowd submission JSON in aicrowd.json
to be scored!
The aicrowd.json
of each submission should contain the following content:
{
"challenge_id": "control-track-citylearn-challenge",
"authors": ["your-aicrowd-username"],
"description": "(optional) description about your awesome agent",
}
This JSON is used to map your submission to the challenge - so please remember to use the correct challenge_id
as specified above.
π submission.md
Best of Luck π π
The control track score (ScoreControl) is the weighted average of a thermal comfort score (ScoreControlComfort), an emissions score (ScoreControlEmissions), a grid score (ScoreControlGrid), and a resilience score (ScoreControlResilience). ScoreControlGrid and ScoreControlResilience are averages of four and two KPIs respectively.
The weights are specified in the table below. In Phase I, the highest weight is given to ScoreControlGrid. By Phase II, ScoreControlResilience is introduced and has the same weight as ScoreControlComfort and ScoreControlGrid. ScoreControlEmissions has the lowest non-zero weight in both phases. The private leaderboard in Phase II makes use of the same weights as the public leaderboard.
Phase | ||||
---|---|---|---|---|
Phase I | 0.3 | 0.1 | 0.6 | 0.0 |
Phase II | 0.3 | 0.1 | 0.3 | 0.3 |
All together, the four scores are made up of eight KPIs namely: carbon emissions (G), discomfort (U), ramping (R), 1 - load factor (L), daily peak (P_d), all-time peak (P_n) 1 - thermal resilience (M), and normalized unserved energy (S), which have been defined in the table below. G, U, M, and S are building-level KPIs that are calculated using each building's net electricity consumption (e) or temperature (T) then averaged to get the neighborhood-level value. R, L, P_d, and P_n are neighborhood-level KPIs that are calculated using the neighborhood's net electricity consumption (E). Except U, M, and S all KPIs are normalized by their baseline value where the baseline is the result from when none of the distributed energy resources (DHW storage system, battery, and heat pump) is controlled.
Name | Formula | Description |
---|---|---|
Carbon emissions | Emissions from imported electricity. | |
Discomfort | Proportion of time steps when a building is occupied, and indoor temperature falls outside a comfort band, |
|
Ramping | Smoothness of the neighborhoodβs consumption profile where low |
|
1 - Load factor | Average ratio of daily average and peak consumption. Load factor is the efficiency of electricity consumption and is bounded between 0 (very inefficient) and 1 (highly efficient) thus, the goal is to maximize the load factor or minimize (1 β load factor) | |
Daily peak | Average, maximum consumption at any time step per day. | |
All-time peak | Maximum consumption at any time step. | |
1 - Thermal resilience | Same as discomfort, |
|
Normalized unserved energy | Proportion of unmet demand due to supply shortage e.g. power outage. |
Where:
$t$ : Time step index;$n$ : Total number of time steps,$t$ , in 1 episode;$h$ : Hours per day (24);$d$ : Day;$i$ : Building index;$b$ : Total number of buildings;$e$ : Building-level net electricity consumption (kWh);$E$ : Neighborhood-level net electricity consumption (kWh);$A$ : Electricity rate ($/kWh);$B$ : Emission rate (kgCO2e/kWh);$T$ : Indoor dry-bulb temperature (oC);$T^{\textrm{setpoint}}$ : Indoor dry-bulb temperature setpoint (oC);$c$ : Thermal comfort band ($\pm T^{\textrm{setpoint}}$ );$O$ : Occupant count (people);$F$ : Power outage signal (Yes/No); and$q$ : Building-level cooling, domestic hot water and non-shiftable load energy demand (kWh).
For Phase I, your agent should complete 1 episode in 30 minutes
. Note that the number of episodes and time can change depending on the phase of the challenge. However we will try to keep the throughput requirement of your agent, so you need not worry about phase changes. We only measure the time taken by your agent. For compute, you will be provided a virutal machine with 2 CPU cores and 12 GB of RAM
.
Participants can run the evaluation protocol for their agent locally with or without any constraint posed by the Challenge to benchmark their agents privately. See local_evaluation.py
for details. You can change it as you like, it will not be used for the competition. You can also change the simulator schema provided under data/schemas/warm_up/schema.json
, this will not be used for the competition.
π You can share your solutions or any other baselines by contributing directly to this repository by opening merge request.
- Add your implemntation as
agents/<your_agent>.py
. - Import it in
user_agent.py
- Test it out using
python local_evaluation.py
. - Add any documentation for your approach at top of your file.
- Create merge request! πππ
-
πͺ Challenge Page: https://www.aicrowd.com/challenges/neurips-2023-citylearn-challenge
-
π£ Discussion Forum: https://discourse.aicrowd.com/c/neurips-2023-citylearn-challenge
-
π Leaderboard: https://www.aicrowd.com/challenges/neurips-2023-citylearn-challenge/problems/control-track-citylearn-challenge/leaderboards