Multi-dataset Time Series Anomaly Detection

Evaluation

• Evaluation for this competition will be done based on the outcomes of Phase II only.

• There will be a public leaderboard showcasing the results instantly as you submit a submission file. 

• The private leaderboard showcasing rank and winner will be released one week after the competition is over on April 16th 2021. This will be the final leaderboard.

• We will use percentage as a metric to compute the forecast.

• Every time-series has exactly one anomaly.

• For every correct identification of the location of anomaly you will get 1 point and 0 points for every incorrect.

• We have added +/- 100 locations on either side of the anomaly range to award the correct answer.

  
  

  Example

  
  

  
  

  
  

  There are 250 files, for every correct answer you will get 1 point and 0 for incorrect. The max score you can obtain is 100% ( as long as you do this in code using an algorithm, no hand labeling). We reserve rights to disqualify any participant if we suspect of any. Please see rules for participating in the competition.
  
  

Rules

We reserve the right to change the rules and data sets if deemed necessary.

Updates for Phase II (from description page):

Q) Why must you submit code with every attempt?
(A) Recall the goal of the contest is not to find the anomalies. The goal of the contest is to produce a single algorithm that can find the anomalies [*]. If your submission turns out to be competitive, your submitted code allows you to create a post-hoc demonstration that it was the result of a clever algorithm.
(Q) Why must you use an official university or company email address to register?
(A) Experience in similar contests suggest that otherwise individuals may register multiple times to glean an advantage. It is hard to prevent multiple registrations, but this policy goes someway to limit the utility of an unfair advantage.
[*] Of course, the “single algorithm” can be an ensemble or a meta-algorithm that automatically chooses which algorithm and which parameters to use. However, having a human to decide which algorithm, or which parameters on a case-by-case basis is not allowed. This is not a test of human skill, this is a test of algorithms.   
 

Note:

The spirit of the contest is to create a general-purpose algorithm for anomaly detection. It is against the spirit of this goal (and explicitly against the rules) to embed a human’s intelligence into the algorithm based on a human inspection of the contest data. For example, a human might look at the data and notice: “When the length of the training data is odd, algorithms that look at the maximum value are best. But when the length of the training data is even, algorithms that look at the variance are best.” , then the human might code up a meta algorithm like “If odd(length(training data)) then invoke … ”. There is a simple test for example above, if we just duplicated the first datapoint, would the outcomes be essentially identical? Of course, an algorithm can be adaptive. If the representational power of your algorithm is able to discover and exploit some regularity, then that is fine. However, an algorithm that logically memorizes which of datasets it looking at, and changes it parameters/behavior based on that observation (not on the intrinsic properties of the data it observes) is cheating. Our code review for the best performing algorithms will attempt to discover any such deliberate overfitting to the contest problems.

Announcements:

To receive announcements and be informed of any change in rules, the participants must provide a valid email to the challenge platform

Conditions of participation:

Participation requires complying with the rules of the Competition. Prize eligibility is restricted by US government export regulations. The organizers, sponsors, their students, close family members (parents, sibling, spouse or children) and household members, as well as any person having had access to the truth values or to any information about the data or the Competition design giving him (or her) an unfair advantage are excluded from participation. A disqualified person may submit one or several entries in the Competition and request to have them evaluated, provided that they notify the organizers of their conflict of interest. If a disqualified person submits an entry, this entry will not be part of the final ranking and does not qualify for prizes. The participants should be aware that the organizers reserve the right to evaluate for scientific purposes any entry made in the challenge, whether or not it qualifies for prizes.

Dissemination:

The Winners will be invited to attend a remote webinar organized by the hosts and present their method.

Registration:

The participants must register to the platform and provide a valid email address. Teams must register only once and provide a group email, which is forwarded to all team members. Teams or solo participants registering multiple times to gain an advantage in the competition may be disqualified.

Intellectual Policy:

All work is open source. 

Anonymity:

The participants who do not present their results at the webinar can elect to remain anonymous by using a pseudonym. Their results will be published on the leaderboard under that pseudonym, and their real name will remain confidential. However, the participants must disclose their real identity to the organizers to claim any prize they might win. 

One account per participant:
You cannot sign up from multiple accounts and therefore you cannot submit from multiple accounts.  

Team Size:

Max team size of 4.

No private sharing outside teams:
Privately sharing code or data outside of teams is not permitted. It's okay to share code if made available to all participants on the forums.

Submission Limits:
You may submit a maximum of 1 entry per day.

Specific Understanding:

1. Use of external data is not permitted. This includes use of pre-trained models.

2. Hand-labeling is not permitted and will be grounds for disqualification.

3. Knowledge between 2 files should not be shared, any violation of this will lead to disqualification

4. Submissions should be reasonably constrained to standard Open source libraries (Python, R, Julia and Octave)

5. If submitted code cannot be run, the team may be contacted, if minor remediation or sufficient information not provided to run the code, the submission will be removed.

6. If an algorithm is stochastic please make sure you save the seeds.

Leaderboard

Getting Started

Overview of the Time Series Anomaly Detection Competition

Detecting Anomaly in univariate time series is a challenge that has been around for more than 50 years. Several attempts have been made but still there is no robust outcome. This year  Prof. Eamonn Keogh and Taposh Roy as part of KDD Cup 2021 are hosting the multi data set time series anomaly detection competition. This goal of this competition is to encourage industry and academia to find a solution for univariate time-series anomaly detection. Prof. Keogh has provided 250 data-sets collected over 20 years of research to further this area. Please review the brief overview video developed by Dr. Keogh.

Here is a simple example show-casing how to find anomaly in a single time series file provided.

#importing the required libraries
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import matrixprofile as mp
from matrixprofile import *

 #reading the dataset
df=pd.read_csv('/Users/code/timeseries/ucr_competition_data/005_UCR_Anomaly_4000.txt', names=['values'])

#set window size
window_size=100
#calculating the matrix profile with window size'4'
profile=mp.compute(df['values'].values, window_size)

#discover motifs
profile = mp.discover.motifs(profile, k=window_size)
print(profile['discords'])