View on GitHub

WSDM2022-Challenge

WSDM2022 Challenge - Large scale temporal graph link prediction

WSDM 2022 Challenge - Temporal Link Prediction

Description

Temporal Link Prediction is one of the classical tasks on temporal graphs. Contrary to link prediction which asks if an edge exists between two nodes on a partially observed graph, Temporal Link Prediction asks if an edge will exist between two nodes within a given time span. It is more useful than traditional link prediction as one can then build multiple applications around the model, such as forecasting the demand of customers in E-commerce, or forecasting what event will happen in a social network, etc.

We are expecting an approach that works well on large-scale temporal graphs in general. In this challenge, we expect a single model (hyperparameters can vary) that works well on two kinds of data simultaneously:

The task will be predicting whether an edge of a given type will exist between two given nodes before a given timestamp.

Description of Dataset A

Dataset A contains the following files:

Description of Dataset B

Dataset B contains a single file:

Note that in this dataset the nodes and edge types do not have features, unlike Dataset A.

Test set and submission guidelines

We will release two CSV file input_A.csv and input_B.csv representing the test queries for dataset A and B respectively. Each file contains the following five columns:

We expect two files output_A.csv and output_B.csv representing your predictions on each test query. Each file should contain the same number of lines as the given input files. Each line should contain a single number representing the predicted probability that the edge connecting from node ID src_id to node ID dst_id with type event_type will be added to the graph at some time between start_time and end_time (inclusive of both endpoints).

It is guaranteed that the timestamps in the test set will be always later than the training set. This is to match a more realistic setting where one learns from the past and predicts the future.

During competition we will release an intermediate test set and a final test set. The prizes will only depend on the performance on the final test set, and you will need to submit supplementary materials such as your code repository URL. You can optionally submit your prediction on the intermediate test set and see how your model performs.

12/23 Update: The intermediate leaderboard has been announced. Note that the ranking on the intermediate test set will not impact the ranking on the final test set and the prize in any way.

Due to the difference between the difficulties of two datasets, we have also changed the ranking metric from harmonic average of AUC to the average of T-scores to encourage balancing the performance on both dataset, instead of sacrificing the performance on one for the other. See the Evaluation Criteria section below for details.

We have also made available a quick evaluation platform for the intermediate test set. During the rest of the competition you can submit your prediction on the intermediate test set there as many times as you like.

The final test set for dataset A and dataset B has been released as well. Submission is open with this Google form until January 20th 2022 23:59:59PM AoE.

12/13 Update:

After inspection of the data and submissions we have found that there are some test set records already appearing in the training set. Moreover, some of the test set records are labeled 0 while they actually appear in the training set. Because of this, we have updated the input_A_initial.csv.gz and input_B_initial.csv.gz files, as well as the intermediate test set input_A.csv.gz and input_B.csv.gz, in the original URL, removing the test records with issues. As a result:

The old intermediate test set are kept here for dataset A and dataset B.

As a result:

11/11 Update:

11/10 Update: We have updated the initial test set for Dataset B so that the nodes not appearing in the training set are removed. The resulting number of test examples is 5,074.

11/4 Update: We have released an initial test set for Dataset A and Dataset B for developing your solutions, as well as a simple baseline.

Example

Say that an edge with type 0 from node 0 to node 1 will appear at timestamp 15000000:

src_id dst_id edge_type timestamp
0 1 0 15000000

You should predict some probability close to 1 for the following query since the timestamp 15000000 is between 14000000 and 16000000:

src_id dst_id edge_type start_time end_time
0 1 0 14000000 16000000

However, you should predict some probability close to 0 for both test queries below:

src_id dst_id edge_type start_time end_time
0 1 0 13000000 14000000
src_id dst_id edge_type start_time end_time
0 1 0 16000000 17000000

Competition Terms and Conditions

At the end of the challenge, each team is encouraged to open source the source code that was used to generate their final challenge solution under the MIT license. To be eligible for the leaderboard or prizes, winning teams are also required to submit papers describing their method to the WSDM Cup Workshop, and present their work at the workshop. 1/19/2022 Update with the message from WSDM Cup Chair, specifically for competition winners:

To claim prizes, those teams who are announced as task winners will be expected to present their solutions at the WSDM Cup 2022 event (virtually), and must also submit a report describing their solution in detail – these reports will be made available on the WSDM Cup website and will be referenceable as scientific reports in the future. Note that the reports are archival, and will not be part of official WSDM proceedings.

The reports must be in English, in PDF format, and should not exceed four (4) pages in the current ACM two-column conference format (including references and figures). Suitable LaTeX and Word templates are available from the ACM Website. Please mention the team name in the authors section, and provide a link to a repository for the open-sourced code within the PDF.

The submission deadline is February 15th, 2020 (AOE timezone). Submissions will be managed via EasyChair: https://easychair.org/conferences/?conf=wsdmcup22

Refer to the “Call for Papers” section on the WSDM Cup 2022 webpage for more details.

Participants are allowed to participate only once, with no concurrent submissions or code sharing between the teams. The same team can submit multiple times, with only the last submission being evaluated.

Participants are not allowed to use external datasets or pretrained models.

We welcome any kinds of model in this challenge, regardless of whether it is a deep learning model or some graph learning algorithm. 1/12 Update: We expect a single model working on both datasets, although hyperparameters could vary. Ensemble methods are not allowed.

Evaluation Criteria

We use Area Under ROC (AUC) as evaluation metric for both datasets, and use the harmonic average of the two AUCs as the score of the submission. Specifically, let AUC_A and AUC_B be the AUC for Dataset A and Dataset B respectively, the final score is 2 / (1 / AUC_A + 1 / AUC_B)

This is to encourage the submissions to work well on both tasks, instead of working extremely well on one while sacrificing the other.

12/23 Update: In order to balance the difficulty between two datasets, we have decided to change the ranking metric from the harmonic average: 2 / (1 / AUC_A + 1 / AUC_B) to the average of T-scores, to better promote our goal on “encouraging the submissions to work well on both instead of working extremely well on one while sacrificing the other”. The T-score of a dataset is computed as

TScore = (AUC - mean(AUC)) / std(AUC) * 0.1 + 0.5

where mean(AUC) and std(AUC) represents the mean and standard deviation of AUC of all participants. The score for ranking will be (TScore_A + TScore_B) / 2.

For reference, we also kept the original harmonic average of AUC in the leaderboard.

Leaderboard for Intermediate Test Set

This is an excerpt from the complete leaderboard; for score computation details please refer here. Note that this will not impact the ranking on the final test set and the prize.

Team Name AUC (Dataset A) AUC (Dataset B) Harmonic Average of AUC Average of T/100 Rank on Harmonic Average of AUC Rank on Average of T/100
IDEAS Lab UT 0.582272719 0.872039558 0.698288604 0.813037237 1 1
DIVE@TAMU 0.495906631 0.756680212 0.599148453 0.585081611 2 2
nothing here 0.500971508 0.632476354 0.559095205 0.534215147 3 3
Graphile 0.52824279 0.501279235 0.51440792 0.523983091 8 4
smallhand 0.516251289 0.539024256 0.527392051 0.518700616 6 5
/tmp/graph 0.519933859 0.519243432 0.519588416 0.516306306 7 6
HUST_D5 0.496201592 0.572936262 0.531815208 0.495548282 5 7
HappyICES 0.507947137 0.500995797 0.504447521 0.48357129 10 8
TopoLab 0.501385353 0.525133491 0.51298472 0.482389216 9 9
10000 Monkeys 0.505363661 0.499368274 0.50234808 0.47764665 11 10
AntGraph 0.503390663 0.5 0.501689602 0.474041453 13 11
zhang 0.501189056 0.502050738 0.501619527 0.470678591 14 12
neutrino 0.500645569 0.501220483 0.500932861 0.469192943 15 13
Tencent_2022 0.499652798 0.500876566 0.500263933 0.467054303 16 14
MegaTron 0.498004853 0.505898455 0.50192062 0.466247299 12 15
marble 0.49767288 0.499622357 0.498645713 0.462510404 18 16
beauty 0.495978453 0.503765895 0.499841844 0.461180372 17 17
luozhhh 0.502644132 0.453959254 0.477062822 0.449979073 20 18
no_free_lunch 0.438312773 0.681825615 0.533599918 0.434084762 4 19
NodeInGraph 0.47341036 0.5 0.486342019 0.414551355 19 20

Leaderboard for Final Test Set

This is an excerpt from the complete leaderboard; for score computation details please refer here.

Team name Dataset A AUC Dataset B AUC Dataset A T-score/100 Dataset B T-score/100 Average of T/100 Ranking Status
AntGraph 0.666001 0.901961 0.676837 0.584637 0.630737 1 1st Prize
nothing here 0.662482 0.906923 0.670374 0.587511 0.628942 2 2nd Prize
NodeInGraph 0.627821 0.865567 0.606713 0.56356 0.585137 3 3rd Prize
We can [mask]! 0.603621 0.898232 0.562266 0.582478 0.572372 4 Runners-up
IDEAS Lab UT 0.605264 0.873949 0.565283 0.568414 0.566849 5 Runners-up
SLi-Rec 0.583935 0.892547 0.526108 0.579186 0.552647 6 Runners-up
no_free_lunch 0.574475 0.911889 0.508733 0.590387 0.54956 7  
DIVE@TAMU 0.585244 0.854917 0.528512 0.557392 0.542952 8  
TopoLab 0.537097 0.896267 0.440082 0.581339 0.510711 9  
smallhand 0.538031 0.726306 0.441797 0.48291 0.462353 10  
10000 Monkeys 0.547856 0.532121 0.459841 0.370451 0.415146 11  
Marble 0.499147 0.592357 0.370378 0.405335 0.387857 12  
/tmp/graph 0.516565 0.523069 0.40237 0.365208 0.383789 13  
MegaTron 0.496068 0.501011 0.364723 0.352434 0.358578 14  
graphile 0.502199 0.460128 0.375984 0.328757 0.35237 15  

The top 3 teams are eligible for prizes. In addition, the runners-up will be also eligible for one free WSDM Cup registration. The 1st-6th place teams are required to submit a technical report and present the solutions to the WSDM Cup conference.

Schedule

Date Event
Oct 15 2021 Website ready and training set available for download.
Nov 11 2021 Intermediate test set release and intermediate submission starts.
Dec 11 2021 Dec 20 2021 Intermediate submission ends.
Dec 16 2021 Dec 22 2021 Intermediate leaderboard result announcement.
Dec 17 2021 Dec 23 2021 Final test set release and final submission starts.
Jan 20 2022 Final submission ends.
Jan 24 2022 Final leaderboard result announcement.
Jan 25 2022 Invitations to top 6 teams for short papers.
Feb 15 2022 Short paper deadline.
Feb 21-25 2022 WSDM Cup conference presentation.

Prizes

The prizes will be determined solely by the performance on the final test set.

We would like to thank Intel for kindly sponsoring this event.

Support or Contact

If you have questions or need clarifications, feel free to join the channel wsdm22-challenge in DGL’s Slack workspace.

WSDM call for cup proposals: https://www.wsdm-conference.org/2022/call-for-wsdm-cup-proposals/