Note
Click here to download the full example code
Training GNN with Neighbor Sampling for Node Classification¶
This tutorial shows how to train a multi-layer GraphSAGE for node
classification on ogbn-arxiv provided by Open Graph
Benchmark (OGB). The dataset contains around
170 thousand nodes and 1 million edges.
By the end of this tutorial, you will be able to
Train a GNN model for node classification on a single GPU with DGL’s neighbor sampling components.
This tutorial assumes that you have read the Introduction of Neighbor Sampling for GNN Training.
Loading Dataset¶
OGB already prepared the data as DGL graph.
import dgl
import torch
import numpy as np
from ogb.nodeproppred import DglNodePropPredDataset
dataset = DglNodePropPredDataset('ogbn-arxiv')
device = 'cpu' # change to 'cuda' for GPU
Out:
WARNING:root:The OGB package is out of date. Your version is 1.2.4, while the latest version is 1.3.3.
Downloading https://snap.stanford.edu/ogb/data/nodeproppred/arxiv.zip
0%| | 0/81 [00:00<?, ?it/s]
Downloaded 0.00 GB: 0%| | 0/81 [00:00<?, ?it/s]
Downloaded 0.00 GB: 1%|1 | 1/81 [00:00<00:25, 3.11it/s]
Downloaded 0.00 GB: 1%|1 | 1/81 [00:00<00:25, 3.11it/s]
Downloaded 0.00 GB: 1%|1 | 1/81 [00:00<00:25, 3.11it/s]
Downloaded 0.00 GB: 4%|3 | 3/81 [00:00<00:09, 7.82it/s]
Downloaded 0.00 GB: 4%|3 | 3/81 [00:00<00:09, 7.82it/s]
Downloaded 0.00 GB: 4%|3 | 3/81 [00:00<00:09, 7.82it/s]
Downloaded 0.01 GB: 4%|3 | 3/81 [00:00<00:09, 7.82it/s]
Downloaded 0.01 GB: 4%|3 | 3/81 [00:00<00:09, 7.82it/s]
Downloaded 0.01 GB: 9%|8 | 7/81 [00:00<00:04, 16.49it/s]
Downloaded 0.01 GB: 9%|8 | 7/81 [00:00<00:04, 16.49it/s]
Downloaded 0.01 GB: 9%|8 | 7/81 [00:00<00:04, 16.49it/s]
Downloaded 0.01 GB: 9%|8 | 7/81 [00:00<00:04, 16.49it/s]
Downloaded 0.01 GB: 9%|8 | 7/81 [00:00<00:04, 16.49it/s]
Downloaded 0.01 GB: 9%|8 | 7/81 [00:00<00:04, 16.49it/s]
Downloaded 0.01 GB: 15%|#4 | 12/81 [00:00<00:02, 24.62it/s]
Downloaded 0.01 GB: 15%|#4 | 12/81 [00:00<00:02, 24.62it/s]
Downloaded 0.01 GB: 15%|#4 | 12/81 [00:00<00:02, 24.62it/s]
Downloaded 0.01 GB: 15%|#4 | 12/81 [00:00<00:02, 24.62it/s]
Downloaded 0.02 GB: 15%|#4 | 12/81 [00:00<00:02, 24.62it/s]
Downloaded 0.02 GB: 20%|#9 | 16/81 [00:00<00:02, 28.86it/s]
Downloaded 0.02 GB: 20%|#9 | 16/81 [00:00<00:02, 28.86it/s]
Downloaded 0.02 GB: 20%|#9 | 16/81 [00:00<00:02, 28.86it/s]
Downloaded 0.02 GB: 20%|#9 | 16/81 [00:00<00:02, 28.86it/s]
Downloaded 0.02 GB: 20%|#9 | 16/81 [00:00<00:02, 28.86it/s]
Downloaded 0.02 GB: 25%|##4 | 20/81 [00:00<00:01, 31.89it/s]
Downloaded 0.02 GB: 25%|##4 | 20/81 [00:00<00:01, 31.89it/s]
Downloaded 0.02 GB: 25%|##4 | 20/81 [00:00<00:01, 31.89it/s]
Downloaded 0.02 GB: 25%|##4 | 20/81 [00:00<00:01, 31.89it/s]
Downloaded 0.02 GB: 25%|##4 | 20/81 [00:00<00:01, 31.89it/s]
Downloaded 0.02 GB: 30%|##9 | 24/81 [00:00<00:01, 33.91it/s]
Downloaded 0.02 GB: 30%|##9 | 24/81 [00:01<00:01, 33.91it/s]
Downloaded 0.03 GB: 30%|##9 | 24/81 [00:01<00:01, 33.91it/s]
Downloaded 0.03 GB: 30%|##9 | 24/81 [00:01<00:01, 33.91it/s]
Downloaded 0.03 GB: 30%|##9 | 24/81 [00:01<00:01, 33.91it/s]
Downloaded 0.03 GB: 30%|##9 | 24/81 [00:01<00:01, 33.91it/s]
Downloaded 0.03 GB: 36%|###5 | 29/81 [00:01<00:01, 36.03it/s]
Downloaded 0.03 GB: 36%|###5 | 29/81 [00:01<00:01, 36.03it/s]
Downloaded 0.03 GB: 36%|###5 | 29/81 [00:01<00:01, 36.03it/s]
Downloaded 0.03 GB: 36%|###5 | 29/81 [00:01<00:01, 36.03it/s]
Downloaded 0.03 GB: 36%|###5 | 29/81 [00:01<00:01, 36.03it/s]
Downloaded 0.03 GB: 36%|###5 | 29/81 [00:01<00:01, 36.03it/s]
Downloaded 0.03 GB: 42%|####1 | 34/81 [00:01<00:01, 37.21it/s]
Downloaded 0.03 GB: 42%|####1 | 34/81 [00:01<00:01, 37.21it/s]
Downloaded 0.04 GB: 42%|####1 | 34/81 [00:01<00:01, 37.21it/s]
Downloaded 0.04 GB: 42%|####1 | 34/81 [00:01<00:01, 37.21it/s]
Downloaded 0.04 GB: 42%|####1 | 34/81 [00:01<00:01, 37.21it/s]
Downloaded 0.04 GB: 42%|####1 | 34/81 [00:01<00:01, 37.21it/s]
Downloaded 0.04 GB: 48%|####8 | 39/81 [00:01<00:01, 38.08it/s]
Downloaded 0.04 GB: 48%|####8 | 39/81 [00:01<00:01, 38.08it/s]
Downloaded 0.04 GB: 48%|####8 | 39/81 [00:01<00:01, 38.08it/s]
Downloaded 0.04 GB: 48%|####8 | 39/81 [00:01<00:01, 38.08it/s]
Downloaded 0.04 GB: 48%|####8 | 39/81 [00:01<00:01, 38.08it/s]
Downloaded 0.04 GB: 53%|#####3 | 43/81 [00:01<00:00, 38.39it/s]
Downloaded 0.04 GB: 53%|#####3 | 43/81 [00:01<00:00, 38.39it/s]
Downloaded 0.04 GB: 53%|#####3 | 43/81 [00:01<00:00, 38.39it/s]
Downloaded 0.04 GB: 53%|#####3 | 43/81 [00:01<00:00, 38.39it/s]
Downloaded 0.05 GB: 53%|#####3 | 43/81 [00:01<00:00, 38.39it/s]
Downloaded 0.05 GB: 58%|#####8 | 47/81 [00:01<00:00, 38.75it/s]
Downloaded 0.05 GB: 58%|#####8 | 47/81 [00:01<00:00, 38.75it/s]
Downloaded 0.05 GB: 58%|#####8 | 47/81 [00:01<00:00, 38.75it/s]
Downloaded 0.05 GB: 58%|#####8 | 47/81 [00:01<00:00, 38.75it/s]
Downloaded 0.05 GB: 58%|#####8 | 47/81 [00:01<00:00, 38.75it/s]
Downloaded 0.05 GB: 63%|######2 | 51/81 [00:01<00:00, 39.00it/s]
Downloaded 0.05 GB: 63%|######2 | 51/81 [00:01<00:00, 39.00it/s]
Downloaded 0.05 GB: 63%|######2 | 51/81 [00:01<00:00, 39.00it/s]
Downloaded 0.05 GB: 63%|######2 | 51/81 [00:01<00:00, 39.00it/s]
Downloaded 0.05 GB: 63%|######2 | 51/81 [00:01<00:00, 39.00it/s]
Downloaded 0.05 GB: 63%|######2 | 51/81 [00:01<00:00, 39.00it/s]
Downloaded 0.05 GB: 69%|######9 | 56/81 [00:01<00:00, 39.67it/s]
Downloaded 0.06 GB: 69%|######9 | 56/81 [00:01<00:00, 39.67it/s]
Downloaded 0.06 GB: 69%|######9 | 56/81 [00:01<00:00, 39.67it/s]
Downloaded 0.06 GB: 69%|######9 | 56/81 [00:01<00:00, 39.67it/s]
Downloaded 0.06 GB: 69%|######9 | 56/81 [00:01<00:00, 39.67it/s]
Downloaded 0.06 GB: 74%|#######4 | 60/81 [00:01<00:00, 39.64it/s]
Downloaded 0.06 GB: 74%|#######4 | 60/81 [00:01<00:00, 39.64it/s]
Downloaded 0.06 GB: 74%|#######4 | 60/81 [00:01<00:00, 39.64it/s]
Downloaded 0.06 GB: 74%|#######4 | 60/81 [00:01<00:00, 39.64it/s]
Downloaded 0.06 GB: 74%|#######4 | 60/81 [00:01<00:00, 39.64it/s]
Downloaded 0.06 GB: 74%|#######4 | 60/81 [00:02<00:00, 39.64it/s]
Downloaded 0.06 GB: 80%|######## | 65/81 [00:02<00:00, 40.38it/s]
Downloaded 0.06 GB: 80%|######## | 65/81 [00:02<00:00, 40.38it/s]
Downloaded 0.07 GB: 80%|######## | 65/81 [00:02<00:00, 40.38it/s]
Downloaded 0.07 GB: 80%|######## | 65/81 [00:02<00:00, 40.38it/s]
Downloaded 0.07 GB: 80%|######## | 65/81 [00:02<00:00, 40.38it/s]
Downloaded 0.07 GB: 80%|######## | 65/81 [00:02<00:00, 40.38it/s]
Downloaded 0.07 GB: 86%|########6 | 70/81 [00:02<00:00, 41.34it/s]
Downloaded 0.07 GB: 86%|########6 | 70/81 [00:02<00:00, 41.34it/s]
Downloaded 0.07 GB: 86%|########6 | 70/81 [00:02<00:00, 41.34it/s]
Downloaded 0.07 GB: 86%|########6 | 70/81 [00:02<00:00, 41.34it/s]
Downloaded 0.07 GB: 86%|########6 | 70/81 [00:02<00:00, 41.34it/s]
Downloaded 0.07 GB: 86%|########6 | 70/81 [00:02<00:00, 41.34it/s]
Downloaded 0.07 GB: 93%|#########2| 75/81 [00:02<00:00, 41.82it/s]
Downloaded 0.07 GB: 93%|#########2| 75/81 [00:02<00:00, 41.82it/s]
Downloaded 0.08 GB: 93%|#########2| 75/81 [00:02<00:00, 41.82it/s]
Downloaded 0.08 GB: 93%|#########2| 75/81 [00:02<00:00, 41.82it/s]
Downloaded 0.08 GB: 93%|#########2| 75/81 [00:02<00:00, 41.82it/s]
Downloaded 0.08 GB: 93%|#########2| 75/81 [00:02<00:00, 41.82it/s]
Downloaded 0.08 GB: 93%|#########2| 75/81 [00:02<00:00, 41.82it/s]
Downloaded 0.08 GB: 100%|##########| 81/81 [00:02<00:00, 34.67it/s]
Extracting dataset/arxiv.zip
Loading necessary files...
This might take a while.
Processing graphs...
0%| | 0/1 [00:00<?, ?it/s]
100%|##########| 1/1 [00:00<00:00, 24528.09it/s]
Converting graphs into DGL objects...
0%| | 0/1 [00:00<?, ?it/s]
100%|##########| 1/1 [00:00<00:00, 251.01it/s]
Saving...
OGB dataset is a collection of graphs and their labels. ogbn-arxiv
dataset only contains a single graph. So you can
simply get the graph and its node labels like this:
graph, node_labels = dataset[0]
# Add reverse edges since ogbn-arxiv is unidirectional.
graph = dgl.add_reverse_edges(graph)
graph.ndata['label'] = node_labels[:, 0]
print(graph)
print(node_labels)
node_features = graph.ndata['feat']
num_features = node_features.shape[1]
num_classes = (node_labels.max() + 1).item()
print('Number of classes:', num_classes)
Out:
Graph(num_nodes=169343, num_edges=2332486,
ndata_schemes={'year': Scheme(shape=(1,), dtype=torch.int64), 'feat': Scheme(shape=(128,), dtype=torch.float32), 'label': Scheme(shape=(), dtype=torch.int64)}
edata_schemes={})
tensor([[ 4],
[ 5],
[28],
...,
[10],
[ 4],
[ 1]])
Number of classes: 40
You can get the training-validation-test split of the nodes with
get_split_idx method.
How DGL Handles Computation Dependency¶
In the previous tutorial, you have seen that the computation dependency for message passing of a single node can be described as a series of message flow graphs (MFG).
Defining Neighbor Sampler and Data Loader in DGL¶
DGL provides tools to iterate over the dataset in minibatches
while generating the computation dependencies to compute their outputs
with the MFGs above. For node classification, you can use
dgl.dataloading.NodeDataLoader for iterating over the dataset.
It accepts a sampler object to control how to generate the computation
dependencies in the form of MFGs. DGL provides
implementations of common sampling algorithms such as
dgl.dataloading.MultiLayerNeighborSampler which randomly picks
a fixed number of neighbors for each node.
Note
To write your own neighbor sampler, please refer to this user guide section.
The syntax of dgl.dataloading.NodeDataLoader is mostly similar to a
PyTorch DataLoader, with the addition that it needs a graph to
generate computation dependency from, a set of node IDs to iterate on,
and the neighbor sampler you defined.
Let’s say that each node will gather messages from 4 neighbors on each layer. The code defining the data loader and neighbor sampler will look like the following.
sampler = dgl.dataloading.MultiLayerNeighborSampler([4, 4])
train_dataloader = dgl.dataloading.NodeDataLoader(
# The following arguments are specific to NodeDataLoader.
graph, # The graph
train_nids, # The node IDs to iterate over in minibatches
sampler, # The neighbor sampler
device=device, # Put the sampled MFGs on CPU or GPU
# The following arguments are inherited from PyTorch DataLoader.
batch_size=1024, # Batch size
shuffle=True, # Whether to shuffle the nodes for every epoch
drop_last=False, # Whether to drop the last incomplete batch
num_workers=0 # Number of sampler processes
)
Note
Since DGL 0.7 neighborhood sampling on GPU is supported. Please refer to 6.7 Using GPU for Neighborhood Sampling if you are interested.
You can iterate over the data loader and see what it yields.
input_nodes, output_nodes, mfgs = example_minibatch = next(iter(train_dataloader))
print(example_minibatch)
print("To compute {} nodes' outputs, we need {} nodes' input features".format(len(output_nodes), len(input_nodes)))
Out:
[tensor([130050, 163662, 133245, ..., 25511, 18764, 76924]), tensor([130050, 163662, 133245, ..., 6542, 107185, 142968]), [Block(num_src_nodes=12920, num_dst_nodes=4108, num_edges=14936), Block(num_src_nodes=4108, num_dst_nodes=1024, num_edges=3295)]]
To compute 1024 nodes' outputs, we need 12920 nodes' input features
NodeDataLoader gives us three items per iteration.
An ID tensor for the input nodes, i.e., nodes whose input features are needed on the first GNN layer for this minibatch.
An ID tensor for the output nodes, i.e. nodes whose representations are to be computed.
A list of MFGs storing the computation dependencies for each GNN layer.
You can get the source and destination node IDs of the MFGs and verify that the first few source nodes are always the same as the destination nodes. As we described in the overview, destination nodes’ own features from the previous layer may also be necessary in the computation of the new features.
Out:
tensor([130050, 163662, 133245, ..., 25511, 18764, 76924])
tensor([130050, 163662, 133245, ..., 144463, 92287, 50391])
True
Defining Model¶
Let’s consider training a 2-layer GraphSAGE with neighbor sampling. The model can be written as follows:
import torch.nn as nn
import torch.nn.functional as F
from dgl.nn import SAGEConv
class Model(nn.Module):
def __init__(self, in_feats, h_feats, num_classes):
super(Model, self).__init__()
self.conv1 = SAGEConv(in_feats, h_feats, aggregator_type='mean')
self.conv2 = SAGEConv(h_feats, num_classes, aggregator_type='mean')
self.h_feats = h_feats
def forward(self, mfgs, x):
# Lines that are changed are marked with an arrow: "<---"
h_dst = x[:mfgs[0].num_dst_nodes()] # <---
h = self.conv1(mfgs[0], (x, h_dst)) # <---
h = F.relu(h)
h_dst = h[:mfgs[1].num_dst_nodes()] # <---
h = self.conv2(mfgs[1], (h, h_dst)) # <---
return h
model = Model(num_features, 128, num_classes).to(device)
If you compare against the code in the introduction, you will notice several differences:
DGL GNN layers on MFGs. Instead of computing on the full graph:
h = self.conv1(g, x)
you only compute on the sampled MFG:
h = self.conv1(mfgs[0], (x, h_dst))
All DGL’s GNN modules support message passing on MFGs, where you supply a pair of features, one for source nodes and another for destination nodes.
Feature slicing for self-dependency. There are statements that perform slicing to obtain the previous-layer representation of the
nodes:
h_dst = x[:mfgs[0].num_dst_nodes()]
num_dst_nodesmethod works with MFGs, where it will return the number of destination nodes.Since the first few source nodes of the yielded MFG are always the same as the destination nodes, these statements obtain the representations of the destination nodes on the previous layer. They are then combined with neighbor aggregation in
dgl.nn.SAGEConvlayer.
Note
See the custom message passing
tutorial for more details on how to
manipulate MFGs produced in this way, such as the usage
of num_dst_nodes.
Defining Training Loop¶
The following initializes the model and defines the optimizer.
opt = torch.optim.Adam(model.parameters())
When computing the validation score for model selection, usually you can also do neighbor sampling. To do that, you need to define another data loader.
The following is a training loop that performs validation every epoch. It also saves the model with the best validation accuracy into a file.
import tqdm
import sklearn.metrics
best_accuracy = 0
best_model_path = 'model.pt'
for epoch in range(10):
model.train()
with tqdm.tqdm(train_dataloader) as tq:
for step, (input_nodes, output_nodes, mfgs) in enumerate(tq):
# feature copy from CPU to GPU takes place here
inputs = mfgs[0].srcdata['feat']
labels = mfgs[-1].dstdata['label']
predictions = model(mfgs, inputs)
loss = F.cross_entropy(predictions, labels)
opt.zero_grad()
loss.backward()
opt.step()
accuracy = sklearn.metrics.accuracy_score(labels.cpu().numpy(), predictions.argmax(1).detach().cpu().numpy())
tq.set_postfix({'loss': '%.03f' % loss.item(), 'acc': '%.03f' % accuracy}, refresh=False)
model.eval()
predictions = []
labels = []
with tqdm.tqdm(valid_dataloader) as tq, torch.no_grad():
for input_nodes, output_nodes, mfgs in tq:
inputs = mfgs[0].srcdata['feat']
labels.append(mfgs[-1].dstdata['label'].cpu().numpy())
predictions.append(model(mfgs, inputs).argmax(1).cpu().numpy())
predictions = np.concatenate(predictions)
labels = np.concatenate(labels)
accuracy = sklearn.metrics.accuracy_score(labels, predictions)
print('Epoch {} Validation Accuracy {}'.format(epoch, accuracy))
if best_accuracy < accuracy:
best_accuracy = accuracy
torch.save(model.state_dict(), best_model_path)
# Note that this tutorial do not train the whole model to the end.
break
Out:
0%| | 0/89 [00:00<?, ?it/s]/home/ubuntu/.pyenv/versions/miniconda3-latest/lib/python3.7/site-packages/torch/autocast_mode.py:141: UserWarning: User provided device_type of 'cuda', but CUDA is not available. Disabling
warnings.warn('User provided device_type of \'cuda\', but CUDA is not available. Disabling')
3%|3 | 3/89 [00:00<00:03, 26.23it/s, loss=3.936, acc=0.023]
7%|6 | 6/89 [00:00<00:03, 26.08it/s, loss=3.504, acc=0.113]
10%|# | 9/89 [00:00<00:03, 25.98it/s, loss=3.219, acc=0.217]
13%|#3 | 12/89 [00:00<00:02, 26.87it/s, loss=3.016, acc=0.209]
17%|#6 | 15/89 [00:00<00:02, 26.20it/s, loss=3.022, acc=0.170]
20%|## | 18/89 [00:00<00:02, 26.17it/s, loss=2.882, acc=0.237]
24%|##3 | 21/89 [00:00<00:02, 26.37it/s, loss=2.770, acc=0.276]
27%|##6 | 24/89 [00:00<00:02, 27.36it/s, loss=2.723, acc=0.321]
30%|### | 27/89 [00:01<00:02, 27.20it/s, loss=2.726, acc=0.300]
34%|###3 | 30/89 [00:01<00:02, 27.13it/s, loss=2.579, acc=0.324]
37%|###7 | 33/89 [00:01<00:02, 27.11it/s, loss=2.534, acc=0.327]
40%|#### | 36/89 [00:01<00:01, 27.34it/s, loss=2.585, acc=0.295]
44%|####3 | 39/89 [00:01<00:01, 27.38it/s, loss=2.469, acc=0.338]
47%|####7 | 42/89 [00:01<00:01, 27.15it/s, loss=2.417, acc=0.348]
51%|##### | 45/89 [00:01<00:01, 27.45it/s, loss=2.438, acc=0.354]
55%|#####5 | 49/89 [00:01<00:01, 28.26it/s, loss=2.302, acc=0.384]
58%|#####8 | 52/89 [00:01<00:01, 27.42it/s, loss=2.224, acc=0.433]
62%|######1 | 55/89 [00:02<00:01, 27.53it/s, loss=2.306, acc=0.378]
65%|######5 | 58/89 [00:02<00:01, 27.12it/s, loss=2.236, acc=0.419]
69%|######8 | 61/89 [00:02<00:01, 26.74it/s, loss=2.076, acc=0.442]
72%|#######1 | 64/89 [00:02<00:00, 27.02it/s, loss=2.177, acc=0.440]
75%|#######5 | 67/89 [00:02<00:00, 27.06it/s, loss=2.116, acc=0.437]
79%|#######8 | 70/89 [00:02<00:00, 26.51it/s, loss=1.954, acc=0.493]
82%|########2 | 73/89 [00:02<00:00, 26.62it/s, loss=1.964, acc=0.482]
85%|########5 | 76/89 [00:02<00:00, 26.48it/s, loss=2.025, acc=0.454]
89%|########8 | 79/89 [00:02<00:00, 26.91it/s, loss=1.923, acc=0.474]
92%|#########2| 82/89 [00:03<00:00, 26.85it/s, loss=1.911, acc=0.496]
96%|#########5| 85/89 [00:03<00:00, 27.16it/s, loss=1.907, acc=0.488]
99%|#########8| 88/89 [00:03<00:00, 27.17it/s, loss=1.897, acc=0.519]
100%|##########| 89/89 [00:03<00:00, 27.03it/s, loss=2.024, acc=0.466]
0%| | 0/30 [00:00<?, ?it/s]
17%|#6 | 5/30 [00:00<00:00, 47.51it/s]
33%|###3 | 10/30 [00:00<00:00, 48.69it/s]
53%|#####3 | 16/30 [00:00<00:00, 48.87it/s]
70%|####### | 21/30 [00:00<00:00, 48.64it/s]
90%|######### | 27/30 [00:00<00:00, 49.24it/s]
100%|##########| 30/30 [00:00<00:00, 50.00it/s]
Epoch 0 Validation Accuracy 0.5355548843920936
Conclusion¶
In this tutorial, you have learned how to train a multi-layer GraphSAGE with neighbor sampling.
What’s next?¶
During inference you may wish to disable neighbor sampling. If so, please refer to the user guide on exact offline inference.
# Thumbnail credits: Stanford CS224W Notes
# sphinx_gallery_thumbnail_path = '_static/blitz_1_introduction.png'
Total running time of the script: ( 0 minutes 11.299 seconds)