Training Run Report

3
Rounds
2
Workers
15.6s
Total Time
4.7MB
Data Transferred
2.3MB
Params Pushed
2.3MB
Weights Pulled
8.7
Final |W|
Push params
Train
Pull weights
Aggregate
Dead / Error
round 0 [2/2] 4.59s 0.0s 0.9s 1.8s 2.8s 3.7s 4.6s scheduler push 399.7KB → node 0 (0.000s) push 399.7KB → node 1 (0.000s) pull 399.7KB ← node 0 (0.000s) pull 399.7KB ← node 1 (0.000s) aggregate: 2 workers, |W|=8.75, Δ=0.0891 worker 0 push 399.7KB → node 0 (0.000s) train node 0: 4.59s exit=0 cpu = _conversion_method_template(device=torch.device("cpu")) pull 399.7KB ← node 0 (0.000s) worker 1 push 399.7KB → node 1 (0.000s) train node 1: 4.31s exit=0 cpu = _conversion_method_template(device=torch.device("cpu")) pull 399.7KB ← node 1 (0.000s) round 1 [2/2] 5.53s 0.0s 1.1s 2.2s 3.3s 4.4s 5.5s scheduler push 399.6KB → node 0 (0.000s) push 399.6KB → node 1 (0.002s) pull 399.7KB ← node 0 (0.000s) pull 399.7KB ← node 1 (0.000s) aggregate: 2 workers, |W|=8.74, Δ=0.0876 worker 0 push 399.6KB → node 0 (0.000s) train node 0: 5.35s exit=0 cpu = _conversion_method_template(device=torch.device("cpu")) pull 399.7KB ← node 0 (0.000s) worker 1 push 399.6KB → node 1 (0.002s) train node 1: 5.53s exit=0 cpu = _conversion_method_template(device=torch.device("cpu")) pull 399.7KB ← node 1 (0.000s) round 2 [2/2] 5.50s 0.0s 1.1s 2.2s 3.3s 4.4s 5.5s scheduler push 399.6KB → node 0 (0.000s) push 399.6KB → node 1 (0.000s) pull 399.7KB ← node 0 (0.000s) pull 399.7KB ← node 1 (0.000s) aggregate: 2 workers, |W|=8.74, Δ=0.0883 worker 0 push 399.6KB → node 0 (0.000s) train node 0: 5.50s exit=0 cpu = _conversion_method_template(device=torch.device("cpu")) pull 399.7KB ← node 0 (0.000s) worker 1 push 399.6KB → node 1 (0.000s) train node 1: 5.38s exit=0 cpu = _conversion_method_template(device=torch.device("cpu")) pull 399.7KB ← node 1 (0.000s) 
[   0.00s] push  node=0  worker.py               1.8KB  0.000s
[   0.00s] push  node=0  job_module.py           1.4KB  0.000s
[   0.00s] push  node=1  worker.py               1.8KB  0.000s
[   0.00s] push  node=1  job_module.py           1.4KB  0.000s
[   0.00s] ──── round 0 start (2 workers) ────────────────
[   0.00s] push  node=0  params.pt             399.7KB  0.000s
[   0.00s] push  node=1  params.pt             399.7KB  0.000s
[   4.32s] exec  node=1  exit=0    4.313s  | cpu = _conversion_method_template(device=torch.device("cpu"))
[   4.59s] exec  node=0  exit=0    4.587s  | cpu = _conversion_method_template(device=torch.device("cpu"))
[   4.59s] pull  node=0  weights.pt            399.7KB  0.000s
[   4.59s] pull  node=1  weights.pt            399.7KB  0.000s
[   4.59s] agg   2 workers  |W|=8.75  Δ=0.0891  0.001s
[   4.59s] ──── round 0 end (2/2 survived, 4.59s) ────────
[   4.59s] ──── round 1 start (2 workers) ────────────────
[   4.59s] push  node=0  params.pt             399.6KB  0.000s
[   4.60s] push  node=1  params.pt             399.6KB  0.002s
[   9.95s] exec  node=0  exit=0    5.350s  | cpu = _conversion_method_template(device=torch.device("cpu"))
[  10.12s] exec  node=1  exit=0    5.525s  | cpu = _conversion_method_template(device=torch.device("cpu"))
[  10.12s] pull  node=0  weights.pt            399.7KB  0.000s
[  10.12s] pull  node=1  weights.pt            399.7KB  0.000s
[  10.13s] agg   2 workers  |W|=8.74  Δ=0.0876  0.001s
[  10.13s] ──── round 1 end (2/2 survived, 5.53s) ────────
[  10.13s] ──── round 2 start (2 workers) ────────────────
[  10.13s] push  node=0  params.pt             399.6KB  0.000s
[  10.13s] push  node=1  params.pt             399.6KB  0.000s
[  15.51s] exec  node=1  exit=0    5.378s  | cpu = _conversion_method_template(device=torch.device("cpu"))
[  15.63s] exec  node=0  exit=0    5.498s  | cpu = _conversion_method_template(device=torch.device("cpu"))
[  15.63s] pull  node=0  weights.pt            399.7KB  0.000s
[  15.63s] pull  node=1  weights.pt            399.7KB  0.000s
[  15.63s] agg   2 workers  |W|=8.74  Δ=0.0883  0.001s
[  15.63s] ──── round 2 end (2/2 survived, 5.50s) ────────