RTNETLINK answers: No such process

Source

A colleague of mine presented me with a weird routing problem today and it took me a while to understand what was going on. The task was simple: add a network route via a certain gateway that can only be reached via a certain network interface. Let's re-create the setup: 

# ip addr change 10.10.0.3/24 dev eth2
# ip link set eth2 up
# ip addr show dev eth2 scope global
3: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 1000
    link/ether 08:00:27:d0:34:51 brd ff:ff:ff:ff:ff:ff
    inet 10.10.0.3/24 scope global eth2
Let's add a new route then: 
# ip route add 10.20.0.0/24 via 10.10.0.1 dev eth2
RTNETLINK answers: No such process
Huh? Our eth2 is UP and should be able to reach 10.10.0.1, right? Let's look at the routing table1): 
# netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
192.168.56.0    0.0.0.0         255.255.255.0   U         0 0          0 eth0
0.0.0.0         192.168.56.1    0.0.0.0         UG        0 0          0 eth0
Aha! For some reason the machine has lost its network route on the eth2 interface. Well, the machine has been online for a while and we don't know which admin did what and why. But although eth2 is configured and UP, it cannot reach its own network w/o a network route. Of course, the "ip addr change" does that automatically2) and we staged the whole thing for illustration purposes.

Let's add the missing route and try again: 
# ip route add 10.10.0.0/24 dev eth2 
# netstat -rn 
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
10.10.0.0       0.0.0.0         255.255.255.0   U         0 0 0 eth2
192.168.56.0    0.0.0.0         255.255.255.0   U         0 0 0 eth0
0.0.0.0         192.168.56.1    0.0.0.0         UG        0 0 0 eth0

# ip route add 10.20.0.0/24 via 10.10.0.1 dev eth2
# netstat -rn 
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
10.20.0.0       10.10.0.1       255.255.255.0   UG        0 0 0 eth2
10.10.0.0       0.0.0.0         255.255.255.0   U         0 0 0 eth2
192.168.56.0    0.0.0.0         255.255.255.0   U         0 0 0 eth0
0.0.0.0         192.168.56.1    0.0.0.0         UG        0 0 0 eth0
Yay! :-)

1) Sometimes the output from the iproute2 tools are not as easy to parse and I'll use good ol' net-tools again.
2) Unless we were to assign a /32 address to the interface, e.g. "ip addr change 10.10.0.0/32 dev eth2"