Read the general checks, then the section matching your setup. One machine
behind a router and a rack behind a shared uplink fail in different ways.
General checks
Run these on every machine you host, on whatever cadence you have settled on. Most produce a number worth writing down, because what matters is this window’s number against your last one.1. Find the interface
Everything below acts on the interface carrying the default route, so resolve it once and let the later commands read it:/devices/pci... is a real NIC. If instead you get $NIC as a
bond, VLAN, bridge, or VPN tunnel (bond0, enp1s0.100, wg0, tailscale0),
it has no physical link of its own, and steps 3 and 4 have to be run against
each physical member interface instead. ethtool will happily report
Duplex: Full and Link detected: yes on a tunnel, so this is worth
confirming rather than assuming.
NIC lives only in the shell session you set it in. If your connection drops,
or you open a second terminal, set it again before continuing.
2. Measure the line
Measure from the machine itself. A laptop on the same network takes a different path through the switch, and on Wi-Fi it is not measuring your machine’s link at all.speedtest-cli is
an unmaintained Python client measuring over HTTP against a public server you
did not choose, so the figure moves with that server’s load and the path to it.
On a fast line, confirm with iperf3 against a public server:
[SUM] line of each, the one marked receiver:
iperf3 servers cap stream count and duration and listen on a
non-default port, so check the terms of whichever you pick.
Measure at a busy hour, not only at night. A line returning 900/900 at 03:00
and 300/150 at 20:00 fails the requirement for most of the day your renters are
actually using it.
3. Check the link itself
Your line can only be as fast as the link the NIC negotiated with whatever it is plugged into:Duplex: Full and Link detected: yes are the only acceptable values on a
physical port. Half duplex collapses throughput under load and points at a
switch port that failed to negotiate.
Speed is the value to record and compare. It is not a fixed property of the
machine: a link renegotiates on every cable reseat, switch reboot, and port
move, and a marginal cable comes back at a lower speed rather than not at all.
A machine that reported 10000Mb/s last window and reports 1000Mb/s now has
a physical problem, even though every other check here will pass.
If the speed is lower than your hardware should manage, read the full output
and compare the supported modes against the advertised ones:
ethtool -s command run by hand or from a systemd unit, a udev rule, or a
systemd-networkd [Link] section. Netplan has no key for link speed, so there
is no point looking for one in your YAML. A NIC that supports 10 Gb where the
link partner does not is a switch port or transceiver question.
On a fibre link, read the transceiver’s own diagnostics as well. Optical
receive power falling window over window is the classic months-long
degradation, and it shows up here before it shows up anywhere else:
4. Read the error counters
This is the check that finds a failing cable before it takes the machine down, and a single reading is worthless. Counters are cumulative since boot, so the number alone means nothing; the number against last window’s, over a known number of days, is the signal. The Network tab in Machine Metrics already charts RX and TX errors and drops per machine, sampled rather than accumulated, so a healthy interface reads a flat zero and anything above it stands out without any arithmetic on your part. Read the trend there first. The commands below are the point-in-time confirmation on the machine itself.1, or no stats available, means this driver has nothing to
offer and ip -s -s link is your only source. Record whatever either prints:
rx_crc_errors climbing means frames are arriving
corrupted, which is a cabling, transceiver, or switch port fault rather than a
software one. rx_missed_errors means the NIC received frames it had no buffer
for, which points at the machine rather than the cable.
Take the uptime alongside it, so your next window can work out a rate:
A count that has not moved since last window is fine however large it is. A
machine that logged 2,000 CRC errors during a switch replacement months ago and
none since has a clean link. One adding to the count steadily does not, and it
reaches your renters as slow transfers long before it reaches you as a failure.
5. Confirm the public IPv4 address
How the platform picks up a changed address is part of machine setup rather
than maintenance. See the
host setup guide.
Sharing one public address across several of your own machines is a different
thing and is fine, as long as each machine has its own forwarded range. See
Allocate non-overlapping port ranges.
6. Pin the machine to one LAN address
Your router forwards the port range to a LAN address. If the machine’s address moves, the rule points at whatever else picked it up, or at nothing.sudo is not optional. On 24.04 the netplan files are root-only, and an
unprivileged grep prints permission errors to stderr and no matches to
stdout, which reads exactly like a machine that is already static.
dhcp4: true means the address is a lease, and a lease is not a promise. It
usually survives a reboot, and then one day it does not: the machine is off
long enough for the lease to expire during a drive replacement, the router
rebuilds its lease table, or another device claims the address first. The
machine comes back healthy on a new address with your forwarding rule pointing
at the old one. Every other maintenance page ends in a reboot, which is what
makes this worth checking rather than assuming.
Fix it in one of two places. Prefer a DHCP reservation on your router, keyed to
the machine’s MAC address: your forwarding rule and the address it points at
then live in the same place, and the machine still boots correctly if you swap
it out.
/etc/cloud/cloud.cfg.d/99-disable-network-config.cfg as well, or cloud-init
regenerates 50-cloud-init.yaml on the next boot and undoes the change.
7. Size the port range against the GPU count
The number of ports the machine needs is a function of how many GPUs are in it, so it changes whenever your hardware does.0 means nvidia-smi failed rather than that the machine has no
GPUs; the error goes to stderr and wc -l counts nothing.
Formula for verification: forwarded ports >= 5 x number of GPUs, with
100 x number of GPUs recommendedExample: 8 GPUs means at least 40 forwarded ports to qualify for
verification, and 800 recommended
Compare that against the range you have forwarded on your router. If you have
added GPUs since setup, the machine is running on a range sized for the
hardware it used to have. Nothing reports this as an error; the machine simply
takes fewer rentals and loses the ones needing many ports.
The Ports value on the machine’s card under Host → Machines in the
console is what the platform believes the machine offers, so check your
arithmetic against it. A machine showing no port data there has not reported a
usable range at all, which How to Self-Test lists as
something to fix before the self-test will run.
Resizing a range is two changes, not one. Widening the rule on your router is
the first; telling the platform its new range is the second, which the
host setup guide covers.
8. Check the firewall and the ephemeral port range
Two host-side settings can break a range your router is forwarding perfectly.ufw is Ubuntu’s default firewall tool and is present on a server install, but
it ships disabled. A machine where you have never turned it on reports:
net.ipv4.ip_local_reserved_ports.
Instances are Docker containers, and Docker publishes their ports through its
own chains rather than through
ufw, so a deny (incoming) default does not
close them. To restrict the Vast range at the host, use the DOCKER-USER
chain, which Docker leaves alone:ufw rules persist across a reboot. Rules added by hand with iptables do
not, unless something restores them:
DOCKER, DOCKER-USER, and DOCKER-ISOLATION chains in that
output; dockerd rebuilds those itself at boot. Only rules you added by hand are
at risk, and the fix for those is to move them into ufw.
9. Test both ends of the range from outside
Testing one port in the middle of your range is the check that misses the failure this step exists for. A rule re-entered after a firmware update over a shorter range than you configured still passes that test and leaves the rest of the range dead. You do not have to test every port. Where your range is one forwarding rule, the lowest and the highest port tell you whether the whole thing arrives, and the highest is the one that matters: a rule truncated at the top end is the common failure, and every renter landing on the working part has a normal experience. If your router made you enter the range as several rules, test the first and last port of each of them. Test from outside your network, on a VPS, a phone hotspot, or anything on a different connection. Running it from inside your LAN tells you nothing, because your router may answer its own public address differently than the internet does. On the machine, start a listener on the lowest port in your range:-u to prove UDP arrives too, since
renters can map UDP ports as well:
If
ufw is active, this test goes through it, because the listener is an
ordinary host socket. A range blocked in ufw fails here even though instance
ports on the same range would work, since Docker publishes those through its
own chains. That is why step 8 comes first.10. Check for loss and DNS drift
A machine passing every check above can still be unreliable if the path to it drops packets.0% packet loss is the expected result, and any sustained loss is worth
chasing: it costs far more throughput than its percentage suggests, because
every lost segment is a retransmit and a congestion-window cut. A high mdev
is jitter, which renters notice on interactive sessions first.
Where the loss happens matters more than that it happens:
DNS Servers is a single point of failure, and that entry is
usually your router. The daemon reaches the Vast controller by name, so a
router reboot or a flapping upstream resolver takes the machine offline while
your line is working perfectly. Add a second resolver, such as 1.1.1.1 or
8.8.8.8, so the machine keeps resolving when the first stops answering.
If this shows no DNS servers while resolution plainly works, the machine has a
hand-written
/etc/resolv.conf bypassing systemd-resolved. Read that file
instead.speedtest-cli at or above 500 Mbit/s both ways, ethtool reporting the speed
you expect at full duplex, no error counter moving since your last window, your
public address unchanged and matching your router’s WAN address, your LAN
address pinned, your range clear of the ephemeral range with the firewall rules
surviving a reboot, both ends of the range answering over TCP and UDP from
outside, and no sustained loss to the final hop. Record the speeds, the link
speed, the error counts, your public address, and the date.For the platform’s own verdict on the same ground, run
vastai self-test machine while the machine is
idle. It checks network speed, stability, and open ports directly.Single-machine hosts
One machine behind one router. Almost everything that drifts here drifts on the router rather than on the machine, so if you only ever log into the machine, you never see it coming.1. Re-verify the range after any router work
Firmware updates, factory resets, and configuration restores all lose or rewrite forwarding rules, and consumer routers rarely tell you they have. Treat any router work as a change to the machine and re-run the port test afterwards. The same applies to a rule still listed that no longer works. Check that its destination address still matches the machine, that it covers both TCP and UDP, and that its range matches what you sized in step 7.2. Check the router is not the bottleneck
Your router can pass a speed test and still cap the machine, because a speed test opens a handful of connections and a busy machine opens thousands. The symptom is a line measuring at full speed while renters report slow or stalled connections during busy periods. Consumer routers have a finite NAT connection table, and the recommended 100 ports per GPU is a lot of concurrent state for a small device. If yours exposes a connection count or NAT table size, record it during a busy period alongside the rest of this check. A router at its limit drops new connections while existing ones keep working, which is what makes it hard to catch. The machine keeps its own table, and it fills the same way as rental density grows:nf_conntrack: table full, dropping packet in dmesg.
The same class of limit applies to the forwarding rules themselves. Some
routers cap the number of entries, and some cannot express a large contiguous
range at all and want it entered as many individual rules. Both bite at the
recommended port count rather than the minimum, so check what yours can express
before sizing a range around 100 ports per GPU.
Multi-machine hosts on one network
Once there is more than one machine behind the same uplink, capacity is shared and the requirement is per machine. Everything here is about what the fleet does to itself, and it gets worse as the fleet grows without any individual machine changing.1. Measure the uplink under load
A machine tested on an otherwise idle network tells you nothing about what it delivers when its neighbors are busy. The requirement is 500 Mbps each way for each machine, at the same time.Derived from the per-machine requirement, not a published Vast figure:
uplink >= 500 Mbps x number of machines, in each direction. Six machines need
3 Gbps symmetric, sustained.
-R, and add the final [SUM] figures
across all the machines. A total that stops rising as you add machines is the
uplink saturating, and every machine on it is below requirement at that moment
regardless of what each measured alone.
2. Check the switch uplink, not just the ISP line
The most common capacity fault in a multi-machine setup is not the ISP link. It is the single cable between the switch your machines are plugged into and the router. Eight machines on 1 Gb access ports feeding one 1 Gb uplink gives the whole fleet 1 Gb to share, about 125 Mbps each under full load, on an internet connection measuring 10 Gb all day from any single machine. Test the fabric directly rather than through the internet. On one machine:ufw rule is needed because iperf3 -s listens on an ordinary host socket
that a deny (incoming) default drops, and the client then just times out with
no indication why. Remove it when you are done.
On another machine:
3. Allocate non-overlapping port ranges
Machines behind one public IPv4 address share one port space. Each needs its own range, sized to its own GPU count, and the ranges must not overlap. Work it out per machine before you touch the router. For a fleet of three, one with 8 GPUs, one with 4, and one with 8, the arithmetic from step 7 gives 800, 400, and 800 ports at the recommended count: 2,000 ports in total, as three non-overlapping ranges, each of which is itself contiguous. Leave gaps between them if you expect GPU counts to change, because growing a range into one another machine is using is an outage for both. Keep the range, the machine, and its LAN address written down together. Your router only shows you the first and last of those three. Watch the ceiling as the fleet grows: one address has 65,535 ports, the ephemeral range claims roughly 28,000 of them by default, and at 800 ports per 8-GPU machine that runs out sooner than it looks. Verify each range with the port test, run once per machine against that machine’s own lowest and highest port. That proves each range is forwarded. It does not prove each reaches the right machine, and adjacent ranges typed into a router are exactly where that goes wrong. Confirm the destination for at least one port per machine. On the machine:4. Size the shared infrastructure for the fleet
Three limits on shared equipment scale with machine count and quietly stop being adequate:- The DHCP pool. Large enough for every machine plus everything else on the network, with a reservation for each so that pinned addresses hold across the fleet.
- The connection tracking table on the router. Every renter connection to every machine is an entry, and this is the limit that turns into fleet-wide connection failures during busy periods while each machine looks healthy.
- Bandwidth fairness. Nothing stops one machine saturating the shared line and pushing the others below 500 Mbps. If your router supports per-client rate limiting, cap each machine at its share.
5. Treat adding a machine as maintenance on all of them
Adding a machine divides the uplink again, consumes another block of the shared port space, and adds load to the router’s connection table. The new machine is the one being installed; the existing machines are the ones that change behavior. Afterwards, re-run the uplink measurement across the whole fleet and confirm the new range overlaps nothing.Changes that need a window
Most of this page reads state. These are the changes it leads to, and all of them interrupt rentals. Schedule them withvastai schedule maintenance.
Applying a netplan change without locking yourself out
A wrong netplan file applied over SSH leaves an unreachable machine, and the recovery is a trip to the machine or its BMC.netplan try exists for this: it
applies the configuration, waits, and rolls back automatically unless you
confirm at the console.
Changing the address means your existing session is connected to the old one.
Open a session against the new address before closing the old one, and keep the
old one open until the new one works.
Recovery
Neither end of the range connects
Your rule is not delivering traffic to the machine at all. In order of likelihood: the rule was lost by a router firmware update or reset; it points at a LAN address the machine no longer holds, which is the DHCP problem; it covers TCP where your range needs UDP too; orufw is blocking the listener, which
step 8 will show.
Re-enter the rule, confirm its destination matches what
ip -4 -brief addr show "$NIC" reports, and test again.