Headless server doesn't detect NAT with port randomization correctly

[rkfg]

I tried to host a server behind two NATs, the router and ISP ones. Pingpong servers replied with my external address and port and I thought it was okay but I couldn't connect to my own server via Internet neither could another player. So I started digging.

I use Debian testing amd64 as my only OS + I have a Zyxel Keenetic router that can be easily "rooted" with a prepared USB flash drive (no firmware update needed for that and it's a native supported functionality, it also supports opkg natively and installs software to the same flash drive. So I can test different cases and watch the actual traffic with tcpdump installed on the router. I also have a VPS in the Netherlands to check what exactly comes from my side. I tried to send a UDP packet from port 10002 to the VPS to port 10002. What I did:

On the router I launched

Code: Select all

tcpdump -n -i ppp0 host <VPS IP> and udp

On the VPS I ran

Code: Select all

tcpdump -n -i eth0 port 10002

Then on the router I ran

Code: Select all

echo "hello" | netcat -u -p 10002 <VPS IP> 10002

This is what router's tcpdump displayed:

Code: Select all

14:53:10.331206 IP 172.21.20.30.10002 > <VPS IP>.10002: UDP, length 6

So far so good. 172.21.20.30 is the IP my ISP assigned to the router, it's a private IP that gets NATted to the WAN IP they use for multiple clients in the same pool. They have several such WAN IPs and sometimes they even assign one to the client but it's rare. When it happens I can just forward a port for a game and it simply works as it should. Technically, I can reconnect many times until I get a WAN IP but it takes time and is unreliable, and I feel like it becomes harder and harder to "win" such an IP with time. So I deal with it and use the local IP from the private net range.

Okay, but what comes to the VPS?

Code: Select all

14:53:10.752733 IP <ISP WAN IP>.14766 > <VPS IP>.10002: UDP, length 6

The port has been randomized to 14766. It stays the same if I send data in a short while but if I wait for about 30 seconds the ISP's conntrack forgets about this mapping and a different port would be assigned. Which is expected.

Now, the problem is that Factorio server doesn't seem to handle this situation correctly. There are no warnings at all though I expect the whole system with multiple pingpong servers to detect this because it's the entire purpose of it! Check with different servers if the external port they see stays the same (and matches the internal port) or if it's randomized and unreliable. In the latter case the game simply can't be set up (without using relays of course) and there should be a big fat error/warning about that, probably the game shouldn't even be published in the public servers directory because it's physically not possible to connect to it. And yet this is what I get in the log:

Code: Select all

   0.000 2020-04-12 17:03:48; Factorio 0.18.18 (build 50828, linux64, headless)
   0.167 Operating system: Linux (Debian testing)
   0.167 Program arguments: "bin/x64/factorio" "--start-server-load-latest" "--server-settings" "data/server-settings.json" 
   0.167 Read data path: /home/rkfg/Games/factorio/data
   0.167 Write data path: /home/rkfg/Games/factorio [265917/937871MB]
   0.167 Binaries path: /home/rkfg/Games/factorio/bin
   0.170 System info: [CPU: Intel(R) Core(TM) i7-7700K CPU @ 4.20GHz, 8 cores, RAM: 32080 MB]
   0.170 Environment: DISPLAY=:0 WAYLAND_DISPLAY=<unset> DESKTOP_SESSION=awesome XDG_SESSION_DESKTOP=awesome XDG_CURRENT_DESKTOP=<unset> __GL_FSAA_MODE=<unset> __GL_LOG_MAX_ANISO=<unset> __GL_SYNC_TO_VBLANK=<unset> __GL_SORT_FBCONFIGS=<unset> __GL_YIELD=<unset>
   0.170 Running in headless mode
   0.172 Loading mod core 0.0.0 (data.lua)
   0.205 Loading mod base 0.18.18 (data.lua)
   0.368 Loading mod base 0.18.18 (data-updates.lua)
   0.456 Checksum for core: 1106864506
   0.456 Checksum of base: 332375697
   0.582 Prototype list checksum: 586383188
   0.612 Info PlayerData.cpp:68: Local player-data.json available, timestamp 1586684661
   0.612 Info PlayerData.cpp:75: Cloud player-data.json unavailable
   0.781 Factorio initialised
   0.781 Info HttpSharedState.cpp:54: Downloading https://auth.factorio.com/api-login?api_version=4
   2.060 Info AuthServerConnector.cpp:133: Got token from auth server for username(rkfg).
   2.061 Info ServerSynchronizer.cpp:28: nextHeartbeatSequenceNumber(0) initialized Synchronizer nextTickClosureTick(0).
   2.061 Info ServerMultiplayerManager.cpp:771: updateTick(4294967295) changing state from(Ready) to(PreparedToHostGame)
   2.061 Info ServerMultiplayerManager.cpp:771: updateTick(4294967295) changing state from(PreparedToHostGame) to(CreatingGame)
   2.061 Loading map /home/rkfg/Games/factorio/saves/mp.zip: 3710690 bytes.
   2.068 Loading level.dat: 9748540 bytes.
   2.071 Info Scenario.cpp:187: Map version 0.18.18-1
   2.192 Loading script.dat: 328 bytes.
   2.193 Checksum for script /home/rkfg/Games/factorio/temp/currently-playing/control.lua: 3441332399
   2.195 Info UDPSocket.cpp:27: Opening socket at (IP ADDR:({0.0.0.0:34197}))
   2.195 Hosting game at IP ADDR:({0.0.0.0:34197})
   2.195 Info HttpSharedState.cpp:54: Downloading https://auth.factorio.com/generate-server-padlock-2?api_version=4
   2.997 Info AuthServerConnector.cpp:66: Obtained serverPadlock for serverHash (npodnkXS14rBifh5LxTXZAiMBPNVR1fJ) from the auth server.
   2.997 Info ServerMultiplayerManager.cpp:771: updateTick(988533) changing state from(CreatingGame) to(InGame)
   3.285 Info ServerRouter.cpp:614: Asking pingpong servers (pingpong1.factorio.com:34197, pingpong2.factorio.com:34197, pingpong3.factorio.com:34197, pingpong4.factorio.com:34197) for own address
   3.285 Info UDPSocket.cpp:39: Opening socket for broadcast
   3.286 Info CommandLineMultiplayer.cpp:303: Maximum segment size = 100; minimum segment size = 25; maximum-segment-size peer count = 10; minimum-segment-size peer count = 20
   3.437 Info ServerRouter.cpp:497: Own address is IP ADDR:({<ISP WAN IP>:64494}) (confirmed by pingpong1)
   3.437 Info ServerRouter.cpp:497: Own address is IP ADDR:({<ISP WAN IP>:64494}) (confirmed by pingpong3)
   3.588 Info ServerRouter.cpp:497: Own address is IP ADDR:({<ISP WAN IP>:64494}) (confirmed by pingpong4)
   4.299 Info MatchingServer.cpp:114: Matching server game `7210794` has been created.
   4.309 Info ServerMultiplayerManager.cpp:703: Matching server connection resumed

In the traffic the exchange happens between 45.79.202.131.34197 > 172.25.10.50.34197 (the latter is my PC address in the home network) so the ports are 34197 on the both sides but pingpong sees that port like 64494. Weird thing is that this mapping seems to be consistent between different IPs but still I can't connect to the server from outside. I don't even see any service packets from the pingpong server when I try to connect (from https://factorio.com/blog/post/fff-143 I get that the Internet player should ask the pingpong server to ask the actual game server behind NAT to send a UDP packet to that player to establish a port mapping but in reality it doesn't happen). When I try to join the server no traffic comes from the pingpong servers at all, the last packet from them contains my external address in ASCII form and that's it. Also the game seemingly sends pings to those servers (1 byte UDP packets) every 15 seconds but gets no response. No clue if it's okay.

tl;dr hosting a game behind a provider NAT doesn't work and the game doesn't warn about that.

[rkfg]

Any thoughts? I'd like to be able to host a game on my home PC but currently I'm limited to forwarding ports using my VPN provider, it works but the ping time is much higher than it could be and there's also a noticeable packet loss because the VPN server is in another country. Just from the traffic analysis I guess it should be possible since the ports aren't randomized between IPs so if the keep alive interval is less than 30 seconds it should stay indefinitely long. But the servers don't pass the reverse connection request to my headless game server so it doesn't work.

[Rseding91]

As far as I know, and can see in code. All we do is send packets to what ever address the game gets when it tries to resolve the pingpong/matching servers.

Past that, the game has no concept of port randomization logic or detection. It sends packets to our servers to keep the connection open and listens on the same socket.

If that's not working for you, then the issue most likely lies in your network configuration.

[rkfg]

Okay, but why do I see the address replies that come from your servers and absolutely nothing from them when I try to connect to my game from the client that I launch from Steam? Should your servers somehow respond to the ping packets that the headless server sends their way? Because I don't see that either. I'm sure the bidirectional traffic works, I can see my external address your servers send me. But that's the last information I receive and then there's nothing. I tried to connect as fast as I could after starting the headless server (in case the port mapping expires too quickly on the ISP side) and still didn't succeed once.

[Rseding91]

I have no idea. But there are loads of servers out there that work just fine, ones that are using nat punching, ones that are using steam networking, and ones with simple port forwarding.

Every time I've seen anyone with server issues it always ends up being a network configuration issue on their side. Some firewall blocking traffic, the ISP doing something dumb, multiple layers of NAT and one of them just isn't working right.

The internet in general is a giant mess of miss-configured systems and this stuff happens all the time.

[rkfg]

I get that but can you help me to debug this issue? I can provide network dumps from tcpdump or any other info you might need. I'm a devops at work and I know how to configure iptables and routing so I doubt it's on my side. I'm not saying it's a bug on your side either but... who knows? If you're willing to help I'm ready to do the testing. It might be something as trivial as too big intervals between the keep alive packets or such.

From the FFFs I figured that you guys love to get to the root of the problem so just shrugging it off with "internet is a mess and it's probably some misconfiguration" is kinda vague and weird to me. I mean I know and agree that it all could be better but we have to work with what we have. In my case it's really easy to reproduce the issue and I'm ready to provide whatever technical information you want. I believe that it works for many other people but in my case it doesn't and from the looks of it I think it should work. So why don't we find out what's wrong? It's a technical help forum after all.

[ckreon]

I'm absolutely baffled by the Developer response to all of these multiplayer threads.

It seems to be an area where they lack detailed knowledge, and it's a niche enough issue to not throw the community into fits. Maybe that's a good enough excuse, but I'm turned off every time I find another thread with these issues, and then see an official response telling them to bug off about it.

Clearly, this IS a Factorio issue, maybe not in the technical sense, but in the practical sense that it fails for a noticeable percentage of users who have zero issue with other titles. Officially it can be written off as user error, but after 5 years of similar issues with the same spiel repeated back to us, I'm not sure how the community is supposed to perceive it as anything other than a Factorio issue.

FWIW, I have the same issue.

[rkfg]

Guys, what happened with It's never what you think it is?

Recently we had a bug report where a modded game would freeze for a minute for seemingly no reason and then continue like nothing went wrong. It being a heavily modded game my first reaction was to blame it on the mods doing something in a very unoptimized way. But I had to test it to figure out which mod was causing the problem. After reproducing it... I was reminded (again) that it's (almost) never what you think it is.

This is exactly what Rseding wrote last year and now that first reaction from the past is the only one. Not even trying to figure it out? I would (kinda) understand if it was an issue of a, let's say, not very advanced user that barely understands what that weird plastic box called "rotor" or something does. I understand that it's extremely hard to get any useful details in such situations because the user simply can't provide them, doesn't know any tools to do so. But I'm not even an "advanced user", I'm a developer and a network admin. I know this stuff, I've been using Linux exclusively for 12 years now, both at home and at work and I'm certainly capable of debugging network issues. The problem here is that I've done everything I could on my side and I'm willing to cooperate. You will not only help just me but many other players in a similar situation. Yes, Steam networking works more or less, but it relays traffic through their servers which causes even more lags and packet drops (the game often freezes and then jumps forward and also it's obvious that my actual position in the world lags behind the visible position when I fight or pick up materials; these postions are closer when there's no relay + there are much less freezes and jumps). The game should be able to tell if it's absolutely impossible to host behind such NAT or it should setup the NAT punching correctly. You've described this scheme in FFF-143 and I guess it should work in my case.

If I write a sample program that tries to replicate this algorithm and if it works for me will I get a bit more attention? Or if it doesn't maybe I will provide the exact reason why it happens so you can also implement it and explicitly tell whether or not it's worth trying.

[rkfg]

So I wrote a program to test if my NAT can be punched correctly and in fact it can. Here's the code: https://github.com/rkfg/nat_punch

How to test:
1) Run server.py on a VPS with a WAN IP (it accepts -p flag to set the port but you can leave it as is):

Code: Select all

python3 server.py

2) You can run server.py on multiple hosts to verify if your port is the same for different IPs (for me it's preserved but if it's different then you won't be able to host Factorio server behind such NAT)
3) Run client.py on one machine behind ISP NAT like this:

Code: Select all

python3 client.py vps-ip:37419

Provide your VPS WAN IP and the port (defaults to 37419) to connect. It should show you your session ID (a random number) and your WAN IP as seen from the outside (for me sometimes it doesn't work from the first try, just do it again). Here you can do it with several VPSes if you have them and see if the port you get back is the same every time. I didn't do that because Factorio headless server already proved the port is preserved.
4) Now run client.py on another machine behind another ISP NAT (your second player that wants to connect to your first PC behind NAT) like this:

Code: Select all

python3 client.py -s 861628577 vps-ip:37419

Here you should use the same VPS WAN IP from the step 3 and supply the session number the first user got in response. Presumably, this session number is published on the "Public games" list but you can send it via email, messenger or whatever.
5) Both clients should receive each other IPs and ports as seen from the outside and start sending data packets. If you see something like this on the both sides then it's working:

Code: Select all

Message with code 4 received: packet number 1
Data packet received from ('5.43.242.176', 14728): packet number 1
Sending packet from 134.19.179.179:37419 to ('5.43.242.176', 14728): 2
Message with code 4 received: packet number 2
Data packet received from ('5.43.242.176', 14728): packet number 2
Sending packet from 134.19.179.179:37419 to ('5.43.242.176', 14728): 3
Message with code 4 received: packet number 3

Code: Select all

Message with code 4 received: packet number 1
Data packet received from ('134.19.179.179', 37419): packet number 1
Sending packet from 5.43.242.176:14728 to ('134.19.179.179', 37419): 2
Message with code 4 received: packet number 2
Data packet received from ('134.19.179.179', 37419): packet number 2
Sending packet from 5.43.242.176:14728 to ('134.19.179.179', 37419): 3
Message with code 4 received: packet number 3
Data packet received from ('134.19.179.179', 37419): packet number 3
Sending packet from 5.43.242.176:14728 to ('134.19.179.179', 37419): 4

Both of these IPs are behind NATs, 5.43.242.176 is behind TWO NATs, ISP and router. And yet they can communicate directly, both ways with no relays needed.

Now, when the network issues are out of question I wonder why Factorio is unable to do the same. Thoughts?

Note: I tried to connect with another user of the same ISP that I use and it didn't succeed (the packets were sent from both sides but weren't received). I could, however, connect with that user using a VPN provider (packets were sent and received by both parties). My guess is that maybe my ISP prevents connections between their WAN IPs for security or other reasons.

Note 2: I tried this utility with my friends, one of them is also behind two NATs, his router gets a LAN IP just like mine. The other one is behind just one NAT. Both tests passed successfully, the packets went both ways with no issues. So probably it's just my ISP preventing any traffic between their users.

Note 3: my guess would be that maybe the pingpong servers use the default port 34197 to communicate with the game server even though the ISP mapped it to a different external port after they send the initial WAN IP detection response. That would explain why I don't see any packets after that (they're being sent to the ISP WAN IP and port 34197 instead of whatever their firewall chose) and at the same time it would explain why it works for some types of NAT (those which don't change the port). Since my ISP preserves the port number between different IPs (and if this speculation is correct) it could be a very trivial fix on your side that would make many people happy: use the port that you've received the server packets from, not the default.

Conclusion: please, fix Factorio NAT punching.

[rkfg]

Any updates on this? What information should I provide to help you fix the issue?

[ospis12]

So if I read you correctly, you want to host your server behind NAT? I just looked up in factorio wiki about headless servers, and they require to have public IP. I also looked at history of this article and change regarding headless servers having public IP was made on 19 of March 2017, more then 3 years ago.

Client hosting works fine through NAT, cause it was design to be behind one. Servers on the other hand were design to be always in public space.

If you want to run server from behind NAT maybe do it with normal game executable?

[rkfg]

So if I read you correctly, you want to host your server behind NAT? I just looked up in factorio wiki about headless servers, and they require to have public IP. I also looked at history of this article and change regarding headless servers having public IP was made on 19 of March 2017, more then 3 years ago.

So https://factorio.com/blog/post/fff-143 doesn't apply anymore? I believe it should because the headless server still contacts the pingpong servers, and not just one of them but three, to confirm that your public IP and port are the same for different IPs. I'd like to have a developer's confirmation on whether NAT punch is actually attempted. If it is, then it's implemented incorrectly for some reason because it should work for me (as I tested with the program above) but it doesn't. If it's not implemented at all then I wonder what happened because in that FFF it's described like it's already in place and working. I hope for the former, of course.

[ospis12]

Gave me a second, will test in virtual box if any packets are sent to punch through NAT.

Edit:
Checked with single NAT to public web and I've put my PC in DMZ, and found, that pingpong servers do not send any requests to Factorio server to ask it to send packets to connecting player.
This leads to conclusion that severs must be either in public address space, or if you are behind NAT(s) you have to forward ports on each NAT to your server.

And to your test inside ISP network, did you recived ICMP: "Destination port unreachable" packets?

[Hornwitser]

There seems to be a bug in how NatPunchRequest packets are handled. If I browse the server list and try to connect to a public server I see no such packets being sent unless I host a public server on the client for a few seconds first.

[Hornwitser]

I did some testing on how Factorio does NAT punching and this is my results.

How it works in 0.16.51:
1. The client sends a NatPunchRequest to a pingpong server with the address of the Factorio server
2. The pingpong server sends a NatPunchRequest to the Factorio server with the address of the Factorio client
3. The Factorio server then sends a NatPunch packet to the Factorio client.

diagram.png (14.86 KiB) Viewed 5348 times

If this is successful you should see something like following line in the server log when the --verbose flag is passed on the command line:

Code: Select all

 411.872 Verbose ServerRouter.cpp:504: Responding to nat punch request (from IP ADDR:({pingpong ip:34197}) to IP ADDR:({client ip:62130}))

And if you want to prove without any shadow of doubt that there's nothing wrong with your network and that it's all a problem with Factorio you can test your setup with version 0.16.51 of Factorio, because in that version nat punching still works.

Somewhere between 0.16.51 and 0.17.79 it all broke, the client no longer sends a NatPunchRequest to a pingpong server when you go into the public server list unless you have hosted a game in the client first.
The dedicated linux server doesn't respond to NatPunchRequest packets, instead it prints the following when the --verbose flag is passed on the command line:

Code: Select all

  97.019 Info UnparsedNetworkMessage.cpp:91: Invalid packet (type NatPunchRequest) received from IP ADDR:({pingpong ip}): multiplayer.invalid-address("", "Name or service not known")

And hosting a game from the client also prints an error when receiving NatPunchRequest packets, slightly different from the dedicated server (also requiring the --verbose flag):

Code: Select all

 603.829 Info UnparsedNetworkMessage.cpp:91: Invalid packet (type NatPunchRequest) received from IP ADDR:({pingpong ip:34197}): garbage after message

0.17.79 was released in December 2019, 4 months before this thread was started.

And this is where the disappointments of the developer response in this thread starts. rkfg has covered the bases pretty well when it comes to both showing good networking knowledge and making sure all packets is captured. The observation is that there's no nat punching going on at all. So what's the response?

If that's not working for you, then the issue most likely lies in your network configuration.

A complete disregard for everything that was said. And when queried for why there's no nat punching coming from your servers you reply with this gem (emphasis mine):

I have no idea. But there are loads of servers out there that work just fine, ones that are using nat punching, ones that are using steam networking, and ones with simple port forwarding.

The emphasised part of the statement is a fabrication. At the time it was written nat punching hadn't worked for months, and less than 1% of the player base was on a version where it worked. It's just disappointing. You had highly knowledged networking people who couldn't get nat punching to work and offered to help diagnose this issue with you and you simply ignored them. Now it's a year later and nat punching still doesn't work.

[rkfg]

Thanks for this detailed write up and insight. Indeed, it was very disappointing, especially after reading so many stellar FFFs where the devs go super deep to solve even the smallest 3rd party mod issues and it turns out to be something unexpected. Those people who reported the bugs gave only a bit of information and yet they got so much attention from the devs and the game became better for everyone as a result. Even for those who don't use those or any mods.

I had high hopes for this case to be solved in a similar manner, spent quite some time writing the probing code and checking my network with several people to make sure that the issue isn't on my side. Getting such an ignorant response ("eh, it's something somewhere, internet is hard and it works for many people and you're not among them so your experience doesn't matter kthxbye") after all these efforts is quite discouraging.

It's even more interesting that this is, in fact, a regression. When 0.15 was the most recent version probably my ISP didn't use CGNAT so my router was receiving a WAN IP and I simply forwarded the port. These days IPv4 addresses are limited and big ISPs don't provide them to their clients and port forwarding/UPnP are simply useless. So I couldn't tell if this is a regression because I never needed NAT punching when it worked and since it doesn't work now I assumed it was never implemented correctly. So I said to myself, maybe it could only work when one side is behind the NAT but not both (which is usually the case today). Good to know it was different!

[Rseding91]

Replying here too in addition to the bug reports around this:

After talking with one of my co-workers I believe it's time for me to eat-crow as the saying goes and admit I was wrong. I'll be looking into this more and seeing if I can figure out what's going on.

Sorry it took so long to convince us something was wrong. But it should be fixed now for the next 1.1.x release! I tested before-and-after with one of my coworkers and was able to reproduce all of the reported issues and fix them.

[rkfg]

Thank you! I appreciate the honesty and am happy to know it's supposedly fixed. Not sure I can test it now but I'll try to setup a new game with my friend and report back here if it works (when the next release is out).