Map download never finishes [14.5] headless windows

[RIscRIpt]

I'm sorry for being a bit `offtopc`, but I cannot PM @Twinsen, so I'll write here:
Please consider removing the PDB, in my opinion it shouldn't be publicly & easily available. Tho it won't harm the multiplayer much, but anyway...

[daniel34]

Please remove the PDB, in my opinion it shouldn't be publicly & easily available. Tho it won't harm the multiplayer much, but anyway...

The PDB file is used to make the stacktrace in a log file easily readable by humans, it cuts down development and bug-finding time when looking for the source of a crash. It doesn't impact the game's performance in any way.
Example here: viewtopic.php?f=49&t=24750#p155261

[bflyblue]

I wonder if it's related to this bug on Juniper routers:

"Dont treat all FFFF checksum packets as DIAG packets"

https://bugs.launchpad.net/juniperopens ... ug/1576506

Checksum 0 gets changed to 0xffff so these packets would hit the same bug

[jsharkey13]

If I've understood the problem correctly, it only occurs when the UDP checksum is a specific value. According to these RFCs: https://tools.ietf.org/html/rfc1624#page-3 and https://tools.ietf.org/html/rfc1122#page-78 there are two special cases for checksums. UDP checksums are never allowed to be 0x0000: these should be rewritten by all compliant hardware to be 0xFFFF, thus making 0xFFFF invalid if the checksum is not actually zero. And it also states that all compliant hardware must silently drop UDP packets that have invalid checksums. So it's not a bug in any of the places where these packets are dropped, they're just conforming to the RFCs. The RFC's also mention, somewhat wryly, "As a result, numerous cases of undetected errors have been reported."

[mexmer]

If I've understood the problem correctly, it only occurs when the UDP checksum is a specific value. According to these RFCs: https://tools.ietf.org/html/rfc1624#page-3 and https://tools.ietf.org/html/rfc1122#page-78 there are two special cases for checksums. UDP checksums are never allowed to be 0x0000: these should be rewritten by all compliant hardware to be 0xFFFF, thus making 0xFFFF invalid if the checksum is not actually zero. And it also states that all compliant hardware must silently drop UDP packets that have invalid checksums. So it's not a bug in any of the places where these packets are dropped, they're just conforming to the RFCs. The RFC's also mention, somewhat wryly, "As a result, numerous cases of undetected errors have been reported."

overwriting 0x0000 to 0xFFFF should not result in invalid checksum, read again that RFC1624 you linked, there are 2 formulas that create complementary checksum and as you can see (example at the end), for 0xFFFF they will both result in 0xFFFF thus valid packet. mind you need to properly handle result overflow (since it operates 16bit numbers, you should use strictly that type, nor larger or smaler).

0x0000 is rewriten to 0xFFFF because of that complementary formula, since on 0x0000 you will get 2 different results 0xFFFF and 0x0000 while 0xFFFF checksum gives 0xFFFF in both cases, unless your implementation of cheksum formulas is wrong (which is pointed out in other document, where some implementation used incremental checksum, which caused such issues)

also that quote you cited is incomplete, that quote referes to case, where UDP checksuming is turned off. You cannot turn off UDP check in standard UDP socket implementation (either it's on windows, linux or mac), you need to forge UDP packets manually and insert them directly to interface (use pcaplib for that for example). So if anything discards UDP checksum, must be any router on the way with bugged firmware.

[ikiris]

It could be anything in the path. You can try to do a traceroute with the packet by walking the TTL up and look for icmp dest unreach / ttl exceed if you have a raw socket, but that will only get you a rough idea where the problem is due to the nature of most networks security policy and how the forwarding planes work. It also requires you to have a known broken test case, and that's easier said than done.

Seriously though, this is pretty normal. You should just include the counter and move on. The nature of the internet basically means that something somewhere is ALWAYS broken, and you just have to deal with it by doing basic things like this to protect yourself. Its why its usually a bad idea to write your own protocols instead of using standard ones as well =) There's a reason the deeper you guys go with this, the closer and closer it starts to look like everything else you could have used to begin with. *chuckles*

*double sigh*
Traceroute uses ICMP protocol, not UDP, and although the checksum algorithm is almost identical, ICMP doesn't allow omission of the checksum and thus doesn't consider 0x0000 to be special. The possibility therefore to reproduce this problem at the same host is next no nothing.
But you're right, there will always be problems and bugs in network equipment, and the Factorio devs will have nothing to do about it, which is exactly why their decision to roll their own protocol is inconsequential to this (and similar) problems.

I didn't say run the command "traceroute", I said do *a* traceroute by walking up the TTL on the specific broken packet. It will show you exactly where the break is within a few hops (due to again, most network's security policy).

As for that second part, if you look at other commonly used protocols, you'll find a glaring counter for exactly this kind of reason, among other things like selective ack, etc etc etc. Reinventing wheels makes for square wheels.

[perceival]

If anyone has ideas on how to narrow down which hop is eating packets I would love to know.

The IP TTL field is probably the way to go as others have mentioned.

You could take a packet capture of the UDP packets you already have with the funny csums, then modify the IP TTL values on each packet so you get a range from 1-30 (say). Then save the modified packets to a different file and replay it using tcpreplay while simultaneously running a packet capture to receive the ICMP TTL expired replies. The source IP of the ICMP TTL expired packet with the highest TTL is either the one thats dropping or a hop before it.

You could also write a program that just generates the .pcap file - couple of nice perl libraries for this like Net::Pcap and Net::Layer. It would be harder than just modifying a .pcap file you already have though because you would need to make a valid routable ethernet header, an IP header, UDP header and your packet data .

Theres also python bindings for libpcap if you are more familiar with python, which may even work better.

I will have a go at writing such a program, would be a lot easier if you could share your work on how you craft UDP packets with the odd checksum.

[sigillum]

Making a program to create the special packets is actually pretty simple, and you don't need to use PCAP or other magic to set the TTL. For example here is a quick hack in C that does the trick pretty simply. It'll run in Linux, someone could probably make a Windows version, or build it with Cygwin/mingw, and you could just run it while running tcpdump/wireshark to watch the ICMP replies.

In any case, you can see how to make a packet that will always have a UDP checksum of 0xFFFF (after it passes your NAT).

Code: Select all

#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <netinet/ip.h>
#include <netinet/udp.h>
#include <stdint.h>
#include <unistd.h>
#include <stdio.h>

uint16_t gen_csum_pad(struct sockaddr_in *wan_addr, struct sockaddr_in *remote_addr)
{
  uint32_t csum_buf;
  uint32_t csum_buf_2;

  csum_buf_2 = ntohl(wan_addr->sin_addr.s_addr);
  csum_buf = (csum_buf_2 & 0xFFFF) + (csum_buf_2 >> 16);
  csum_buf += ntohs(wan_addr->sin_port);

  csum_buf_2 = ntohl(remote_addr->sin_addr.s_addr);
  csum_buf += (csum_buf_2 & 0xFFFF) + (csum_buf_2 >> 16);
  csum_buf += ntohs(remote_addr->sin_port);

  /* protocol */
  csum_buf += 17;

  /* UDP length */
  csum_buf += 10;

  /* length */
  csum_buf += 10;

  /* 1's complement fold */
  while (csum_buf & 0xFFFF0000)
    csum_buf = (csum_buf & 0xFFFF) + (csum_buf >> 16);

  /* calculate pad to make overall checksum FFFF */
  return htons(0xFFFF - csum_buf);
}

int main(int argc, char **argv)
{
  int sockfd;
  uint16_t data;
  struct sockaddr_in lan_addr;
  struct sockaddr_in wan_addr;
  struct sockaddr_in remote_addr;
  int ttl;

  sockfd = socket(AF_INET, SOCK_DGRAM, 0);

  lan_addr.sin_family = AF_INET;
  lan_addr.sin_addr.s_addr = htonl(0x########); /* source IP in hex, e.g. 192.168.0.16 = 0xc0a80010 */
  lan_addr.sin_port = htons(20000);

  wan_addr.sin_family = AF_INET;
  wan_addr.sin_addr.s_addr = htonl(0x########); /* source IP after NAT (your WAN address) as above */
  wan_addr.sin_port = htons(20000);

  remote_addr.sin_family = AF_INET;
  remote_addr.sin_addr.s_addr = htonl(0x########); /* remote IP to send packets to */
  remote_addr.sin_port = htons(20000);

  bind(sockfd, (struct sockaddr *)&lan_addr, sizeof(struct sockaddr_in));

  data = gen_csum_pad(&wan_addr, &remote_addr);

  printf("Pad is %04X\n", ntohs(data));

  for (ttl = 1; ttl < 30; ttl++)
  {
     printf("Sending with TTL %d...\n", ttl);
     setsockopt(sockfd, IPPROTO_IP, IP_TTL, &ttl, sizeof(ttl));
     sendto(sockfd, &data, 2, 0, (struct sockaddr *)&remote_addr, sizeof(struct sockaddr_in));
     sleep(1);
  }

  return 0;
}

Works for me, YMMV, etc., etc. Good luck with your testing.

[gallomimia]

Guys, this problem was solved already by the world of bit torrenters. The culprit is a router with "game mode" enabled. It inspects packet DATA and not headers, performing NAT on the data as well as the headers. It can be a useful "feature" for badly designed network code in certain console games and others, but for the most part, it merely gives you a 1 in 2^32 chance of colliding with that address in every single 4-bytes of data in your network transactions.

In short, the bug is not factorio's. It is a bug in the router. Your choices are to turn the "feature" in the router off, or throw the router in the garbage. Or possibly to tunnel your game traffic inside an encrypted protocol like ssh so that the data is different values.

[AyrA]

Guys, this problem was solved already by the world of bit torrenters. [...] In short, the bug is not factorio's. It is a bug in the router [...]

We solved this issue pages ago and came to the conclusion, that adding a simple counter is better than telling the user their equipment is shit. People will just abandon factorio otherwise. Adding a counter is a very simple solution to any problem that could ever arise for checksum errors.