[twinsen] [0.18.32] DNS/hostname with both IPv6 and IPv4, uses only IPv4, when using --mp-connect or GUI

[movax20h]

It is a rather minor issue, but.

Code: Select all

   0.173 2020-06-25 20:34:55; Factorio 0.18.32 (build 52799, linux64, alpha)
   0.338 Operating system: Linux (Debian testing)

I have a headless server running on IPv6 only, using `--bind [::1]:8181--rcon-bind [::1]:8182`

When I try to connect to it using `factorio --mp-connect localhost:8181`, or use "localhost:8181" in the GUI to connect, factorio only tries 127.0.0.1 for the network address:

Code: Select all

88357.782 Joining game IP ADDR:({127.0.0.1:8181})

Which does fail with the error eventually. It never tries IPv6 first (despite this being my configuration AFAIK), or fallbacks to other addresses after IPv4 failed.

My configuration is very standard and not modified from default:

Code: Select all

$ getent hosts localhost
::1             localhost ip6-localhost ip6-loopback
$

Code: Select all

$ getent ahosts localhost
::1             STREAM localhost
::1             DGRAM  
::1             RAW    
127.0.0.1       STREAM 
127.0.0.1       DGRAM  
127.0.0.1       RAW    
$ getent ahostsv4 localhost
127.0.0.1       STREAM localhost
127.0.0.1       DGRAM  
127.0.0.1       RAW    
127.0.0.1       STREAM 
127.0.0.1       DGRAM  
127.0.0.1       RAW    
$ getent ahostsv6 localhost
::1             STREAM localhost
::1             DGRAM  
::1             RAW    

Code: Select all

$ cat /etc/hosts | grep localhost
127.0.0.1       localhost debian
::1             localhost ip6-localhost ip6-loopback
$

Code: Select all

$ grep hosts /etc/nsswitch.conf 
hosts:          files mdns4_minimal [NOTFOUND=return] dns myhostname mymachines
$

(files mean to check `/etc/hosts.conf` first, before trying DNS resolves are configured in `resolv.conf` - standard behavior).

Code: Select all

$ egrep -v '^#' /etc/gai.conf 
$

Code: Select all

$ cat /etc/host.conf 
multi on  #  If set to on, the resolver library will return all valid addresses for a host that appears in the `/etc/hosts` file, instead of only the first.

Code: Select all

$ host localhost
localhost has address 127.0.0.1
localhost has IPv6 address ::1
$

I am guessing when there are multiple results for the host, factorio uses just the first one. I would suggest to trying all in order returned by the resolver? Similar to how web browsers maybe do it? If the system doesn't have IPv6 available (this can be tested by opening the socket to destination - if there are no route to it or the protocol is disabled system wise, it will fail opening in `connect` immedietly, with `ENETUNREACH` error), the testing can be skipped. Otherwise factorio could be trying all address in the system default order, and only return back an error if all addresses failed?

But there must be more to it, because even if I change the gai.conf to:

Code: Select all

$ egrep -v '^#' /etc/gai.conf 
label ::1/128       0   # IPv6 localhost will be returned first
label ::/0          1
label 2002::/16     2
label ::/96         3
label ::ffff:0:0/96 4    # IPv4 in "tunneled IPv6 form", will be returned, but last
precedence  ::1/128       50
precedence  ::/0          40
precedence  2002::/16     30
precedence ::/96          20
precedence ::ffff:0:0/96  10
$

it still doesn't work (factorio is still trying IPv4 address by default).

Similarly, if I use, ask DNS resolver (which is not used for `localhost`, because it is handled by hosts resolver), to return tunneled form for IPv4 addresses, it still doesn't.

Code: Select all

$ cat /etc/resolv.conf
nameserver 2001:1620:2777:1::10
nameserver 2001:1620:2777:2::20
options inet6  # Ask `gethostbyname` to do AAAA query before A query, and wrap IPv4 addresses (A records) as "tunneled form" back to app if there are no AAAA records. AFAIK it is ignored by `getaddrinfo`, which requires explicit `AI_V4MAPPED` in the app for same functionality.

If I change the `/etc/hosts` to only list IPv6 for localhost it does work:

Code: Select all

# cat  /etc/hosts
127.0.0.1       ipv4-localhost debian
::1             localhost ip6-localhost ip6-loopback
#

Connect using "localhost":

Code: Select all

  13.228 Joining game IP ADDR:({[::1]:34197})

So I think still it has something to do with only trying the first address.

I know in reality if the factorio server has DNS address (and clients asked to use it), the factorio server process should listen on all IP addresses exposed via this DNS address, so any address would work (not doing so would be a server misconfiguration really). The problem is it is not totally possible with factorio headless server. It can only listen on ALL (current and future) interfaces (using `--port`), or on only one (via `--bind`). For non-public server, it doesn't is exactly a good option, because it forces one to listen on all interfaces or one.

However I do think the factorio is not respecting current standards:

RFC 6724: https://tools.ietf.org/rfc/rfc6724.txt - Default Address Selection for Internet Protocol Version 6 (IPv6)
(obsoletes RFC 3484)
RFC 3493: https://tools.ietf.org/rfc/rfc3493.txt - Basic Socket Interface Extensions for IPv6 (obsoletes RFC 2553)
RFC 4291: https://tools.ietf.org/rfc/rfc4291.txt - IP Version 6 Addressing Architecture (obsoletes RFC 3513)
RFC 4862: https://tools.ietf.org/rfc/rfc4862.txt - IPv6 Stateless Address Autoconfiguration (obsoletes RFC 2462)


Quote from RFC 6724 , section 2:

2. Context in Which the Algorithms Operate
...

Well-behaved applications SHOULD NOT simply use the first address
returned from an API such as getaddrinfo() and then give up if it
fails.

...

And more details about treatment of IPv4 and IPv6:

...
3.2. IPv4 Addresses and IPv4-Mapped Addresses

The destination address selection algorithm operates on both IPv6 and
IPv4 addresses. For this purpose, IPv4 addresses MUST be represented
as IPv4-mapped addresses [RFC4291]. For example, to look up the
precedence or other attributes of an IPv4 address in the policy
table, look up the corresponding IPv4-mapped IPv6 address.

IPv4 addresses are assigned scopes as follows. IPv4 auto-
configuration addresses [RFC3927], which have the prefix 169.254/16,
are assigned link-local scope. IPv4 loopback addresses (Section
4.2.2.11 of [RFC1812]), which have the prefix 127/8, are assigned
link-local scope (analogously to the treatment of the IPv6 loopback
address (Section 4 of [RFC4007])). Other IPv4 addresses (including
IPv4 private addresses [RFC1918] and Shared Address Space addresses
[RFC6598]) are assigned global scope.

IPv4 addresses MUST be treated as having "preferred" (in the RFC 4862
sense) configuration status.

3.3. Other IPv6 Addresses with Embedded IPv4 Addresses

IPv4-compatible addresses [RFC4291], IPv4-mapped [RFC4291], IPv4-
converted [RFC6145], IPv4-translatable [RFC6145], and 6to4 addresses
[RFC3056] contain an embedded IPv4 address. For the purposes of this
document, these addresses MUST be treated as having global scope.

IPv4-compatible, IPv4-mapped, and IPv4-converted addresses MUST be
treated as having "preferred" (in the RFC 4862 sense) configuration
status.

3.4. IPv6 Loopback Address and Other Format Prefixes

The loopback address MUST be treated as having link-local scope
(Section 4 of [RFC4007]) and "preferred" (in the RFC 4862 sense)
configuration status.

...

"preferred" (in the RFC 4862) only means that they can be used unrestricted by the applications (i.e. they are not stale, or link-local). It doesn't determinine priority of which address to use. All "preferred" addresses (which is some set IPv6 and IPv4 addresses) are considered equal in this sense:

From RFC 4862:

...
preferred address - an address assigned to an interface whose use by
upper-layer protocols is unrestricted. Preferred addresses may be
used as the source (or destination) address of packets sent from
(or to) the interface.
...

A non-preferred address types are tentative addresses and deprecated address. These will not be used. It is more for the purpose of selecting the source address, not the destination address, or the host resolution priorities.

As far as I know `getaddrinfo` in Linux (glibc 2.30 in my case), does respect this behaviors (as tested in other apps), but Factorio is not.

It somehow feels factorio is using own non-glibc resolver, and/or sets own address sorting, and only tries the first address.

It does appear factorio uses `getaddrinfo`, as I checked with gdb and ltrace:

Code: Select all

Thread 1 "factorio" hit Breakpoint 1, __GI_getaddrinfo (name=0x7ffd6deb5ca0 "localhost", service=0x7ffd6deb5cc0 "34197", hints=0x7ffd6deb5d10, pai=0x7ffd6deb5c88) at ../sysdeps/posix/getaddrinfo.c:2161
2161	in ../sysdeps/posix/getaddrinfo.c
(gdb) bt
#0  __GI_getaddrinfo (name=0x7ffd6deb5ca0 "localhost", service=0x7ffd6deb5cc0 "34197", hints=0x7ffd6deb5d10, pai=0x7ffd6deb5c88) at ../sysdeps/posix/getaddrinfo.c:2161
#1  0x0000000000ff69d8 in SocketAddress::SocketAddress () at /tmp/factorio-build-6v7ua5/src/Net/SocketAddress.cpp:117
#2  0x0000000000ff6b5b in NamedSocketAddress::resolve () at /tmp/factorio-build-6v7ua5/src/Net/NamedSocketAddress.cpp:12
#3  0x0000000001001164 in NamedSocketAddress::getAddress () at /tmp/factorio-build-6v7ua5/src/Net/NamedSocketAddress.cpp:5
#4  MultiplayerConnectSettings::getAddress () at /tmp/factorio-build-6v7ua5/src/Net/MultiplayerConnectSettings.cpp:27
#5  ClientRouter::joinGame () at /tmp/factorio-build-6v7ua5/src/Net/ClientRouter.cpp:40
#6  0x00000000011c3b22 in ClientMultiplayerManager::joinGame () at /tmp/factorio-build-6v7ua5/src/Net/ClientMultiplayerManager.cpp:271
...
(gdb) print *hints
$4 = {ai_flags = 1024, ai_family = 0, ai_socktype = 2, ai_protocol = 0, ai_addrlen = 0, ai_addr = 0x0, ai_canonname = 0x0, ai_next = 0x0}
(gdb) 

`ai_flags = 1024`, is equivelent to `ai_flags = AI_NUMERICSERV`, which is OK (it asks getaddrinfo to only resolve the `service` from string to integer, without checking service resolution, i.e. in `/etc/services` or DNS SRV). That is OK.

`ai_family = 0` means `AF_UNSPEC`, which is also OK (return both AF_INET and AF_INET6).

`ai_socktype = 2` means `SOCK_DGRAM`, that is OK.

All is good, and if I trace it manually after the return:

Code: Select all

Thread 1 "factorio" hit Breakpoint 1, __GI_getaddrinfo (name=0x7ffd6deb5ca0 "localhost", service=0x7ffd6deb5cc0 "34197", hints=0x7ffd6deb5d10, pai=0x7ffd6deb5c88) at ../sysdeps/posix/getaddrinfo.c:2161
2161	../sysdeps/posix/getaddrinfo.c: No such file or directory.
(gdb) print pai
$8 = (struct addrinfo **) 0x7ffd6deb5c88

(gdb) advance getaddrinfo
0x0000000000ff69d8 in SocketAddress::SocketAddress () at /tmp/factorio-build-6v7ua5/src/Net/SocketAddress.cpp:117
117	/tmp/factorio-build-6v7ua5/src/Net/SocketAddress.cpp: No such file or directory.

(gdb) print **(struct addrinfo**)(0x7ffd6deb5c88)
$17 = {ai_flags = 1024, ai_family = 10, ai_socktype = 2, ai_protocol = 17, ai_addrlen = 28, ai_addr = 0x4a996f0, ai_canonname = 0x0, ai_next = 0xae6dc10}
(gdb) print *(**(struct addrinfo**)(0x7ffd6deb5c88))->ai_next
$20 = {ai_flags = 1024, ai_family = 2, ai_socktype = 2, ai_protocol = 17, ai_addrlen = 16, ai_addr = 0xae6dc40, ai_canonname = 0x0, ai_next = 0x0}

(gdb) print *(**(struct addrinfo**)(0x7ffd6deb5c88)).ai_addr
$22 = {sa_family = 10, sa_data = "\205\225", '\000' <repeats 11 times>}
(gdb) print *(*(**(struct addrinfo**)(0x7ffd6deb5c88))->ai_next).ai_addr
$26 = {sa_family = 2, sa_data = "\205\225\177\000\000\001\000\000\000\000\000\000\000"}

I can see glibc does return two addresses for "localhost", one IPv6 and one IPv4.

Code: Select all

"\205\225", '\000' <repeats 11 times> corresponds to (sockaddr_in6 value) of [::1]:34197
'\205\225\177\000\000\001...' corresponds to (sockaddr_in value) of 127.0.0.1:34197  (0205 * 256 + 0225 == 34197)

(gdb) print ((**(struct addrinfo**)(0x7ffd6deb5c88)).ai_addr.sa_data)[/*port*/2+/*flow_info*/4+/*address_length*/16-1/*last_digit*/]
$73 = 1 '\001'   # 1 in `0000:0000:0000:0000:0000:0000:0000:0001`

So factorio knows about both addresses, and as you can see glibc returns IPv6 as the first one.

[Rseding91]

For other developers: Factorio iterates all of the network adapters on a computer and finds the first IPv4 one it can, or falls back to IPv6. I'm not sure if it forces ipv6 if you use --bind.

From all of my testing it's almost impossible to force Factorio to use IPv6 without completely uninstalling ipv4 from the system. Once an address is chosen nothing ever "tries the other if one fails". You basically have to force the game to use one and then it uses it; otherwise you get IPv4.

[Rseding91]

Some more info:

The issue as I see it on our side: If the game has to choose between an IPv4 or an IPv6 adapter when doing network communication - the chances that a given IPv6 address will have internet connection *and* be what the user wants to use (host on or otherwise) is so near zero that we're statistically always better off choosing the IPv4 adapter.

At least - on windows - every system out there since Vista has an IPv6 LAN address that only works for LAN communication and will have no internet connection unless that user happens to have an IPv6 address from their ISP or has a router with 4to6 or some variant configured.

So in the end: if Factorio sees 2 network adapters - one with IPv4 and one with IPv6 - we're always better off choosing the IPv4 one.

I don't know if that story is the same on Linux but it's why Factorio never chooses IPv6 as the network adapter to use unless it can't find an IPv4 one.

[orzelek]

Is IPv6 really so rare?
I have functioning IPv6 address from at least few years.

[Rseding91]

Is IPv6 really so rare?
I have functioning IPv6 address from at least few years.

According to a quick search: yes. 20% to 22% of the world has them. Now, which % has them and uses them as their primary connection I have no idea - some smaller % still.

[Squelch]

This is the current state of hosts announcing IPv6 according to RIPE NCC

ie just under 29%

IPv6 statistics

[SoShootMe]

The issue as I see it on our side: If the game has to choose between an IPv4 or an IPv6 adapter when doing network communication - the chances that a given IPv6 address will have internet connection *and* be what the user wants to use (host on or otherwise) is so near zero that we're statistically always better off choosing the IPv4 adapter.

The issue as I see it, is that the game chooses in the first place. Discounting advertisement/discovery communication, if all clients talk only to the server (as I understand is the case), I don't see why either the client or server need to choose.

As it happens I have network client/server software development experience (and so I realise it is far from a trivial problem, particularly on the server side, and perhaps an even bigger problem to fix now) but it is plain for anyone to see that other software manages IPv4/IPv6 just fine.

[Rseding91]

As it happens I have network client/server software development experience (and so I realise it is far from a trivial problem, particularly on the server side, and perhaps an even bigger problem to fix now) but it is plain for anyone to see that other software manages IPv4/IPv6 just fine.

I'd be interested to see how other software handles it. Every piece of game software i've seen requires you give it the host IP address for it to bind to. Or it just hosts through steam networking which then falls back to steam to do it.

[SoShootMe]

As it happens I have network client/server software development experience (and so I realise it is far from a trivial problem, particularly on the server side, and perhaps an even bigger problem to fix now) but it is plain for anyone to see that other software manages IPv4/IPv6 just fine.

I'd be interested to see how other software handles it. Every piece of game software i've seen requires you give it the host IP address for it to bind to. Or it just hosts through steam networking which then falls back to steam to do it.

With regards to binding sockets, the "TL;DR" version is that in my experience, typical clients don't explicitly bind to a particular address (or port) but may provide an option to do so, and typical servers bind to what they're configured to, with a default of either any (one socket; TCP and sometimes UDP) or all addresses (one socket per address; only UDP). This is separate to IP version considerations.

The IPv4/IPv6 side is different on clients and servers. For clients, because of the above it's basically about choosing a destination address: trying each in turn, in the order provided, whether IPv4 or IPv6, is one approach indicated in RFC6724 (as quoted by the OP) and works well in many cases: attempting an IPv6 address will fail immediately as the OP described in the (common, currently) case of full IPv4 connectivity but the only IPv6 being link-local. For servers, portability is probably the biggest challenge due to variation between (and within) platforms, notably around IPv6-mapped IPv4, but network code portability in general is not a new problem.

[justarandomgeek]

I'd be interested to see how other software handles it. Every piece of game software i've seen requires you give it the host IP address for it to bind to. Or it just hosts through steam networking which then falls back to steam to do it.

Most software i've interacted with defaults to binding to all addresses on the machine unless configured to a single interface/address (or, in really nice ones, a list of them).

[blahfasel2000]

According to a quick search: yes. 20% to 22% of the world has them. Now, which % has them and uses them as their primary connection I have no idea - some smaller % still.

Google reports that 30% of their users are using their services through native IPv6: https://www.google.com/intl/en/ipv6/statistics.html
In some countries (Germany and India for example) IPv6 adoption has reached 50% already. And the numbers are going up steadily.

Also, in more and more countries you can't actually get a new internet connection with native IPv4 anymore, unless you pay extra for a business account. Access to IPv4 is provided through DS-Lite tunneling, which means you are basically behind a NAT shared with many other customers, with no control over it (no port forwarding!). So for more and more people IPv6 is the only option when they want to host a game.

The issue as I see it on our side: If the game has to choose between an IPv4 or an IPv6 adapter when doing network communication - the chances that a given IPv6 address will have internet connection

The once common problem of broken systems having IPv6 configured for some reason but having no internet connectivity has been more or less weeded out years ago (see https://en.wikipedia.org/wiki/World_IPv ... Launch_Day). Today you can basically just assume that if a system has a public IPv6 address configured then it will have proper IPv6 internet connectivity.

[Squelch]

Maybe this example code for the Happy Eyeballs algorithm may be of help?

https://github.com/shtrom/happy-eyeballs-c

[Twinsen]

So I looked into this. To my understanding this is what happens:


When resolving addresses Factorio will call getaddrinfo and then use the first ipv4 address, otherwise the first ipv6 address. The preference for ipv4, while not technically correct, is not the problem.

The problem is not respecting RFC 6724, section 2. Currently Factorio takes the first address and uses it everywhere.

Unfortunately implementing this correctly will mean a serious refactor of how we handle and save addresses.
Also since we use UDP, correctly and quickly choosing the correct address to connect to probably means updating our network protocol.
I'm not sure if this is worth the effort. In practice, I don't really see the current implementation causing problems that can't be worked around(in the most common use cases).

Maybe this example code for the Happy Eyeballs algorithm may be of help?
https://github.com/shtrom/happy-eyeballs-c

That implementation works for TCP, but we use UDP.
Also, in my opinion RFC 6555 is solving a problem that really shouldn't exist, but that's irrelevant.

[justarandomgeek]

In practice, I don't really see the current implementation causing problems that can't be worked around(in the most common use cases).

The big one i see is that it seems like it would be impossible to host a game that's accessible to both ipv4-only and ipv6-only clients, if the server is only binding to a single address (as opposed to the more common all-addresses-on-machine)

[Squelch]

That implementation works for TCP, but we use UDP.
Also, in my opinion RFC 6555 is solving a problem that really shouldn't exist, but that's irrelevant.

I meant it as a hint under the assumption that TCP was being used for initial server/client setup. I later found RFC 6936 outlining the use of zero checksum datagrams over IPv6. It would appear that it's not simple, and if it also needs a major netcode refactor, then that makes it just that bit harder. I do foresee it becoming more of a problem as time goes on, so there's only so long before it will need to be addressed.

[Twinsen]

The big one i see is that it seems like it would be impossible to host a game that's accessible to both ipv4-only and ipv6-only clients, if the server is only binding to a single address (as opposed to the more common all-addresses-on-machine)

That's more of a feature request of being able to bind to a list of addresses instead of just one, which probably makes sense given the whole ipv4 and ipv6 transition (and will probably make more sense in the future)

[SoShootMe]

The big one i see is that it seems like it would be impossible to host a game that's accessible to both ipv4-only and ipv6-only clients, if the server is only binding to a single address (as opposed to the more common all-addresses-on-machine)

That's more of a feature request of being able to bind to a list of addresses instead of just one, which probably makes sense given the whole ipv4 and ipv6 transition (and will probably make more sense in the future)

It's not strictly necessary to allow binding to a list of addresses to support IPv4-only and IPv6-only clients on the same server: most platforms allow a single IPv6 socket bound to the wildcard address, by default or with appropriate setup (IPV6_V6ONLY off), to be used to serve both IPv4 and IPv6 clients.

This works well for a TCP server, and usually also for a UDP server, but there are certain network configurations - regardless of IP version or versions - where a single socket bound to the wildcard address can precipitate problems for a UDP server. Without going into details (I'm happy to if there's interest), the end result is that multiple sockets is the best way to support such configurations in a UDP server.