increase the newGRF limit

[andreasaspenberg]

previously the limit was shown i believe but, it is not now so, i downloaded a lot of stuff not knowing about the limit. i can not use all of it because of the limit. it made sense before when the content slowed down the game but, not now. they should at least increase the limit but, maybe they should just remove it entirely now the game runs so smoothly.

[fridaemon]

previously the limit was shown i believe but, it is not now so, i downloaded a lot of stuff not knowing about the limit. i can not use all of it because of the limit. it made sense before when the content slowed down the game but, not now. they should at least increase the limit but, maybe they should just remove it entirely now the game runs so smoothly.

Download this patched version viewtopic.php?f=33&t=73469&start=2800 - u can use much more newgrfs (256 I think ...)

[andreasaspenberg]

that is not compatible with any of the graphics i have. it support neither original_windows now openGFX, both of which i have installed. on the other hand: i did not post this to get around the built-in limit but to try to get the developers to increase it.

[Auge]

Hello

that is not compatible with any of the graphics i have. it support neither original_windows now openGFX, both of which i have installed.

That is not true. Both the original and the openGFX and also every other base set is supported in JGRPatchpack. If you can't use any of the base sets, then your installation was gone wrong.

on the other hand: i did not post this to get around the built-in limit but to try to get the developers to increase it.

OpenTTD allows you to select up to 59 different NewGRFs for each new game you start.

Source: OpenTTD-Wiki

59 items are really insufficient? Holy anything, I myself never touched this limit …

Tschö, Auge

[andreasaspenberg]

considering how many different newGRF sets there is now, the limit is fairly mean. the modded version of the game does not have an installer at all. the limit made sense before when each newGRF set slowed the game but, that is no longer a problem. now i experience no slowdowns with newGRF at all, though i might have trouble finding stuff because i have downloaded so many different things. ways to organise newGRF additions based on what pack they come in would certainly help.

[kamnet]

the modded version of the game does not have an installer at all.

Why do you need an installer? Just unpack it in a directory of your choice and run. There are no drivers or OS configuration required.

though i might have trouble finding stuff because i have downloaded so many different things. ways to organise newGRF additions based on what pack they come in would certainly help.

NewGRFs don't care if they come in packs or not. They're all displayed individually by the game anyhow.

[Auge]

Hello

considering how many different newGRF sets there is now, the limit is fairly mean. … the limit made sense before when each newGRF set slowed the game but, that is no longer a problem. now i experience no slowdowns with newGRF at all

If NewGRFs in itself ever slowed down games and you (on your hardware) do not encounter that problem anymore, this means not, that someone else can run into this problem (with her/his hardware). Ok, one can say, noone is forced to outbid the limits, but so also you aren't.

I myself never ran into such a problem because of the number of used NewGRFs, even on my (very) old computer (Pentium III, 450 MHz with 256 MB RAM). When the game slowed down, the cause was the number of running vehicles towns and industries on the map, in other words, the number of objects thats simulation has to be computed. At the moment I use a preset with around 40 to 45 NewGRFs in my personal games.

though i might have trouble finding stuff because i have downloaded so many different things. ways to organise newGRF additions based on what pack they come in would certainly help.

Hmm, one should use only one industry set in a game. One matching rail vehicle set [1] (maybe two), one or two matching road vehicle sets [1], a matching ship set [1], an aircraft set [1], one to three town sets, that's in sum six to ten NewGRFs. That's IMHO enough space for eyecandy stuff (stations, buffers, signals, depots, trees, objects and so on). You are not forced (with exception by yourself) to use all available NewGRFs in one game.

Tip: Start a few test games with different presets to select your favourite combinations of NewGRFs. At the end you will not use all of them at once. To be more clear, there will be sets you'll never use again. Sets you do not have to load into the next game, sets that will not flood your menus if you do not load them.

Tschö, Auge

[1]: "Matching * [vehicle] set" means a set that is able to transport the products of the activated industry set.

[edit]: a few typos

[Eddi]

"slowing down" was never the reason for the newgrf limit. the online server list is why this limit exists, as every server must send a complete list of all used NewGRFs, and the size of this reply package is the reason for the current limit.

[jfs]

The limit does not relate to performance, it relates to network protocol. I actually thought the limit only applied to multiplayer games, and there wasn't one in singleplayer.
Anyway, the reason for the current limit to number of NewGRFs is that network packets have a limited size, and all the information needed to describe a server needs to fit in a single packet. That limits the number of NewGRFs that can be described and hence how many can be used.
It's possible to extend the limit by extending the network protocol, and yes someone should just do that already. (But it's not as trivial as just making a number bigger, it needs some structural changes to how network game info is communicated.)

[andreasaspenberg]

then the limit could be removed for lan and single player and kept for online. the part of the game that was slowed by newGRF was startup. the more stuff i had installed, the longer it took to start.

[_dp_]

What on earth do you need 60 grfs for? Have you tried to actually think what you add not just add everything because it exists? Newgrf count is not the only limit in the game, there are plenty of others like 64 cargo types, 256 or smth house types etc and they aren't going anywhere. Furthermore many newgrfs make no sense together even if they can technically be used. E.g. you only need one ground set, one indusry set, one or two house sets, etc. IMO if you just add 60 grfs randomly you're pretty much guaranteed to end up with a broken gameplay.

[andreasaspenberg]

i do not even use house sets or industry sets. i use vehicle sets(trains and trucks and ships) and track sets. even those is too many for the limit.

[jfs]

It sounds strange to me to want hundreds, possibly a thousand, different locomotives to choose between. Any one single set normally provides a wide variety covering all use cases.

Two misunderstandings:
1. LAN multiplayer and internet multiplayer are the same thing, they are both restricted in package size by the same MTU values. It does not have to do with transfer speed or reliability. It has to do with how layer 2 protocols frame data and that all layer 2 protocols have frame size limits of around 1300-1500 bytes.
The limits apply equally to Ethernet, WiFi, ATM, PPP, Frame Relay, FDDI, TokenRing, and whatever else you could imagine running the game over. The game's protocol has to be designed to fit the lowest common denominator, because packets can route through many networks (unknowable during setup, and can change during play) and it needs to work regardless of what type of connections are in use on the route between server and clients.

2. Raising the limit on singleplayer games (if there is not already a separate one, still not sure) creates a complexity in that some savegames will not be loadable as a basis for starting a multiplayer game.

[andreasaspenberg]

as of right now, the limit is for all games. multiplayer however is limited to the second largest maps because no players can connect to the largest maps. as for the modded version: i put it where i wanted and it could not find compatible graphics. i have however been running the official version with both graphics sets and it is running perfectly in the background right now. different trains differ in costs and performance. i want to have a wide selection. most connections these days is strong enough to handle an increased amount of data over what is currently being sent.

[jfs]

Please don't try to argue network protocols if you don't understand the internet protocol suite (TCP, UDP, IP), link layer protocols, internet routing, and socket API programming. The way you write your posts it seems you don't actually understand the root cause of the problem, and explaining all the details required to understand it would be an essay of several thousand words and technical illustrations. The short version is that OpenTTD requires the full server info to fit in a single UDP datagram, UDP datagrams are routed by the IP protocol suite across many distinct link layer protocols, different link layer protocols have different maximum transfer units (MTU, maximum frame size), if a UDP datagram does not fit within the MTU of any single hop along the route the datagram is discarded, because UDP does not allow fragmentation. Because of that, the server info UDP datagrams sent by OpenTTD need to fit within the MTU of any link layer protocol the datagram could possibly cross during the routing from server to client, regardless of whether server and client are on the same physical network segment, or one is in a submarine and the other is on the back side of the Moon.

Besides, increasing the maximum server info packet size from 1460 bytes (current value of SEND_MTU in src/network/core/config.h) to 1500 bytes (maximum permitted by Ethernet and 802.11 WiFi) to assume a local home network would give at most space for 2 or 3 additional NewGRFs in the server info packet.

[andreasaspenberg]

here it is you who do not understand. many games can transfer much more data between computers than what openttd can. warcraft 3 for example have room for 8 megabytes of custom data including graphics. that is much more than openttd sends. openttd after all have the data stored locally, while warcraft 3 transfers all of the custom data. openttd can therefore remove the newGRF limit with proper programming. i will not respond to further posts from you, since you attack me.

[andythenorth]

This thread is probably now better locked.

It's quite hard to have a decent, civil discussion with people who feel attacked by factual explanations of complex things.

Thanks.

[andreasaspenberg]

when people attack me, i ignore them. my education is computer based. people should not make stuff up and call them facts.

[andythenorth]

In that case, please engage in civil discussion.

If you dispute the facts, please explain why the facts are incorrect.

Saying the facts are wrong, and refusing to talk further about it easily fits the definition of passive aggression.

Labelling anything you disagree with as an attack easily fits the definition of passive aggression.

So please, come forward, and join us in a civil discussion, it's welcomed here.

[jfs]

Here, let me show you the actual code relevant.

src/network/core/config.h where various constants are defined. An excerpt of values relevant to this topic:

Code: Select all

static const uint16 SEND_MTU                      = 1460;         ///< Number of bytes we can pack in a single packet

/**
 * Maximum number of GRFs that can be sent.
 * This limit is reached when PACKET_UDP_SERVER_RESPONSE reaches the maximum size of SEND_MTU bytes.
 */
static const uint NETWORK_MAX_GRF_COUNT           =   62;

src/network/core/udp.cpp contains the code that sends the NetworkServerInfo packet, including the NewGRF configuration. This is the code that actually sends the NewGRF configuration:

Code: Select all

		/* Only send the GRF Identification (GRF_ID and MD5 checksum) of
		 * the GRFs that are needed, i.e. the ones that the server has
		 * selected in the NewGRF GUI and not the ones that are used due
		 * to the fact that they are in [newgrf-static] in openttd.cfg */
		const GRFConfig *c;
		uint count = 0;

		/* Count number of GRFs to send information about */
		for (c = info->grfconfig; c != nullptr; c = c->next) {
			if (!HasBit(c->flags, GCF_STATIC)) count++;
		}
		p->Send_uint8 (count); // Send number of GRFs

		/* Send actual GRF Identifications */
		for (c = info->grfconfig; c != nullptr; c = c->next) {
			if (!HasBit(c->flags, GCF_STATIC)) this->SendGRFIdentifier(p, &c->ident);
		}

Sending the NewGRF configuration uses the SendGRFIdentifier() function, which is defined in src/network/core/core.cpp as follows:

Code: Select all

/**
 * Serializes the GRFIdentifier (GRF ID and MD5 checksum) to the packet
 * @param p   the packet to write the data to
 * @param grf the GRFIdentifier to serialize
 */
void NetworkSocketHandler::SendGRFIdentifier(Packet *p, const GRFIdentifier *grf)
{
	uint j;
	p->Send_uint32(grf->grfid);
	for (j = 0; j < sizeof(grf->md5sum); j++) {
		p->Send_uint8 (grf->md5sum[j]);
	}
}

The function sends 4 bytes of GRF ID, and 16 bytes of MD5 checksum, total 20 bytes per NewGRF.

20 bytes per NewGRF, times 62 maximum, gives up to 1240 bytes spent on NewGRF configuration in a NetworkServerInfo packet. Subtract that from the SEND_MTU of 1460 bytes and that leaves 220 bytes for remaining server information, such as name of the game, server version, number of players current and max, size of map, and more. Really not a lot of spare space.

What would it take to expand this limit? It would mean the server info has to be expanded to multiple packets instead of a single one. Since the packets are sent as connectionless UDP data, OpenTTD would have to keep track of which servers the various NewGRF data received, belongs to, and assemble the data that can potentially arrive out of order, or incomplete because of random packet loss. Implementing that is possible, but it's non-trivial.
If you think you can do it, either implement it from scratch or backport it from JGRPP, you are welcome to do so and submit a pull request. Right now, everyone working on OpenTTD just thinks there are more important/interesting things to work on than the network protocol, which is why nobody is doing this. But pull requests submitted will be reviewed, and if good enough quality will be merged.

It's not impossible to change, but it's not trivial, and the limit (still) has nothing to do with transmission speeds or bandwidth limits.