Updated extra-large-maps patch, version 12 (for r23992)

[CARST]

What does that mean exactly?

Is so much memory and CPU power needed only for making the map or for every player who plays it.


What i want to say is, i usually play only with two friends, they have average computers. Mine computer on the other hand is the bleeding-edge technolagy available. So if i make the map and host it (meaning i save it too for all players) will we all able to play a 16384x8192-map?


And if you say i could make a map that large already, what do i have to change in the config files?

[Bilbo]

What does that mean exactly?

Is so much memory and CPU power needed only for making the map or for every player who plays it.


What i want to say is, i usually play only with two friends, they have average computers. Mine computer on the other hand is the bleeding-edge technolagy available. So if i make the map and host it (meaning i save it too for all players) will we all able to play a 16384x8192-map?


And if you say i could make a map that large already, what do i have to change in the config files?

Generating such large map takes also quite long time (few minutes), but tnis is done once and only on the server. Playing the map (running the map, vehicles i it, etc ...) then needs to be run on each of the machines. So everybody need strong computer, as when playing over network, each player runs their own copy which is synchronized with server (basically, server send you the commands from you and other players and you update your state of the simulation with them)

Once the map become too complex for some player to simulate, he will drop out inevitably.

And if you want to have even larger maps - you will need to get sources, apply my patch for large maps and then modify src/map.h:

Find line with
#define MAX_MAP_TILES (4096 * 4096)
Change it to
#define MAX_MAP_TILES (16384 * 16384)
(for example, although actually trying 16K x 16K maps will only cause OpenTTD to run out of memory, unless you happen to have 64bit system and 3 GB of memory (6 GB if you want to load or save such large map))

That number is the only limit limiting number of tiles. Then compile openttd and you're done :)

Also, for such large maps, I recommend adding also smallmap-zoom patch from http://bugs.openttd.org/task/54 for more comfort navigation through the world.

[CARST]

How do i apply it to the patch?

I created a dir with a fresh ttd installation, made an TortoiseSVN-Checkout, then applied your patch with TortoiseSVN. Max mapsize is still 2048x2048.

Any idea what i am doing wrong?


And as a second thought, any possibility to apply it to ChrisIN?

[Bilbo]

How do i apply it to the patch?

I created a dir with a fresh ttd installation, made an TortoiseSVN-Checkout, then applied your patch with TortoiseSVN. Max mapsize is still 2048x2048.

Any idea what i am doing wrong?


And as a second thought, any possibility to apply it to ChrisIN?

Weird. Did you get any warnings or errors from applying the patch?

Current ChrisIN already contains my large maps patch. So if you want to have even larger maps, just get ChrisIN source, modify MAX_MAP_TILES and compile it.

Well, if it won't help, I'll do svn export, apply my patch, raise the limit of MAX_MAP_TILES and put the sources somewhere so you will just download it and compile it.

[CARST]

Hmm ... i'm sure i am doing something wrong ...

I created a fresh ttd-directory, made svn-checkout, edited the line of your patch in the chrisIN.diff (map.h => max_map_tiles) and then applied the diff to the directory.

Now, i can choose all map sizes. Even 16384x16384 and so on, but if choose a size greater than 4096x4096 it says the map has too much tiles...

[Bilbo]

if you are selecting larger map size in one dimension, you have to select smaller in another - there is limit on number of tiles, which is about 16 million. so you can have 4096x4096, 8192x2048, 16384x1024,32768x512, etc ...
16384x16384 would require about 5 GB of memory, and 4096x4096 is largest size that most computers can handle...

None of currently manufactured CPU's can handle 16384x16384 in realtime anyway, so there is no point in enabling this size...

[CARST]

Hi.

I would like to create a 16384x8192-map. That should be possible for 2GB oder 4Gb computers. Why not let the players decide how large the maps should be? (The could be some kind of a warning showing how much memory will be needed.)

I would create the map, host it and save it for my friends i play with. So i think it should be possible to play such a large map (16384x8192)


Regards,
CARST

[sforget]

Under 32 bit Windows, each program is limited to 2GB of RAM - regardless of how much ram teh computer actually has, each program can only use 2GB. This is a Limitation of th Windows core and not easy to get around. Maintaining windows compatibility means that you can NEVER allow the program to exceed 2GB in Memory requirements. So Extremely Large maps are probably not possible nor practical.

A person could make a Linux (or other OS) specific version that can handle more RAM usage, but it wouldn't be usable in windows, nor would window users be able to access a server or saved game based off a specific deisgn like this.

[Bilbo]

32bit windows programs, compiled with special flag ("3 gb aware") can be run on 32bit windows run with special flag, that will allow using 3gb of memory (rest of 4gb virtual memory (=1gb) is reserved for kernel space). Linux have same option, although I think the binaries there does not need any special flag. But for most cases, in 32bit OSes (both windows and linux), 2GB of memory is limit for one process. I'll remake the patch a bit, into one version, that have quite "reasonable" limits in terms of harware (4m tiles + max size 8192, which would then likely be accepted in trunk) and then second "test" version possibly without any imposed limits at all (but more than 16M tiles won't work well anyway on most machines) - as the limits are set in constants, such patch would be quite simple :)

[CARST]

Okay, i understand that, but i have Windows Vista 64bit with 4GB memory, so i think i should be able to create a large map like 16000x8000.

Well, not everyone has the possibility to play, host or save such maps, but i think there should be the option to enable "ultra-large-maps"; maps that are bigger then the "extra-large-maps".


Everyone has the choice then and if the maps are too large, the person has just to disable "ultra-large-maps" and use the "extra-large-map"-mode then.

[athanasios]

Won't such a huge map be unplayable in multiplayer?

Note: Under 4GB, memory is reserved for graphics card memory. So if you have an 640 MB graphics card this will be removed from your 4MB memory. It is crazy because it is not used (memory hole), but that is how it works. There are workarounds, but sometimes they don't work. So someone has to exclude this from his memory when calculating how much memory he needs to play a huge map.

[Bilbo]

Okay, i understand that, but i have Windows Vista 64bit with 4GB memory, so i think i should be able to create a large map like 16000x8000.

Well, not everyone has the possibility to play, host or save such maps, but i think there should be the option to enable "ultra-large-maps"; maps that are bigger then the "extra-large-maps".


Everyone has the choice then and if the maps are too large, the person has just to disable "ultra-large-maps" and use the "extra-large-map"-mode then.

It will work, but you have to recompile openttd as 64bit application. First I'll concentrate of making extra-large to get into trunk with reasonable limits for current PC's, then I can concentrate on making some ultra-large patch (64K x 64K would be probably the hard-to-break limit, as coords are stored as 32bit int I think - such large map would work only in 64bit systems and you'll need 80GB ram for it. I think most people will stick to something much lower that that limit :)

[CARST]

So does that mean it won't be possible that you just release a version which allows 16000x8000 maps?

I don't care about 16000x16000 and larger, just up to 16000x8000.

I don't need a recompiled OpenTTD for 64bit, just that you set the limits a little bit higher.

Would you be able to lift the limits up? Or are these maximum tiles "hard-coded" into the game itself?



I can only repeat i really would appreciate this,
regards,
CARST

[DaleStan]

Note: Under 4GB, memory is reserved for graphics card memory. So if you have an 640 MB graphics card this will be removed from your 4MB memory.

ITYM "4GB". And yes, this is the way it works, so that software can write directly to graphics memory without all sorts of nasty hacks or expensive instructions.
There are two memory "holes" on most x86 systems: One hole from 640K to 1M, for the 286 and up, which is used, most notably, for VGA memory. The other hole, which first possibly appeared with the 386, is usually in the 3.5GB to 4GB range, for memory on PCI and AGP cards.

The RAM that would normally service the addresses in the hole is usually ignored, but some systems may remap the RAM that would otherwise service the upper hole to addresses above the 4GB barrier.

For more on this subject, see, among other things, these pages.

[Bilbo]

So does that mean it won't be possible that you just release a version which allows 16000x8000 maps?

I don't care about 16000x16000 and larger, just up to 16000x8000.

I don't need a recompiled OpenTTD for 64bit, just that you set the limits a little bit higher.

Would you be able to lift the limits up? Or are these maximum tiles "hard-coded" into the game itself?



I can only repeat i really would appreciate this,
regards,
CARST

16384*8192 won't work on any 32bit machine.
Actually it would work, but you won't be able to save or load the game (there is not enough adressable memory for that and attempt to load or save will end up with crash)
Unless the kernel is using 3gb/1gb for user/kernelspace instead of standard 2gb/2gb - then it would work even for 32bits.

You will need a 64bit system (and 64bit openttd) to run such large maps. If you have 64bit Vista, then take any 64bit compiler, raise the limits in the patch (make MAX_MAP_TILES equal to (16384*8192)), recompile openttd using 64bit compiler and it should work then .... for this size Openttd will need ~ 2.5 GB memory at most (when saving/loading), so 4GB is fine.

[DaleStan]

16384*8192 won't work on any 32bit machine.

Well, not without PAE and AWE. The former requires both OS and hardware support, and the latter requires making potentially Interesting changes to the OpenTTD source.

[Mchl]

ITYM "4GB". And yes, this is the way it works, so that software can write directly to graphics memory without all sorts of nasty hacks or expensive instructions.
There are two memory "holes" on most x86 systems: One hole from 640K to 1M, for the 286 and up, which is used, most notably, for VGA memory. The other hole, which first possibly appeared with the 386, is usually in the 3.5GB to 4GB range, for memory on PCI and AGP cards.

At last! Someone has explained the question, to which even Gigabyte's technical support couldn't (or wouldn't) give a reasonable answer to me!

[Bilbo]

16384*8192 won't work on any 32bit machine.

Well, not without PAE and AWE. The former requires both OS and hardware support, and the latter requires making potentially Interesting changes to the OpenTTD source.

PAE will help OS to support more physical memory, but virtual memory would still be limited (2gb per process, or 3gb with 3/2 user/kernel split)
AWE (Address Windowing Extensions) is specific to MS windows and with PAE together it will make it possible, however, access to the tile array is rather random and constant remapping of memory would probably slow down the process. May be nice to try it, but I doubt anybody would want to program that, as switching to 64bit is usually much easier :)

[DaleStan]

Well, for that *particular* size, with 3/1 split (I note you're ignoring things like the vehicle array, so I'll do so too) it's just a matter putting the SL copy of the map in a different window than the gameplay copy. But for larger maps, yes, you'd have to have multiple windows for the map array.

Just be thankful that Intel didn't give us the lovecraftian tentacled horror that would be 32-bit segmented addressing!
Does [ebp+ebx] default to ss or ds? How about [ebx+ebp]? Do the answers to either of those depend on the assembler or assembler options used? Would the answers to any of the above be different with esp instead of ebp?

[Bilbo]

Well, for that *particular* size, with 3/1 split (I note you're ignoring things like the vehicle array, so I'll do so too) it's just a matter putting the SL copy of the map in a different window than the gameplay copy. But for larger maps, yes, you'd have to have multiple windows for the map array.

Just be thankful that Intel didn't give us the lovecraftian tentacled horror that would be 32-bit segmented addressing!
Does [ebp+ebx] default to ss or ds? How about [ebx+ebp]? Do the answers to either of those depend on the assembler or assembler options used? Would the answers to any of the above be different with esp instead of ebp?

Size of the vehicle array is quite neglibigle. I think there is hard limit of 5000 vehicles of each type per player, independent of the map size (which would need probably a raise to take full advantatgen of such large space), so you end up with maximum of 160000 vehicles. I don't have the sources here currently, but I just assume that openttd excluding the map array (code, vehicles, all other data not directly dependent on the map size) can fit in about 20-30 mb. Which is quite neglibigle once map array reach sizes in order of gigabytes.