[Patch] NoGo/NoAI town build

[R2dical]

So I'm working on implementing Zuu's NoGo/NoAI patch (hope this is OK )

I've updated to current trunk and added a required #include but can't get it to compile. I get this error (MSVC 2012):

Code: Select all

Error	3	error C2129: static function 'T SQConvert::GetParam<Targ4>(SQConvert::ForceType<T>,HSQUIRRELVM,int,SQConvert::SQAutoFreePointers *)'
 declared but not defined	\trunk\src\core\smallvec_type.hpp	36	1	openttd

The line of code being:

Code: Select all

SmallVector() : data(NULL), items(0), capacity(0) { }

This .hpp was untouched by the patch, which compiled fine before. With this I am quite lost as it seems to be an error with the Squirrel implementation, usually a c2129 error is "A forward reference is made to a static function that is never defined."

Anyone have an idea?
Attached is the patch.

[YNM]

Surely it's not included in the trunk itself, right ?

[R2dical]

You mean is this patch part of current trunk code causing this error on trying to compile the clean nightly?

Then no, this patch was started by Zuu to allow the AI and GS to found a town (and rename one), it is incomplete and not part of trunk.
I am interested in this feature so want to try finish the patch.

[Zuu]

I actually forgot about this patch. I wrote it while I had no access to an OpenTTD compilation environment and I have therefore not had the chance to use a compiler to verify the correctness of the code. Which is also a reason why the patch was never committed.

Thanks for reminding me.

Edit: One of the missing things to do is to run the code generation scripts for squirrel. This is one reason why you get a compilation error.

[R2dical]

I actually forgot about this patch. I wrote it while I had no access to an OpenTTD compilation environment and I have therefore not had the chance to use a compiler to verify the correctness of the code. Which is also a reason why the patch was never committed.

Thanks for reminding me.

Ahh I see, cool

Edit: One of the missing things to do is to run the code generation scripts for squirrel. This is one reason why you get a compilation error.

I see (well not really squirrel integration is beyond me ). So I assume you will retake the reigns on this? At least the patch in the top post was updated to trunk...

[Zuu]

Edit: One of the missing things to do is to run the code generation scripts for squirrel. This is one reason why you get a compilation error.

I see (well not really squirrel integration is beyond me ). So I assume you will retake the reigns on this? At least the patch in the top post was updated to trunk...

Update to r25785 or later, and you will find the new API methods available. Allow up to 24 hours for the docs web to update, but as you did have a look at the patch file, you probably know where to look to figure out the API even before the docs are updated.

[R2dical]

Many thanks and nice work!

Looks like its working great:

TestRun.png (65.86 KiB) Viewed 1057 times

[R2dical]

Not sure if this should go here or in the AI thread, but this is opened already so...

So 2 things, first and most importantly I managed to freeze/cause heavy and continuous lag in OpenTTD with my GS using this new function. It works ok for small maps but on a 1024 x 1024 the game becomes unresponsive. This is the function in my GS which is doing this, let me know if the whole GS will be easier to test:

Code: Select all

// Function call to build towns
function IndustryConstructor::BuildTown() {
	// Check GS continue, to exit
	if(CONTINUE_GS == false) return;
			
	// Display status msg
	Log.Info("+==============================+", Log.LVL_INFO);
	Log.Info("Building towns...", Log.LVL_INFO);

	// Check initiaized
	if (this.INIT_PERFORMED == false){
		// Display error msg
		Log.Error(">IndustryConstructor.BuildTown: Script uninitialized!", Log.LVL_INFO);
		return;
	}
			
	// Loop for number of towns modified by map size, not including initial 1
	local MAX_TOWNS = GSController.GetSetting("DENSITY_TOWN_TOTAL") * MAP_SCALE;
	local CITY_COUNT = 0;
	for(local i = 1; i < MAX_TOWNS; i++){	
		// Random build
		local BUILD_TRIES = 2000;
		local TILE_ID;
		local TOWN_SIZE;
		local IS_CITY = false;
		// Loop until correct tile
		while(BUILD_TRIES > 0){
			// Get a random tile
			TILE_ID = Tile.GetRandomTile();
			
			// Get a random size
			TOWN_SIZE = GSBase.RandRange(4);
			
			// Set city
			if(CITY_COUNT < GSGameSettings.GetValue("larger_towns"))
			{
				IS_CITY = true;
				CITY_COUNT++;
			}
			
			local BUILT_TOWN = GSTown.FoundTown(TILE_ID, TOWN_SIZE, IS_CITY, GSGameSettings.GetValue("town_layout"), null);
			
			// Try build
			if(BUILT_TOWN == true){
				// Multiply size if city
				if(IS_CITY == true)
				{
					// - Get town id
					local TOWN_ID = GSTile.GetTownAuthority(TILE_ID);
					// - Get house count
					local TOWN_HOUSES = GSTown.GetHouseCount(TOWN_ID);
					// Loop till size
					for(local j = 0; j < GSGameSettings.GetValue("initial_city_size"); j++)
						GSTown.ExpandTown(TOWN_ID, TOWN_HOUSES);
				}
				break;
			}	
			BUILD_TRIES--;
		}
		// Display error msg 
		if(BUILD_TRIES == 0)
			Log.Error(">IndustryConstructor.BuildTown: Failed to build town!", Log.LVL_INFO);	
	}
	
	// Display status msg
	local MAP_TOWNS = GSTown.GetTownCount();
	Log.Info("Built " + (MAP_TOWNS - CITY_COUNT) + " towns and " + CITY_COUNT + " cities, for a total of " +
		 MAP_TOWNS + " / " + MAX_TOWNS, Log.LVL_INFO);
}

Yes I know there is lots of inefficient code, but this was just a test Also I am using a clean version of r25785.

Secondly, calling this function (specifically the GSTown.FoundTown) after map generation causes the GS to "skip" its 2500 tick quota...why is this?

[Zuu]

Not sure if this should go here or in the AI thread, but this is opened already so...

So 2 things, first and most importantly I managed to freeze/cause heavy and continuous lag in OpenTTD with my GS using this new function. It works ok for small maps but on a 1024 x 1024 the game becomes unresponsive. This is the function in my GS which is doing this, let me know if the whole GS will be easier to test:

Does this freeze/lag happen when the game is not running in fast forward?

Yes I know there is lots of inefficient code, but this was just a test Also I am using a clean version of r25785.

Secondly, calling this function (specifically the GSTown.FoundTown) after map generation causes the GS to "skip" its 2500 tick quota...why is this?

I'm not sure. Can you provide a minimal GS that shows this?

Edit: I've confirmed this with my test GS. I'll have to look into why this happens.

[R2dical]

Does this freeze/lag happen when the game is not running in fast forward?

Yes, I think this may have something to do with the number of towns my script is set to generate. With the default setup on a 1024 x 1024 it will try build 640 towns, but it freezes way before it gets that far...

Can you provide a minimal GS that shows this?

Attached is a hack and slash version of my GS which replicates all the effects. There is a bit of collateral but I think its simple enough for testing

EDIT: Ninja'd

[Zuu]

Trying to reproduce the lag, I started a temperate 1024x1024 game with your TownConstructor. However it doesn't build any towns. (using r25789)

Edit: Added save game

[R2dical]

Looking into it, but does it do the same without NewGrfs?

Also something really wacky, I updated to r25789 and now everything works fine for me! Both this test GS and my main GS which I have not touched after update. The beginning quota is used and no big map lag Although the ticks in quota seem to run very slowly and scale with map size...

EDIT: I get the same error as you with your newgrfs but works fine with no newgrfs. This test script is very basic (200 lines of code), uses game settings from cfg, null name and a random tile for each try of the new GS build town. If 2000 tries fail it gives the message in the screenshot.

[Zuu]

With about 5000+ towns I've made these observations (using my own test script randomly builds large towns):

• When the town directory window is closed, there is no log
• When the town directory is open and use sort by name or sort by rating there is a lag. (sort by population gives no lag)
• If I disable the script, there is no lag with or without the town directory open (with eg. name sorter)

To conclude, it looks like it is re-building and sorting the list of towns in the Town Directory window while there are lots of tows that is what make the game lag. With the script enabled this has to be done once every tick and the operation probably takes long enough that OpenTTD can't keep up with the normal 74 ticks per second.

My tests was made on a computer with i7 3.6 GHz and with a map size of 2048x2048.

[R2dical]

First thanks for looking into this today.

So looks like the update to r25789 solved the tick quota being skipped thing.

I think I found the cause of the lag, at least when using my scripts. I am using a town names grf (South African Town Names), with this on large maps the ticks go VERY slow then the game stops responding. This happens on my laptop (i3 2.13GHz) and PC (i5 2.66GHz).

I also see the lag with the town list window but thats nothing to the above when in the "Running script" part of new game creation.

If you still have your script could you post it? I would like a peak

[Zuu]

Attaching my test GS. Despite the name, it does only build towns in the current form.

The GS was created to test the APIs to build industries. I now later used it to test the FoundTown API.

[R2dical]

Hi Zuu.

Old discussion I know but I have run into this same issue with lag again

I have rewrote my GS from scratch and now when trying to do the same thing have run into the lag again. This time it is worse however...

During the "running script" set of 2500 ticks my script runs incredibly slowly for up to 5 or 6 then essentially freezes the game.

I tried your script posted above but noticed it performs its build "while the game is running" (not in initial 2500 ticks), using a Sleep(1). I modified it slightly to emulate my script (heavy build-count case admittedly) by making it build 2560 towns with 50 random tile tries for each loop. Sure as nuts same laggy behaviour, but on my second run I got a game crash! (repeatable - r26183 no grfs)

Code: Select all

Crash reason:
 Exception: E1212012
 Location:  7551812F
 Message:   Out of memory. Cannot allocate 33554432 bytes

OpenTTD version:
 Version:    r26181M (2)
 NewGRF ver: 14006645
 Bits:       32
 Endian:     little
 Dedicated:  no
 Build date: Dec 26 2013 13:57:57

Registers:
 EAX: 0012E74C EBX: 00000001 ECX: 00000000 EDX: 77D070F4
 ESI: 00000016 EDI: 00000000 EBP: 0012E79C ESP: 0012E74C
 EIP: 7551812F EFLAGS: 00200246

 Bytes at instruction pointer:
 C9 C2 10 00 89 45 C0 EB ED 3D 01 01 00 00 0F 85 6D 96 FF FF E9 49 96 FF

This happened after the usual heavy lag period, should this go in bugtracker too?

Attached is the GS, yours slightly modified as described.

[Alberth]

Running out of memory is not fixable, other than you buying more memory for your computer, or by you rewriting your script so it uses less memory to do its work.

[R2dical]

Well thats kind of my surprise and point. The above script is super basic, maybe 5 memory variables total and the only DoCommand is the FoundTown one. After running this for the 2560 towns OpenTTD memory footprint jumps from ~150mb to ~1.8 gb. Subsequent runs cause a crash on my 32bit win with 4gb ram.

Running the exact same script but with a Sleep(1) before the build loop (so it runs in game) keeps memory footprint to ~300mb.

So this is normal? Is there something about the initial 2500 ticks that has to be kept in mind to avoid this memory buildup?

[Zuu]

The large memory usage difference sounds strange. It sounds like there might be a leak when towns are funded during world gen. Im a bit interested if this only happens for fund town or can be triggered also by other methods that construct objects. eg new goal, or if the problem is isolated to fund towns.

[R2dical]

From what I have seen it seems isolated to the found town function, building industries and roads in my GS under the same circumstances is fine, but I could be wrong. Should be easy to test though as the memory leak is quite easy to see and repeat as in the previous modified version of your GS.

Maybe this is on the wrong track but the leak may involve town name generation? Last time I came across this, before I saw the leak, I thought I had "solved" it by disabling South African Town Names (which are longer on average). Of course since my rewrite SA names on/off both cause crash now but maybe at different rates?