Page 1 of 4

GS: Industry Constructor

Posted: 29 Jul 2013 19:52
by R2dical
This is a game script, inspired by Aphids excellent IndustryStabilizer, that allows much higher customization as to the initial placement of industries, as well as being able to maintain industry numbers by founding random new ones every X months if below the usual amount.

It has a lot of settings, so it is recommended you read the readme (via ingame button) to take a look how it works. Especially since 2 will change some Advanced game settings.

Features:

Set a number for industries.
Set proportions each for primary, secondary, tertiary and special types.
Choose a distribution algorithm for each above class:
- Scattered: Evenly distributed, with a min distance from towns and other industries.
- Clusters: Same industries in groups, also with a min distance from towns and other industries.
- Town: Close to a town, with a limit per town.
- Random: Same as standard (but strangely loads much faster for big maps)
It can also maintain the numbers on the map, using a refresh period, and a min / max % chance.

Comments, suggestions and bugs are welcome.

Note: To use with PBI, set Maximum distance from town factor = 0.

Read the Readme for hints to avoid some industries not building.

Screenshots:
Constructor.gif
Constructor.gif (70.33 KiB) Viewed 3098 times
Download:
IndustryConstructor-3.tar
(121 KiB) Downloaded 974 times
Readme:
readme.txt
(6.46 KiB) Downloaded 1103 times
Source repository:
Bitbucket

Re: GS: Industry Constructor

Posted: 29 Jul 2013 20:06
by Zuu
R2dical wrote:Its not finished yet, I am posting in hopes of getting some help with finishing and polishing the squirrel code as I am not very familiar with it :)
If you want any feedback on the code, you can attach the GS tar to the forum or link to a public repository where the code can be found.

Re: GS: Industry Constructor

Posted: 29 Jul 2013 20:38
by R2dical
Thanks, I attached my current version .tar.

Unfortunately I use the spiral walker function from the Minchinweb library, a version I adapted to work with GS (only an AI version published), so this won't work without that. What is the best solution? Write my own version? Include what I used in my GS? Release a GS converted version?

EDIT: Also the code is a bit messy, as still WIP, not to mention quite long for a GS (~1200 lines with all comments). If there are some things I would like to ask about specifically is data storage. I'm thinking I need to keep some lists in memory and save containing info on which nodes have industries (for my ClusterBuildMethod function, I have industry ID + number of industries + tile ID) and which towns have industries, for my coming TownBuildMethod function.
In c++ I would use 2D vector, or maybe linked list, but in squirrel all I could get working was 3 1D arrays of same length to use for the same purpose.

Also is there a way to increase the ticks allowed to be used by GS on game start? On large maps (1024x1024) my cluster builder can take a few game months to build all industries. Hmm, I guess my code needs to be more efficient maybe.

Edit: Attachments removed due to update.

Re: GS: Industry Constructor

Posted: 30 Jul 2013 14:36
by MinchinWeb
R2dical wrote:Unfortunately I use the spiral walker function from the Minchinweb library, a version I adapted to work with GS (only an AI version published), so this won't work without that. What is the best solution? Write my own version? Include what I used in my GS? Release a GS converted version?
Why don't you post your unofficial version here and I'll upload an official release to Bananas as soon as I can. I glad to hear someone else finds my library useful :D

Re: GS: Industry Constructor

Posted: 30 Jul 2013 16:48
by R2dical
Great, thanks and posted!

Yeah there are some nice functions in your library that I am using for this script and my AI (especially your pathfinder wrapper). Have you and Zuu considered combining it with superlib to make one single custom lib?

For the converted version, I just batch renamed all "AI" to "GS" and the only conflict I found was in Pathfinder.Road.nut where you had a variable named "RAIL" which (maybe) needed to be named back from "RGSL". API wise I have had no issues so far with missing functions.

Some suggestions, if you would like, are the GetTownTiles function discussed in the API suggestions thread which I use quite a lot as well as Tile.IsIndustry, Tile.GetClosestIndustry and an errorhandler function which seems could be generic for most GS and AI. Fairly simple functions but seem like they could be used often.

Otherwise, regarding this GS, I have completed my TownBuilderMethod (see image). Aside from some of the issues I posted above, I now just need to handle special industries (refineries, banks, watertowers, etc), other climates (temperate only so far tested), improve the speed of existing algorithms and test :)

New version if anyone would like a look.

Edit: Attachments removed due to update.

Re: GS: Industry Constructor

Posted: 01 Aug 2013 21:34
by R2dical
I've now completed this script and this release can be considered the proper version 1 :D Updated first post.

So far all intended features are working and the script performs as expected :P I will put it on Bananas once the NoGo version of the Minchinweb library is up.

Re: GS: Industry Constructor

Posted: 02 Aug 2013 02:51
by MinchinWeb
My library, translated for GameScripts, has been uploaded to Bananas. I tested your game script and nothing broke, so I'll take that as a good sign. If something breaks in my library, let me know and I'll fix it right away!

Here's a link to the file -> MinchinWeb's MetaLibrary v6 for GameScripts

Re: GS: Industry Constructor

Posted: 02 Aug 2013 12:30
by R2dical
Great, thanks for upping that so quickly :D Industry Constructor now on bananas.

Re: GS: Industry Constructor

Posted: 03 Aug 2013 19:48
by Dave
I've been waiting for this for ages. Thank you.

What are the settings I need to make it consistent with Pikka's PBI?

As it stands it creates Secondary and Tertiary industries but not primary ones, no matter what the settings.

Re: GS: Industry Constructor

Posted: 03 Aug 2013 23:44
by R2dical
NP, I've also wanted this feature and now finally learned enough coding to make it :)

I also use the PBI, and tried to describe how to use this with newgrf in the readme, but it does depend a bit on the combination of settings you set. The defaults are designed to work most efficiently with standard game for all map sizes, but more industry types can affect the proportions to make chances 0 sometimes.

Also on higher map/industry count setting the script will overshoot its 2500 tick quota so you will have to wait a bit in game for all the industries to be built, this can happen while paused or while playing ( there is supposed to be some ingame msgs telling player the status but they do not work for some reason). The best is to check the gs debug window for progress (debug lvl 2 gives more details).

The only specific thing I got so far with PBI is that its industries have a max distance of 20 from towns, so you have to set the appropriate setting for the respective algorithms.

All that said mine works fine with default script settings and PBI. What settings are you using, with other newgrfs, map size and climate? Set debug to lvl 2 and see if you get any errors while building the industries in the GS debug.


Also, found some bugs and upped a new version (2) to bananas and 1st post:

Fix: Crash when saving array of industry types, when using certain newgrfs.
Fix: Typo bugs in parameters file, now 2 town settings can be changed correctly.

Changed: Some default settings to make loading a little quicker.
Changed: Loading a game no longer runs the build immidiately, but waits for the set period as it should.
Changed: Updated to Superlib v36.

Re: GS: Industry Constructor

Posted: 04 Aug 2013 00:02
by Zuu
If you want to inform gamers that init is not done yet, you need to detect when the game has started (company exist), and only then show the message. Showing a message before the company exists will not do anything.

I do see the problem though that checking too often for if a company has started before init is done may cost additional ticks and in some situations just make you overshot the budget.

Re: GS: Industry Constructor

Posted: 04 Aug 2013 13:21
by Dave
R2dical wrote:NP, I've also wanted this feature and now finally learned enough coding to make it :)

I also use the PBI, and tried to describe how to use this with newgrf in the readme, but it does depend a bit on the combination of settings you set. The defaults are designed to work most efficiently with standard game for all map sizes, but more industry types can affect the proportions to make chances 0 sometimes.

Also on higher map/industry count setting the script will overshoot its 2500 tick quota so you will have to wait a bit in game for all the industries to be built, this can happen while paused or while playing ( there is supposed to be some ingame msgs telling player the status but they do not work for some reason). The best is to check the gs debug window for progress (debug lvl 2 gives more details).

The only specific thing I got so far with PBI is that its industries have a max distance of 20 from towns, so you have to set the appropriate setting for the respective algorithms.

All that said mine works fine with default script settings and PBI. What settings are you using, with other newgrfs, map size and climate? Set debug to lvl 2 and see if you get any errors while building the industries in the GS debug.


Also, found some bugs and upped a new version (2) to bananas and 1st post:

Fix: Crash when saving array of industry types, when using certain newgrfs.
Fix: Typo bugs in parameters file, now 2 town settings can be changed correctly.

Changed: Some default settings to make loading a little quicker.
Changed: Loading a game no longer runs the build immidiately, but waits for the set period as it should.
Changed: Updated to Superlib v36.
512*512 map, normal industries. Starting 1845. I find all the industries build after the original initialisation. That's not a problem as I start paused anyway.

My first thought was that I use the Early Houses beta, which slashes the population of towns to a 1/4 before 1900. This isn't a problem when PBI usually loads as it disregards it in the first instance, but using the funding method like the GS, this would stop a lot being built. However, removing the Early Houses beta hasn't changed the story.

The debug log is showing the script getting stuck at Oil Refineries. I changed the year to 1880 and it worked, but took a while, so it may just be it takes a while rather than doesn't work in 1845. I will try again.

Now it's done, I realise the normal setting is far too aggressive for me - so I'll probably go back and tweak the figures. 15 of each primary industry on 512*512 is lots :P But at least it's working!

Does the script log say when the script is completed?

I don't play too many games so the long wait time isn't much of an issue for me, but I imagine it will put some off - it could take a couple of hours for massive maps and high industry settings. You do make that very clear though, and that's my only negative critique, as the placement is absolutely superb.

Great job!!! Totally solves a problem I've had with the industry generator since the original TTD!

Re: GS: Industry Constructor

Posted: 04 Aug 2013 14:48
by R2dical
Zuu wrote:If you want to inform gamers that init is not done yet, you need to detect when the game has started (company exist), and only then show the message. Showing a message before the company exists will not do anything.

I do see the problem though that checking too often for if a company has started before init is done may cost additional ticks and in some situations just make you overshot the budget.
Ahh I see, thanks Zuu, this is what I am doing wrong. A problem to fix though, as init runs very quickly but the "build" function is the one that takes long and will usually overshoot the quota on larger maps. I need to send the msgs between these two functions, looping for a check on company existence for a msg will slow down the build script I think as it runs LOTS of nested loops ?(
Dave wrote:My first thought was that I use the Early Houses beta, which slashes the population of towns to a 1/4 before 1900. This isn't a problem when PBI usually loads as it disregards it in the first instance, but using the funding method like the GS, this would stop a lot being built. However, removing the Early Houses beta hasn't changed the story.

The debug log is showing the script getting stuck at Oil Refineries. I changed the year to 1880 and it worked, but took a while, so it may just be it takes a while rather than doesn't work in 1845. I will try again.
The town algorithm in this GS is the most complicated of the lot, and takes the longest. Given that it searches for towns above X pop the Early Houses mod will indeed slow it down as there will be fewer towns that meet the minimum. Try lowering the Town: Minimum population. Oil refineries are abnormal in that they also have the distance from edge requirement, try increasing General: Max distance from edge for Oil Refineries.

If it is still getting stuck you can turn on General: Prospect abnormal industries rather than use methods?, which I included for this specific reason. It will build the Oil Refinery (in temperate) using the original (random) method. This is recommended for larger maps and you can activate it while the script is running too 8)
Dave wrote:I don't play too many games so the long wait time isn't much of an issue for me, but I imagine it will put some off - it could take a couple of hours for massive maps and high industry settings. You do make that very clear though, and that's my only negative critique, as the placement is absolutely superb.

Great job!!! Totally solves a problem I've had with the industry generator since the original TTD!
Thanks for your comments, it is worth noting though that the waiting time is not directly proportional to map size, more towns and more space will increase the chances a lot. Using standard GS settings (and PBI) a 2048 * 2048 (year 1935) took 3 game years to populate (about 15 minutes). You can play while this is happening, and FF speed runs the script faster too. Like you say this is quite a lot of industries also (1600 total), these are the same numbers as the standard also btw (in the Density: Total industries the settings = original) just the proportions modifiers can make it seem drastic sometimes :lol:

I am looking at speeding up the "town" algorithm though, it takes the longest by far :x
Dave wrote:Does the script log say when the script is completed?
Its supposed to display a news event (thanks to Zuu I will fix this maybe), but in the GS window it will say "All industry class types built", once done. The "built current total Y / X" is more for debugging as it disregards the "minimum" modifiers and non-buildable industries.

Re: GS: Industry Constructor

Posted: 04 Aug 2013 15:25
by R2dical
I was wondering if any other coding guys have a suggestion for speeding up the town build algorithm?

Here is the psudocode

Code: Select all

// Loop for each industry to build within each industry type

// Check if industry is not buildable

// Create town list
// Valuate by population
// Remove towns below population (parameter)

// Loop through town list
// Remove towns with current ind ID in if cant build multi per town (parameter)
// Remove towns with max industries already (parameter)

// Check if the list is not empty

// Loop until tries are maxed (tries = town_list_count * 3)
// Sort towns by random
// Get town border size (function)
// Create list of town max/min X/Y size (function)
// Create list for border tiles (town size + (parameter))
// Remove the town rectangle
// Sort tiles by random

// Loop for each tile in list

// Check abnormal industries (to otherwise prospect randomly)

//Check dist from other industries (function)
// - If not null (null - no industries)
// - If less than minimum, re loop

// Try build
And the actual code for the function:
TownBuildFunction.txt
(6.32 KiB) Downloaded 469 times

Re: GS: Industry Constructor

Posted: 04 Aug 2013 17:54
by Zuu
1. If you upload the code with .nut instead of .txt, colour coding will work automatically for more text editors. Also, some may load .nut and .txt in different editors.

2. Code review

In a large map LOCAL_TOWN_LIST will contain quite many elements. At each build try, you valuate all items with a random value. I cannot see that you call .Sort() anywhere, so from my understanding of GSList, it will not actually sort the list based on the new values, but I could be wrong.

If your intention is to pick towns by random each time, I would put all towns in a Squirrel array (as it supports random access). Compute a random array index using GSBase.Rand() and use the random array index to pick a random item. This way the operation to pick a random town will be cheaper than re-ordering the whole list each time.

Code: Select all

		// Sort by random
		LOCAL_TOWN_LIST.Valuate(GSBase.RandItem);
		// Get random ID
		local TOWN_ID = LOCAL_TOWN_LIST.Begin();
I don't have full understanding of the problem you try to solve, but I would recommend to time the various bits of that method and see what takes time. Another way is to time the method as whole, make changes and then use 'restart' in the game console to try again and see what effect your change had on execution time. It is important that you retry using 'restart' and not 'newgame' as you want to use the same random seed etc. each time.

As your method uses randomization in it, you may want to disable that (unless it is that thing you try to speed up) while trying to improve the method as you can otherwise find improvements that in general is not an improvement but happen to improve things with just the random numbers that you got.

Re: GS: Industry Constructor

Posted: 04 Aug 2013 19:25
by krinn
instead of shaking your list randomly to pickup the first item.

random by the size of the list and pickup the item from that result, you'll get the same, a random choice, but you just never randomize the list (that may be huge because of # of towns)

Code: Select all

//LOCAL_TOWN_LIST.Valuate(GSBase.RandItem);
local TOWN_ID =  GSBase.RandRangeItem(0, LOCAL_TOWN_LIST.Count());
LOCAL_TOWN_LIST.RemoveBottom ( LOCAL_TOWN_LIST.Count() - TOWN_ID );
LOCAL_TOWN_LIST.RemoveTop( TOWN_ID - 1);
It depend how AIList implement them, but logically this should gives better results than sorting.
edit:
Note that this alter your LOCAL_TOWN_LIST, so you better work on a copy.

this one should do better job.

Code: Select all

local TOWN_ID =  GSBase.RandRangeItem(0, LOCAL_TOWN_LIST.Count());
local z = 0;
foreach (town, _ in LOCAL_TOWN_LIST)
          {
          if (z == TOWN_ID)    { TOWN_ID = town; break; }
          z++
          }

Re: GS: Industry Constructor

Posted: 04 Aug 2013 20:09
by Zuu
Assuming it is rather a rule than an exception that there are multiple tries, I would guess that a solution like this is faster as it use direct access into an array rather than altering a list or iterating over it to find the item.

This solution however involves copying the GSTownList to a squirrel array which is non-free, but that will only happen once. Subsequent iterations in the while loop will only draw a new random number as index into the array.

Code: Select all

local list = GSTownList();
local array = [];
foreach (t, _ in list) {
    array.append(t);
}
local len = array.len();

while (BUILD_TRIES > 0) { // <--- The loop over tries

    local index = GSBase.RandRange(len);
    local town_id = array[index];

    // try with town town_id

    BUILD_TRIES--;
}

Re: GS: Industry Constructor

Posted: 04 Aug 2013 23:43
by R2dical
Firstly thank both of you very much for taking time to look at this.

I was hoping there would be some obvious program flow mistake but I guess not...

So I spent a while trying your various approaches to randomising the town list, as it sounds like working with lists is what is the CPU hog here. It was hard to tell if they were working or not, as you say Zuu due to the randomising. Similar to your approach krinn I eventually settled on:

Code: Select all

		// Get start ID
		local TOWN_ID = LOCAL_TOWN_LIST.Begin();
		// Get random ID
		for(local i = 0; i < GSBase.RandRange(LOCAL_TOWN_LIST.Count()); i++){
			TOWN_ID = LOCAL_TOWN_LIST.Next();
		}
But still the script seems to take a long time! Eventually I went full hardcore and added debug msgs and signs after every few lines of code to find the culprit, and sadly it seems to be mostly my town tile function :cry: (the one I posted about in the other thread). The build tries loop for each random tile around the town can also take up to around 50% of the time per each build but this is random chance.

So I switched my lengthy town tile function for:

Code: Select all

local TOWN_RADIUS = (GSTown.GetHouseCount(TOWN_ID).tofloat() * 0.3).tointeger();
		local TOWN_TILE_LIST = Tile.MakeTileRectAroundTile(GSTown.GetLocation(TOWN_ID),TOWN_RADIUS);
And now, rather annoyingly, its much quicker :oops:

Only problem is now it gets stuck trying to build Oil Refineries around towns far from the edge. I tried to remove towns further from the edge than max using:

Code: Select all

if(IND_NAME == "Oil Refinery"){
	// Valuate by edge distance
	LOCAL_TOWN_LIST.Valuate(GSMap.DistanceFromEdge(GSTown.GetLocation));
	// Remove towns farther than max
	LOCAL_TOWN_LIST.RemoveAboveValue(GSGameSettings.GetValue("oil_refinery_limit"));
	}
But this results in an error. Is it possible to valuate towns directly like this?
EDIT: Nevermind, I found the sneaky function example in the NoGo docs, got this working. Now the script can populate a 1024 * 1024 in under 2 mins :) Thanks for your help guys.

Also I tried using 2D arrays for towns ID + industry ID's built so far, but I think my syntax is wrong. Can squirrel do multi dimension arrays?

Re: GS: Industry Constructor

Posted: 05 Aug 2013 02:59
by krinn
No you cannot the valuate will pass townID, so you cannot use it with a GSMap function.
but you can do your own that will do it

Code: Select all

function yourclass::TownDistanceFromEdge(townID)
{
       return GSMap.DistanceFromEdge(GSTown.GetLocation(townID));
}
and then do :
LOCAL_TOWN_LIST.Valuate(yourclass.TownDistanceFromEdge);

But you better not do that, the RemoveAboveValue will remove all towns, and not only towns that may prospect oil, but all that have DistanceFromEdge > oil_refinery_limit.

Coudn't find sleep, so here's my optimize version of your code. I don't know if it works, but you have something to read now.

Re: GS: Industry Constructor

Posted: 05 Aug 2013 07:23
by Zuu
R2dical wrote:Also I tried using 2D arrays for towns ID + industry ID's built so far, but I think my syntax is wrong. Can squirrel do multi dimension arrays?
An array can store any Squirrel data type including arrays. Thus you can make a nested array.

Code: Select all

local array = [];
for (local i = 0; i < 10; i++) {
    local item = [];
    for (local j = 0; j < 10; j++) {
        item.append(0);
    }
    array.append(item);
}

// now you can do eg. array[3][2].