Creating a REAL map with real geography and cities

OpenTTD is a fully open-sourced reimplementation of TTD, written in C++, boasting improved gameplay and many new features.

Moderator: OpenTTD Developers

Zanman777
Engineer
Engineer
Posts: 32
Joined: 02 Feb 2012 00:09

Creating a REAL map with real geography and cities

Post by Zanman777 » 13 Oct 2013 09:55

This is a topic to teach and discuss the creation of real maps based on the real world, with the real cities on their correct geographic positions and correct approximate ingame population (scaled to the real world, that is). All of this in a way that prevents sticking to any newGRFs during the creation of the map.

The principle

The problem with making a scenario of a country and saving it in .scn is you burn some newGRFs into the scenario; if you want to change newGRFs, you have to do the scenario all over again.

Well, the only alternative to prepare agnostic maps would be the heightmap system.

So far, heightmaps have been used as a foundation to create realistic (or not) maps, by making everything except terrain shaping by hand. Some people have tried to create "heightmap bumps" or marks for the city locations. I've decided to take this one step further.

In the beginning of the efforts,wWhat I have done is collect the coordinates and population figures of 152 of the most important Portuguese cities & towns, one by one, searching in google. I then created a .txt file ready to be imported into microDEM via GPS->WayPoints; Read From File. (for those of you who don't know, MicroDEM is a program that opens and edits heightmaps). MicroDEM would import the city coordinates as waypoints, which could then be "painted" (in the correct place, of course) into the heightmap with a symbol of your choice. This way, one could create a heightmap WITH terrain symbols on the places the cities should be.

Regarding the populations, I collected the populations of all the 152 towns & cities (again, one by one). Next, I categorized the cities according to population ranges (5000 to 10000 people, 10000 to 15000, etc). This way, I could easily scale the Portuguese towns. All I had to do was choose which OpenTTD population the biggest city would have (in my case, Lisbon). I did a "template conversion" so that Lisbon had 10,000 ingame population. It was a good number to scale up/down mentally; if I wanted a smaller Lisbon, say 5,000, I'd just divide all the other pre-calculated population ranges by 2.

A better method

Moki found an even better way to do this. He found a method that no longer requires the search of city locations & populations one by one. Here he explains it step by step:
Moki wrote:That's how it's done:

1. Download and merge the DEMs for the whole area
2. Download the proper location files from http://earth-info.nga.mil/gns/html/namefiles.htm
3. Import the txt into Open Office Calc. This is done by open, changing the file type to Text CSV and opening the file. The settings should be changed to "Character Set = UTF-8" and "separated by Tab". This and the following steps can probably be done with Microsoft Excel too, but I have no idea, how that works.
4. Data -> Filter -> Standard Filter ... filter by the PC column (that's the importance value/size). For my map, I used PC <= 3, so all cities with values 1-3 are used. This works well for whole country maps. For a regional map, you might want to use PC <= 4 or 5 and for an international map PC <= 2. If you want to use all towns (even the smallest ones without any rating), filter the FC column for P instead. This leaves all "populated places" and removes all the rivers, forests, etc. that you probably don't want to label on your map.
5. Now delete all columns that aren't needed. You need LAT (latitude) LONG (longitude) and FULL_NAME_RG (name). The PC column can also be deleted now as we don't need it anymore.
6. Rearrange the columns in the following order: LAT, LONG, NAME. MicroDEM needs this format or it won't work.
7. Press ctrl-a -> ctrl-v (copy everything)
8. Open the text editor of your choice (notepad or similar), make a new document and press ctrl-v (paste)
9. Save that txt file somewhere. Doesn't really matter where, just remember the directory.
10 Go back into microDEM, and select GPS -> Waypoints from the Menu. A Waypoints window will pop up.
11. In that window click Read from file and select your previously created txt file. All the locations should appear in the waypoints list. The additional data like MGRS, Easting, Northing and Elevation will automatically be calculated.
12. Click on WPTs and define how you want your points to look
13. Check Waypoint names on map near the bottom of the window, if you want the names to be displayed
14. Click Add on map and Plot on map
15. If the points aren't displayed on the map yet, click Force redraw (that blue arrow thing) in the map window.
16. Now you can zoom and move around your map as usual and of course save the map image. All further steps are done with Photoshop/Gimp and of course in OpenTTD.
The city import patch - importing cities directly instead of creating heightmap references

Zydeco kindly uploaded a patch he had made some time ago which allows importing the cities into the scenario editor. This way, you no longer have to use heightmap bump references to place cities manually.
Zydeco wrote:When I was making a few maps from heightmaps a few years ago, I made a patch that would read towns and coordinates from a CSV file, and place them automatically, or place a sign if the towns failed to found.
I've updated it for the current version, and tried with your data (although I set all the town sizes to small, it would require manually growing them.
It also fails to place a few, because my edge coordinates are a bit off, but it saves lots of work finding where to place towns.

I've uploaded the patch, heightmap and the towns file I made by rearranging yours into the format needed by my patch, and the heightmap resulting from running the import_towns path/to/towns-pt.txt command in the editor, without doing any further adjustments
So, using Moki's method and Zydeco's patch, we can directly fetch a geographic map and city locations & populations from the net, and then import this data into OpenTTD. By other words, we would take steps 1 to 4 of Moki's method, and then use the import town patch. As Moki said:
Moki wrote:With a tiny bit of reformatting (which can be automated), the data should also be usable with Zydeco's patch. It's definitely possible to automatically translate the PC ratings directly into town sizes and the coordinates are already in the right format.
To-do

We can now fine-tune the patch so that it can take input from Moki's method directly, whilst preventing the coordinate offset he mentioned. Next, all that's left is taking the input of the cities' population and scaling them directly to ingame population, maybe according to some user-defined parameters like "minimum city population" or "maximum city population"... We're really close to having the whole process complete :D



Next is the town import patch provided by Zydeco, along with a Win32 Build compiled by Moki with the import_towns patch applied.

The patch adds a command to OpenTTD console: import_towns. So, in order to use it, you open the scenario editor, open the console (pressing the \ button over the tab keyboard button) and write import_towns <file_name>. At least this is the principle of it, not sure if the very specific details are correct (a more solid guidance for this shall be written here later). Stay tuned!
Attachments
import_towns.patch
The town import patch kindly provided by Zydeco, who is the true author of the patch.
(6 KiB) Downloaded 227 times
openttd_r25866_import_towns.zip
Moki's build of OpenTTD with import_town's patch already applied. Ready to fire up!
(5.82 MiB) Downloaded 216 times
Last edited by Zanman777 on 17 Oct 2013 23:42, edited 11 times in total.

moki
Traffic Manager
Traffic Manager
Posts: 131
Joined: 06 Jul 2007 08:48
Location: Wismar, Germany

Re: THE ULTIMATE Portugal map toolkit - usable for others to

Post by moki » 13 Oct 2013 10:12

Oh my, that's genius! And you tell me about this after I just finished my current map?! :twisted:

I'm a bit confused about the coordinates text file... The first 2 numbers (38.713800 -9.139400) are obviously decimal coordinates, followed by the name, but what does the rest mean? Did you write the files yourself or did you use some tool to compile it?

Zanman777
Engineer
Engineer
Posts: 32
Joined: 02 Feb 2012 00:09

Re: THE ULTIMATE Portugal map toolkit - usable for others to

Post by Zanman777 » 13 Oct 2013 11:02

What I did was input the decimal coordinates on the waypoint dialog of microDEM. After you insert the decimals, microDEM calculates those other values and exports them on the txt file too. And I used microDEM itself to create the .txt, btw ;). It was VERY lengthy, took me like 6 hours to add the coordinates one by one... Must be a better way, though! :D

moki
Traffic Manager
Traffic Manager
Posts: 131
Joined: 06 Jul 2007 08:48
Location: Wismar, Germany

Re: THE ULTIMATE Portugal map toolkit - usable for others to

Post by moki » 13 Oct 2013 11:45

Ah, ok. It's one of those methods that doesn't make the task faster or easier, but more precise.

I'm not an expert with microDEM, but I'll definitely have a try these days. It's one powerful tool, so there's probably an easier way that's not as obvious or well documented.

As I mentioned in the other thread, I usually just place the biggest (or most important) cities by exact map data and estimate the rest. It works like "There are those 2 big cities and about 2/3 of the way between them there's a smaller town"... I just place markers for the cities and then guess where the small town should be.

[edit] There were a few decent sites that sounded promising for coordinates in bulk... mostly unavailable now, thanks to the US government :evil:

Zydeco
Engineer
Engineer
Posts: 11
Joined: 01 Nov 2007 08:57
Skype: maczydeco
Contact:

Re: THE ULTIMATE Portugal map toolkit - usable for others to

Post by Zydeco » 14 Oct 2013 10:17

When I was making a few maps from heightmaps a few years ago, I made a patch that would read towns and coordinates from a CSV file, and place them automatically, or place a sign if the towns failed to found.
I've updated it for the current version, and tried with your data (although I set all the town sizes to small, it would require manually growing them.
It also fails to place a few, because my edge coordinates are a bit off, but it saves lots of work finding where to place towns.

I've uploaded the patch, heightmap and the towns file I made by rearranging yours into the format needed by my patch, and the heightmap resulting from running the import_towns path/to/towns-pt.txt command in the editor, without doing any further adjustments
Attachments
import_towns.patch
patch that adds the import_towns command
(6 KiB) Downloaded 192 times
towns-pt.txt
csv file with map and town coords
(5.25 KiB) Downloaded 248 times
Portugal.png
heightmap I used
(164.35 KiB) Downloaded 7 times

Zydeco
Engineer
Engineer
Posts: 11
Joined: 01 Nov 2007 08:57
Skype: maczydeco
Contact:

Re: THE ULTIMATE Portugal map toolkit - usable for others to

Post by Zydeco » 14 Oct 2013 10:18

maximum number of attachments is 3, here is the generated scenario:
Attachments
portugal-test.scn
(232.48 KiB) Downloaded 232 times

moki
Traffic Manager
Traffic Manager
Posts: 131
Joined: 06 Jul 2007 08:48
Location: Wismar, Germany

Re: THE ULTIMATE Portugal map toolkit - usable for others to

Post by moki » 14 Oct 2013 21:40

@Zydeco:
Could you provide a windows binary for that patch (no need for updating it to current trunk)? It looks very useful, but I never got the hang of patching and compiling... :oops:



I found a nice source for coordinates. You can download the "GeoLite city" CSV from here and import it into Open Office Calc. The database is rather extensive (about 440000 locations worldwide and even towns with less than 5000 inhabitants are listed) and already gives you proper decimal coordinates. It's not perfect, but easier than finding every single location by google maps or similar tools.
Maybe it's even possible to export that data from Open Office in a format that's already microDEM-friendly after removing the cities you don't want to include.

Sadly, no luck for decent population data so far. For Germany the data is pretty easy to find (we count and document everything here ;) ), but a comprehensive list like the one for the coordinates is hard to find.


[EDIT]
Woah, forget that last link... go directly to http://earth-info.nga.mil/gns/html/namefiles.htm and be stunned. They got regularly updated data on a huge number of cities and geographic features for the whole world sorted by countries. I just downloaded the data for Germany and there are about 83000 "populated places" (that's everything but the very smallest villages) and about 87000 other locations like administrative regions, vegetation features (forests, basically) and a few others.
There's no exact population data, but a rather useful "populated place classification" that sorts the towns by importance from 1 to 5. I have no idea how they do this classification, but the biggest towns are usually the most important, so it's a good starting point for town sizes in OpenTTD.
They got no data on the US, though... just a link to the United States Geological Survey which just got (you guessed it) shut down :(

[EDIT 2]
The file from the previous link can be imported into microDEM and it should be possible to generate a map overlay from it. I have no idea, how to do that, though... even if nobody finds out how to do that, it's still good data and a convenient format, though.

Zanman777
Engineer
Engineer
Posts: 32
Joined: 02 Feb 2012 00:09

Re: THE ULTIMATE Portugal map toolkit - usable for others to

Post by Zanman777 » 16 Oct 2013 17:18

WOOW!

Zydeco, that patch is AWESOME!

I've been wondering if it would be possible to do that for a long time, but it seemed like a complex thing to try on my own.

Heck, I know C++ (even though I'm a beginner), I think you just gave me a lust to get the hang of patching and creating/editing patches for OpenTTD! :D

Moki, that's a great link you got there. Nice find :D

moki
Traffic Manager
Traffic Manager
Posts: 131
Joined: 06 Jul 2007 08:48
Location: Wismar, Germany

Re: THE ULTIMATE Portugal map toolkit - usable for others to

Post by moki » 17 Oct 2013 00:41

Sooo... I found a way to make microDEM do almost exactly what I want :D See the attached image for the result. This is not the finished heightmap, but the perfect basis for further work. Of course, the names can be turned off and the color and size of the points can be changed as needed.

That's how it's done:

1. Download and merge the DEMs for the whole area
2. Download the proper location files from http://earth-info.nga.mil/gns/html/namefiles.htm
3. Import the txt into Open Office Calc. This is done by open, changing the file type to Text CSV and opening the file. The settings should be changed to "Character Set = UTF-8" and "separated by Tab". This and the following steps can probably be done with Microsoft Excel too, but I have no idea, how that works.
4. Data -> Filter -> Standard Filter ... filter by the PC column (that's the importance value/size). For my map, I used PC <= 3, so all cities with values 1-3 are used. This works well for whole country maps. For a regional map, you might want to use PC <= 4 or 5 and for an international map PC <= 2. If you want to use all towns (even the smallest ones without any rating), filter the FC column for P instead. This leaves all "populated places" and removes all the rivers, forests, etc. that you probably don't want to label on your map.
5. Now delete all columns that aren't needed. You need LAT (latitude) LONG (longitude) and FULL_NAME_RG (name). The PC column can also be deleted now as we don't need it anymore.
6. Rearrange the columns in the following order: LAT, LONG, NAME. MicroDEM needs this format or it won't work.
7. Press ctrl-a -> ctrl-v (copy everything)
8. Open the text editor of your choice (notepad or similar), make a new document and press ctrl-v (paste)
9. Save that txt file somewhere. Doesn't really matter where, just remember the directory.
10 Go back into microDEM, and select GPS -> Waypoints from the Menu. A Waypoints window will pop up.
11. In that window click Read from file and select your previously created txt file. All the locations should appear in the waypoints list. The additional data like MGRS, Easting, Northing and Elevation will automatically be calculated.
12. Click on WPTs and define how you want your points to look
13. Check Waypoint names on map near the bottom of the window, if you want the names to be displayed
14. Click Add on map and Plot on map
15. If the points aren't displayed on the map yet, click Force redraw (that blue arrow thing) in the map window.
16. Now you can zoom and move around your map as usual and of course save the map image. All further steps are done with Photoshop/Gimp and of course in OpenTTD.

With a tiny bit of reformatting (which can be automated), the data should also be usable with Zydeco's patch. Since I didn't find the time to work out compiling, I didn't look into that any further. It's definitely possible to automatically translate the PC ratings directly into town sizes and the coordinates are already in the right format. Almost automatic real world map generation, here I come ;)
Attachments
GERMANY_PC123.PNG
(2.02 MiB) Downloaded 7 times

TinyMusic
Tycoon
Tycoon
Posts: 1423
Joined: 02 Feb 2013 02:03

Re: THE ULTIMATE Portugal map toolkit - usable for others to

Post by TinyMusic » 17 Oct 2013 04:44

Zydeco wrote:When I was making a few maps from heightmaps a few years ago, I made a patch that would read towns and coordinates from a CSV file, and place them automatically, or place a sign if the towns failed to found.
I've updated it for the current version, and tried with your data (although I set all the town sizes to small, it would require manually growing them.
It also fails to place a few, because my edge coordinates are a bit off, but it saves lots of work finding where to place towns.

I've uploaded the patch, heightmap and the towns file I made by rearranging yours into the format needed by my patch, and the heightmap resulting from running the import_towns path/to/towns-pt.txt command in the editor, without doing any further adjustments
Which revision did you use? It doesn't compile well with r25868
Formerly known as UseYourIllusion.
Java Scenario Found Here - Version 2 out
[tweɪ̂ pû tɕʰì wɔ̀ mǐlɤ lû tɕʰìŋ nì pɑ́ŋmɑ̌ŋ]

Zanman777
Engineer
Engineer
Posts: 32
Joined: 02 Feb 2012 00:09

Re: THE ULTIMATE Portugal map toolkit - usable for others to

Post by Zanman777 » 17 Oct 2013 13:35

This keeps getting better and better... Mods, I think this topic should be stickied! :D We're figuring out how to make REAL maps almost instantly, many people might be interested in that!

EDIT: btw UseYourIllusion, he used revison 25866, I believe (it's in the patch file).

moki
Traffic Manager
Traffic Manager
Posts: 131
Joined: 06 Jul 2007 08:48
Location: Wismar, Germany

Re: Creating a REAL map with real geography and cities

Post by moki » 17 Oct 2013 17:58

I tried applying Zydeco's patch but didn't succeed. I tried r25866, but make gives an error (screenshot attached) and doesn't create openttd.exe. Compiling the clean trunk with MinGW (Windows 7 64bit) works fine, so it must be something about the patch... as I said, I've got no experience with programming, patching or compiling and have no idea, where I could have gone wrong.
Attachments
compile_error.png
compile_error.png (3.25 KiB) Viewed 7491 times

TinyMusic
Tycoon
Tycoon
Posts: 1423
Joined: 02 Feb 2013 02:03

Re: Creating a REAL map with real geography and cities

Post by TinyMusic » 17 Oct 2013 18:00

moki wrote:I tried applying Zydeco's patch but didn't succeed. I tried r25866, but make gives an error (screenshot attached) and doesn't create openttd.exe. Compiling the clean trunk with MinGW (Windows 7 64bit) works fine, so it must be something about the patch... as I said, I've got no experience with programming, patching or compiling and have no idea, where I could have gone wrong.
I think I have the same error too.
Formerly known as UseYourIllusion.
Java Scenario Found Here - Version 2 out
[tweɪ̂ pû tɕʰì wɔ̀ mǐlɤ lû tɕʰìŋ nì pɑ́ŋmɑ̌ŋ]

Zydeco
Engineer
Engineer
Posts: 11
Joined: 01 Nov 2007 08:57
Skype: maczydeco
Contact:

Re: Creating a REAL map with real geography and cities

Post by Zydeco » 17 Oct 2013 20:11

I didn't realise windows has no asprintf function, this updated patch should work on all platforms.

Updated: will it work now?
Attachments
import_towns_3.patch
(6.03 KiB) Downloaded 176 times
Last edited by Zydeco on 17 Oct 2013 20:40, edited 1 time in total.

moki
Traffic Manager
Traffic Manager
Posts: 131
Joined: 06 Jul 2007 08:48
Location: Wismar, Germany

Re: Creating a REAL map with real geography and cities

Post by moki » 17 Oct 2013 20:22

Just tried (again with r25866) and got another error this time. Writing code that works for all platforms seems to be really complicated :?
Attachments
compile_error_02.png
compile_error_02.png (3.19 KiB) Viewed 7452 times

Zydeco
Engineer
Engineer
Posts: 11
Joined: 01 Nov 2007 08:57
Skype: maczydeco
Contact:

Re: Creating a REAL map with real geography and cities

Post by Zydeco » 17 Oct 2013 20:41

moki wrote:Just tried (again with r25866) and got another error this time. Writing code that works for all platforms seems to be really complicated :?
It seems I was missing a (char*) cast, I've updated it in import_towns_3.patch (in my previous post). I hope it works now, I sadly don't have any windows build system set up for testing myself

moki
Traffic Manager
Traffic Manager
Posts: 131
Joined: 06 Jul 2007 08:48
Location: Wismar, Germany

Re: Creating a REAL map with real geography and cities

Post by moki » 17 Oct 2013 23:12

Thanks you, Zydeco! :D
After several tries it finally worked. I attached the windows build for those who want to experiment with the patch but aren't willing to learn about patching and compiling under Windows. It took me almost a whole day to get everything working, so I can fully understand that ;)
Just start the game as usual, go to the scenario editor, import your heightmap and type "import_towns" (without the ") into the console and go from there. I'm sure you'll figure it out.

Now I'm off to find the fastest and most effective way to use the combined DEM and NGA data for this patch. I hope to find the time for writing a comprehensive tutorial (from heightmap download to finished scenario) tomorrow. Almost automatic scenario generation for the ambitious mapmaker (AASGAM, maybe?) is only a step away.

@Zanman777: Could you please link this in the first post? People might find it useful.
Attachments
openttd_r25866_import_towns.zip
Import Towns patch by Zydeco. Win32 build.
(5.82 MiB) Downloaded 112 times

Zanman777
Engineer
Engineer
Posts: 32
Joined: 02 Feb 2012 00:09

Re: Creating a REAL map with real geography and cities

Post by Zanman777 » 17 Oct 2013 23:44

Done, Moki! ;)

Haha, loved the AASGAM idea! :lol:

Hey, Moki, did you by chance use Code::Blocks? I've been trying to open & compile OpenTTD with Code::Blocks to no avail... Fails at importing the "generate" subproject. Ah well.

Hmm... How about EARWASTE? (Easy Automated Real World Area Sampler for Tycoon Editor)? :lol:

moki
Traffic Manager
Traffic Manager
Posts: 131
Joined: 06 Jul 2007 08:48
Location: Wismar, Germany

Re: Creating a REAL map with real geography and cities

Post by moki » 18 Oct 2013 02:45

@Zanman777
No, I used MinGW/Msys and followed the tutorial step by step. Took me a while, but in the end it worked remarkably well.
And please do choose whatever name you like. Silly acronyms seem to be a tradition around here, so I won't object at all :lol:

@Zydeco
I may have found a rather obvious bug in your patch. I just tried importing towns into a testing map and get the "Error reading edge coordinates..." every time when East-Coordinates are involved. For example 50N,20W,40N,30W works fine while 50N,20E,40N,10E or 50N,10E,30N,10W give me the error. The same thing happens for towns. If a town has East-coordinates, the console says "Coordinates for [Town] are wrong...", no matter if the town's in the selected area or not. South and North work fine, though - even with an area like 10N,20W,10S,40W. Everything's fine as long as the eastern hemisphere isn't involved. So the patch is great for Portugal but problematic for Germany ;)
I just had a look at the code (love it - very clean and well-commented), but it's 4:45am and my head doesn't want to learn about programming anymore.

Zydeco
Engineer
Engineer
Posts: 11
Joined: 01 Nov 2007 08:57
Skype: maczydeco
Contact:

Re: Creating a REAL map with real geography and cities

Post by Zydeco » 18 Oct 2013 08:34

moki wrote:@Zydeco
I may have found a rather obvious bug in your patch. I just tried importing towns into a testing map and get the "Error reading edge coordinates..." every time when East-Coordinates are involved. For example 50N,20W,40N,30W works fine while 50N,20E,40N,10E or 50N,10E,30N,10W give me the error. The same thing happens for towns. If a town has East-coordinates, the console says "Coordinates for [Town] are wrong..."
I've tried myself (on Mac OS X) and I haven't had problems with edges like 50N,20E,40N,10E or 50N,10E,30N,10W, and towns in it.
I think the problem may be your locale expecting comma instead of period as decimal separator, I'm not sure how that works on windows, on unix I'd run the program with LC_ALL=C
I can probably update the patch later today to parse decimal numbers in a locale-independent way

Post Reply

Return to “General OpenTTD”

Who is online

Users browsing this forum: No registered users and 2 guests