SquirrelGRF: Squirrel scripting for GRF-like files

Forum for technical discussions regarding development. If you have a general suggestion, problem or comment, please use one of the other forums.

Moderator: OpenTTD Developers

Michi_cc
OpenTTD Developer
OpenTTD Developer
Posts: 619
Joined: 14 Jun 2004 23:27
Location: Berlin, Germany
Contact:

Re: SquirrelGRF: Squirrel scripting for GRF-like files

Post by Michi_cc »

cirdan wrote:Which design, if any, would make trunk inclusion more likely?
Please note that this is my personal opinion only and does not reflect what any of the other devs might think.

I've already mentioned, and I'll restate it only for completeness, that a basic 1:1 copy of the NewGRF-system has no advantage over implementing a translator program that transforms something text-like into NFO (or GRF). The code maintenance burden of having a parallel infrastructure is just not worth it. To make it worth, the new system has to provide things NewGRF does not provide. As NewGRF provides a lot of things, this is not easy.

One way to do that is to move the focus away from 'scripting a GRF file' to 'scripting OpenTTD'. Basically, I mean a framework akin to NoAI that provides hooks to directly change the behavior of vehicles, industries, and so on. A vehicle for example has a function to determine speed and acceleration, which is currently built-in into OpenTTD. A hook might now allow the script file to provide it's own acceleration function which means that every vehicle can have a tailored behavior caring for any imaginable traction type. (Side note: given enough basic primitives, even a NewGRF callback could in theory do that, but the resulting interpreter would be more and more like a script VM.) A more basic interface for simply setting graphics and properties could be realised for example by a standard script library. Proper hooks at the proper place could allow for "patches" that can really be loaded just like NewGRFs today, a thing often hoped for by users.

Actually pulling this off would be a hell of a job though. The realm of possible problems is abundant, perfomance and network safety just to pick some. And the resulting diff would probaly be so huge that trunk inclusion would again be very questionable (huge diff = more or less impossible to review). (Side note 2: Ever heard of OpenTTD.GPMI? No? Well, that was a try to make a modular and fully scriptable OpenTTD. You can probably guess its fate though.) All in all, I just can't see such a patch ever be finished, just by the sheer workload alone. You'd have to be very crazy to try.

-- Michael Lutz
DaleStan
TTDPatch Developer
TTDPatch Developer
Posts: 10285
Joined: 18 Feb 2004 03:06
Contact:

Re: SquirrelGRF: Squirrel scripting for GRF-like files

Post by DaleStan »

maquinista wrote:
Michi_cc wrote:No, of course not. But that's not the point, a to-NFO compiler can do legible as well. The point is that something that adds lines over lines without any technical benefit has a chance near zero to ever make it into trunk.
I see lots of technical benefits on squirrel. NFO is difficult. In Spanish set I only can copy the code of other trains and adapt it carefully. Also, I can't add new features, only new trains or translations. Squirrel could make it much easier.
But what technical benefit is there for an in-OpenTTD solution over an external solution that outputs NFO?
Michi_cc wrote:To make it worth, the new system has to provide things NewGRF does not provide. As NewGRF provides a lot of things, this is not easy.
"And can relatively easily be extended to support new things."

For an example of what NFO can be extended to do, I believe that AIs comparable to NoAI's could be implemented in NFO. The biggest challenge would probably be packing the entire AI into 32K sprites, since the callback structure can't loop or recurse. However, since the callback structure can't loop or recurse, the system to break overlong loops/recursions would be essentially superfluous -- it would be replaced with a "I don't know yet; please call me again" response.
To get a good answer, ask a Smart Question. Similarly, if you want a bug fixed, write a Useful Bug Report. No TTDPatch crashlog? Then follow directions.
Projects: NFORenum (download) | PlaneSet (Website) | grfcodec (download) | grfdebug.log parser
User avatar
CommanderZ
Tycoon
Tycoon
Posts: 1872
Joined: 07 Apr 2008 18:29
Location: Czech Republic
Contact:

Re: SquirrelGRF: Squirrel scripting for GRF-like files

Post by CommanderZ »

For an example of what NFO can be extended to do, I believe that AIs comparable to NoAI's could be implemented in NFO. The biggest challenge would probably be packing the entire AI into 32K sprites, since the callback structure can't loop or recurse. However, since the callback structure can't loop or recurse, the system to break overlong loops/recursions would be essentially superfluous -- it would be replaced with a "I don't know yet; please call me again" response.
Oh, so you are telling that the criticism that applies to SquirrelGRF applies to NoAI too? That an extension of NFO could make NoAI obsolete? I would like to see, how many (and how smart) AIs would we have then.
Eddi
Tycoon
Tycoon
Posts: 8272
Joined: 17 Jan 2007 00:14

Re: SquirrelGRF: Squirrel scripting for GRF-like files

Post by Eddi »

you are missing the point. nobody argues that a high level language would not be easier to get people started in writing GRFs/AIs. but that is only the frontend (the user-visible language)

the discussion here is about the backend strategy. NoAI had no such discussion because there was no existing (NFO or otherwise) system, so it was completely free in chosing the backend that fit the purpose. (in this case, a squirrel VM, which later revealed its own kind of problems, but that is not a point in this discussion)

with GRFs the situation is different, there is already a heritage system that Does The Job (tm). it is just a tiny bit clumsy to program in, which is not enough of a valid reason to ditch the existing GRF support, which has taken years to implement properly.

the options at this point are:

1) develop and maintain the two systems in parallel

this has been repeatedly opposed by several of the devs. the maintenance overhead will by far not make up for the questionable benefits of a higher level language, plus you drop TTDP compatibility or have a double squared development and maintenance overhead, because you have to maintain two frontends in two backends, which need to be completely interchangable.

2) develop a system to translate Squirrel (or any other language, see previous attempts) into NFO/GRF

the obvious advantage here is that this requires absolutely no change in OpenTTD, and you have no effort to maintain TTDP compatibility. this is even more encouraged since you have given no evidence that anything that would be possible in your proposed SquirrelGRF that is not already possible in NFO, or that NFO is not easily extendible to.

3) develop a system to translate NFO into Squirrel

has the advantage that you could drop the existing GRF backend, replace it with your SquirrelGRF backend, and provide a legacy layer for loading old GRFs. good luck pulling that off, though. i can't see this happening
User avatar
Korenn
Tycoon
Tycoon
Posts: 1735
Joined: 26 Mar 2004 01:27
Location: Netherlands
Contact:

Re: SquirrelGRF: Squirrel scripting for GRF-like files

Post by Korenn »

Eddi wrote:3) develop a system to translate NFO into Squirrel

has the advantage that you could drop the existing GRF backend, replace it with your SquirrelGRF backend, and provide a legacy layer for loading old GRFs. good luck pulling that off, though. i can't see this happening
Although I agree that it's not very likely this will happen, I think it's the more elegant solution. The squirrel VM is already there and providing functionality anyway, so adding grf functionality to it and dropping the legacy system makes sense. Too bad some people can no longer think outside the NFO/GRF box.
User avatar
Expresso
Tycoon
Tycoon
Posts: 1760
Joined: 09 Aug 2004 00:14
Location: Gouda, the Netherlands

Re: SquirrelGRF: Squirrel scripting for GRF-like files

Post by Expresso »

The thing which has been bouncing around my head is instead of openttd asking the script how to behave, it should be openttd ripping the script appart and converting it to an internal hierarchy.

A simple explanation of this would be a JIT compiler (this implies translation to machine code, but that's not really what I mean).

There are two things contained:
  • The first is data, this shouldn't be a problem to translate to structures internal to openttd.
  • Then there's code, which can be divided in performance critical code, and non-performance critical code. Building a vehicle (for example) doesn't quite need a lot of performance (as the game has to wait for the user to deliver input), so these parts can safely be kept as scripts. However, animations and production callbacks need to be translated to quite a complex tree (mind you, once this is done, all you need to do is traverse the tree and not search, fetch and interpret a bunch of data). Introduction dates might become a simple linked list with an event and a date contained in each node (to name but something simple. Same goes for when a game object becomes obsolete.
A production callback could be divided in a simple tree of classes, however I'm not sure how much performance benefit will result from this. I might be able to provide an example of a factory (using a stockpile and different production input requirements), if need be.
AndyLandy
Engineer
Engineer
Posts: 30
Joined: 10 Sep 2008 20:09

Re: SquirrelGRF: Squirrel scripting for GRF-like files

Post by AndyLandy »

Expresso wrote:The thing which has been bouncing around my head is instead of openttd asking the script how to behave, it should be openttd ripping the script appart and converting it to an internal hierarchy.
I think this tallies up with my thoughts on the matter. I'm not convinced you need a "programming language" to define structures in TTD. A "configuration file" would be a more-appropriate term, and (Open)TTD(Patch) could convert that to whatever internal structures they choose to use.
User avatar
Ralph
Engineer
Engineer
Posts: 87
Joined: 21 Jun 2004 15:25

Re: SquirrelGRF: Squirrel scripting for GRF-like files

Post by Ralph »

AndyLandy wrote:
Expresso wrote:The thing which has been bouncing around my head is instead of openttd asking the script how to behave, it should be openttd ripping the script appart and converting it to an internal hierarchy.
I think this tallies up with my thoughts on the matter. I'm not convinced you need a "programming language" to define structures in TTD. A "configuration file" would be a more-appropriate term, and (Open)TTD(Patch) could convert that to whatever internal structures they choose to use.
This is a good approach up until it comes to defining complex behaviour that may depend on several factors. For defining basic vehicles like the original TTD ones a config file would be fine, but when you want to do interesting stuff like change how a vehicle looks depending on its age, or maybe what engine its attached too, it starts to fall apart. You need 'logic' rather than just 'data' and before you know it your config file is very like a programming language.
User avatar
cirdan
Director
Director
Posts: 539
Joined: 07 Apr 2007 18:08

Re: SquirrelGRF: Squirrel scripting for GRF-like files

Post by cirdan »

Michi_cc wrote:I've already mentioned, and I'll restate it only for completeness, that a basic 1:1 copy of the NewGRF-system has no advantage over implementing a translator program that transforms something text-like into NFO (or GRF).
I guess having a working implementation doesn't count as an advantage...
Michi_cc wrote:One way to do that is to move the focus away from 'scripting a GRF file' to 'scripting OpenTTD'.
Scripting OpenTTD would certainly be the ultimate goal. But one has to start with something.
Michi_cc wrote:Basically, I mean a framework akin to NoAI that provides hooks to directly change the behavior of vehicles, industries, and so on. A vehicle for example has a function to determine speed and acceleration, which is currently built-in into OpenTTD. A hook might now allow the script file to provide it's own acceleration function which means that every vehicle can have a tailored behavior caring for any imaginable traction type. (Side note: given enough basic primitives, even a NewGRF callback could in theory do that, but the resulting interpreter would be more and more like a script VM.) A more basic interface for simply setting graphics and properties could be realised for example by a standard script library. Proper hooks at the proper place could allow for "patches" that can really be loaded just like NewGRFs today, a thing often hoped for by users.
Those hooks are, as I see it, the callbacks. And OpenTTD has as many callbacks as specified by NFO. Adding a new callback to handle 'current vehicle acceleration' (if my physics is right you can compute speed from that) would be as trivial as giving it a name and allowing SquirrelGRF (or whatever) to define it. And a standard script library would be the natural evolution, of course.
Michi_cc wrote:Actually pulling this off would be a hell of a job though. The realm of possible problems is abundant, perfomance and network safety just to pick some. And the resulting diff would probaly be so huge that trunk inclusion would again be very questionable (huge diff = more or less impossible to review). (Side note 2: Ever heard of OpenTTD.GPMI? No? Well, that was a try to make a modular and fully scriptable OpenTTD. You can probably guess its fate though.) All in all, I just can't see such a patch ever be finished, just by the sheer workload alone.
So the probability of inclusion is already zero, and it decreases with complexity...
Michi_cc wrote:You'd have to be very crazy to try.
You don't know how crazy I am. ;-) Undertaking a large project doesn't put me off. Knowing that it will be in vain does.
Korenn wrote:Although I agree that it's not very likely this will happen, I think it's the more elegant solution. The squirrel VM is already there and providing functionality anyway, so adding grf functionality to it and dropping the legacy system makes sense. Too bad some people can no longer think outside the NFO/GRF box.
Err, no, I agree that adding GRF functionality to the Squirrel VM makes sense, but dropping NFO support does not, from my point of view. It's called "backwards compatibility", and it's something important to have. There's a reason why a linux binary compiled for 1.0 still runs today (ever heard users of another OS ask "will the programs that I already have run on this new OS version?").
AndyLandy wrote:I think this tallies up with my thoughts on the matter. I'm not convinced you need a "programming language" to define structures in TTD. A "configuration file" would be a more-appropriate term, and (Open)TTD(Patch) could convert that to whatever internal structures they choose to use.
A "configuration file" would be all right as long as you only want to define static data, that is, a vehicle/house/industry whose behaviour only depends on its given, fixed set of characteristics; in fact, you could even do that in XML. But this doesn't work if you want dynamic behaviour; for instance, a locomotive whose appearance changes depending on the wagons it has attached. NFO goes to great lengths to define a "bytecode language" to encode the use of several variables and the various operations that can be done on them; what I'm proposing is a new system where those variables and operations are encoded as "GetConsistLength() == 6".
Michi_cc
OpenTTD Developer
OpenTTD Developer
Posts: 619
Joined: 14 Jun 2004 23:27
Location: Berlin, Germany
Contact:

Re: SquirrelGRF: Squirrel scripting for GRF-like files

Post by Michi_cc »

cirdan wrote:I guess having a working implementation doesn't count as an advantage...
Only if the there wasn't a working NewGRF implementation.
cirdan wrote:So the probability of inclusion is already zero, and it decreases with complexity...
Not zero, but very low. (Of course, if a patch for example disregards coding style completely, the chance really is zero, even if it is a small patch.)

The bigger the patch, the better it has to be. Making sure the patch can be easily understood is the most important part. One of the key points to achive that is to use either git or mercurial (or even svk) to maintain a patch series instead of one single huge patch. Small, self-contained patches are much easier to review.

-- Michael Lutz
User avatar
Korenn
Tycoon
Tycoon
Posts: 1735
Joined: 26 Mar 2004 01:27
Location: Netherlands
Contact:

Re: SquirrelGRF: Squirrel scripting for GRF-like files

Post by Korenn »

cirdan wrote:
Korenn wrote:Although I agree that it's not very likely this will happen, I think it's the more elegant solution. The squirrel VM is already there and providing functionality anyway, so adding grf functionality to it and dropping the legacy system makes sense. Too bad some people can no longer think outside the NFO/GRF box.
Err, no, I agree that adding GRF functionality to the Squirrel VM makes sense, but dropping NFO support does not, from my point of view. It's called "backwards compatibility", and it's something important to have. There's a reason why a linux binary compiled for 1.0 still runs today (ever heard users of another OS ask "will the programs that I already have run on this new OS version?").
You didn't read what I quoted correctly. The idea there is to drop straight grf support and add a layer on top of the squirrel VM that would translate GRF instructions to the proper VM commands. It would mean both grf and script could be supported while maintaining only one system.

That's also backwards compatibility, but in a different way. Like how the Playstation 3 supports PS2 games through an emulator, but not directly on the system itself.

But as Eddi wrote, it's highly unlikely to ever happen ;)
DaleStan
TTDPatch Developer
TTDPatch Developer
Posts: 10285
Joined: 18 Feb 2004 03:06
Contact:

Re: SquirrelGRF: Squirrel scripting for GRF-like files

Post by DaleStan »

Korenn wrote:You didn't read what I quoted correctly. The idea there is to drop straight grf support and add a layer on top of the squirrel VM that would translate GRF instructions to the proper VM commands. It would mean both grf and script could be supported while maintaining only one system.

That's also backwards compatibility, but in a different way. Like how the Playstation 3 supports PS2 games through an emulator,
Nope. The PS2 hardware is static. NFO is not, so you'd have to maintain the NFO -> Squirrel converter. Plus, you assume that NFO -> Squirrel is even possible. I'm not convinced, mainly due to the troublesome little thing known as "action 6".
To get a good answer, ask a Smart Question. Similarly, if you want a bug fixed, write a Useful Bug Report. No TTDPatch crashlog? Then follow directions.
Projects: NFORenum (download) | PlaneSet (Website) | grfcodec (download) | grfdebug.log parser
User avatar
Ralph
Engineer
Engineer
Posts: 87
Joined: 21 Jun 2004 15:25

Re: SquirrelGRF: Squirrel scripting for GRF-like files

Post by Ralph »

DaleStan wrote:
Korenn wrote:You didn't read what I quoted correctly. The idea there is to drop straight grf support and add a layer on top of the squirrel VM that would translate GRF instructions to the proper VM commands. It would mean both grf and script could be supported while maintaining only one system.

That's also backwards compatibility, but in a different way. Like how the Playstation 3 supports PS2 games through an emulator,
Nope. The PS2 hardware is static. NFO is not, so you'd have to maintain the NFO -> Squirrel converter. Plus, you assume that NFO -> Squirrel is even possible. I'm not convinced, mainly due to the troublesome little thing known as "action 6".
Given we have a NFO parser written in C++, I don't see why one could not replicate it in Squirrel given the right API. It might not be pretty or fast, but possible, unless this action6 is something magical that can be done in one Turing-complete language but not another.
frosch
OpenTTD Developer
OpenTTD Developer
Posts: 988
Joined: 20 Dec 2006 13:31
Location: Aschaffenburg

Re: SquirrelGRF: Squirrel scripting for GRF-like files

Post by frosch »

Note: The following monologue is not directed to the author of the patch. But to all those, who participated in the discussion like "squirrel is the solution for everything". [0]

So, let's see what an AI can do, has to do, and is allowed to do:
  • Most of the time an AI is just dealing with itself, and maybe testing some stuff on the map. Sporadically it executes some command.
  • It does not matter when it does those things.
  • It does not matter how long it takes to do that.
  • There is only one instance of the AI
  • Nothing has to be synchronized during multiplayer. The commands are justs transfered as if they were done by a human.
  • It can use memory as it likes.
  • If it fails, noone cares. The AI is terminated, and one can start a new one if (s)he wants.
Overall this sounds like an ordinary task of any imperative language. You can call commands and you can declare variables as usual and use them where and as you like. And - oh, see - squirrel is an imperative language. And it even seems to work. (Except NAIL is of course better)

Let's see what newgraphics can do, have to do, and are allowed to do:
  • They have to deliver the requested information.
  • The information is needed immediatelly.
  • It is needed on all clients at the same time. And it has to return the same value.
  • But sometimes information is only requested on a single client using the same methods. This may not change the results of subsequent requests which have to be in sync on all clients. So basically you have to store and restore the state like it was before the single-client information was requested.
  • There are thousands of independent small instances, which may not interact. Same reason as before.
  • If it fails, and no default value is useable, the game has to stop. Likely by crashing.
These issues dramatically reduce the amount of memory available for a single unit, and it what ways it can access and modify the memory.
Now let's start coding a nice newgraphics using squirrel.
What do I have to keep in mind:
  • I must not use loops or recursions while the game is running.
  • I must not use write to global variables.
  • I have hardly a byte memory for member variables.
  • The game has to store and restore the VM state all the time to keep the clients in sync.
Uh, that sounds complicated: I can hardly use any of the features an imperative language provides. The restrictions are silly and just chicanery!

Ok - fun aside. I hope I could express why squirrel is a good decision for an AI, and why it is an absolutely bad decision for newgraphics. In fact the restrictions that need to be applied would be so non-intuitive, that using it might turn out harder than nfo. Especially as you might notice to late that something does not work as you are used to from usual imperative-language-tasks.

Additionally there is still the performance question. Why did noone yet profiled it? There is already enough support to do it. Just create a game with 200 trains consisting of 10 units, and enjoy watching them. As the VM-store and -restore stuff is missing, you can expect it to at least not become faster over the time. (Note: I started profiling myself last week, but got interrupted by a hardware failure, that forced me to reboot. After that I did neither found time nor motivation to start it again.)

Finally let's take a look at some nfo properties, that enforce the above mentioned general restrictions.
  • It is Touring-complete on initialisation.
  • It is not Touring-complete while the game is running.
  • While the game is running it can usally only read certain variables and request some information.
  • Certain variables can be changed at well defined points.
So, you think those restrictions are silly, and need to be removed? Nfo is the only language with those silly restrictions? Nfo is doomed to hex values, I can hardly understand half of the action0 page.
  • Nfo/newgrf is not the only language with such restrictions. I know another one:
    PovRay (http://www.povray.org) is a raytracer which you can use to create 3D images. It is not like the usual "modellers" where you click vertexes and move them around until the figure looks as you like.
    PovRay is text-based. Its code is interpreted in two phases:
    - The "parsing": The code is read in, and executed to generate the scene in memory. This part is Touring-complete.
    - The "rendering": The szene is evaluated for every pixel of the final image. You can also "execute" certain code in this phase, but it is not Touring-complete, you cannot access memory as you like, you cannot interact with other stuff, and similiar.
    This does not restrict the szene construction, but keeps the rendering of a single pixel fast (i.e. sane, possible at all, ...) and allows independent evaluations.
    So does that sound similiar to nfo, its task and its restrictions?
    Note: I have never heard someone saying, PovRay is restrictive and hard to understand. Usually one fails dealing with 3D stuff without inmediatelly seeing the result.
  • I guess most people who want to take a look at nfo start at the action0 page. But - uhm - action0 is as complicated as TTD, as its features are closely related to what TTD's default vehicles/industries/houses do. Even more important: A lot of action0 stuff is just obsolete, deprecated, not used by anyone, maybe never used at all. There are better and easier ways to do certain things by callback.
Thanks for reading :)


[0] I have meet a lot people in my life which said C is the best language for every task. I am quite annoyed by those. Usually it turns out that C is the only language they know a bit. And they will never understand that different tasks need different tools. E.g. doing a sed-Task with C is incredible stupid, but it is seen quite often.
⢇⡸⢸⠢⡇⡇⢎⡁⢎⡱⢸⡱⢸⣭⠀⢸⢜⢸⢸⣀⢸⣀⢸⣭⢸⡱⠀⢰⠭⡆⣫⠰⣉⢸⢸⠀⢰⠭⡆⡯⡆⢹⠁⠀⢐⠰⡁
DaleStan
TTDPatch Developer
TTDPatch Developer
Posts: 10285
Joined: 18 Feb 2004 03:06
Contact:

Re: SquirrelGRF: Squirrel scripting for GRF-like files

Post by DaleStan »

Ralph wrote:Given we have a NFO parser written in C++, I don't see why one could not replicate it in Squirrel given the right API.
A parser is not a converter. Korren's idea (AIUI) is to decompile NFO to a sequence of squirrel instructions, and then execute those squirrel instructions in place of the original NFO.

And no, action 6 is not a Turing-breaker. It's a decompiler-breaker. It writes arbitrary bytes to arbitrary locations in the following sprite.[0] And it is, in at least one case, the only sane way to do things.

[0] At least it doesn't write arbitrary bytes to arbitrary locations in arbitrary sprites, as self-modifying ASM code can do.
To get a good answer, ask a Smart Question. Similarly, if you want a bug fixed, write a Useful Bug Report. No TTDPatch crashlog? Then follow directions.
Projects: NFORenum (download) | PlaneSet (Website) | grfcodec (download) | grfdebug.log parser
User avatar
Expresso
Tycoon
Tycoon
Posts: 1760
Joined: 09 Aug 2004 00:14
Location: Gouda, the Netherlands

Re: SquirrelGRF: Squirrel scripting for GRF-like files

Post by Expresso »

Well, what could solve this situation and increase the performance of newgrf is the implementation of some dynamic state machine (in native code), which has a callback to a parser to tell it how to build the tree. This will (as far as I can see) increase performance and isolate NFO from the rest of openttd (which is a good thing). In my opinion this is the best solution.

The basic idea is that whatever openttd is fed in NFO or (maybe, hopefully someday) squirrelgrf is ripped apart and turned in internal structures, so the program doesn't need to execute slow bytecode or even slower scripts.

This state machine basically consists of a bunch of classes dynamically linked together (except for the one keeping track of all relevant information) which can each on it own do only 1 thing, and do it well:
  • An executer class, a class whose only job it is to run through the state machine and building/altering the tree.
  • A register class: A class which contains some registers and a pointer to the next node. Further pointers and strings don't need storage here.
  • A class to fetch data from wherever it may be needed, using a callback, and storing the returned information in a register.
  • A class to return/send (processed) data to wherever it must be put from a register.
  • Some classes to make a boolean comparison between two registers in various ways and be able to branch based on the result of the comparison.
  • Some classes to perform arithmetics on registers (one for each operation) (add, subtract, divide, multiply).
  • Some classes to perform bitwise operations (one for each operation) (and, or, not, xor).
It's hard to imagine something which can't be expressed in something as flexible as this. The advantage to this is that the run-time interpretation gets completely eliminated. Perhaps somebody can dream up something like this idea that's more efficient? ;)

Action 6 might be resolved by keeping track of the affected parts and, if needed, rerouting the pointers temporarily or permanently (basically, yet another simple callback).
User avatar
Ralph
Engineer
Engineer
Posts: 87
Joined: 21 Jun 2004 15:25

Re: SquirrelGRF: Squirrel scripting for GRF-like files

Post by Ralph »

Some thoughts, not really about the topic but design in general.
frosch wrote: [*] The information is needed immediatelly.
[*] It is needed on all clients at the same time. And it has to return the same value.
[*] But sometimes information is only requested on a single client using the same methods. This may not change the results of subsequent requests which have to be in sync on all clients. So basically you have to store and restore the state like it was before the single-client information was requested.
[*] There are thousands of independent small instances, which may not interact. Same reason as before.
...
[*] I must not use write to global variables.
[*] I have hardly a byte memory for member variables.
To deal with multiplayer issues I think you just have to be very firm on what the client side script is allowed to do, by all means execute script that controls animation and sprite selection on the client, but anything that alters the state of the game (industry production, changing vehicle properties, etc) only execute it on the server then tell the clients the result*. Could perhaps be done by defining a subset of the API that the game is allowed to use when its in 'client' mode. Also by all means have the client execute the server side stuff and use the results as a best guess for updating the interface until it can confirm the true result with the server.

*I suspect (but don't know) that the openTTD netcode is probably not a classic client/server model, but rather one that attempts to keep the game's state in sync on all the clients.

On variables: I think its better to keep one instance of the VM for each addon, and then pass it an identifier when dealing with individual vehicles. This way the game's state can be kept simple, and the VM can use the identifier as a handle to store additional information about the vehicle if it wants.
DaleStan
TTDPatch Developer
TTDPatch Developer
Posts: 10285
Joined: 18 Feb 2004 03:06
Contact:

Re: SquirrelGRF: Squirrel scripting for GRF-like files

Post by DaleStan »

Expresso wrote:Well, what could solve this situation and increase the performance of newgrf is the implementation of some dynamic state machine
Does NFO really need a state machine for in-game execution? In game, it's hardly more than a dumb tree-traversal, and, with only a few well-defined exceptions, no node can change the values any other node tests. The game engine can change those values to its heart's content, but the nodes themselves cannot.
Expresso wrote:The basic idea is that whatever openttd is fed in NFO or (maybe, hopefully someday) squirrelgrf is ripped apart and turned in internal structures,
Is this not the way it already works? At least in Open. Patch was built the other way around; NFO *is* its internal structures. And, because NFO was designed primarily to be fast (traversed quickly), it is fast, and you will be hard pressed to generate a faster system, even if you build a JIT compiler into Open. The primary bottleneck is not the tree-traversal, but rather in the calculation of certain 40+x and 60+x variables, which is already native code. If the object is speed, your time is likely much better spend caching those variables[0] than trying to JIT NFO (or anything else) into native code.

JITting not-NFO into NFO, on the other hand, would be a reasonable operation, except that there's no reason for that to be JIT. A static external compiler would work just as well, and likely result in smaller distributions, both for the GRF and for OpenTTD.

[0] In TTDPatch, we cache most of the 40+x vehicle variables. Caching vehicle variable 60 and most of the 40+x station variables has been on the todo approximately forever.
Ralph wrote:To deal with multiplayer issues I think you just have to be very firm on what the client side script is allowed to do,
The NFO language (the language, not the interpreter) already enforces this.[1] Why create a new engine that has to actually enforce this, instead of letting the language do the enforcing?

[1] It's not that the interpreter won't service requests to write to memory, it's that the language provides no way to make such requests.
To get a good answer, ask a Smart Question. Similarly, if you want a bug fixed, write a Useful Bug Report. No TTDPatch crashlog? Then follow directions.
Projects: NFORenum (download) | PlaneSet (Website) | grfcodec (download) | grfdebug.log parser
User avatar
cirdan
Director
Director
Posts: 539
Joined: 07 Apr 2007 18:08

Re: SquirrelGRF: Squirrel scripting for GRF-like files

Post by cirdan »

frosch wrote:...
Thank you again for your insight. I truly appreciate that people are posting constructive comments in this thread.
frosch wrote:Note: The following monologue is not directed to the author of the patch. But to all those, who participated in the discussion like "squirrel is the solution for everything".
Certainly, I'm not one of those! As I said in my initial post, I only chose Squirrel because it was already being used by OpenTTD, so it reduced the changes needed in the compiling-Makefile machinery.
frosch wrote:Let's see what newgraphics can do, have to do, and are allowed to do:
...
You raise several good points here, of which determinism is, my opinion, the most problematic one. Nothing stops a Squirrel script from doing

Code: Select all

function CanWagonBeAttached (wagon)
{
        return Random() % 2;
}
and chaos will ensue. We could prevent this by declaring what constitutes "defined behaviour" for a script, but we don't want the SquirrelGRF spec to end up as the C spec, do we?
frosch wrote:Additionally there is still the performance question. Why did noone yet profiled it? There is already enough support to do it. Just create a game with 200 trains consisting of 10 units, and enjoy watching them. As the VM-store and -restore stuff is missing, you can expect it to at least not become faster over the time. (Note: I started profiling myself last week, but got interrupted by a hardware failure, that forced me to reboot. After that I did neither found time nor motivation to start it again.)
I haven't done any profiling because I don't think there's anything to profile yet. In the last version I posted, the Squirrel code only ran (i) during GRF activation; (ii) in the wagon-attach callback. Since I agree with belugas in that slowdowns at game start are acceptable, and that wagon attachment is a highly interactive process (where a performance decrease would be masked by user interaction), I don't see any interest in having profiling runs now.
frosch wrote:Finally let's take a look at some nfo properties, that enforce the above mentioned general restrictions.
...
So, you think those restrictions are silly, and need to be removed? Nfo is the only language with those silly restrictions? Nfo is doomed to hex values, I can hardly understand half of the action0 page.
I don't find NFO restrictions silly. The're there for a reason, and I clearly understand what it is. This is not about "Hey, I don't like that and I don't understand it either, so let's get rid of it.".
frosch wrote:Nfo/newgrf is not the only language with such restrictions. I know another one:
PovRay (http://www.povray.org) is a raytracer which you can use to create 3D images. It is not like the usual "modellers" where you click vertexes and move them around until the figure looks as you like.
PovRay is text-based. Its code is interpreted in two phases:
- The "parsing": The code is read in, and executed to generate the scene in memory. This part is Touring-complete.
- The "rendering": The szene is evaluated for every pixel of the final image. You can also "execute" certain code in this phase, but it is not Touring-complete, you cannot access memory as you like, you cannot interact with other stuff, and similiar.
This does not restrict the szene construction, but keeps the rendering of a single pixel fast (i.e. sane, possible at all, ...) and allows independent evaluations.
So does that sound similiar to nfo, its task and its restrictions?
Note: I have never heard someone saying, PovRay is restrictive and hard to understand. Usually one fails dealing with 3D stuff without inmediatelly seeing the result.
Well, that may very well be because PovRay is easy to understand, and NFO isn't. (I'm not guessing that PovRay isn't hard--I know it isn't). Could it have something to do with the fact that PovRay is, as you said, text-based, while NFO is hexcode-based?
frosch wrote:Thanks for reading :)
Thanks for writing. :)
frosch wrote:I have meet a lot people in my life which said C is the best language for every task. I am quite annoyed by those. Usually it turns out that C is the only language they know a bit. And they will never understand that different tasks need different tools. E.g. doing a sed-Task with C is incredible stupid, but it is seen quite often.
I'm an experienced programmer, and I know a dozen of languages, so I know that there isn't a cure-all language. (You can do away for most tasks with C and Perl though.) :mrgreen:

Do we at least agree in that NFO is a language designed to be easily read by a computer and not by a human, and that it takes (most) humans longer to read and write than the average human-friendly language? If Squirrel is deemed to be a bad choice as an alternative, and a better option is found (easily parsed by both humans and computers), would OpenTTD be willing to accept it?
User avatar
Korenn
Tycoon
Tycoon
Posts: 1735
Joined: 26 Mar 2004 01:27
Location: Netherlands
Contact:

Re: SquirrelGRF: Squirrel scripting for GRF-like files

Post by Korenn »

frosch wrote:But to all those, who participated in the discussion like "squirrel is the solution for everything"
I hope people didn't get the idea I think squirrelGRF is the answer to everything...

I just think it's wrong to shoot it down before seeing how far it can go, especially with someone willing to invest the time to find out.
Post Reply

Return to “OpenTTD Development”

Who is online

Users browsing this forum: No registered users and 11 guests