Engine Progress (uzurpator)

[Purno]

64x64 pix? OMFG... that's... huge...

I think I'll ask my bro if he wants to draw something. 64x64 pix is just too huge for me, I think...

What's the style you want? Modern (like Verdana, Arial), Old (like Times, Courier), Cartoonish (like Comic Sans), Silly (like... well... hmm...)?

[Hyronymus]

This is not the livestock market where you make deals by bidding, uzurpator. There is no such thing as "Hyronymus' way" so I don't see what there is to gain for TE in accepting your "offer". Neither did I say that I want you out of the project. What I did say is that I want you to stop raging like a bull in a China shop and start coding along the guidelines we did specify yet.

[uzurpator]

64x64 pix? OMFG... that's... huge...

I think I'll ask my bro if he wants to draw something. 64x64 pix is just too huge for me, I think...

What's the style you want? Modern (like Verdana, Arial), Old (like Times, Courier), Cartoonish (like Comic Sans), Silly (like... well... hmm...)?

Pioneer law - If you consider 64x64 too big, make smaller, if you see Times fitting - use time - you decide. I code.

Hyro: EOT.

[Hyronymus]

No, not EOT. This topic will continu (I hope). The discussion about which way to go shall continu elsewhere though.

[XeryusTC]

Uzurpator: I would like to point out that my name is XeryusTC, there is no O in it. Until you get it right I don't have to take you seriously, unless you try to pronounce it in which case I can handle some variety (not too much though).

Further, you didn't even look at the doxygen created for TRoS (which can be found here). It has some pages that are not generated by code but that are completely written by hand. They currently don't really include any explenation on how to use TRoS though. All the other design features are mostly stored on photos of Seniltai's whiteboard or in some random text files which are very lickely not stored in the repository. But currently the source is your best documentation friend, although some of our dependencies' documentation also counts for TRoS a bit (look at RakNet for example).
I would also like to note that both Seniltai and me are still in high school and both doing VWO (pre university) so we spend loads of our time on school or other RL stuff so we don't have that much spare time to work on TRoS or its documentation.
And the fact that TRoS is hacked ATM is mostly the case with the tools/client because some key features are not implemented into the engine yet and will have to be done seperately.

And concidering Pioneer Law, I came up with TRoS before you came up with UzuEngine, so effectively TRoS should be used.

[uzurpator]

Uzurpator: I would like to point out that my name is XeryusTC, there is no O in it. Until you get it right I don't have to take you seriously, unless you try to pronounce it in which case I can handle some variety (not too much though).

Then don't take me seriously.

Further, you didn't even look at the doxygen created for TRoS (which can be found here). It has some pages that are not generated by code but that are completely written by hand. They currently don't really include any explenation on how to use TRoS though.

So why do you mention them?

All the other design features are mostly stored on photos of Seniltai's whiteboard or in some random text files which are very lickely not stored in the repository. But currently the source is your best documentation friend, although some of our dependencies' documentation also counts for TRoS a bit (look at RakNet for example).

Id really prefer to ask on ogre forum what to do, then mow through sources.

I would also like to note that both Seniltai and me are still in high school and both doing VWO (pre university) so we spend loads of our time on school or other RL stuff so we don't have that much spare time to work on TRoS or its documentation.

So it _me_ who is supposed to work on that? _I_ am supposed to document _your_ engine?

And the fact that TRoS is hacked ATM is mostly the case with the tools/client because some key features are not implemented into the engine yet and will have to be done seperately.

ie - TRoS is incomplete and hyped.

And concidering Pioneer Law, I came up with TRoS before you came up with UzuEngine, so effectively TRoS should be used.

Concidering PL none of us came with a workable solution yet - or more specifically - none of us presented an app which shows any aspect of the gameplay, which is documented well enough to be used.

[Hyronymus]

Uzurpator, stop reading between the lines please. XeryusTC doesn't say anywhere that he expects you to write the documentation, he's providing an insight into the hwo and why of unsufficient documentation.

[eis_os]

Could simple both write their engine? And we see what best fits?

And we don't have doxygen anymore, I poked Seniltai numerous times (and will do again now Wink ) to set up doxygen on the server as I was running it before but I didn't install a new http server after a clean windows install.

Now you write:

Further, you didn't even look at the doxygen created for TRoS http://tros.ath.cx/Doxygen/

While I don't want to take any sides, but thats a bit unfair....

PS: I am sorry to name it UzuEngine

[XeryusTC]

Could simple both write their engine? And we see what best fits?

And we don't have doxygen anymore, I poked Seniltai numerous times (and will do again now Wink ) to set up doxygen on the server as I was running it before but I didn't install a new http server after a clean windows install.

Now you write:

Further, you didn't even look at the doxygen created for TRoS http://tros.ath.cx/Doxygen/

While I don't want to take any sides, but thats a bit unfair....

The doxygen is online again since yesterday evening/this morning, but he could have grabbed a version of doxygen himself and compiled it.

So why do you mention them?

Because it shows that our doxygen isn't only made for documentating code.

So it _me_ who is supposed to work on that? _I_ am supposed to document _your_ engine?

If you want to you can go ahead .

ie - TRoS is incomplete and hyped.

Yes, and UzuEngine doesn't exist and is hyped too. I wonder which is worse .

Concidering PL none of us came with a workable solution yet - or more specifically - none of us presented an app which shows any aspect of the gameplay, which is documented well enough to be used.

Why did you mention it in your post then anyway? You are bending your rules to your will ATM, there is no justice in that as I can't ever be right to your rules.

[uzurpator]

The doxygen is online again since yesterday evening/this morning, but he could have grabbed a version of doxygen himself and compiled it.

I think we already established, that I cannot do that, because i cannot download SVN snapshot, and I sure as hell am not going to download file by file.

And I think we also already established, that I will download SVN snapshot, to evaluate TRoS anyway.

Because it shows that our doxygen isn't only made for documentating code.

Lovely. I am supposed to know that? It is your job to advertise, document and make the documentation available and easy to digest.

If you want to you can go ahead .

Having little time to do anything in particular, I'll leave the joys of doing your job, to you.

Yes, and UzuEngine doesn't exist and is hyped too. I wonder which is worse .

Please, find where I have used a construct "is there, but not implemented yet". Besides, you are the least entitled to make such statement, since you are, apart from me, the only person who actually got to see the source.

Why did you mention it in your post then anyway? You are bending your rules to your will ATM, there is no justice in that as I can't ever be right to your rules.

Great. Where is your usable solution? I mean, no toolbox, because toolbox is easy to obtain. I want to see specific TE code - you know - display map, move it around, add some gui, modify the terrain. Basics. And let us not forget the documentation, so a developer like me can jump in and extend.

[uzurpator]

TE task architecture

/*****************************************************************************/

Execution List
Concept:

Execution list is a realisation of a streaming processing of data. The concept
is to divide the data and its operations to a small chunks of code. Each chunk
is somehow dependant on other chunks. The thread of execution takes a first
chunk that can be processed and returns its result. This should unlock other
chunks which now can be processed. The execution is not deterministic - you
cannot tell when a given chunk will be executed.

What is so cool about it?

It scales very well as a number of execution units. If you have a 4 cpu
computer then the program should run abour 4 times faster then on a single
without any modifications, have 8 cpus, 8 times faster*

*this is actually less, due to certain protocol overhead.

To show you an example:

Consider a simple arythmetical expression

a + b * c + d / f * (g + h +j) * i + c - d - e

this can be expressed as:

((a + (b * c)) + ( ( ((d * i) * g) + ((d * i) * h) + ((d * i) * j) ) / f ) + ((c - d) - e))

this forms a graph of execution:
res = result of operation

Code: Select all

        | col1    | col2    | col3    | col4    | col5    |
step 1: | (b * c) |           (d * i)           | (c - d) |
step 2: | (a+res) | (res*g) | (res*h) | (res*j) | (res-e) |
step 3: |         | (res    +  res)   |         |         |
step 4: |         | (res              +  res)   |         |   
step 5: |         | (res/f)                     |         |
step 6: | (res     +  res)                      |         |
step 7: | (res                                  +  res)
step 8: |  res

as you can see to count the expression 13 operations need to be performed
Each step shows operations which can be performed in parallel. 1 CPU will
need 13 ops.
How many ops for 2 cpus:

Code: Select all

step1: (b * c) and (d * i) unlock 4 ops - outstanding ops: 5 
step2: (a+res) and (c + d) unlock 1 op  - outstanding ops: 4
step3: (res*g) and (res*h) unlock 1 op  - outstanding ops: 3
step4: (res*j) and (res-e) unlock 0 ops - outstanding ops: 1
step5: (res+res) unlocks 1 op           - outstanding ops: 1
step6: (res+res) unlocks 1 op           - outstanding ops: 1
step7: (res/f)   unlocks 1 op           - outstanding ops: 1
step8: (res+res) unlocks 1 op           - outstanding ops: 1
step9: (res+res) unlocks 1 op           - outstanding ops: 1
step10: result

so - adding 1 cpu increased speed by 30%.

The list of ops is
(b*c) | (d*i) | (c-d) | (a+res) | (res*g) | (res*h) | (res*j) |
(res-e) | (res+res) | (res+res) | (res/f) | (res+res) | (res+res)

OK - that does not seem too hot, 30% is a bit low. However, the main point is here
no need to recompile to add more processing power. 5 cpus will process this in 8 steps.

OK - still, not too hot.

To get the most of this it is best to consider the type of task you are making, this
scheme works best in:

Games operating on huge game object base (strategies ferinstance)
Rendering and ray tracing
Generally, where there is a huge base of objects that require
some action
Other I haven thought of

-------------------------------------------------------------------------------

Terms:

EXECUTOR:
Executor is the entity which traverses the container of tasks and performs them
one at a time

EXECUTION CONTAINER Exco
Exco is the entity which holds all tasks and contains logic needed to dispatch,
organize, add and remove tasks

EXECUTION CONTAINER ITERATOR Exco::iterator
Execution container iterator is an entity which, if properely created, always
points to a viable task in the Exco

TASK
Task is a single operation that can be invoked by an iterator

Use:

EXECUTOR traverses EXCO with the use of EXCO::iterator and does TASK until
all TASKS are completed

-------------------------------------------------------------------------------

IMPLEMENTATION NOTES:

EXCO uses std::list

There are 3 types of tasks

blocking_task - a task that will gather up to x executors and fire one controlled
task by one of them
single_task - a task that will allow 1 executor to activate it
multi_task - a task that will allow mutliple executors to activate it

-------------------------------------------------------------------------------

Usage:

1.Create your execution list.

ec::exco ex(numOfUsers);

num of users is amount of executors that will traverse a list. Note! It has to
be correct - otherwise all block tasks inside your exco will block indefinetly!

2. define your task

class my_task : public ec::single_task
{
my_task(exco& list) : single_task(list){};
~my_task(){};
void logic() // logic of your task, since this is a single task, then only
// one executor will enter here
{
std::cout << "my_task";
}
};

3. instantiate your task

for (int x = 0;x < 10;++x)
{
ex.insert(my_task(ex));
}

4. instantiate executor

std::vector<executor> execs;
for (int x = 0;x < numOfUsers;++x)
{
execs.push_back(executor(ex,"my_executor"));
}

5. Unless your app exits earlier, you should see 10 times the "my_task"
printed on the screen

--------------------------------------------------------------------------------

classes:

ec::exco - main execution list
ec::single_task - single task
ec::multi_task - multiple task
ec::block_task - blocking task
ec::executor - excutor which will create a thread
ec::thredlessExecutor - executor which will not create a thread

[uzurpator]

Map structure

TE map geometry strucuture.

TE map is comps in the following way

1. the outermost entity is 'terrain', it acts as a container
class and usually has no functionality on its own.

terrain contains leafs - ie small snippets of geometry.

2. Leaf.
Leaf is a collection of 8x8 tiles. It is used for version control.
Version control is used in transferring terrain leafs. IF a client
has a version of a leaf that is older then the version on the server
then the new version will be sent to the client.

Tiles were grouped into leafs to avoid hight memory and runtime
overhead if versioning was applied to tiles.

8x8 size comes from a compromise between runtime overhead, memory
overhead and granulity of leaf division

from the leaf division comes limitation, that map has to be of size
which is divisible by 8 (example: 808*328 tiles)

3. Tile
Tile is the basic unit of map division. It comes in two forms. Encoded
and not encoded.

Not encoded version of a tile is a structure that contains all tile
information.

Encoded version is a 64 bit identifier which is stored in the map array.

'class tileData' hosts encoded version
'class tile' hosts not encoded version

encodig scheme is as such:

The tiles are in clockwise orientation with lower left corner
being the lowest

2-3
| |
1-4

the division to triangles is highest-first, that is, the highest corner
is the origin of the triange division.

One of corner heights must be 0

eccoding (tiles are measured in units - what unit is - see below)
bit 0-3 - 1st corner height (range 0 - 15)
bit 4-7 - 2nd corner height
bit 8-11 - 3rd corner height
bit 12-15 - 4th corner height
bit 16-27 - height of a tile -> 4096 values
bit 28-63 - flags
bit 28 - tile is visible
bit 29-31 - reserved
bit 32-37 - tile's owner ID -> 64 possible owners
bit 38-43 - tile's type -> up to 64 possible types
bit 44-47 - vegetation density (8 values)
bit 48-63 - reserved

the map is orientated in such way

^ lattitude
|
|
|
(0,0)---> longitude

4. nodes

A basic mesurement of a geometry is 1 unit. 1 Units equals 50cm, or 0.5m.
tile has size of 32x32 units and its corners may differ in 15 units.

this gives a tile 32x32 possible points to plant something. Each point is
a node, which may be an origin of a strucutre.


--

Tile types

Tile type is an identifier of tile apperance and behavior functions.

- tile apperance may be connected to tile slopeness, elevation, time of
year, owner ID and other I have not thought of yet
- tile behavior may also be connected to tile slopeness, elevation, time
of year (and other) and may cause the tile to change:
- cost of landscaping
- cost of clearing
- upkeep cost of structures on it
- special effects on the tile
- allowable construction options
- other

[uzurpator]

Transport/Transactions between server and client

The client and server talk to each other with a set of
transactions - usually initiaded by the client.

The tracastions start with an Action rised by the originating entity
(client or server) and are pased to transport layer to process. Transport
layer forwards the data to reaction part of the target entity which
fulfills the request.

The actions are realized by classes inherited from 'action' interface.
Reactions are realized by classes inherited from 'reaction' interface.
Data is sent with the use of 'datagram' class
Instances of commLink class are used as a buffer for reception/sendin
of datagrams.

Example.

The client wants the server to crate a map.

1. client calls its map_make request handler

Code: Select all

-	this_->my_actions_[map_make]->setSize(128,128);
-	this_->my_actions_[map_male]->invoke();

(look at event.h to see how it is handled in the code. setSize is a method
used only by action_map_make class while my actions is general
action pointer associative container. Hence in the code setSize will
be static_casted!)

2. invoke method inserts the datagram with data on the my_out_comm of client.
Then transport recepts it and checks if it needs processing, if not it is forwarded
to my_out_comm of transport where it is read by the server.

3. Server processes the datagram and creates a map according to it.

complete transaction

Code: Select all

client.fetchInput
	eventReader2
		static_cast<action_map_make*>(my_actions_[map_make].get())->setSize(128,128);
		my_actions_[map_make]->invoke();
			my_out_comm_.dispatch(datagram);
transport
	my_in_comm_.fetch(datagram);
		// operate on the datagram data, map make has none, so pass
		my_out_comm_.dispatch(datagram);
server.processCommands
	my_in_comm_.fetch(datagram);
	my_reactions_[datagram.command_]->setParams(datagram);
	my_reactions_[datagram.command_]->invoke();
	
[
//-----------------------------------------------------------------------------
note: datagrams are passed by reference - in exceptiuon to transport layer
which makes a copy, it is marked with an arrow
//-----------------------------------------------------------------------------

Transaction:		exit

Brief:				trigger server to cause application exit
Originatior:		client
Enumeration code:	exit
Datagram params:	none
Postcondition:		server triggers application exit
Trigger:			ESC key press

Flow:
1.client.action.exit
2.transport.reaction.exit -> transport.action.exit
3.server.exit

Additional:			-

//-----------------------------------------------------------------------------

Transaction:		make new map

Brief:				request the server to create a new terrain
Originatior:		client
Enumeration code:	map_make
Datagram params:    (client -> server)
					params[0] - longitunal size of the map in tiles
					params[1] - lattitunal size of the map in tiles
					(server -> client)
					params[0] - longitunal size of the map in tiles
					params[1] - lattitunal size of the map in tiles

Postcondition:		server makes a new map and sends the client new map size
Trigger:			M key press

Flow:
1.client.action.map_make
2.transport.reaction.map_make -> transport.action.map_make
3.server.raction_map_make

4.server.action_set_map_size
5.transport.reaction.set_map_size -> transport.action.map_make
6.client.reaction_set_map_size

Additional			Call setSize method of action_map_make to set params

//-----------------------------------------------------------------------------

Transaction:		require area update

Brief:				request the server to update certain area
originator:			client
Enumeration code:	area_req, leaf_req, leaf_send
Datagram params:    (client -> server)
					note: coords are in LEAFS not TILES!
					params[0] - longitunal coord of lower left corner of requested area
					params[1] - lattitunal coord of lower left corner of requested area
					params[2] - longitunal coord of upper right corner of requested area
					params[3] - longitunal coord of upper right corner of requested area
					params[x] - [y] -  versions of these leafs on the client in the row
					                   ordering ex: for 3,3-5,5
					                   
					                   par[4] == ver of 3,3
					                   par[5] == ver of 3,4
					                   par[6] == ver of 3,5
					                   
					                   par[7] == ver of 4,3
					                   par[8] == ver of 4,4
					                   etc
					                   
					params[0] < params[2]
					params[1] < params[3]
					(server -> client) (1 dgram only, a stream of dgrams is sent)
					params[0] - longitunal coord of a leaf
					params[1] - lattitunal coord of a leaf
					params[2] - msb of leaf(0,0)
					params[4] - lsb of leaf(0,0)
					params[5] - msb of leaf(0,1)
					params[6] - lsb of leaf(0,1)
					:
					:
					params[128] - msb of leaf(7,7)
					params[129] - lsb of leaf(7,7)
					params[130] - version of leaf
					

Postcondition:		client recieves an update on requested leafs
Trigger:			sent every frame
					
Flow:
1.client.action.area_req
2.transport.reaction.area_req -> transport.action.area_req
3.server.reaction.area_req
3.server.action.leaf_send
4.transport.reaction.leaf_send -> transport.action.leaf_send
6.client.raection.leaf_send

Additional:			call setArea of action_area_update to set params

Verbose:			When updating te view area the client does not know wether leafs
                    which make this ara have been updated or not. Henceforth a caching
                    mechanism is introduced, to save on bandwidth when updating an area
                    
                    this works this way:
                    
                    client requests an update on ara 3,3 - 6,6 - this is 4x4 leafs,
                    1024 tiles. 2kB od data. with 20 updates/second it is 40 kB/s.
                    abit to much for typical connection if we were to update each
                    frame.
                    
                    but if, of these 4x4 leafs have not changed, then from the first
                    frame, none needd to be sent if the client stores all leafs
                    locally.
                    
                    Implementation.
                    
                    The client has its own terrain array. Whenever it wants to display
                    some portion of it, it asks the server to update certain area
                    
                    client-side transport layer passes the request though the network
                    to server-side transport which calculates versions of client and
                    server
					
						ex:
							leaf 4,5 was last updated to version 12
							leaf 4,5 was last sent to client was version 12
						conclusion:
							client does not need an update
							
						ex2:
						
							leaf 4,5 was last updated to version 23
							leaf 4,5 was last accessed by client on version 12
						conclusion:
							client needs an update
						
					for every leaf needing an update in response the server
					will emit a series of leaf_send datagrams
					to the client with leafs that need to be updated.
					
					NOTE: leaf_send can be triggered without corresponding area_req
					sever may send leaf updates whenewer it pleases
					
//-----------------------------------------------------------------------------

[uzurpator]

Lastly - some source for our viewing pleasure. Note - this is early, so it is not well documented. That work is in progress.

Currently - frame structure, client server and basic communication are done. Now - display.

[uzurpator]

Nothing much yesterday, just got rid a a few warnings.

Today - render area basics.

[uzurpator]

Twas a conceptual work day, but a successful one.

The required logic has been determined, now to implement the render area interfaces.

[uzurpator]

Implementing 10%
EDIT:
20%

[eis_os]

Hmm, I have to admit I haven't yet time time to really get the whole overview of the code. Sorry

Some points I have to say:
- Will you add some folders for key elements?
I think there will be a lot more files in the future and then you can lose the overview.
- Why is the class "action" defined in action_map.h and not in action.h?
I guess you haven't yet found the right names for all stuff

- will there be a naming convention
I think it's better readable if Interfaces are prefixed.

- usage of size_t
Platforms may have different sizes for the types (say a 32bit or 64bit machine), so I don't think thats a wise to use native types. (say a savegame, network code) Other projects solve this with a types header file. Like U32 S32 and so on... Ohh, the network need to handle endianness aswell...

[uzurpator]

1. Folders

Already doing that. The engine parts are getting compartmentized - I really don't want a bazillion of files scattered left and right

2. Names & Naming convention

Well - for now I want to do as much as possible before my enthusiasm wanes. The more I do, the more will be to later pick up.

3. Types

I am alraedy doing this for much more important reason - to embed some information what kind of value a given variable holds eg: leaf_size means that the variable (or more specifically - function param) holds some sort of leaf data.

Also - mostly member functions are called

Code: Select all

blahBlahBlah()

member variables are

Code: Select all

my_blah_blah_blah_

with exceltion of bool which are

Code: Select all

isSomethingImportant_
// usage
if (isSomethingImportant_)
{
doStuff();
}

functions return by reference, if a function is non void then it either returns bool, POD or a reference to a class (the last used for the 'square bracket' like usage ie:

my_map_(12,14) = 10;

I am also appending empty defines to signify more interesting parts of code for example:

Code: Select all

#define IO // in-out parameter
void someFunction(someClass& blah)
// goes into
void someFunction(someClass& IO blah);

generally I want a program that is 'read' not 'decoded' by a human.

[eis_os]

Well - for now I want to do as much as possible before my enthusiasm wanes. The more I do, the more will be to later pick up.

Sure, code as you like Every line of new code is more progress towards to a working TEmpire.
I wonder where the other people of the Project are?