[concept] Storing each leg of a transfer for cargopackets

[Eddi]

it has been a long standing problem that transferring cargo causes unpleasant side effects with regards to transfers, like negative final profit. on the other hand, you cannot simply store all transfer legs in one cargo packet, since that blows up the memory requirement significantly.

this idea tries to circumvent that problem by only increasing the memory usage if transfers are actually used.

Idea: create a new cargo packet for each transfer leg.

Concept: each time a cargo packet is loaded into a vehicle, a pointer to the vehicle is stored in the cargo packet, to have a link to the vehicle so its payment stats can be updated later. each time a cargo packet is transfered, a new cargo packet is created, which gets the same cargo amount, and a pointer to the old cargo packet. upon final delivery, this "linked list" of cargo packets needs to be traversed to get the original source, and each transfer leg payment can then be calculated as share according to the actually travelled distance and time in transit, so over-estimation can be compensated. for that, the "estimated" transfer profit is removed from the vehicle profits, and the actual "final" transfer profit is added to the vehicle and the vehicle's company.

Implementation details: Member "loaded_at_xy" can be removed, since it will always be the same as "source_xy". member "feeder_share" can be removed, because it can be deterministically recalculated. An index into the vehicle pool and a pointer into the cargopacket pool must be added. some failsafe must be installed when vehicles already got sold when the final delivery is calculated.

[Michi_cc]

Why use a whole cargo packet? Make a linked list from a struct that contains a destination tile, vehicle pointer and a tick counter. Together with the starting tile from the cargo packet, you have the complete movement history of the packet. Storing that is no problem IMHO, OpenTTD is using almost no memory at all compared to other current games.

The more interesting question is how you want payment to be calculated. One feature of the current system is that the final sum of all shown vehicle profits is what your account receives. If you calculate the profit for all vehicle at the end of the journey, the amounts shown earlier don't add up any more. Or just don't show any transfer profit at all.

-- Michael Lutz

[Rubidium]

Let me be pessimistic

What about abusing this to cause denial of service attacks by filling the cargo packet pool? Specifically the case where you move cargo between two stations with forced unloads. You'd be creating cargo packets all the time, filling the pool and increasing the savegame without anyone noticing it until someone starts looking at the savegame.

[ChillCore]

Or just don't show any transfer profit at all.

That might be a good idea in fact ... just show transfer instead of a value and do payments (tranfers and final) only on final delivery.
As tranfer instead of value is shown when transfering, player knows that a payment, later on, is to be expected.
At the same time the "problem" with negative profits on final delivery would be gone?

Let me be pessimistic

What about abusing this to cause denial of service attacks by filling the cargo packet pool? Specifically the case where you move cargo between two stations with forced unloads. You'd be creating cargo packets all the time, filling the pool and increasing the savegame without anyone noticing it until someone starts looking at the savegame.

Maybe a stupid idea but ...
In the assumption visited stations are remembers in order to be able to calculate the payments afterwards (I do not know if this is already the case) ...
If a cargopacket visits the same station twice, clean the pool (¿partially?) and forget everything that happened with it before (¿between the "double station" visits?), except perhaps the payment rate decay?

[Eddi]

Why use a whole cargo packet? Make a linked list from a struct that contains a destination tile, vehicle pointer and a tick counter.

Hm, that may also be viable, especially if you include more stuff like destination/next hop information, which stays either constant, or is useless between each transfer leg.

OpenTTD is using almost no memory at all compared to other current games.

well, yes, but that is because people make thoughts about memory usage first.

The more interesting question is how you want payment to be calculated. One feature of the current system is that the final sum of all shown vehicle profits is what your account receives. If you calculate the profit for all vehicle at the end of the journey, the amounts shown earlier don't add up any more. Or just don't show any transfer profit at all.

i was thinking the final payment calculation should not change. sum of all days in transit, and (manhattan?) distance between original and final destination. then you divide that by some weighted mean between the vehicles [formula to be decided]. the difference between final leg payment and estimated (formerly "transfer") payment may be shown at the last vehicle, or at each transfer vehicle [that may be spammy with town destinations].

example:
packet travelled in vehicle A and B of equal length, but slight detour, leg A was estimated 1000€ transfer income, final income is calculated as 1800€, with a 50:50 split, makes 900€ per leg. vehicle B may now show "Income: 1800€, Transfer: 800€", or vehicle B "Income: 900€" and vehicle A "Income: 900€, Transfer: -100€"

If a cargopacket visits the same station twice, clean the pool (¿partially?) and forget everything that happened with it before (¿between the "double station" visits?), except perhaps the payment rate decay?

that might not be as stupid as it sounds. this way, the number of transfer packets has an upper bound in the order of number of stations, but this comes at the cost of traversing the list in each unload action instead of just the final delivery.

[Expresso]

Perhaps a hash table combining stationid and cargopacketid could be created?

If a cargopacket arrives at a station the game (more precisely: the cargopacket) simply needs to check whether the hash is already known.

If the cargopacket knows tge station, it could simply be deleted or remain in the vehicle. Another option is to have some counter (tracked in the hash?) which allows a cargopacket to visit the same station more then once (to allow for weird constructs).

The delivery and pickup dates of each leg could be tracked in the cargo packet by means of a linked list (struct containing pickup and delivery dates and vehicleid).

Cleanup is also relatively fast this way and payment could be calculated very precisely.

[Tafidis]

Ok, suggested weighted mean formula for transfer chain vehicles 1->2->3->......->N

p(i): estimated profit for each transfer (including last leg making final delivery).
P: actual profit based on manhattan distance and days is in transit (a.k.a route profit going in the bank)

Both the above are calculated same as in existing code.

fp(i): final payment for vehicle i, calculated as follows:

fp(i) = p(i) * P / [ p(1) + p(2) + ... + p(N) ]

The sum fp(1) + fp(2) + ... fp(n) equals P, which means that payments "add up" to what you get in the bank.

Thus, the only thing you need to make this is to accumulate all transfer profits in the cargo packets (which is already done in member feeder_share, only a minor change is required for the last leg to be added).

Now, the individual vehicles' final profits need to be all updated at once when the final delivery is made.
What if a vehicle has been sold? Well, remove its p(i) from the formula, thus the rest of the vehicles share the profit.
Example: vehicle A transfer profit: 500. Vehicle B (delivering) final leg profit: 500. Route profit: 800. With the above formula:

fp(A) = 500 * 800/1000 = 400, and same for fp(B), both vehicles get equal profit, obviously.

But, if you sold vehicle A in the meantime, p(A) is removed from the formula, so

fp(B) = 500 * 800 / 500 = 800 (vehicle B gets the whole profit).

For this to work, each cargo packet must "know" which vehicles it has been loaded on, not stations, basically a list of the following type:

vehicleID, transfer_profit

If a vehicle is deleted, then its entry in the list must also be deleted. Also, its transfer_profit has to be deducted from the feeder_share

Upon final delivery, the list is iterated once and each vehicle in the list gets paid: transferprofit * route profit / feeder_share

Thoughts?

[TERdON]

But, if you sold vehicle A in the meantime, p(A) is removed from the formula, so

Bad idea.

1. We should probably make a new approach mostly compatible with a likely infrastructure sharing patch. Your suggestion would mean transfer cargo would give their payment to the competitor...
2. Even in vanilla OpenTTD, some cargo may be routed via oil rigs, which are accessible to all companies. This is not usually done, though, so it's not a huge problem today.
3. With NewGRFs, the problem is worsened - in e.g. FIRS you have other shared station types.

I suggest instead that either you save a "ghost vehicle" for every sold vehicle until all transfer payments have been made,

or (probably better)

that you have a single ghost vehicle of each type (road, train, boat, plane) and company, to which transports of sold vehicles are assigned. One of each type since we need to split the payment for the official bookkeeping.

Then your approach could be applied as-is. Albeit I might vote for some different weighting function. Specifically, it would be a good idea to use a function that makes sure the partial payments add up to the total one, even in the case of unfortunate rounding... Or a function which gives more payment to the shorter-distance transports, or similar.

Otherwise I agree with your proposal and hope it's implementable. The number of cargo packets can't really be huge, can it? Even if the data structure is e.g. 16 bytes, we would fit millions of such packets. Optimization (merging packets if possible) should of course be done if possible (think some of that was done in the cargodist patch).

[Eddi]

so that means the company must be stored in the cargo/transfer packet, so the income can still be allocated to the correct company even if the vehicle was sold, only the vehicle statistics won't be updated (like which vehicle type made the income, for both vehicle and general company statistics)

[Tafidis]

To be honest, I don't see this happening at all. I mean, what's the benefit of all this? Right now, you can already alleviate the problem (s?) of negative profit for final vehicle making delivery by
(a) adjusting the percentage of payment made to transfer vehicles and/or
(b) designing your network in such a manner that cargo is not transfered further from its destination w.r.t the origin and then moved back towards its final destination.

I don't see a clean-and-easy solution that could be applied directly upon the current system (that includes my "suggestion"). Check PayFinalDelivery() in economy.cpp to understand what I mean. When a delivery is made, each cargopacket is passed to this function and the income for each packet is added to a variable for that final delivering vehicle only. How would you sum up the profits for the transfer vehicles? A CargoPayment for each one in the linked list? Too complex imho and for what gain exactly? You can directly assign the profits to transfer vehicles for each cargo packet of course, but then without income animations for the transfer vehicles because they would be constantly showing tiny amounts of income.

As for infrastructure sharing and the like, I don't know anything about that, so could you please give me some more info. Do you mean a situation where company A makes a transfer and company B makes the final delivery? How does this work currently? I kind of see how it could happen at oil rigs, but then who forces company A to "transfer" there? Company B? It could be a problem for YACD though...

use a function that makes sure the partial payments add up to the total one, even in the case of unfortunate rounding...

Yeah, that can be done with small adjustments (like final vehicle gets paid the "rest"), not a real problem.

Oh, and why exactly should company A still get paid if they sold their transfer vehicle (in our hypothetical "system")?

[TERdON]

To be honest, I don't see this happening at all. I mean, what's the benefit of all this? Right now, you can already alleviate the problem (s?) of negative profit for final vehicle making delivery by
(a) adjusting the percentage of payment made to transfer vehicles and/or
(b) designing your network in such a manner that cargo is not transfered further from its destination w.r.t the origin and then moved back towards its final destination.

Assumption B horribly breaks in transport networks in yacd or cargodist games, where you don't have direct links between each and every station. It also horribly breaks in case you are using feeder transports (common for some playstyles, especially "realistic" ones).

Approach A simply is a very ugly workaround - it works badly when you combine different transport modes - you need to set the percentage VERY low in order to completely avoid the problem if you have e.g. a bus transfer from the airport that only works well one-way (Since the air transport is so much faster). Simply the current approach only works decently if you have transports that have approximately the same speed, if not, vehicles doing the last mile for fast transports will have huge variation in profits - methinks this variation should rather land on the fastest transportation mode. The vehicle stats will simply be useless.

Even worse: If you have a transfer distance which goes "backwards" (A->B->C->D, where C actually is closer to A than D, e.g. due to obstacles in the map), odd things happen, usually implying negative (not even zero) payment. This implies that at least something should be done to the payment system in the long run. Such oddity transport commonly occur if you apply transfers.

I don't see a clean-and-easy solution that could be applied directly upon the current system (that includes my "suggestion"). Check PayFinalDelivery() in economy.cpp to understand what I mean. When a delivery is made, each cargopacket is passed to this function and the income for each packet is added to a variable for that final delivering vehicle only. How would you sum up the profits for the transfer vehicles? A CargoPayment for each one in the linked list?

Yes

Too complex imho and for what gain exactly?

To fix the above mentioned problems. Complexity is still under debate (but I appreciate that it is not obvious that it is at all feasible, even in small-scale games)

You can directly assign the profits to transfer vehicles for each cargo packet of course, but then without income animations for the transfer vehicles because they would be constantly showing tiny amounts of income.

It for sure won't be a clean, small change. That's why I have written that I hope it would work, and not just assumed so. One problem you haven't mentioned is e.g. that payments may spread over several years...

I agree about the animation for all included vehicles probably being unnecessary though (the payment can simply be animated where the goods is delivered, I don't see it as necessarily indicating the vehicle that earned it).

As for infrastructure sharing and the like, I don't know anything about that, so could you please give me some more info. Do you mean a situation where company A makes a transfer and company B makes the final delivery? How does this work currently? I kind of see how it could happen at oil rigs, but then who forces company A to "transfer" there? Company B? It could be a problem for YACD though...

With IS patches, any station (or at least any station belonging to a player allowing it) can be used by any transportation company, opening for new playing styles (e.g. one infrastructure builder catering many different players, etc). Potentially, every station can be used as oil rigs today (theoretically) can be done, although it's very seldomly done. This of course makes the (in vanilla openttd mostly academic) problem of transfers between companies far worse.

use a function that makes sure the partial payments add up to the total one, even in the case of unfortunate rounding...

Yeah, that can be done with small adjustments (like final vehicle gets paid the "rest"), not a real problem.

I was going for something slightly more elaborate (for very long transfers, maybe distributing it iteratively), but that probably is sufficiently good. Maybe I would let the most expensive part of the journey get the remainder, though, instead of simply the last vehicle.

Oh, and why exactly should company A still get paid if they sold their transfer vehicle (in our hypothetical "system")?

Why should they not? The transport has been performed. (I can see the technical issues in implementing it, but not the principle that would be used as a motivation, unless it is "too hard to implement"...).

[Eddi]

I don't see a clean-and-easy solution that could be applied directly upon the current system (that includes my "suggestion"). Check PayFinalDelivery() in economy.cpp to understand what I mean. When a delivery is made, each cargopacket is passed to this function and the income for each packet is added to a variable for that final delivering vehicle only. How would you sum up the profits for the transfer vehicles? A CargoPayment for each one in the linked list? Too complex imho and for what gain exactly? You can directly assign the profits to transfer vehicles for each cargo packet of course, but then without income animations for the transfer vehicles because they would be constantly showing tiny amounts of income.

why do you think a CargoPayment for each vehicle in the list is so impossible/implausible?

[Tafidis]

e.g. that payments may spread over several years

Why is this a problem? If you mean the transfer was made last year and the delivery this year, the transfer vehicle gets paid THIS year. I thought we had established that.

Why should they not? The transport has been performed. (I can see the technical issues in implementing it, but not the principle that would be used as a motivation, unless it is "too hard to implement"...).

Shouldn't be, as each vehicle in the game has a unique identifier iirc.

why do you think a CargoPayment for each vehicle in the list is so impossible/implausible?

I don't. It just seems a lot when you add everything we have mentioned together. Certainly not a small task. We should hire an experienced programmer

"Ghost" vehicles and other hacks are not likely to be included in trunk, though. What would happen IRL? Company A or bus driver A (with his own bus, "hired" by your company) would get paid for his leg by you (airline B), probably before you carried the passengers across the ocean.
So what if the transfer vehicles get paid their provisional profit, and their final payment is adjusted as per my formula or the one Terdon is hiding under his shelf. Thus, transfer animations stay as they are now, and adjustments are made per cargopacket (without the need for extra animations). If vehicle gets sold, then delete from list as per my suggestion. Re-adjust the profits of the other vehicles in the chain by keeping provisional profits of sold vehicles as "constants" (can no longer be changed).

[Eddi]

of course, if it was a small task, i would have done a patch myself...

this thread exists because it is not a small task, and it deserves discussion, idea gathering and problem analysis first.

[Expresso]

One additional point I did not mention in my post:
As pickup and delivery dates are tracked, packets waiting at a station actually COST money; this creates incentive to make sure that cargo is moving around and not waiting at a station. Add to this that generation date (the date the cargo packet first appeared at a station, can also be tracked) and you have a very realistic cargo payment system.

Merging cargo packets becomes a serious problem, as you still need to manage all the indevidual pickup and delivery dates. There are solutions for that, but an already slower payment mechanism becomes even more complex.... and splitting packets could become even more fun.:S

Hhhmmmm.... different solution needed. Why not track with a cargopacket which vehicle has a stake in it (how much, pickup and delivery dates). The cargopacket could then simply track which vehicles have a stake....

Errrmmm... accounting system needed? Unless, ofcourse merging and splitting cargo packets is completely forgotten... which is bad for performance.

[Tafidis]

I too find myself leaning towards a "timestamp" system as a way to manage both this proposed feature, as well as other time-related features in the game as well (e.g. "daylength").

Also, cargo not aging when waiting at a station after being transfered is not intuitive at all, so I would agree on that, too.

[Eddi]

i'm of the opposite opinion, cargo that waits at a station, and even cargo loaded into a vehicle that is still "loading", should not age. only when the vehicle is actually moving [that includes waiting at a signal or in a traffic jam] the cargo should age.

this should make sure that transfer systems e.g. from train to ship, where capacities usually do not exactly match, to not be penalized.

[Tafidis]

this should make sure that transfer systems e.g. from train to ship, where capacities usually do not exactly match, to not be penalized.

But even if they don't match you can you use more/less vehicles. In any case, there should be an incentive to not let them "pile up" (other than the quantity of cargo itself decaying if the rating of that transfer station is very low).

Also, I believe currently the cargo already loaded on the vehicle waiting for a "full load" does age (it has already been moved to the vehicle cargo list), please correct if I am wrong.

Also, w.r.t. calculating partial profits based on time and distance, the formula I proposed is based on the transfer profits because they are already a function of both. However, there is also a "minor" detail here.

Consider the usual feeder chain A->B->C->D.

1) Vehicle transferring at B gets its (transfer) profit calculated as P1 = f (Quantity, Days_in_Transit, Loaded_at_XY)
2) Vehicle transferring at C gets its (transfer) profit calculated as P2 = f(Quantity, Days_in_Transit, Loaded_at_XY)
3) Vehicle delivering at D gets its (normal) profit calculated as P3 = P - (P1+P2), where P=f(Quantity, Days_in_Transit, Source_XY)

In (1), Loaded_at_XY is the same as Source_XY (the origin of the cargo). Thus, this profit calculation is quite straightforward. It is the same as it would be if this was a delivery (see (3))

In (2), the vehicle is paid only for leg B->C, BUT the days_in_transit is always counted from when the cargo was first loaded (station A). Thus, the cargo is already aged when loaded at the beginning of this leg. So, if vehicles (1) and (2) travel the same distance at the same time and for the same quantity of cargo, the transfer profit of vehicle (2) will be less than that of vehicle (1).

In step (3), Days_in_Transit again correctly applies to the equivalent distance (since the distance from the origin Source_XY is considered in this case).

I think the behaviour in (2) is inconsistent because the days_in_transit does not correspond to the distance traveled in that time. Not sure how relevant this is to your concept suggestion Eddi. I just thought this would be the most relevant thread to mention this.

A "timestamp" system would possibly be able to alleviate this "problem" too.

[Eddi]

Also, I believe currently the cargo already loaded on the vehicle waiting for a "full load" does age (it has already been moved to the vehicle cargo list), please correct if I am wrong.

yes, and i think this behaviour is bad.

A "timestamp" system would possibly be able to alleviate this "problem" too.

i didn't quite understood what you meant with "timestamp system". the times are likely not stored because they take too much memory (date alone is 32 bit, tick accuracy would take another 7 at least, plus possible daylength)

[Michi_cc]

A "timestamp" system would possibly be able to alleviate this "problem" too.

How would your timestamp behave with respect to the change date cheat?

-- Michael Lutz