Grf Crawler 1.0 - RSS - Category System - RSS feed - XML API

[eis_os]

Hello

In the process of archiving all my TTDPatch related stuff,
I plan to shutdown GRF Crawler December 2011.

Best regards

[Nite Owl]

Might I ask why? I understand that the fruit store is now the place to go for all things OpenTTD. I would also understand if you no longer have the desire or the time to manage the site. I would suggest however that it be left in place even if it is no longer updated just for the sake of having another archive around. I may be mistaken but are there not still some things on the GrfCrawler site that can be found nowhere else? Even if they can all be found somewhere else the site remains useful as an archive due to its ease of use.

[eis_os]

Well it runs on it's own already since ages. My Software tries to be as compatible as possible. So I didn't do any maintenance too.
The last thing was the switch to php5 on the server and I think Owen changed the database connection already two times without me. As you can see the last post here was 2009 so most people aren't really see GRFCrawler important anyway.

I really think this service isn't important any more and people really should think about the future handling of GRFs.
I don't want some kind of savegame depot graveyard (Sorry Owen), so there are two ways:

a) Someone convince me that it needs to stay
b) I will shut it down.

Reference: http://www.tt-forums.net/viewtopic.php?p=928688#p928688

If I ordered my thoughts I will do an post on General TTDPatch about all the TTD related stuff.

[kamnet]

If good software does what it is supposed to do, then why would people need to comment on it? People only tend to comment on software when it's broken or not meeting their needs. GRFCrawler does a very good job. The lack of comments over the last few years tells me that you've done an excellent job with it.

Honestly, if there was no objections, I'd be more than happy to crawl through tt-forums and start adding in all the newgrfs that aren't listed, and update the ones that are. As a user, I much prefer using GRFCrawler to both the forums and even BaNaNaS for finding NewGRFs.

[michael blunck]

Sorry Oskar, I don´t see any need for GRFCrawler to be "shut down":

- apparently, it works as intended w/o any problems now for a long time, hence no excessive maintenance needed,
- people are using it, so there´s obviously a need for it,
- it´s on the tt-forums server, hence no extra fee for bandwidth,
- <to be continued>

If it´s only the "responsibility" you´d like to get rid off, someone else could take it over?

regards
Michael

[FooBar]

Also, new stuff is added every now and then, so from that angle it's also still used.

To me, the added benefit of grfcrawler over bananas is that grfcrawler:
- shows recent changes/additions
- shows pictures
- categorizes it's content

So I'm all in favour of keeping it, even if you want to pull your hands off it yourself.

[Leanden]

What foobar said, grfcrawler makes it easier to browse for new things as they are catagorised, something you cant do on bananas, perhaps you could open GRFcrawler to the tt community so they can upload and maintain their own GRFs, then you don't have to maintain the database yourself, just host the site

EDIT: Just realised that is how it is and that people have been adding GRFs to it

[orudge]

If it´s only the "responsibility" you´d like to get rid off, someone else could take it over?

As I've said to Oskar personally, I would be happy to do this, and I would much rather take control over it than have it shut down. It would be my intention too to actually maintain it (at least in terms of ensuring the content it points to is up-to-date, etc), and potentially we can look into ways of integrating it with Bananas in the future, too. Certainly, though, I see absolutely no reason why it should be shut down, considering it's hosted by me anyway.

[eis_os]

As said in the reference link, I don't like a graveyard or a broken site like ttdgraphics.cjb.net.

About Bananas integration, ttdgraphics.cjb.net was once searched as well, so it needs some kind of official way to be indexed once a day. And provide the necessary stuff to be useful, I don't see much benefit without some categories. Otherwise you still have a bananas like list.

Interface wise: json or XML data. For less traffic and easy loading I would prefer some kind of json dump.

While Uploading wasn't GRFCrawlers key point, I do have some kind of 2.0 Version that I have never finished as the MySQL Database wasn't powerful enough for freetext searching capabilities of GRFCrawler. Aka you need only two letters for searching and get exact matches. (The querybuilder currently creates simply huge sql queries, so it's like a zero optimized table scan = very bad)

Open-Sourcing isn't an option, sorry. The real heart is only ~40KB, but can't be released to individuals, so the integration into other projects isn't possible. The database can be shared...

[Lordmwa]

I love using GRF crawler - will be sad to see it go if it must

[orudge]

For what it's worth, it seems that GRFCrawler had some 9547 visitors last month (350,908 hits in total), excluding common web spiders, etc. Last year as a whole, there were 140,962 visitors generating 5,544,628 total hits. While there may not be all that many new GRFs being added or modified, it seems there's definitely still demand for the site itself.

As I say, if I were to take over the general maintenance of the site, I would ensure that links are kept up-to-date (even if it meant appointing somebody else to oversee that when I can't). And long-term, we can potentially work out a replacement for GRFCrawler as part of another site, if that seems to be what people think is best (not using any existing code, if you're not wanting to allow that). It just seems very stupid and rather sad to effectively close the site. But then I personally hate getting rid of things.

[eis_os]

You can't trust that stats. Only 4937 unique visitors and one host already generating 1645 hits.

Let's ask a bit different, what features does a grf search system need? What features are missing...

Btw. general the owner of the entry needs to maintain the links... Not the admin...

[kamnet]

I don't know what else to say. you asked us to "convince you" that there should be a reason to shut down GRFCrawler. We've all refuted the idea that the service is unwanted, unused and has no interest. This apparently is not enough. We can't read your mind though, so we don't know what you're actually thinking.

And you're unwilling to continue to maintain it yourself, and unwilling to turn it over to somebody or make the code open source.

What was the point of notifying us or asking our opinions of it?

[ChillCore]

Maybe mostly repeating what has alreay been said but I just wanted to make some noise supporting the "don't shut it down please" voice.

Granted I do not use it as often anymore as I once did when I was new to the (Open)TTD(patch) community, no point in downloading the same thing 20 times is there?
The last time I downloaded a grf I needed was not even three weeks ago ... and I do check from time to time for new stuff.
I have found many nice little, and not so little, gems on there. Which I can not find anywhere else and am not allowed to re-distribute myself without violating their lisence, if I want to share it with someone.

Also ...

Only 4937 unique visitors ...

That number is not so low IMHO and like I said ... there is no point in downloading the same thing 20 times and/or visiting 4 times a day.


Like everybody before me said, there is a demand and a use for GRFCrawler, the hits statistics and people still adding new stuff prove that.
I really hope you and orudge can come to some sort of agreement to keep it alive, honestly, it'd be a shame for such a usefull site to dissappear and it will be a sad day when it will be gone.

Is the core code really that topsecret that you can not trust orudge with it? I am sure he will keep it away from public view if really needed.

[andythenorth]

I am thinking about rewriting the Bananas web front-end and extending it to do some of what grf crawler does.

This is partly a project so I can learn the Pyramid web framework. If you're interested in collaborating catch me on the openttd irc.

cheers,

Andy

[eis_os]

Why I ask about it? If people only search for text description, it will be easy to add the necessary changes to "bananas".
The discussion already shows that bananas already allows different licenses, looks like even non-free:
http://www.tt-forums.net/viewtopic.php?p=949403#p949403

We have seen good reason why I shouldn't shut down GRFCrawler, but simply I didn't want to comment on it yet.
I will try to invest time to update and maintain it the next years if I decide to keep it.
As soon as I would allow up/downloading of grfs specially by md5, GRFCrawler would be in direct competition to bananas. I don't want to make bananas maintainer angry. And as I read andythenorth already tries to replicate GRFCrawlers interface.

I still think the statistics are simply wrong, 1645 hits from one host doesn't sound very reasonable or it's some kind of proxy.

About the code, I share code between projects, I don't want to risk security breakage, neither on GRFCrawler nor on other projects. I trust Owen. He trust me too, otherwise I wouldn't have a link to the forums database . I do think Owen shouldn't worry about GRFCrawler, he has enough other stuff to do. Neither do I want that GRFCrawler stresses Owens server, I try to be a good citizen.

Flaws of GRFCrawler:
Non UTF8 safe, breaks xml processing, aka rss feeds. (specific for GRFCrawler, did started on PHP4 only host)
Problems with gzip compression.
Breaking the forums database results in broken html and broken xml.
Ineffective search on the sql server because of table scans. (GRFCrawler does use a ton LIKE % ... % queries)
No negative boolean operator.
Old Smarty version with problems for PHP5.3. (I do have a patched version, but not installed yet)
No caching, except http not modified header.
No real admin interface.
Upload system for (preview-)graphics.

[michael blunck]

["flaws" of current GRFCrawler implementation]

So, the question isn´t really whether to keep GRFCrawler´s current implementation at all, but whether to improve it further, which you´re unsure about due to "little" usage by the audience and lack of time.

At least that´s what I read out of your posts.


Let me assure you that I´m using GRFCrawler on a regular base.

regards
Michael

[andythenorth]

I don't want to make bananas maintainer angry

From discussion, I think there's no risk of that

[FooBar]

Besides, there are still enough people who play TTDPatch. For TTDP, there's no such thing as Bananas, so for TTDP players Grfcrawler is the main source to find newgrfs.

I think the main reason to keep Grfcrawler around is that there is currently no alternative that provides nearly the same functionality. For as long as that's the case, we should keep Grfcrawler, and there will always be people who use it.
In fact, if the plug gets pulled on Grfcrawler any time soon, I'm going to make a grfcrawler myself. And that's how strong I feel it shouldn't be just killed off.

[Ammler]

TTDPatch user can use bananas too, I would rather recommend to merge grfcrawler and bananas. There is no need to maintain 2 such lists. Specially as both current Apps need new maintainers...