Assisting with the (digimob) Project -

Regarding the size, I agree that Google Documents is not there yet, but it is true for other programs. I use to do this type of file indexing on my hard disks and until Excell 2003 the number of row in a spreadsheet was limited to 65536. Spreadsheets are not databases, but you can go a long way with them today.
In fact for the file format, I would suggest to stick to the basic: csv (comma separated value). You can do a lot of work on it using simple tools like grep and only convert to more powerful format at the last minute. That will also speed up the time you need to generate your file.

Also, until you are ready to crowd source the sanitization of your data, the database is not really necessary.

In fact for the file format, I would suggest to stick to the basic: csv (comma separated value). You can do a lot of work on it using simple tools like grep and only convert to more powerful format at the last minute. That will also speed up the time you need to generate your file.

Well... some file names have commas in them for one thing. But for another, currently the text data is parsed into a list of objects (classes) in python. I can use those objects to render their data into any format I want. I can see no good reason to render first to one file format and then another. This would actually lengthen the time needed to generate the spreadsheet. String searching, etc, can all be done very easily from within python.

Also, until you are ready to crowd source the sanitization of your data, the database is not really necessary.

Maybe, maybe not. There would be some advantages. For now, I'm thinking that Google Docs is unlikely to be satisfactory for searching through the output. In the meanwhile, though, it's true that there are at least some things I can fix up with the script, which will be useful regardless of the final output format. I'll get on those things in the next few days, hopefully. I think they'll be pretty straightforward. The real "bugs" that remain, though, are in one way or another spreadsheet-related.

Google Docs may well be insufficient to our needs... As for hosting a database, could we use free hosting? These forums are hosted through Forumotion, and I have to figure we might could find free hosting for that, too. But, given my ignorance of these matters, perhaps I'm completely mistaken.

Google Docs may well be insufficient to our needs... As for hosting a database, could we use free hosting?

I'll do a little bit of research, but at first glance, the answer seems to be yes. I'm also fairly ignorant on this topic, so I'll see what I can figure out. I'm looking through these sites first:

http://www.free-webhosts.com/free-mysql-database.php

And I'll also look to see if I can find other sites that might not require the extra coding.

Just a bit of an update:

I've found a MySQL hosting site that's free and looks like it could be fine. The only drawback is that I may have to code an interface in PHP, I'm not sure. But in any case, there are a few other testing steps to take. I checked out one other site and it had terms and conditions that made me think twice. I might look at a couple more in the next few days, but I do have some deadlines coming up...

I've dedicated very little time to this in the last week, and that may continue for a little while. I have a feeling that whenever I manage to sit with this for a few hours, I'll be able to basically figure out what to do and how, so hopefully there should be more to report in another week or two. I feel a bit lame about the snail's pace at the moment, this stuff shouldn't have to take quite this long, but it's not quite priority #1 right now. Nevertheless, work continues, however slowly.

There's no race.

In the early days, trying to get the quickest turn-around possible, we released digests every few weeks... It got maddening. Now, with quarterly releases, we have a better opportunity to organize and compile digests, and hopefully that translates into superior formatting.

Take your time. Maybe some more assistance will manifest.

Just did a little work and thought I'd post a bit of an update. I've just checked that things are working with python and SQL at the site I'm doing testing on, and all seems well for now. What this means is that next time I sit down with it, I'll be trying to populate a test database with entered values like what I showed before on the google docs spreadsheet -- but (if we're lucky) without omissions. From there, setting up a simple web interface shouldn't be too much of a jump. So, that's my current plan, and where things are at.

neutralrobotboy wrote:Just did a little work and thought I'd post a bit of an update. I've just checked that things are working with python and SQL at the site I'm doing testing on, and all seems well for now. What this means is that next time I sit down with it, I'll be trying to populate a test database with entered values like what I showed before on the google docs spreadsheet -- but (if we're lucky) without omissions. From there, setting up a simple web interface shouldn't be too much of a jump. So, that's my current plan, and where things are at.

Excellent!

Ok, I've got my script outputting to a database and so far it looks good! What this means is that we have a kajillion entries sitting there in a MySQL database right now. Next step is the web interface for searching, I guess. I'm no big PHP wiz, but I think getting the basic functionality up and running shouldn't be too hard. Then there will be a test site up somewhere where searches are possible. After that, some scheme for the remaining data entry tasks can be worked out. As before, I can't give any guarantees about timeframe for doing this, but I'll try to get on it when I can.

Been very consumed recently,
My apology for lack of input.
the last two weeks in september I should have much more free time.
trying to imbibe such a large dose of info on a foreign substance (non urban//successful long term out door living) has been eating my time.
It has been a while since I have checked the releases info..... but I tried to acquire as much as possible for a while so if me having a good amount of the docs on my machine and me having some spare time can help let me know.
((i may be slightly tech-tarded >.<))

o if me having a good amount of the docs on my machine and me having some spare time can help let me know.
((i may be slightly tech-tarded >.<))

Actually, this could be VERY helpful! Once things are set up properly, it would be excellent to have people helping to verify and classify entries. I'm hoping to set something up where people can just do a few at a time whenever it suits them. I'm a bit swamped with non-digimob things right now, but when that's up and running, it'd be pretty neato to get people involved that way.

Soo i got alot of the torrents on my server.
Lacking alot thou but well 600Gb and umetered bandwhidth and unlimited Mysql database.
If you want a databse based FULL library we can use my hosting.
as for the files already there (no worries its a free singup lo just to keep googlebots outl)
link to library
i would be happy if someone that actually is handling this stuff would co operate and we could build a full library.
i am also building a occult info page lacking tons of info for now just a basic idea of a page for now.
www.occultbook.info
and a community website www.occultbook.org
All theese meduims could be used to bring this information to the public and for thoose of intrest
also a simple forum basically empty for now www.occultbook.net
to lazy to make actual links lol

Dude, this is awesome to hear! We were talking about hosting recently and didn't seem to have any really good ideas... Maybe this could work out well.

lol i hope so we could also setup a Ad free forum for this if anyone likes the idea lol

Glad to hear talk of evolutions.

ankh_f_n_khonsu wrote:Glad to hear talk of evolutions.

Indeed.

i dont quite know whos reply i am waiting on on this matter either lol

jlx wrote:i dont quite know whos reply i am waiting on on this matter either lol

Have you sent an email the group's account or PMed admin?

nope lol i,ll gte to that then
(god i love that bounce smiley lol)

Soo anyone got a smexy internet connection
and some basic knowledge and willing to start working on this?
Since my 1Mb down and 512Kb/s up.
Well to put it short we all be dead before its done uploading.lol

Hosting w/o bandwidth is like oreos w/o milk.

Well i has bandwhidt on the hosting but not on my pc lol
actually plenty of it on my hosting

Ugh how did this end up here please move it to the assisting topic

so i,m quite sure there is a way to automagically create a SQL index of all files.
and i now got unlimited bandwihdt and space to use.
So who ahs a smexy upload speed?
and basic knowledge of sql?
i,m gonna look for something to do the sql database whit.
what i am thinking is this search by name title author etc.
once its found direct download.
Sound good?

jlx wrote:Ugh how did this end up here please move it to the assisting topic

so i,m quite sure there is a way to automagically create a SQL index of all files.

I haven't updated here in a while, but this has essentially been done! If all files are on one server, it may be possible to further automagicalize the whole thing, but a mysql database exists with an index of filenames, extensions, and their corresponding torrents/sub-directories. It was generated by a python script. A simple search page has also been set up, but it's hush-hush at the moment because it needs a home.

jlx wrote:
and i now got unlimited bandwihdt and space to use.
So who ahs a smexy upload speed?
and basic knowledge of sql?
i,m gonna look for something to do the sql database whit.
what i am thinking is this search by name title author etc.
once its found direct download.
Sound good?

a home whit love and care lol.
But i lack bandwhidth to upload and sh*t.
i got 1mb down 512kb up and unlimited on my host lol
So lets do this sh*t aye?