Assisting with the (digimob) Project -

Page 2 of 3 Previous  1, 2, 3  Next

View previous topic View next topic Go down

Re: Assisting with the (digimob) Project -

Post  Schinder on Fri Jul 31, 2009 2:27 am

Regarding the size, I agree that Google Documents is not there yet, but it is true for other programs. I use to do this type of file indexing on my hard disks and until Excell 2003 the number of row in a spreadsheet was limited to 65536. Spreadsheets are not databases, but you can go a long way with them today.
In fact for the file format, I would suggest to stick to the basic: csv (comma separated value). You can do a lot of work on it using simple tools like grep and only convert to more powerful format at the last minute. That will also speed up the time you need to generate your file.

Also, until you are ready to crowd source the sanitization of your data, the database is not really necessary.

Schinder

Location : Beijing, China
Number of posts : 4
Registration date : 2009-07-28

View user profile

Back to top Go down

Re: Assisting with the (digimob) Project -

Post  neutralrobotboy on Sat Aug 01, 2009 7:54 am

In fact for the file format, I would suggest to stick to the basic: csv (comma separated value). You can do a lot of work on it using simple tools like grep and only convert to more powerful format at the last minute. That will also speed up the time you need to generate your file.

Well... some file names have commas in them for one thing. But for another, currently the text data is parsed into a list of objects (classes) in python. I can use those objects to render their data into any format I want. I can see no good reason to render first to one file format and then another. This would actually lengthen the time needed to generate the spreadsheet. String searching, etc, can all be done very easily from within python.

Also, until you are ready to crowd source the sanitization of your data, the database is not really necessary.

Maybe, maybe not. There would be some advantages. For now, I'm thinking that Google Docs is unlikely to be satisfactory for searching through the output. In the meanwhile, though, it's true that there are at least some things I can fix up with the script, which will be useful regardless of the final output format. I'll get on those things in the next few days, hopefully. I think they'll be pretty straightforward. The real "bugs" that remain, though, are in one way or another spreadsheet-related.

neutralrobotboy

Age : 36
Number of posts : 255
Registration date : 2008-12-23

View user profile http://www.namelessnumberhead.com

Back to top Go down

Re: Assisting with the (digimob) Project -

Post  Khephra on Sat Aug 01, 2009 7:06 pm

Google Docs may well be insufficient to our needs... As for hosting a database, could we use free hosting? These forums are hosted through Forumotion, and I have to figure we might could find free hosting for that, too. But, given my ignorance of these matters, perhaps I'm completely mistaken. Wink

_________________
"Sacred Activism is the fusion of the mystic's passion for God with the activist's passion for justice, creating a third fire, which is the burning sacred heart that longs to help, preserve, and nurture every living thing." - Andrew Harvey
avatar
Khephra

Age : 52
Number of posts : 897
Registration date : 2008-08-10

View user profile

Back to top Go down

Re: Assisting with the (digimob) Project -

Post  neutralrobotboy on Sat Aug 01, 2009 8:31 pm

Google Docs may well be insufficient to our needs... As for hosting a database, could we use free hosting?

I'll do a little bit of research, but at first glance, the answer seems to be yes. I'm also fairly ignorant on this topic, so I'll see what I can figure out. I'm looking through these sites first:

http://www.free-webhosts.com/free-mysql-database.php

And I'll also look to see if I can find other sites that might not require the extra coding.

neutralrobotboy

Age : 36
Number of posts : 255
Registration date : 2008-12-23

View user profile http://www.namelessnumberhead.com

Back to top Go down

Re: Assisting with the (digimob) Project -

Post  neutralrobotboy on Sat Aug 08, 2009 2:46 am

Just a bit of an update:

I've found a MySQL hosting site that's free and looks like it could be fine. The only drawback is that I may have to code an interface in PHP, I'm not sure. But in any case, there are a few other testing steps to take. I checked out one other site and it had terms and conditions that made me think twice. I might look at a couple more in the next few days, but I do have some deadlines coming up...

I've dedicated very little time to this in the last week, and that may continue for a little while. I have a feeling that whenever I manage to sit with this for a few hours, I'll be able to basically figure out what to do and how, so hopefully there should be more to report in another week or two. I feel a bit lame about the snail's pace at the moment, this stuff shouldn't have to take quite this long, but it's not quite priority #1 right now. Nevertheless, work continues, however slowly.

neutralrobotboy

Age : 36
Number of posts : 255
Registration date : 2008-12-23

View user profile http://www.namelessnumberhead.com

Back to top Go down

Re: Assisting with the (digimob) Project -

Post  Khephra on Sat Aug 08, 2009 8:43 am

There's no race. Smile

In the early days, trying to get the quickest turn-around possible, we released digests every few weeks... It got maddening. Now, with quarterly releases, we have a better opportunity to organize and compile digests, and hopefully that translates into superior formatting.

Take your time. Maybe some more assistance will manifest. Smile

_________________
"Sacred Activism is the fusion of the mystic's passion for God with the activist's passion for justice, creating a third fire, which is the burning sacred heart that longs to help, preserve, and nurture every living thing." - Andrew Harvey
avatar
Khephra

Age : 52
Number of posts : 897
Registration date : 2008-08-10

View user profile

Back to top Go down

Re: Assisting with the (digimob) Project -

Post  neutralrobotboy on Wed Aug 26, 2009 4:16 am

Just did a little work and thought I'd post a bit of an update. I've just checked that things are working with python and SQL at the site I'm doing testing on, and all seems well for now. What this means is that next time I sit down with it, I'll be trying to populate a test database with entered values like what I showed before on the google docs spreadsheet -- but (if we're lucky) without omissions. From there, setting up a simple web interface shouldn't be too much of a jump. So, that's my current plan, and where things are at.

neutralrobotboy

Age : 36
Number of posts : 255
Registration date : 2008-12-23

View user profile http://www.namelessnumberhead.com

Back to top Go down

Re: Assisting with the (digimob) Project -

Post  Khephra on Thu Aug 27, 2009 8:42 am

neutralrobotboy wrote:Just did a little work and thought I'd post a bit of an update. I've just checked that things are working with python and SQL at the site I'm doing testing on, and all seems well for now. What this means is that next time I sit down with it, I'll be trying to populate a test database with entered values like what I showed before on the google docs spreadsheet -- but (if we're lucky) without omissions. From there, setting up a simple web interface shouldn't be too much of a jump. So, that's my current plan, and where things are at.

Excellent! Very Happy

_________________
"Sacred Activism is the fusion of the mystic's passion for God with the activist's passion for justice, creating a third fire, which is the burning sacred heart that longs to help, preserve, and nurture every living thing." - Andrew Harvey
avatar
Khephra

Age : 52
Number of posts : 897
Registration date : 2008-08-10

View user profile

Back to top Go down

Re: Assisting with the (digimob) Project -

Post  neutralrobotboy on Mon Aug 31, 2009 3:36 am

Ok, I've got my script outputting to a database and so far it looks good! What this means is that we have a kajillion entries sitting there in a MySQL database right now. Next step is the web interface for searching, I guess. I'm no big PHP wiz, but I think getting the basic functionality up and running shouldn't be too hard. Then there will be a test site up somewhere where searches are possible. After that, some scheme for the remaining data entry tasks can be worked out. As before, I can't give any guarantees about timeframe for doing this, but I'll try to get on it when I can.

neutralrobotboy

Age : 36
Number of posts : 255
Registration date : 2008-12-23

View user profile http://www.namelessnumberhead.com

Back to top Go down

SorrY.

Post  SoyCello on Thu Sep 03, 2009 8:56 am

Been very consumed recently,
My apology for lack of input.
the last two weeks in september I should have much more free time.
trying to imbibe such a large dose of info on a foreign substance (non urban//successful long term out door living) has been eating my time.
It has been a while since I have checked the releases info..... but I tried to acquire as much as possible for a while so if me having a good amount of the docs on my machine and me having some spare time can help let me know.
((i may be slightly tech-tarded >.<))
avatar
SoyCello

Age : 34
Location : Richmond, VA, US
Number of posts : 32
Registration date : 2008-09-27

View user profile http://www.facebook.com/SoyCello

Back to top Go down

Re: Assisting with the (digimob) Project -

Post  neutralrobotboy on Thu Sep 03, 2009 7:21 pm

o if me having a good amount of the docs on my machine and me having some spare time can help let me know.
((i may be slightly tech-tarded >.<))

Actually, this could be VERY helpful! Once things are set up properly, it would be excellent to have people helping to verify and classify entries. I'm hoping to set something up where people can just do a few at a time whenever it suits them. I'm a bit swamped with non-digimob things right now, but when that's up and running, it'd be pretty neato to get people involved that way.

neutralrobotboy

Age : 36
Number of posts : 255
Registration date : 2008-12-23

View user profile http://www.namelessnumberhead.com

Back to top Go down

mmkays

Post  jlx on Fri Nov 13, 2009 3:02 pm

Soo i got alot of the torrents on my server.
Lacking alot thou but well 600Gb and umetered bandwhidth and unlimited Mysql database.
If you want a databse based FULL library we can use my hosting.
as for the files already there (no worries its a free singup lo just to keep googlebots outl)
link to library
i would be happy if someone that actually is handling this stuff would co operate and we could build a full library.
i am also building a occult info page lacking tons of info for now just a basic idea of a page for now.
www.occultbook.info
and a community website www.occultbook.org
All theese meduims could be used to bring this information to the public and for thoose of intrest
also a simple forum basically empty for now www.occultbook.net
to lazy to make actual links lol


Last edited by jlx on Fri Nov 13, 2009 10:07 pm; edited 1 time in total (Reason for editing : modofied link to look more proper)

jlx

Number of posts : 12
Registration date : 2009-11-13

View user profile http://www.occultbook.org

Back to top Go down

Re: Assisting with the (digimob) Project -

Post  neutralrobotboy on Fri Nov 13, 2009 6:15 pm

Dude, this is awesome to hear! We were talking about hosting recently and didn't seem to have any really good ideas... Maybe this could work out well.

neutralrobotboy

Age : 36
Number of posts : 255
Registration date : 2008-12-23

View user profile http://www.namelessnumberhead.com

Back to top Go down

i hope so

Post  jlx on Fri Nov 13, 2009 9:42 pm

lol i hope so Smile we could also setup a Ad free forum for this if anyone likes the idea lol

cat bounce

jlx

Number of posts : 12
Registration date : 2009-11-13

View user profile http://www.occultbook.org

Back to top Go down

Re: Assisting with the (digimob) Project -

Post  ankh_f_n_khonsu on Sat Nov 14, 2009 2:09 pm

Glad to hear talk of evolutions. Smile
avatar
ankh_f_n_khonsu

Number of posts : 545
Registration date : 2008-09-15

View user profile

Back to top Go down

Re: Assisting with the (digimob) Project -

Post  Khephra on Mon Nov 16, 2009 10:02 am

ankh_f_n_khonsu wrote:Glad to hear talk of evolutions.

Indeed. Smile

_________________
"Sacred Activism is the fusion of the mystic's passion for God with the activist's passion for justice, creating a third fire, which is the burning sacred heart that longs to help, preserve, and nurture every living thing." - Andrew Harvey
avatar
Khephra

Age : 52
Number of posts : 897
Registration date : 2008-08-10

View user profile

Back to top Go down

funny part

Post  jlx on Mon Nov 16, 2009 10:07 am

i dont quite know whos reply i am waiting on on this matter either lol
bounce

jlx

Number of posts : 12
Registration date : 2009-11-13

View user profile http://www.occultbook.org

Back to top Go down

Re: Assisting with the (digimob) Project -

Post  ankh_f_n_khonsu on Mon Nov 16, 2009 12:38 pm

jlx wrote:i dont quite know whos reply i am waiting on on this matter either lol
bounce

Have you sent an email the group's account or PMed admin?
avatar
ankh_f_n_khonsu

Number of posts : 545
Registration date : 2008-09-15

View user profile

Back to top Go down

hmm

Post  jlx on Mon Nov 16, 2009 1:00 pm

nope lol i,ll gte to that then Smile bounce
(god i love that bounce smiley lol)

jlx

Number of posts : 12
Registration date : 2009-11-13

View user profile http://www.occultbook.org

Back to top Go down

Re: Assisting with the (digimob) Project -

Post  jlx on Wed Nov 18, 2009 10:49 am

Soo anyone got a smexy internet connection
and some basic knowledge and willing to start working on this?
Since my 1Mb down and 512Kb/s up.
Well to put it short we all be dead before its done uploading.lol

jlx

Number of posts : 12
Registration date : 2009-11-13

View user profile http://www.occultbook.org

Back to top Go down

Re: Assisting with the (digimob) Project -

Post  Khephra on Wed Nov 18, 2009 1:01 pm

Hosting w/o bandwidth is like oreos w/o milk. Wink

_________________
"Sacred Activism is the fusion of the mystic's passion for God with the activist's passion for justice, creating a third fire, which is the burning sacred heart that longs to help, preserve, and nurture every living thing." - Andrew Harvey
avatar
Khephra

Age : 52
Number of posts : 897
Registration date : 2008-08-10

View user profile

Back to top Go down

Re: Assisting with the (digimob) Project -

Post  jlx on Wed Nov 18, 2009 3:53 pm

Well i has bandwhidt on the hosting but not on my pc lol
actually plenty of it on my hosting Smile

jlx

Number of posts : 12
Registration date : 2009-11-13

View user profile http://www.occultbook.org

Back to top Go down

ok some updates

Post  jlx on Tue Dec 01, 2009 5:22 pm

Ugh how did this end up here Shocked please move it to the assisting topic

so i,m quite sure there is a way to automagically create a SQL index of all files.Suspect
and i now got unlimited bandwihdt and space to use.
So who ahs a smexy upload speed?
and basic knowledge of sql?
i,m gonna look for something to do the sql database whit.
what i am thinking is this search by name title author etc.study
once its found direct download.
Sound good? afro


Last edited by jlx on Tue Dec 01, 2009 5:44 pm; edited 1 time in total (Reason for editing : went to wrong place O_o)

jlx

Number of posts : 12
Registration date : 2009-11-13

View user profile http://www.occultbook.org

Back to top Go down

Re: Assisting with the (digimob) Project -

Post  neutralrobotboy on Tue Dec 01, 2009 9:10 pm

jlx wrote:Ugh how did this end up here Shocked please move it to the assisting topic

so i,m quite sure there is a way to automagically create a SQL index of all files.Suspect

I haven't updated here in a while, but this has essentially been done! If all files are on one server, it may be possible to further automagicalize the whole thing, but a mysql database exists with an index of filenames, extensions, and their corresponding torrents/sub-directories. It was generated by a python script. A simple search page has also been set up, but it's hush-hush at the moment because it needs a home.

jlx wrote:
and i now got unlimited bandwihdt and space to use.
So who ahs a smexy upload speed?
and basic knowledge of sql?
i,m gonna look for something to do the sql database whit.
what i am thinking is this search by name title author etc.study
once its found direct download.
Sound good? afro

neutralrobotboy

Age : 36
Number of posts : 255
Registration date : 2008-12-23

View user profile http://www.namelessnumberhead.com

Back to top Go down

i haz home

Post  jlx on Tue Dec 01, 2009 9:17 pm

a home whit love and care lol.
But i lack bandwhidth to upload and sh*t.
i got 1mb down 512kb up and unlimited on my host lol
So lets do this sh*t aye?


Last edited by jlx on Wed Dec 02, 2009 7:14 am; edited 1 time in total (Reason for editing : self censorship lol)

jlx

Number of posts : 12
Registration date : 2009-11-13

View user profile http://www.occultbook.org

Back to top Go down

Re: Assisting with the (digimob) Project -

Post  Sponsored content


Sponsored content


Back to top Go down

Page 2 of 3 Previous  1, 2, 3  Next

View previous topic View next topic Back to top

- Similar topics

 
Permissions in this forum:
You cannot reply to topics in this forum