Assisting with the (digimob) Project -

We get lots of emails asking how people could assist with Occult.Digital.Mobilization, but one particular role has been especially difficult to procure: archivists. As Occult.Digital.Mobilization's archive continues to expand, the urgency for a searchable archival list increases. Without a searchable index, our ability to cross-check for duplicates is reduced and interested students have no cumulative referencing. After several failed attempts to delegate this momentous responsibility, we thought we might pose the problem to the community and perhaps organize an incremental strategy.

Several months ago, Khephra began organizing guidelines for this responsibility. Here's a snippet of what he sent out:

Ideally we'd like a database of titles arranged by author's name with references to the digest in which it's included. If it were a database, it'd make it possible to prioritize based on author's name, digest release #, type of medium (e.g. pdf, avi, mp3) and genre.

Obviously, in the case of genre there needs to be some discussion on a basic map. Perhaps it could use the same "classification system" as we have on the forums. Perhaps the sub-forums offer a ready-made skeleton, or perhaps more distinctions need to be drawn. Through collaboration, we'll end-up with a superior artifact.

How you go about accomplishing the task is up to you, but with transparency and collaboration efforts can be maximized.

After discussing it a little further, we think Google Documents or Scribd would be ideal. Among other benefits, a public spreadsheet would allow volunteers to incrementally add entries as new digests are released.

So - anyone interested?

I'm glad someone got all those privileges sorted... the thread is unlikely to gain much traction if people can't reply.

ankh_f_n_khonsu wrote:I'm glad someone got all those privileges sorted... the thread is unlikely to gain much traction if people can't reply.

Even having sorted the privileges, it's still a bit quiet. ...

Ok, I am now in posession of one brand new shiny hint!

I'd be happy to do this if I can. However, I don't actually have all the digimob releases on my machine. I don't think that's important, though, to get the ball rolling.

Now, I haven't looked into scribd, but google documents looks like it has a good system for controlling who has write-access to the spreadsheet. This might be good, as it would probably be best to only allow write-access to forum members. It also has "forms" which can be sent out via email so that pieces of the spreadsheet can be filled out pretty easily as needed. Ideally, it would also be good to have some kind of versioning system if possible (I'm not 100% sure if google does this with google docs, but it's not the end of the world if not).

So! If it tickles digimob's fancy, I can start a spreadsheet and enter some of the releases I currently have into a google docs spreadsheet, and we can just see what issues arrise.

In order to archive all of digimob's releases, it would be excellent to have at least some kind of list of all releases and their contents. This can be easily achieved by anyone who has all of digimob's releases in one folder somewhere: In any unix-like environment (Mac OSX or any flavor of gnu/linux), open up a shell prompt, cd to the digimob directory, and type "ls -l -R > digimob_archives.txt" for example. The archivist could then enter these things into the spreadsheet with relative ease.

Of course, many hands make light work, and it'd be excellent if a few people could contribute at least a little, but I'm willing to put my hand up and see what I can do.

Woo! Woo!

Ahh, and using the current forum topics as a basis for putting down the subject sounds like a good idea to me. Actually, I think it would be ideal to have subject codes which the spreadsheet automagically translates into text in another column. I think this can be done, but I don't know how yet. This would ultimately save quite a bit of typing and also cut down on typos. If I were to type out "Western Magickal Tradition > Alchemy" for every alchemical text, a simple sort would exclude any file where I had made any kind of typo, including an accidental trailing space, for example, which would be very hard to track down.

Here's a little preview of what google docs might look like: http://spreadsheets.google.com/ccc?key=0AsJ3oKUxrWrJck9GaFNDa0s4OUJJRkhESnBmeEMyM0E&hl=en

Though, actually, I would prefer not to have the final version done from my main google account if possible.

Somewhat off-putting to have heard no feedback, feels like I'm talking to myself here...

In any case, I've done some thinking and realized that doing all of this by hand will be extremely labor-intensive and there may be ways to save time. It would be much better to have the bulk of the work done by script. Some things, like categorization, may not be possible by script, however, or at least not without working out a system for future releases. If I can find a way to send entries to google documents via script, that would be ideal. If that isn't possible, uhh... Thinking caps go on.

Not talking to yourself... Some of us spin quite a few plates.

Actually, I think it would be ideal to have subject codes which the spreadsheet automagically translates into text in another column. I think this can be done, but I don't know how yet.

That would be super-convenient, but I'm not sure how it would work... Databases aren't my expertise though, so my ignorance shouldn't be an indication of impossibility.

Though, actually, I would prefer not to have the final version done from my main google account if possible.

Agreed. Another gmail account could be made specifically for the task, or we could create a document using the group's gmail and share it with those who are helping on this project. We've heard whispers of others helping, but we'll see how that pans out...

In any case, I've done some thinking and realized that doing all of this by hand will be extremely labor-intensive and there may be ways to save time. It would be much better to have the bulk of the work done by script. Some things, like categorization, may not be possible by script, however, or at least not without working out a system for future releases. If I can find a way to send entries to google documents via script, that would be ideal. If that isn't possible, uhh... Thinking caps go on.

Yes, 'labour-intensive' probably describes it well... Generating a list of files included in the archive wouldn't be too tough, but there are many directories zipped, and oodles of dangling jpgs (tarot + etc). I'll follow your advice and generate a list and PM it to you.

Not talking to yourself... Some of us spin quite a few plates.

Groovy, man. I don't mean to be impatient. So long as there's some sign I'm not being left hanging, I'm good.

Another gmail account could be made specifically for the task, or we could create a document using the group's gmail and share it with those who are helping on this project. We've heard whispers of others helping, but we'll see how that pans out...

Cool. Either of those solutions sounds good to me.

Yes, 'labour-intensive' probably describes it well... Generating a list of files included in the archive wouldn't be too tough, but there are many directories zipped, and oodles of dangling jpgs (tarot + etc). I'll follow your advice and generate a list and PM it to you.

File received, thanks! I'm going to look into parsing this with a script. I think I can hack up a python script to do that lickety-split, the only trouble is having the script then output into a database. BUT! The good news is that if the script-based archiving can get up and running, it should also be able to implement the subject-code method very easily (databases are also not really my forte), and future releases should become relatively easy to archive.

Yes, 'labour-intensive' probably describes it well... Generating a list of files included in the archive wouldn't be too tough, but there are many directories zipped, and oodles of dangling jpgs (tarot + etc). I'll follow your advice and generate a list and PM it to you.

Hmm... I see what you mean. I've also noticed just now that there are occasionally files like "(digimob) BOOK LIST.txt" in #18, which already has a partial list... This is a tricky matter indeed. On the plus side, with a script parsing all of this stuff, the jpegs should wind up being pretty easy to organize. An entry like "FILES 'dramatic universe vol 3-1.jpg' THROUGH 'dramatic universe vol 3-60.jpg'", or whatever other convention seems clearest, could be automated simply in a script in most cases, I think. The .zip files are another matter, maybe, but I would hope that files aren't being hidden in .zips unless the directory name and file name give a pretty good indication of what's inside the file itself. So, I would hope that a listing of the containing directory and filename would suffice.

Sometime in the next few days, I'll try to work out more of the details of this process. It's inevitable that some fields will have to be put in by hand and the output of the script will have to be checked by people, but I think a lot of time should be saved anyhow. Thanks again!

I'm happy to see this progressing, and think it's going to make a big difference!

Ok, I've done a bit of research and just barely started feeling out a script. Basically, this is my current plan for the flow of the whole process:

1. Parse files and directory structure from ls output (thanks, Khepra!). Information that can definitely be extracted from this: File names, file sizes, digimob release numbers, media type, exact directory the file was located in. Because the titles of the digimob files aren't all consistent, it may not be possible to extract author names at all -- I'll look into this further, though. There's a chance that many (probably not all) authors can be put in automagically based on filenames.

2. Generate an .ods (open document spreadsheet) file from 1. I've decided to use .ods as the intermediary file type because it's got a straghtforward open spec that's been known for a while, and there are python modules already in existence to make writing an .ods file from a script a fainly simple process.

3. Import .ods file into Google docs and paste into master spreadsheet. I tested this process out and it's very simple. This should go over without a hitch.

4. Input the rest of the data by hand. This is where other people help out or else it's gonna take a LOOOONG time.

Once I get a working mock-up of steps 1-3, we need to pin down the subject structure and codes (if any). I think actually that using the digimob forum structure might not be so good after all. For example: I noticed a number of .pdf's dealing with learning Hebrew. Where do those go? The forums don't seem to have an "Instructional" section. There may be other cases like that.

After getting that straightened out, I think it would be a good idea to figure out a way to distribute the work that needs to be done, so people can do small chunks. If a form could be made that emails out, say, 10 incomplete entries at a time, people could fill it out bit by bit. This would mean that at any time, someone could say, "Yeah, I've got a spare 15 minutes today, send me some entries."

Here's a start:

http://spreadsheets.google.com/ccc?key=t26Byflmotq77FB398TNkUA&hl=en

Took much longer to get that up and running than I expected, but there it is. I"ll write up remaining issues sometime when my brain isn't fried.

neutralrobotboy wrote:Here's a start:

http://spreadsheets.google.com/ccc?key=t26Byflmotq77FB398TNkUA&hl=en

Took much longer to get that up and running than I expected, but there it is. I"ll write up remaining issues sometime when my brain isn't fried.

Tried, but didn't have authorization. Is it publicly viewable?

neutralrobotboy wrote:
1. Parse files and directory structure from ls output (thanks, Khepra!).

Excellent! I was going to send a PM to see if he'd gotten it on to you yet.

Any bit of the process that could be automated and make the load lighter seems ideal.

Generate an .ods (open document spreadsheet) file from 1.

Import .ods file into Google docs and paste into master spreadsheet.

Input the rest of the data by hand. This is where other people help out or else it's gonna take a LOOOONG time.

Yes, I suspect there'll be a good bit to enter manually, but we might can incrementally make things easier. The notation used in digests has progressively gotten better (hopefully!). Ideally we could enact a format that was suitable for labour-saving updates.

Once I get a working mock-up of steps 1-3, we need to pin down the subject structure and codes (if any). I think actually that using the digimob forum structure might not be so good after all. For example: I noticed a number of .pdf's dealing with learning Hebrew. Where do those go? The forums don't seem to have an "Instructional" section. There may be other cases like that.

No matter how many hairs we split, there will still be plenty of ambiguity. Part of the idea behind matching the genres to the structure of the fora was to help confine things a bit. Otherwise we could get really bogged down in sub-genres. However, I'm certainly open to other ways to categorize, so if you - or anyone else - has any ideas, pass 'em on!

After getting that straightened out, I think it would be a good idea to figure out a way to distribute the work that needs to be done, so people can do small chunks. If a form could be made that emails out, say, 10 incomplete entries at a time, people could fill it out bit by bit. This would mean that at any time, someone could say, "Yeah, I've got a spare 15 minutes today, send me some entries."

I'd <3 to see this much community involvement!

Tried, but didn't have authorization. Is it publicly viewable?

Ahh, thanks. Forgot to do that. Should be viewable now.

neutralrobotboy wrote:Ahh, thanks. Forgot to do that. Should be viewable now.

Fantastic progress!

Ideally we could enact a format that was suitable for labour-saving updates.

Yeah, if it doesn't add labor to the releasing team, it would be excellent to have something like this -- future releases could be catalogued almost entirely by script.

Part of the idea behind matching the genres to the structure of the fora was to help confine things a bit. Otherwise we could get really bogged down in sub-genres. However, I'm certainly open to other ways to categorize, so if you - or anyone else - has any ideas, pass 'em on!

Yeah, makes sense. So something on learning Hebrew would be under "Western Magickal Tradition > Qabalah", say?

Anyway, the remaining issues with the script are:

1. Some entries can't currently be put in by the script at all. These are entries with special characters, such as accented vowels. I've figured out how to put these all in their own text file, so they can probably be copied + pasted from that. There aren't that many of them, in any case. The problem seems to somehow be related to unicode/ascii conversion somewhere, but it's a real headache.

2. Unidentified bug: Makes it crash after parsing about 8700 lines of the ls output. Still tracking that one down, it's very strange. The database I linked to earlier is the result of a cheap hack to circumvent it. That's why some of the "Occult Carrot" entries aren't labeled properly.

3. File grouping not implemented. This is the idea I proposed of grouping files that are clearly numbered as part of a series. I'll try to get on this soon-ish, but I'm trying to spend a bit less time at the computer at the moment, and my workload is soon to increase...

After these issues are resolved (can't guarantee a timeframe at the moment), the script will be ready to go "live", basically. Then we can work out how to do the community participation thing, etc.

neutralrobotboy wrote:Anyway, the remaining issues with the script are:

Wow! It's exciting to watch this progress!

Great work! I certainly don't want to dissuade you from this project, but I've got a couple of questions.

1) How does someone look up an item, if they don't have the exact file name? The hypothetical pdf I wish to contribute might have been named different at some point, either with the title of the book first, rather than the author's name, or the author's first name and yours starts with the author's last name, or one of them gets listed under "the" (as is the case in mega torrent #2.3,) etc.

2) Also, currently these aren't in alphabetical order. Will the program correct that, when you're done entering the information?

1) How does someone look up an item, if they don't have the exact file name? The hypothetical pdf I wish to contribute might have been named different at some point, either with the title of the book first, rather than the author's name, or the author's first name and yours starts with the author's last name, or one of them gets listed under "the" (as is the case in mega torrent #2.3,) etc.

Actually, I was going to bring this up too... I think Google Docs has some search functions, if not, the solutions aren't terribly convenient. The thing is that even with things as they are now, you *should* be able to do searches like, [Author's last name] in the "File Name" column, actually, and have all files that include the author's last name anywhere in the filename turn up. I can look into this. It would be excellent if someone else could also look into this, since I'm not at the computer too much at the moment. In any case, searching should be flexible. Also, if you had a pdf to contribute, I assume the releasers would take care of renaming the file if that needed to be done.

2) Also, currently these aren't in alphabetical order. Will the program correct that, when you're done entering the information?

This can be done with simple column sorting. The script *could* arrange things in alphabetical order, but this can also be done simply even in Google Docs. Part of the idea of a database like this (I assume) is that it can be sorted in a variety of ways, leaving the question of how to sort the data up to the user.

All that said, a more convenient solution in the long run would be a custom web app. This would mean hosting and bandwidth costs, though... Not to mention development of the app itself and database administration, which would all become a real pain real quick. Though I could write the web app in theory (probably using Django and MySQL, which I've played with before), I wouldn't trust myself with admin -- It's just not something I have experience with.

At the moment, I assume the "preferred" solution is for the user to actually download (i.e., export) the database and use it locally. Like, I would grab it as an .ods or .xls file and open it in OpenOffice.org and do my searching and sorting that way. But again, that's not terribly convenient and I haven't looked into the best way to do it from within Google Docs.

Just looked into the search+sort issue, the solution seems to be "gadgets". I think it'll be easy to figure out, actually.

Yes, in preparation for digests a text file is manually searched for duplicates... having the archival list searchable is a high priority.

Yes, in preparation for digests a text file is manually searched for duplicates... having the archival list searchable is a high priority.

I've looked into google docs' "gadgets" again for this, and it's still a jumble to me. For a database this large, it may be a big headache. I'm actually not convinced that google docs will wind up being the best solution in terms of searchability, but I'll keep digging and hopefully figure it out. The good thing about the "gadgets" thing is that I can write one in JavaScript if need be. That said, I'm still not totally convinced that this is right considering the number of entries we're dealing with (this affects how long it takes for a gadget to refresh, which happens often). Figuring this out will take a bit of time, unless someone else has already dealt with this in google docs and knows what to do.

Dealing with this stuff, I'm tempted to just host the database myself and code the interface by hand in python using django. It's been a while since I've done that kinda thing, but it still may be worth it over google docs atm. I have a suspicion it'd take less effort and produce better results for flexible searching. The only thing is badwidth. More on this later.

Sorry to jump in, I appreciate the progresses made so far, but I would like to give some comments.

For a database this large, it may be a big headache.

It ain't big yet. But we need to consider scalability.

Dealing with this stuff, I'm tempted to just host the database myself and code the interface by hand in python using django.

I would suggest that you spend some extra time refining the import to spreadsheet. I think some good use can be made out of spreadsheets and it can turn to be a useful tool until we get to the database level.
Some suggestion for improvement: convert extension to lower case.

Sorry to jump in, I appreciate the progresses made so far, but I would like to give some comments.

No need to apologize! I'm glad to get more input.

It ain't big yet. But we need to consider scalability.

It's true that it ain't big yet. However: When playing with google docs' "gadgets", it seems that they refresh quite often, which seems to mean a complete refresh of all 8,000 (ish) rows of data, which means it takes a long time to do anything with it. Since I'm new to google docs, I'm hoping there's something I just don't understand, or some simple work-around. But as it stands, it's pretty annoying to work with. If you want to try messing around with it to see if you can find a convenient and flexible "gadget"-y way to search by varying criteria, though, please feel free to relay any of your findings.

I would suggest that you spend some extra time refining the import to spreadsheet. I think some good use can be made out of spreadsheets and it can turn to be a useful tool until we get to the database level.

One reason I'm bringing this up now is that I have a strong suspicion that telling my script to output to a database directly would resolve the outstanding problems with the script itself. That is, I don't think that the unicode conversion weirdess would persist, since it's an artifact of the module that's writing to spreadsheet format. Actually, the other outstanding problem with the script is also related to format conversion somewhere in that module. I should probably look at other python modules that have the same functionality, see if they fare any better.

Anyway, my thinking with the database is not so much about scaleability as flexibility, ease of use, and ease of maintenance. At this stage, maybe it's not really the way to go, I'm not sure, but it *may* turn out to be easiest in the long-run to be in total control of all archive-related functionality.

Even at this early stage, google docs is only practical if we find some way of providing reasonable search functionality. With things as they are, the only way I can think to do that is by allowing people to download the spreadsheet and search using their favorite compatible spreadsheet program (probably Excel or OpenOffice.org). Which I guess is fine, but it puts the burden of sorting and searching on the user, if this is meant for public consumption.