Assisting with the (digimob) Project -

We get lots of emails asking how people could assist with Occult.Digital.Mobilization, but one particular role has been especially difficult to procure: archivists. As Occult.Digital.Mobilization's archive continues to expand, the urgency for a searchable archival list increases. Without a searchable index, our ability to cross-check for duplicates is reduced and interested students have no cumulative referencing. After several failed attempts to delegate this momentous responsibility, we thought we might pose the problem to the community and perhaps organize an incremental strategy.

Several months ago, Khephra began organizing guidelines for this responsibility. Here's a snippet of what he sent out:

Ideally we'd like a database of titles arranged by author's name with references to the digest in which it's included. If it were a database, it'd make it possible to prioritize based on author's name, digest release #, type of medium (e.g. pdf, avi, mp3) and genre.

Obviously, in the case of genre there needs to be some discussion on a basic map. Perhaps it could use the same "classification system" as we have on the forums. Perhaps the sub-forums offer a ready-made skeleton, or perhaps more distinctions need to be drawn. Through collaboration, we'll end-up with a superior artifact.

How you go about accomplishing the task is up to you, but with transparency and collaboration efforts can be maximized.

After discussing it a little further, we think Google Documents or Scribd would be ideal. Among other benefits, a public spreadsheet would allow volunteers to incrementally add entries as new digests are released.

So - anyone interested?

I'm glad someone got all those privileges sorted... the thread is unlikely to gain much traction if people can't reply.

ankh_f_n_khonsu wrote:I'm glad someone got all those privileges sorted... the thread is unlikely to gain much traction if people can't reply.

Even having sorted the privileges, it's still a bit quiet. ...

Ok, I am now in posession of one brand new shiny hint!

I'd be happy to do this if I can. However, I don't actually have all the digimob releases on my machine. I don't think that's important, though, to get the ball rolling.

Now, I haven't looked into scribd, but google documents looks like it has a good system for controlling who has write-access to the spreadsheet. This might be good, as it would probably be best to only allow write-access to forum members. It also has "forms" which can be sent out via email so that pieces of the spreadsheet can be filled out pretty easily as needed. Ideally, it would also be good to have some kind of versioning system if possible (I'm not 100% sure if google does this with google docs, but it's not the end of the world if not).

So! If it tickles digimob's fancy, I can start a spreadsheet and enter some of the releases I currently have into a google docs spreadsheet, and we can just see what issues arrise.

In order to archive all of digimob's releases, it would be excellent to have at least some kind of list of all releases and their contents. This can be easily achieved by anyone who has all of digimob's releases in one folder somewhere: In any unix-like environment (Mac OSX or any flavor of gnu/linux), open up a shell prompt, cd to the digimob directory, and type "ls -l -R > digimob_archives.txt" for example. The archivist could then enter these things into the spreadsheet with relative ease.

Of course, many hands make light work, and it'd be excellent if a few people could contribute at least a little, but I'm willing to put my hand up and see what I can do.

Woo! Woo!

Ahh, and using the current forum topics as a basis for putting down the subject sounds like a good idea to me. Actually, I think it would be ideal to have subject codes which the spreadsheet automagically translates into text in another column. I think this can be done, but I don't know how yet. This would ultimately save quite a bit of typing and also cut down on typos. If I were to type out "Western Magickal Tradition > Alchemy" for every alchemical text, a simple sort would exclude any file where I had made any kind of typo, including an accidental trailing space, for example, which would be very hard to track down.

Here's a little preview of what google docs might look like: http://spreadsheets.google.com/ccc?key=0AsJ3oKUxrWrJck9GaFNDa0s4OUJJRkhESnBmeEMyM0E&hl=en

Though, actually, I would prefer not to have the final version done from my main google account if possible.

Somewhat off-putting to have heard no feedback, feels like I'm talking to myself here...

In any case, I've done some thinking and realized that doing all of this by hand will be extremely labor-intensive and there may be ways to save time. It would be much better to have the bulk of the work done by script. Some things, like categorization, may not be possible by script, however, or at least not without working out a system for future releases. If I can find a way to send entries to google documents via script, that would be ideal. If that isn't possible, uhh... Thinking caps go on.

Not talking to yourself... Some of us spin quite a few plates.

Actually, I think it would be ideal to have subject codes which the spreadsheet automagically translates into text in another column. I think this can be done, but I don't know how yet.

That would be super-convenient, but I'm not sure how it would work... Databases aren't my expertise though, so my ignorance shouldn't be an indication of impossibility.

Though, actually, I would prefer not to have the final version done from my main google account if possible.

Agreed. Another gmail account could be made specifically for the task, or we could create a document using the group's gmail and share it with those who are helping on this project. We've heard whispers of others helping, but we'll see how that pans out...

In any case, I've done some thinking and realized that doing all of this by hand will be extremely labor-intensive and there may be ways to save time. It would be much better to have the bulk of the work done by script. Some things, like categorization, may not be possible by script, however, or at least not without working out a system for future releases. If I can find a way to send entries to google documents via script, that would be ideal. If that isn't possible, uhh... Thinking caps go on.

Yes, 'labour-intensive' probably describes it well... Generating a list of files included in the archive wouldn't be too tough, but there are many directories zipped, and oodles of dangling jpgs (tarot + etc). I'll follow your advice and generate a list and PM it to you.

Not talking to yourself... Some of us spin quite a few plates.

Groovy, man. I don't mean to be impatient. So long as there's some sign I'm not being left hanging, I'm good.

Another gmail account could be made specifically for the task, or we could create a document using the group's gmail and share it with those who are helping on this project. We've heard whispers of others helping, but we'll see how that pans out...

Cool. Either of those solutions sounds good to me.

Yes, 'labour-intensive' probably describes it well... Generating a list of files included in the archive wouldn't be too tough, but there are many directories zipped, and oodles of dangling jpgs (tarot + etc). I'll follow your advice and generate a list and PM it to you.

File received, thanks! I'm going to look into parsing this with a script. I think I can hack up a python script to do that lickety-split, the only trouble is having the script then output into a database. BUT! The good news is that if the script-based archiving can get up and running, it should also be able to implement the subject-code method very easily (databases are also not really my forte), and future releases should become relatively easy to archive.

Yes, 'labour-intensive' probably describes it well... Generating a list of files included in the archive wouldn't be too tough, but there are many directories zipped, and oodles of dangling jpgs (tarot + etc). I'll follow your advice and generate a list and PM it to you.

Hmm... I see what you mean. I've also noticed just now that there are occasionally files like "(digimob) BOOK LIST.txt" in #18, which already has a partial list... This is a tricky matter indeed. On the plus side, with a script parsing all of this stuff, the jpegs should wind up being pretty easy to organize. An entry like "FILES 'dramatic universe vol 3-1.jpg' THROUGH 'dramatic universe vol 3-60.jpg'", or whatever other convention seems clearest, could be automated simply in a script in most cases, I think. The .zip files are another matter, maybe, but I would hope that files aren't being hidden in .zips unless the directory name and file name give a pretty good indication of what's inside the file itself. So, I would hope that a listing of the containing directory and filename would suffice.

Sometime in the next few days, I'll try to work out more of the details of this process. It's inevitable that some fields will have to be put in by hand and the output of the script will have to be checked by people, but I think a lot of time should be saved anyhow. Thanks again!

I'm happy to see this progressing, and think it's going to make a big difference!

Ok, I've done a bit of research and just barely started feeling out a script. Basically, this is my current plan for the flow of the whole process:

1. Parse files and directory structure from ls output (thanks, Khepra!). Information that can definitely be extracted from this: File names, file sizes, digimob release numbers, media type, exact directory the file was located in. Because the titles of the digimob files aren't all consistent, it may not be possible to extract author names at all -- I'll look into this further, though. There's a chance that many (probably not all) authors can be put in automagically based on filenames.

2. Generate an .ods (open document spreadsheet) file from 1. I've decided to use .ods as the intermediary file type because it's got a straghtforward open spec that's been known for a while, and there are python modules already in existence to make writing an .ods file from a script a fainly simple process.

3. Import .ods file into Google docs and paste into master spreadsheet. I tested this process out and it's very simple. This should go over without a hitch.

4. Input the rest of the data by hand. This is where other people help out or else it's gonna take a LOOOONG time.

Once I get a working mock-up of steps 1-3, we need to pin down the subject structure and codes (if any). I think actually that using the digimob forum structure might not be so good after all. For example: I noticed a number of .pdf's dealing with learning Hebrew. Where do those go? The forums don't seem to have an "Instructional" section. There may be other cases like that.

After getting that straightened out, I think it would be a good idea to figure out a way to distribute the work that needs to be done, so people can do small chunks. If a form could be made that emails out, say, 10 incomplete entries at a time, people could fill it out bit by bit. This would mean that at any time, someone could say, "Yeah, I've got a spare 15 minutes today, send me some entries."

Here's a start:

http://spreadsheets.google.com/ccc?key=t26Byflmotq77FB398TNkUA&hl=en

Took much longer to get that up and running than I expected, but there it is. I"ll write up remaining issues sometime when my brain isn't fried.

neutralrobotboy wrote:Here's a start:

http://spreadsheets.google.com/ccc?key=t26Byflmotq77FB398TNkUA&hl=en

Took much longer to get that up and running than I expected, but there it is. I"ll write up remaining issues sometime when my brain isn't fried.

Tried, but didn't have authorization. Is it publicly viewable?

neutralrobotboy wrote:
1. Parse files and directory structure from ls output (thanks, Khepra!).

Excellent! I was going to send a PM to see if he'd gotten it on to you yet.

Any bit of the process that could be automated and make the load lighter seems ideal.

Generate an .ods (open document spreadsheet) file from 1.

Import .ods file into Google docs and paste into master spreadsheet.

Input the rest of the data by hand. This is where other people help out or else it's gonna take a LOOOONG time.

Yes, I suspect there'll be a good bit to enter manually, but we might can incrementally make things easier. The notation used in digests has progressively gotten better (hopefully!). Ideally we could enact a format that was suitable for labour-saving updates.

Once I get a working mock-up of steps 1-3, we need to pin down the subject structure and codes (if any). I think actually that using the digimob forum structure might not be so good after all. For example: I noticed a number of .pdf's dealing with learning Hebrew. Where do those go? The forums don't seem to have an "Instructional" section. There may be other cases like that.

No matter how many hairs we split, there will still be plenty of ambiguity. Part of the idea behind matching the genres to the structure of the fora was to help confine things a bit. Otherwise we could get really bogged down in sub-genres. However, I'm certainly open to other ways to categorize, so if you - or anyone else - has any ideas, pass 'em on!

After getting that straightened out, I think it would be a good idea to figure out a way to distribute the work that needs to be done, so people can do small chunks. If a form could be made that emails out, say, 10 incomplete entries at a time, people could fill it out bit by bit. This would mean that at any time, someone could say, "Yeah, I've got a spare 15 minutes today, send me some entries."

I'd <3 to see this much community involvement!

Tried, but didn't have authorization. Is it publicly viewable?

Ahh, thanks. Forgot to do that. Should be viewable now.