The Making of the Archives

After well over a year of dreaming, planning, and plain old hard work, the archive of No Depression's 75 print issues finally launched today.

It has been a long daunting undertaking and had I known what I was getting into at the time I'm not sure I would have taken it on. I expect that once the dust settles and time marches on the frustrations and stress of last year will fade and I will only remember the good parts and the bad parts won't seem so bad, but at this moment I am completely wrung out.

There was considerable effort on the part of many people in making the archive site a reality and I want to share some of the details of the process that was undertaken to get 75 issues of printed content in a form you are now able to enjoy, search and sort on this website.

The data from the print publications was in early incarnations of PageMaker software and also existed in Word files. The final data for each issue was in the PageMaker files so ideally we wanted to use those. Considerable testing was done by a number of developers to try and determine a way to auto extract the data from the PageMaker files but it was determined that there was no way to automate the process and do a mass import. A Rails database was built to house and process the data and the Word files were used as it was possible to mass import all the data for each issue into separate fields for each article, with the exception of the record reviews which all ended up in one article body field and had to be cut and pasted into separate articles. The Word files contained close to the final version of each article but any edits that were made for articles to fit on pages and last minute corrections or updates that were made to the articles as they were being sent to the printer are not included in the Word files so in some cases you will find that articles in the archive are slightly different than what appeared in print.

Banner ads were run on the No Depression website for several weeks early in the year and 22 volunteers signed on and then followed through to do the first phase of the work. Each was assigned issues of the magazine to work on and the first step in the process was to open a page in the database for each article that had been imported and ascertain which of 17 different article types they were looking at based on the text in the article body field (feature, screen door, live review, record review, essay, etc.). Next they chose an article type from a drop down menu and then cut and paste all the meta data for each article type into the proper fields (record label, headline, subhead, cd title, live show location, etc.).

After all the content was cut and pasted for each issue an inventory was done comparing the printed magazine to the articles in the database and a list of any articles missing from the database that were in the magazine, or in the database and not in the magazine was created. Next all the articles that were missing from the database for each issue were cut and pasted out of the PageMaker files and added to the database. For some of the early issues there weren't PageMaker files so all the missing articles had to be retyped. This was done for all 75 issues over a period of about a month and I estimate it took somewhere around 500 hours to complete this part of the process.

Meanwhile the development of a highly customized Word Press web site was underway. When things were far enough along the data from the original Rails database was imported into the Word Press database. The development of the site was a long involved process that took close to three months to complete. As these things go, it took twice as long and cost twice as much as expected. (And it's not yet finished at this moment).

Once the web site was functioning another batch of 20 volunteers, 10 who had helped on the first phase of the project and 10 new folks, started in on the next phase of data management. They were each assigned issues and combed through the table of contents for each issue clicking on every link to make sure the articles displayed correctly, looking for misplaced punctuation, capitalization in the headings, consistency of display and incorrect album art displaying on the reviews. A script had been written to mass import the album art mp3 clips player widget on each record review and the widget often pulled in the incorrect album so the volunteers had to check each review for the correct album art and make a note if it was incorrect so it could be fixed along with the other corrections that were needed.

Each volunteer then compiled a report with the url from each article page that had issues, noting what those issues were, and then the edits and corrections were made for each issue. While the edits were being made a few volunteers were formatting the letters to the editor, field reportings, Top 40 retail chart, Film @11, Opinion poll, adding page breaks to all the feature stories and consolidating artist tags.

Every artist name in the "browse by artist" feature had to be looked at and the articles consolidated under one name as there were often multiple tags for each artist with several variations. For example; Willie Nelson & Ray Price, Willie Nelson and Friends, Willie Nelson & Friends, Willie Nelson and Family, Willie Nelson Family Fourth, etc.- the articles attached to each of those names had to be consolidated under one listing for each artist.

Another task was publishing all of the content from the old web site which had also been imported to the Word Press database. Article type fields were added for each different web archive article type and each article had to be reviewed, artist names and tags added where necessary and then each individual article was published to the site.

In the end well over 1000 hours of time were generously donated by the 34 people listed below. There are 7,299 articles in the archive covering close to 3700 artists and each and every article was touched multiple times by just 34 volunteers. Several who put in well over a hundred hours individually over the last few months. They came from all over the world; London, Spain and Australia in addition to many from the United States. I know nothing about most of them other than their names, yet they stepped up and gave generously of their time and continually asked what else they could do to help when they had completed their assigned task. I'm not usually one to ask for help and their generosity and willingness to spend so much of their time and energy helping me get the archives online was humbling and moving. Their words of support and encouragement along the way gave me the boosts I needed to continue on with this overwhelming undertaking.

Without them the archives would not be available online. There is absolutely no way I could have done all this by myself and I will be forever grateful to them for the time they devoted to this project. I hope that as you read articles in the archives you will think of them cutting and pasting every last scrap of information into the database so it can be searched, sorted and enjoyed by music fans all over the world for years to come.