More "Pause" than "End"

Some years back, on a particularly busy night, a new hire rubbed eyes and blearily asked the Nightstalker crew, "Just how long DO you work?"

I answered in the only possible way: "Until the job is done."

That, truly, is what makes the overnight hours in Data Services interesting. I say "overnight" because my hours can be a bit fuzzy- and this is true for everyone in our department. Some folks come in fairly early, some come in around the traditional office start time of 0800, and some come in at 2200 or even later. We have one person who works a traditional second shift, and the remainder of the crew work something of a hybrid 3rd shift, starting anywhere from 1800 to almost 2300. I don't believe there's anything like an official policy regarding this; it's more of a rule of thumb: as long as one works one's 40 hours, everyone is happy. But, back to "until the job is done."

I think I may have previously mentioned that our plant can be considered a "job shop" or "contract manufacturer." In other words, the product that we manufacture- direct mail- does not "belong" to us. You will typically not see our name on it anywhere. We may have some involvement in the design process, and we are often involved in the postal and logistics components of a job, but at the end of the day, we're simply the folks who manufacture something for someone else.

Sometimes...

Sometimes things just need to get done. One of our top clients has a job that runs every month, and it keeps the programmer pretty busy for around a week. However, once the client approves the job, yours truly ("me") typically swings into action. This particular job requires a lot of hard copy:  proofs for press and prepress, and tray tags for press. Printing, collating and marking up of this takes ~12 hours, and is best done when the office is empty. Why? Because one simply needs a lot of space to do the job correctly and efficiently, as well as tying up two printers.

So, the Friday before last, I came in with the expectation that I would be working on this job. I had put a dent in it when I left Saturday morning, and came back Saturday evening. I got home around midnight, and everything was done.

In a nutshell, my weekend was more of a pause than an actual end to the week. The obvious downside to weekends like this is that they are short. Jennifer and I got in a nice walk on the following Sunday- almost three miles. I did a bit of work in the Secret Underground Lair, as did Mr. T, and great progress has been made. My plan for the weekend, though, had been to get to the unused desk and hook up my Raspberry Pi. That hadn't happened yet, but we're on the cusp of it!

One other huge thing that happened the previous week was the Great Book Cull, during which time a few hundred books were donated to our local library. Although both Jennifer and I hate getting rid of actual paper books, there were a few compelling reasons for eliminating these particular volumes from our collection:

1) They were classics (Shakespeare, Tolstoi, etc)- these are readily available online and downloadable as plain text files

2) Cold War fiction- entertaining reading when it was written, but no longer plausible.

3) Out of date software texts- bye, bye, Windows 95 For Dummies, etc... I  kid you not.

So that was the two weekends ago. I visited the opthamologist for the last post surgery follow up for my right eye. (all is good- come back in a year). Don't get me wrong- if you're in the Des Plaines area, Dr. Winkler is the best. I'm just ready to get on with my life!

Pause, continued... this past weekend (Dec 19-20): I got out on time on Saturday morning for a change, but I knew I'd be coming back on Saturday evening. I figured a few hours would take care of a few quick production QC's- boy was I wrong.

The first job went smoothly, and it was done fairly quickly. The second, though, took much longer, as some paperwork was missing. It's not unusual for this to happen with this particular job, though. The third job was unexpected, but pretty easy- especially since the lead programmer was also in the office to explain a few things. The last job, though.

In a nutshell, it was programmed correctly but the output was set up incorrectly, and consequently compiled incorrectly, and the output was just plain wrong. As it was late and hadn't brought my meds to the office, I remade the files and recompiled the job, but did not perform the production QC.  

Finally, data.

I need to say something about my writing process, which will eventually relate to data. I promise.

When I write a blog, I generally have an idea of where I want it to go. Sometimes it goes there, sometimes it doesn't. This blog is a case of: IT DOESN'T CARE WHERE I WANT TO GO.

Point: my idea for this blog actually had to do with a concept for music opinion data analysis.

Way back when, in my college days, I started my music database. It was on looseleaf paper, and it was unwieldy and incredibly difficult to update or edit. At some point, I decided I'd create something of a personal "Top 40". Only looking at my stats was somewhat biased, I quickly surmised.  Undaunted, I expanded my pursuit to include what others were interested in. When I shared my results, however, some (many) of the participants were upset when I told them the results were weighted. That is, their top votes received 10 points, and their bottom votes received one point. Huh?

Participants in the survey were asked to vote for up to 10 of their favorite albums, and 10 of their favorite songs. I asked that they please put them in order of preference. When the results had been tabulated, I shared them. In my circle of friends and acquaintances, no one could believe that "Bread's Greatest Hits"  was more popular than "Barry Manilow Live" (joke).

I did it this way for a couple of (what I thought) were very good reasons. First: everyone probably has a favorite song. Hey, if its YOUR fave, why not weight it? Second, I wanted to be able to track which songs were actually more popular than which songs were most often voted for. Taking the two most "popular" songs of this era (from my polls), the hands-down winners were The Beatles' "Hey Jude" and Led Zepplin's "Stairway To Heaven". As my survey was quite small, let's say I had 10 voters: 4 went with the Beatles, and 4 with Led Zepplin. Let's say the other two went with Lynyrd Skynyrd's "Freebird" as their favorite. If Hey Jude == 1, and Stairway == 1 and Freebird == 1, then we have 4 to 4 to 2. What matters next is what is the next greatest song.  So, two of the "Beatles" guys vote for  "I Saw Her Standing There", and a couple of Zepplin fans vote for "Rock and Roll". After that, each of the other Beatles and Zepplin fans votes for Hey Jude and Stairway, furthering the tie. Then, we factor in the Skynyrd fans, who generally might like the Zep guitars, but can sing to Hey Jude. So, the final score ends up being Beatles: 5.8 (among Beatles fans), Zep: 5.8 (among Zepplin fans). The two Skynyrd fans, though, both vote Hey Jude #2, and Stairway #3, giving the Beatles a total of 6.6 and Led Zep 5.8. Therefore, in this tiny, theoretical sample, "Hey Jude" is the greatest rock and roll song of all time.

Sorry about the length of this... I told you all of that to tell you this: I have a new data project which is based on my previous data project.

I know what songs I like and those which I don't like. I plan to quantify this data and publish my findings in a new format.

As always, I am hochspeyer, blogging data analysis and management so you don't have to,

Geomorphology, meterology and Big Data

Edit: I found a glaring quantitative error, and corrected it.


I'd like to open with an apology to my long-suffering and patient loyal readers.

To be polite, my recent writing has been sparse at best. The reason is that I've been working a lot- I haven't had a "real" day off in three weeks. I think- this is my third weekend without a whole day off. I took Saturday evening off and will be back in the office on Sunday. And what, one might ask, does the author do for a living that requires so much work?

I'm a programmer, working in direct mail (a.k.a. "junk mail"). And what causes overtime in this field?

To keep it simple, there are really only two factors: workload and workforce. Here in the United States of America, the month of October is the time for senior citizens (folks who are 65 years of age or older) to make some choices regarding their prescription drug benefit provider. I'm not an expert on this, but the bottom line for my company is that in September and October we experience a huge spike in this business from these clients. This year, however, workforce came into play. There is another manufacturing plant that our plant has a fairly close relationship with, and they are currently shorthanded (as is our plant) and they also have a client that is doing a similar ad campaign. The deadlines have been tight, and everyone's resources- human and physical- have been severely stressed. My role in this has been pretty much support- but it's been for both plants. Or, wL > wF.

Having said that, I wanted to take a bit of a light-hearted look at Big Data.

There's been a few topics that I've been wanting to write about, but some recent twitter activity led me here. Just for the record, I'm currently listening to Steely Dan's "Midnight Cruiser", and thinking about data.

Data. Steely Dan. Yeah, not much of a connection there.

I'm not sure if the average reader realizes that data geeks even enjoy music.  To be blunt, we do.

But... back to data. Geomorphology is a real word. I was introduced to this term by my wife, who has a geology degree. Geololgy one- liner: she has rocks in her head. She said so. Anyway,  I find the big data landscape falling somewhat messily onto this collision of mismatched terminologies.

As I am not a true "data" person, I often laugh at data terminology and enjoy extending it to its ridiculous, but plausible limits limits.

Point:"data lakes".

Everyone pretty much understands (more or less) what "big data" is. Pretty much like everyone understands what "crime" is. Or "pornography". Alles klar?

In other words, aside from I.T. insiders and those who follow big data, no one really knows what big data is- or how pivotal it can be.

So, I suppose this is a call to action: how do you define your data?

I do not have a lot of data, relatively speaking. "Relatively speaking", of course, is a HUGE qualifier.

When I think about my personal data, I think in terms of things that matter to me- in the "real world",  these things have little value. In the real world, I tend to generate lots of data which has no value to me personally. For example, I've been on twitter for around three and a half years, and in that time have posted nearly 2900 tweets.That sounds like a lot of tweeting, but in reality it's far less than three tweets per day. What would be interesting to me would be a breakdown of my top hashtags.

But, as usual, I have digressed.

The personal data that I track is only in a few categories. I use data to catalog stuff, for the most part: books, videos, music and Legos. I also keep a pedometer log.

Most- if not all- of this data is useless to pretty much anyone except me. But, here we get a peek into the actual application of data science IRL. All data is data, but of all that data, which is most relevant to you? Does Lego care how many 3001 blue elements I own? I think not. They probably do care, however, about my age, where I purchase Lego products, and how much I spend on Lego in a month or year.

This is truly the science and ART of data science. Much of what I tweet on the subject of data science and data analysis is somewhat technical, focused on languages, algorithms and "sciencey" stuff... but business and ethics are also huge, and seem to be marginalized.

"What is the greatest Rock 'N' Roll song of all time"? A valid question. Of course, it is a question that cannot be answered- at least, not with data. Usually, there seem to be three contenders: "Hey Jude" (The Beatles), "Stairway To Heaven" (Led Zepplin) and "Freebird" (Lynyrd Skynyrd).

Likewise, a data scientist must be in tune with business: what is your best product/service? Data science should not only answer that question, but give stakeholders the answers to the five great press questions: Who? What? When? Why? and How? When a data scientist returns valid, data-based answers that are clearly communicated to these questions, the stakeholder has a valid representation of their business based on science and art.

Sorry- I never got around to the humor of Big Data... maybe another time.

As always, I am hochspeyer, blogging data analysis and management so you don't have to.

I Want To Tell You

George Harrison really doesn't get a lot of credit as far as the Beatles go... well, apart from being the reason the Beatles were deported from Germany, at least. "I Want to Tell You" finally rose to the top for today's blog title after considering a number of rain related titles.

Yesterday was the third consecutive day that I tried to cut the grass, as well as the third consecutive day that we experienced rain, and storms, so my first choice was Eurythmics' "Here Comes the Rain Again," which was appropriate. Then, another Harrison song, "Rain" came to mind. I had almost decided to use CCR's "Who'll Stop the Rain?" when Duran Duran's "Hold Back the Rain" came to mind. Sheesh! What's a blogger to do? The thing that made the decision for me was this: I didn't start out to write about the rain, but it's pretty much what got today's post going.

I woke up unnaturally early on Tuesday morning, fifteen minutes before my 0730 alarm. I did all of the usual morning stuff, and by 0800 had my first cup of tea in hand. The weather was threatening rain, but I figured that as long as the rain was light, I would cut the grass. Rain came (but there wasn't much actual precipitation), thunder came, and then it was clear. The weather went through a few cycles like this. After my fifth cup of tea (~1300CST), I had decided I'd had enough. It was relatively dry and the grass was taunting me. I changed into my grass cutting gear and headed out.

As we had been rained out the previous day, I knew the gas tank was full. I proceeded to pull on the starter handle; I was rewarded with the handle coming off the string! I made a bigger knot which actually survived a few pulls before coming off again. Jennifer must have been watching, because she came out of the house. In the most measured and polite voice that I could muster under the circumstances, I handed the green plastic handle to her and said, "Fix it. Please." I then went to the fence and pulled down some of the ubiquitous wild morning glories.

After a few minutes she had reattached the handle and given the motor a few unsuccessful cranks. Undaunted, she went into the garage and got out Ye Olde Sparkplug Socket and removed Ye Olde Sparkplug. It was somewhat fouled, so I got a paper towel, dipped it in some gasoline, and gave it a good cleaning. She replaced Ye Olde Sparkplug, reattached the wire, and it started on the second crank. Then, I proceeded to cut the grass. The grass was higher than we usually allow it to grow, so it took around forty-five minutes to cut. It was also incredibly humid: by the time I was done, there was sweat on top of my perspiration.

I put the mower away, went inside and cleaned up. I made myself a delicious concoction of spicy seafood ramen, Progresso Tomato Basil soup and cheddar cheese and started this blog entry. I infrequently go to work in the "front half" (Monday, Tuesday, Wednesday) of the week, but I text or email every day to see if I'm needed. I was pleasantly surprised when the asked me to come in.

And then it rained some more. Now, about twenty-four hours later, we have flooding, flood warnings, flash flood warnings and more rain is forecast.

Whew! After all of that, what I Want To Tell You is this: I found a couple of pieces in cyberspace that I wanted to share. The first one claims that "80% is the new 100%" and other one through a finely detailed chain of events example effectively refutes the first. I think the author of the first piece might have been speaking from his own experience about growing into a more effective manager, but in the world I live in, 80% almost NEVER cuts it. In my world, when I am proofreading something which has an expiration of July 17, 2013 11:59, that isn't good enough- because sometimes my world is based in internet time, 24/7, and 11:59 not only needs to be followed by A.M. or P.M., but the time zone also needs to be specified. I could go on and on and on, but I think you get the idea.

Lastly, Jennifer and I were at the library today, pretty much intending just to drop a few things off. However, I thought we'd go up to the fourth floor to see if we could see any of the flooding. This time, there was no water. We went to the New Books shelf, and a title caught my eye. With great excitement, I grabbed R For Dummies before anyone else could! I had actually been looking for something like this for the past several months! I showed it to Jennifer, and she naturally said, "What's R?" I took a breath, gathered my thoughts and gave her the best answer I could think of: "R is the freeware version of S." I got the deer-in-the-headlights look. I eventually was able to explain that R is a language used to graphically represent statistics. It's easy enough to understand, but it's not always that easy to explain.  At this point reality came crashing through the R-induced fog, and Jennifer remembered that Mr. T needed a new book to read. I had a thought that he might enjoy a nice biography of Nikolai Tesla, so I went off in search of that while she browsed the electronic catalog. Surprisingly, every one of the Tesla books missing from the shelves, so I went to check the electronic catalog to confirm this. They were indeed all checked out, the catalog confirmed, but I found a few other interesting titles. I went into the stacks and grabbed The Physics Book. It is a softcover book approximately 500 pages in length, and the pages are equally divided between text and pictures or illustrations. It covers milestones, persons and generally presents the history of physics in a very readable format. Another book that caught my eye but I did not check out was Physics for Entertainment. It was written in 1913 and apparently was a best-seller in the 1930's at one time. Curious, curious title.

As always, I am hochspeyer, blogging data analysis and management so you don't have to.

Timehack 062713 0532

Something New

This is my 10th blog entry, and I figured I'd do a bit of e-celebrating by giving it a special title. "Something New?" one might say. Not particularly original, especially since I'm using it in the context of the Beatles' 10th album release, according to a Beatles discography on wikipedia. So, huzzah! Or, as John Lennon so famously put it,  "For our last number, I'd like to ask your help. Would the people in the cheaper seats clap your hands, and the rest of you, if you'd just rattle your jewelry." 


The Beatles actually are the perfect segue into today's thrilling database update. After many months, I've started to populate the Songs table! I decided to start with The Beatles because there are songs that on more than one album, so in addition to progress, I've got some real time testing going on. So far, everything looks good here. Another table, however, has a problem that I was aware of but hadn't chosen to address until now. The Time table has 43.2K records, which is twelve hours in one second increments. I was so happy when I created this in Excel- it was exactly what I needed for the database. The only problem was I was looking at Excel formatted values. When I imported it into Access, I discovered that my simple solution was only a partial solution, because Excel might have displayed 00:02, but Access displayed 12:00:02 AM. So, unless I can figure out a better way, I am looking at no fewer than 43.2K edits to this table BEFORE the table is usable.

I pretty much hate data entry, but really enjoy working with tables, so it's okay. Still, 43.2K+ edits is a BIG number. The good news is that this particular table does not contain data which is vital to the database. But, its 43.2k+ of edits.

In other news, I'm having a time getting Microsoft Office 2010 charts to present data properly. So far, every time I create a chart, Office inverts the data in the chart. No biggie, as I can just resort the data, but its somewhat annoying as a chart should present data as it is presented.