Monday, August 12, 2013

#notsobigdata Query fail

As Mel the cook on (the television show) Alice is purported to have said, "The best D fence is a good O fence". If true, that Mel was a man wise beyond his years. I'll bet he also moonlighted as a data analyst when he wasn't flipping greasy burgers.

Once again, I've lead off with a comment that is seemingly unconnected to anything- but when it comes to relational databases in which most everything is related to nearly everything in some way, shape or form, as the administrator, manager or designer one needs to be constantly aware of what is going on with the Precious. Er, I mean, the data.

A few days ago I had the opportunity to do a little bit of Show and Tell with a buddy and my database. As the database started out as a catalog of our audio and video collections, it's still very heavy with those sorts of records. My buddy was fairly amazed at the amount of movies we have (I'm not sure I've mentioned this before, but I coined the term "videot" with our son Daniel in mind, as he's constantly spurting out factoids and sundry other movie trivia... if I didn't know better, I'd think the "D" in IMDB.com stood for "Daniel" rather than "Data"). I've tried very hard from the onset to make the database accessible (no pun intended) and logical, and judging by my buddy's reaction, I think I succeeded. We were looking at all sorts of records and I was explaining as we went along some of the features and why it had been set up in the way that it was. Then he said, "Wouldn't it be pretty easy and useful to add another column in here that tells where each item is stored?" That, gentle readers, was an epiphany: in a instant he went from one who had never seen a database before and barely knowing what it could do, to making a suggestion to make it more powerful and useful. I told him, "Yes, thanks- I'd been planning on doing that, but it will take one or two more tables to do it properly."

Here's where planning and design come into play. I could add a column, and have a drop-down list for my locations... IF the number of locations was immutable and fairly small. I've done this in other databases where the dataset itself was finite and would not ever shrink or grow. In my situation, our organization is sometimes quite good, and at other times nonexistent. Additionally, we're always looking for ways (well, I am, at least) of improving storage... it does not always work as planned. In fact, a thought just occurred to me that involves the P-Touch. The more I think about this particular idea, the more I feel the need to utilize Visio and Excel- I have a rudimentary asset tag system, but it needs a bit of tweaking....

STOP! Find a happy data place, find a happy data place, find a happy data place.

This week's goal is to continue with data entry and to work on the physical area around Jennifer's PC, as I have some stacks of paper that need to go to a better place. Also, ALL of the queries need to be rebuilt. Although I do use ad hoc queries fairly often, I rely quite heavily on permanent queries for many things, such as deduping and making lists to output to Excel for analysis and counting. And yes, I am aware that in the #notsobigdata world, these tasks could be done in Access, but my little black book of best software practices says that one should use the best tool for the job. The reason for all that? I was going to show my buddy how queries worked, but I had broken all of them when I added a new lookup table to the primary table, so now they are all gone and need to be reconstructed. Vigilance is the price one must pay for clean data.

And now, it's time to wrap this up. I'm checking out of the Inn at the Stream of Consciousness and getting back to work. And, ...

As always, I am hochspeyer, blogging data analysis and management so you don't have to.

Postscript- Every so often, I will feature some older posts- here are today's group.



 

No comments:

Post a Comment