Feeds, Needs and Speeds

Well, it's starting to get real. The database, that is. As I posted earlier on Twitter, 200 rows of data in a single column do not a RDBMS make, but it's a start (the count is currently 224). As Jackie Gleason quipped in one of his signature lines, "And away we go!"

The casual reader may be wondering at this point: there are actually folks out there still building relational databases? Isn't the non-relational NoSQL model more popular in terms of new deployments, versatility and just plain coolness?

I'd like to explain a bit of my personal journey that brought me to Forty-Two, the database.

Long ago, like teenagers everywhere, I faced highschool graduation without a plan. Not just a clear plan, mind you- NO PLAN. Computer Science was in its infancy at the time, and nearly nonexistent in most high schools. I was not one of the cool kids, nor was I a jock or a brain, but I was also not a nerd. However, I knew one or two nerds. And the nerds were highly focused in their scholarly discipline: they were not into math and science; they were into math. They were geometry slingers, wearing leather slide rule holsters on their hips. They had glasses, thick glasses. Below average complexion. Few social skills. Pocket protectors in their left shirt pockets. And behind the pocket protectors... punch cards for their next "program".They were the late 70's analogues of Drs. Sheldon Cooper and Leonard Hofstadler (The Big Bang Theory).

I was not them. Well, sort of not like them. I liked history. I was (probably) the worst kind of history nerd: I was a military history buff. I started out with Avalon Hill games like France 1940 and Panzer Blitz, and progressed to Tobruk and Squad Leader, eventually culminating in the non-Avalon Hill classic Fletcher Pratt's Naval Wargame. The point is this: as the complexity increased, the playability decreased, as did the number of folks willing to take on the rules. But, I digress.

I was accepted to Rosary College (later Dominican University). I declared history as my major, and spent two unremarkable years there. I eventually had five colleges or universities under my belt, with no undergraduate degree to show for all of the buckazoids invested.

Long before this became a mainstream theory in education, I discovered that we do not all learn in the same way, and that higher education was not really the best choice for me.

Fast forward a few decades. I've mentioned this fairly recently- I was working as a data analyst at an "action sports" company. The company designed paintball equipment, and had it manufactured overseas. My job as the data analyst was to take Wal Mart RetailLink data and dice and slice it into what my employer could use.

The problem was my employer was using either Office 2000 or 2003, which limited Excel to a maximum of ~64K  (I believe it was 63,536) rows. As time went on, my data often exceeded this artificial limitation, and I was forced to use Access just to grab the Monday morning data. Once again, skipping several steps, I became adept at moving data between Excel and Access.

Fast forward one more time to today. I use all sorts of tools to do my job; Access and Excel aren't really part of my professional portfolio of commonly used programs on the job, but I use them at home,.. pause for effect.

Yes, although I may have mentioned a bit about Forty-Two before, I don't think I've said too much beyond it was my own little database dev world. Here's where the title comes in: way back when I was an I.T. reseller, we used to often qualify sales by talking about speeds and feeds- equipment specifications. Needs are also important (besides completing the alliterative trilogy).

So, Forty-Two is an obvious reference to Douglas Adams works, and is so named because its' goal is to answer that elusive question: what is the meaning of Life, the Universe, and Everything. It is starting out life as an Access database, currently with only one table. Previous iterations have taught me to take it easy with adding tables, so my aim is to get this Titles table to be mostly complete before adding additional single- or very few-column tables and then finally starting to create the relationships.The first table is called Titles simply because it holds titles: books, videos, software... if its media, then its name goes here. Why? Forced normalization: why go through the normalization process when I can start out with a relatively clean dataset?

This is starting to turn into a wall of words (by my standards, anyway!), so stay tuned... Lego is next!

As always, I am hochspeyer, blogging data analysis and management so you don't have to.

Data, defined

Although it is not my intent, I am certain that this post has the potential to step on a few toes, possibly bruise an ego or two, or ruffle some feathers. I may even get someone mad. Really e-mad.

For starters, I do not have any letters, diplomas, certifications and am not currently professionally employed in whatever one might consider the "data community". Whatever that might be. I do not claim to be an expert or have any special expertise or training in the areas of Big Data, the Internet of Things/Everything, Statistics, Analytics or The Cloud. I was once employed as a data analyst working with Small Data for a short time.

Whew!

So, who and what exactly am I?

I'm a guy who tweets (and retweets) primarily on the subjects of Big Data, IoT, programming and related topics. As far back as high school- maybe even earlier- I've been interested in data. It was either my music collection or Fletcher Pratt's Naval Wargame that gave me my start in classifying and quantifying. I remember even attempting to do a few music surveys way back when, and some of the respondents were unhappy because the polls were not simple popularity contests, but the answers were weighted based upon their position on the poll. Fast forward to today. I'm currently building a flat database of my Lego collection in Excel 2007 (why 2007? Because that's what I have on the computer nearest to the Legos!). This, in turn, will be added to my master database Forty-Two- so named because it answers the question of Life, the Universe and Everything.

Having said ALL of that, I'd like to start off by saying that the term "data" may not be as concrete as we are lead to believe. In my world, data comes in the following flavors: Big Data, Not-So-Big Data, Small Data, Micro Data, and Statistics. Depending upon the size of the dataset(s) and one's perspective, most- if not all- data can fit into more than one classification. Really? Sure. Case: say there's a hypothetical high school senior who is one of the stars of his basketball team. He's a good defender, doesn't get a great deal of fouls (below the league average), and is about average in scoring- except he leads the league in free throw percentage. Several colleges and universities are interested in him- they've got data on this fellow going back to 6th grade. That's data- to them. To me, a person who could care less about basketball- it's nothing more than a bunch of irrelevant stats. On the other hand, these same scouts would not be impressed by the number of PhD's that follow me on Twitter.

So, how big is a Big Data dataset? I asked a coworker. He wasn't sure, but thought a mail list might qualify. Don't laugh too soon- some of the mail lists I've seen have more than 10 million names. To me, though, I'd put that in the Not-So-Big Data or Small Data categories. The IoT,  Amazon, Google, Youtube and Wikipedia definitely fit into the Big Data category, but to the average person, these can be tough to visualize. So, for what I think might be a decent, understandable Big Data dataset, I propose the 2010 U.S. Census. It was a 10 item questionnaire (with a few extra answers possible) that mailed to 135,000,000 addresses representing approximately 309,000,000 persons.

Small Data could be a database, a website or the phone directory of a small to medium sized city- the lines are pretty fuzzy here.

Lastly, there's microdata. I'm not sure if this term is used anywhere else, but I find it to be a convenient term for personal data- data generated and maintained by one person or one family for their own use and not often formally shared. A cataloged collection of coins, stamps, recipes, exercise/workout logs or Legos- all of these are Microdata in my worldview.

Thanks for your patience- I hope you enjoyed this. I generally write a lot less... I'm not a fan of writing or reading walls of words!

As always, I am hochspeyer, blogging data analysis and management so you don't have to.

Meerkat's Greatest Adventure (of 2014), Part 2

Y'all know that the unifying theme of this blog is data. It's my data primarily, with occasional nods to Big Data and the Internet of Things (IoT). Meerkat, our Subaru Outback, and Ziggy, our Garmin GPS, are members of the Internet of Things. But don't say that near them... they're people, as M*A*S*H*'s Radar O'Reilly might say.

Since Merkat is bigger and older than Ziggy, I've decided to turn the keyboard over to him (I think Meerkat is a him...), and let him revisit our trip. Take it away, Meerkat....

*********************************************************************************

Okay... so there we were,

With 1.9 miles on the tripmeter, we departed the SUL. Ziggy immediately started chirping instructions. It was Monday morning- 0738 local, and we found ourselves enmeshed in rush hour traffic. If the pilot had any common sense, we would have been moving through traffic at least fifteen minutes earlier. As it was, we found ourselves ensconced in a "popular" intersection, with nothing to do but wait our turn at the left arrow.

We finally got our turn and proceeded south.

To make this story more interesting, we drove south and west for a long time and arrived in Texas. I sat in the driveway for a day, and the rest of the time I slept in the street, while Ziggy got to go inside.

Afterwards, things got interesting.

The humans did all of their "visiting" and had all of their meetings, and then it was time to go home. Because of all of the unrest in Ferguson, MO, we decided to spend our tourism dollars in safer areas and avoided the St.Louis area entirely.

As we approached Oklahoma City, I sensed a irregularity in one of the mechanical systems, and promptly alerted the pilot. Not long after that, help arrived. A squad of shock troops deployed, led by a dude that could only be described as a cowboy: while the rest of the unit was in full battle dress armor, there he was in cowboy boots and denims, 9mm strapped to his side as if he did this every day.

In any event, he got us to where we needed to be, barked some orders, and in a few hours we were back on the road.

********************************************************************************

I promise that's the last time I will EVER let Meerkat write a blog entry! Well, I'll never let ANY individual that can be classified as a member of the IoT guest write in my blog. I really thought Meerkat was going to make it really interesting... I was wrong- sorry.

In the wonderful world of data, I FINALLY backed up my Omron pedometer data. Some of you might recall that in February (One simply does not raid the SUL) I lamented the loss of fifteen months worth of pedometer data due to a HDD crash. Well, even some us "old hands" are a little slow at times, but tonight I FINALLY downloaded the data to the desktop. I'll back it up to the external HDD once I clean it up a bit.

Clean?

Yes, the data- specifically the dates- are improperly formatted for my use. When I downloaded the software from the Omron site, I got the European version rather than the American version of the software. It doesn't really matter too much as I'm okay with kilometers and the energy expenditure is shown in kcal's rather than joules, but even though I'm okay with European dating, my copy of Excel sorts American style (for the unaware, Euro=day/month/year, American=month/day/year) So, the dates need to be edited before being backed up. There's a funny story about those dates... well, it's funny now, anyway.

Back when I was in high school, I was introduced to Fletcher Pratt's Naval Wargame (see "Imagination"). The resources I used for this game were all published in England, so I became accustomed to the Euro dating system, so much so that when I got my driver's license, I entered my birthday Euro style without thinking, and when I received my license, my birthmonth was listed as May rather than September. I had to go back to the license facility, explain what had happened and get them to issue a new license.

There may be some "big" data news in the next blog... probably. There may be a Christmas blog as  well.

As always, I am hochspeyer, blogging data analysis and management so you don't have to

A cautionary tale of fixed values used in calculating extrapolated fields

Since writing A Tale of Two Databases, I've dusted off my Excel worksheet that I use to keep tabs on blog readership. Yes, deep, deep, way deep down inside of me- somewhere- there's a tiny bit of statistician. And this statistician, who lives somewhere on the Continuum-, er, that is, the Spectrum- likes to keep track of statistics about the blog.

A bit of background is in order at this juncture. I've long been a fan of collecting facts and figures- at least since high school. The year I was a freshman in high school, a new store opened that was on my way to school. The shop was named "The Sutler's Wagon", and Jeff Urquhart (the store has long since closed) was the proprietor. It was a hobby shop. Even back then, I LOVED hobby shops.

Prior to The Sutler's wagon opening, there were three shops I frequented whenever I could get my Dad to take me to them. One was Pilot. I don't remember the whole name of the business, but Pilot was more of a hardware store with a pretty bustling O scale and S scale tinplate selection and HO scale model railroading. Pilot was actually pretty close to Jeff's shop, and was derisively referred to as "the Pirate". Stanton in Chicago was a fairly large store, stocking everything from N to G scale railroading, as well as rockets, RC, plastic models, flying balsa planes... pretty much anything one could ask for in hobbies. The last store was Hill's in Park Ridge, which seemed to have a product range similar to Stanton but smaller selection.

The Sutler's Wagon was different. No trains. A couple of rockets. Some model cars. Everything else was military. Books. Models. Boardgames. Paint. Brushes. Miniatures. Miniatures in 15mm, 25mm, 54mm, 1/35th, 1/72nd, 1/144th, 1/700th, 1/1200th  and 1/2400th scale. The 15's, 25's, 1/144th, and 1/2400th were generally diecast metal; the others were generally plastic. He had rules for the games- these miniatures were not just for looking pretty- they were meant for action!

And this has what to do with anything?

I've mentioned this game before. In a very real way, Fletcher Pratt's Naval Wargame taught me long division. While playing the game is fun. I think I had almost as much fun making the ship (data) cards. In the game, each ship has a point value. The point value is based on a number of data which need to be factored into an equation. That's the easy part. From the total point value, one must then create a damage table which shows how a ship's performance is degraded as she takes damage. So, say a ship has a point value of 10,000  and a max speed of 20 knots. The ship takes 1000 points of damage. 10,000/20= 500, so for every 500 points of damage, the ship would lose 1 knot of maximum speed.  It's not perfect, and is designed to work best with WWI and WWII vessels, but it does work rather well. There are other factors that can make the game quite involved, but overall it is great fun.

From this point, I started looking at music. On the same block as The Sutler's Wagon was a music store called RPM records. Along with Rolling Stone Records in Chicago, I would scour their bargain "cutout" bins. I discovered a lot of great music in these bargain bins, and by the time I got to college, I had a few hundred albums.  My earliest music databases were handwritten on loose leaf paper- I counted an album as being played when I had listened to at least one side.- this worked for both LPs and cassettes. I did have some 8-tracks, but I did not track these.

Flash forward many years. On a normal day I'm on a PC or two at work, and three PCs (simultaneously) at home. In all honesty, a lot of what I do currently is online gaming, but I also write this blog, do a bit of research, maintain a few spreadsheets, and work on my database, Sometimes, I even take a bit of time off to play a game or two on Steam (user= drehenthalerhof).

Here's a lesson for all of you budding data analysts, data scientists, statisticians and anyone else who uses spreadsheets and databases: if the datum doesn't look right, don't assume it is. Check it- even if you are the one that entered it!

So there I was Saturday night, updating the Excel blog readership spreadsheet. This is not a particularly complicated spreadsheet- fifty rows and thirteen columns. As I've repeatedly mentioned, Google's free analytics aren't too deep- they're pretty much stats, and they only show the Top 10. And- because they're real time, if I miss a day or two, I can potentially miss some data. So I've had extrapolate to get a more accurate picture. At some point, though, I accidentally overwrote a formula- and my beautiful table of extrapolated data suddenly was being calculated by a constant instead of a variable! I figured out the error quickly enough, but one can't count on an easy fix every time. When I suspect a problem, or when I need to do a QC on an Excel worksheet, CTRL+~ is my goto.

As always, I am hochspeyer, blogging data analysis and management so you don't have to.

Imagination

I've always had a very vivid imagination, and have a moderately strong "artsy" side. I enjoyed comic books in high school, and was a regular reader of some of the readily available (but not necessarily mainstream titles) such as Howard The Duck, Conan the Barbarian (the large format, black-and white one) and Heavy Metal. I had wanted to draw in the style of the Conan comics, and after much practice had become underwhelmingly mediocre at drawing some body parts, so I abandoned that venture. I wanted to paint for the longest time, but just never got around to putting down cash  for paints, brushes, easel, etc; the closest I ever got to this was painting 25mm miniature wargaming figures (soldiers), 1/32 scale military models and 1/700 scale WWII waterline warship models. The "25's" I was not bad at, the 1/32 scale stuff was pretty hit-or-miss, but I was pretty good at the waterline models. My favorites were Imperial Japanese Navy (IJN) destroyers- I had a book that had a detailed history of the IJN which gave brief unit histories of nearly every ship from the Meiji Restoration until the end of the Second World War. The book was a British publication with very high quality paper and lots of photographs- I used it as my painting guide (I loved painting Japanese characters on the ships' sides) and for modelling some of the ships' rigging (the wires or ropes which stretch from mast to mast on a ship); it's quite amazing what just a few pieces of rigging do for a ship. This book was also my main data source for creating ship data cards for Fletcher Pratt's Naval Wargame. In short, I was one of "those" kids who had weird hobbies and not a ton of friends who wanted share in the hobbies and games.

Oh, and I am an only child.

I mentioned I have a pretty vivid imagination. Not always having playmates available, it was not uncommon for me to play a game of Monopoly or Risk by myself... with the maximum number of players! I also starting writing stories in high school, generally science fiction; most didn't get past a chapter or two. My enjoyment of writing has carried on into adulthood, and this blog is it's latest manifestation.

The other "artsy" thing I've always enjoyed was photography. Before anyone knew what a "selfie" was, I took one in my Dad's 1965 Chevy Impala wagon. I had to be around ten years old, and it was on a trip to visit relatives in Minnesota. (It should be noted that this was truly "back in the day"- the car had seatbelts, but their use was not required). I was in the back seat where the green Coleman cooler was. I put my head over the front bench seat and looked down- there was the camera. Unable to resist, I grabbed the camera and took a picture of... ME! It's a great picture, and we have it in a photo album somewhere.

That was the first picture I remember taking. I've taken many more since then. At one point, I even took Photo Tech classes- long before digital. I've shot 126, 35mm and even Kodak disk. This past week, though, I entered the digital age with our acquisition of a Nikon D3200.

Back in my 35mm days, I owned Minolta XG-M and X-700 camera bodies; I still have the X-700, along with a 50mm, 35-70mm and the prize of my old lens collection: a 250mm catadiaptropic lens! The "cat" is a special telephoto lens that is much smaller and lighter than a typical 35mm lens. It is unique in that the aperture is fixed, and that it uses a pair of mirrors inside the lens to attain the greater focal length. When I took the hood off of my cat, it had pretty much the same dimensions as a standard 50mm lens; to the casual observer, I was using a standard lens- the weight ( and therefore balance) were also similar. The downsides are twofold. First, the lens is slow- mine was an f5.6, and could be further stepped down by a rear-mounted, screw-in gray filter to f11. The second downside can also be exploited as an in-camera effect: the doughnuts.

This is my lens on a later-model body with a motor drive.

The doughnuts were formed by the back mirror, as I recall. If you look closely at the picture, you'll see something at the front of the lens that looks like a new moon- that is the front mirror. If you were looking at the lens head-on, there would be a large red circle (the back of the front mirror) surrounded by the silver of the rear mirror. The rear mirror has a hole in it which accomodates the image gathered by the rear mirror and reflected by the front mirror. When looked at through the rear of the lens, the front mirror is tiny. These lenses are also more delicate than a standard SLR lens, because the position of the front mirror is critical, and any hard handling can make these lenses useless.  

That's all for now. No cool photos to post or data to report... but hopefully I should have some soon!

As always, I am hochspeyer, blogging data analysis and management so you don't have to.