Although it is not my intent, I am certain that this post has the potential to step on a few toes, possibly bruise an ego or two, or ruffle some feathers. I may even get someone mad. Really e-mad.
For starters, I do not have any letters, diplomas, certifications and am not currently professionally employed in whatever one might consider the "data community". Whatever that might be. I do not claim to be an expert or have any special expertise or training in the areas of Big Data, the Internet of Things/Everything, Statistics, Analytics or The Cloud. I was once employed as a data analyst working with Small Data for a short time.
Whew!
So, who and what exactly am I?
I'm a guy who tweets (and retweets) primarily on the subjects of Big Data, IoT, programming and related topics. As far back as high school- maybe even earlier- I've been interested in data. It was either my music collection or Fletcher Pratt's Naval Wargame that gave me my start in classifying and quantifying. I remember even attempting to do a few music surveys way back when, and some of the respondents were unhappy because the polls were not simple popularity contests, but the answers were weighted based upon their position on the poll. Fast forward to today. I'm currently building a flat database of my Lego collection in Excel 2007 (why 2007? Because that's what I have on the computer nearest to the Legos!). This, in turn, will be added to my master database Forty-Two- so named because it answers the question of Life, the Universe and Everything.
Having said ALL of that, I'd like to start off by saying that the term "data" may not be as concrete as we are lead to believe. In my world, data comes in the following flavors: Big Data, Not-So-Big Data, Small Data, Micro Data, and Statistics. Depending upon the size of the dataset(s) and one's perspective, most- if not all- data can fit into more than one classification. Really? Sure. Case: say there's a hypothetical high school senior who is one of the stars of his basketball team. He's a good defender, doesn't get a great deal of fouls (below the league average), and is about average in scoring- except he leads the league in free throw percentage. Several colleges and universities are interested in him- they've got data on this fellow going back to 6th grade. That's data- to them. To me, a person who could care less about basketball- it's nothing more than a bunch of irrelevant stats. On the other hand, these same scouts would not be impressed by the number of PhD's that follow me on Twitter.
So, how big is a Big Data dataset? I asked a coworker. He wasn't sure, but thought a mail list might qualify. Don't laugh too soon- some of the mail lists I've seen have more than 10 million names. To me, though, I'd put that in the Not-So-Big Data or Small Data categories. The IoT, Amazon, Google, Youtube and Wikipedia definitely fit into the Big Data category, but to the average person, these can be tough to visualize. So, for what I think might be a decent, understandable Big Data dataset, I propose the 2010 U.S. Census. It was a 10 item questionnaire (with a few extra answers possible) that mailed to 135,000,000 addresses representing approximately 309,000,000 persons.
Small Data could be a database, a website or the phone directory of a small to medium sized city- the lines are pretty fuzzy here.
Lastly, there's microdata. I'm not sure if this term is used anywhere else, but I find it to be a convenient term for personal data- data generated and maintained by one person or one family for their own use and not often formally shared. A cataloged collection of coins, stamps, recipes, exercise/workout logs or Legos- all of these are Microdata in my worldview.
Thanks for your patience- I hope you enjoyed this. I generally write a lot less... I'm not a fan of writing or reading walls of words!
As always, I am hochspeyer, blogging data analysis and management so you don't have to.
Showing posts with label Not-So-Big-Data. Show all posts
Showing posts with label Not-So-Big-Data. Show all posts
Wednesday, May 13, 2015
Saturday, August 10, 2013
#notsobigdata
Yes, I did create the twitter hashtag #notsobigdata in the wee hours of Wednesday, August 7. It is for my Not-so-big-data theme. And even though I realize that Big Data is the future, Not-so-big-data is the here, and the now, ... and the foreseeable future.
Not-so-big-data can still be a lot of data. For example, I once worked as a data analyst for a Wal Mart vendor. This was back in the Excel 2000 days, when Excel had a limitation of 65,536 rows. Even though I reported on only six to eight SKUs on each weekly report, there were weeks where the data exceeded Excel's capabilities, and I had to grab the data in Access, and then slice and dice it into chunks that the Excel of the day could handle. It was also at this time that I became familiar with what is seriously the best mouse ever devised- the Logitech Trackman Marble. I had never had to select 60,000+ rows of Excel data before, but this mouse made that task easy. We own four of them.
Getting back to not-so-big-data...
Not-so-big-data is what we deal with on a daily or weekly or monthly basis. Some of it is eclectic and ad hoc, often never gracing a ledger or spreadsheet... data like fuel economy or utility usage, but maybe bits and pieces of the family or organizational budget. The bottom line is that these dollars and cents (or whatever one's local currency happens to be) are important, at least for the day or week or other short period of time in question. Jennifer is a perfect example of a not-so-big-data consumer.
And not only is she a perfect example of the not-so big-data consumer, she's also an expert not-so-big-data manager and analyst. As an example, our has meters for gas, water and electricity. Even though these utilities employ meter readers, we send in our readings every month, because when the utilities don't read the meters, they estimate usage. And sometimes, the meter readers just plain make mistakes. We were on the wrong end of a meter reading error once- a 100 USD mistake. Jennifer caught this, and the mistake was corrected.
Along a similar line of thought, I was doing some data entry in one of my Access tables on Thursday before going in to work, and I ran into some duplicate records, or "dupes" for short. There were at least six sets of dupes. Egad! I was not happy to find them, as I am typically pretty vigilant about my rules. This particular table ("Names") currently only has two columns- the autonumber and the name. Right now, the only purpose the table serves is to force normalization in the current primary ("Media") table. The Names table is the sole source of data for the Artist/Author column in the media table- I do not allow free data entry into areas that are prone to seeing repetitive data. As I was working with some books, I glanced over at the bookshelf and replaced the dupes with authors which had not yet been entered into this table. Adding insult to injury, I know exactly when and how the errors occurred.
I had been entering the data of some compilation music cassettes a few weeks ago, and as each cassette has a number of artists, I got a bit careless and just entered several- instead of checking to see if there already was a record. And so for a moment's complacence, I had to spend time fixing rather than creating. I hope to eliminate this from ever happening again by double-checking to see that there are no more dupes in the table, and then setting the duplicates property of the field to == no duplicates.
As always, I am hochspeyer, blogging data analysis and management so you don't have to.
Not-so-big-data can still be a lot of data. For example, I once worked as a data analyst for a Wal Mart vendor. This was back in the Excel 2000 days, when Excel had a limitation of 65,536 rows. Even though I reported on only six to eight SKUs on each weekly report, there were weeks where the data exceeded Excel's capabilities, and I had to grab the data in Access, and then slice and dice it into chunks that the Excel of the day could handle. It was also at this time that I became familiar with what is seriously the best mouse ever devised- the Logitech Trackman Marble. I had never had to select 60,000+ rows of Excel data before, but this mouse made that task easy. We own four of them.
Getting back to not-so-big-data...
Not-so-big-data is what we deal with on a daily or weekly or monthly basis. Some of it is eclectic and ad hoc, often never gracing a ledger or spreadsheet... data like fuel economy or utility usage, but maybe bits and pieces of the family or organizational budget. The bottom line is that these dollars and cents (or whatever one's local currency happens to be) are important, at least for the day or week or other short period of time in question. Jennifer is a perfect example of a not-so-big-data consumer.
And not only is she a perfect example of the not-so big-data consumer, she's also an expert not-so-big-data manager and analyst. As an example, our has meters for gas, water and electricity. Even though these utilities employ meter readers, we send in our readings every month, because when the utilities don't read the meters, they estimate usage. And sometimes, the meter readers just plain make mistakes. We were on the wrong end of a meter reading error once- a 100 USD mistake. Jennifer caught this, and the mistake was corrected.
Along a similar line of thought, I was doing some data entry in one of my Access tables on Thursday before going in to work, and I ran into some duplicate records, or "dupes" for short. There were at least six sets of dupes. Egad! I was not happy to find them, as I am typically pretty vigilant about my rules. This particular table ("Names") currently only has two columns- the autonumber and the name. Right now, the only purpose the table serves is to force normalization in the current primary ("Media") table. The Names table is the sole source of data for the Artist/Author column in the media table- I do not allow free data entry into areas that are prone to seeing repetitive data. As I was working with some books, I glanced over at the bookshelf and replaced the dupes with authors which had not yet been entered into this table. Adding insult to injury, I know exactly when and how the errors occurred.
I had been entering the data of some compilation music cassettes a few weeks ago, and as each cassette has a number of artists, I got a bit careless and just entered several- instead of checking to see if there already was a record. And so for a moment's complacence, I had to spend time fixing rather than creating. I hope to eliminate this from ever happening again by double-checking to see that there are no more dupes in the table, and then setting the duplicates property of the field to == no duplicates.
As always, I am hochspeyer, blogging data analysis and management so you don't have to.
Monday, June 24, 2013
Lazing on a Sunday Afternoon
I like the concept of leading off with an appropriate song title, a la "Here Is the News", and so I present another today. I'm fairly confident that this has all the signs of becoming a regular blog feature, so today's cut is from Queen's A Night at the Opera, which also includes the perhaps more well-known (even renowned) "Bohemian Rhapsody". While fact-checking this, I also ran across two other Queen (if we're talking Queen here, "favourite" must be spelled in Freddie and Brian's native English) favourites which I haven't heard in sometime: " '39" and "I'm in Love with My Car", both of which were on my personal heavy rotation back in the day.
So, how and why does one laze on a Sunday afternoon?
Well, for one, the temperature was predicted to be about 90F (~32C) today. As Jennifer and I are not fans of humidity or hot weather-- and these two are mates here in the Greater Chicago area-- we were pleased to see the temps pretty much sticking to the low 80's (~26-29C) with a slight breeze. There was not a hint of rain in the sky, although the forecast included a 30% chance and so we decided to do a bit of shopping.
The first stop was our local pet store. We've been doing a bit of searching for a second cat carrier, as Schwarz and Tinka need to be in a carrier when they have their annual veterinarian's exam- and the one we've been using for the past several years is just too small. We had both done a bit of looking on Amazon for something similar to the one we currently own, but nothing struck our fancy. At the store, we saw a few of the carriers which we had seen online, and decided to keep on searching.
The next stop was the Goodwill store. Sundays and Mondays, Goodwill has half-price specials, so we visit a few times per month on the weekends. Our final stop was Aldi (yes, that Aldi!) We picked up a few things, and then headed home.
I've had this "thing" laying around for a while, so long I've forgotten exactly what it is. It was part of a printer at one time, and I had thought it might be interesting to disassemble some day. Today was that day. I gathered several screwdrivers, needle-nosed pliers and a few other tools and began to disassemble the thing. I did this outside at the picnic table, as I didn't know if it had any residual toner in it. Fortunately it didn't, and after about half an hour of disconnecting, unscrewing and sweating in place, I had a pile of parts- plastic, sheet metal pieces, screws and springs. Apart from the plastic, everything went into our spare parts collection; the plastic, unfortunately, was not recyclable, so it went into the trash.
By the time I had finished, it had gotten noticeably cooler and the breeze had picked up... perfect for cutting grass!
Except that the breeze was developing into a wind, the light was fading as the clouds expanded, and it looked like the 30% prediction was going to turn into 100%. As the passing seconds became passing minutes, the clouds became darker and more ominous, and the grass seemed to heave with the mounting winds. And there was thunder... thunder, but no lightning. By this time, I had decided that the grass could wait another day. Jennifer was making what would be a supremely delicious curry by now, so I just messed around a bit on the computer. When the rain finally arrived, it was incredibly underwhelming, evaporating almost as soon as it came down. By the time the thunder had passed out of earshot, you couldn't tell there had been any precipitation whatsoever. And we ate.
And that is how we were found to be Lazing on a Sunday Afternoon. In the wonderful world of Not-So-Big-Data, all that happened was a bit more work on the Lego spreadsheet. I haven't even set a deadline for the completion of this part of the project as yet, because data entry is just one part of the Lego component of the database. I still need to design the other Lego look up tables before importing it into the database.
As always, I am hochspeyer, blogging data analysis and management so you don't have to.
Timehack 0845 062413 (2)
Timehack 1725 062413 (3)
So, how and why does one laze on a Sunday afternoon?
Well, for one, the temperature was predicted to be about 90F (~32C) today. As Jennifer and I are not fans of humidity or hot weather-- and these two are mates here in the Greater Chicago area-- we were pleased to see the temps pretty much sticking to the low 80's (~26-29C) with a slight breeze. There was not a hint of rain in the sky, although the forecast included a 30% chance and so we decided to do a bit of shopping.
The first stop was our local pet store. We've been doing a bit of searching for a second cat carrier, as Schwarz and Tinka need to be in a carrier when they have their annual veterinarian's exam- and the one we've been using for the past several years is just too small. We had both done a bit of looking on Amazon for something similar to the one we currently own, but nothing struck our fancy. At the store, we saw a few of the carriers which we had seen online, and decided to keep on searching.
The next stop was the Goodwill store. Sundays and Mondays, Goodwill has half-price specials, so we visit a few times per month on the weekends. Our final stop was Aldi (yes, that Aldi!) We picked up a few things, and then headed home.
*******
I've had this "thing" laying around for a while, so long I've forgotten exactly what it is. It was part of a printer at one time, and I had thought it might be interesting to disassemble some day. Today was that day. I gathered several screwdrivers, needle-nosed pliers and a few other tools and began to disassemble the thing. I did this outside at the picnic table, as I didn't know if it had any residual toner in it. Fortunately it didn't, and after about half an hour of disconnecting, unscrewing and sweating in place, I had a pile of parts- plastic, sheet metal pieces, screws and springs. Apart from the plastic, everything went into our spare parts collection; the plastic, unfortunately, was not recyclable, so it went into the trash.
By the time I had finished, it had gotten noticeably cooler and the breeze had picked up... perfect for cutting grass!
Except that the breeze was developing into a wind, the light was fading as the clouds expanded, and it looked like the 30% prediction was going to turn into 100%. As the passing seconds became passing minutes, the clouds became darker and more ominous, and the grass seemed to heave with the mounting winds. And there was thunder... thunder, but no lightning. By this time, I had decided that the grass could wait another day. Jennifer was making what would be a supremely delicious curry by now, so I just messed around a bit on the computer. When the rain finally arrived, it was incredibly underwhelming, evaporating almost as soon as it came down. By the time the thunder had passed out of earshot, you couldn't tell there had been any precipitation whatsoever. And we ate.
And that is how we were found to be Lazing on a Sunday Afternoon. In the wonderful world of Not-So-Big-Data, all that happened was a bit more work on the Lego spreadsheet. I haven't even set a deadline for the completion of this part of the project as yet, because data entry is just one part of the Lego component of the database. I still need to design the other Lego look up tables before importing it into the database.
As always, I am hochspeyer, blogging data analysis and management so you don't have to.
Timehack 0845 062413 (2)
Timehack 1725 062413 (3)
Subscribe to:
Posts (Atom)