This September has pretty much been THAT sort of month for me- and while I was worrying about September, October rolled in! And if you're wondering what "THAT sort of month" means, nearly every word in these first two sentences needed to be corrected BEFORE the proofreading stage of posting this blog.
FRACK!
For those long-suffering, loyal readers who might have been wondering where this blog had gone... rest assured, it is still here. Obviously.
What's been going on in my life?
Well, for starters, I'm still butting heads with R. For those unacquainted, ""R" is a statistical programming language.
EH?
My main Twitter account, @cjoelharrison, deals primarily with data: Big Data, the Internet of Things, Statistics and Mathematics. Now, before you write me off as a propellerhead with no foundation in the real world, let me explain what I do on Twitter.
My main twitter account exists primarily to gather and repeat information and news about Big Data, the Internet of Things and anything related to these topics. Lots of verticals, especially ag, petro, transport and health, and the IoT/IoE applications therein. (And for those who know me IRL... WTF???)
Yes, I actually think about these things.
Anyway, moving right along- I finally installed R and RStudio on our Windows 10 machine... somewhat successfully. R is okay, but the 'swirl' library is not working, so I made a post on the R Studio forum, and hope for a fix.
That's all for now, I suppose.
As always, I am hochspeyer, blogging data analysis and management so you don't have to.
Showing posts with label Big Data. Show all posts
Showing posts with label Big Data. Show all posts
Monday, October 3, 2016
Monday, October 12, 2015
Geomorphology, meterology and Big Data
Edit: I found a glaring quantitative error, and corrected it.
I'd like to open with an apology to my long-suffering and patient loyal readers.
To be polite, my recent writing has been sparse at best. The reason is that I've been working a lot- I haven't had a "real" day off in three weeks. I think- this is my third weekend without a whole day off. I took Saturday evening off and will be back in the office on Sunday. And what, one might ask, does the author do for a living that requires so much work?
I'm a programmer, working in direct mail (a.k.a. "junk mail"). And what causes overtime in this field?
To keep it simple, there are really only two factors: workload and workforce. Here in the United States of America, the month of October is the time for senior citizens (folks who are 65 years of age or older) to make some choices regarding their prescription drug benefit provider. I'm not an expert on this, but the bottom line for my company is that in September and October we experience a huge spike in this business from these clients. This year, however, workforce came into play. There is another manufacturing plant that our plant has a fairly close relationship with, and they are currently shorthanded (as is our plant) and they also have a client that is doing a similar ad campaign. The deadlines have been tight, and everyone's resources- human and physical- have been severely stressed. My role in this has been pretty much support- but it's been for both plants. Or, wL > wF.
Having said that, I wanted to take a bit of a light-hearted look at Big Data.
There's been a few topics that I've been wanting to write about, but some recent twitter activity led me here. Just for the record, I'm currently listening to Steely Dan's "Midnight Cruiser", and thinking about data.
Data. Steely Dan. Yeah, not much of a connection there.
I'm not sure if the average reader realizes that data geeks even enjoy music. To be blunt, we do.
But... back to data. Geomorphology is a real word. I was introduced to this term by my wife, who has a geology degree. Geololgy one- liner: she has rocks in her head. She said so. Anyway, I find the big data landscape falling somewhat messily onto this collision of mismatched terminologies.
As I am not a true "data" person, I often laugh at data terminology and enjoy extending it to its ridiculous, but plausible limits limits.
Point:"data lakes".
Everyone pretty much understands (more or less) what "big data" is. Pretty much like everyone understands what "crime" is. Or "pornography". Alles klar?
In other words, aside from I.T. insiders and those who follow big data, no one really knows what big data is- or how pivotal it can be.
So, I suppose this is a call to action: how do you define your data?
I do not have a lot of data, relatively speaking. "Relatively speaking", of course, is a HUGE qualifier.
When I think about my personal data, I think in terms of things that matter to me- in the "real world", these things have little value. In the real world, I tend to generate lots of data which has no value to me personally. For example, I've been on twitter for around three and a half years, and in that time have posted nearly 2900 tweets.That sounds like a lot of tweeting, but in reality it's far less than three tweets per day. What would be interesting to me would be a breakdown of my top hashtags.
But, as usual, I have digressed.
The personal data that I track is only in a few categories. I use data to catalog stuff, for the most part: books, videos, music and Legos. I also keep a pedometer log.
Most- if not all- of this data is useless to pretty much anyone except me. But, here we get a peek into the actual application of data science IRL. All data is data, but of all that data, which is most relevant to you? Does Lego care how many 3001 blue elements I own? I think not. They probably do care, however, about my age, where I purchase Lego products, and how much I spend on Lego in a month or year.
This is truly the science and ART of data science. Much of what I tweet on the subject of data science and data analysis is somewhat technical, focused on languages, algorithms and "sciencey" stuff... but business and ethics are also huge, and seem to be marginalized.
"What is the greatest Rock 'N' Roll song of all time"? A valid question. Of course, it is a question that cannot be answered- at least, not with data. Usually, there seem to be three contenders: "Hey Jude" (The Beatles), "Stairway To Heaven" (Led Zepplin) and "Freebird" (Lynyrd Skynyrd).
Likewise, a data scientist must be in tune with business: what is your best product/service? Data science should not only answer that question, but give stakeholders the answers to the five great press questions: Who? What? When? Why? and How? When a data scientist returns valid, data-based answers that are clearly communicated to these questions, the stakeholder has a valid representation of their business based on science and art.
Sorry- I never got around to the humor of Big Data... maybe another time.
As always, I am hochspeyer, blogging data analysis and management so you don't have to.
I'd like to open with an apology to my long-suffering and patient loyal readers.
To be polite, my recent writing has been sparse at best. The reason is that I've been working a lot- I haven't had a "real" day off in three weeks. I think- this is my third weekend without a whole day off. I took Saturday evening off and will be back in the office on Sunday. And what, one might ask, does the author do for a living that requires so much work?
I'm a programmer, working in direct mail (a.k.a. "junk mail"). And what causes overtime in this field?
To keep it simple, there are really only two factors: workload and workforce. Here in the United States of America, the month of October is the time for senior citizens (folks who are 65 years of age or older) to make some choices regarding their prescription drug benefit provider. I'm not an expert on this, but the bottom line for my company is that in September and October we experience a huge spike in this business from these clients. This year, however, workforce came into play. There is another manufacturing plant that our plant has a fairly close relationship with, and they are currently shorthanded (as is our plant) and they also have a client that is doing a similar ad campaign. The deadlines have been tight, and everyone's resources- human and physical- have been severely stressed. My role in this has been pretty much support- but it's been for both plants. Or, wL > wF.
Having said that, I wanted to take a bit of a light-hearted look at Big Data.
There's been a few topics that I've been wanting to write about, but some recent twitter activity led me here. Just for the record, I'm currently listening to Steely Dan's "Midnight Cruiser", and thinking about data.
Data. Steely Dan. Yeah, not much of a connection there.
I'm not sure if the average reader realizes that data geeks even enjoy music. To be blunt, we do.
But... back to data. Geomorphology is a real word. I was introduced to this term by my wife, who has a geology degree. Geololgy one- liner: she has rocks in her head. She said so. Anyway, I find the big data landscape falling somewhat messily onto this collision of mismatched terminologies.
As I am not a true "data" person, I often laugh at data terminology and enjoy extending it to its ridiculous, but plausible limits limits.
Point:"data lakes".
Everyone pretty much understands (more or less) what "big data" is. Pretty much like everyone understands what "crime" is. Or "pornography". Alles klar?
In other words, aside from I.T. insiders and those who follow big data, no one really knows what big data is- or how pivotal it can be.
So, I suppose this is a call to action: how do you define your data?
I do not have a lot of data, relatively speaking. "Relatively speaking", of course, is a HUGE qualifier.
When I think about my personal data, I think in terms of things that matter to me- in the "real world", these things have little value. In the real world, I tend to generate lots of data which has no value to me personally. For example, I've been on twitter for around three and a half years, and in that time have posted nearly 2900 tweets.That sounds like a lot of tweeting, but in reality it's far less than three tweets per day. What would be interesting to me would be a breakdown of my top hashtags.
But, as usual, I have digressed.
The personal data that I track is only in a few categories. I use data to catalog stuff, for the most part: books, videos, music and Legos. I also keep a pedometer log.
Most- if not all- of this data is useless to pretty much anyone except me. But, here we get a peek into the actual application of data science IRL. All data is data, but of all that data, which is most relevant to you? Does Lego care how many 3001 blue elements I own? I think not. They probably do care, however, about my age, where I purchase Lego products, and how much I spend on Lego in a month or year.
This is truly the science and ART of data science. Much of what I tweet on the subject of data science and data analysis is somewhat technical, focused on languages, algorithms and "sciencey" stuff... but business and ethics are also huge, and seem to be marginalized.
"What is the greatest Rock 'N' Roll song of all time"? A valid question. Of course, it is a question that cannot be answered- at least, not with data. Usually, there seem to be three contenders: "Hey Jude" (The Beatles), "Stairway To Heaven" (Led Zepplin) and "Freebird" (Lynyrd Skynyrd).
Likewise, a data scientist must be in tune with business: what is your best product/service? Data science should not only answer that question, but give stakeholders the answers to the five great press questions: Who? What? When? Why? and How? When a data scientist returns valid, data-based answers that are clearly communicated to these questions, the stakeholder has a valid representation of their business based on science and art.
Sorry- I never got around to the humor of Big Data... maybe another time.
As always, I am hochspeyer, blogging data analysis and management so you don't have to.
Thursday, June 4, 2015
(Around) 2.5 years, or, my personal Interwebz v2.1
Around two and a half years ago, I posted my first blog- December 4th, 2012 (it's a pretty short post- even by my current blogging standards!). Even way back then (figuratively speaking) I strove to keep data in the mix as the unifying theme of this blog. And at the end of May, 2015, I'm still blogging about life in general with data as the underlying theme.
It's June of 2015 now, and I've been working on my Twitter account. I consider Twitter serious work primarily because it is an integral element of my online presence strategy- and two and a half years ago, I didn't even know what an online presence strategy was, let alone seeing the need for one. After all, Twitter is one of the places where this blogs posts, but more importantly, it's where I interact with the world of #bigdata and #IoT.
I'm a firm believer in cross-pollenization and shameless self-promotion when it comes to creating a larger presence on the internet, so here's an invitation: if you have an understanding of any of the following terms, have a personal or professional interest in them, or possibly have heard one or more of them before and would like to learn more about them... here's a sample of the topics I address on my (hochspeyer) account on twitter:
#bigdata, #notsobigdata, #smalldata, #microdata, #database, #analytics, #stats, #IoT (Internet of Things), #IoE (Internet of Everything), #M2M (machine to machine), #machinelearning and, of course, #hadoop and #python. I touch on a lot of related topics on that account, but if Big Data and the Internet of Things are topics which you are interested in, you may enjoy following.
My other twitter account (hochspeyer1) still has a tech focus, but has more of a personal touch, looking more at #maker and #programming topics, as well as life in general.
Whew!
The state of the state of my microdata projects: as of today (Jun 4, 2015), the SUL (Secret Underground Lair) is still theoretically experiencing remodeling; by this I mean that while all of the furniture and shelving are in their (probable) final positions, everything that came out and came from upstairs still is in need of homes. Forty-two. my database project, is currently on hold. The Lego database (currently an Excel 2007 workbook) is currently in a data entry phase, and will eventually be a part of Forty-two. Even the Raspberry Pi, Arduino and Python projects are all on hold pending the completion of the SUL upgrade/remodel. Lastly, I'm building my media library, one song at a time. I'm using Windows Media Player, which I suppose is extremely lazy, but the interface is familiar and it does pretty much all that I want, so for now it suffices.
That's all from the SUL for now. As always, I am hochspeyer, blogging data analysis and management so you don't have to.
It's June of 2015 now, and I've been working on my Twitter account. I consider Twitter serious work primarily because it is an integral element of my online presence strategy- and two and a half years ago, I didn't even know what an online presence strategy was, let alone seeing the need for one. After all, Twitter is one of the places where this blogs posts, but more importantly, it's where I interact with the world of #bigdata and #IoT.
I'm a firm believer in cross-pollenization and shameless self-promotion when it comes to creating a larger presence on the internet, so here's an invitation: if you have an understanding of any of the following terms, have a personal or professional interest in them, or possibly have heard one or more of them before and would like to learn more about them... here's a sample of the topics I address on my (hochspeyer) account on twitter:
#bigdata, #notsobigdata, #smalldata, #microdata, #database, #analytics, #stats, #IoT (Internet of Things), #IoE (Internet of Everything), #M2M (machine to machine), #machinelearning and, of course, #hadoop and #python. I touch on a lot of related topics on that account, but if Big Data and the Internet of Things are topics which you are interested in, you may enjoy following.
My other twitter account (hochspeyer1) still has a tech focus, but has more of a personal touch, looking more at #maker and #programming topics, as well as life in general.
Whew!
The state of the state of my microdata projects: as of today (Jun 4, 2015), the SUL (Secret Underground Lair) is still theoretically experiencing remodeling; by this I mean that while all of the furniture and shelving are in their (probable) final positions, everything that came out and came from upstairs still is in need of homes. Forty-two. my database project, is currently on hold. The Lego database (currently an Excel 2007 workbook) is currently in a data entry phase, and will eventually be a part of Forty-two. Even the Raspberry Pi, Arduino and Python projects are all on hold pending the completion of the SUL upgrade/remodel. Lastly, I'm building my media library, one song at a time. I'm using Windows Media Player, which I suppose is extremely lazy, but the interface is familiar and it does pretty much all that I want, so for now it suffices.
That's all from the SUL for now. As always, I am hochspeyer, blogging data analysis and management so you don't have to.
Wednesday, May 13, 2015
Data, defined
Although it is not my intent, I am certain that this post has the potential to step on a few toes, possibly bruise an ego or two, or ruffle some feathers. I may even get someone mad. Really e-mad.
For starters, I do not have any letters, diplomas, certifications and am not currently professionally employed in whatever one might consider the "data community". Whatever that might be. I do not claim to be an expert or have any special expertise or training in the areas of Big Data, the Internet of Things/Everything, Statistics, Analytics or The Cloud. I was once employed as a data analyst working with Small Data for a short time.
Whew!
So, who and what exactly am I?
I'm a guy who tweets (and retweets) primarily on the subjects of Big Data, IoT, programming and related topics. As far back as high school- maybe even earlier- I've been interested in data. It was either my music collection or Fletcher Pratt's Naval Wargame that gave me my start in classifying and quantifying. I remember even attempting to do a few music surveys way back when, and some of the respondents were unhappy because the polls were not simple popularity contests, but the answers were weighted based upon their position on the poll. Fast forward to today. I'm currently building a flat database of my Lego collection in Excel 2007 (why 2007? Because that's what I have on the computer nearest to the Legos!). This, in turn, will be added to my master database Forty-Two- so named because it answers the question of Life, the Universe and Everything.
Having said ALL of that, I'd like to start off by saying that the term "data" may not be as concrete as we are lead to believe. In my world, data comes in the following flavors: Big Data, Not-So-Big Data, Small Data, Micro Data, and Statistics. Depending upon the size of the dataset(s) and one's perspective, most- if not all- data can fit into more than one classification. Really? Sure. Case: say there's a hypothetical high school senior who is one of the stars of his basketball team. He's a good defender, doesn't get a great deal of fouls (below the league average), and is about average in scoring- except he leads the league in free throw percentage. Several colleges and universities are interested in him- they've got data on this fellow going back to 6th grade. That's data- to them. To me, a person who could care less about basketball- it's nothing more than a bunch of irrelevant stats. On the other hand, these same scouts would not be impressed by the number of PhD's that follow me on Twitter.
So, how big is a Big Data dataset? I asked a coworker. He wasn't sure, but thought a mail list might qualify. Don't laugh too soon- some of the mail lists I've seen have more than 10 million names. To me, though, I'd put that in the Not-So-Big Data or Small Data categories. The IoT, Amazon, Google, Youtube and Wikipedia definitely fit into the Big Data category, but to the average person, these can be tough to visualize. So, for what I think might be a decent, understandable Big Data dataset, I propose the 2010 U.S. Census. It was a 10 item questionnaire (with a few extra answers possible) that mailed to 135,000,000 addresses representing approximately 309,000,000 persons.
Small Data could be a database, a website or the phone directory of a small to medium sized city- the lines are pretty fuzzy here.
Lastly, there's microdata. I'm not sure if this term is used anywhere else, but I find it to be a convenient term for personal data- data generated and maintained by one person or one family for their own use and not often formally shared. A cataloged collection of coins, stamps, recipes, exercise/workout logs or Legos- all of these are Microdata in my worldview.
Thanks for your patience- I hope you enjoyed this. I generally write a lot less... I'm not a fan of writing or reading walls of words!
As always, I am hochspeyer, blogging data analysis and management so you don't have to.
For starters, I do not have any letters, diplomas, certifications and am not currently professionally employed in whatever one might consider the "data community". Whatever that might be. I do not claim to be an expert or have any special expertise or training in the areas of Big Data, the Internet of Things/Everything, Statistics, Analytics or The Cloud. I was once employed as a data analyst working with Small Data for a short time.
Whew!
So, who and what exactly am I?
I'm a guy who tweets (and retweets) primarily on the subjects of Big Data, IoT, programming and related topics. As far back as high school- maybe even earlier- I've been interested in data. It was either my music collection or Fletcher Pratt's Naval Wargame that gave me my start in classifying and quantifying. I remember even attempting to do a few music surveys way back when, and some of the respondents were unhappy because the polls were not simple popularity contests, but the answers were weighted based upon their position on the poll. Fast forward to today. I'm currently building a flat database of my Lego collection in Excel 2007 (why 2007? Because that's what I have on the computer nearest to the Legos!). This, in turn, will be added to my master database Forty-Two- so named because it answers the question of Life, the Universe and Everything.
Having said ALL of that, I'd like to start off by saying that the term "data" may not be as concrete as we are lead to believe. In my world, data comes in the following flavors: Big Data, Not-So-Big Data, Small Data, Micro Data, and Statistics. Depending upon the size of the dataset(s) and one's perspective, most- if not all- data can fit into more than one classification. Really? Sure. Case: say there's a hypothetical high school senior who is one of the stars of his basketball team. He's a good defender, doesn't get a great deal of fouls (below the league average), and is about average in scoring- except he leads the league in free throw percentage. Several colleges and universities are interested in him- they've got data on this fellow going back to 6th grade. That's data- to them. To me, a person who could care less about basketball- it's nothing more than a bunch of irrelevant stats. On the other hand, these same scouts would not be impressed by the number of PhD's that follow me on Twitter.
So, how big is a Big Data dataset? I asked a coworker. He wasn't sure, but thought a mail list might qualify. Don't laugh too soon- some of the mail lists I've seen have more than 10 million names. To me, though, I'd put that in the Not-So-Big Data or Small Data categories. The IoT, Amazon, Google, Youtube and Wikipedia definitely fit into the Big Data category, but to the average person, these can be tough to visualize. So, for what I think might be a decent, understandable Big Data dataset, I propose the 2010 U.S. Census. It was a 10 item questionnaire (with a few extra answers possible) that mailed to 135,000,000 addresses representing approximately 309,000,000 persons.
Small Data could be a database, a website or the phone directory of a small to medium sized city- the lines are pretty fuzzy here.
Lastly, there's microdata. I'm not sure if this term is used anywhere else, but I find it to be a convenient term for personal data- data generated and maintained by one person or one family for their own use and not often formally shared. A cataloged collection of coins, stamps, recipes, exercise/workout logs or Legos- all of these are Microdata in my worldview.
Thanks for your patience- I hope you enjoyed this. I generally write a lot less... I'm not a fan of writing or reading walls of words!
As always, I am hochspeyer, blogging data analysis and management so you don't have to.
Saturday, May 9, 2015
Life in the Twittersphere
I'm not going to do the parody lyrics here, but bring up the Eagles' "Life In The Fast Lane" from their "Hotel California" album and you can sort of hum along- it pretty much works. I used to do a lot of parody lyrics, as a matter of fact, but as far as this blog goes, I really didn't make the connection with the Eagles until I actually saw the title of the blog.
For what its worth, this was going to be another "Hoodie Migration" blog, but that was before I read a few posts from the Outmannedmommy blog. To be fair, the Outmannedmommy blog is quite funny, but it may be NSFW (not safe for work) as F-bombs occasionally detonate- especially from guest writers! Still, I enjoy Mary Widdicks' writing- because her style is great, and I can relate to her writing.
Also, strangely (or not?), I don't read a lot of other blogs, Mary's is the only funny one I normally read; most of the rest are retweets on Twitter (nothing wrong with that) about the IoT and bigdata.
And here's the crux, the nexus of this post: Me and Twitter (if you ant to try your hand at parodying this, try Tom T. Hall's most excellent "Me and Jesus" or "Faster Horses, Younger Women, Older Whiskey and More Money".
Twitter is by far the number one reason for my dearth of blogs as of late. I've been contributing a fair amount to the Twittersphere, focusing primarily on Big Data and the Internet of Things. One of my coworkers constantly tells me that I'm a fraud (he's kidding... mostly) because I'm not a PhD, a data scientist, a statistician, or any number of other things associated with Big Data and the IoT... and at that point I always remind him that I make no claims to be anything more than a catalyst or a conduit for ideas. And even though I'm no expert, I've become fairly adept at picking interesting and informative stories, factoids, infographs and blogs. This is evidenced by several folks that ARE PhD's, data scientists and statisticians who follow (and retweet) me.
That's the really short version of Life in the Twittersphere, but I need to get to bed, so I am going to close this and hopefully get another blog out very soon.
News from the Secret Underground Lair (SUL): I think I had mentioned that Mr. T and I were planning a bit of remodeling in the SUL. I'm happy to report that the majority of that work is now complete. Well, the furniture moving is done. Everything that got moved out now needs to be moved back in- I wish there was a defrag utility for small offices... press a button on the wall, and all of the books, boxes, doodads, computer components, computers, etc would find the optimal place to rest and automatically go there.
Finally, data news (this blog is about data, right?) I've got a spreadsheet roughed out, and have put a bit of test data in. So far, everything looks good: according to my flat database, I have 71 Lego elements.
As always, I am hochspeyer, blogging data analysis and management so you don't have to.
For what its worth, this was going to be another "Hoodie Migration" blog, but that was before I read a few posts from the Outmannedmommy blog. To be fair, the Outmannedmommy blog is quite funny, but it may be NSFW (not safe for work) as F-bombs occasionally detonate- especially from guest writers! Still, I enjoy Mary Widdicks' writing- because her style is great, and I can relate to her writing.
Also, strangely (or not?), I don't read a lot of other blogs, Mary's is the only funny one I normally read; most of the rest are retweets on Twitter (nothing wrong with that) about the IoT and bigdata.
And here's the crux, the nexus of this post: Me and Twitter (if you ant to try your hand at parodying this, try Tom T. Hall's most excellent "Me and Jesus" or "Faster Horses, Younger Women, Older Whiskey and More Money".
Twitter is by far the number one reason for my dearth of blogs as of late. I've been contributing a fair amount to the Twittersphere, focusing primarily on Big Data and the Internet of Things. One of my coworkers constantly tells me that I'm a fraud (he's kidding... mostly) because I'm not a PhD, a data scientist, a statistician, or any number of other things associated with Big Data and the IoT... and at that point I always remind him that I make no claims to be anything more than a catalyst or a conduit for ideas. And even though I'm no expert, I've become fairly adept at picking interesting and informative stories, factoids, infographs and blogs. This is evidenced by several folks that ARE PhD's, data scientists and statisticians who follow (and retweet) me.
That's the really short version of Life in the Twittersphere, but I need to get to bed, so I am going to close this and hopefully get another blog out very soon.
News from the Secret Underground Lair (SUL): I think I had mentioned that Mr. T and I were planning a bit of remodeling in the SUL. I'm happy to report that the majority of that work is now complete. Well, the furniture moving is done. Everything that got moved out now needs to be moved back in- I wish there was a defrag utility for small offices... press a button on the wall, and all of the books, boxes, doodads, computer components, computers, etc would find the optimal place to rest and automatically go there.
Finally, data news (this blog is about data, right?) I've got a spreadsheet roughed out, and have put a bit of test data in. So far, everything looks good: according to my flat database, I have 71 Lego elements.
As always, I am hochspeyer, blogging data analysis and management so you don't have to.
Subscribe to:
Posts (Atom)