Showing posts with label Python. Show all posts
Showing posts with label Python. Show all posts

Wednesday, July 13, 2016

Big Data, Small Data, and 42 Data

Its been a bit since I've written, so a quick update is in order.

Work had not been particularly busy as of late; my reason for the delay in writing has more to do with software than anything else. I finally had the opportunity to upgrade my laptop to Microsoft Office Pro 2010 from 2007, which has been on my to-do list ever since the hard drive replacement a while back. With the installation of 2010, I've FINALLY gotten back to working on the database.

I seem to be unable to stop referring to von Moltke the Elder, and I'm not stopping now. I found an older, saved version of the database, and decided it was not usable, so I'm starting afresh (again!) with Forty-Two, with a much-improved (I hope!) design.

Before I got too deep into this database, I thought I'd do a bit of web scraping and see if I could find a "music template" for an Access database. After a fair amount of searching, I discovered that Access templates- generally speaking- do not exist... at least not in the same league as Excel or Word templates. The best explanation that I've found for this was on an Access board, and I paraphrase here: "Access is pretty much a sandbox developer's environment. You won't find many templates. Period."

So, I'm back to doing it the way I've always done it: making it up as I go. Well, LEARNING as I go.

I suppose I should take this opportunity to make once of my periodic disclaimers: I'm not an expert, but I have a deep interest in Big Data, the Internet of Things (sometimes referred to as the Internet of Everything), data analysis, databases, STEM and the Maker Movement. Okay, back to our regularly scheduled program.

Some time ago- not long after I'd discovered the joy of caring for and feeding databases- I ran into a statement which I thought was a bit curious. It was about database design, and the author stated that the best way to design a database is with pencil and paper. I eventually understood his premise and agreed with him up to a point. My personal perspective is that this can be a great starting point, especially if you're completely new to databases, are looking at a completely new database, or if you're designing with a certain goal in mind. I've built small databases, for example, that were for crunching data in a small project (<200 data points) of mostly text. I've used existing databases and added my own queries to provide quality and efficiency reports for ISO 9001:2008 (at least, I THINK that's the spec!). There have been others as well, but my project....

Let me introduce (or reintroduce) everyone to my pet project, Forty-Two. It's called that quite simply because its ultimate function will be to answer the question, "What is the meaning of Life, the Universe and Everything?", which, for those who who are not immersed in in-print memes, is a reference to Douglas Adams' "The Hitchikers' Guide to the Galaxy".  And this was before I had even heard of Python, and Guido van Rossum's homage to Monty Python's Flying Circus. This is THE truth that I have found: I.T. and literature are strange bedfellows.

So here I sit writing about the database.

Once again back to von Moltke: the best plan can fail. Vis à vis my database. I had a grand thought for normalization: make a names table.The names table is as simple as it sounds: first name, middle name or last name- all are contained in one table.  The problem with this theory reared its ugly head almost immediately: music groups do not fit this model. So, there is a new- unplanned  table: music groups. Although the individual members of said groups may be part of the database at some shining point in the future, for now there is a groups table that just lists the names of groups- its the only way to make soundtracks and other compilations work.

That's it for now- I want to publish this entry.

As always, I am hochspeyer, blogging data analysis and management so you don't have to.

.

Wednesday, July 29, 2015

The post that needs to be written

It is now the waning hours of 7/29 in the United States (that's 29/7 in many other parts of the world).

One of the reasons for my fascination with dates and times and other units of measure stems from just trying to be both accurate and understood. In grade school, it didn't matter as much- if you weren't understood in English, then everyone switched to Polish. That was the sort of neighborhood I grew up in.

Later on, I joined the Air Force, and not only had to contend with cultures, I had to deal with time zones. Jennifer had some related experiences, and we're pretty much both in tune with time zones, measurements and cultures.

So there I was on the morning of the 28th of July, trying to finish one more blog entry for the month. I eventually decided that the last theme for the month of July was going to be "fitness". Specifically personal fitness (i.e. my fitness).

A couple of days ago, I had announced to my SIL Donna that I was joining her on a weight-loss quest. I'm no stranger to this, but I've not tried very hard in the past, either, so this time, I'm starting out with a very concrete, attainable goal: I want to drop ten pounds in four weeks (28 days). Now, most things I've read say one pound per week is perfectly safe and attainable. Conventional wisdom would seem to indicate that 2.5lbs (1kg== 2.2 lbs) per week is safely doable. My biggest concern is whether I'll hit that target.

Now, as all of this is PII (personally identifiable information), I may not be able or willing to share all of the data even after I'm done with the process. However, ... getting back to Donna.

We had had a small text convo about my goals, and she said something to the effect of, "Oh, you're going keto!" To which I replied, "No, I'm going carb restriction." We pretty much agreed to disagree at that point.

After further research, though, I'm fairly certain that my new diet is not "keto". It's merely carb restriction. Keto is actually the addition of fat to replace carbs. At this point, I do not intend to do that.

In fact, for the first week or two, I don't plan on doing much of anything resembling what is conventionally referred to as a "diet," although I understand that anything one ingests is one's "diet."

Like nearly everything else, keto comes in a wide variety of forms, and there is a "bit" of disagreement of the proponents (a.k.a. "True Believers") in each plan as to what actually constitutes keto. In my early research, I was much surprised to learned that keto seems to have started as an epilepsy treatment rather than a weight loss treatment.

But-as far as my goals go- I'm not doing a strict keto regimen. Keto seems to do a direct substitution of fats for carbs, with the goal of training one's body to get accustomed to using fat for energy rather than carbs. My plan doesn't go that far, at least initially, and is really not that structured. I start out sensible rather than radical: elimination of pasta and packaged foods. No chocolate or sweet snacks. More exercise. 

So, basics for a week or two is the final operational plan. We'll see how that goes, and then tweak as necessary. Numbers will be posted on a "need to know" basis.

Wink, wink, nudge, nudge. Say no more, say no more. (Python and data!). :)

As always, I am hochspeyer, blogging data management and analysis so you don't have to.  

Thursday, June 4, 2015

(Around) 2.5 years, or, my personal Interwebz v2.1

Around two and a half years ago, I posted my first blog- December 4th, 2012 (it's a pretty short post- even by my current blogging standards!). Even way back then (figuratively speaking) I strove to keep data in the mix as the unifying theme of this blog. And at the end of May, 2015, I'm still blogging about life in general with data as the underlying theme.

It's June of 2015 now, and I've been working on my Twitter account. I consider Twitter serious work primarily because it is an integral element of my online presence strategy- and two and a half years ago, I didn't even know what an online presence strategy was, let alone seeing the need for one. After all, Twitter is one of the places where this blogs posts, but more importantly, it's where I interact with the world of #bigdata and #IoT.

I'm a firm believer in cross-pollenization and shameless self-promotion when it comes to creating a larger presence on the internet, so here's an invitation: if you have an understanding of any of the following terms, have a personal or professional interest in them, or possibly have heard one or more of them before and would like to learn more about them... here's a sample of the topics I address on my (hochspeyer) account on twitter:

#bigdata, #notsobigdata, #smalldata, #microdata, #database, #analytics, #stats, #IoT (Internet of Things), #IoE (Internet of Everything), #M2M (machine to machine), #machinelearning and, of course, #hadoop and #python. I touch on a lot of related topics on that account, but if Big Data and the Internet of Things are topics which you are interested in, you may enjoy following.

My other twitter account (hochspeyer1) still has a tech focus, but has more of a personal touch, looking more at #maker and #programming topics, as well as life in general.

Whew!

The state of the state of my microdata projects: as of today (Jun 4, 2015), the SUL (Secret Underground Lair) is still theoretically experiencing remodeling; by this I mean that while all of the furniture and shelving are in their (probable) final positions, everything that came out and came from upstairs still is in need of homes. Forty-two. my database project, is currently on hold. The Lego database (currently an Excel 2007 workbook) is currently in a data entry phase, and will eventually be a part of Forty-two. Even the Raspberry Pi, Arduino and Python projects are all on hold pending the completion of the SUL upgrade/remodel.  Lastly, I'm building my media library, one song at a time. I'm using Windows Media Player, which I suppose is extremely lazy, but the interface is familiar and it does pretty much all that I want, so for now it suffices.

That's all from the SUL for now. As always, I am hochspeyer, blogging data analysis and management so you don't have to.


Tuesday, May 12, 2015

Batch v2.0

I figured I'd go with a very "I.T.-sounding" title for a change, and quite honestly, anytime you throw out verbiage resembling "v2.0", it implies that this is the new and improved version of something, and it should be checked out. I'm glad you've stuck with me thus far, as I'm about to possibly let some of the wind out of your sails of enthusiasm: this is not about a new an improved app or some software product. It's not a Python library or a Hadoop project.

Actually, it's short for the decidedly unsexy sounding "Bachelorhood Revisited", which is more less where I find myself this morning. The short version of the story is this: Jennifer's Mom had an upcoming medical procedure which required at least a one day hospital stay, and then some assistance once she got home. As Jennifer was pretty much the only family member available for these tasks, she flew over and is spending pretty much all of May with her folks.

So this leaves me in an odd state of quasi-bachelorhood: on the one hand, I can come and go whenever, turn the lights on in the bedroom at all hours, and generally don't have to be to quiet when I come in. Those are the "pluses".

In the debit column, I have to take care of the bills, meter reading, garbage- pretty much everything household related that Jennifer normally does (of course, she helps out tremendously in a remote capacity). I do some of the cooking and food prep... not much, really, but much more that when she's here. I do the vast majority of the dishes, and all of the laundry. And I have to warm the bed up all on my own. This all happens before or after work and on what passes for a weekend.

In summary: this bachelor thing is for the birds. I can't wait for her to get back!

Data news: I've been watching the (free) Google analytics for my blog, and I'm at a loss for this month's performance, which is a personal best for me in terms of viewership: with a little over a third of the month elapsed, I've surpassed my previous record month by 9.2%. And there are over two weeks to go! Woo hoo!

Lego data news:  The Lego spreadsheet continues to grow and evolve. I finished the count on the white 3001 elements (the basic, ubiquitous 2x4 Lego brick); next on the list are the red 3001 elements. The current grand total stands at 1792 pieces. I should note here that when you're holding some of those nifty molded ABS plastic toys in your hand, they are not parts or pieces or bricks... they are elements.

SUL news: The Secret Underground Lair officially has all of its furniture repositioned in the new and improved floorplan. There have been some unofficial murmurings, however, that one more move may be in the offing, although this would require disassembling my half of the office. The jury is still out on this... more news as it becomes available.

As always, I am hochspeyer, blogging data analysis and management so you don't have to.

Monday, February 23, 2015

CL

I try to make "milestone" post a little special; in this case, it is my 150th post, so I've decided to go with "CL" as the title... CL being 150 rendered in Roman numerals.

Big news on the technology homefront. Jennifer has been without a computer for nearly a year. Well, not actually without a computer- she has a very nice Lenovo laptop that she uses. Her desktop expired in May or June of 2014, however. We built her a desktop sometime in June or July, but we had a problem with the motherboard, so she's been using the laptop ever since then. Two Wednesdays ago, Mr. T decided that we should rectify the situation.

We replaced the motherboard.

Now, the odd part about those four words is this:

Mr. T and I were able to get together to get the job done... almost done, that is.  It didn't work- some of the fans spun up, but otherwise the PC was inert. Last Sunday, we took a fresh look at the machine. Mr. T suggested I take a look inside. I did. Much to his surprise, I started touching things- fans, jacks, cables, anything that we had attached. I may not have an A+ certification, but I know my way around a motherboard. And I actually saw the problem before I touched it: one of the memory sticks was not seated. I pushed it down until it clicked into place, and we were in business.

Everyone was happy.

Bouyed by his construction success, Mr. T called in a chit. A year or so ago, we had informally promised Mr. T a new computer after Jennifer got a new PC. With hers being nonfunctional for such a long time, we didn't have to make good on that promise. Now, though, having demonstrated his ability to build, it was time.

After a few configurations, he was finally able to put together a nice Intel-based I5 system and get everything through Newegg. Everything arrived on Friday. Mr. T got up early on Saturday, had a nice breakfast, and for six hours worked on the box.

No go.

As Bill Clinton once famously stated, "I worked harder than I did in my entire life".  Well, with apologies to the former president, its pretty close to what he said.

Frustrated, he asked me if I would look at it with him today (Sunday) Sure, that's what Dads are for.

We opened it up, and pretty much repeated the diagnostic process of the past Sunday. I had to laugh- Mr. T had admonished me for equipping Jennifer's CPU with a masive dual fan equipped heatsink; it turns out that his is just as huge! We poked and prodded for about an hour. The main issue with his box was that the main case fan would not power up.There was also a boot order issue, but we found a work around for that. After testing, we determined that there was a dead terminal on the motherboard. I have to say I was impressed with the quick response I received from Newegg. I went to their chat, entered my issue, and in about 10 minutes I had an RMA.

Jennifer's PC, on the other hand, required a bit of work. After a few days with a new motherboard, I realized that it had a problem: it would not install Windows updates. I finally figured out that Windows 7 needed to be reinstalled. Once that happened, the machine ran flawlessly.

Sorry, once again no data, but a lot of hardware- that's how I roll when I'm the I.T. guy! I did start working on a spreadsheet which will be imported as a table into the new database, and did a bit of Python work as well.

As always, I am hochspeyer, blogging data analysis and management so you don't have to.


Tuesday, February 10, 2015

Forty-Two, v7 (or so)

I've been spending a fair amount of time on Twitter (@CjoelHarrison) lately... so much, in fact, that other elements of my e-life have been suffering. Case in point: this blog. In January of 2015, while I was establishing my presence in the twittersphere, I did not post a single blog. Possibly even more tragic, my playing time on Rail Nation has suffered tremendously. And don't even get me started on my lack of quality Steam time... I'm not certain when the last time I played Sid Meier's Civilization V was.

In any event, here I am once more heralding the latest iteration of my pet database project, Forty-Two. This may indeed be Version Seven; then again, I've not kept track, so it could be any number greater than three. This iteration, though, is different from previous ones: it was not necessitated by a hardware crash or a software glitch. Rather, it is simply a fresh start with a few ideas to make a (relatively) large dataset more organized.

So, I suppose the logical question would be this: why would a home user with no database training want to even try to build an Access database in the first place?

I suppose I could just write it off as an Aspie thing- a project that goes on forever without a realistic chance of being completed (I'm not sure that's exactly an Aspie trait!) It could be that I haven't learned MySQL yet, or haven't even tried NoSQL, and let's not even bring Hadoop into the discussion!

The reality is a lot more pedestrian. And practical: I want to know what I don't know.

In plainer speech, I want to learn how to build a small relational database from the ground up. I want this database to be useful to my family. And, possibly the most important reason for building the database: should anything bad happen, I'd like to have it as a record for an insurance claim.

So there, that's what drives me to keep on trying to build Forty-Two, the database that answers the question of Life, the Universe, and Everything.

The other side projects that have been taking up my time are coding. Professionally, I'm a programmer. The interesting thing about that is that even if I spent the majority of my time programming, the world would probably not recognize me as a programmer, because the tools that I use are very specialized and are more related to composition and layout than programming- and when we DO actually write some code, there is absolutely no way it con be confused with a Turing complete language. Still, some of these tools DO have actual programming, and to help myself to better understand these areas, I've taken to studying the Python language.

Gasp- it's official now: yours truly is studying Python.

print("That's all for now folks!")

Sunday, December 28, 2014

Another look at resolutions

I don't know how things are in your world, but in my world there seem to be a lot of "gotchas". A "gotcha!" is something like, "the other guy blinked" or, "I should've seen that coming". Today's gotcha was the latter. It happened at The Bridge this evening.

I had experienced a fairly fitful sleep Saturday, which was to be expected, I suppose, after being off for two days and then coming to work for one. This time of the year can be something of a sleep challenge for shift workers such as myself. I normally start work around 2000, and am generally done before 0500 the following morning. Our company gives us Dec 24th and 25th, as well as Dec 31st and Jan 1st as holidays. The problem is that this year the 24th fell on Wednesday, and the 25th on Thursday.

As best as I can figure, for the two day holiday, I actually net about a day and a half-after all, I've gotta sleep sometime!

So there I was, short of sleep on Saturday afternoon. I got up, bathed quickly and had a bit of tea. Then I poured a can of Monster into a plastic bottle, we piled into Meerkat, and were off. We arrived at the Bridge and sat down on one of the padded benches in the vestiblue where we usually wait for the doors to the auditorium to open. Tonight's wait was a bit longer than usual, and there were none of the customary sound checks. I overheard someone saying that some technical difficulties had been exerienced. Not long after, however, the doors opened and we funneled in. As we headed toward our seats, I noticed Karl Clauson with a microphone. It was going to be a great service.

I think I've mentioned Karl once or twice before in this blog. His life can only be described as "amazing".

Here's where everything sort of hits the proverbial fan, though. This blog is entitled, "Another look at resolutions".

I don't really recall if it was Friday or Saturday, but Jennifer and I had briefly discussed resolutions. I've never been a fan of resolutions (see Resolutions and other useless things), and Jennifer and I agreed on this point.

Karl spoke on resolutions. Gotcha!

I don't know what it is about our church, but the "gotchas" seem to happen all of the time! Karl's teaching should be up on The Bridge's website in a few days- click on the "Watch or listen to last week's sermon" link to see/hear it.

No data today. The floor of the Secret Underground Lair is covered in the confetti that is the byproduct of shredding old paperwork.  I took that picture about an hour ago- it looks worse now! I think that shredder manufacturers really need to come up with a more user friendly way of transferring shredded paper into one's recycle or trash. Each and every "consumer/SOHO" shredder I've ever used has had this challenge- and this is only a small part of the mess.

I stopped not long after bag #3 was almost full. Tomorrow is Sunday, and I tentatively have three things on my agenda: to finish the shredding and clean up the mess, to put the new motherboard in Jennifer's computer, and to test my new monitor. I guess that's actually four.

At some point in time, I need to get back to working on my database, and to resume my adventures in programming in Python. I suppose that those are worthy tekkie resolutions for 2015.


As always, I am hochspeyer, blogging data analysis and management so you don't have to.