Curious Observation About Engineering

For a while, I've wanted to develop a small application that I can use to sync my mp3 player with my computer, or vice versa. It tends to be a fairly common thing, as I'm always adding/removing songs from one or the other. It also comes up a lot if I do a new install of an operating system and I'd like to have some music on there temporarily; it's just easier to transfer it from my mp3 player rather than another file system on a hard drive somewhere.

Yesterday, I started thinking about this application again, and today I began fleshing out some things on paper. I was going over things such as how you determine if two files are identical (despite file name, file size, etc... which don't really tell you anything about uniqueness), what sort of sync behavior should be used (mutual-add, mutual-add/delete, etc...), and how to handle the more real-life situation where you've got a number of folders with sub-directories as opposed to just a list of files. I had all of these great ideas of how to solve these problems, and my brain was working overtime for the problems I didn't yet have an answer to. I was looking at source code for some common core UNIX file utilities like diff, cmp, cp, etc... looking for pointers...

And then it happened. I realized that I didn't need to make any sort of application. I realized my "problem" was non-existent. Why, you ask?

With the use of the built-in UNIX 'cp' command (file copy), and a few command line switches, I could achieve my sync behavior.

cp --recursive --update

Done.

--recursive handles the issue with multiple levels of directories, and --update handles the issue with determining if files are identical or not, as well as handling the mutual-add syncing.

The curious part comes in with my reaction to this discovery: I was totally bummed! As an engineer, you tend to want to have these problems to solve. Then, you are excited once you solve them. However, realizing there was never a problem to solve in the first place doesn't bring that same excitement, it in fact has the opposite effect.

It does illustrate a few important concepts of engineering though.

1. Don't re-invent the wheel.
2. Keep it simple, stupid!

Some engineers have a tendency to want to use complex or advanced solutions to problems which don't need them. I myself was bummed that such a simple solution would solve this problem, as I had envisioned a much more "sexy" method. This only leads to trouble down the road, and if at all possible, it's better to realize there's a better way to do something before you've finished doing it the wrong way.

There's more to coding than...well.... coding.

Learning how to write software is really a tiny piece of what it means to be a programmer. Let's be honest, writing code is actually pretty easy to do. You spend a few years learning how the basic logic structures work, the looping, function calls, return values, blah blah blah. After that, the most work you put in is studying some new API or learning the nuances of some new language. All in all though, it's not terribly hard to do.

Unfortunately, there's a lot more to it than that. The book series "Effective C++" by Scott Meyers (I've provided links at the end of this entry to all of the items I rave about, so no need to fire up Google) really opened me up to this notion. It was the first book I had read about programming that didn't teach you how to write code, but how to write proper code. Actually, not just proper code, but great code. It was such a breath of fresh air to pick up a book about programming and not have to see the same step-by-step teaching process. Books like this (you should have already gone and checked it out from your library by now...if not, stop reading and go. now. I'm waiting.) focus on things that too many programmers neglect: writing proper, efficient code that does exactly what you intend it to do, and does so very nicely.

But again, that's just another piece of the puzzle. I've been browsing some programming blogs recently and I came across this one: http://www.gamesfromwithin.com/ There is some really great content in there and I encourage you to check it out. The relevance for this discussion is that the author is a big proponent of agile development and test driven development. If you aren't familiar with those concepts, I hear Google works quite well. The basic run down of agile development is that it's a production mechanism of sorts that stresses quick iteration of software based on client feedback (whether the clients are other programmers working on other features or your actual end-users depending on the type of project you have). A main element of agile development tends to be a team oriented approach that involves things like XP (2 programmers working side-by-side on the same piece of code).

For an example of how well these things can work, check out this entry in the above mentioned blog: http://www.gamesfromwithin.com/articles/0602/000104.html You'll be amazed at how much work gets accomplished with this method.

Test driven development follows with the iterative theme in that you essentially reverse the coding process. How? With the use of a unit testing framework, you write your tests before you write your code. Clearly, the test should fail. Then, you write just enough code to make that simple test pass. The key word there is simple. The tests you design should be very small and simple to implement. The author of that blog suggests between 1-2 minutes to design the test, and not much more than 15 to implement the code to pass the test. "Test Driven Development" by Kent Beck seems to be a godsend book on this design methodology. I just picked it up from the library yesterday and it's very helpful in outlining the process, go get it!

These are just 2 examples of design methodologies in software engineering, but there are a ton of other ones. Some work better for certain things, but the point is that this stuff matters. It's vital in a professional software development environment. Without things like this, nothing would get done, and even if it did, the result would be awful. I encourage you to start learning about this stuff while you are still in school, because chances are you won't have too much meaningful class time devoted to things like this. All of it can only serve to make you a better programmer.

This is where the distinction between "Computer Science" and "Software Engineering" comes in. Software engineering is devoted to stuff like this. It's all about the design, the planning, the process. Computer science tends to be focused on the implementation alone. The curriculum in colleges today leaves much to be desired in my book. How can you send CS majors out into the work field with very little knowledge of software engineering principals? To me, they go hand in hand. You can't write any sort of decent code without a design or a planning process. What about a roadmap or schedule for the project? You can't just "wing it" and expect any worthwhile results to come out the other end. Some people think all of this "management" should be the responsibility of the projects "manager". Maybe so, but I don't think it's black and white. The project manager can't write your test cases for you. The project manager (in the sense that most people think of him/her in) can't set realistic and useful code milestones for you. As a programmer, you can't get to where you need/want to be unless you know where that place is before you start.

I encourage you to read as many blogs by professional developers as you can. These people are wise beyond my years, so I could go on and on, but you're better off hearing it out of their mouths (or fingers...) than having me repeat to you what I've learned from them. The "Games From Within" blog above is really outstanding. I learned more about software engineering in a few hours than I have in 3 years of college. In addition, look for some language-specific blogs by some seasoned vets as well. These blogs tend to focus on many similar topics that the "Effective C++" series does. That is to say, the "how to do it right" as opposed to just the "how to do it". I found this site the other day and it has a wealth of information from blogs by over a dozen different professionals, to regular articles: http://www.artima.com/index.jsp

Good luck, and remember: learning how to program is just the first small step in being a software developer.

Resources You Need To Check Out

Websites / Weblogs

• http://www.artima.com/index.jsp
  
• http://www.gamesfromwithin.com/
• http://powerof2games.com/
• http://www.aristeia.com/

Books

• http://www.amazon.com/Test-Driven-Development-Addison-Wesley-Signature/dp/0321146530
• http://www.amazon.com/Effective-C%2B%2B-Addison-Wesley-Professional-Computing/dp/0321334876/ref=pd_bbs_sr_1?ie=UTF8&s=books&qid=1205521782&sr=1-1

There's room for everyone

One of my biggest problems to date has been that I get discouraged when I look at what's already been accomplished in this field. When you are sitting down to work on a trivial idea you've come up with, it's easy to stop and say "Wow, do you have any idea how complex the GCC compiler is? What the hell am I doing with this simple program?!" It's an easy trap to fall into, but a deadly one.

We can't all be John Carmack, Bill Gates, Linus Torvalds, etc etc etc... but that doesn't mean we should find a new profession. Implementing those trivial ideas are essential for growing as a developer, and if you don't go through that process you'll never get to where you want to be. John Carmack didn't roll out of bed one day and write a 3D game engine. He had years of programming experience. Granted, he's a very intelligent man when it comes to math and science, regardless of his programming abilities, but again, he's one of the many exceptions. Most people simply aren't that gifted. However, if programming is the field you want to be in, you can work hard enough at any area of it and become proficient.

The only remedy I've found for this issue is that when you have an idea for a piece of software you'd like to create, don't "Google it". Most ideas that most of us have will be based off of some piece of software that is already in use. Going to Google to check out some of those applications is only going to worsen this issue. I like to get off of my computer, grab a pen and paper and start designing the system "offline". It's actually a lot more productive than sitting in front of your monitor, and you can't get discouraged if you pretend that nobody has ever made this piece of software before.

Going back to the trivial nature of some of the programs we may write... Just as programming books start you off simple before branching into pointers, objects, templates, data structures, etc... you should be doing that with your code. Now, this tends to happen naturally anyways since you can't well write code involving things you don't yet know how to do, but I know I've certainly tried. Not to say that it's always a bad thing to test new waters, but you shouldn't really make a habit of doing something until you understand it. A book progresses rather quickly, but your projects should progress much more slowly in terms of the complexity of the underlying software. This is paralleled by the fact that you cannot simply read a programming book front to back as if it were a novel, it must be an interactive experience in which you are typing the code from the book, doing the examples, doing the projects, etc... The more time you spend building the complexity of your projects up, the better off you'll be in the long run because you'll have an expert-level understanding of everything you've done, which will then make the transition into the really advanced stuff easier.

A great way to do this is to develop your software idea just like a mainstream piece of software. Have iterative releases of your application, adding new features and refactoring code along the way. Make a habit of creating a "TODO" list after each release so that you have specific points to focus on. Ideally you'd have setup some sort of road map at the outset of the project, so you should already have some direction in this regard. Set stern, but reasonable deadlines for your releases, manage a blog documenting the process, invite others to use your applications (friends are fine, as long as they will give honest feedback), etc... Doing all of these things will keep you excited about your project, focused on your project, and you'll absorb a lot more of the techniques than if you just fleshed out your initial idea and say "Meh, that's cool I guess...".

Hope this helps.