Small and Medium Business

The major problems with tape backup are:

1. It is unreliable.  Tapes don’t get changed, they overheat, they are terribly hard and slow to verify.
2. Restoration is slow because tape is slow and because you must rebuild the OS and restore files separately.  It is time consuming to test the tape as well.
3. If you run out of room you typically throw away your investment and start over unless you can have someone change tapes and run a two tape backup which is usually very problematic.

These  issues lead to many minor issues such as never REALLY knowing if you have a good backup.  Over the last one to two years a new method of backup has become available to small to medium business.  Very low disk prices has made it affordable to leave unreliable and slow tapes behind and use disk drives to backup disk drives (disk to disk backup).  Disks offer several benefits over tape as a backup medium:

1. Tape is a sequential access medium, meaning to read the last file on the tape you have to read ALL the other files.  Disk is random access, meaning to get to any file on the disk you just look it up in a directory and read it.
2. It is far faster and easier to validate that a disk based backup is working
3. Disks media is more reliable than tape media
4. Disk allows the backup of another disk as an image, this means you not only get the files but you get the Operating System exactly as it is running

Image based backup means you keep an exact copy of the server or workstation you are backing up.  This image is so complete that you can actually mount it as a virtual drive and run it.  More importantly new technology allows the image to be put on dissimilar hardware and the software finds all the drivers that need replacing and takes care of getting that done.  Is is extremely smart about it and can put an image on from most hardware onto must any other hardware with no human intervention.  Rarely you will have to find a driver on the web and deliver it to the machine but it will tell you easily when this is needed.  To test the backup you can easily mount the image and see the files that are backed up.  With the image mounted you can copy a version of a file back to your real hard disk.  Finally incremental “backups” can be made by simply repeating the changes, bit by bit, that have changed on your real hard drive, onto your image making the amount of data to change far lower.  This means you can have several versions of files backed up by having several incrementals on the drive.

Image based, disk to disk backup is far superior to ANY tape backup I have ever seen.  It is affordable, reliable and fast.  That said there are many ways to set it up and it is very important that you use an experienced engineer to create your image backup scenario for the maximum reliablity and ease of use.  We went through about 20 different iterations before finding what we currently consider the right one.

Having written that Tape Backup is not recommended, I should now write about what we are recommending for a solution.  Prudent marketing might make me say……email me to find out, but that’s not in the spirit of a good blog and frankly it’s not that hard to figure out what I might be thinking about.  But before I do the “what” article I think some important background information is necessary.  It is important to understand what you are protecting against.   Backup by itself is really meaningless, what you need is recovery.  There are actually a lot of things to think about when you think about recovery.  Recovery can be categorized as needed in three cases.

1. I have overwritten or deleted a file. I need a copy that was saved yesterday or last week or last month.
2. I have just had a disk drive crash, a motherboard fail, a controller card fail, etc. etc.
3. I have just lost access to my servers physically and electronically either through fire, flood, theft, earthquake, etc.

For each case there is a consequence and a probability of occurring.  In the first case the consequence is annoying and perhaps costly.  It is rarely if ever catastrophic, although it can be in the case of a legal brief that must be filed by a certain time.  It is, on the other hand, very likely that this will occur multiple times.  The second case will very likely happen to you at some point in your business life, perhaps multiple times.  The consequence is loss of the services supported by that hardware for some period of time and perhaps loss of a lot of important data.  The third case, which is the one talked about the most in the tape backup days, is the absolute least likely to occur but the most catastrophic because you will have lost access to all your servers and services.   When figuring out a restore plan you should think about Recovery Time Objective and Recovery Point Objective for each of your systems.

Recover Time Objective is the amount of time you are willing to wait in the event of a failure of a particular category on a particular system

Recovery Point Objective is the point in time to which you need to recover in the event of a failure of a particular category on a particular system.

The following examples are not suggested times these are examples of how the results of your analysis would be documented, your RPO and RTO could be wildly different.

1. Email System,   Category 2: RPO - loss of no more than 1 hour of email.  RTO - 3 hours
2. Electronic Medical Records System, Category 3: RPO - Loss of no more than 1 minute, RTO - 1 hour
3. Word Processing Files, Category 2: RPO - Loss of a day is acceptable.  RTO - 24 hours

You may not need to get your email system back that quickly and you may easily be able to sustain the loss of a days email.  There are clients, on the other hand, who will set the objective at zero lost emails and recovery times of less than 30 seconds.  As you might expect the tighter the recovery point objective and the shorter the recovery time objective the more the solution is going to cost to secure and maintain.

Is there a suggested solution that is cost effective and gives good RTO and RPO?  Glad you asked.  See the article, Image Based Disk to Disk Backup, the New Standard (available by end of January 09)

You have an idea for a computer program, what do you do if your not a programmer?  The marketing literature says it’s really easy but of course it’s not really your thing.  But if it’s that easy you know a guy who’s kid is a guru, he seems to know everything about computers and loves programming.  Then there is also the guy down the block who is a professional and says his company has done lot’s of good programs.  It is very compelling to hire a person who “knows how to program” and they listen to your idea and get started writing code.  Your question is how long will it take and the answer might be something like, well about 6 weeks or 3 months or some number.  A few weeks into the process you start seeing screens and other things that look like a computer program, you might be able to enter data, stuff like that.  A few weeks/months later it’s reported that more of the program is done.  As the time and money you expected to spend approaches you hear they are 90% done.  As the time and money you expected to spend is exceeded you start to hear things like, well there is this one part that we had to do differently but we are almost there.  You ask what that means and they show you.  But the way they “solved” the problem doesn’t really solve your problem even though technically it does some of what you asked, sort of.   You tell them that won’t work and you are told, well the way you want it will take another month we think.  After a big sigh you say ok and a month later you get told that it’s harder than they thought but they almost have it done.  In the mean time you have found several other things wrong with the program you ask them to fix.  A month later you ask about it and they say, oh well those other things got in the way but we fixed those and we are back on the big problem.  You wonder why they didn’t tell you that was going to happen, your still writing checks and can’t use the program.  And on it goes…….

While the exact scenario has lots of different ways of playing out, the point is that creating an important computer program is a non-trivial exercise.  It is an exercise that has much more to do with communication, problem solving, risk managment and goal seeking than it does with learning how to write code.   While many applications have gotten done like the one above they are at best more costly than they needed to be and have fewer functions than you wanted.  Likely they will have a shorter life and when needing enhancements will be much more prone to needing complete rewriting.

In the next article I will outline the basics of a real custom application project.  Just doing it in the proper steps, while helpful, is not a guarantee of success but if you get a developer who would never think of doing it any other way you at least have a chance of having one that knows the value of proper project management.

To my employees after I made a presentation on a tough economy.

“I made a point last night that this year was going to be tough.  Clearly anyone listening to the news of any sort can not come to any other conclusion.  Google, Apple, Honeywell, the banks, the auto industry, etc., etc. are all laying off large numbers of people.  Locally one of our clients is 1/2 the size it was,  another is down, a third is at least somewhat challenged as evidenced by forced vacation.  It would be silly and self defeating to put our head in the sand and not recognize that this is happening.

That said some of the best times I have had personally and in business have been in tough times.  Tough times bring out the best in good people and we are a bunch of good people.  We are going into this year with 2.5 months of working capital including our line of credit.  Marketing is getting our name out there and Mike is beginning to knock ‘em down.  Although SAC Ear, Nose and Throat did not take services immediately Mike has established a relationship into the account.  I am confident that as Mike keeps working that relationship we have a good chance of getting some labor dollars.  Sacramento Coca Cola is looking for more and more work.   Even our client who is half the size is still asking for our help regularly.  The application we just finished is likely to serve a business 1/2 the size of their largest year but they need us more than ever for the efficiency and solid information we can give them with a proper program.  One of our financial management clients get’s much of their income from a percentage of managed investment accounts, we know that investment accounts are down so their income must be down but they increased their plan level with us to Platinum.

Customer service is going to be key to our success.  It always is but it’s even more important during bad times.  Customer Service means many things, it means professionalism, looking and acting sharp.  It means performing for clients quickly, happily and with enthusiasm.  It means always keeping an eye on the value AS THE CLIENT sees it, of every dollar they pay.  If you don’t know the value they see then ask, clients are happy to tell you.  They are much more happy than if they are surprised.  This value needs to be reinforced at the start by making sure they understand how much something will cost, executing on the expectations you have set and bringing up immediately when you can’t meet those expectations due to discoveries you make on the way.  There is nothing wrong with not knowing everything about everything or all about what you might discover in a diagnosis, just don’t try hiding it, that rarely works.  When you are done with a job the followup enforces the value as well.  What you were asked to do and what you did in an email, work order and/or on the time record must be written to show the value of what you’ve done.

Work with each other, when someone get’s a great review ask them what they did and make it your own.  The next time you’ll be the one who get’s the great recommendation and they will be coming to you.

We have a great opportunity in 2009.  Being a tough economy is part of that great opportunity.  The more we “up our game” the more success we will have.

I look forward eagerly to how each of you will use your creativity and excellence to “up your game” and how that will work together to up all our games.”

Bill Pennock

Seriously there is still a place for tape backup but it’s not in most small to medium business networks.  Here are the problems:

1.  It is not random access so it’s difficult to retrieve single files or folders.

2.  It is subject to degradation easily based on heat and moisture.

3.  The tapes are expensive and wear out.

4.  You must find the same tape drive to read tapes in the event of system loss (theft/fire).

5.  You must restore data separately from the operating system making restoring a complete system time consuming and error prone.

6.  When your technician finds out there is a failure that requires restoring from tape, ask them if there is fear in the pit of their stomach.  Truthful caring one’s will tell you they are nervous.

Tape was the only way we had of reasonably backing up systems till just a year or two ago.  But there are now very inexpensive disk storage devices on the market and great software to take advantage of disk as a backup medium.  These two things make tapes a VERY bad choice for new backup in nearly all cases.   Next installment on the blog….What about off-site backup?.

Every company is similar in one way, they all depend on people to get the work done at some level.  People require communication and communication is hard.  As a mater of fact there are quite a few difficult problems in business. This is what makes it both fun and frustrating. But one thing I believe, the better you get the hard parts done especially if your solution is unique, the better value you can bring to the table for your vendors, customers and employees and ultimately for you, the business owner.

Originally this article described one scenario where custom application development was of value. I have since created another article on another very different value that application development can bring. I hope to add more to this series in the future. Till then here are the two articles each with a different value proposition to a business, both very real examples I have seen in action where custom application programming created significant benefit to the companies who chose to invest.

Work Flow
Getting your company to operate the way you want it too is an amazingly difficult task. You post policy and procedure and repeatedly you find people not understanding it the way you expected, resulting in inconsistent and sometimes negative value for your clients. Work flow applications allow you to create a structure where the process must be followed. The rules of the process are built into the program. These programs work well when, to get the job done, the user must interact with the computer for a significant percentage of the time. It is so important though that you are beginning to see computers in places where they traditionally have not been. For instance in transportation where the computer now guides the dispatcher/dispatched rather than leaving it to the whim of the personalities.

Knowing your costs
In this example a companies competitive advantage is based on knowing the exact cost of a large number of inputs and possible outputs of their value process. The better they know the costs the better then can manage supply and labor creating a win for everyone. This article also touches on the value of custom application development to create intellectual property or trade secret value in your business.

This is a post about why Application Development has been a big part of my life for the last 25 years.   I started life more as a mechanic, from refrigerators to huge steam plants and journeyman Air Conditioning I loved fixing things.  In the 70’s though I got a job that allowed me to play on an early HP “personal” computer.  Two big cabinets and a monitor and keyboard, single user.  It was an engineering computer and had an early “basic” language on it.  I got hooked but could not afford to do anything with it till years later when the Commodore 64 was introduced.  From then on I could not get away from it.  I programmed early in the morning and late at night.  I taught myself databases and assembler code (the language of the machine itself) and loved it.   Eventually friends asked me to write programs for them and I hacked some things out and made a few bucks.  But one thing I figured out early on, reading the manual and then hacking away till something worked would be fine for awhile but even then there were people developing methodologies and processes to create better programs.  Mostly they worked on bigger programs for bigger companies but the principles were still the same.  So I made it my mission to learn all I could about how “the big boys” did it and bring the relevant amounts of that process and procedure to the work I was doing on small database projects.  In the process I developed a four step process modeled from many of the things I was learning.  I spent hours on the old Compuserve forums learning from the best in the business.  I sent my code out into the world to get criticised by them also.  That was tough, like sending your baby pictures out only to be told they aren’t that cute.  Eventually that changed and I started getting more kudo’s than cut’s thankfully.

Several things I learned over the years.  One, many people who program for a living don’t take the time to learn their craft and they typically have to fix and fix and re-fix problems.  Two, the right amount of project management is required for every project, none is NEVER the right amount.  You must go through the steps of defining what you are going to build, designing the way you are going to build it and then building it according to the design, testing that all the way.  For a small project the time it takes is minimal.  For a larger project, $100k to $500k the process must be more rigorous.  But it MUST be there even for projects you are just doing for fun yourself, or they will fail.  Three, all businesses, even ones that seem exactly alike from the outside, are not alike.  In fact most businesses take on the personality of their leaders and it’s an amazing and fun thing to see.  Four, communication is an art and not easy.  The biggest impediment to communication is the illusion that it’s actually happening.  If you could take a picture of the images in two people’s brain when they were talking you would, I’m sure, be amazed how different they were.  This means that if you are trying to build software, pure idea’s to start with, you have to question, and re-question what you thought you heard in order to get to the real picture.  That is fun and results in very interesting discoveries.  For instance most companies have words they use frequently that mean entirely different things depending on the context but they don’t even recognize it.  As a non-participant though the words cause immediate confusion.  Likely, though, frequently inside the company these same confusions occur but more subtly because people have learned to adjust.  This can be a source of inefficiency and conflict frequently with misunderstanding as to the source.  As an analyst trying to create a model of the business a computer can use I have to be very clear on definitions, computers are not good at situational meanings.  Through this process I have heard more than once, “I know my business better today than I did before you started”.  That is one of the greatest complements I have or will receive.   Last, for now, I frequently see application development that is done by modeling the individual views of each stakeholder and then trying to link those views together.  In fact there is always an underlying model for how the business runs which then can be viewed through various lenses at various angles to show the right person the right information at the right time.  Miss this and the program is inevitably very complex, bulky and not as effective.  Get the underlying model correct and all sorts of changes can occur for years without ever needing a rewrite.

I love teasing out the real underlying model of the business that serves all the stakeholders.  It’s like looking in a fog at first and poking around till you find the substance.  Application development is a passion for me.

Custom Application Development is not for every business.  In many cases there are standard programs off the shelf that can perform all the tasks you need sufficiently.  Of course programs like Microsoft Business Systems Accounting and Quickbooks and many others can and should handle standard accounting tasks.   They are good at it because Accounting is a very standardized process and is unlikely to be your competitive edge.  But small business is a reflection of the management team and frequently of one or two people who start a company.  Just as personalities are different companies are different and have their own personalities.  It is frequently the case that two companies that make competing products or have competing services actually operate completely differently.  In the competition for both customers and great employees that personality can make the difference between mediocre and great success. 

So how does Application Development fit in?  There are operations in many businesses that require a great deal of information and frequently that needs to be acted on by many people over a process.  Getting the right information to the right people at the right time using paper is cumbersome.  Computers have aided that process by managing it and frequently by not loosing the information on some one’s desk or in their briefcase.  But applications that are standardized can not handle the specific personality of your business and if that’s your niche, your competitive edge, then custom application development becomes an option. 

How do you know if this applies to you?  Do you have databases that have sprung up in Access or other personal databases that are now being relied on by many people?  Do you have someone who is creating huge unwieldy spreadsheets that many people need to keep up?  Do you a person called a programmer or Access guru that you live in fear will disappear?  All of these are indicators that you have a need.  Other indicators are frequent questions around the office like “Do you know if ……”, “What is the status of …..”, “Who knows where we are with ……”.  These are indicators that the right information is not available when needed. 

Squaretree has analysed many companies over the years and helped a good percentage of those.  Sometimes we decide with the client that Custom Application Development is not necessary for the circumstances, the systems and processes are likely to fit just fine in a standard program.  Many times though we are able to steer a company through a successful application development cycle that results in a reliable, manageable program that meets their needs and saves time and money through increases in productivity.

We hear that remark in various ways all the time.  From “I want a computer that just works” to “I want a network that just works” to “I want a mobile device that just works” and on and on.  It makes sense, all of these things are tools to get some other job done. It is the other job that is valuable.  The tool is only valuable because it enables that other job to be accomplished.  If the tool get’s in the way of the thing you are using it to do it is not helpful. 

Squaretree has turned that into an internal slogan.  We do not deliver servers, or server maintenance, or desktop support or router configuration or messaging expertise……we deliver systems that just work.  If we are not doing that directly then we are helping internal I.T. Staff to do that for their companies.  There are alot of benefits to thinking this way, not the least of which is engineers become focused on the solution no mater if it leads to the server, the routers and networking gear or the workstation and are equally proud to work on any of them since they are delivering the “just works” network.

So…the other day I said to my wife, “don’t bother me I’m trying to get done and my computer is fighting me all the way”.  And she said…..you guessed it…..”and you deliver what????”. 

So I had to start thinking about that.  One of the things that we all have to remember, like it or not, is the computer that “just works” is probably not the same computer with which you push the envelope of connectivity and usabilty constantly.  For instance the thing I was having trouble with was getting my second monitor to work.  Sounds easy.  But I have a high end laptop with a very specialized high resolution video card that delivers a very crisp high resolution display on my laptop.  It can easily attach to a high resolution second monitor.  But of course I don’t stop there I suspend the machine and connect it to another high resolution monitor that is different make, model and resolution than the first without letting it reboot or anything.   Oh and between times I’ve decided to use it without the second monitor at all, even though I had windows open on that second monitor.  And I expect it to keep up without a pause during all this perfectly while running Java graphic applications and Windows media player videos both embedded in web sites and stand alone……AND…..I have perhaps 5 web pages, 7 emails, outlook, calendar and 4 programs running all in different windows.  Oh and two web browser brands also.  You know it’s a wonder it keeps up over 99% of the time. 

We can deliver the network or the computer that just works. But much like other appliances you need to define exactly what it is you want it to Just Work on.  Then we can do it.  But if you want it to also be able to just work you also have to live with the fact that you can’t try to confuse it constantly.  If you want push the envelope you have to remember that it will likely take a bit more patience on occassion to make it Just Work.

Incidentally I got what I needed to do accomplished and only 5 minutes later than I was targeting which was hours and hours faster than if I had not had a computer at all.  So ultimately the tool really did “Just Work”

Continued from What is Managed Service really? Some history

From 2000 on I have been looking for ways to use the large systems management programs to make our engineering department and our network clients life better.  I kept trying to get our engineers to put a really large square peg in a pretty tiny round hole.  Products like Tivoli, HP Open View and Microsoft Systems Management Server were robust, full featured and mature products but designed and priced to meet the needs of multi-national corporations with thousands of P.C.s.  They could be scaled down to networks around a thousand and even to 500 to 700 PC networks but they broke down when working with multiple domains meaning multiple separate companies which are the networks we support. 

In 2005 I discovered several programs to address this problem that were finally at a level of maturity to be worth looking at.  The programs we looked at were N-Able, Level Platforms and ManageIT.  All had pretty good capabilities but N-Able added a business training aspect that would they claimed would take you from a reactive to a proactive mind set.  You might remember the program about a Law School where the opening was the professor saying he wasn’t going to teach them the law he was going to teach them to “think like lawyers”.  N-Able understood that it was not enough to simply add a  monitoring system.  The mindset of the engineering department had to change. 

I was tired of the fact that we could only react to problems after they showed up.  I was tired of EVERYTHING being an emergency.  I was tired of not being able to see in advance when a network was overloaded or when a disk drive was showing errors days before it actually went out and we had no reliable way to catch the problem.  I was tired of never being sure that a clients backup was actually going to restore since I had no control over the removal and storage of tapes.  In late 2005 we purchased the program and embarked on a path of change that was a far more costly and difficult path than I would have imagined.  During that time we replaced all our engineers and not just once.  We have had setbacks and successes, rousing accolades and a few more challenging moments.  But through it all I was absolutely resolute in the knowledge that the only way to change the way network support was done was to give the engineers the tools we had just discovered and then work hard to get them to see the vision of using the tools to look into the future and start making more things planned and less things reactions.

That was alot harder than I thought.  The imaturity of the business itself was fighting me, there was not a shared peer concensus on the best way to deal with networks.  Engineers were typically passionate about computers and learned by getting in and doing but lacked alot of discipline around the processes.  The best of them developed the discipline out of neccessity but they were all inventing thier own.  One of the most pervasive problems though was the desire to be a hero.  While it’s stressful and difficult to deal with the user who is frantically trying to get to a network resource it is immensely rewarding to fix the problem and literally have people call you superman or my hero.  It was hard to get some engineers to let that go. 

The managed systems software we have in place today allows us to work to two major goals.  1.  Every thing on your network that will result in a user symptom or risk will send an alert before or when it happens and we will know about it.  2.  Every alert we get is something we actually care about.

The more closely we can get to these two goals the more perfectly we can achieve a truely managed service and the more we can get away from everything is an emergency.

Next:  What a managed network looks like