Encapsulate

Information hiding is a simple, proven concept that results in software that is easier to test and easier still to maintain. Most software modules should hide some information from all other software. This information could be the structure of data; the contents of data; an algorithm; a design decision; or an interface to hardware, to a user, or to another piece of software. Information hiding aids in isolating faults because, when the hidden information becomes unacceptable in some manner (such as when it fails or it must be changed to accommodate a new requirement), only the piece of software hiding that information need be examined or altered. Encapsulation refers to a uniform set of rules about which types of information should be hidden. For example, encapsulation in object-oriented design usually refers to the hiding of attributes (data) and methods (algorithms) inside each object. No other objects may affect the values of the attributes except via requests to the methods.


Reference:
Parnas, D., "On the Criteria to Be Used in Decomposing Systems into Modules," Communications of the ACM, December 1972.

Don't Reinvent the Wheel

When electrical engineers design new printed circuit boards, they go to a catalog of available integrated circuits to select the most appropriate components. When architects design homes, they go to catalogs of prefabricated doors, windows, moldings, and other components. All this is called "engineering." Software engineers usually reinvent components over and over again; they rarely salvage existing software components. It is interesting that the software industry calls this rare practice "reuse" rather than "engineering."

Magic Square Corollary

The following definition for a Magic Square series is from the EDU-BLOG:

In recreational mathematics, a magic square of order ‘n’ is an arrangement of n2 numbers, usually distinct integers, in a square, such that the n numbers in all rows, all columns, and both diagonals sum to the same constant.  A normal magic square contains the integers from 1 to n2. The term “magic square” is also sometimes used to refer to any of various types of word square.

The constant sum in every row, column and diagonal is called the magic constant or magic sum, M. The magic constant of a normal magic square depends only on n and has the value 

Thus the magic square series is like this: 1, 5, 15, 34, 65, 111, 175, 260…

Often when I want to practice programing I'll write some code to calculate some interesting mathematical number or series. Recently I picked the magic square series. When I finished I noticed that for each order of magnitude beyond n = 20 there was a pattern.

Magic Square Corollary

for y = 0 to ∞

n	M(n)
20	4,010
200	4,000,100
2,000	4,000,001,000
20,000	4,000,000,010,000
...	...

This was done solely for the enjoyment of playing around with some numbers. I used Roger's Online Equation Editor to make the above equation image. Very handy tool.

Next I noticed that any value of n = 10, 20, 30, ..., 90 can have the above equation applied to it. For example, n = 10:

M(10) = 505

Split 505 at the two least significant digits. Maintain the order of the left half to get 500 and 5. Then apply those values on the left and right of the plus sign, as so:

M(n) = M(10 * 10^y) = 500*10^(3y)+5*10^y

M(100) = 500,050

M(1,000) = 500,000,500

...

M(90) = 364,545

M(n) = M(90 * 10^y) = 3,645*10^(3y)+45*10^y

M(900) = 364,500,450

M(9,000) = 364,500,004,500

...

Avoid Numerous Special Cases

There are often exceptional situations to an algorithm's design. Exceptional situations cause special cases to be added to the algorithm. Every special case makes it more difficult to debug, modify, maintain, and enhance an algorithm.

If you find too many special cases, you probably have an inappropriate algorithm. Rethink and redesign the algorithm.

Minimize Intellectual Distance

Edsger Dijkstra defined intellectual distance as the distance between the real-world problem and the computerized solution to that problem. Richard Fairley argues that the smaller the intellectual distance, the easier it will be to maintain the software.

To do this, the structure of the software should as closely as possible mimic the structure of the real world. You can minimize intellectual distance using any design approach. Be aware that the structure of the real-world is not unique. As pointed out so well by Jawed Siddiqi in his March 1994 article in IEEE Software, entitled "Challenging Universal Truths of Requirements Engineering," different people often perceive different structures when examining the same thing and thus construct quite diverse constructed realities.

Keep Design Under Intellectual Control

A design is under intellectual control if it has been created and documented in a manner that enables its creators and maintainers to fully understand it.

An essential attribute of such a design is that it is constructed hierarchically and with multiple views. Hierarchies enable readers to comprehend the entire system abstractly, and then comprehend finer and finer levels of details as they move down the hierarchy. At each level the component should be described from an external point of view only. Furthermore, any single component (at any level in the hierarchy) should exhibit simplicity and elegance.

Maintain Conceptual Integrity

Conceptual integrity is an attribute of a quality design. It implies that a limited number of design "forms" are used and that they are used uniformly. Design forms include the way components inform their callers of error conditions, how the software informs users of error conditions, how data structures are organized, mechanisms for component communication, documentation standards, and so on.

When a design is complete, it should look as if one person created it all, even though it is the product of many devoted people. During the design process, there are often temptations to diverge from accepted forms. It is okay to give in to such temptations if the justification is for additional integrity, elegance, simplicity, or performance of the system. It is not okay to give in solely to ensure the designer x has left his mark on the design. Ego satisfaction is not as important as conceptual integrity.

Conceptual Errors Are More Significant Than Syntactic Errors

When creating software, whether writing requirements specifications, design specifications, code, or tests, considerable effort is made to remove syntactic errors. This is laudable. However, the real difficulty in constructing software comes from conceptual errors. Syntactic errors often look like silly mistakes that can be laughed off. In contrast, developers often feel flawed, or incompetent, when a conceptual error is located. No matter how good you are, you will make conceptual errors. Look for them.

Ask yourself key questions throughout the development process.

• When you read the requirements ask yourself, "Is this what the customer really wants?"
• While you are designing a solution, "Will this architecture behave appropriately under stress?" or "Does this algorithm really work in all situations?"
• During coding, "Does this code do what I think it does?" or "Does this code correctly implement the algorithm?"
• During test, "Does the execution of this test convince me of anything?"

Reference:
Brooks, F., "No Silver Bullet: Essence and Accidents of Software Engineering," IEEE Computer, April 1987.

Use Coupling and Cohesion

Coupling and cohesion were defined in the 1970s by Larry Constantine and Edward Yourdon. They are the best ways we know of measuring the inherent maintainability and adaptability of a software system. In short, coupling is a measure of how interrelated two software components are. Cohesion is a measure of how related the functions performed by a software component are. We want to strive for low coupling and high cohesion. High coupling implies that, when we change a component, changes to other components are likely. Low cohesion implies difficulty in isolating the causes of errors or places to adapt to meet new requirements. Constantine and Yourdon even provided a simple-to-use way to measure the two concepts. Learn it. Use the measure to guide your design decisions.

You want:

• low coupling
• high cohesion

The image above is a special forces team that gives an alternative example of low coupling with high cohesion. The sniper team is another such example.

Design for Change

During software development, we regularly uncover errors, new requirements, or the results of earlier miscommunications. All these cause the design to change even before it is baselined. Which should be expected - change during development is inevitable. Bersoff, Henderson, and Siegel defined the first law of system engineering as,

No matter where you are in the system life cycle, the system will change, and the desire to change it will persist throughout the life cycle.

Furthermore, after baselining the design and delivering the product, even more new requirements will appear. All this means that you must select architectures, components, and specification techniques to accommodate major and incessant change.

To accommodate change, the design should be:

• Modular - composed of independent parts that can be easily upgraded or replaced with a minimum of impact on other parts.
• Portable - easily altered to accommodate new host machines and operating systems.
• Malleable - flexible to accommodate new requirements that had not been anticipated.
• Of minimal intellectual distance.
• Edsger Dijkstra defined intellectual distance as the distance between the real-world problem and the computerized solution to that problem. Richard Fairley argues that the smaller the intellectual distance, the easier it will be to maintain the software.
• Under intellectual control.
• A design is under intellectual control if it has been created and documented in a manner that enables its creators and maintainers to fully understand it.
• Such that it exhibits conceptual integrity.
• Conceptual integrity is an attribute of a quality design. It implies that a limited number of design "forms" are used and that they are used uniformly. When a design is complete, it should look as if one person created it all, even though it is the product of many devoted people.

* Image taken from http://www.greenville-home-remodeling.com/010-greenville-home-remodel-rare-design-before-and-after-kupersmith-front-elevation/. If you like it, check out the cool remodels they do.

Design for Maintenance

The largest post-design cost risk for non-software products is manufacturing. This makes design for manufacturability is a major design driver.

The largest post-design cost risk for software products is maintenance. Unfortunately, design for maintainability is not the standard for software. People often ask what software maintenance is and they'll joke that you don't have to grease code. Well, you don't have to literally grease code but figuratively greasing code to ensure that it is easy and quick to change does need to happen.

For example, at my workplace Apache Ant was a primary build tool. It has been replaced with Gradle and as a consequence when development teams have to work on a service that is built with Ant they complain. The maintenance work here would be to change the build code from ANT to Gradle to ensure that work is easy and quick. Of course, whether to make that change depends on whether that service is getting modified often enough to justify the cost. The architectural question is was this service designed in such a way that changing the build code to use a different build tool easy. We approached this by ensuring that the service's build code was loosely coupled to the service source code.

A designer has the responsibility to select an optimal software architecture to satisfy the requirements. Obviously, the appropriateness of this architecture will have a profound effect on system performance. However, the selection of this architecture also has a profound effect on the maintainability of the final product. Specifically, architecture selection is more significant than algorithms or code as far as its effect on maintainability.

Design for Errors

Errors in software are to be expected. Since you expect error you should make design decisions to optimize the likelihood that:

1. Errors are not introduced.
• Use code reviews to catch functional errors.
• Use static analysis to catch coding errors.
2. Errors that are introduced are easily detected.
• Ensure that your code is setup log unexpected errors.
• Use tools like AppDynamics to view the ranking of errors over different time frames.
• Use tools, such as, Splunk to graph errors by type over time. Especially, critical to do this before and after deploying to highlight new errors introduced by the latest deployment.
3. Errors that remain in the software after deployment are either noncritical or are compensated for during execution so that the error does not cause a disaster.
• This is all about testing that things work in a way that you want when they are broken. Here's some examples:
1. Database is down.
   Don't let your service return anything that makes clients think that their order was saved. You have to make the call based on your business if it is better to return error responses or just refuse to accept requests all together. Which of those could get you into more trouble? But don't just code for that. Actually kill your database in the test environment and test it. Setup a functional test that does this every time you build.
2. Disk is full and you can no longer log.
   Same thing as above. Setup a functional test that points the logs at a very small virtual drive and make this test happen with every build.
3. System encoding changed.
   Seriously, this is a thing. On Macs the default is utf-8, on Windows it is ascii, on Linux it is something. I have some Python 3 code that I originally wrote on a Mac but when I moved it to a Windows machine it choked on the encoding all because I didn't set an encoding. I was [unknowingly] depending on the system default. Servers are group property. Maybe only your OPS team can touch them but you probably have more than one Operations Engineer. Or your servers are configured via Chef or something else and everyone misses that single line where someone decided it would be good to explicitly define the system encoding. Boom! Code that was working fails for what most would consider a trivial bit of code.

Such robustness is not easy to incorporate into a design. Some of the ideas that help include the following:

1. Never fall out of a case statement. For example, if there are four possible values for a variable, don't check just for three and assume that the fourth is the only remaining possibility. Instead, assume the impossible; check for the fourth value and trap the error condition early.
2. Predict as many "impossible" conditions that you can and develop strategies for recovery.
3. To eliminate conditions that may cause disasters, do fault tree analysis for predictable unsafe conditions.

Tourism - Family 2016

Home - I've always loved it. I think it is pretty great and that it has a lot of potential to be better.

Design Principles

Design is the set of activities including:

1. defining an architecture for the software that satisfies the requirements
2. specifying an algorithm for each software component in the architecture

The architecture includes a specification of all the building blocks of the software, how they interface with each other, how they are composed of one another, and how copies of components are instantiated (made in memory and executed) and destroyed. The final product of design is a design specification.

How to Ease Growing Pains

As companies get bigger, even child companies (brands, if you prefer), there comes a time when the startup mentality might not be helpful. Here are five ways to make dealing with success easier.

1. Recognize that management is a skill.
   • As you have more development teams, it becomes harder to focus everyone toward the same goal. Good management does that.
2. Hire a COO.
   • It makes sense for young companies to emphasize product development and sales over operations. However, there comes a point when it is more important to hone operations and remove any and all sloppiness.
   • “Customers have to be sure you can deliver.” – Robert Shar
   • Midsized Companies Can’t Afford Operational Glitches
3. Get comfortable saying no.
   • Don’t take on every project that comes your way. Really think about the odds of success. Focus on what is important to your business.
4. Be focused on always getting better at what you already do.
   • New projects can be an unreliable route to growth.
   • “You have to get out of that mode of doing everything differently and turn to repetitive sets of tasks performed by people that become more and more expert in them.” – Derek Lidow
5. Understand that it is not do or die.
   • If you lack clear strategy then there is nothing wrong with focusing on what is already working for you.

The above is a ripoff from your-awkward-phase-and-why-you-should-love-it.

The Paradox of Success

The “paradox of success” can be summed up in four phases:

1. When we have clarity of purpose, it enables us to succeed at our endeavor.
2. When we have success, we gain a reputation as a “go to” person. We are presented with increased options and opportunities.
3. When we have increased options and opportunities, we have increased demands upon our time and energies, it leads to diffused efforts.
4. We become distracted from what would otherwise be our highest level of contribution. The effect of our success has been to undermine the very clarity that led to our success in the first place.

The pursuit of success can be a catalyst for failure.
===
Success can distract us from focusing on the essential things that produce success in the first place.

Software Architects Need Feedback

"Typically, [architects] are farmed out to various projects as consultants, with the aim of ensuring that the project takes off on the right track and avoids mistakes it might make without the wisdom of the consultants. The intent of this practice is laudable, but the outcome is usually sobering: because the consultants are so valuable, having given their advice, they are moved to the next project long before implementation is finished, let alone testing and delivery. By the time the consultants have moved on, any problems with their earlier sage advice are no longer their problems, but the problems of a project they have long since left behind. In other words, the consultants never get to live through the consequences of their own design decisions, which is a perfect way to breed them into incompetence. The way to keep designers sharp and honest is to make them eat their own dog food. Any process that deprives designers of that feedback is ultimately doomed to failure."

from API: Design Matters

Micro-Services: The Modern Modular Programming Paradigm

Modular programming is a software design technique that emphasizes separating the functionality of a program into independent, interchangeable modules, such that each contains everything necessary to execute only one aspect of the desired functionality.

As Service Oriented Architecture (SOA) has become popular the modular programming definition could easily be rewritten to read:

The micro-services style of service oriented architecture is a software design technique that emphasizes separating the functionality of a service into  independent, interchangeable services, such that each contains everything necessary to execute only one aspect of the desired functionality.

The term modular programming was coined in the late 1960s. It has been a key part of good software engineering since.

Here are some positive points about micro-services style of SOA:

• Small, easy to understand code base. Because micro-service application is responsible only for one thing, it requires less code, is easy to understand, is easy to reason about, is easy to test, and has less risk of changes.
• Easy to scale. With a large, monolithic application you have to scale everything together. You have to scale the whole thing and waste resources. Whereas, if demand for a particular service temporarily increased then you would only need to scale the specific service.
• Easy to Deploy. With monolithic applications even one line code change require redeployment and re-testing of the whole application. This could be a big problem for many organizations with its high degree of risk and disruption. Deploying micro-services is much simpler because the scope of deployment is much smaller. Plus you know where to look for problems if such arise.
• Ability to use a different technology stack. With micro-services the approach should be to use the best tools and languages for the job, instead of one size fit’s all. The same goes for databases. For example, recommendation micro-services can use neo4j and python because python has a lot of machine-learning libraries; event-processing micro-services may use java and cassandra because of multithreading properties of jvm and high scalability of cassandra. It also allow teams to try new technologies on small services without major disruption.
  Smaller teams. It’s much easier and faster to work with a collocated, small team. Each small team can own micro-service and access other services via high level api. For example, a team in New York can be responsible for recommendation services and a team in India for content micro-service.
• System resilience. If an monoliths application stop working, then a lot of functionality stop working. On the opposite side, If one of the micro-services stop working – only small, particular functionality will be lost. It’s much more simple to build some resilience around smaller service. For example, failure of the order-processing system can be mitigated with message queues.

Regression Testing - Testing Variability

When you need to test that a code change has made an improvement or, at least, didn't reduce the current level of performance, it is called regression testing. A level of statistical analysis is needed to ensure that perceived differences are not the result of random chance. The rigorous way to accomplish this is to use the Student's t-test to compare the results. The t-test can tell you the probability that a regression exists, but it doesn't tell you which regressions should be ignored and which must be pursued. However, provided the cost of running additional tests a cost-benefit analysis could tell you that.

Check out the Apache Commons Mathematics Library (TTest) for an implementation for Student's t-test.

Get To It

"Our great business in life is not to see what lies dimly at a distance, but to do what lies clearly at hand." - Thomas Carlyle

My sister, Joyce, and I write letters back and forth. It is a bit of a tradition that we include a quote either in the letter or on the back of the envelope. The above quote is what I included with my most recent letter. I picked it because all to often at work it seems that we spend so much time worrying about what may come along in the future that we don't do what lies clearly at hand.


If that seems too short sighted to anyone then I implore you to remember the popular quote of Lao-tzu. "A journey of a thousand miles begins with a single step."