2020 Update: Since writing this post, I’ve co-founded Foxhound Systems, a software development company that specializes in building software and training development teams on using functional programming. If you’re considering using Elm in production, have an existing Elm code base that you need help with, or are looking for specialized training for your team, we can help. Take a look at our Services for more details or email me directly at christian@foxhound.systems with questions.

At my job, we make a SaaS product used widely by both students and staff at university dorms across the united states. Our software provides an array of features that range from form-based tools, to email and text message broadcasting, to a central news feed that acts as both a communications tool and data aggregator.

The company was founded in 2013, and since its inception, a combination of plain JavaScript, jQuery, and an assortment of jQuery-esque libraries had been what the entire front end of the application was built with. In October of 2016, I realized that we were long past due for an upgrade. The straw that broke the camel’s back occurred when a feature we had worked on over the summer and released in August had already started feeling like a legacy application. Built as a single-page application (SPA) making very heavy use of Ajax calls to a JSON-based RESTful API as its back end, its 5,000 lines of front end code were already becoming very difficult to work with and modify, hardly a month after release. “When did this turn into jQuery spaghetti?” I thought.

Trying Elm

The search for a replacement front end framework had me considering React and Vue.js as the top candidates for several weeks. But I had a feeling that I should examine Elm more closely. I had read about Elm in the past and it had been showcased at a Haskell meetup I attended in Chicago. I was not by any means an advanced Haskeller when making the consideration to use Elm, but I knew that several years of writing small Haskell programs and reading about the language would have left me with knowledge transferable to Elm. It was also my belief in the benefits of statically typed functional programming that made the opportunity to at least try Elm too tempting to pass up.

I decided to follow the general strategy outlined in Evan Czaplicki’s How to Use Elm at Work post. One day in November of 2016 I set out to make a small internal tool that solved the problem of having no UI to configure a certain one of our features. The project took me about three full days to complete. The experience involved a lot of fighting against the compiler and resisting the feeling of being trapped by having to write in a functional style. By the end, I had written a bit over 500 lines of Elm. I realize that despite running into compiler errors often, and having to completely eschew techniques that were commonplace in imperative code, the constraints imposed by Elm were actually rather helpful. The compiler offered protection against silly mistakes, and functional code was easy to read and naturally highly composable. These were the types of benefits that I had read about when learning the fundamentals of Haskell.

Teaching Elm

A few weeks after finishing my little project, I introduced one of my engineers to Elm. He would be writing another internal tool that was quite a bit larger than mine. More importantly, he had no previous functional programming experience whatsoever. In order to be able to adopt Elm at our company, it was absolutely imperative that I would be able to get him productive in what was to him a completely foreign language rather quickly. In order to accomplish this, my approach was to:

1. Pair program with him on a significant portion of this initial project, but make him write most of the code
2. Emphasize the importance of reading type signatures, both in our own code and in any documentation we were referencing
3. Treat compiler errors as helpful feedback, rather than as a signal indicating failure
4. Practice approaching each problem with a functional mindset (e.g. “How can we apply List.map or List.filter rather than a for loop and array mutation?”)

This project took him a few weeks to complete. The end result was highly successful, and both he and I learned quite a bit about Elm as we worked on it together. Most importantly, by the end, my engineer was comfortable enough with Elm as to work independently for extended periods of time. The lesson had been a success, and it also proved that about a week of intense training, someone who has never written functional code can build a solid understanding of the basics of Elm. Within a month, they should be able to work independently on code that will eventually make it into production.

Work has gotten really interesting again.

Another important factor that emerged during this process was a human one: writing Elm code was both fun and interesting. My favorite quote from my new-to-Elm engineer during this process was the one above. Hearing this was not a top initial priority when looking for a new tool, but it was a very reassuring thing to hear. As the Chief Technology Officer of a company whose main focus is to build software, my responsibilities do not end at ensuring my team is productive. I view it as an obligation to ensure that each team member feels the importance of their work, engages in work that they have a personal affinity towards, and continually develops professionally. Simply writing code in Elm was immediately hitting two out of three of these goals.

Using Elm

With these two trials of Elm being very successful, I had all the evidence I needed: we were going to move forward with Elm. Our first user-facing application of Elm would come in short order. Without going into extensive detail, we spent the entire first half of 2017 making the largest and most complex feature that our software had ever seen: a highly customizable form-builder system with integrations to the rest of the data in our software.

With almost every single piece of data on each page in this feature being dynamic (questions, input types, order values, tags, answers, form template and submission edit histories, etc.), the need for managing all of this data effectively was paramount. Moreover, this data would have to be shared across multiple views seamlessly: an edit to a form template would have to be reflected in the corresponding form submission creation page immediately; a new form submission would have to be visible in the multiple tabular views in addition to its own individual page view.

Not only was the scope and complexity of this feature to be extremely broad, but it would also serve as a replacement for two existing features that were no longer up-to-par, and not worth updating. This would easily be the most high-stakes project we had ever undertaken.

With all of this in mind, I was quite convinced that Elm would be the best tool for the job. And so, we used Elm for the entire front end.

Released in early June, it is now over 22,000 lines of Elm code in the form of a single Elm application. Feedback from our summer users has been nothing short of glowing. I am certain that it was the decision to use Elm that make it possible to build such an intricate front end to such a high degree complexity without making any compromises in performance or reliability.

Reflecting on Elm

In the rest of this post I am going to outline what we have learned as we have used Elm; both its strengths and its weaknesses. This is not meant to be an Elm tutorial, but it is meant to inform someone with little-to-no knowledge of the language of its distinguishing features. Every point discussed below is aimed at addressing the experiences of using Elm in a production setting. That is, software that will likely be written by several people, that must be bug-free, performant, address a certain set of functionality requirements, and will see significant use by end users.

Elm has an incredibly powerful type system

Relative to other front end tools, Elm’s type system is its most distinct and powerful feature. Elm is statically typed, meaning all code is verified during the compilation process (more on that later). More importantly, Elm allows for the creation of Algebraic Data Types, which are referred to as Union Types in Elm. This allows the programmer to model much of the business logic of the application in type system, to be verified statically by the compiler, rather than in code that will be evaluated at runtime.

One simple example is that of a three-way state. Suppose I have a tag input field. When I arrive to the page with this input, certain tags may already be present. I can then edit the tag list, either by adding or removing tags. But here’s the catch: if I press “Cancel” to return to a previous view in the application, the tag list must revert to its original state. If I press “Save” the changes must be applied.

There are a number of ways to do this in JavaScript, one solution would be:

1. in addition to the master tag array for that field, create temporary arrays for added and for removed tags
2. for each tag change, apply it to the master array and keep a record of the change in the corresponding temporary array
   • if a tag is removed, remove it from the master array and add it to the removed array
   • if a tag is added, add it to the master array and to the added array
3. apply or revert the changes depending on the final user action
   • if the “Save” button is pressed, use the tags stored in both temporary arrays to permanently save the changes (e.g. http request to the back end)
   • if the “Cancel” button is pressed, use the temporary arrays to identify which tags to add back into or remove from the master array
4. clear the temporary arrays

In JavaScript, our solution relies on using several data structures to help with the bookkeeping of keeping track of which tags were added or removed. This might work, but I have to make many considerations in order to avoid possible errors. What if something is not re-initialized correctly after the user visits the page for a second time? Are we resetting them every time that we need to? How do we keep the tag list that the user sees in the DOM in sync with the state?

I chose this approach because there is no great way to model the current state of each tag in the tag itself. Attempting to do so might involve adding a status field containing a string that indicates one of the three possible states. Or worse yet, I could try to model this by juggling several boolean fields (e.g. added_status, removed_status).

Even the single-field approach likely to result in chains of if-statements that perform string comparisons in several places in the program. I will also have to ensure that my status field is always initialized with any tag object in order to protect against runtime errors. I could attempt to solve this latter problem by creating prototype functions that extend a Tag constructor object, but there is nothing forcing anyone to use these prototypal functions to create Tag objects to begin with. In short, adding such a field to my tag object is an encumbrance, as I have to remember to handle this extra field throughout my program.

By contrast, in Elm, modeling the possible states of each tag is incredibly easy. We can write this as follows:

-- The type representing the possible states of the tag
type TagState = Current | Added | Removed


-- The type representing a single tag
type alias Tag =
    { tagId : Int
    , tagName : String
    , tagStatus : TagState
    }

In the above example, every value of type Tag will contain a tagStatus field, which will contain a value of type TagState, which in turn has to be one of the three states I want to represent. The important thing to note here is that now every tag value must have a tagStatus field, and it must always contain one of the three defined TagState values. If this field is not initialized (e.g. at the JSON decoder for a tag) or its values are not handled exhaustively (e.g. in my view code), the program will not compile. I will show an example of the latter scenario below.

Elm has a great compiler

Elm’s compiler is what does the heavy lifting of enforcing the constraints of the type system, and I would consider it to be a huge asset when writing Elm code. Other statically typed languages have compilers, but Elm’s is in a league of its own.

Let’s consider the snippet of code in the section above. With the above types, every Tag in my application to always have a TagState. Let’s see how we would use this to our advantage to address the problem of keeping our DOM in sync with our data:

-- Function that takes a tag value and returns an html
-- value that will be rendered by the Elm program
displayTag : Tag -> Html Msg
displayTag tag =
    -- perform a match against the tagStatus field of the tag parameter
    case tag.tagStatus of

        Current ->
            -- Display a standard tag
            div [ class "tag" ] [ text tag.tagName ]

        Added ->
            -- Display a tag, but include the 'tag-added' class
            -- so that we can style these tags differently
            div [ class "tag tag-added" ] [ text tag.tagName ]

        Removed ->
            -- Display an empty div*
            div [] []

* Note that it would be possible to restructure this code in a way that does not display an empty <div> in the case of a removed tag, but the above code results in the clearest example.

But what if we had forgotten to handle one of our cases? Suppose we did not include the Removed case.

==================================== ERRORS ====================================

-- MISSING PATTERNS --------------------------------------------------- TagDisplay.elm

This `case` does not have branches for all possibilities.

72|>    case tag.tagStatus of
73|>        Current ->
74|>            div [ class "tag" ] [ text tag.tagName ]
75|>
76|>        Added ->
77|>            div [ class "tag tag-added" ] [ text tag.tagName ]

You need to account for the following values:

    Removed

Add a branch to cover this pattern!

If you are seeing this error for the first time, check out these hints:
<https://github.com/elm-lang/elm-compiler/blob/0.18.0/hints/missing-patterns.md>
The recommendations about wildcard patterns and `Debug.crash` are important!

Detected errors in 1 module.

The compiler tells us what the cause of the error is, exactly where it is, and what we need to do to fix it. The large majority of compiler errors in Elm are written in this manner. This is wildly helpful when dealing with something like potentially dozens of user-defined types, each with numerous possible values. This is particularly helpful when one developer might be editing another developer’s code: it is not necessary to look at the definition of the TagState type in order to safely edit code that relies on it; if I miss something the compiler will let me know.

There is one other key benefit that comes out of this combination of static typing a powerful compiler: Elm absolutely never encounters runtime exceptions. In a talk recorded in April of 2017, Richard Feldman from NoRedInk describes how the 100,000 lines of Elm code they have built up since 2015 have never thrown a runtime exception.

Writing in a functional style has significant productivity benefits

Everyone who knows about Elm knows that it is a functional language. However, I think that relatively few developers have written enough functional code to build an appreciation for just how pleasant reading and writing functional code is, and the productivity gains that come as a result. Here are some of the hallmark features of Elm code:

Pure Functions - Virtually all functions in Elm are considered ‘pure’. This means that given a set of parameters, a function will always produce the same result. Such functions also referred to as being referentially transparent, meaning they can be replaced with their corresponding return values without altering the behavior of the program. Because of the constraints that enforce this property, pure functions lack the ability to produce side effects (making HTTP requests, changing HTML on the page, printing output somewhere, etc.).

These traits combine to result in a significantly lower cognitive load required to read Elm code. For example, if you see a function whose type signature is Int -> Int, meaning that it takes one Int parameter and returns an Int value, you can safely assume that it will at most be doing some sort of numerical manipulation without any other side effects. If you are searching for code that validates email addresses, you know that you can look elsewhere (perhaps for a function that takes a String and returns a Bool).

Immutable Values - All values in Elm are immutable; they cannot be changed after they are set. This may seem limiting to someone coming from writing JavaScript, but in a functional paradigm, changing values is not necessary. The standard approach is to return a new value rather that overwrite an old one. Immutability eliminates a whole array of possible issues in a program, ranging from race conditions caused by concurrent code, to uncontrolled global state modification.

Higher-Order Functions - A higher order function is simply a function that takes another function as a parameter, or returns a function as its result. This style of programming is very common in Elm and leads to code that is well suited function composition, and in turn, reusability. Take a look at an example of the map function below (which takes a function and applies that function to every element in a list):

square : Int -> Int
square x = x * x

normalList = [1, 2, 3, 4, 5]

squaredList = List.map square normalList

In the above code, the square function is applied to every element in normalList. When this code is evaluated, squaredList will contain [1, 4, 9, 16, 25].

Pattern Matching - This is without a doubt one of the most useful features in Elm. Pattern matching allows you to to write code that will only get evaluated when the “shape” of the value being examined matches the defined pattern. Consider the following example:

type PermissionLevel = AdministratorPermissionLevel | StandardPermissionLevel

type UserGroup = AdministratorUserGroup | StandardUserGroup


-- Function checks whether a user can edit a post

checkIfUserCanEditPost : PermissionLevel -> UserGroup -> Bool
checkIfUserCanEditPost requiredPermissionLevel currentUserGroup =
    case (requiredPermissionLevel, currentUserGroup) of
        -- A standard user can edit a standard post
        (StandardPermissionLevel, StandardUserGroup) ->
            True

        -- An administrator can edit any post
        (_, AdministratorUserGroup) ->
            True

        -- Deny any other possible combination of values
        _ ->
            False

In the above code, we combine the requiredPermissionLevel and currentUserGroup parameters into a tuple in the case statement and evaluate them together as we try to match one of the cases. The _ value will match any value in the case statement. We use it to avoid having to define (StandardPermissionLevel, AdministratorUserGroup) and (AdministratorPermissionLevel, AdministratorUserGroup) as two separate cases. Instead, we tell compiler to produce code that evaluates to true anytime the user is an administrator. We also use the _ value in the final case statement as a catch-all, to deny all other possible combinations of required permission level and user group.

Elm’s Model-Update-View architecture is very well-suited for building web applications

The core architectural pattern in every Elm web application is what is referred to as The Elm Architecture, consisting of three main parts:

The Model is the state of the application. The Model consists of a single data structure that contains every piece of data used in the application. It will usually grow incrementally as an application grows in features and complexity, but how it is structured is up to the developer. In most applications the Model will take the form of a record type (similar to an object in JavaScript) which may have any number of top level fields that may be any type of data structure (including themselves being record types).

The Update is the portion of the application that handles both changes to the Model as well as any I/O that the Elm application has to perform (http requests, calling external JavaScript functions, etc.). No other portion of the program can change the state of the Model or perform I/O. The Update is called anytime a Msg value is produced in the application. Such a Msg value will usually represent the action and may have additional data bound to it (e.g. UserSearchInput "john").

The View is the portion of the program that renders HTML and handles user inputs. The View always takes the Model as an argument, so any conditional logic that uses data to dictate how the HTML on the page is changed during runtime must be based off of data in the Model. The View is automatically called by the Elm runtime anytime any value in the Model changes. User inputs in the View will produce Msg values, which will result in the Elm runtime invoking the Update.

So in general, the execution of an Elm program is as follows:

1. The Model enters a particular state
2. The View is rendered based off of the state of the Model
3. The user interacts with the application, a Msg value is produced
4. The Update is called, receiving the Msg value as a parameter, which results in a change to the Model (return to step 1)

This structure and the separation of concerns between the different portions of the application make it easy to both build and later refactor even extremely large applications. This structure also eliminates nearly all issues with data going out of sync with the DOM, or different DOM elements being out of sync from one-another, as the View will always re-render the DOM based off of the contents of the Model. To someone who is not used to using this type of architecture, seeing it in action for the first time may feel like magic. It is not uncommon to think “wow all I did was change the value in the model and all of the HTML that relies on that value updated automatically”.

Elm has a very powerful debugger

One of the features released in the latest version of Elm (0.18) is known throughout the Elm community as the Time Traveling Debugger. When opened, the debugger displays the current state of the program as well as the history of Msg values produced as the user has interacted with the program. When one of the older Msg values in the list is clicked, the entire Elm application will revert to that point in history in the execution of the program. This which will include the state at that point in time as well as the entire contents of the DOM. Clicking through the Msg list effectively allows one to replay the entire history of the current session.

What’s more is that this entire history can be exported as a JSON file. So a user can be asked to reproduce the steps that led to a bug, export the history, and send that history file to a developer that will fix the problem. The debugger is easily enabled via a --debug flag appended during compilation time, and requires no external tools or plugins.

Elm has a readable syntax

This may be a point of contention, but it is my opinion that Elm’s syntax (largely taken from Haskell), is very readable:

• It tends to be quite terse, with individual lines of Elm code being very short.
• It does away with unnecessary curly brackets and semicolons.
• Function parameters are delineated via spaces.
• Functions and variables start with lower case letters and are always written in camelCase.
• Types and module names always start with uppercase letters and are written in PascalCase.
• Elm code is indented only with spaces (killing the tabs vs. spaces debate).
• Indentation largely does not matter in most places to begin with (though most of the community uses a utility to auto-format their code, discussed below).

Elm has a great set of standardized development tools

The Elm community has developed a number of tools over time that remove many of the pain points of development.

• The standard elm binary comes with four primary utilities:
  • elm make - the compiler (which is also what enables the debugger via the --debug flag)
  • elm package - the package build tool; can be used to quickly install new packages or download all dependencies in an existing project
  • elm repl - a simple read-eval-print loop; a good way to test code and ideas in the command line
  • elm reactor - a simple web server and websocket-based live reload used for rudimentary development
• elm-format - a utility that auto-formats code to the community standards; most editors will have plugins to have to auto-run each time a file is saved
• elm-upgrade - a utility that helps automatically upgrade much of your code when a new version of Elm is released (more on this later)

The Elm community is very friendly and helpful

In my experiences, the following three communities are the best place for Elm help and discussion:

• /r/elm - The Elm Reddit community. A place for news and general discussion.
• Elm Slack - The Elm Slack team. A great place to go for help on a certain topic. There are always users active here, and in my experiences, skill levels range from total beginner to expert. I have been surprised to hear answers from people who obviously have very advanced knowledge of things like the Elm compiler; it is reassuring to know that language experts have a willingness to help Elm novices.
• Elm Discuss - The Elm Google Group. Like the other two, this community is very active, with discussions ranging from beginners asking for help, to discussions of very advanced topics, project proposals, etc.

Elm applications have excellent performance and additional optimization is easy

Although Elm is a very high level language, the JavaScript that the compiler produces is extremely fast.

In August of 2016, Elm 0.17 (last version of Elm at the time of this writing), even non-optimized Elm code was able to outperform React, Angular 2, and Ember.

In a different set of benchmarks posted in May of 2017, Elm’s performance was on par with React and Angular 4, which were all among the fastest frameworks in the benchmark (having a slowdown of 1.30 - 1.40 relative to the optimal vanilla JavaScript implementation of the benchmark). The fastest framework intended for production use, Inferno, scored 1.07.

In practice, runtime performance should rarely be a concern with Elm for normal web applications. In the event that additional optimizations are required, elm has two libraries which can increase the performance of the application.

• Html.Lazy - contains functions that can be used to cache the results of view functions, reducing the need to re-render certain elements.
• Html.Keyed - contains functions that can be used to optimize situations when elements are getting added, removed, or re-ordered (such as in a list)

Functions in both of the above modules are drop-in replacements for functions from the standard Html module, meaning no substantial rearchitecting of code is required to apply such functions.

Elm applications can be rather large when compiled

The amount of code produced by the Elm compiler can be somewhat lengthy. Our 22,000 line application compiles to a file with over 53,000 lines of JavaScript that is 1.6MB in size.

Fortunately, using the Google Closure Compiler with simple optimizations enabled, this file can be reduced to be a mere 450KB in size. A minor downside of using the GCC is that it is written in Java, so the Java Runtime is required to run it.

There is a version written in Node.js, but it is considered an experimental release, and in my experiences, this compiler is far slower on large JavaScript files such as the one in our case. I have a powerful Core i7 6700K @ 4.00Ghz on my Linux machine, and whereas the Java version takes 2-3 seconds to complete, the Node.js version takes 43 seconds. On substantially weaker hardware, like an EC2 instance used for a development or build environment, the JavaScript version would take several minutes to run.

The Elm core libraries do not yet have certain web API bindings, but interoperability with JavaScript code is safe and easy

Not everything is possible in native Elm code. For example, there is no official library for using the localStorage API, as the Elm API is still being developed. However, the good news is that Elm has very well thought out JavaScript interoperability. There are two primary methods for doing so:

• Ports can be used to talk to any JavaScript code outside of the Elm application during runtime. This technique can also be used to send data into Elm with functions that the Elm application object exposes.
• Flags can be used to set values in Elm during the initialization of the application. This may be used to set values passed to the browser from the web server that Elm must have access to, such as user configuration settings or authentication tokens. This technique would likely be used when Elm is being used in a full stack MVC framework (such as Ruby on Rails or Laravel).

A distinguishing feature of this interoperability model is that it ensures type safety and the integrity of the program. Even though an Elm application makes the use of flags and ports, it will still use compile-time type checking and never encounter runtime exceptions.

The Elm language and core libraries are prone to change as new versions of the compiler are released

This point may sound scarier than it is, but its implications should certainly be considered before using Elm for production use. Elm is currently on version 0.18, with a 0.19 release coming likely in the next few months. 0.17 was released in December 2016, right as we were finishing our second small Elm project. As a result, we experienced the process of upgrading Elm in our first two small applications. To give a summary of the experience:

• The process was well documented and relatively simple. There were several syntax changes, but most of this was handled by the elm-upgrade tool I mentioned above.
• There were a few changes to the way the HTTP library worked, which we had to spend some time on. This portion of the process likely took a lot longer than it would today, as we were so new to Elm.
• We had to wait to upgrade until all of our dependency libraries were upgraded as well.

I will add that this last point may have huge potential ramifications in a production application. I have been somewhat reserved in using external libraries in our application, particularly avoiding ones with large dependency trees or a lot of code. I can foresee a scenario where some dependency used by your application gets abandoned, and your team may have to support it. Certainly this is a risk with any library in any language, but the Elm compiler will refuse to even attempt to compile your application unless every dependency in your project supports the current version.

If such an abandonment scenario takes place with single module library that has 150 lines of code, maintaining that library will likely be quite straightforward. It may not even require any changes, short of a version bump in the package file. But I would stay far away from a library like elm-mdl, a material design implementation in Elm, which contains 10,000 lines of Elm code. If your application becomes highly dependent on such a library, and the maintainer stops supporting it, you will have to make the decision between forking the library and maintaining it on your own, or never upgrading to the newest version of Elm.

In Elm, doing what might seem hard can actually be quite easy, and the inverse is also true

All of Elm’s unique features often come together to produce a language that often flips the definitions of ‘easy task’ and ‘difficult task’ on their heads.

• Want to add a new possible value to one of your union types, which will necessitate changes to the code in dozens of places in your application? Easy, just start by changing your type definition and the compiler errors will help you find everywhere that needs updating.

• Want to decode some JSON? Hard, especially if the JSON is heavily nested and it must be decoded to custom types defined in your application. Doing this will require an understanding of how JSON decoding works in Elm and will usually result in quite a bit of code (our application contains over 900 lines of JSON decoders alone).

• Want to create multiple different views in your application that each have complex data interdependencies? Easy, The Elm Architecture’s Model-View relationship make this type of thing almost trivial to do. It will also be almost impossible for the data to get out of sync across different views.

• Want to measure the height of an element on the page at the moment a user clicks on it? Hard, in order to do this we had to make heavy use of event bubbling and writing JSON decoders for the native event.target object that is produced by an onclick event.

Generally speaking, however, the trade off is worth it. The easy tasks that become difficult usually do so because you gain some sort of benefit (such as type safety) as you do them in Elm. The difficult tasks that become easy usually result in massive productivity and reliability gains, particularly as an Elm application reaches large sizes. Our 22,000 line Elm application is easier to refactor than our 5,000 line jQuery-based application by a wide margin. The Elm application will age well, only becoming more reliable and performant as we make improvements and add features over time. The jQuery-based one will not, and will be slated for replacement when its limitations become too prominent.

Conclusion

Using Elm in production has been a very successful endeavor at our company. Our latest project, with a front end written solely in Elm, has exceeded all expectations, both those of our users as well as our own. We have managed to take a set of functionality that would have been exceptionally difficult to build using our old methods, and using the strengths of the Elm language and architecture, successfully developed the largest feature in our entire software product to date. All of this done with a very high degree of maintainability and reliability. This post has been a record of our experiences with Elm up to this point.

The decision to use Elm for the first time was difficult due to the risks associated with the unknowns that would come with a departure from normalcy. The decision to continue using Elm will not be.

Software in an omnipresent aspect in the world of today. The vast majority of people today have a continuous interaction with software. Nearly every moment of every day for most is in some way reliant on code somebody has written. For this large majority of people, software is a sine qua non.

But the minority of people who create software that sees use by others know that building it is remarkably difficult. Making it comes with obstacles that are unique to it. In this post I will describe a few of these defining challenges.

1. Seemingly insignificant details are often crucial

In software, what seems inconsequential will often be just the opposite. Perhaps the most famous example of this is that of the Ariane 5 rocket, whose 1996 maiden flight resulted in it veering off course and exploding less than a minute after takeoff. The cause of this issue was an improper conversion of a floating point number to an integer somewhere in the rocket’s guidance software. A single line of code that resulted in a $370 million mistake.

More recent issues in other industries have occurred as well. In 2012, the trading firm Knight Capital lost $440 million in 45 minutes because someone forgot to remove an unintended block of code from one of their eight trading servers. Just a few weeks ago, in February of this year, an engineer at Amazon Web Services doing server maintenance typed a command with a few erroneous parameters, resulting in thousands of websites and other services going offline for several hours. Dozens of well known organizations such as Expedia, Kickstarter, and U.S. Securities and Exchange Comission were affected.

High profile examples such as these get people’s attention, but they also serve to demonstrate that seemingly minor mistakes can and do result in critical software failures. When building software, the details matter.

2. Software complexity grows exponentially

Let’s play a game. I am going to give you a certain number of coins and you need to tell me how many possible arrangements of “heads” and “tails” there are. For example, if I give you one coin, there are obviously only two arrangements, one for each face of the coin. If I give you two coins, there are four possible arrangements…

Three coins? Let’s see…

That’s eight possible arrangements. Now, how many coins would I have to give you for you to make at least 1,000 arrangements? Have a guess in mind? The answer: 10 coins. The formula for calculating this is:

How many coins for at least a million arrangements? Turns out it’s only 20.

Going from 10 to 20 coins is a thousand-fold increase in the number of possible arrangements. Why is this the case? A single coin only has two possibilities, but adding it to a set of coins results in a doubling of the number of possibilities for the entire set. This is an exponential rate of increase.

Software complexity grows in much the same way. When creating a piece of software, making a small addition can be rather simple. But it can result in an exponential increase in complexity on the entire program. Suppose you have a settings page with five settings, each setting having two possible states. That’s 32 possible configurations. Adding a sixth setting will result in 64 possible setting configurations. Each one of these configurations must be tested to ensure everything functions as intended.

This rapid growth in complexity forces developers to be careful when designing large applications. Those that are not often pay the price in the form of buggy or generally unreliable code that can be very difficult and time consuming to work with.

3. Creating the user interface is just as difficult as writing the rest of the code

A key component of building most software is creating its user interface. Even after all much of the issues with underlying complexity and architectural details have been addressed, making an application that is usable by people requires designing and building an interface that allows for interacting with the software.

User interfaces are software that must be built much in the same way as the rest of an application, but they have the unique requirement of being intuitive and usually visually appealing. Making them requires not only technical ability, but also knowledge of design principles, as well as an intimate understanding of the types of users that will be consuming the software. Making a health insurance website for people over the age of 50 will require a different design approach than a social media application intended to be used by college students. Both will certainly require design expertise.

User interfaces also take a very long time to make. The end user will never see the long hours that go into interating over designs, perfecting layouts, and deciding on things like color, spacing, and numerous other details that can completely change an end user’s experience. Doing this requires a lot of time, regardless of how experienced a software developer is at making user interfaces.

The process of creating a user interface can be compared to drawing a portrait. Most people can recognize a well drawn portrait. But few people will be able to draw one themselves. Those that will be able to will have had years of practice, and the process will still require time and patience. Software developers that make user interfaces, even if they do it by writing code, require a similar progression of skill development that can take years to achieve mastery in.

4. Nobody really knows how long building something will take

We are going to go on a road trip from New York City to San Francisco. Let’s pull out our United States road atlas and plan the route we will take. We trace a path following Interstate 80 westbound from New York to our destination. Looking at the map key, we try to estimate the distance; about 3,000 miles.

Assuming that we will average 75 miles per hour, we will be driving for 40 hours. That’s quite a bit further than we can comfortably drive nonstop, so let’s plan on spending a night in a motel somewhere in the middle of our trip. This should take about 10 hours. Let’s also not forget that we will have to stop for gas several times. Assuming 6 stops of 10 minutes each, we will add an hour to our trip. So our whole trip will take 51 hours in total.

But in reality, we will come to discover that this estimate fails to account for numerous delays that only become known to us over the course of our trip. Starting the trip during Monday morning rush hour costs us several hours. An accident that causes traffic in Pennsylvania does the same. Our stop in a motel in Illinois ends up taking 13 hours rather than 10. We get pulled over for speeding in Nebraska. We need to stop at a mechanic shop in Wyoming because our car’s oil level is far too low. We end up spending a second night in a motel, because it turns out that driving 1,500 miles per day is only feasible in theory. We get caught in rush hour traffic again in Sacramento.

So how long does our trip take? By the time we check the time in San Francisco, we find that it has been 76 hours since we have left New York City! Our initial estimate of 51 hours was off by nearly 50%. We did not account for numerous factors, from an extra motel stay to getting pulled over, to being stuck in traffic several times. But how could we? Most of these only became apparent once we were driving.

Estimating the length of a software project is in many ways like planning our road trip. But whereas the road trip is measured in hours or days, a software project is measured in weeks or months. Even after spending several days planning a large piece of software, issues will arise over the course of development. Problems will emerge only as certain details are decided and small issues are resolved. This cycle of discovery and resolution of problems is an inherent trait of software development. This is also exactly why a component that is slated to take two weeks may end up taking five. A project planned to take 16 weeks can end up spaning 28.

To complicate matters further, the cause of a delay may have nothing to do with the project itself. A bug in an unrelated feature may require several hours of attention in order to fix. A completely unrelated meeting may end up taking an afternoon. Small delays over a period of months can add up to weeks. When planning a large piece of software, the best we can do is have rough guess as to how long building it will take. It is prudent to realize that any such guess is subject to change.

5. Sometimes the best solution is to delete something

When a physical object gets worn out and need to be discarded or replaced, the need to do so ends up being quite apparent. A bicycle tire gets worn down to its innertube, causing a flat. A computer’s motherboard fails after several years, preventing it from turning on. A car’s transmission blows out after it has driven 350,000 miles and it will no longer start. Even a derelict apartment building may need to be demolished as renovating it may not be economical.

Software is different. Software never gets worn out. A piece of code running for the first time is no different than when it runs for the billionth time. As long as the underlying hardware is functional, code can continue to run ad infinitum. This is where one of the major challenges of software development occurs. Sometimes the best solution is to delete something.

The need for this type of deletion of software can manifest itself in many ways. Perhaps there is an old feature that is only utilized by a small minority, yet those that do use it frequently report issues. Other times there may be a large component that sees very heavy use, yet it is poorly built and updating it to meet user requests does not make sense. In such a case, building an improved feature that will replace the old one may make sense.

In all cases, deletion is challenging. Users may get frustrated and annoyed. In the case of a replacement, new documentation has to be created. The time spent making the old component may seem wasted. Or in some cases, the condemned component interacts with several others, so there will be work involved in ensuring that the peripheral components continue to function without it.

But at the same time, the consequences for not deleting something can be worse than pruning it. The time spent on supporting an old piece of software can end up amounting to more than building an improved replacement. Slow and buggy features may drive users to give up on using a feature altogether. An old feature may be a glutton for system resources, slowing the entire application down for everybody. In such cases, the benefits of removal often outweigh the costs.

In this post, I am going to share a few of the key elements that make a software development team effective and enable the type of growth mentioned above. I will discuss our drive to innovate, our emphasis on learning and education, and the essential tools and techniques of our software development methodology.

Innovation

Perhaps the most fundamental aspect of our software development team is our drive to innovate. We continuously seek out new techniques, strive for a more performant and better organized architecture, and develop more effective ways to write clean and reusable code. These measures ultimately help us build features that have fewer bugs, take less time to build, and provide a better user experience.

One specific example of the application of such innovation is in the design of our user interfaces. Newer features are created with painstaking effort to ensure their interfaces are clean and intuitive while giving users the ability to perform their intended actions as easily as possible.

The design of our new features begins with planning a page layout. Here, we must determine the overall structure of the page—making decisions about where components such as user inputs and relevant data will be. This may involve asking ourselves questions like:

• Should this input be a dropdown or a search bar?
• Should the data be presented as a list or a more interactive table?
• Which pieces of data are relevant to someone who will be using this feature?

Once a layout is determined, we make use of things like animation, color, and icons to provide visual cues to the user about where to click, where to focus their attention, and feedback as to whether an action is completed successfully or not. This entire process of developing a layout and then adding styling and animation can be iterated over several times before a design is deemed suitable for the final product.

The results of this extra effort are evident. Our newest features are very heavily used, yet they receive the smallest number of related support requests.

Another more technical example of our innovation is in the front end architecture of our newest features. Every major feature we have released this year has been what can be referred to as a “single page application” (SPA). What this means is that once a user arrives at one of these features, either through sidebar navigation or via direct link, their browser will download all of the data necessary to perform every function of the feature.

Although this early loading of the data may seem unnecessary, the result is a better user experience. For example, when searching for users to tag in a dynamic form field, the search results in an SPA will always appear instantaneously, regardless of whether the search is being performed at an institution with a few hundred or several thousand users.

This new approach differs significantly from the one utilized in our older features, where each search query was sent to the server and then used to perform a database lookup. This often resulted in noticeable delay between search input and results display, and would be most apparent when the list of results was very long.

Other benefits of an SPA, such as smoother and faster “page” navigation (SPAs can give the illusion of having multiple pages), lead to a better experience as well.

Education

Another major aspect of the success of our development team is a focus on education. Software development requires substantial breadth and depth of knowledge spanning many distinct skill sets. And while everyone on our development team has an academic background in Computer Science, this is not sufficient. Although there is no better background for software engineering than a degree in Computer Science, most curriculums cover only a small portion of the skills required for even rudimentary software development. Continuing to learn after graduation is a necessity.

Our emphasis on learning begins immediately. Each new team member is given small tasks that will build their skills as a productive software engineer. As time goes on, larger projects require a composition of the skills that are learned early on. Even interns begin working independently within a few days of starting their internships. There is no better way to learn software development than to be actively engaged in it.

Within the first three weeks of my internship with the company, I was able to learn more than I have ever learned in an entire year of schooling.

One other component of education that goes beyond building knowledge is developing understanding. Everyone on the development team is challenged to not only learn how to build software, but to also understand why it is built the way it is. This may range from understanding why a single line of code works the way it does, to why it is better to use one database schema over another, to why architectural decisions that initially seem appropriate may be problematic in the long run.

Building an understanding of why involves asking questions and critical thinking. Everyone on our development team is encouraged to scrutinize their own work. This may involve asking questions like: “Why is this solution the best one?” or “Will this cause any problems when we scale this feature?” Other times this will involve taking the time to explain a topic in detail, often beyond what is necessary for the task at hand. Through questioning and examination, everyone learns not only the hows, but also the whys of building software.

This understanding of why is vital for long term growth as a software engineer. Understanding why transcends individual projects and programming languages. The piece of software being worked on may be different, or the syntax of the code might be new, but the fundamental principles seldom change. Knowledge of why is also the essence of the distinction between familiarity and expertise in a given topic.

Methodology

A third key aspect of the success of our development team is is our methodology. This ranges from the tools we use to help us stay organized, to the process our code goes through before it reaches our production servers.

One of the most important tools we use while building software is called “version control.” Version control allows us to keep track of every single change ever made in our code in a central repository. Should a feature break, or should we simply need to refer to how something was written previously, version control allows us to browse our entire project history.

Another feature of our version control system that is absolutely crucial to effective software development is called “branching.” This allows each engineer to create their own “branch” in the code repository, which they can then use to build new features without interfering with other developers. More importantly, branching allows us to build features and make changes without breaking the production website. Once a feature or set of changes are completed, a branch can be seamlessly merged back into the main line of code.

A component of our software development methodology that goes beyond use of tools is our code review process. This is a procedure that we undergo each time a set of code changes are completed. Here, the author of the code will sit with at least one other engineer (the ‘reviewer’) and walk them through their changes. The reviewer will provide feedback, ask questions, and point out any issues with the code that the author is submitting.

This process may be somewhat daunting for new engineers, as their work is examined to a degree of closeness that is not common outside of software development, but any feelings of anxiety quickly turn into confidence with the right approach. The purpose of a code review is to educate the author as well as any reviewers, to catch bugs early on, and to improve code quality. With a reassuring attitude from the reviewers, code review becomes a collaborative effort to the benefit of everyone involved.

Continual Improvement

The aspects of a software development team I described in this post are key contributors to its success. However, if there is one additional point to end this post on, it is to mention that we are continually seeking to improve each of these aspects of our team.

This quarter, we have begun to put additional focus on the deliberate learning of specific technologies and concepts, rather than just a broad improvement of software development skill. Just last week we developed a plan for a better code review process, one that will allow for more reviews of smaller segments of code rather than lengthy reviews that had only come at the end of projects up until now. And even at this very moment, we are in the process of evaluating new tools to improve the way in which we build our user interfaces with the goal of producing code that is more terse and comprehensible.

As our product and our company continue to grow and develop, I look forward to the opportunity to continue to innovate in our software, to overcome any challenges we may face, and to continue to build a team of effective software engineers.

What is website encryption?

As of today the pages on this site are being served up over SSL. This means that the sever this site is hosted on is encrypting each webpage before it serves it to your browser. Encryption makes the data unreadable to third parties, and also ensures that the data that you see has not been tampered with.

An encrypted website can be identified by the https:// as opposed to the http:// at the beginning of the address bar, and a (usually green) lock icon just to the left of the actual address text. The state of the lock icon also serves to warn against possible faults in the encryption. In Google Chrome, a red lock icon and https being stricken out indicates a serious fault in the SSL on the site (either the encryption is broken or the identity of the host has not been verified). An orange lock icon is less severe, and typically indicates that domain may be encrypted but some of the elements on the page are not.

As a rule of thumb, if the connection is not in the “green” status, you should never enter sensitive data (credit card numbers, SSNs, personal info) anywhere on the site. Other browsers such as Firefox may not adhere to this same color system for indicating whether SSL is working on a site, but you can always click the lock icon for more information about the encryption status. This should be true for all browsers on all operating systems.

How is this site encrypted?

Quite a while ago, an organization called Let’s Encrypt was created to serve as a new certificate authority. A certificate authority serves as a trusted, unbiased organization that independently verifies the authenticity of servers that want to serve encrypted web content to a browser.

Before Let’s Encrypt, a website owner had to pay money to a certificate authority like DigiCert, Comodo, or Symantec in order to get an SSL certificate. Prices of SSL certificates from these authorities are in my opinion very expensive. The cheapest one I can see is a single domain certificate from Comodo for $76 per year, ranging all the way up to a wildcard certificate (covering all of a website’s subdomains) from Symantec for a whopping $1,999 per year! For a website like this one, paying over $6 a month just to serve my visitors lightly styled HTML pages over SSL is a waste of money.

But this is where Let’s Encrypt comes back into the picture. Let’s Encrypt is a free service that is the product of a collaboration between organizations like Mozilla, Akami, Cisco, and the EFF that aims to bring free SSL certificates to the entire internet. What’s more is that Let’s Encrypt is a totally automated service, not requiring any human intervention on behalf of the certificate authority to issue a certificate for a particular host.

A couple weeks ago, I signed up to request an invite to the Let’s Encrypt beta program. The opportunity to serve my site over SSL without having to pay a ton of money (in this case, for nothing other than a bit of my time) was a good prospect to me. I did not hesitate to submit this site to participate in the beta program. Today, I got an invite.

Encryption Steps

I will not go into much detail into how the actual SSL setup process, because there really was not much to it. Anyone with a moderate amount of Linux experience should be able to follow the instructions and complete the process within 20 minutes. The email I received inviting me to the beta program contained most of the instructions for setting up SSL. In short, the steps were like this:

1. Clone the letsencrypt client (a command line tool written in Python) from the Let’s Encrypt GitHub repo
2. cd into the cloned repo and run a script with ./letsencrypt and a few option flags (provided in the email)
3. Follow the wizard and select the domain that is to be served over SSL (based off of available Apache/Nginx configurations)
4. Add a redirect from HTTP to HTTPS in my /etc/apache2/sites-enabled/ configuration file for this domain
5. Ensure that none of the assets (like Disqus comments) on my site are being served over HTTP (to avoid the “orange lock” warning mentioned earlier)

Why Encrypt?

The lingering question here is quite simple: Why bother encrypting? This is just a blog website that has a relatively small amount of content. The most sensitive piece of information here is probably my email address, and the biggest risk associated with that is my inbox getting hit with spam. It is not like encryption is doing anything to mitigate that either. It would appear as though me taking any time at all to encrypt this site is utterly pointless.

But I believe that there is a reason. I am a proponent of encryption becoming a de facto standard for how we share data over the internet. Encrypted data should not just be the content your bank website sends you when you check your account balance. Nor should it be just your payment gateway connection when you click “Checkout with PayPal” on a store’s site. I believe that everything should be encrypted, all of the time. That means all communication, all transactions, all entertainment, and all information access in general.

When encryption is the standard, the discrepancy between important encrypted data (like your medical records on your hospital’s website) and unimportant data (like this blog post) vanishes. By encrypting everything, important encrypted content becomes obscured by all of the unimportant encrypted content. This makes it very difficult for a malicious entity to target important encrypted data. An attacker may waste time and money attempting to decrypt data exchanged between your computer and some server, only to find that the data is a sad pepe meme, for example.

The other component of my belief for encrypting everything is simply from a privacy standpoint. I believe that nobody should have the ability to track what you do on the internet, and making encryption a standard is perhaps the most important thing we can do to protect against that. This is particularly important for protecting people who are not tech savvy, and who are most likely to have their information compromised or make dangerous decisions on the internet.

Fortunately, this belief is not novel. Companies like Google and Apple are moving rapidly towards an encryption by default standard. Brucer Schneier also made a good post about the importance of encrypting everything. And as of next month, Let’s Encrypt will go public and anyone will be able to use the service to serve encrypted versions of their own website.

The bottom line is that while the decision to encrypt my website looked at in isolation seems pointless, looking at it as part of growing standard of encrypting everything paints a very different picture.

In March of this year, Google released the new Chromebook Pixel. Having used ChromeOS for most of a year on the Acer C720 Chromebook and very much enjoying it, I did not hesitate to retire the C720 and buy the Chromebook Pixel LS. This “Ludicrous Speed” version of the Pixel comes with a Core i7 processor, 16 GB of RAM, and a 64 GB SSD. Both versions of the new Pixel sport a fantastic 2560 x 1700 pixel resolution screen (which has a unique 3:2 aspect ratio), a great keyboard, touchpad, and a 12 hour battery life. Each Pixel also has two USB-C ports, which can be used for charging, data transfer, display and audio output, and even ethernet.

The Chromebook Pixel LS is a fantastic device, with very fairly priced hardware. I have absolutely no complaints about the build quality and hardware features at all, although I will admit that I would have happily paid a bit more for a 128 GB SSD. But this post is not a Chromebook Pixel LS review. You can find hundreds of those all over the internet. Instead, I am going to talk about my software development environment and workflow on the Pixel. After using it on a daily basis for nearly 3 months now, I can share the experience of using the Pixel for this purpose with detail.

Environment Setup

When I used the C720, I did all of my software development in Linux via crouton. This allowed me to run Linux side-by-side with ChromeOS, and switch between the two seamlessly. In Linux, I was able to do all the things I would normally do: install packages, run daemons, host a local MySQL and Apache server, etc.

Going into the purchase of the Chromebook Pixel, my research revealed that it was actually quite possible to do all of my software development without relying on Linux through crouton. I will discuss the motivations for this decision later on. But doing this required a bit of a setup. I will walk through everything I did in the first few weeks of using the Chromebook Pixel.

Developer Mode

Putting the Chromebook into developer mode is the first step towards getting the most out of it. Developer mode primarily allows for access to a full featured Linux shell (via the shell command) in the terminal. This is necessary for the successful execution of the next few items on this list.

To put the Chromebook in developer mode:

1. Hold down the ESC and Refresh (F3) keys.
2. While holding the keys, press the Power button. This will cause the Chromebook to reboot.
3. When you are met with the white recovery screen, press Ctrl-D and follow the instructions.

That is all there is to it. But it is worth noting that this process clears everything stored locally on your Chromebook. Any important files should be backed up prior to putting a Chromebook into developer mode.

dev_install

Once the Chromebook is in developer mode, you can install the dev_install script, which gives access to a number of useful packages aimed at Chromium OS developers. These are tools that you would find on most Linux systems. My purpose for running this script was to get access to sshfs, but since installation takes up a bit of SSD space, it is worth mentioning that there is now a Chrome app that acts as an alternative to this process (described in the next section).

To run the script:

1. Open a terminal with Ctrl+Alt+T.
2. Type shell to go into the true Linux shell (which is what developer mode enabled).
3. Type dev_install. This begins the install process and prompts the user with a few questions.

Once this is complete, the emerge command becomes available, which can be used to install the various aforementioned packages. Unfortunately, emerge does not give access to all the packages one would expect to be able to install (it is no apt-get or pacman), but it can be useful.

sshfs / sftp

sshfs is a utility that allows an remote filesystem to be mounted in a local directory. Simply put, you can take a directory that is on some server somewhere and turn it into a ChromeOS folder. Doing this allows you to manipulate all of the files in said folder with any ChromeOS applications.

sshfs can be installed with the emerge command. Installing the sshfs-fuse package will enable the sshfs command. The syntax for its usage is

sshfs [user]@[host-ip-address]:[/path/to/directory] [/path/to/mountpoint]

There are a number of other flags you can specify (such as -p for port). I found that the command would give me an error if I used the host’s domain name rather than IP address, but your mileage may vary.

The other way to mount a remote directory is through the SFTP File System Chrome app. I have not personally tested this functionality, but it looks like this is actually a great potential alternative to the need for developer mode and dev_install altogether. I discovered this app thanks to a recommendation from the ChromeOS subreddit.

Caret

Caret is an awesome Sublime-like text editor for ChromeOS. It is a Chrome app, so it gets its own dedicated window and shares a large number of Sublime’s features. Used in conjunction with sshfs or sftp, Caret can be used to edit remote files on any server. Make sure to set the disableReload option to true in the Caret configuration file to prevent the cursor from bouncing to the top of the file every few seconds (which happens only when editing remote files).

Chrome MySQL Admin

Chrome MySQL Admin is a lightweight MySQL Chrome app that is very easy to use. It is not quite as powerful as MySQL workbench or even phpMyadmin, but it gets the job done for database servers that do not have phpMyadmin installed.

One of the great aspects of Chrome MySQL Admin which I have not seen anywhere else is that the app will sync your saved database connections to Chrome. This means that if you access the app on another computer later on, your connections are available to you as long as you are logged into Chrome with your Google account. If security is an issue, simply do not add a connection to the app’s favorites list or do not log into your Google account on a computer where it may be dangerous to do so.

crouton

crouton, mentioned earlier, allows for the installation of Linux into a chroot side-by-side with ChromeOS. crouton allows for the installation of various operating systems, including a few versions of Ubuntu, Kali Linux, Debian, etc. There is also chroagh which is a fork of crouton made specifically to cater to Arch Linux.

When installed, crouton can be accessed via the ChromeOS terminal directly in a tab using only a command line interface, as well as with a graphical desktop environment (Xfce being the default). You can configure crouton to start its own window session and then switch between ChromeOS and Linux using keyboard shortcuts, or as of a few months ago, you can now use the xiwi target in your installation to make the Linux GUI run inside of a resizeable (and full-screenable) ChromeOS window.

Even though I am not using crouton for most of my development, there are certain situations where having acess to full blown Linux is useful. I consider my crouton installation a sort of emergency backup that only gets used once in a while when I need access to specific tools locally.

Vim

Vim is certainly not directly related to ChromeOS, but its use in my workflow has become very significant. Vim is a lightweight and very old command line based text editor that has a very strong following amongst a large number of software developers. Vim also has a significant learning curve. I will discuss the role that Vim plays in my development setup more later.

tmux

Where Vim is the yin, tmux is the yang. tmux (short for “Terminal Multiplexer”) is a utility that allows for the splitting of a terminal into multiple panes. This may seem pointless if you are using a local machine with a desktop environment where you can just open as many terminal windows as you need, but it becomes a necessity when working over ssh. Overall, tmux is a significant time saver to anyone who does a lot of command line work.

Environment Decisions

As I said towards the beginning of this post, my goal was to do development on the Pixel without using crouton, and only having it installed as a sort of emergency backup. I had a few reasons for this:

• While crouton is usually reliable, experience has shown that silent updates to ChromeOS will break functionality that crouton depends on. Fixing these issues is usually as simple as updating crouton, but sometimes it may take a few days for the update to actually be released by the crouton developers.
• Configuration of all of the various daemons and servers to replicate the actual application I am developing to a local environment is a Sisyphean task. In the past year, due to various operating system reinstalls and even an SSD failure, I have to reinstall the application infrastructure at least five separate times. The arrival of the Pixel prompted me to move this infrastructure to a remote development server and I am fairly confident I will never have to go through this process again.
• Actually running all of the daemons and servers mentioned above as local applications takes up system resources. The Chromebook Pixel LS can absolutely handle doing this, but with how easy it is to avoid doing this, it is a better option to outsource this work. By not having to run things like a MySQL database, a Redis instance, a NodeJS server and an Apache server locally, I can preserve the Pixel’s CPU, RAM, and particularly the SSD space for other things that cannot be offloaded. Just as importantly, a reduction in load on the CPU means a longer battery life.
• The super high resolution screen of the Chromebook Pixel is not handled very well by desktop environments like Xfce. Text in many panels and tabs ends up being absolutely tiny and the interface overall does not scale very well. Changing configuration can fix a few things but it is quite obvious that the environment is not yet ready to handle such a screen perfectly. This problem could be avoided by using crouton in command line mode only straight out of the ChromeOS shell, but at that point the usage becomes indistinguishable from using a remote server.

Looking at these points, we can see that there is no single overwhelming reason not to use a local instance of Linux via crouton to do development on the Pixel. But in aggregate, these points add up to make a solid case against using doing so. What’s more is that they also build a case against using a local development application setup in general, regardless of operating system. Offloading the replication of the development application to a remote server seems like a better option all around, whenever doing so is possible. This is something that I had not really thought so much about until I was faced with this decision around the time I bought the Pixel.

Evolution of the Workflow

So the question is, what do I actually do? I have listed my tools and I just described what I opted not to do. Now, onto what I actually do use. To begin with, my actual development application is hosted on a remote Linux server. This server is an Ubuntu 14.04 VPN that replicates our production servers but on a much smaller scale.

My initial plans for using the Pixel included strongly relying on sshfs and Caret to do my development. That is, I would mount the remote directory that contained the project source code using sshfs, and then use Caret for the actual editing. This was the most appealing option initially, particularly because I had been coming from using Sublime.

I found that using Caret was okay, but not perfect. The main reason for this was the fact that I was editing a rather large project (hundreds of files with well over 100k lines of code) remotely. Loading all of these files as a project into Caret took a substantial amount of time (longer than even the slowest of IDEs opening), and I had to go through this process every time I would remount the remote directory, which was at least once a day. I also experienced minor synchronization issues with the files themselves and the file tree displayed in the Caret sidebar.

At this point I realized that I had a few options. I could either deal with these minor issues and trudge onwards with Caret, or I could attempt a few fixes by making permutations of the same overall setup (such as using sftp instead of sshfs, and seeing if that changed anything). I also had the option to do all of the work using a different approach altogether. The option of using Vim was staring me in the face.

I was somewhat apprehensive about switching to using Vim initially, as I knew it would probably slow me down to begin with. This proved to be an unnecessary concern. The basic knowledge I had picked up of Vim from using it in college (knowing things like normal vs insert mode, and commands like dd, yy, p, o, cw, :##, and :wq) was sufficient to get started. As time passed, I read a few tutorials and gradually added new commands to my regular usage. As this happened, my .vimrc file also grew. Using tmux also was a challenge at first, as its key bindings are somewhat unintuitive. But with a bit of looking at reference material, and through sheer persistence, I was able to become comfortable with Vim in conjunction with tmux within a few weeks.

At this point, I have moved away from using Caret for nearly all programming and do virtually everything in Vim. Vim lives on the remote server that I do my development on. When I connect to my server, the first thing I do is start tmux and then usually create at least three panes. One is for Vim, and the others will usually be for some sort of cli utilities or server console output. It is worth mentioning that the combination of the high resolution and aspect ratio of the Pixel’s screen makes looking even at very small text on multiple panes really easy.

I do still use Caret for some things. When I download a file that I want to open and look at, Caret is the best option to use on ChromeOS. Additionally, if I open a log file or something similar, Caret is no worse of an option than Vim is for the purpose of viewing it. So to sum up, I found that remotely using Vim is the best option to do any serious programming, but to simply view files, using Caret locally is comparable, if not better.

Finally, for database management I use a conjunction of the Chrome MySQL Admin app and phpMyAdmin depending on which servers I am working with. In the case of exporting and particularly importing very large database files, I will simply ssh into the database server and use the command line to get any work done. When I am working with Redis, I will use the command line application to do everything.

Parting Thoughts

Taking the time to write this post has allowed me to realize that even though the tools and environment setup on my Chromebook Pixel are far removed from what the average ChromeOS user would use on a daily basis, I am still using the operating system in the underlying manner that goes in line with what was envisioned by Google. That is, all of my work is done remotely, in “the cloud”, with only a minor reliance on the actual software on the Chromebook itself. Looks like Google was right all along.

A corollary of all of this is that this setup is not actually bound to the Chromebook Pixel itself. On days that I work from home, I will use a full sized PC, which is running Manjaro Linux. To get started, all I have to do is ssh into my development server and I pick up exactly where I left off from the office the day before. The Pixel might not even leave my bag on those days.

Furthermore, if my Pixel got smashed, stolen, ran over by a train, or stopped a bullet, I would be able to continue working the moment I got my hands on any computer that I could access ssh on. And since I ended up not even using sshfs in the end, I could quite easily avoid putting a replacement Chromebook in developer mode and going through the first few steps of my setup altogether. The only thing I would need to do on a fresh Chromebook would simply be to log into my Google account. Caret and Chrome MySQL Admin would already be installed thanks to the sync features ChromeOS has.

All in all, I am very pleased with this setup and it suits my need perfectly. I get access to the Chromebook Pixel’s fantastic hardware, get to use the extremely snappy and minimal ChromeOS, and ultimately work in an environment that is totally decoupled from the actual device, lending itself to maximum portability by being accessible from any computer that I would ever need to work from.

A Few Words of Wisdom

I was browsing reddit today I stumbled across a post titled “I want to be a Software Architect”, where the author described his goal of improving his technical expertise and laid out a number of different topics, frameworks, and programming languages to study. The end goal behind this improvement, as the title of the post suggests, would be to help one day become a Software Architect or Principal Software Engineer.

I began reading the replies and I found what I believe to be one of the most valuable pieces of advice I have ever read on reddit. I am reposting it here in part to share it with others, but also to ensure that I can refer back to it as time goes on. The following reply to the aforementioned post comes from user /u/justanothersde:

Just a couple notes: In some companies the word “architect” is a dirty word. It implies a hands-off ivory tower fellow who needs other people to get things done. These days having skills to be an individual and independent builder is very important, even if you end up being a principal engineer in a large company where you spend all your time telling everyone else how they should write their code.

Another thing is that getting sharp engineering skills is only half of the equation to moving into a senior role. Only your first or second promotion is really about your mastery of all things tech, beyond that you are promoted more around other competencies like your ability to leverage yourself and get work done through other engineers. You do this by learning to first be right and say important things in public, and then to convince everyone around you that your design is correct. Many engineers never quite realize that they have to speak their ideas out loud, in public, under scrutiny from their peers and their management. Many engineers never learn how to write or articulate themselves in ways that naturally influence people. They use the wrong level of specificity, too many words, focus on the hard things without concisely conveying the big picture. They forget that they are writing for a specific audience and not just clones of themselves. They focus on calling out the hard or stupid aspects of the problem over bring attention to all the things that can be done easily and which have a lot of return of value. Many engineers never bother to understand the business or their customer. I’ve seen so many designs for beautiful or complex systems but as soon as I start asking questions about the flow of money and the business efficiencies realized by this or that tradeoff, some eyes just cloud over. Senior engineers need to be able to manage upwards, and help their management chain just as much as they write new beautiful code for some abstract purpose.

All the subjects in your list are great and should be fun to learn. You will get insight and wisdom learning and using these things. They will also all be irrelevant by the time you become a principal. Interviews for principal engineers usually include coding problems that can be solved in any language, and usually can be quickly knocked out by someone who has only read K&R “The C Programming Language”. The test for coding questions at this level has very little to do with the semantics of the language used or the power of this or that framework, but everything to do with the clarity of thinking and problem solving displayed while going through the motions. Abstract problem solving and the ability to quickly see problems and solutions is almost the only thing that is permanent, while everything else is transient and depends on the time and place and company and whatever other subject environment things that won’t exist in the job tomorrow. PEs are expected to be powerful tomorrow in situations heretofore undefined, and that just means that they are wise, adaptable, dynamic and potent.

As for getting there, don’t overthink it. It’s a long road, but you have nothing but time. Software engineering can be extremely fun, so focus on that and as time flies by you will suddenly realize that you have become a master. It turns out everyone else around you will as well, and they will recognize that with lofty titles that at the end of the day still don’t really quantify all of the problems you can solve and things that you can figure out how to build.

Motivations

Earlier this year I decided to begin learning Haskell. I have been working my way through the Learn You a Haskell textbook. So far it has been a very interesting experience and a pleasant departure from the normalcy of imperative programming that I am used to. As I get further into it, however, I am realizing that the learning curve is only getting steeper. I think the following image is true:

At this point I am fairly sure that I am somewhere between a low and high point in the initial series of peaks. Either way, since Learn You a Haskell is a little slow in terms of how quickly it introduces material, I figured it would be useful to begin doing something practical with Haskell with what I have learned. I read about Hakyll last year, a while after initially making this blog in Jekyll. At the time I was somewhat intimidated by it. Not knowing any Haskell at all can make looking at even the relatively short site.hs file feel like reading hieroglyphics.

Now that I am fairly comfortable with the basic syntax of Haskell, I realize that there is not really all that much to Hakyll in terms of complexity. Getting a blog up with Hakyll might be even faster than getting a blog up with Jekyll (barring the couple couple of gigabytes of data and dozens of dependencies required to get Hakyll working).

The Experience so Far

The initial switch to Hakyll was incredibly simple and easy. In fact, I was not even intending on doing it on the night that I did, I only meant to take a read through the website and had planned on doing it some weekend soon. Instead, after updating hakyll, compiling the default site.hs file, and then running a local preview server with the executable’s watch command, I realized that that was all there was to it.

Experience with both Markdown and Jekyll did definitely help me understand what I was doing. I moved the content from my Jekyll directory over to my Hakyll directory, built the site, initiated a git repository, and after a push to GitHub and a pull on my droplet, the Hakyll version of the blog was live.

I find the simplicity of Hakyll to be really appealing, although I realize that there must be a ton of moving parts abstracted away from the user that go into making the whole thing work. With little to no premade Hakyll themes floating around, and what seems like far fewer directories and files then Jekyll throws at the user by default, I get the impression that building a site with Hakyll will feel a lot more like a “ground up” approach.

Not All Positives

Although I have an overall positive impression of Hakyll at the moment, even in my very short time using it I have encountered a string of issues. I will say that the initial process of getting the site up and running was very smoothly done on my Manjaro Linux PC. I am grateful for Manjaro’s access to the Arch User Repository, where I have been able to download every single application I have ever needed since I started using it. That last statement is not an exaggeration; literally every single one.

The issues that I had encountered, however, have been on both my Digital Ocean droplet (where this blog is hosted) as well as in the crouton chroot of my Chromebook. In both the droplet and in crouton, I am running Ubuntu 14.04. I began by attempting to install Hakyll on the droplet, and the first thing I got was strange exit errors even when trying to update cabal. It turns out that even updating cabal requires a lot of memory, so I scaled up my droplet from 500 MB to 2 GB.

Resizing the droplet solved the memory issue, but allowed me to discover that the install process would crash when attempting to install pandoc. After doing some research it appeared that the issue was caused by a cabal bug.

I found instructions on how to move past Ubuntu’s default ghc version of 7.6.3 to 7.10.1, along with a newer cabal. After quite a bit of waiting for everything to install multiple times, I finally managed to get Hakyll running and my site.hs file compiled on my droplet. Except, when I resized my droplet back down to 500 MB it appeared that ghc was broken, my site executable file was broken, and cabal was having issues.

Having conceded that getting Hakyll working on the droplet was a lost cause, I decided to try to get it working on the Chromebook. I went through the entire 45 minute long install process on the Chromebook, only to find that when I finally ran the build command with the site executable, I was getting some sort of text encoding error from pandoc.

At this point I have given up on both the Chromebook and the droplet. I assume the Chromebook’s issue is fixable. For the time being I am content with only being able to compile and build my website from my Manjaro PC.

A Few Lessons Learned

I think I have heard the phrase “cabal hell” somewhere on the Haskell subreddit a few times. I have also heard mention of things like different ghc versions leaving people with uncompilable code. I think that I am now catching a glimpse of both of these.

These issues are both unpleasant and costly, particularly in terms of time. I certainly hope that other experiences I have plan on having with Haskell do not turn into more of the same. I have faith in Haskell and its incredibly helpful community, but I have now witnessed firsthand not everything always being hunky-dory.

The Information Problem

A couple of months ago, while faced with a number of design decisions in a project I was working on, I decided to sit down and qualify the fundamental principles of providing information to someone. I felt that there was certainly a systematic way to approach the problem of attempting to serve someone the information that they were either looking for or expecting to receive.

It quickly became quite evident that there were certain decisions I could make that would be desirable to the end user, and other decisions that could make receiving the desired information a painful process. Ultimately, after a lot of thinking, I came up with what I believe to be four fundamental principles that are always relevant when the goal is to convey information in any way to another individual.

The four principles are as follows:

1. Clarity
2. Relevance
3. Accessibility
4. Density

I will explain each of these principles in detail.

Clarity

Information clarity is perhaps the most easily identifiable principle in regards to how information should be handled. It is also probably the least abstract of the four principles. This principle is relatively simple; information must be provided and presented in an intelligible and unambiguous manner.

Information that is being presented should not be confusing. It should also not require supplementary material or additional searching to understand (at least not when the goal is to be sufficiently thorough).

If the information you are providing to someone else (be it an end user or a friend receiving an email) causes them to do any of the following, your information is lacking clarity (this list is certainly not exhaustive):

• Your recipient has to Google search something you said to understand it.
• Your recipient does not understand the context of what you are saying.
• Your recipient has to decipher errors (spelling, grammar, formatting).
• Your recipient has to ask followup questions before taking further action.

It is important to realize that having mostly clear information is challenging but quite achievable. Having crystal clear information, however, is significantly harder. Getting a reply email with followup questions from a colleague is certainly going to happen at some point, even when you thought you explained everything thoroughly. There are users out there who will not understand what you assume everyone understands, forcing them to Google search for what you are attempting to explain. But, this is not necessarily going to be your fault. This may be because they accidentally stumble into material that is too advanced for them, which they are definitely not the target audience of. What is clear to some will not always be clear to others.

Providing clear information requires a strong attention to detail, a meticulous approach towards providing explanation, and a very good understanding of the audience you are reaching out to. It might be pretty easy to know what your colleague or friend knows, but it is insanely difficult to try to anticipate what is clear for all of the 25,000 users that visit your website each month.

As the entity providing the information, you are responsible for its clarity. Your assumptions about your recipient(s) must be reasonable, and you must cater to their needs. Failure to do this will result in confusion, ineffectiveness, and eventually even negligence in regards to your information on the part of your recipient.

Clarity is a sort of foundation for providing information. If your information is horribly unclear, none of the other principles I am going to discuss will even begin to matter. A reasonable level of clarity absolutely must be present.

Relevance

Information relevance is the idea that the information you are providing should be useful to the person you are providing it to. Ideally, our goal with regards to relevance is to provide information whose subject matter is what the recipient expects. That is, we explain or touch on everything they need, without any additional, superfluous or unneeded content.

Achieving relevance is a balancing act. In my opinion, it is better to err on the slide of slightly irrelevant rather than on the side of slightly lacking. Something that is slightly irrelevant may be an annoyance or cause the person who is receiving your information to take a bit longer to digest it. But, by contrast, something that is slightly lacking may actually be lacking a key detail. The last thing you want to do is force your recipient to have to go hunt down this detail on their own, or wait hours or days because they need a followup question answered.

If your recipient does any of the following while taking in your information, your information is likely not completely relevant:

• Your recipient’s focus is shifted to something else as he is processing your information.
• Your recipient has to skim or jump through what you are sending him.
• Your recipient has to ignore opinions or other non-factual content.

As with clarity, providing information that is relevant to your recipient requires an understanding of your recipient. Why are they accessing this information? What do they expect? What do they not expect but still need?

Google search is a service implements the principle of relevance extremely well. For example, if I search for ‘Python’ in Google, I will not get any references to the animal. All of the results on my screen will be related to the Python programming language. If a zoologist were to perform the same search, it is quite likely that they would get the animal instead (unless they were also a hobbyist Python programmer). The reason for this is that Google catalogs my data and my behavior. In a certain sense, Google understands me and is able to provide what is relevant to me as a result of this. Google search manages to hit information relevance right on the head in this way.

It is almost a challenge to provide absolutely no relevant information. Typically what you write or provide will have some relevance in some way. The goal with regards to relevance, as mentioned earlier, is to find the meeting point between information that is superfluous and information that is lacking.

Accessibility

Information accessibility is about the flow of actions that the recipient of your information has to take before he actually receives it. Unlike relevance and clarity, accessibility has nothing to do with the actual content of the information itself. Accessibility is also not directly applicable to simple forms of communication (such as an email) or basic information (such as a static webpage with no navigation whatsoever).

Accessibility is the idea that your recipient should have to take as few steps as possible to reach or find the information that he is looking for. Accessibility also means that the recipient should be able to easily find the information he is looking for, even if he does not know exactly which steps to take to find it.

Many services fall far short of making information (or functionality) easily accessible to their users. An example of very poor accessibility I encountered recently was when I was attempting to manage my recurring payments in my PayPal account. As it currently stands, PayPal makes cancelling a recurring payment a five step process. Worse yet, I was unable to find this solution on my own, I had to Google search “paypal cancel recurring payments” because the option to view and manage them is buried so far down in the user options.

As a PayPal user, it is quite evident that managing where my money is going should be one of the most important things to me. PayPal’s decision to make it so difficult to manage recurring payments (coupled with the fact that you are forced into recurring payments with certain services you purchase using PayPal) is an example of very poor accessibility.

A general example of excellent accessibility is the availability of an ‘omnisearch’ that many services and products now offer. This feature is typically a search bar located on the homepage or dashboard of the service. Typing into the search bar will usually provide a comprehensive search of all types of options and content. For example, typing ‘change password’ into this type of search bar in most services will instantaneously load a link to the account management page that allows you to change your password. This is fantastic, because as the user, I just need to type in what I want to do and perform one click, and I am in a spot to perform said action.

In the end, determining whether the principle of information accessibility is being applied well enough is subjective. Generally speaking, the more easily accessible, the better.

Density

Finally, information density is about providing the proper scope of information to your recipient. This principle is about how compact or ‘condensed’ the information you are providing is. Compared to clarity or relevance, this is a relatively abstract idea that I will explain carefully.

Information can be clear, relevant, and accessible yet be provided with the wrong level of density, making it significantly less useful than it otherwise could be. Information density is also unique in that it is manifested both through actual content (like clarity and relevance) as well as through design and functionality (like accessibility).

The general idea behind information density is to supply the recipient with information in a way that allows him to spend as little time as possible digesting it, while still giving him everything he needs or expects.

This may initially sound like information relevance. That is not the case, however. If we go back to my ‘Python’ Google search example. Google returns results about the programming language and not the reptile because it knows that the programming language is more relevant to me. This has nothing to do with density.

A great example of the principle of information density in action in the form of an activity digest email that many services provide. Suppose I get a daily digest of new posts of a particular forum I visit. The daily digest is designed to summarize the new content from each day, emailed automatically at the end of the day. The daily digest will likely list at least the title of each post, and a link directly to it. But there are other pieces of information which might also be worth including.

For example, the digest may also include the specific time that each post was made, or the author of each post. It may include the number of replies each post has, or even include the first few sentences of each post. And to contrast this, we can think about the daily digest of a forum administrator rather than a regular user. The forum administrator may be more concerned with user activity and statistics rather than individual posts, so he would prefer that the daily digest simply included the number of posts made that day, in addition to the number of user logins, pageviews, replies, etc.

None of these pieces of information are irrelevant. Yet changing which ones are included changes the density of the information. One other facet of information density is that most ‘condensed’ information is provided with an option to expand the information. In the case of the user facing daily digest, this is a link to each forum post, allowing him to access the full post.

Altering the density of the information that you are providing to your users can completely change how your product or service is used. It can change how much time users spend on it, how many visit, and most fundamentally, how the service itself ends up getting used. It is this fact that makes information density a crucially important principle to consider when designing a service that provides information.

Summary: ‘I-CRAD’

I believe that these four principles are a comprehensive approach to providing information. They are applicable to virtually every type of environment, service, or product in some way. Their implementation can also spell the difference between a booming success and a total failure.

To recap, the principles are:

• Clarity: Information should be intelligible and unambiguous
• Relevance: Information should be of the correct subject matter
• Accessibility: Information should be easy to find and quick to navigate to
• Density: Information should be compressed to an optimally useful level

The way I have come to remember these four principles is using the acronym ‘I-CRAD’. That is, Information Clarity, Relevance, Accessibility, and Density.

What is an Algorithm?

I think that when most people hear the word ‘algorithm’ they tend to imagine some mathematician or software engineer doing some complex mathematical or computer related task. This is a correct assumption. Mathematicians, programmers, software developers, and engineers all regularly use algorithms to complete various professional and academic tasks. It is algorithms that play a key role in critical developments in science and technology. Algorithms have been and continue to be used in developing the world we live in today. But, I think that people oftentimes do not really understand what algorithms really are, and that their reach extends far beyond the realm of the various aforementioned professionals and significant scientific developments.

So what is an algorithm? Here is a definition from Professor Jeff Erickson from the University of Illinois:

An algorithm is an explicit, precise, unambiguous, mechanically-executable sequence of elementary instructions.

This is one of the first things that I was met with when I took his Algorithms course as a Senior. This definition is great. I like this definition because it captures exactly what an algorithm is in just a few words. However, this definition is still very intimidating. I do not think it helps people move away from the image of the mathematician or software engineer I described above.

Let’s try another definition. The following is my own definition of the word ‘algorithm’. I am going to set aside the rigorous academic requirements for an algorithm and instead focus on what I believe is the very essence of an algorithm, in more simple terms. My definition is as follows:

An algorithm is a sequence of steps that, when followed, will complete a specific task.

This definition is an excellent starting point for algorithmic thinking.

Algorithms Everywhere

I believe that a person can think in terms of algorithms in many different contexts. These are situations and environments that reach far past a academia or a technical job. What I mean by this is that it is possible to think of common day tasks algorithmically.

Why? Well, since I took the algorithms course I mentioned above, my approach to most problems changed. I adopted an algorithmic mindset towards virtually every task I had to perform. I found that thinking about the algorithm involved in executing a particular task is a recipe for great productivity. By doing this, it becomes easier to get started, harder to get distracted, makes mistakes during execution less likely, and can make the actual time the task makes shorter than when not using this mindset.

So what does this really entail? Well just like a computer scientist would do, thinking algorithmically outside of the realm of computers involves breaking a task or problem down to fundamental units, or small steps, and designing a plan or procedure on how to complete these units in order. I find that thinking about the small details of a task helps as well.

For example, suppose that I have to do the laundry. I am sitting in my room, in front of my laptop. I can just think, “well, I guess it’s time to go do my laundry,” and then just go start doing things. There is no doubt that this is an acceptable way for me to do the laundry. In my opinion it just is not ideal.

Suppose I have to get my laundry done, but instead of just deciding to do it, I develop an algorithm to do it. The algorithm to get the laundry done is as follows:

1. Perform a quick scan of my room for any miscellaneous articles of clothing that are not in the laundry basket.
2. Pick up every article of clothing.
3. Put them all in the laundry basket.
4. Grab the laundry detergent from the closet.
5. Put the laundry detergent on top of the basket.
6. Carry the basket downstairs to the laundry room.
7. Set the washing machine.
8. Start a timer on my watch for 35 minutes.
9. Bring the laundry detergent back upstairs, leave the laundry basket.

Another, shorter algorithm, similar to steps 7-9 would follow a while later for setting the laundry to dry. This seems extremely tedious, right? Why bother doing it this way? To begin with, there is no need to actually write down an algorithm like this. This is just something you should lend about 5-15 seconds of thought to, immediately before you begin to perform the task.

The advantages of this approach, in this and most cases are:

1. Efficiency - By using this approach, I am making a plan that will be at least as efficient as if I just got up and started moving. For example, suppose I forgot my laundry detergent in my closet, and went to the laundry room without it. Now, I have to spend an extra 90 seconds running back to my second floor bedroom and then back down to the laundry room. Not that big of a deal, but could have been avoided with about 10 seconds of planning.
2. Ease of Execution - This may not be the ideal term, but the general idea here is after acknowledging the specific steps a task takes to get from start to finish, I find that the task ends up appearing substantially easier to execute than just looking at the task holistically. In this case, “doing my laundry” seems very tedious. However, scanning my room is quick, grabbing the laundry detergent is quick, carrying the basket is quick, setting the laundry is quick. Looking at it this way, I have a chain of quick, well defined tasks. This is easier to digest and start working on than a single, ambiguous task.
3. Focus - The last element of this type of approach I find is that one develops a better sense of focus during the task. No step in doing the laundry is difficult, but it is still possible to make mistakes during the process. For example, setting the load as cotton sturdy instead of permanent press may end up shrinking and wrecking your clothes. Not starting the timer on my watch can lead me to forget about the load of laundry until bed time when I realize I still have a wet load in the washing machine. Thinking algorithmically about a task increases focus during execution, decreasing the frequency of mistakes.

The other point to realize here is that this type of approach really can be applied to virtually everything. Doing the laundry, cleaning your room, making a sandwich, grocery shopping, completing a series of errands involving driving around, studying for an exam, mowing the lawn, performing a workout, or my personal favorite, making coffee. It would not be difficult to add a dozen more things to this list.

Adopting this type of approach for most of the tasks you do takes some time. The ideal way to do this is to turn this type of algorithmic thinking into a habit. Eventually, you will find yourself completing more tasks, more quickly, with fewer errors and less time wasted.

Ultimately, I believe that your time is the most valuable thing you own. Saving 90 seconds because you made sure to bring your laundry detergent may not seem like a big deal. But, saving 90 seconds four times a day is three hours a month, which is one and a half days per year. This adds up to a couple of months in a lifetime. This, coupled with the fact that the things you do do are performed more accurately, makes such an algorithmic approach worthwhile. Before you know it, you will have lived a lifetime of slightly enhanced efficiency.

The Reign of Adblock Plus

I started using Adblock Plus in 2006, the year it was released for Mozilla Firefox. This was two years before Google Chrome was even released. For anyone who was even slightly technologically savvy, the advent of Adblock Plus had a huge impact on web browsing. It changed the playing field and gave people a huge reason to migrate to Firefox and away from Internet Explorer. Between addons (specifically Adblock Plus), and tabbed browsing, Firefox was a big deal.

In 2011, I encountered an issue with Firefox where it simply refused to open on my Windows machine. Even a reinstall wouldn’t fix whatever problem I had at the time, so I migrated to Google Chrome. I’ve been using Chrome ever since. What came with me, however, was Adblock Plus.

A few months ago I read an article about ad blocking extensions exactly like Adblock Plus. The article came with somewhat surprising news. It turns out that blocking ads in the browser is a demanding process. It uses a bit of CPU power but in particular, it uses a lot of RAM and slows down page load times. This was something I had never really bothered to think about or consider before. Blocking ads seems like it would do nothing but help one’s browser performance.

This news came to me around a time where my browser usage was becoming more and more intensive. With Google making Chrome more and more into its own self-sufficient ecosystem of applications and dynamic content, I had fewer reasons to have regular applications open and more reasons to have many tabs open. It was (and still is) not uncommon for me to have over 10 tabs open when I am browsing the internet. Between Gmail, Google Music, sometimes Facebook, a few Reddit tabs, and maybe a YouTube tab, I found that I was frequently devoting over 2 gigabytes of RAM to Google Chrome alone. Google Chrome had become sluggish, no longer operating at the high speed Google always claimed it was king of.

The hosts file

Enter the hosts file. Well, reenter the hosts file. Originally implemented in the 1970s, the hosts file was used by operating systems to map hostnames (such as google.com) to IP addresses (such as 74.125.224.72). The hosts file was used to navigate ARPANET, a precursor to the Internet. Users would either need to create and update, or get their hands on a copy of an updated hosts file to actively be able to browse ARPANET via domain names. In 1984, a system which replaced the need for a manually updated hosts file was released. It was called the Domain Name System, which we typically refer to today as just DNS.

But the hosts file was here to stay, and had become a key part of virtually every operating system with any sort of networking capability. It gave the user the ability to locally map any hostname to any IP address. Here is an example of what most default hosts files will look like (this one taken from Arch Linux):

#
# /etc/hosts: static lookup table for host names
#

#<ip-address>	<hostname.domain.org>	<hostname>
127.0.0.1	localhost.localdomain	localhost
::1		localhost.localdomain	localhost

# End of file

Any line that starts with # is a comment, and does nothing when the file is actually parsed by the computer. At the start of every other line we have an IP address, and at the end we have a hostname. We can see that the file’s role in its current state is to map the “localhost” hostname to the default loopback IP (127.0.0.1 or 0.0.0.0 in IPv4 and ::1 in IPv6) so that anytime we enter “localhost” in our browser address bar (or anytime any application uses it as a hostname), the computer will connect to itself. This shortcut is often used by anyone who does any sort of web development or other network application work.

The hosts file above also demonstrates another capability which we can take advantage of. We can, in effect, block connections from any website if we route the site’s hostname to 127.0.0.1 using the hosts file, as the contents of the hosts file takes precedent over any DNS mappings we receive from the internet. So if we add a line such as this:

127.0.0.1 reddit.com

we tell our computer to take us to 127.0.0.1 instead of reddit’s normal IP address everytime we enter reddit.com in our browser. So, by redirecting all traffic that would otherwise go to reddit.com to ourselves, reddit becomes unreachable. All we need now is a list of ad serving domains that we want to block and the hosts file serves the exact same purpose as Adblock. The difference being that the hosts file uses no RAM and no CPU. It does not slow down any browser, and it completely eliminates connections to any domain listed in it. So the hosts file affects Firefox, Chrome, and any other browser and application we may be using.

Fortunately, there are kind people on the internet that have compile pre-existing hosts files filled with thousands of malicious and ad-serving domains. The first site I found is http://winhelp2002.mvps.org/hosts.htm, which includes not only a hosts file in both .zip and .txt form, but a great explanation of how the hosts file works and various types of domains to block. The second is http://someonewhocares.org/hosts/, which also offers a great description as well as instructions of how to update the hosts file on your PC.

Time for some automation

I would like to take it a step further and write a short bash script to automatically update our hosts file using the hosts files provided by both of these sites. This will be written to work on Linux, but should probably work on OSX as well. Ideally, every time we run the script we want to grab the most updated versions of each site’s hosts file, so we have the most up-to-date version of a list of malicious and ad-serving domains.

So, the algorithm we want to perform will be something like this:

1. Make a copy of our original hosts file as a backup (only if this is our first time running the script)
2. Download the pre-loaded hosts file from mvps.org
3. Append (not overwrite) it to our existing hosts file
4. Download the pre-loaded hosts file from someonewhocares.org
5. Append (not overwrite) it to our hosts file
6. Delete the files we downloaded from both sites, as we do not need them anymore

In addition to this, if we run our script a second time or any additional time, instead of making a copy of the current hosts file, we want to overwrite our primary hosts file with the copy we took we ran the first time. This is so that we are starting with a clean hosts file every time we run the script, and not adding tens of thousands of redundant lines.

After a little bit of experimenting, the script is as follows:

#!/bin/bash

echo "Checking if copy of original hosts file exists..."
if [ -e /etc/hosts-original ]
	then
		echo "Copy of original exists."
		cat /etc/hosts-original > /etc/hosts	
	else
		echo "Copy of original does not exist.  Copying..."
		cat /etc/hosts > /etc/hosts-original
fi

echo "Downloading mvps.org hosts file..."
wget http://winhelp2002.mvps.org/hosts.txt --output-document=hosts-mvps

echo "Downloading someonewhocares.org hosts file..."
wget http://someonewhocares.org/hosts/zero/hosts --output-document=hosts-swc

echo "Adding mvps.org hosts file content to main hosts file..."
cat hosts-mvps >> /etc/hosts
echo "Adding someonewhocares.org hosts file content to main hosts file..."
cat hosts-swc >> /etc/hosts

echo "Cleaning up..."
rm hosts-mvps
rm hosts-swc

echo "Completed!"

You can view and download the same file on GitHub. I encourage you to open it in a text editor after downloading it to confirm it is exactly what I have in the box above. You will need to run the script with root permissions by prefixing with the sudo command because the entire /etc/ directory is read only.

If you are a new Linux user (or an old, forgetful, Linux user) attempting to run this script for the first time, open a terminal and type the following commands:

cd ~/Downloads
sudo sh hosts-update.sh

The first line will change to the Downloads directory of your home folder. The second line will actually execute the script. The first line is not necessary if you know what you are doing and/or have moved the script to a different directory. You will be prompted to enter your UNIX password to give yourself the root privileges mentioned above.

Windows users, unfortunately you will have to do this process manually. You will need to download the contents of both hosts files and open up the one on your computer in a text editor, and then paste both inside. Once again, do not overwrite the original hosts file content. Also, do not open long files in regular Notepad. It will crash. Use Sublime or Notepad++. Detailed instructions on how to update the Windows hosts file are on both of the hosts file sites I linked above.

Android users, if you have rooted your phone you have an option like this as well. There is an app on F-Droid called AdAway, which does exactly what we did here. Installing the app and giving it root permissions will then automatically update your /system/etc/hosts file with its own proprietary list of ad-serving domains.

ChromeOS users, unfortunately there are no good options for ChromeOS. First of all, editing the file even using sudo does not work. There is a workaround method, however every time ChromeOS updates the hosts file will be reset.

With that, enjoy your Adblock alternative that takes absolutely zero system resources. You will see empty boxes where ads would be on certain websites, or the ads should just be gone all together. Feel free to contact me with any questions. @charukiewicz

Update (Feb, 15, 2016): There has been a lot of material that has either come out or I have read about since writing this article. Namely, there are a few really good scripts out there that do what my bash script does, but are more feature filled. They pull from more sources, allow you to include your own whitelist, etc.