The only reason this matters is that this means my stack is Node.js-based because of this decision. Since this app will be heavily data-driven I will need a database, but it's also very much an MVP that is changing week to week as we tweak it. I wanted a database ORM that would give me migrations to make it easier to deploy changes. This led me to Prisma, the "next-generation Node.js and TypeScript ORM."

Great! Now, since I am building this on the cheap I did not want to deal with database hosting. I wanted to deploy the application to one of my existing servers so I picked SQLite. Since I am using an ORM I could easily switch to a different relational database later on and migrate the data.

So far everything has been working great, at least it was until I needed to start building reports. After failing to use the ORM to build the joins I needed, I resorted to just writing the SQL I needed. The data is normalized in a sane way, so the calculation I needed was a simple join between three tables and a GROUP BY and SUM().

Here was where I ran into my first issue.

Prisma has you define all the schemas in a schema.prisma file using their own syntax. It is nothing complicated, but you define the tables and columns, and it uses this to write migrations as well as generate ORM objects to use in the code. For a game we have a createdAt and a submittedAt column that I have set to a Datetime type.

// schema.prisma
model Game {
  id          Int             @id @default(autoincrement())
  title       String
  createdAt   DateTime        @default(now())
  submittedAt DateTime?
  owner       Player          @relation(fields: [ownerId], references: [id])
  ownerId     Int
  objectives  GameObjective[]
  players     GamePlayer[]
}

The report I was writing was for a month's worth of time, so I wrote a basic SQL query to find all the games that were submitted within the month. The best part was I got no results back.

SELECT * FROM Game
WHERE submittedAt >= '2023-07-01 00:00:00' AND submittedAt < '2023-08-01 00:00:00'

No results! So I decided to look at the data as there was no reason such a simple query should fail. I knew I had data there.

This was when I learned a very important thing about SQLite - it does not have a DateTime data type. It does have date-time functions and can work with date-time values though, we just need to store string dates as TEXT and in the ISO8601 format, REAL for Julian day numbers, or an INTEGER for Unix Timestamps.

Looking at the data in the table, I noticed large numbers. Awesome, we have Unix Timestamps. This is not the best as I would have preferred strings, but it will do.

SELECT * FROM Game
WHERE submittedAt >= 1688169600 AND submittedAt < 1690848000

This still returned no results. If I took off the AND ... part I got the expected games, but I needed to restrain the games by a single month. I went down another rabbit hole trying to figure this out, because none of the date-time functions worked either. I was looking at the data and realized that these numbers were stored in a text column.

This was where I learned another thing about SQLite - SQLite uses "Type Affinity" for data.

In order to maximize compatibility between SQLite and other database engines, and so that the example above will work on SQLite as it does on other SQL database engines, SQLite supports the concept of "type affinity" on columns. The type affinity of a column is the recommended type for data stored in that column. The important idea here is that the type is recommended, not required. Any column can still store any type of data. It is just that some columns, given the choice, will prefer to use one storage class over another. The preferred storage class for a column is called its "affinity".

The type of the column is merely a suggestion, and you can store any type of data in any type of column. SQLite will just make it work.

I tried to convert the column to an integer, which was also when I learned you could not change a column's type. You have to make a temporary table with the new schema, copy the data over, delete the original table, and rename the temporary table.

This still did not fix it.

OK, fine, whatever. I swapped the table back to being a TEXT field. It was after this I noticed that I was now getting errors saying that the createdAt and submittedAt values were too large for the INTEGER datatype, and to switch to BIGINT. Why was I getting this now? Shouldn't a Unix Timestamp fit inside an integer? I should have a good fifteen years until 2038 when this will be a problem.

I then realized the problem. My data was not a proper Unix Timestamp, which is the number of seconds since the epoch (1970-01-01). My data was the number of milliseconds since the epoch, which is not a Unix Timestamp.

This is why none of the date-time functions would work for me, because the data I was passing in was not a valid Unix Timestamp. My sample queries were not working because the greater-than-or-equal value was small enough to find my games, but the less-than value I was using was too small. The comparison worked fine, I was just using proper Unix Timestamps to generate the query so my WHERE condition failed.

But wait, aren't I using an ORM that should be properly storing my data?

So I tested my sample query with milliseconds and it worked perfectly.

It turns out Prisma is storing the SQLite DateTime type as a millisecond-since-epoch value, not any of the three values that SQLite will interpret as DateTime values. Prisma just seems to be turning date strings into a new Date() object and just dumping it using Date().getTime(). It is not doing the best for compatibility (a date-time string in ISO8601 format) but doing the least amount of work to store the data. Prisma is not even using Date().toISOString() to get a date.

Two hours of my time went down the drain because SQLite allows duck typing on data, and Prisma decided to store dates in a non-standard format.

Thanks, JavaScript.

Why Are There Multiple Versions of Linux?

The ecosystem around Linux is built upon the idea of sharing software. The Linux project produces what is called a Kernel, or a program that exists at the core of an operating system to run everything else. The kernel itself needs software to run alongside it, which is usually termed "userland software." This includes things like text editors, web browsers, desktop managers, and essentially everything except super low-level things like hardware access. All of this software is grouped together to create a "distribution," and it really is nothing more than a collection of software with no more commonality than the use of the Linux kernel. You can swap out the kernel or the software and (generally) everything is fine. Collectively this is called a "distribution."

This is a little bit different than operating systems like FreeBSD, where the kernel and software are packaged together as a singular source. You install a version of FreeBSD which has a specific kernel and set of software, and something else like OpenBSD has its own kernel and set of software. Proprietary operating systems like Windows and macOS work this way as well.

The Linux ecosystem has spawned a series of distributions that are based on other distributions. Ubuntu is based on Debian, as is Linux Mint. Why does this happen? Usually, because one distribution has restrictions some users do not like. Debian tends to heavily value stability so has a very strict policy for upgrading packages, making many Debian packages based on older code. Ubuntu has a much more relaxed policy but uses Debian packages as a base. PopOS! builds on top of Ubuntu as Ubuntu is newer than Debian, and came into being because Canonical decided to stop work on Unity, the window manager Ubuntu used to use. It now focuses on supporting the hardware that System76 builds and has a different policy for non-free software than Debian or Ubuntu do.

In the case of Red Hat, the flow of code works differently. The base distribution for Red Hat is Fedora, which includes the most bleeding edge code. As code is validated as stable and decisions are made on versions, this flows into CentOS Stream. CentOS Stream is eventually locked down to specific packages and released as Red Hat Enterprise Linux, which has a very long support lifecycle and therefore is considered very stable. The issue people have with using RHEL is the cost, as RHEL is a commercial product that includes support.

Almost all software distributed for Linux is considered "Open Source," which means that raw source code for the software is made available to just about anyone. This allows users to modify software any way they see fit, and distribute those changes to other people. This is accomplished through a variety of open-source licenses, which I've talked about before, and in most cases, this source code is easy for people to access. The internet has made it easy to distribute source code through sites like GitHub or GitLab, and running a standalone source code repository or even an FTP server with source code are ways that distributions have also supplied source code.

Red Hat used to provide source code as a git repository that they hosted. Users could download the source code and compile the packages themselves, and if they did you would have a copy of RHEL without any RHEL branding. This brought into being CentOS, which used to be a third-party distribution based on RHEL source code. Users could download a CentOS release and could be safe in the knowledge that something like CentOS 6.3 was compatible with RHEL 6.3, and was completely free (as in cost). You did not get support like you would if you purchased RHEL through Red Hat directly, but many users did not or could not afford a full Red Hat license. All of this was legal thanks to how open-source software license work.

Consolidation

In 2014 Red Hat acquired the CentOS project but promised that nothing would change. In 2020, however, Red Hat announced that CentOS as a project would be discontinued and in 2021 would move to a rolling, testing release for RHEL. This meant that CentOS would be more stable than Fedora, but would not guarantee to be 100% compatible with RHEL as it would become what amounted to "the next version" of RHEL at all times. In response to this other distributions popped up like Rocky Linux and AlmaLinux to fill the gap that older CentOS releases filled. During this entire time, we also had Oracle Linux, a second commercial distribution based on RHEL source code and operated by Oracle Corporation.

The change announced by Red Hat now means that projects like Oracle Linux, Rocky Linux, and AlmaLinux now do not have direct source code access to RHEL, and just CentOS Stream. That means they cannot promise exact compatibility for the current versions of RHEL, but just "the next version," and since CentOS Stream has the potential to not be as stable they cannot promise the stability that RHEL provides. They can no longer fill the gap that some users wanted them to fill. If users want RHEL, they will have to become a customer of Red Hat.

Why Do This?

It is clear that Red Hat clearly wants to make more people pay for access to RHEL, and sees these derivatives as lost sales. In many cases though the reason users went with CentOS originally, or free alternatives like Rocky or AlmaLinux, was because the pricing structure supplied by Red Hat did not work. A license for a single server started at $349 and that includes no support, is restricted to physical (not virtual) servers, cannot be combined with any other RHEL licenses, and is labeled as not production ready. This then means for a "production ready" license you need to spend at least $799. Oh, and these are yearly subscription licenses.

For many users, this is not economical. Why pay $349 a year for a license with no support when you could just use Rocky or AlmaLinux instead? If these derivatives did not exist users would just use some other stable distribution like Debian or SUSE. RHEL is treating RHEL alternatives as some sort of piracy situation where each install of Rocky equates to a sale that was "stolen."

In response to a post on LinkedIn by Jeff Geerling Mike McGrath, Vice President of Core Platforms at Red Hat, confronted the term "freeloaders."

Finally, I wanted to say something about the term "freeloaders" I've seen many use it. This is a mostly internal term we have at Red Hat, it looks like at some point it slipped out in the public. So what does it mean? A freeloader is when a large enterprise business has 20 RHEL licenses, 150,000 community rebuild systems, and sometimes hundreds of user accounts and hundreds of kbase searches per month. It's not the enthusiasts, it's not the hackers and coders, it's not the academics, and it's not the people that use rebuilders because they can't afford it. We really try not to use the term, but when we do, it's about the large companies that can afford to pay but don't.

-- Mike McGrath, 2023

So to coerce large users who could pay they went with a solution that kills access to all non-paying customers including the users who they claim are not the problem.

How Can They Do This?

Most people coming into this ask how Red Hat can legally do this. Isn't almost all the source code used and published by Red Hat some variation of the GNU Public License, which requires the source code to be made available? Yes, yes it is. Then how can they lock access behind a paywall?

The GPL specifies that if you receive a compiled version of a piece of software, say in an RPM or DEB package, the packager must allow you access to the source code upon request. They are not required to ship the source with the binary but must make it available as conveniently as possible for a user who requests it. There is no requirement that the source be available for non-users or every person on the planet, or that the source code be available in a format that is easy to download and consume. This allows GPL software to be sold commercially and is all totally within the guidelines of the GPL. Red Hat is only required to give source code to users, which they may deem as people who have paid for RHEL, and only when they request it.

The second argument comes up with the fact that to be a RHEL customer, you must agree to a secondary End User License Agreement. This EULA imposes additional restrictions on how you can use the software, such as limiting installation to physical hardware, and includes provisions for revoking user subscriptions and accounts if a user shares RHEL source code. The GPL, however, states that a developer cannot impose any additional limitations above-and-beyond what is in the GPL. How do they get away with this?

Simple - the GPL still allows this. The GPL is singularly focused on making sure that software that you receive has a list of freedoms, including the ability to be modified and shared, and forcing the developer to make those modifications and sources available. Nothing in the additional EULAs and agreements stops this from happening. If I were to sign up for a RHEL subscription, download all the source code, and redistribute it, I am allowed to do so under the GPL. I could not be sued for this or be considered in breach of the license.

What the GPL does not provide, nor care about, is continued access to other software. If you purchase software from a vendor and it includes GPL software, the vendor is under no obligation to supply you with future or past software. If you purchased "Accounting Software v1.45", the vendor is only required to give you source access to v1.45 of that software. They are under no obligation to give you access to 1.44 or 1.46. All the vendor has to legally do is supply you with the source for the software that was delivered to you. If you do something that causes them to no longer consider you a customer and, therefore, stop receiving the software, they are under no obligation to give you access to the software in the future.

The GPL does not provide perpetual access to software, it only provides source access to a binary distribution you received. It does not account for and therefore does not differentiate, subscriptions or single transaction purposes. So Red Hat is entirely within its rights to restrict source code from non-users, and if you do something to cause yourself to no longer be a customer, Red Hat is under no obligation to continue to give you anything more than the source code for binaries you have already received. They are only obligated to give you the source code for the software you have already received.

Letter of the Law vs Spirit of the Law

There is a concept of "Letter of the law vs the Spirit of the Law" which is the idea that laws can be interpreted in two ways - a strict interpretation of law based on the wording, and a looser interpretation based on the intent of a law. This exists because some laws a written in a very vague way that can cause unintended consequences, so the safest way to interpret a law is by the actual verbiage of that law. In many cases, very innocuous laws can become very harmful depending on how the law itself is read.

Many people, especially in the age of the internet, look at the GPL from the "spirit of the law" side. The GPL is designed as a way for code to be forever shareable, and attempts to lock code down are frowned upon. Thanks to TiVo, the GPL v3 was rewritten in such a way to close loopholes that large corporations have found in wrapping GPL code so as to not provide source access to end users. The point of the GPL is to make sure the software source is available to everyone.

The disconnect is that this is not how the license is written. The GPL does not directly care for the end user or the developer. The GPL treats software as an entity with inherent rights, one of those being that software must be "free" in a liberty sense. It must be free to be modified, and users should have the freedom to share those modifications. Nothing that Red Hat is doing is infringing on that letter of the law.

There is an argument that they are in violation of the spirit of the law. Considering that for decades the source code for Red Hat, then Red Hat Enterprise Linux, Fedora, and now CentOS Stream have been available it rings a bit hollow that this is a "technical" decision and it is too hard to continue to make the source available. The source code for RHEL is being stored in version control somewhere, and it is conceivable that Red Hat could continue to make it available, just like the CentOS Stream source is made available. What has actually changed in the landscape that made Red Hat decide on this change?

I would say it is IBM, and it want to squeeze as much revenue from any product as possible. Nothing more, nothing less. Red Hat is in their full legal right to do what they are doing, but from an ethical standard they are ignoring what makes open source open source.

Where to go From Here?

Do the same thing that we have continued to do since the dawn of software - refuse to accept what corporations want and do what makes sense for you. For me, even though I have loved Fedora and spent years running RHEL and CentOS servers in businesses, I will be switching to something else. My current plan is to investigate NixOS as I am intrigued by not only the immutability aspect but also the reproducibility aspects. For servers, I may switch back to Debian, or potentially move to NixOS.

For you, do not forget that there are multiple alternatives out there that may better align with your ethics. If you want stability, look at OpenSUSE or Debian, or continue to support Rocky Linux or AlmaLinux. If you want a good desktop, there is always Ubuntu, Arch, or dozens of others that focus on specific niches like gaming or productivity.

Vote with your wallet and your downloads.

git is a time machine. It allows you to travel up and down the history of your project. Each commit is a blip on that timeline. A commit is a valuable, recorded piece of history.

git can allow you to do a great many things as a time traveler. One of the simplest is to just go back in history and observe. git checkout <hash> will move the view of your timeline back to the specific hash, but it does not destroy any work anyone has done. You can always git checkout HEAD to go back to the end of the timeline, or "now." In a way, this is a safe version of time travel. You cannot really mess anything up jumping around commits like this.

When you go back in time, git lets you know that anything you do in this checked-out state is ephemeral. If you want to make a change, git tells you to create a new branch. A branch is just a parallel timeline. Branches do not interact with the main timeline until you are ready to "merge" the parallel timeline. You can create, alter, and destroy branches as much as you want without affecting the main timeline.

When you are ready to merge a branch, the contents of the branch are added to the timeline as if it was there the whole time. git does some reconciliation to bring the branch into the main timeline (this is what the "merge" commits are in the history), and that alternate timeline/branch is now a part of the main timeline's history.

None of this alters existing history, it just appends to it. This is why (generally) branching, committing, and merging are safe workflows.

Rewriting History

This is great and all, but what happens when you need to actually change history, not just append to it. That's where git rebase comes into play. git rebase is a way to rewrite history. That power is as wonderous and scary as it sounds.

There are three times that I find git rebase is useful daily. The first is a branch sync, where I'm working on a remote branch and want to make sure everything is synced up. The second is hiding inconsequential commits from my local history before I push a branch up. The third is because a branch's parent has updated drastically, and I want the new code in my branch. Each of these instances requires me to rewrite what my local history looks like, so we need git rebase.

Why do we go through with all of this? The simplest reason for me is having a clean commit history. Rebasing allows us to make sure that a timeline looks clean with branches coming out and back in, and that the commits that exist matter. Anyone who comes along to work on the code should be able to quickly pinpoint various changes and lifecycles from the timeline. Rebasing allows us to keep a cleaner timeline through the ability to manipulate it.

Rebasing encompasses a list of different things under the hood, so let's look at the actual usages I find common and what is going on.

Branch Syncing

A command I use daily is git pull --rebase origin <branch> to pull remote changes into a local copy of a branch, especially the main branch of a project. While git pull by itself will bring down the changes, it does so by bringing down the changes and stuffing it around any local changes I have made but have not published. This is not a huge problem, but it can become a larger issue when conflicts occur.

Let's say our main branch histories look like this, and my local branch has an extra merge I did locally because of a bug fix:

// Remote
244426b0 2 hours ago - Merged branch feature/time-feature
a7ec8a44 3 days ago - Added Time class
a9cad80d 1 month ago - Deleted unneeded dependency on Vendor\Baz
21420248 1 month ago - Initial Commit

// Local
e454fa2e 10 hours ago - Merged branch bugfix/broken-dependencies
c23d5765 10 hours ago - Updated dependencies
a9cad80d 1 month ago - Deleted unneeded dependency on Vendor\Baz
21420248 1 month ago - Initial Commit

git pull will bring down the two missing commits, but the order will not be as clean as it will order the timeline based on commit time, not necessarily the order in which things were added to the timeline.

244426b0 2 hours ago - Merged branch feature/time-feature
e454fa2e 10 hours ago - Merged branch bugfix/broken-dependencies
c23d5765 10 hours ago - Updated dependencies
a7ec8a44 3 days ago - Added Time class
a9cad80d 1 month ago - Deleted unneeded dependency on Vendor\Baz
21420248 1 month ago - Initial Commit

This is not the worst thing in the world, but since I have not pushed my local commit I would prefer it not get mixed in with the existing timeline. git pull --rebase goes a step further and moves any local commits to the end of the timeline, after any remote commits. Our history ends up looking like this:

e454fa2e 1 minute ago - Merged branch bugfix/broken-dependencies
c23d5765 1 minute ago - Updated dependencies
244426b0 2 hours ago - Merged branch feature/time-feature
a7ec8a44 3 days ago - Added Time class
a9cad80d 1 month ago - Deleted unneeded dependency on Vendor\Baz
21420248 1 month ago - Initial Commit

It is a small change, but now our local work is moved and timestamped as after the other work. To me, this is a cleaner history and better group work being done together.

When is this safe?

This is generally safe on most branches. git will only rebase and move your local commits if they do not exist in the remote repository. You may still run into conflicts if you edit the same file as someone else, but the rebase will stop and you get a chance to fix things. Overall this works best when you keep this branch up-to-date as much as possible and are pushing your commits as much as possible. In the worst cases, you can git rebase --abort to stop the pull and rollback and do a traditional pull.

Hiding Unneeded Commits

The other daily rebase I do is when it comes to hiding commits that no one cares about. Have you ever seen a commit log like this?

$ git log --pretty=format:"%h %cr - %s"
b2b99fb 25 seconds ago - Updated dependencies
244426b 4 months ago - Added The Contracts of Open Source
a7ec8a4 1 year, 7 months ago - Fixed comments maybe
a9cad80 1 year, 7 months ago - Typos and fixes
2142024 1 year, 7 months ago - Added some responsiveness
1ba90ea 1 year, 7 months ago - Added false promise post
c23d576 1 year, 10 months ago - Fixed some more grammar issues
e454fa2 1 year, 10 months ago - Fixed typo
e1fd605 1 year, 11 months ago - Upgraded to sculpin 3
be8b7ec 2 years, 1 month ago - Overhaul of the talks and updated a ton of data
9f670ee 3 years, 1 month ago - Update .gitignore
00980fd 3 years, 5 months ago - Clarified a sentence
3ddd2f1 3 years, 5 months ago - Fixed typos
2df0ba3 3 years, 5 months ago - Added cron expression post

Commits in git should be atomic. Atomic commits are commits that do a single thing, but that single thing is a unit of work. In the case above, adding the "false promise" post is actually four total commits - 1ba90ea, 2142024, a9cad80, and a7ec8a4. The last three commits are fixing issues in the post, but the entire unit of work is comprised of those four commits. I should rewrite them as a single commit.

An interactive git rebase is the perfect tool for this job. In this case, we will tell git we want to rebase everything after a certain point, and by making it interactive git will let us manipulate history to clean it up. The first thing we'll want to do is figure out when we want to start to rewrite history. In our case, the new blog post was originally added in 1ba90ea, so we will want to go one step earlier to c23d576. This is the first hangup many people run into. The hash you specify is not included in the list to manipulate.

Now we just tell git to remove the locks and let us work:

git rebase -i c23d576

git will open a text editor and place all of the commits after (but not including) c23d576 into a nice little list for us. We are going back a bit in time so we are dragging in changes after the post, which is another potentially confusing area. What are we looking at?

pick 1ba90ea Added false promise article
pick 2142024 Added some responsiveness
pick a9cad80 Typos and fixes
pick a7ec8a4 Fixed comments maybe
pick 244426b Added The Contracts of Open Source
pick b2b99fb Updated dependencies

# Rebase c23d576..b2b99fb onto a7ec8a4 (6 commands)
#
# Commands:
# p, pick <commit> = use commit
# r, reword <commit> = use commit, but edit the commit message
# e, edit <commit> = use commit, but stop for amending
# s, squash <commit> = use commit, but meld into previous commit
# f, fixup <commit> = like "squash", but discard this commit's log message

The first four lines are the commits we can manipulate. The first word is a command, which the commented area at the bottom will detail. I have listed the most common five things you can do. The second column is the commit hash, and the rest of the line is the commit message. This screen allows us to queue up what we want to do and will execute it when we save and close this file.

What commands do we normally use?

• pickup - Just use the commit as is
• reword - Use the commit, but change the commit message
• edit - Use the commit, but stop and allow us to amend it with further changes
• squash - Use the commit, but merge it and the commit message with the previous commit
• fixup - Like squash, but ignore the commit message

What we want to do is create a single unit of work for adding the post, but we do not want to lose the last three commits. We have two options - squash or fixup. Since history does not care that I made typos or fixed some issues with responsiveness, we will go with "fixup". We will edit the lines to look like this:

pick 1ba90ea Added false promise article
fixup 2142024 Added some responsiveness
fixup a9cad80 Typos and fixes
fixup a7ec8a4 Fixed comments maybe
pick 244426b Added The Contracts of Open Source
pick b2b99fb Updated dependencies

We can then save the file and exit the text editor (^X in nano, :wq in vim). git will then start to do the commands we told it to.

1. "pick" 1ba90ea and use it as-is
2. "fixup" 1ba90ea by adding the changes from 2142024
3. "fixup" 1ba90ea by adding the changes from a9cad80
4. "fixup" 1ba90ea by adding the changes from a7ec8a4
5. "pick" 244426b and use it as-is
6. "pick" b2b99fb and use it as-is

If we look at the log now we will see that those other commits have disappeared, but if we look at the files all those changes are still intact:

$ git log --pretty=format:"%h %cr - %s"
ed4d525 64 seconds ago - Updated dependencies
231fe64 64 seconds ago - Added The Contracts of Open Source
8eda557 64 seconds ago - Added false promise article
c23d576 1 year, 10 months ago - Fixed some more grammar issues
e454fa2 1 year, 10 months ago - Fixed typo
e1fd605 1 year, 11 months ago - Upgraded to sculpin 3

We rewrote history to get rid of my typos! Now no one will need to know that I am a poor speller (no comments on this post about spelling mistakes. That's what Twitter DMs are for).

What happened? git rolled back our repository to c23d576 - we went back in time and undid everything after that point. git, however, remembers all the commits after that, so began to rebuild history at that point using our instructions. git began layering on those old commits, but this was new work so we get new hashes. The three "fixup" commits are effectively wiped from history as the pointers shift to make it look like c23d576 went directly to a7ec8a4. A new commit is generated, 8eda557, to symbolize that new history. The last two commits are layered in the same way, with new hashes to symbolize their new place in the timeline.

A side effect of this is the timestamps now all show our fixed-up commit, as well as everything after it, looks like they were done "now" (or 64 seconds ago as I live test all of this). That is because those changes were rewritten 64 seconds ago. We altered history, so git accurately reflects that. It will not lie, and it is not a lie to say we changed 231fe64 and ed4d525 64 seconds ago - they were part of the rebase.

In the end, we are left with a single, atomic commit for a post. This provides a cleaner history and a much more useful history. We can see when the post was committed, but we hide those fixes because ultimately all history needs to worry about is that the post was added.

What happened to those original commits? They still exist, but we no longer look at them in the timeline. Technically the timeline actually looks more like this, with those original commits almost like a branch from our rebase start. You could actually checkout b2b99fb and trace its history back to the original rebase commit as it exists in something called the "reflog", which is the full commit history of the repository. For all intents and purposes, it is no longer part of the timeline we are on.

When is this safe?

This type of rebasing is only safe 100% of the time in two instances. The first is on code YOU HAVE NOT PUSHED. I routinely make small commits working toward a goal or fixing bugs I introduce in code. Having those commits are great ways to roll back if things go off the rails. When you are happy with your code, rebase and fixup/squash all those down. This is easy to deal with as it does not deal with force pushing.

The other time this type of rebasing is safe is when YOU are the only one to be working on a branch. In many instances, we will be working in separate feature branches all on our own. In those cases, it's perfectly fine to rebase commits in that branch and push them up for review. I normally do this when I have very small units of work that need to be done, like with a few lines of code. If I have a larger unit of work, like say a new feature, I will tend to fixup locally but keep a history of the larger blocks of work being done. Keep in mind that this will require a force push, which will forcibly reset the remote branch to mirror your local branch.

If someone else is working on a branch with you, DO NOT REBASE AND FORCE PUSH to a remote branch. This will cause issues with other collaborators. Yes, it's all solvable, but it can cause a lot of issues as everyone tries to reconcile the force pushes.

Pulling In Parent Branch Changes

This situation is the one where most people run into issues. This workflow is when you have a feature branch that comes off your mainline code branch, which we will call main. main is getting updated constantly with other feature branches being merged in, and your feature branch needs to be updated to take advantage of those changes. You have two options. The first is to pull down and merge main into your feature branch. This will create a merge commit, and git will do its best to figure out how to merge the changes together.

The second option is a rebase on main. As with our interactive rebase this will effectively move all of our feature branch commits to be after the current version of main and start our branch there. Our branch structure is a bit cleaner as we can better see that we depend on code from the 5aac552 commit rather than the cfe88cd commit we originally branched off of.

When is this safe?

It's always safe.

But Chris, this always causes conflicts!

Many times it does, but not because of the reason you think. The conflict occurs when git attempts to take your existing commit and reconcile it against the new files. If one of those files from main is a file you also edited, you may get a conflict. git tries its best to figure out how to merge changes but sometimes it cannot. This means you have to figure out how to reconcile it, or git rebase --abort and stop the rebase.

The larger issue at hand is that two developers changed the same blocks of code during two different feature sets. The solution is not to stop using rebase, but to better understand the scope of feature branches and make sure that work is not being done concurrently on the same portion of the codebase. This problem is exacerbated when a feature branch is very wide in scope or has a long lifetime as this widens the number of files that can be altered.

If you are constantly running into issues rebasing on a parent branch, first look at the work being scheduled. Make sure that the issues being worked on do not overlap in scope or code. Second, make sure that feature branches are short-lived. A branch that exists and is worked on for weeks or months at a time is grossly over-scoped. Break it into smaller feature branches or units of work in your planning. A good rule of thumb is that any issue that takes more than 8 person-hours is not broken up enough.

Your other option is to not use a rebase and just merge up from main. Rebasing is not an always or never situation.

"I'm worried about losing work"

Rebasing generally does not cause work to get lost, but resolving merge commits do. This is the biggest issue with most rebasing problems.

One of the most common mistakes is taking a feature branch and rebasing the parent. If there are a lot of differences as outlined above, you get a merge conflict. You now have to look at the code and figure out the best way to fix this, and there is this monster staring, urging you to finish the rebase as quickly as possible. You are stuck until you fix this.

You resolve the merge, and after-the-fact realize you lost a chunk of code, or you accidentally reverted the code to an earlier version. Now you are out of sync with main. You squashed something wrong and now the commit you needed is gone. What do you do?

The nice thing about git is that nothing is ever lost. If you need to find a commit, that is where the git reflog comes into play. The reflog is an activity log of what you have done with your local repository. You can use the reflog to find older commits and check them out, or reset them, or cherry-pick them back into the current timeline. It is not a complete history, however, and is designed to clean itself out every so often. The reflog is not a magical backup tool.

$ git reflog show
d4d525 (HEAD -> master) HEAD@{0}: rebase -i (finish): returning to refs/heads/master
ed4d525 (HEAD -> master) HEAD@{1}: rebase -i (pick): Updated dependencies
231fe64 HEAD@{2}: rebase -i (pick): Added The Contracts of Open Source
8eda557 HEAD@{3}: rebase -i (fixup): Added false promise article
5eec634 HEAD@{4}: rebase -i (fixup): # This is a combination of 3 commits.
083b44f HEAD@{5}: rebase -i (fixup): # This is a combination of 2 commits.
1ba90ea HEAD@{6}: rebase -i (start): checkout c23d576
b2b99fb HEAD@{7}: checkout: moving from 244426b03db6d06d62d4a6bedcb64c4baef3acb4 to master
244426b (origin/master, origin/HEAD) HEAD@{8}: pull --rebase origin master: checkout 244426b03db6d06d62d4a6bedcb64c4baef3acb4

Remember that earlier example where we ran a rebase and "fixup" 'd a few commits? This is what that rebase looks like in the reflog. I can git reset or git checkout any of the commits in the reflog. By default the reflog only shows the current branch, you can use git reflog show --all to set all the activity no matter the branch.

"Force pushing is scary"

Force pushing is scary because it can cause a lot of problems if other people do not realize you force pushed. Force pushing resets the timeline so that the remote branch matches your local one completely. The issue here is that if multiple people are also pulling down a branch from a central repository, a force push can quickly get them out of sync. This is most destructive when multiple people are committing to the same branch, and someone force pushes to that shared branch.

The good thing is the solution is easy! NEVER FORCE PUSH TO A SHARED BRANCH. Just do not do it.

If you must force push to a branch, then alert the other developers working on that branch. They will have two things they can do. The first is to fetch the new branch and git reset themselves to match. The reflog can be used to git cherry-pick their lost commits over.

The second is to do a git pull --rebase to try and automatically reconcile the branch. Depending on what caused the force push in the first place you may have merge conflicts, but hopefully, the reason and changes are clearly communicated by whoever did the force push.

Worst case you can always create temporary branches to play around in before you mess with your local branch.

"I hate all the merge conflicts and constant merges"

This goes back to one of my earlier notes: try and limit the scope of feature branches and make sure branches are short-lived. If you have lots of overlapping work between branches you are going to run into conflicts at some point anyway.

"I wish there was a way to see the outcome first"

The best way to do this is to create a temporary branch. Let's say we have a feature branch and we want to rebase on main, but see how that looks first.

// Assuming we are on our feature branch `feature/cool-feature`
$ git fetch origin main:main
$ git checkout -b dr-rebase-cool-feature
$ git rebase main
// See what happens and abort of fails

Since git allows you to create and throw away branches with ease, branching should be taken advantage of when possible.

Hopefully, this helps with showing when rebasing can be used, and what to do when things go wrong. Always remember you can git reset and use the reflog to pull back missing commits, and never force push to a shared branch!

I changed PHPUnit's policy for PHP support going forward. I am working hard on adding support for PHP 8 to PHPUnit 8. Yet I am still confronted with a constant stream of "Sebastian is not doing enough" and "Sebastian harms the PHP ecosystem".

Many people do not realize how old the concept of open-source software is, or that that software basically started as a shared development experience. In the 1950s computers did not come with software for the most part. Developers would be forced to develop software for the platforms that they had access to. Since many of these early developers were introduced to computers during college, one of the few institutions that could afford even machines like the TX-0 or the A-2, the general ideas of academia bled into software development.

These developers, be they students or faculty at the schools, would share their software just like they would share knowledge. This fed quite well with the early hacking ideas of "Information Should Be Free." If someone developed an algorithm or a utility it was shared amongst everyone else. Early hacking culture introduced the idea that it was also fine to modify the software. The unwritten rule was that it was shared.

Software in its infancy was open source, we just did not have a name for it. The idea of commercial software was not even considered. Software from even the machine vendors was just considered part of the machine. You purchased hardware, not software. You created the software and shared it with your colleagues. If you had a problem with the software you could talk to the author, or you fixed it yourself. There was no expectation other than the sharing of knowledge.

Fast forward a few decades and commercial software start to appear. As machines become somewhat more standardized and development starts to become more costly, developers like Micro-Soft start not sharing their software but selling their software. Software becomes a product and begins to unbundle from the hardware. Need a compiler? That will be an extra cost of $60-$75 on top of the hardware.

The ideas of open source never went away. Systems like the Berkeley Software Distribution for Unix flourished despite AT&T's attempts to lock down the licensing of the early Unix source code. The GNU Project was formed in 1983 with the explicit purpose of making sure that users keep control over the software that they run, and have the freedom to modify it as they see fit. In keeping with the early developers and hackers, the GNU Projects forced developers to share their modifications but gave them the ability to study and copy software as well.

Much of the modern web only exists because of open-source software. Apache's httpd has and continues to power a huge swath of the Internet as a whole. Subsystems for communication like e-mail are more often than not powered by open-source software. Windows had a networking stack lifted from BSD. Companies like Mozilla and Redhat power the internet and give developers and users open-source options for their software and operating systems. If open-source software would disappear, the tech world would come to a screeching halt.

When a developer decides to put their time toward open-source software, they are giving up their time and sharing their expertise. Despite all the grumblings about not owing users anything, most open source developers spend time working with and listening to users and other collaborators to make their software better. They build tools and programs to make their lives, and by extension others' lives, better.

There is a contract that is established when using open-source software. I do not mean the license, a legal document that spells out what you can and cannot do with a piece of software. There is a deeper moral contract that both the developer and the end-user enter into.

The contract says that the developer of the software is giving up their time for the greater good. To make sure you can use the software as you see fit, they give up the secrets of that software so that the user has access to their knowledge. The user can dig into the code and see how it works. The user can change the software as they see fit, no matter what the original developer designed. The user should share their knowledge about the software with the rest of the world through patches and collaboration.

The contract does not state that the original developer must follow the whims of the users. The contract does not state that the original developer owes the users anything. If a user feels that something is lacking, the user is expected to collaborate or do the work themselves. Are the docs lacking? Anyone can write docs for open-source software. The original developer is free to ignore the work done by users if they see fit. The users are just as free to post the docs even if they are not wanted.

I find it despicable what Sebastian had to endure with PHPUnit and PHP 8 support. Sebastian, as the original developer and maintainer of PHPUnit, is more than free to dictate how he spends his time working on the software. If there is a demand for PHP 8 support, he is more than free to weigh the pros and cons of adding that to older versions despite what he has already noted as his support structure for releases.

If you wanted PHP 8 support in older versions, you are more than welcome to fork the software and add support for PHP 8. You are welcome to work with Sebastian to try and get it into the official releases, but Sebastian is not under any obligation to give in to the masses. He has already given the tools needed to modify PHPUnit freely to the world. His knowledge is laid bare in the source code.

If PHPUnit is not to your liking, study the code, and modify it yourself. Distribute the patches and the forks back out to the world just like Sebastian did. That is the power that open source gives you under the contract.

I think a large number of developers forget that many open source projects are run by individuals. These individuals seldom make money directly on the software, but hold other jobs to pay the bills or maybe sell ancillary services around the software they build. Unless you are specifically paying Sebastian as a contractor to modify PHPUnit to your liking, you do not get to demand anything of him or any maintainer. You can ask, and he can say "No." If you do not like that answer, the source code is there for you to change.

When a library is changing too quickly for your liking, you are free to stay on an older version. Most developers do not remove older unsupported versions of libraries or applications just because a new version is out. If a developer wants to support older versions of their software, that is a decision they make. That is not a decision the users make. You are making a conscious decision to stay on that older version, and you need to live with the consequences of that decision.

If your hands are tied because of an outside force and are unable to upgrade, the only advice I can give to you is to implore those that have the power to make that decision to upgrade. If it is a boss or a CEO, make the case. Explain why upgrading is beneficial, and how staying behind is becoming a drain on development. You must complain to those in power about your situation, not to the developer of the libraries and tools you use. Those developers' obligation to you ends when they share their knowledge and software.

When you use open source software, remember that it was built on the ideas of "Information should be free," collaboration, and the betterment of everyone.

There are humans behind that source code you are using.

As a "hot take" to this, Sherri W. (@SyntaxSeed) responded with:

Hot take: #PHP's release cycle is too fast. 😰 https://t.co/YfvBC7yGRa

— SyntaxSeed (Sherri W) (@SyntaxSeed) August 31, 2019

I responded to this with my own thoughts:

The alternative is too slow :(

Right now it's roughly 12 months. Before 5.4 it swung between 6 months to nearly 3 years, depending on the release. At least now it is consistent.

Not that anyone ever upgrades anyway. Java and Python devs stay on outdated versions all the time.

— Chris Tankersley (@dragonmantank) August 31, 2019

From there other people joined into a bit of discourse over whether or not a long or short release cycle helps developers. Developers weighed in on both sides.

The Arguments For an LTS Release Cycle

Clients Won't Pay for Upgrades

From Sherri's perspective as a freelancer with 20-30 clients, it is hard to get a client to pay money just because the underlying language has upgraded. We already have problems trying to justify why we should pay for testing, so coming back to a client a year or two after a project is finished just to pay for a non-functionality-adding upgrading can be a hard sell.

I understand the reasoning. I had two clients that were on PHP 5.2 for a very, very long time. When I say "long time," I mean PHP 5.2 had been released in 2006, and these projects were from still in use well into PHP 5.5's release.

The first was a small Bed and Breakfast site that was written into Wordpress. The reservation system that they used was source encrypted with IonCube, a source encryption extension for PHP. The client refused to pay for an upgrade for this plugin, and since it was wrapped in IonCube we could not manually upgrade it. It refused to work on PHP 5.3 or anything higher. I and the original contractor who worked with her could not get it to work.

The second project was a local government project. They had a loaned server that had been paid with through donations that ran Windows 2000 and was hosted at a local library. Since it had been paid and maintained through donations, it was locked to this hardware. The library would only support the machine if it worked with the AD controller. That left us on Windows 2000.

This was around what would be the end of PHP 5.3's life. When 5.4 was released I contacted them about upgrading, especially because Zend Framework 1 was well outdated as well. There was no money for an upgrade at the time.

In both cases, it was a business decision motivated by money that these pieces of software stay at 5.2. They both stayed at 5.2 for a very, very long time.

From Sherri's tweets, she is in much the same boat - many customers just do not want to pay for arbitrary upgrades for infrastructure. You could try and bundle it with new features, but then they may balk at the cost and still decline the project. If releases were slower, they could be tied to more major upgrades.

Business Can Move Slow

Lars Moelleken (@suckup_de) mentioned that sometimes business processes move slower than release cycles. This means that businesses that need stability look toward an LTS release to provide that stability while the business can still provide value for the life of a project.

I have seen this as well. One project I worked on had many different requirements, which included the list of allowed operating systems and software requirements. We had to work with a series of hardware that only worked with specific Linux kernels, and some distributions shipped with versions of libraries we needed (specifically a version of OpenSSL that had some hardening patches applied).

We also had to be very cognizant of changes to the codebase. We had to be careful not to break anything, as loss of functionality could have some very bad consequences for our uses. Getting patches installed for bugs was done in months, not days.

This meant that many of our software stayed on older versions of languages or libraries. Python and PHP were both well out of EOL when I started at the company. When I left, at least PHP was at 5.6, and Python had started a crawl toward Python 3. The underlying OS had not changed, but because the distro did not support rolling upgrades we could do little for in-place upgrades.

We had planned on upgrading all of this, but much of it was tied to a sales cycle and maintenance timeframes. We would have to maintain two versions of the software which was an additional cost for us. It was decided that we would try and upgrade what we could when we had time, and try to push the customer to a new sales cycle which would allow us to switch them over.

We Just Can't

This argument came up during our weekly group get-togethers where we just have a video call and hang out for an hour. The main focus had been a coworker who used to work for an insurance company that did most of their work in Java 6.

When he started, he wanted to use some newer best practices and libraries that would have made their lives easier. There was a lot of pushback from doing this from various sides.

Many of the arguments revolved around either that there was not enough time, or previous consultants had already decided that "Solution X" was a bad fit for the company. A few developers mentioned that some of the new things would just never work for the current solution due to "technical restraints."

In this case, the development team decided that it would be too much work to push forward with an upgrade. Java 6 still worked, so it was better to just continue to deliver functionality with the current setup. Maybe new projects could allower newer setups.

Why LTS is Bad

In all three of the above excuses, actual reasons had been put forth as to why a slower release cycle would be better.

All of them are completely invalid and just excuses for not doing work. I understand the why of each argument. I just do not accept them because, in the long run, they are just causing more work and more pain in the upgrade process. This makes it even hard to justify upgrades because they will cost more and take more time, and are much more prone to failure.

If There Isn't Money Now, There Won't Be In The Future

In the Bed and Breakfast case, she ended up paying for a server all to herself running PHP 5.2 and an older operating system. This also required her to sign off on a security waiver stating that she understood. I am not 100 percent sure she really did understand, otherwise, she could have paid for the new version of the plugin. As a contractor, I had to protect myself.

By the time I had stopped consulting for her, the plugin was not even maintained anymore - she would have to pay for an entirely revamped reservation system. The cost went from what I think was $75 (at the time) for the plugin upgrade to nearly $2,000 to just replicate what the old plugin did.

The Zend Framework 1 application is still in use. I just checked and it was moved to a host running PHP 7.0. The project was never upgraded. In fact, I know this because there were some workarounds I had to do to get Zend Framework to run under Window 2000's version of IIS. The site is now running under Apache httpd, according to the headers, but still has those workarounds. They just moved it. It was a simple application with no private data so I am not really worried from a security standpoint, but no one has bothered to upgrade it.

They have contacted me on-and-off through the years about doing upgrades, but each time the cost is bundled with an upgrade to something newer, and well outside of the price range they want to pay for changes.

If a framework, OS distribution, or language has an LTS release, this increases the length of time between supported releases. This adds additional complexity to upgrades, which increases costs. The increased cost and time are usually seen as a waste because of no new tangible benefit from the increase. Why pay for something that does not add new features or revenue?

Frameworks like Symfony do a good job of having their final releases in a version be somewhat compatible with the new version, making upgrades easier. Even with 3.4 being LTS, the next LTS is 4.4... which if developers are not upgrading now, that means they are waiting for the next LTS, which is going to take longer and therefore cost more, to implement.

If Your Business Moves at a Glacial Pace, That's Your Fault

Saying that a business moves slowly, and therefore release cycles should move slowly, is a farce. I never accept this is a good answer. In fact, Sara Golemon can back this up:

5.6 got extended support. Rather than it giving those users more time to transition, it just gave those users more time to get stuck behind an ever more daunting wall of upgrades.

— SaraMG (@SaraMG) August 31, 2019

Much like putting off an upgrade because there is no money in the budget, purposefully putting off upgrades leads to the exact same problem - you push an upgrade off until the point it's painful, and the amount of time and money has now increased. Going for Symfony 3.4 to 5.x will not be straightforward. Moving from Ubuntu 14.04 to 18.04 will cause a lot of things to break.

You now are forced to spend more money and time than if you had just kept up with upgrades and updates. Rewriting software from scratch is more expensive than refactoring.

I fought very hard to move to PHP 7 and Python 3 on the one project, and to upgrade the underlying OS. During my tenure, we went from 5.4 to 5.6 with nothing but package upgrades and got those into production without the clients ever noticing.

We did actually get the PHP 5.6 to 7.2 code migration finished (just not put into production) by the time I left. Since my predecessor and myself took great care to use best practices, the actual number of things that phpstan found were fixed in a few hours. Unit tests were added around them to make sure nothing broke.

The Python code was a mess and primarily 2.6, so it was mostly a lost cause. A rewrite was started that included tests upfront. It was not completed when I left, but it was light-years ahead of where the 2.6 code was. The only problem was it meant pulling our lead Python developer off for a few months to do the work, pushing back a release.

I cannot find the tweet for the life of me, but someone brought up the longevity of developers. Since job movement is fairly frequent in our industry, leaving an upgrade for two to three years can mean the loss of knowledge that is required for these upgrades to go smoothly.

By saying that your business processes move slow, and accepting that, you are only making it harder on yourself, or the people that come after you. You are costing your company more money in the long run.

You Can, You Just Don't Want To

This is usually where most developers end up when it comes to legacy code. The code is in such bad shape that it is hard to fix, so there is an unconscious bias to upgrading it. They are worried about having to go to their boss and explain why they need to upgrade and are afraid of being shot down. It is easier to just stay the course and develop features.

As with most of these situations, you are just delaying the inevitable. You are going to have to upgrade someday, and you do not want that someday to be when a massive CVE comes out of nowhere that you have to handle.

Ben Ramsey (@ramsey) does bring up a good point:

The major Linux distros all maintain what are effectively LTS versions of PHP. They are official to the respective distros, just not to the PHP project itself. The distro maintainers backport security patches, even when PHP core does not.

— Ben Ramsey (@ramsey) August 31, 2019

Except that you are at the mercy of a maintainer who then decides if a security feature should be backported. As these are backported, they introduce divergences in the codebases, both from the upstream provider (say PHP Internals) as well as have the possibility of changing behavior. There will be some security fixes that cannot be backported because they deal with newer or changed code, so now the maintainer has to decide on re-implementing the fix or leaving it out.

If you hold off because "someone else provides support," or "we just use what is in the repositories," you are giving up and deciding to stay where you are. It will only end in you still being behind and spending more time and money when you have to make an upgrade.

And if you are holding off because you still use mysql_* functions... Stop. Get off your butt and change it. You've literally had years to fix this.

What Can You Do?

First and foremost, start your planning now. Depending on the quality of your application and the age of your infrastructure, you may have little or a lot of work to do. The sooner you start planning the easier time you will have.

Sell The Upgrade

You can start small and start to make some changes right away while you make the business case. Explain to upper management how not doing these upgrades are going to leave you in a bad spot. Here are a handful of things you can use:

• Finding developers who want to work on older software is always hard. Finding developers who want to work in old languages is harder.
• Even with backported security you are at a security disadvantage. It takes time for the backports to happen if they actually happen at all.
• Libraries and tools move on. You will be left with substandard tooling compared to competitors who stay up-to-date.
• As libraries update, that leaves you to maintain them (especially manually back-porting security patches). This is more work for your developers, and less time you can put toward new features that matter.
• To do an upgrade at a later date means spending even more time not working on new features. This can put you behind competitors.
• If you use AWS/Azure, moving to newer PHP versions can get an automatic optimization, meaning fewer servers, which means less cost.

If you find it impossible to sell doing the upgrades, you have two options - do it anyway, or leave.

If you think you can get away with it, or you have the power to do it, go ahead and just do the upgrades. If you really look into it you might find straight version upgrades are trivial, but worst case you will get an accurate estimate on how long the upgrade will take. Remember, the longer you delay, the longer the upgrade will take.

If a company cannot take the time to understand why being up-to-date is a business advantage, then move on to a company that does understand it.

Run Multiple Versions of PHP

I use both Docker and phpenv to handle multiple versions of PHP on a single machine. I can switch between them with some changes and try out my code as I upgrade.

For Docker, you should just need to change your Dockerfile or switch containers out. It will depend on how your setup is configured. A huge selling point of Docker is the ability to swap out containers, so if you are using Docker (and honestly, if you are using Docker but can't upgrade PHP... WTF?!), this should be fairly easy.

For locally installed PHP, I love phpenv. It allows you to have multiple versions installed at once, and has directions for setting up both PHP-FPM and Apache httpd.

Laravel Homestead is one other option. It is a vagrant-based virtual machine with PHP 5.6 through PHP 7.3. Even if you do not use Laravel, you can throw a normal PHP application in there as well and start switching PHP versions.

PHP tries very hard to keep backward compatibility, so unless you are using a deprecated feature like mysql_* functions your app might just work out of the box.

Figure Out Code Changes

Look at upgrading your PHP version first. If you are on PHP 7.0 or 7.1, great! PHP does an awesome job at adhering to SemVer, so there should be little work you need to do for the minor versions. The PHP manual contains release and migration notes for each version since 5.0. Read the migration notes for each version:

• 5.x to 7.0
• 7.0 to 7.1
• 7.1 to 7.2
• 7.2 to 7.3

Ignore new features and focus on any changes you need to make.

Tools like phpstan can check your code against PHP 7 and make suggestions on things you will need to change. As I mentioned before, I took our PHP 5.6 codebase and ran it against PHP 7.2 and only had a handful of things to change. You may have more, but it gives you a detailed list of what needs to be fixed.

Actually Upgrade

Most mainline distributions have good maintainers that keep PHP up-to-date. These packages go through the same process for inclusion as any other package, so convince your systems or operations team to update. If they provide pushback, you will need to come up with a good business reason (the packages are official, safe, newer PHP has better security support, is faster, etc). Since they are official repositories, it is not that hard for them to get added into a system. It's not like the operations team needs to compile it themselves.

For Ubuntu/Debian there is the set of packages from Ondřej Surý, available at https://deb.sury.org/. He has worked for years to provide high-quality Debian packages, and all of the PHP packages are either directly from him or based on his packages.

On RHEL/CentOS/Fedora, you have packages from Remi Collet, available at https://rpms.remirepo.net/. He maintains packages for core PHP as well as a bunch of extensions, for various versions of PHP. As with Ondřej, Remi is the package maintainer for Fedora, so these are as official and safe as you are going to find for RPM-based systems.

Don't Delay, Start Now

I hope at this point I have convinced you why something as nice sounding as LTS releases are not as cozy and safe as they make themselves out to be. You are sacrificing time and money later for perceived stability today.

A project that stays up-to-date, and puts into processes that help update in real-time, will be able to stay competitive longer. If security is something that is an ingrained part of software development, why isn't upgrading? Like security, upgrading isn't something you bolt-on, or do later.

Stop making excuses, and start upgrading.

First off, I deleted my old sculpin.json and sculpin.lock files. I still used them because I'm a horrible person, and the old deploy system I used to use still used the phar version. Funny enough, I used vendor/bin/sculpin locally since I had long since stopped using PHP 5, but my old build system for the blog still ran PHP 5.6 and used the phar. I know, I know. That's all been updated now.

I then deleted my composer.lock file and updated my composer.json to use the version 3 tags for Sculpin:

{
    "require": {
        "sculpin/sculpin": "~3.0"
    }
}

As expected, when I ran composer install it nuked a bunch of old libraries and dragged in all the new ones. This was a super simple composer.json, so there was not any other conflicts.

I then ran vendor/bin/sculpin generate --server --watch to build the site and see what happened. I did have two very minor issues. One was that I had a title: front matter that started with the back tick ("`"). I removed that because it was not a big deal. The other issue was that one of my source files was missing an extension, and it would give the following error:

Exception: Argument 2 passed to Sculpin\Core\Formatter\FormatterManager::formatBlocks() must be of the type string, null given

This took a bit of tracking down, but it looks like if the extension is missing Sculpin cannot figure out the type of content it is, and therefore does not process it. When it grabs the content, it gets a "null" value, and the system complains. Another small issue, so adding ".md" to that file fixed it right away! I'll be submitting a bug report for that.

Overall though, the process took about 20 minutes to upgrade, with almost all of that time being the missing extension problem. Sculpin continues to be a very simple, robust, and yet easy-to-use system, and I love it for that.

For those that do not know what the library does, cron-expression just checks to see if a cron expression (something like 0 0 * * *) is valid, can check to see if it matches the current time and needs to run, and can determine future run dates. If you need a simple way to schedule things, cron itself is a very useful and well understood syntax. cron-expression does not run your code though, it is mostly a validation library.

Much like Sculpin, it's a pretty stable project so there wasn't a ton of movement on it development-wise. Some bug fixes here, few enchancements there, but nothing major. At the beginning of 2017 I pushed out a 1.2.0 release. I had decided that I would only support PHP 7.0 going forward. By this time I had learned that Laravel was using this library under the hood, so I wanted to get one final release done under the older PHP 5.x branch. v2.x and later would all be PHP 7.x compatible.

Then I started digging into a bug, and that bug turned into a few bugs, in regards to validation. As it turns out, the regex that the library used was really loose and let a lot of stuff through. This did not seem to affect valid expressions, but it allowed a lot of junk through. As time went on more and more reports with this started coming through. The underlying logic had to change. I started working on this as the main focus.

Then I got this bug report - #153, "Wrong nextRunDate for * rules". Long story short, step ranges in the library were broken. Someone had discovered a bug in Laravel's cron system that caused the expression to validate on the incorrect set of months. Even I misunderstood how that worked, so I ended up diving into the source code for cronie, one of the main cron systems shipped with Linux systems.

cron-expression had gotten our implementation completely wrong. I re-implemented a bunch of our validation logic to the same basic way that cronie does, and this actually ended up not only fixing our stepping issue but also our data validation woes. The new code was a bit more compact and more unified in how the library does validations. Overall this was good.

There was a big problem though, and that was that this was a huge backward compatibility break. When a bug has survived long enough and people rely on that behavior, it is no longer a bug - it's a feature. So our bad stepping fix has the potentional for breaking a huge number of systems, even if that behavior is bad. People rely on it.

Sufficient time has past for a v2.0.0 release. That will be happening today. All the fixes will be available in packagist as soon as it updates.

The repo will also be moving to a new repository: dragonmantank/cron-expression. The reasons for this are twofold - one, I am not and cannot be the admin of the original repo as it is not an organization, it is a personal repo. I cannot wire in new build or checking systems at all. Two, this is the perfect time to do this break. v2.0.0 is incompatible with v1.0.0 because of the stepping issue, and this will let frameworks or other installs that rely on the v1.x branch to move at their leisure without breaking them.

The old repo will no longer be maintained, but I will still watch the issues. The existing issues will still be evaluated and looked at, just implemented in the new repo. The old package will remain in packagist for those that need it. All new work will be done in the new repo against the 2.x branch.

If anyone has any questions, feel free to hit me up on twitter at @dragonmantank.

My youngest son said it would be cool to have one, and asked if we could get one. I said no. My wife and I are on the same page about this, but the idea of a device, which I have no control over, listening to everything being said is not something we want in our house. It's not just me not liking the Amazon Echo, either - I don't want a Google Home in the house either.

That lead to a discussion about why having a listening device in the home is bad. We expect a certain amount of privacy in our own home regardless of the fact that we are not doing anything against the law. I just do not want my private conversations overheard by a device that sends all of that back to a server, where it sits forever. Police have already tried to get Echo recordings for a murder, though if Amazon is to be believed unless someone said "Alexa, help me!" nothing should have been recorded. Even if it had recorded something, Amazon states that such voice recordings are encrypted.

Knowing how well software is built and how often "encrypted" data gets accessed means I do not want my words recorded and stored on Amazon's, or anyone else's, servers. Hell, I work for a company who designs and sells a network appliance to find bad traffic on networks. When someone has access to servers or the network, getting access to information is trivial. Amazon now also sells Echo Look, which is a camera that currently helps you dress fashionably. I do not even have to talk about how creeped out that would make me feel.

We grow increasingly reliant upon companies that make our lives more convenient. I've used Google's e-mail, calendaring, and document storage services for years because it was easy to use, worked directly with my phone, and meant I did not have to worry about e-mail. There are some nice perks to that, like online document editing, having airline data directly parsed and made available, intelligent spam filtering, and device syncing, all to name a few.

If I do not want my speech hosted on Amazon or Google servers... why my textual life hosted and sifted through by Google?

Taking Back My E-mail

The first thing I've decided to move off of Google, and back into my own control, is my e-mail.

I have a lot of e-mail addresses, and I have been attempting to consolidate them into just a few. Google made that pretty easy. I'm grandfathered into the old G Suite setup of it being free for 100 users, but I took liberal advantage of domain aliases and catchall e-mail addresses.

I looked at services like FastMail, ProtonMail, and Kolab Now. All three of them are highly regarded, with Kolab and ProtonMail being open source projects. Moving my domains and setting up aliases though, that would end up being very, very costly. Kolab charges around $50 for just setting up a single domain alias. FastMail and ProtonMail would start to get very pricy as I moved all my domains over.

ProtonMail also lost points as I would have to use a web browser on my desktop. I want my e-mail in any app of my chosing. I am not paranoid enough to think someone is trying to get into my e-mail, so the security aspect of ProtonMail was not a huge selling point.

I decided to host my own e-mail.

Running My Own Server

"Email is one of the bastions of the decentralised Internet and we should hang onto it" - Nux, Hacker News

I'm not afraid of servers or their maintenance at all. My career started with maintaining servers and dealing with configuring them, so why not just run my own e-mail server?

I know, I know. I should not run my own e-mail server because:

• There are lots of moving parts
• It's not just e-mail, its virus scanning, spam filtering, e-mail access
• Maintenance is time consuming
• Blacklist maintainers are cold, heartless beings that never remove IPs
• Russians will hack me
• E-mail isn't secure
• I have to trust my host

Frankly, most of the above is FUD. If we, as developers, are telling people to run things like Docker or set up their own VPS because "it's the right way to run a web app," then running an e-mail server should not be some scary thing. Granted, I am not going into this blind as I've set up an e-mail server before, but come on people. It isn't that bad.

I do want to cut down on the amount of work I have to do. I first looked at Mail-in-a-Box, which is a set of scripts that sets up a mail server. I decided against it as it is pretty much all or nothing. You run and set up the box the way it wants to be set up and that's it. Want to do something else with the box? Too bad.

I then found sovereign. It is a set of Ansible playbooks that set up a server that includes e-mail as one of the various services. Since it is just based on Ansible configuration and I know how to work with that, I decided on sovereign.

Setting up the Server

The Server

I use Digital Ocean for a lot of projects. As I said before, privacy from foreign powers is not a current concern I have so hosting a server in the US is fine for the moment. I created a VPS with Debian 8 as that was what sovereign recommended.

The next thing I did was check the assigned IP on http://multirbl.valli.org/. This site will check a bunch of well used DNS blacklists to see if the IP that Digital Ocean gave me has had a shady history. The first one... well, once it hit twenty blacklists I deleted the VM and rebuilt it on a different server.

The second one was only on four blacklists. That is a much more manageable number to deal with. Most blacklists are fairly easy to get removed from, and if I'm only on four I will take my chances.

With that sorted out I followed through the rest of the instructions in the sovereign README file. It took only a few minutes of prep before running the Ansible playbooks.

I started off with a domain that did not previously have e-mail associated with it, to test things out. That way if it all went to Hell I wouldn't lose any e-mail. Ran the scripts and after about 15 minutes ran out of memory on the server.

I tried to work around it, but with everything running 512mb was not big enough. I deleted the server and reprovisioned a bigger one. Not only did it have more memory, it also had more hard drive space.

That worked better. About 20 minutes later I had a server up and running!

Shutting down Services

sovereign comes with a bunch of services installed, and since this was my first run through I let it install everything. Once I confirmed everything was working well, I SSH'd into the server and disabled a bunch of stuff I did not need, like ZNC. I happily pay IRCCloud for IRC bouncing.

Most servers are compromised because of services running on the box. It is rare that an actual OS exploit is the problem. I removed the services I did not need from the site.yml file, and shut down services I did not need.

I did want to keep the webmail so I just disabled a bunch of vhosts as well. So far so good.

Multiple Domains

sovereign actually makes it pretty simple to set up multiple domains on a single install. group_vars/sovereign houses all of the domains and accounts you want to set up. Adding a second domain was a simple as adding a new entry under mail_virtual_domains, and the associated accounts under mail_virtual_users.

Another Ansible run, and my legit domains I wanted to move off of Google were all set up. I tested logging in via Evolution, the e-mail client that comes with GNOME and what I use on my desktop and laptop. Auto config did not work, but I manually set up IMAP+ with no issues. I could send e-mail to and from accounts without a problem.

That left me figuring out how to get catchall e-mail addresses to work. There was an open issue on the Github project, so I dug around a bit. sovereign uses a Postgresql-backed e-mail system for the users, so finding how to do catchall addresses was a bit of a pain. Turns out it is really hard and not well documented. This wasn't a problem with sovereign, but postfix itself.

I found instructions for how to do it at https://workaround.org/ispmail/wheezy/connecting-postfix-to-the-database. I created a new file at roles/mailserver/templates/etc_postfix_pgsql-email2email.cf.j2 and modified the Ansible scripts to use it per the instructions on workaround.org.

Another Ansible deploy, and I tested it from my old Hotmail address.

I did not get my e-mails.

Checking the logs I was getting greylisting errors. Turns out Hotmail/Outlook.com get flagged quite regularly for spam, so my server was greylisting them. I added the following to /etc/postgrey/whitelist_clients and restarted postgrey:

# Outlook.com
104.47.0.0/17
40.107.0.0/16
/.*outbound.protection.outlook.com$/
/outlook/

I sent another e-mail, and my catchall started working! Well, technically, it was working before, just my greylist service was slowing Outlook.com down.

Moving from Google

After all my testing, I was ready. I went into my DNS providers and added the needed DKIM, DMARC, and MX records to point to my new server. I waited about fifteen minutes, as the TTL on all the records was 900 seconds, and tried to send an e-mail. It showed up in my new inbox.

I actually started recieving legitimate e-mail as well. I noticed some, like e-mails from Twitter, were coming in about 2 hours later than their timestamp. Quick look at the logs showed I'm greylisting Twitter's servers as well. Everything was working though, as grey listing is a normal part of day-to-day e-mail. If I'm greylisting someone and it's important, there are many other ways to get in touch with me ASAP.

I have years worth of e-mail sitting in GMail though. I wanted to move all of that over.

After some searching I came across imapsync, which is an open source tool that syncs mail from one IMAP server to another. I followed the directions at http://blog.jgrossi.com/2013/migrating-emails-using-imap-imapsync-tofrom-gmail-yahoo-etc/ on compiling and setting it up on my Ubuntu 17.04 desktop.

I then followed the directions at https://imapsync.lamiral.info/FAQ.d/FAQ.Gmail.txt for syncing from GMail to my local server. I settled on the following command to run:

imapsync \
           --host1 imap.gmail.com \
           --ssl1 \
           --user1 me@googlehostedemailaddress.com \
           --password1 p@ssw0rd \
           --authmech1 plain \
           --host2 mail.newmailserver.com \
           --ssl2 \
           --user2 me@googlehostedemailaddress.com \
           --password2 n3wp@ssw0rd \
           --useheader="X-Gmail-Received" \
           --useheader "Message-Id" \
           --automap \
           --regextrans2 "s,\[Gmail\].,," \
           --skipcrossduplicates \
           --folderlast  "[Gmail]/All Mail"

GMail has a 2.5GB limit on mail transfer per day, but I was below that limit. I fired up the command and was immediately shut down by Google. They consider PLAIN authentication an insecure way to authenticate (for good reason), but they provided a link and explanation. I followed the directions and ran the command again.

Nearly 48 hours to download all of the e-mail. It worked though. I started to see all of my folders and e-mail show up in my new server.

With that, I was off of Google's mail servers.

Security Concerns

E-mail is not secure. It was never designed to be. Even running something like ProtonMail, which touts it's encryption, does nothing to encrypt e-mails once it leaves their servers. Anyone can sniff e-mail on the wire. That's the nature of e-mail.

What is a concern is authentication, and access to the box.

SSH access is locked down to key-based authentication. No users have passwords. sovereign also sets up fail2ban, which should stop any brute force attacks. I'll probably supplement that with ossec. I should be able to get that installed with a new Ansible role.

For any virtual hosts on the machine as well as IMAP, sovereign sets up Let's Encrypt for SSL certificates, as well as scripts to renew them when needed. sovereign sets up Roundcube for web mail, which is protected with this, and any new subdomains it activates will be protected as well (with the appropriate changes to Ansible).

E-mail access and sending require authentication. Most servers get blacklisted due to the lack of authentication on the sending portion. Authentication is set up by default with sovereign, and all of the authentication happens over SSL/TLS.

My only main job is to update the base OS and packages every so often. I think I'm pretty well set up other than that.

Step One Completed

It's been a few days now and so far so good. The only hard thing thus far was setting up the catchall addresses. I'm getting e-mail on my laptop, desktop, and phone without an issue. I've tested sending mail to different services and so far have not been blocked. The e-mail transfer from GMail to the new server has been taking a while, but it's pretty hands off once it starts.

I am not totally off of Google yet. Next step is to move all of my calenders, which I believe I can do with ownCloud. ownCloud is an open source file, calendar, and contacts, storage/sharing service that gets installed as part of sovereign. ownCloud should actually handle both moving my calendar from Google Calendar, but also my files from Google Drive.

I also have a few patches that I want to clean up and send to sovereign. One nice one is the catchall setup, but then I've also been working with the Ansible scripts a bit to make it smaller to run. By default it runs all the tasks, but for something like adding a single e-mail address that means a 15-20 minute run.

So far I've been impressed with sovereign. I'd highly suggest looking into it if you want to run your own server.

If you aren’t familiar with how Open Source came to be the way it is today, please read “A History of Open Source,” which is effectively Part 1 of this small series of posts.

“younger devs today are about POSS — Post open source software. fuck the license and governance, just commit to github.” — James Govenor

There are two basic licensing camps in the Open Source world — the world of copyleft and the GPL, and the permissive realm of BSD/MIT. Since 2000, a shift has been made toward permissive licensing.

Is one better than the other? If so, why?

The trend does seem to indicate that the current development environment is favoring developer ease-of-use for code (permissive licensing) over a requirement of code sharing (copyleft). The general idea of permissive licensing is to make the Developer’s life easier, but what if there was an even more permissive license than permissive licenses?

“Empowerment of individuals is a key part of what makes open source work, since in the end, innovations tend to come from small groups, not from large, structured efforts. ” — Tim O’Reilly

As with everything, the internet change how we shared code.

Github did what no other source code sharing system did, and that was make it easy to share code. Now before you jump down my throat, let me clarify. While there had been Sourceforge for open source projects, and Google Code for sharing code, neither were that great, let alone for a new developer getting started.

Github made it easy for anyone to throw code up on a website, and made it easy to get that code down to your machine. They invested in teaching people to use git and made the case for why you should use them. They made open source project hosting free.

For many years Github actively made the decision to not enforce a license on code that was uploaded as open source repositories. Github left it up to the maintainer to sort that out. 80–90% did not bother with a license. That is even after a change in 2013 where Github decided to start asking about licensing when new projects were created.

“Software is like sex: it’s better when it’s free.” — Linus Torvalds

“All information should be free” has been a tenant of hackers since the 1960’s. Instead of restricting usage of code, why not just make it free? Completely free?

There has been a recent trend toward the idea of releasing software under much more lax licenses that veer more toward Public Domain than they do an established Open Source license. In some extreme cases code is being released without any license as to how it can be used, under the assumption that no license is the same as Public Domain.

The driving force behind this idea is “I don’t care what you do with my code.” It’s a noble idea that hearkens back to the 1960s. Code does not need all of these rules around sharing and usage, just take my code and do what you want.

There are even licenses that support this, due to the way that copyright works. Licenses such as WTFPL (Do What the Fuck You Want to Public License) and the DBAD (Don’t be a Dick) Public License are designed to get out of the nitty-gritty thinking when it comes to sharing code — here is code, just use it.

The First Fallacy — No License is OK

“Linux is not in the public domain. Linux is a cancer that attaches itself in an intellectual property sense to everything it touches. That’s the way that the license works.” — Steve Ballmer

Licensing is restrictive no matter which camp you are in, and by making licenses you make it harder to integrate software. For example, the company you work for probably will not use GPL software for fear of having to release the source code of their flagship product which contains many proprietary ideas and business rules.

In the US, copyright is automatically assigned. There isn’t a special form you have to send into the government, when you create something copyright is assigned to you, or whomever hired you to do the work. There are things you can do to further prove that you are a copyright holder, but simply publishing code online marks you as the copyright holder. Created works do not automatically go into the public domain anymore.

Copyright holders hold all the cards. Just because you can see the source code for a piece of software doesn’t mean you can use it without repercussion, just like finding a $100 bill on the ground doesn’t automatically make it yours.

We live in a world controlled by copyright, and until such a time as copyright laws change, releasing software without a license is a dangerous move, even potentially more dangerous than other licenses.

Unless you have something in your hand that says you are allowed to use the software, you are right back at an AT&T Unix situation. Otherwise the copyright holder can pick up their ball and go home, or worse, sue you for using their software.

The Second Fallacy — Lax Licenses are Open Source

”From my point of view, the Jedi are evil!” — Anakin Skywalker

The current development landscape very much carries a “Fuck It, Ship It” attitude. It is a core mentality of many tech startups and developers. Getting an MVP out and validated is more important than wasting time thinking about licensing. We are developers that use open source tools so we feel the need to give back, so we release what code we can.

In an ideal world you might just release your software as Public Domain, but there are many countries that do not recognize public domain, and public domain has different definitions depending on where you are. You need some sort of licensing.

In a world where Public Domain is not really a good thing to release code under, we end up with these licenses that absolve the original developer from putting restrictions on the code.

• Don’t Be a Dick
• Do What The Fuck You Want
• Don’t Be Evil

Developers do not want to have to mess around with licensing. Public domain is not a viable choice. “I just want to release code.” Developers ended up coming up with very lax software licenses where they basically say they don’t care what you do with the code.

Public Licenses are also littered vague concepts, like the DBAD. What defines being a dick? Who defines it? While there are examples in the licenses, DBAD even says that it is not limited to the examples given. What happens when someone decides you are being a dick with their software when you don’t think you’re being a dick? Douglas Crockford famously added “The Software shall be used for Good, not Evil” to the MIT license used for JSMin. Who determines what is evil?

These lax licenses are coming from a good place, and the people that come up with them are not ignorant or stupid people. The only problem is that the legal system doesn’t like vague concepts, and from a business standpoint vague definitions can really put you in a bad spot if someone decides you are being a dick, or doing something evil.

Developers that are fed up with licenses and procedure and bureaucracy are, in my mind, ignoring sixty years of history in computing. The “Just Ship It” attitude and the “Just Commit It” culture of many groups feeds into this idea that the early MIT hackers would have loved — make the software available and good things will come of it.

As humans though, we screw it up. We tried sharing software without licensing and, honestly, that did not end up working out. Hell, we cannot even agree on how software should be shared. Should be be copy-left? Should it be permissive? Can’t I just give it away?

Open Source licenses were chosen because they had been vetted and have the legal verbiage to make their usage cases safe (permissive or copyleft). While it might suck to have to put a license on something, sometimes the right, and safe, thing to do is suck it up and spend thirty seconds deciding if you want a permissive license or a copyleft license.

Saying that we are beyond Open Source and the need for licenses is just a lie developers are telling themselves when they don’t want to think about what happens to their code. You created it, take thirty seconds make sure that the code is released properly and will be used properly.

Go to https://opensource.org/licenses/alphabetical and take a look at the licenses that are available. There are many out there, as well as the venerable GPL and BSD licenses. If that list is daunting, check out http://choosealicense.com/ from Github.

Don’t ignore sixty years of history.

The world of computers is an odd place. In the span of my own lifetime, I’ve gone from not owning a computer because it was too expensive to owning a watch that has more computing power than the first computer I ever owned. The amount of computing power in my house is mind boggling when I think about it compared to twenty years ago.

Software, too, has evolved. I started off with DOS, then switched to Windows 3.1. I never personally owned a modern Mac until a few years ago, but used them throughout school. There was always the PC vs Mac rivalry but I didn’t care for the most part. I used a PC because it played games. That was up until I found Linux.

Somewhere around 2000, I was at a book store and came across a boxed set for Linux Mandrake. I think it was something like fifty dollars and I had enough cash for it. I installed it on a second machine I had and was amazed.

I quickly ran into problems running it and had to search for help online. I started to learn about sharing source code, how to patch and recompile programs, and this whole world of sharing code. The GPL made all of this possible.

This GPL thing intrigued me though. Here was this document that told me I was allowed to modify and share the source code to software as long as I made my changes public. That all made sense. If something didn’t correctly I should be able to fix it and let other people know of the fix. I could not do that with Windows, or Microsoft Office, or Photoshop on the Macs at school.

Why did you need this documentation, this proof that I was allowed to do this and not get in trouble?

That’s the world we live in.

How did we get here?

“All Information should be free” — Steven Levy, “Hackers: Heros of the Computer Revolution”, on the Hacker Ethics

In 1956, the Lincoln Laboratory designed the TX-0, one of the earliest transistorized computers. In 1958 it was loaned to MIT while Lincoln worked on the TX-2.

The TX-0 amazed the early computer hackers at MIT. It didn’t use cards, and it wasn’t cloistered away like the hulking behemoth of a machine from IBM that most people at MIT programmed against. You typed your program onto a ribbon of thin paper, fed it into the console, and your program ran.

Most importantly the TX-0 was not nearly as guarded as the holy IBM 704. Most of the hackers were free to do what they wanted with the machine. There was one problem, and it was somewhat of a large on — the TX-0 had no software.

So the hackers at MIT created what they needed.

Most of the software was kept in drawers and when you needed something, you reached in and grabbed it. The best version of a tool would always be available, and anyone could improve it at any time. Everyone was working to make the computer and the software better for everyone else.

“All information should be free” was a core tenant of the hacker culture at MIT. No one needed permission to modify the software as everyone was interested in making the software, and thereby the TX-0, better.

As the machines changed and the software changed, this ethos did not. Software would be shared and changed to work on many different types of hardware, and improvements were added over time. Needed the latest copy? Just ask for it. Need to fix it? Just fix it.

“To me, the most critical thing in the hobby market right now is the lack of good software courses, books, and software itself. […] Almost a year ago, Paul Allen and myself, expecting the hobby market to expand, hired Monte Davidoff and developed Altair BASIC. […] The feedback we have gotten from the hundreds of people who say they are using BASIC has all been positive. Two surprising things are apparent, however. 1) Most of these “users” never bought BASIC […]” — Bill Gates, “An Open Letter of Hobbyists”

Fast forward to 1976. Computers have left the halls of universities that had the physical space needed in the 50’s and 60’s to house them and are entering people’s homes. They aren’t necessarily like the computers we have today, but all computers need software.

The ideals that the hacker culture at MIT did not change as it spread westward and as these computers invaded the lives of hobbyists. What has changed is the business around computers, and like anything when it comes to humans, there is always money to be made.

“All information should be free” reared it’s head when the tape containing Altair BASIC disappeared from a seminar put on by MITS at Rickey’s Hyatt House in Palo Alto, California. Why? Ed Roberts, the “father of the personal computer” and the founder of MITS (Micro Instrumentation and Telemetry Systems) had decided to not give the Altair BASIC software to customers for free and instead charged $200 for the ability to write software.

For better or for worse, copies of Altair BASIC started appearing and being shared.

The landscape of computers and software development was changing. You no longer had one or two giant machines sitting in a university that had paid staff who could write software for them. With the TX-0 at MIT, it did not cost them anything extra to make and distribute software because there was no downside — there was not any money being exchanged. Just increases in workflow (and better gaming).

By the 1970s the need for software was seeing an ideological shift .Up until this point the creation of software was paid for indirectly by the universities and companies that needed it. Since most software was built by university developers, it was infused with the academic idea of sharing knowledge. Now software developers were seeing the need to develop generic software that many people would need to use.

That costs money, because developers have themselves and their families to support.

“Those who do not understand UNIX are condemned to reinvent it, poorly” — Henry Spencer

The 1970s also saw the development of the Unix operating system developed at AT&T by Ken Thompson, Dennis Ritchie, and others. Much like the original tools built by the hackers at MIT on the TX-0, Unix grew as it was licensed to other companies and universities.

Unix was alluring because it was portable, handled multiple users and multi-tasking. Standards help people develop software, and Unix became one of those standards. Before this was Multics for the GE-645 mainframe, but it was not without its faults.

AT&T, however, was not allowed to get into the computer business due to an antitrust case that was settled in 1958. Unix was not able to be sold as a product, and Bell Labs (owned by AT&T) was required to license its non-telephone technology to anyone who asked.

Ken Thompson did just that.

Unix was handed out with a licenses that dictated the terms of usage, as the software was distributed in source form. The only people who had requested Unix were ones that could afford the servers, namely universities and corporations. The same entities that were used to just sharing software.

The open nature of Unix allowed researchers to extend Unix as they saw fit, much as they were used to doing with most software. As fixes were developed or things were improved, they were folded into mainstream Unix.

The University of California in Berkeley was one of the most sought-after versions of the Unix code base, and started distributing their own variant of BSD in 1978, known as 1BSD, as an add-on to Version 6 Unix.

There was a hitch though. AT&T owned the copyright to the original Unix software. As time went on AT&T used software from projects outside of themselves, including the Computer Sciences Research Group from Berkeley.

Eventually AT&T was allowed to sell Unix, but their commercially available version of Unix was missing pieces that were showing up in the Berkeley variant, and BSD tapes contained AT&T code which meant users of BSD required a usage license from AT&T.

The BSD extensions were what we would eventually call “Open Source,” in a permissive sense. BSD was rewritten to remove the AT&T source code, and while it maintained many of the core concepts of and compatibility with the AT&T Unix, it was legally different.

Much like with Bill Gates and Micro-Soft’s (eventually Microsoft) Altair BASIC, we start to see the business side of software start to conflict with the academic side of software, or more it conflicting with the hacker idea of software.

We also see one of the first true Open Source licenses come from this, which distinctly grants the end user special rights on what they can and can’t do with the software. Unix had its own license which up until commercialization (and a growing market) had been fairly liberal, but BSD wanted to make sure that Unix would be available to whomever needed it.

“Whether gods exist or not, there is no way to get absolute certainty about ethics. Without absolute certainty, what do we do? We do the best we can.” — Richard Stallman

In 1980, copyright law was extended to include computer programs. Before that most software had freely been shared or sold on a good faith basis. You either released your software for everyone to use as public domain, or you sold it with the expectation that someone wouldn’t turn around and give it away for free.

Richard Stallman was, and probably is, one of the last true Hackers from the MIT era. In a sort of hipster-y kind of way he yearned for the time when software could be free, not shackled by laws or corporations. In a sense, software was meant to be shared and wanted to be shared. “All information should be free.”

Stallman announced the GNU project in 1983, which was an attempt to create a Unix-compatible operating system that was not proprietary. NDAs and restricted licenses were antithetical to the ideals of free software that he loved.

The Free Software Foundation was founded in 1985, and along with it came the idea of “copyleft.” Software was meant to be free, and GNU Manifesto shared his ideas on GNU project and software in general. Whether you agreed with it or not, the GNU Manifesto was a fundamental part of what we now consider Open Source.

Stallman then conglomerated his three licenses, the GNU Emacs, the GNU Debugger, and the GNU C Compiler, into a single licenses to better serve software distribution — the GPL v1, in 1989.

The release of the GPL, the release of a non-AT&T BSD Unix, and the flood of commerical software of the 80’s and 90’s, lead us to where we are today, and are the three major ideals that exist:

• Software should always be free — Copyleft
• Software should be easy to use and make the developers lives easier — Permissive
• Software should be handled as the creator sees fit — Commercial

“younger devs today are about POSS — Post open source software. fuck the license and governance, just commit to github.” — James Govenor

Since 2000, a shift has been made toward permissive licensing. One would argue that the GPL is dying. One could argue that developers are more interested in helping themselves than the actual idea of free software.

There is no denying that there are two camps when it comes to Open Source software, with the crux of the problem being exactly what software is supposed to be, or do for us.

The GPL says it should be free. In a way, Software is a living, breathing thing that wants to have the freedom to become the best possible piece of Software. It cannot do that when it can be locked up, chained, and held back from the passions that developers have for making software better. You, the end user, are better because Software can be changed to make everyone’s lives better, and you are better because you can change the Software.

The other camp is more pragmatic in a way. Permissive licensing wants software to be free because that helps Developers. You, the Developer, release software to make people’s lives better. You, the Developer, are more interested in knowing that people can use your software or code in a way that they see fit. The end user is better because the Developer had the freedom to change the software to make everyone’s lives better.

Is one better than the other?

Or should we just throw it all to the wind and ignore sixty years of computer history and forget about licenses?