6 posts tagged with "webdev"

The singleton pattern has got a bad reputation over the years due to be widely overused in the incorrect use cases. With the proliferation of microservices, have APIs become the new singleton?

The Problem​

APIs, or application programming interfaces, have become a ubiquitous part of modern software development. They allow different systems and applications to communicate with one another, enabling the creation of complex, interconnected systems that can share data and functionality. However, there has been a growing concern that APIs are being overused, leading to a proliferation of unnecessarily complex and fragile systems that are difficult to maintain and scale.

One reason for the perceived overuse of APIs is the ease with which they can be implemented. With the abundance of API management tools and frameworks available, it is relatively straightforward to expose a set of functionality as an API and make it available to other systems. This has led to a proliferation of APIs, many of which are redundant or unnecessary, adding unnecessary complexity to the overall system.

Another issue is the lack of standardization in the API ecosystem. Each API is typically designed to meet the specific needs of the system it was created for, resulting in a wide variety of different designs and conventions. This can make it difficult for developers to understand and use APIs from other systems, as they may have to learn and adapt to new conventions and patterns each time they encounter a new API.

In addition to these issues, the reliance on APIs can also lead to fragile systems that are difficult to maintain and scale. When multiple systems are tightly coupled through APIs, a change to one system can have cascading effects on others, leading to unexpected behavior and potential failures. This can make it difficult to make changes or updates to a system without the risk of breaking something else.

There are also concerns about the security of APIs. As they allow systems to communicate with one another, they can also provide a potential entry point for attackers to gain access to sensitive data or functionality. Properly securing APIs can be a complex and time-consuming task, and if not done correctly, can lead to significant vulnerabilities.

The Solution​

So, what can be done to address these issues? One solution is to use APIs more judiciously, carefully evaluating whether an API is truly necessary before implementing it. This can help reduce the overall complexity of the system and make it easier to maintain and scale.

It's also important to adopt API design standards and guidelines, which can help ensure that APIs are consistent and easy to understand and use. Finally, proper API security practices should be implemented to protect against potential vulnerabilities.

The Context​

Cloud and infrastructure as code have revolutionized our industry. They allowed us to be able to procure infrastructure in a simple, adaptable way.
This allowed us to move from writing huge monolithic applications to write microservices that interact between them.
One of the most accepted definition of a microservice can be expressed as:

A self-contained portion of code that does not share resources with other services, can be deployed independently, and should be easy to rewrite in a small portion of time.

This sounds great when we talk about individual parts of a software projects. Nevertheless, when thinking about systems and how they operate, There is a point to make about granularity as software does never work fully isolated. It requires interactions with other systems to fulfill their purpose.

Most of the monolithic applications of the past had an issue of being over-engineered to allow changes that might never happen.

Could that also happen with microservices?

The Issues​

Clarity Of The Domain​

When a system grows too much in small pieces, it becomes more and more complex to understand the big picture.
When pieces are too small, domain events start becoming exchange of information in between nodes of a network. All this removes cohesion on the knowledge over the domain of a system, making it difficult to grasp the real intention and capabilities of concepts and actors across a system.

Babel tower Issue​

The more parts a system has, the less heterogeneous it becomes. This at the same time translates into a more complex environment with more integrations, frameworks and bigger learning curves that affects delivery. There need to be a balance of when and where in a system a new technology is added. Decisions must be based on needs and not on preferences.

Implicit runtime dependencies​

The more a system get split, the more dependency on certain node it will have. This tends to cause more dependencies in between the pieces of your infrastructure-based puzzle where you start having god infrastructure points that become single point of failure, or you have a chain of dependent infra that need to be deployed in a go or certain order.

Hidden Complexity​

The more your microservice environment grows, the more it requires a growing support infrastructure for monitoring, alerting and other services not used as part of the main system. This normally is a separate effort which has its cost. The more a system grows, those hidden complexities become a dependency for all the nodes in the system, making it a complex task to evolve and change those dependencies.

Why… if YAGNI​

One of the main ideas of microservices was to be able to validate assumptions fast. Before bootstrapping new services or infrastructure, there is a need to ask ourselves about the existence of a service or infrastructure that contains the domain knowledge required for the experiment in the current ecosystem. If we are not careful, experiments won't be experiments. They will be MVPs, where domain knowledge is re-implemented, just for having it as a standalone node on the system.

Repeating Yourself​

When we create pieces of code that are independent, there is always a certain level of bootstrapping that is required and repeated in each node of our systems. This will cause not only a set of duplicated code, but also has a development time cost attached to it. Bootstrapping a project in a high granularity system can be complex to standardize.

Microservices, the cloud, and infrastructure as a service have definitely revolutionized our industry, nevertheless as in everything there is a need for balance. Making sure we use the right tool for the job, and we don't over-engineer things, not only at a code level but also at infrastructure level, as everything has a cost.

Conclusion​

In conclusion, a macro infrastructure due to microservice obsession can lead to increased complexity and overhead costs, as well as challenges in making changes and updates to the system. While microservices can offer benefits such as increased scalability and flexibility, it is important for organizations to carefully consider their specific needs and choose the right level of granularity for their architecture.

I have observed quite a few articles lately that elaborate on issues with TDD. Nevertheless, they focused on the first letter but miss the focus of the other two letters.

Not A Testing Strategy​

If you take anything out of this article, please think about this quote:

If TDD was about testing it would have been called TDT (test driven testing).

The fact that we do test upfront in TDD does not mean at all that there is a direct relationship with a testing strategy, and as many preach, unit testing is not enough to create robust software.

A Design Strategy​

TDD is actually a Design Strategy, this is why the 2 last letter are for driven development. This means that your final code is being moved by those tests and not the other way around.

The design that TDD will move you towards to is minimalistic. Reducing the tendency of overengineering solutions when you don't need them. This brings a reducing time to market, by reducing the accidental complexity.

When doing TDD most developers have the complexity of letting go their egos, the problem when people fight against the practices is because they think to know better. Nevertheless, it tends to generate waste because most code optimizations tend to be premature and most extensibility points will never be modified.

There are places where TDD does not fit, for example while investigating a technology through a spike or PoC because in these cases, the person is exploring knowledge not generating value. In other cases, TDD allows you to bring value in the shortest way possible.

Conclusion​

If you are an experienced developer, do not discard TDD because you think you know better, allow it to challenge you. If you are a new developer, learn from the different ways of doing things and understand the value, don't take articles at face value.

Why are messages important?​

Commit messages are part of the collaboration we do day to day inside a team, it works as a record of what has happened.

Every time you perform a commit, you’re recording a snapshot of your project that you can revert to or compare to later.

— Pro Git Book

Commit messages are used in many ways, including:

• To help a future reader quickly understand what changed and why it changed
• To assist with easily undoing specific changes
• To prepare change notes or bump versions for a release

All three of these use cases require a clean and consistent commit message style.

Easy Commit messages with Commitizen​

This tool purpose is to define a standard way of committing rules and communicating it. The reasoning behind it is that it is easier to read, and enforces writing descriptive commits. Removing the ambiguity of options and the mental load of following the standard manually.

Commitizen will prompt you a series of questions that will generate the final commit message. It has multiple adapters, in my case I prefer to be controlling the questions, so I use cz-format-extension.

You can add commitizen to your project with the next command line

npm install commitizen --save-dev # npm
yarn add commitizen -D  # Yarn

Add any of the available adapters, in my case cz-format-extension:

    npm install cz-format-extension --save-dev # npm
    yarn add cz-format-extension -D  # Yarn

In your package.json you will need to add the next section:

  ...
  "config": {
    ...
    "commitizen": {
      "path": "cz-format-extension"
    }
  }
  ...

The Adapter cz-format-extension allows a massive flexibility as the questions can be defined in a .czfrec.js file. An example is:

const { contributors } = require('./package.json')

module.exports = {
  questions({inquirer}) {
    return [
      {
        type: "list",
        name: "type",
        message: "'What is the type of this change:",
        choices: [
           {
              type: "list",
              name: "type",
              message: "'What is the type of this change:",
              choices: [
          {
            "name": "feat: A new feature",
            "value": "feat"
          },
          {
            "name": "fix: A bug fix",
            "value": "fix"
          },
          {
            "name": "docs: Documentation only changes",
            "value": "docs"
          },
          ...
        ]
      },
      {
        type: 'list',
        name: 'scope',
        message: 'What is the scope of this change:',
        choices: [
            {
              "name": "core: base system of the application",
              "value": "core"
            },
            {
              "name": "extensions: systems that are observed",
              "value": "extensions"
            },
            {
              "name": "tools: other things in the project",
              "value": "tools"
            },
        ]
      },
      {
        type: 'input',
        name: 'message',
        message: "Write a short, imperative tense description of the change\n",
        validate: (message) => message.length === 0 ? 'message is required' : true
      },
      {
        type: 'input',
        name: 'body',
        message: 'Provide a longer description of the change: (press enter to skip)\n',
      },
      {
        type: 'confirm',
        name: 'isBreaking',
        message: 'Are there any breaking changes?',
        default: false
      },
      {
        type: 'input',
        name: 'breaking',
        message: 'Describe the breaking changes:\n',
        when: answers => answers.isBreaking
      },
      {
        type: 'confirm',
        name: 'isIssueAffected',
        message: 'Does this change affect any open issues?',
        default: false
      },
      {
        type: 'input',
        name: 'issues',
        message: 'Add issue references:\n',
        when: answers => answers.isIssueAffected,
        default: undefined,
        validate: (issues) => issues.length === 0 ? 'issues is required' : true
      },
      {
        type: 'checkbox',
        name: 'coauthors',
        message: 'Select Co-Authors if any:',
        choices: contributors.map(contributor => ({
            name: contributor.name,
            value: `Co-authored-by: ${contributor.name} <${contributor.email}>`,
        }))
      },
    ]
  },
  commitMessage({answers}) {
    const scope = answers.scope ? `(${answers.scope})` : '';
    const head = `${answers.type}${scope}: ${answers.message}`;
    const body = answers.body ? answers.body : '';
    const breaking = answers.breaking ? `BREAKING CHANGE: ${answers.breaking}` : '';
    const issues = answers.issues ? answers.issues : '';
    const coauthors = answers.coauthors.join('\n');

    return [head, body, breaking, issues, coauthors].join('\n\n').trim()
  }
}

The file creates a process of questions for:

• type: align with semantic release message specification
• scope: affected part of the application
• message: the imperative written message
• body: longer description
• breaking: to determine if it's a breaking change for semantic release
• Issue: related issue of our ticketing system
• Co-Authors: people working in the tasks while pair programming

All these questions are asked interactively and not by the brain power of doing manual work.

And you can then add some nice npm scripts in your package.json file pointing to the local version of Commitizen:

  ...
  "scripts": {
    "commit": "cz"
  }

This will be more convenient for your users because then if they want to do a commit, all they need to do is run npm run commit and they will get the prompts needed to start a commit!

NOTE: If you are using precommit hooks thanks to something like husky, you will need to name your script something other than "commit" (e.g. "cm": "cz"). The reason is because npm scripts has a "feature" where it automatically runs scripts with the name prexxx where xxx is the name of another script. In essence, npm and husky will run "precommit" scripts twice if you name the script "commit", and the workaround is to prevent the npm-triggered precommit script.

That is all :) . I will do a special mention to commitlint that is a very useful tool to lint commit messages. I do not use it anymore as it has some overlap with commitizen.

What & Why Git hooks?​

Git hooks are scripts that Git executes locally before or after events such as commit, push, and receive.

These hooks are completely programmable trough bash scripting. Examples of what can be done:

• pre-commit: Enforce project coding standards.
• pre-push: Run tests.

This allows us to make sure we are committing the correct things at the correct time. Not breaking our code just because of the mental load of doing things as a manual process that can be forgotten.

How to Start​

Add Husky​

Husky is a tool that allows Git hooks using JavaScript configured using individual files for hooks in a .husky/ directory.

The fastest way to install husky is by using husky-init, a one-time command to quickly initialize a project with husky:

npx husky-init && npm install       # npm
npx husky-init && yarn              # Yarn 1
yarn dlx husky-init --yarn2 && yarn # Yarn 2+
pnpm dlx husky-init && pnpm install # pnpm

It will set up husky, modify package.json and create a sample pre-commit hook that you can edit. By default, it will run test when you commit.

To add another hook, use husky add.

If you are not comfortable using husky-init you can find other options here.

Add lint-staged​

Husky is very useful, but it will run natively to git and not focus the commands in our bash scripts for all the files, not only the ones we want to commit.

Lint Staged appear to resolve this problem. It allows you to run the process against staged git files that match a pattern.

Install lint-staged by adding it to your local project.

npm install lint-staged --save-dev
yarn add lint-staged -D 

In your package.json add it as a script("lint-staged": "lint-staged",) and refer it through a pre-commit hook. If using Husky, this can be found in .husky/pre-commit with the next content:

#!/bin/sh
. "$(dirname "$0")/_/husky.sh"

yarn lint-staged

There are multiple ways to configure lint-staged. One of them is having a lint-staged.config.js file in your project root folder. In this file, you can express what process you want to run for what types of files. For example:

module.exports = {
  '*.{ts,tsx}': [() => 'yarn tsc:check', 'yarn format', 'yarn lint:fix', 'yarn test', 'git add .'],
};

The previous snipped runs the compiler check, formatting, linting and test before adding the fixed staged files to the current commit.

Conclusion​

With this two tools, we will now be pushing code that will pass similar checks than our CI/CD system.

Over the last few years, some practices appear to be more a dogma than a value adding practice. One of this is Pull Requests.

Why PR's exist​

• Malicious Code Prevention: Pull requests exist mostly as a practice accepted for zero trust environments (ex. Open Source). An attack vector on this type of environment is the ability of anyone to contribute, meaning you could inject code that could create known vulnerabilities that packages will inherit. That is why maintainers validate code from unknown users.

• Highly Distributed Teams: PR's can be use for highly distributed teams (around the clock) as a way to do knowledge sharing. If someone in side A of the world can follow and understand the changes for a feature that is being developed in side B of the world.

The issue​

IS there any value of doing PRs when people work collocated? What is the cost of PRs in trust environments?

The value that normally people give to PRs is the one of having a peer review process. Nevertheless, we will see later in this article that there are less invasive ways to do this.

Some costs of PRs are:

• Slow Delivery: PRs are a start and stop strategy where there is a gateway to merge code. This is time that needs to be taken by developers (writting & preparing a PR) and reviewers (reviewing, commenting, etc) to go through the process. At the same time is more time the code takes to get to production (merging, re-testing, etc). This is significant for features but also for fixes, meaning you can go from a response time of minutes to hours.
• Isolation work: When working on branches, devs work on code that works isolated but needs to be merged with a continuous stream of changes. This means that any test isolated will probably be invalidated upon merging.
• Lack of ownership: As work is done isolated, the developer who creates a PR delegates part or the responsibility to the reviewer. Humans don't have compilers or containers to run the code in our brain, meaning catching errors tends to be out of our realm.
• Egos: As catching errors tends to be out of our human realm, PRs tend to become a thing related to preferences (Style, patterns, etc). This hardly provides any value to the code as either tools like linters can do this automatically or it brings premature optimizations.
• Late feedback: Any valid recommendation is actually provided quite late in the process, when the code has already been written and validated. Causing rework that takes time.

The Alternatives​

Pull requests are just one of the asynchronous peer code reviews styles. Is not the only way of doing peer reviews.

Some other types of peer reviews that I, personally, like are:

• Over-the-shoulder: The bases of this is to have a conversation over what has been or is being implemented. This creates a synchronous feedback loop on an async process. It does not fix all the shortcomings of a PR, but it creates a faster feedback loop.
• Pair Programming / Mob Programming: The idea is that multiple developers work in the same code base in the same computer, creating a synchronous feedback loop in a synchronous process. This with Trunk-based development allows very fast feedback loops at product level, and with the correct tools generates resilience and ownership among developers.

The disclaimer here is I have worked doing pair programming, TDD and trunk-based development for more than 5 years in multiple size companies, and it has always been a bliss.