Hao's Learning Log

Reflecting on AI Dev Tools in 2024

Sat, 09 Nov 2024 06:17:00 GMT

The last couple of years have been a magical time for AI fans, with new tools, AI models, services, or apps popping up every other week. As someone who was very skeptical at the start, I thought it would be interesting to reflect on how AI has changed my workflow and review what I’m currently using.

A quick disclaimer: I’m not a fan of the term “AI” since nothing we have right now deserves to be called “artificial intelligence.” These are just machine learning data models. But for the sake of consistency, I will refer to them as “AI” since it is now the accepted way to refer to machine learning.

Coding

I recently moved off GitHub’s Copilot and adopted Cursor with the Claude data model, and I really enjoy the new workflow.

With Copilot, my usage was mostly limited to the code auto-complete feature, which helped my coding a little. I would switch between turning it on and off because it was slow to respond to my changes and often disrupted my workflow.

Cursor has been a big enough improvement that I decided to pay the monthly subscription. The difference isn’t world-shattering, but it is significant enough for someone like me—who hates subscriptions—to bite the bullet. For one, the Claude model is much better than Copilot, which is based on ChatGPT. And secondly, Cursor’s AI editor-level features are much more intuitive to use than VS Code.

After using Cursor + Claude for two weeks, I noticed myself reaching out for AI assistance more frequently, and my productivity has gone up as a result. I would even dare to say my developer experience has improved because boring, repetitive tasks now feel less of a drag.

Problem Solving

When I’m stuck on an issue now, I open up the side panel, add the right context (tag files, etc.), and see what Claude comes up with. There have definitely been times when it was able to give me a solution right away, although that is rare. Even if it doesn’t give me the answer, I find that the back-and-forth conversation makes the debugging process more bearable.

Maybe I’m a bit old-school, but I still prefer to tackle the problem myself and only reach for AI’s help when I feel like I’m not making any progress. I want to maintain and practice my problem-solving abilities—both to keep myself employed and because this is why I do this for a living.

Just a side note: I still Google things most of the time. Especially for simple questions, Google can often give me an answer in a few seconds (search + click a couple of links) faster than the generative AI can. Maybe I just hate waiting for the text to be generated and want to feel like I’m doing something, but I find using Google is still less taxing when I’m in the zone.

Rubber Ducking

As software engineers, we all know too well the feeling of being stuck on a problem for hours, if not days, only to have someone jump on a call and find the solution before we finish explaining the problem. This is known as rubber ducking or rubber duck debugging.

It’s very common to get so deep into a problem that we lose sight of the bigger picture and forget to step back. This is why many people swear by pair programming. Personally, I prefer to pair-program intentionally; blindly pairing on all tasks is a waste of engineering time. It’s often frustrating both when I’m in the zone and the constant interruptions cause extra context switching, and when I need help but can’t find anyone to pair with. AI solves this problem.

When using Claude to debug issues, it’s the perfect rubber duck. It has no context, so I have to explain the problem. And while explaining the problem, I get a chance to reset and look at the issue again more holistically.

Code Complete

This is probably what most people think of when AI coding comes up. I still use it, and it’s nice when it helps me finish writing a function or a line. It would be a lie to ignore the time saved here, but for me, this isn’t the most useful part, which is why I would turn off Copilot sometimes out of frustration. Personally, I find the ability to have a conversation about a problem, with the AI understanding the context, much more useful.

Writing

For software engineers, written communication is often part of the job, especially as they move up in seniority. This could be writing documentation for other engineers, communicating architecture or design changes to wider engineering communities, presenting technical analysis to non-technical stakeholders, or simply writing a good PR description. Whatever the scenario, my experience has been that the majority of developers lack strong writing skills (myself included). I’ve seen some beautifully written documents from senior engineers I’ve worked with in the past, but I wouldn’t expect that level of writing from everyone.

Polish And Refine Text

One thing I’ve noticed myself doing more over the last year or so is leaning on ChatGPT to polish what I’ve written. I’ll brain dump what I want to say, then paste it into ChatGPT and ask it to consolidate and reword it. The result is often better than what I had initially. It does have a tendency to reword things in a more formal or even academic tone, which then requires further refinement, or I’ll just manually update them myself if there isn’t much to change.

Generating Ideas

One of the things I struggle with in writing is getting started. When preparing an analysis document, for example, there might be several angles to tackle the topic. In this situation, I write down some key points I want to cover, then use ChatGPT to generate a skeleton document and see what points I might have missed. Once I have a list of points to cover, it becomes easier to structure and present them.

Summarising Text

I’ve started using ChatGPT to summarise text more recently. That said, I still prefer to read through the original text. It’s the same reason I read books instead of book summaries—I find a lot of valuable information is lost in the summarisation process. Maybe I’ll change my mind as AI models improve, but for now, I’ll keep this usage to a minimum.

Final Words

There has been a lot of hype around AI in recent years—just look at all the money thrown at AI startups. Not surprisingly, there has also been fear around AI taking over jobs, especially software engineering jobs. I have no doubt that as AI models get better, the number of software engineering jobs will plateau and maybe even slowly decrease over time.

As I mentioned at the start of the article, “AI” has no intelligence; these are just machine learning models. So no matter how good they become, they won’t directly “replace” a skilled engineer. Even if they could, in the short to medium term, the compute costs would be so high that companies might as well hire a person to do the job—although this could change as AI chips become more powerful. In the near future, AI tools will likely improve engineers’ efficiency, so fewer team members will be needed to accomplish the same amount of work. Junior engineers will suffer most during this transition, as we’re already seeing.

I don’t have a crystal ball, so I am not sure which direction things will go in the future. But at least in the short term, I’d advise all engineers to start leveraging these AI tools and get comfortable adopting them into their daily workflows to improve efficiency. From my experience, I’d estimate that using AI tools like ChatGPT and Claude with Cursor has increased my work efficiency by more than 20%.

I wrote an article called A web dev engineer's review: Learning Swift using ChatGPT back in April 2023. It might be an interesting read also.

PS: I only used AI to fix some grammar mistakes in this article.

GraphQL at scale: schema stitching v.s. schema federation

Fri, 18 Aug 2023 16:20:00 GMT

I have been helping to lead a GraphQL team at a large organization. The goal of my team is to create a scalable GraphQL platform and increase its adoption within this company. When I joined the team, we'd already gone through several iterations and failed to onboard any consumer in the 1 - 2 year period. The approach at the time of my joining was to use schema stitching with an open-source framework called GraphQL Mesh. After experiencing numerous challenges and we are now shifting to Apollo Federation as an alternative, which as the name suggests federates the schema instead of stitching them. In this article, I will share the differences between these two approaches and walk through my experiences with them including why we making the switch, particularly in regards to scalability and stability.

What is GraphQL schema and why is it important?

A GraphQL schema is a way of defining the types, queries, and mutations that can be used in a GraphQL service. If you are unfamiliar, the closest thing in the REST world is probably an API specification.

Here's an example of what a GraphQL schema might look like:

type Query {
 user(id: ID!): User
 users: [User]
}

type User {
 id: ID!
 name: String
 email: String
}

One could say the GraphQL schema serves as the backbone of a GraphQL API since it defines the structure and shape of the data that can be queried or mutated. The schema act as the contract between the backend and the frontend, which can also be viewed as documentation. With tools such as graphql-codegen, the consumer of a GraphQL service can generate boilerplate code and type definitions from the schema, saving engineers' time and effort.

The single GraphQL service approach doesn't scale

The common implementation of a GraphQL service is to set it up as a layer in front of other data providers, such as API services or databases. All requests from clients go through the GraphQL service in the form of GraphQL operations (queries, mutations, and subscriptions). The GraphQL service then processes these requests and forwards them to the appropriate data providers. Once the data has been retrieved, the GraphQL service touches up the data and returns it to the client. The diagram below for an illustration of this process.

In this setup, often teams follow a BFF (backend for frontend) approach, formatting and tweaking the schema to match the expectations of the consumer.

Initially, teams will start by converting or adding a GraphQL service in front of one existing backend service. Then over time, as it proves to be beneficial, teams will start adding more backend services to the same GraphQL service, as shown in the diagram below.

While this approach works well for small organizations with a limited number of data providers, it can become problematic in larger organizations.

Adding new data providers to the GraphQL service can be time-consuming and error-prone since the schema must be defined manually for each provider.
Additionally, as the number of data providers increases, the GraphQL service can become a monorepo that slows down development due to its growing internal complexities.
The GraphQL team can become a bottleneck, as all changes to data providers must be coordinated with changes to the GraphQL schema to ensure that the schema and data responses remain in sync.

In the beginning, the biggest issue for our team was the very first point. There were already many APIs within the organisation which we could integrate with, but it took a long time to get them added to the earlier iterations of the GraphQL service. To workaround this problem, we looked toward adopting the schema stitching strategy that came with GraphQL Mesh. It allowed us to add new data providers more easily by automating the conversion of API specs to GraphQL schemas. GraphQL Mesh also helps to reduce the complexity of coordinating changes between the GraphQL schema and data providers with the help of a service called Hive (we'll touch on this later).

Schema stitching

Schema stitching in essence is the process of combining multiple GraphQL schemas into a single, unified schema that is presented to the consumer. This was a concept initially proposed by Apollo, which they later abandoned and then was picked back up by The Guild who built GraphQL Mesh on top of it. The approach is great if a company is looking to migrate from traditional REST APIs to GraphQL, because with Mesh offers a plug-n-play integration flow, assuming of course each REST API came with API specs (e.g. Swagger, OpenAPI, RAML, etc).

With Mesh, naturally, we adopted its sibling SaSS service Hive as well. Hive, an open-source tool also created by The Guild, is a schema registry with some handy features that synergy well with Mesh or any GraphQL service in general. Hive is a schema CDN, that hosts the latest schema and stores previous versions of the schema. This means it has a historical view of how the schema evolved over time but also offers the ability to track each iteration of the schema on each data source (e.g. REST API). Schema is like a contract, and it is very important for all parties to know when a contract has been broken so it can be quickly fixed. Hive supports the ability to perform schema checks and notify of any breaking changes. We also used the paid feature which allows us to send usage metrics to Hive and see relevant monitoring stats. Hive can then use these usage data to let us know if something is safe to be removed.

After implementing schema stitching, the biggest win for us was how quickly and easily we could add data providers to the GraphQL service. At this point, our architecture looked something like this:

Now, schema stitching wasn't a flawless solution:

Schema stitching requires accurate and reliable API specifications, unfortunately, most of the REST APIs we were dealing with didn't have good or any API specs. This meant we had to either generate a schema from a raw API response (yes Mesh can do that) or we had to create and maintain the Swagger (or alternative) API specs manually. Both approaches were less than ideal and required massive maintenance effort.
This is still a monolith solution, the issue mentioned in the previous section still applies here.
We are still the bottleneck between consumer and data provider teams due to the lack of accurate API specs.
When an issue occurs, we are the first to get notified but the issue often comes from the underlying data provider.
A new problem was that Mesh packages are still in pre-stable phase and not following Semantic Versioning, so we often encountered bugs that led to a lot of headaches and slowed down development.

Given the above challenges, we explored alternatives and decided to move to Apollo Federation.

Schema Federation

Schema federation refers to the process of combining multiple individual schemas into a single, federated schema. In a schema federation, each schema retains its structure and integrity, but a new, overarching schema is created that combines the various schemas into a single logical unit. This allows for seamless querying and integration of the different data sources, without the need to manually combine or transform the data.

Using Apollo Federation, you can create a single GraphQL schema that is composed of multiple independent services called subgraphs. This allows you to build complex and powerful GraphQL APIs without having to create a monolithic schema that is managed by a single team or service. Instead, each service can manage its portion of the schema, and the federated schema is built automatically by combining the individual schemas.

Just like Mesh has Hive as its schema registry, Apollo offers its SaSS service called Apollo GraphOS which offers all the features of Hive and more. You can read more about it in Apollo's docs.

How is it different to schema stitching?

In many ways, schema federation has many similarities to schema stitching. GraphQL schema federation and schema stitching are both techniques for combining multiple GraphQL schemas into a single, cohesive schema. However, there are some key differences between the two approaches.

Schema stitching follows a central governance model, where each data provider doesn't need to know about each other. The GraphQL service that stitches all the schema is both the gateway and the brain that decide how schemas get stitched. Federation, on the other hand, has a router service but uses a decentralised governance model. It requires each data source to know about each other and define their relations between themselves.

Schema federation is a built-in feature of the Apollo GraphQL platform, whereas schema stitching is a technique that can be used with any GraphQL implementation. This means that schema federation is designed to be easy to use and integrate with other Apollo tools, whereas schema stitching is a more general-purpose solution that may require more work to set up and use.

Finally, schema federation uses a set of conventions and APIs to define how the individual schemas should be combined, whereas schema stitching typically involves writing custom code to manually merge the individual schemas. This means that schema federation can be easier to use and maintain, especially for large and complex GraphQL APIs, whereas schema stitching can be more flexible and allow for more customization.

How we use schema federation

With Federation, our new architecture now looks something like the following diagram.

The idea is to have a separate team manage a single subgraph and its data provider(s). We then took this idea further by making these teams into domain-based backend teams, who are responsible for the domain subgraph and the data providers (in our case they are mostly REST APIs).

Which is the right one for you?

As mentioned already, we decided to ditch schema stitching and migrate over to federation. Partly because our current data providers (REST APIs) have room for improvement. As part of the migration, we are going to rewrite the current APIs into domain APIs with GraphQL subgraphs in front of them. The wider business decision to move to a domain-driven backend has very good synergy with Apollo Federation. Of course, these are probably unique to our situation and others may not be so lucky. So let's discuss some common pros and cons of each approach.

Benefits of schema federation

There are several other benefits to using schema federation over schema stitching when building a distributed GraphQL schema. Some of the key benefits include:

Convenience: Schema federation is a built-in feature of the Apollo GraphQL platform, so it is easy to use and integrate with other Apollo tools. This means you can quickly and easily create a federated schema without having to write a lot of custom code.
Maintainability: Schema federation uses a set of conventions and APIs to define how the individual schemas should be combined, which makes it easier to maintain and update the federated schema over time. This can be especially helpful for large and complex GraphQL APIs.
Compatibility: Schema federation is fully compatible with the Apollo Client and other popular GraphQL tools, so you can use it to build rich and powerful GraphQL-powered applications.
Performance: Schema federation is designed to optimize performance and minimize network overhead, which can be important for large and complex GraphQL APIs.
Stability: From our testing Apollo software is more stable and mature compared to GraphQL Mesh which is the main if not only schema stitching tool out there.

Benefits of schema stitching

There are several benefits to using schema stitching over schema federation when building a distributed GraphQL schema. Some of the key benefits include:

Flexibility: With schema stitching, you have complete control over how the individual schemas are combined. This allows you to customize the schema stitching process to fit the specific needs of your application.
Compatibility: Schema stitching can be used with any GraphQL implementation, whereas schema federation is only available on the Apollo platform. This means that schema stitching may be a better option if you are using a different GraphQL implementation or if you want to use a mix of different GraphQL implementations in your schema.
Customization: With schema stitching, you can write custom code to merge the individual schemas in any way you want. This allows you to add custom logic and behaviour to the schema stitching process, which can be useful for complex or specialized GraphQL APIs.
Open source: all Mesh code is open source, so anyone could see and fork them as they see fit. The Guild even said they'll be making Hive (schema registry) open source as well.

Conclusion

Choosing the approach is important, but it is also worth thinking about the tools behind them as well. From our research, if you want to go with schema stitching then GraphQL Mesh is the only implementation right now, and similarly Apollo Federation is the only choice if you pick schema federation. While GraphQL Mesh gets additional points for being open source, it is still a few steps behind Apollo both in terms of scale and software maturity. For example, many of the Mesh packages are still not following Semantic Version at the time of writing, however, based on their roadmap this might not be the case for long. Until then, Apollo might be the better choice for teams wanting a stable platform to build their GraphQL services.

As mentioned in the "How is it different to schema stitching?" section, the two approaches use different governance models. If you work in a large company, then I would suggest Federation because sooner or later you will reach a point where a single GraphQL team isn't enough to provide support, maintenance and development to meet the demand from all the consumers and data providers. Handling schema changes is not only time-consuming but requires careful coordination, in my team we found ourselves spending a decent amount of time preventing breaking changes and supporting new schema releases. In smaller companies or even medium-sized companies (several consumers and data providers instead of tens or even hundreds), schema stitching provides more flexibility and be more easily slotted into an existing REST backend architecture.

Ultimately, the choice between schema stitching and schema federation will depend on the specific needs of your application, so it's important to evaluate both options and choose the one that best fits your requirements. This is the perfect situation to create PoCs and test out each approach before committing.

A web dev engineer's review: Learning Swift using ChatGPT

Sat, 08 Apr 2023 22:17:00 GMT

ChatGPT was released back in November 2022, and it caused quite a stir online. Scaremongers have been saying it will change everything and AI is ready to take jobs from us. There are also sceptics which said it won't change anything. I was interested in how good ChatGPT was, so I decided to do an experiment and have been testing it out myself in the past couple of months. I wanted to see if it could help me to learn iOS mobile app development and how it faired compared to widely used methods such as "googling" and following text or video tutorials.

My background

I've mainly been working in the web sphere during my career, which means I know next to nothing about native app development, especially in Apple's developer ecosystem. I briefly tested out Android and hybrid app development a few years back, but never checkout any of Apple's platforms. So I came into this experiment with zero knowledge about Xcode, the IDE for developing apps for any of Apple's platforms, or Swift, the programming language for developing apps. Thus this should be a perfect scenario to test out ChatGPT's limits.

Chosen project

What better way is there to learn about app development than creating an app? I love reading Chinese online novels, unfortunately, there isn't a good free reading app on iOS (Android users have a lot more choices). So I decided this would be my project to learn Swift. The app would essentially be an HTML scrapper that given some instructions (e.g. URLs and CSS selectors) can parse HTML into data which is then rendered on the screen. It'll need to be able to search multiple websites for books and pull information about a book such as name, author, table of contents URL, etc. Then using the table of contents URL it needs to parse a list of chapters, and finally, get the content from the currently reading chapter. Some other features will provide a better reading experience as well, such as the ability to change background colour, support multiple fonts, text-to-speech, data caching, etc.

In my head, a lot of the functionalities boiled down to UI-specific stuff, and I was pretty confident I'd be able to get them implemented without much issue. Others might be more difficult, especially learning how to parse HTML in Swift.

In summary, I was very curious to learn more about both ChatGPT and iOS app development. With my motivation high, I cracked on at the beginning of Feb 2023.

ChatGPT only provide text-based answers

After I opened up Xcode, I was very lost. Although some of the interface made sense to me, most of it was alien as I've always preferred text editors such as Sublime Text and VS Code instead of IDEs. This highlighted a massive limitation with ChatGPT, it doesn't do images. When learning something new, images can be worth more than a thousand words. Although it gave a lot of good background information about Xcode and some high-level features, it did not provide a good basic Xcode interface walkthrough. I had to result to watching a few YouTube videos showing me the basics of the different UI buttons and panels of Xcode.

Maybe one-day ChatGPT 10 will be able to utilise more data formats but right now it is severely limited by the fact it can only communicate via text. This means for anyone who intends to use ChatGPT that relies heavily on visual aspects, might be disappointed with what ChatGPT can do.

ChatGPT is amazing at answering basic questions

I was massively impressed with how easily ChatGPT answered my basic and stupid questions early on regarding both Swift and SwiftUI. It not only generated code to further illustrate its points but was also able to contextually continue the conversation when I asked follow-up questions.

There is an example where I asked it the difference between @State and @StateObject in Swift. Then without providing any context, I asked it to update its code example in the previous answer to showcase a simpler use case of @State.

[INSERT IMAGES]

With ChatGPT's help, I made a lot of progress at the beginning and quickly built up a basic book page screen that contained book cover images, author, word count, description of the book, etc.

ChatGPT even helped me to correct my Xcode configuration when I had to request data from an HTTP URL instead of HTTPS. It turns out Apple blocks non-secure connections by default, and unless I add an exception in my Info.plist the app refused to make the connection.

ChatGPT struggles with complex code problems

Later, ChatGPT showed its limits when I hit a more complex problem. I wanted to take all the text from a single chapter (long string) and split it up into multiple chunks, where each chunk fits within the current available screen space. I needed this because I want to allow people to swipe horizontally and flip through the pages like many other reading apps.

It turns out this is a non-trivial problem in Swift, and I struggled to get ChatGPT to give good answers. Some of the shortcomings of ChatGPT include:

ChatGPT gave answers that did not satisfy my requirements. I tried to reword my request differently but it was never able to provide the code to split up the pages efficiently. It only managed to provide several half-complete and less optimal solutions.
Some answers contained basic coding errors such as referring to APIs that did not exist in Swift. Interestingly, when I pasted in the Xcode error it was able to recognise its mistakes and apologised.
ChatGPT started losing context between questions. This not only meant my follow-up question's changes were added to some new snippet of code or that changing the previous code introduced other errors. But most often, it would give me brand new code solutions out of nowhere when asked to change something in the previous code example.
It was trained using old data (limited knowledge of the world and events after 2021) which means ChatGPT is not aware of the newer programming language changes. This can lead to some silly answers sometimes by returning deprecated code, etc.

This was a problem that I stuck on for days, ChatGPT and I went around in circles and it was frustrating. From this experience it is easy to see that ChatGPT is just a better Google search, it understands the question better and filters the results out with more context. But at the end of the date, it is just that, a better Google and it is not perfect.

ChatGPT can be very slow

There have been times when I simply gave up on asking ChatGPT questions because it was taking a very long time to respond. This is especially true when I used it during peak hours (free tier). However, its normal responding time can be long as well. Since it progressively generates its answers, where each word doesn't take long to generate but the time to generate them does not get any faster. This always means complex questions that require longer answers will take longer to generate than short answers. This is very different to Googling for results, some of the results' relevance level might be low, but it'll return results quickly.

When ChatGPT is working well, having it on the side feels like doing pair programming with another engineer, bouncing questions off it is a great way to learn. However, often the feedback loop was too long, which ended up taking me out of the "zone" and I found myself juggling more ideas in my head than usual as I try to break down questions into more manageable chunks for ChatGPT. Then holding on to the extra information in my head as I watch it slowly generate each word back to me.

I slowly stopped using ChatGPT

As I got better and more comfortable with Swift and Xcode, I used ChatGPT less and less. This should highlight the fact ChatGPT is just another tool in an engineer's toolbox and another source of information like Google results, YouTube videos, etc.

At the start of the project, ChatGPT was contributing more to the project than I was. But as soon I started grasping the basics of Swift, I soon took over and only asked clarification questions such as "Is it better to use @AppStorage or UserDefaults in Swift" to confirm my understanding and avoid bad practices. ChatGPT's answers always made me smile by returning diplomatic answers that explain both concepts and conclude that "it was dependent on my use case".

Conclusion

In summary, I have been very impressed with ChatGPT in the past couple of months. It has helped me work through some obscure issues, but at the same time when it came to complex programming problems, it struggled. So I had no choice but to architecting the app and take on the complex issues myself.

I have to admit ChatGPT made coding more fun, especially when working on a personal project when there are no colleagues to chat with about approaches and problems. When I get stuck on a coding problem I often find it hard to find the motivation to continue. ChatGPT has made it less daunting when I hit a brick wall. It is like a patient and knowledgeable friend, guiding and pushing me towards my goal.

Finally, in terms of the project, I'm happy to share that after around two months the app is almost done. There are a few more features I want to implement, then I'll be asking ChatGPT's help to refactor the code as right now it is a mess!

Afterwords

The media love to add "AI" to anything that isn't achieved by a human, but we should be clear that ChatGPT is strictly not a true AI. It is still based on a Machine Learning data model, albeit the best one in the world so far. It has no consciousness and when it replies it is not writing out of its understanding of the world. This showed when I asked it the complex code problem explained above. ChatGPT is great at putting publically available knowledge together in a short and easy-to-consume result, aka generating information, but it cannot "create" anything that never existed on the internet. Again, my "split this text into multiple pages" problem was a good example of this.

This also led me to question what would happen to the future ChatGPT versions. Before 2021, there was little to no "AI" generated content online (at least good content). But with the release of ChatGPT, I can imagine the internet will be plagued with generated content on so many blog posts (I'm referring to people typing a title into ChatGPT and then publishing its response as a blog post). I am very curious how future "AI" will be trained, because if it starts to consume the content generated by other "AI" then it would go into a self-approving spiral. If a wrong piece of information appears frequently enough in its training data then it could be considered a fact, just look at the Google’s AI chatbot Bard makes factual error in first demo example. But since the authors of ChatGPT have already altered how it responds to some questions, I'm sure it's not an impossible task, may just require a lot of manual tinkering.

Today I saw an article about how good ChatGPT was at medical diagnosis The newest version of ChatGPT passed the US medical licensing exam with flying colors — and diagnosed a 1 in 100,000 condition in seconds. Based on my coding experience with it so far, ChatGPT might be a good place to get some initial ideas about something, but I wouldn't trust it with my medical diagnosis yet.

Note: this article was written with zero help from ChatGPT

When is GraphQL the right choice for your Project or Business?

Sun, 12 Feb 2023 22:21:00 GMT

Building a GraphQL API requires a deep understanding of the technology and best practices. It offers developers a high level of control and freedom, which is why it is so easy to get started. However, it is also why people can design themselves into a corner if they are not careful. Security and performance implications are just typical issues of software development. GraphQL also requires that the schema is well-designed and easy to maintain; otherwise, the development team will run into a world of pain. Additionally, the development team must be trained and skilled in using GraphQL, as it requires a different way of thinking about data and APIs. In this article, let's discuss when it is a good choice to adopt GraphQL and who it is for.

What is GraphQL and why is it so good?

GraphQL provides a flexible and efficient way to query and manipulate data, as opposed to traditional REST APIs with a set structure and limitations in data retrieval. With GraphQL, the client has more control over the data it receives and can request exactly what it needs without over or under-fetching. This can result in faster and more efficient applications. Like REST, GraphQL can be used by any platform e.g. native mobile app, web, desktop app, etc.

Here are some of the main selling points of GraphQL. Many of these addresses pain points seen in the REST world:

Flexibility: GraphQL allows the client to specify exactly what data it needs, reducing the amount of over- or under-fetching data compared to traditional REST APIs. It is easier to fetch the data required for a specific use case, leading to faster and more efficient applications.
Performance: GraphQL can often result in fewer round trips to the server than REST APIs, as all the required data can be requested in a single request. It can lead to improved performance, especially in low-bandwidth or high-latency environments.
Type System: GraphQL has a built-in type system that allows you to define the structure of your data, making it easier to validate user input and ensure that your API is always in a consistent state.
Easy to use: GraphQL provides a simple yet powerful syntax for querying and manipulating data, making it easy for developers to start using it immediately.
Better documentation: The GraphQL schema acts as documentation for your API, making it easier for other developers to understand how to use your API correctly.

GraphQL isn't a REST replacement

There are many direct comparisons between GraphQL and REST. While it is good to have those discussions to understand the role of GraphQL in an organisation, the comparison is inaccurate at best. Because GraphQL isn't here to replace REST. Instead, it works best when acting as the interfacing layer that sits in front of REST.

I have worked on projects where the architecture looks like the following diagram:

We had GraphQL services that talked directly to SQL databases. It sounds great in theory but a nightmare in practice. We wrote SQL queries in GraphQL resolvers without help from the typical ORM database tools. It could have been a better dev experience and error-prone. The team did its best to refactor things and apply abstractions to reduce code logic duplications, but it didn't work. It was still hard to understand and messy. Not to mention testing anything is incredibly difficult.

GraphQL offers the most value when it doesn't have to worry about the business logic and focuses on what it does best: distributing requests and aggregating results. Of course, it doesn't mean we shouldn't have business logic in GraphQL, but based on my experience, the less logic the better.

I recommend that anyone thinking about adopting GraphQL or working on a new green field project consider the following architecture.

PS: there are SaSS services out there which can make generating some of the CRUD operations from the database easier. This might be attractive for small projects or startups. Still, it will be best for the business to leave these SaSS services and build out the REST layer internally for better control and support those customisations needed as part of the business growth journey.

GraphQL doesn't make getting your product out faster

GraphQL can be misconceiving when it comes to development time because it is perfect at some things and gives the wrong impression that it will always speed up your project.

If you agree with what I talked about in the previous section, the first elephant in the room is that adopting GraphQL means building up one additional service and layer in your architecture. This translates to more development work, both in terms of writing the schema and resolvers. On the DevOps side, there are deployment pipelines, infrastructure setup, monitoring and alerting, security, etc. All of this can take more time than implementing the GraphQL service itself.

As mentioned already, to build and use GraphQL effectively, the development team must have a deep understanding of GraphQL and consider security and performance implications. Wrong design decisions or choices could lead to more development time spent on rectifying these mistakes.

Who's best suited to adopt GraphQL?

GraphQL shines when it is used in medium or large companies with many APIs and client teams. Since GraphQL provides a single entry point for all the APIs client teams need. The added benefits of reduced over and under-fetching could improve client-side performance massively on low-end devices or customers with poor internet connections.

Introducing GraphQL is easy, but the coordination needed to make GraphQL work well in an organisation is the real test. If not careful, it will cause more overhead and delays than it was intended to improve. Schema changes become a massive talking point because it impacts everyone, both the client teams consuming from GraphQL and the underlying API teams providing data to it. Small companies or startups don't benefit enough from GraphQL to justify the cost it brings. Typically speed and release quickly is the main driver for companies that those stages, and adopting something like tRPC might be a better choice. Of course, it is always possible to migrate to GraphQL later as a replacement (or even build on top of it?) at a later point.

For medium-sized companies, who are already starting to see innovation slowdowns caused by an increasing number of REST APIs or hearing complaints from client teams about the backend, it might be time to start investing in GraphQL. There are so many benefits that come with GraphQL when done right.

GraphQL would still be a massive help for large companies, but there might be limits to what it can solve at the scale. Instead of adopting a monolith GraphQL service, they should look at more scalable options such as schema-stitching with GraphQL Mesh or schema federation with Apollo Federation. We'll get into it at another time 🙂

Fixing esbuild related CPU architecture error on Apple silicon Macs

Sun, 12 Feb 2023 15:33:00 GMT

I ran into this issue recently where esbuild was complaining about the CPU architecture in VS Code. The project was running SvelteKit (v1) for some context. The error didn't disrupt anything I was doing, but I was sick of seeing the error message everywhere and decided to fix it. I've always run Rosetta v2 on my Apple silicon Macs because of compatibility concerns. Here's my article on Setting up ZSH with VS Code on Apple silicon Mac (M1 chip). But this was the first time Rosetta caused any conflicts in my workflows. I found a fix at the end and decided to share it, given how little people have written about this.

Esbuild complaining I had installed the wrong package for my platform

Everything went smoothly until I tried to add SCSS to the project, which I have covered in a previous article How to add SCSS or SASS to SvelteKit. Later I tried to use TypeScript in the same project and saw the same error. In both cases, the error looked like this:

You installed esbuild on another platform than the one you're currently using.
This won't work because esbuild is written with native code and needs to
install a platform-specific binary executable.

Specifically the "@esbuild/darwin-x64" package is present but this platform
needs the "@esbuild/darwin-arm64" package instead. People often get into this
situation by installing esbuild with npm running inside of Rosetta 2 and then
trying to use it with node running outside of Rosetta 2, or vice versa 
(Rosetta 2 is Apple's on-the-fly x86_64-to-arm64 translation service).

If you are installing with npm, you can try not copying the "node_modules"
directory when you copy the files over, and running "npm ci" or "npm install"
on the destination platform after the copy. Or you could consider using yarn
instead which has built-in support for installing a package on multiple
platforms simultaneously.

If you are installing with yarn, you can try listing both this platform and the
other platform in your ".yarnrc.yml" file using the "supportedArchitectures"
feature: https://yarnpkg.com/configuration/yarnrc/#supportedArchitectures
Keep in mind that this means multiple copies of esbuild will be present.

Another alternative is to use the "esbuild-wasm" package instead, which works
the same way on all platforms. But it comes with a heavy performance cost and
can sometimes be 10x slower than the "esbuild" package, so you may also not
want to do that.

Debugging

First attempt - switch over from PNPM to Yarn

As shown in the error message, it recommends using Yarn due to its ability to support and configure multiple architectures via .yarnrc.yml. I was using PNPM at the time, so I thought this was what I was missing. So I tried switching over to Yarn, added the config file, and filled in the necessary configs, but sadly, it did nothing.

supportedArchitectures:
  os:
    - 'current'
  cpu:
    - 'arm64'

Second attempt - Switching over to ZSH shell instead of Rosetta shell

I thought it might be the way I am installing NPM packages. I have been running yarn install using Rosetta shell, which I've configured to run in x64 CPU architecture mode instead of the default arm64.

Here's a screenshot proving Rosetta shell (the one on the right) is running in x64 architecture mode.

I checked my package.json, which doesn't include esbuild, so it must be a nested dependency. After checking yarn.lock file, I was happy with that theory.

Then I removed my node_modules folder and installed it again. Unfortunately, nothing changed.

Third attempt - reinstalling node

Finally, I stumbled on to this Github issue comment, which suggested running node -p "process.arch" command to see which architecture Node is running on. And both ZSH shell and Rosetta shell were returning x64. So there is the issue.

The fix

I use NVM to manage my Node versions on my machine, so your fix may vary. The goal is to uninstall the current node and reinstall it using the terminal shell in arm64 mode. These were the commands I used, given I have NVM:

nvm uninstall 16
nvm install 16

After that was done, I ran node -p "process.arch" in the ZSH shell it returned arm64.

The next step is to remove and reinstall all the node packages again.

rm -rf node_modules
yarn

Now check VS Code (it may need a reload), and the annoying errors should be gone!

What I learned about JS Date from writing a simple date library

Sun, 27 Nov 2022 18:05:00 GMT

The built-in Date object is notoriously difficult to use. It is why there are several date libraries out there, such as Day.js, luxon, date-fns, etc. Recently, I've implemented a date library that acts as a light abstraction layer, providing syntactic sugar like the date libraries mentioned above. I won't go into detail about why I didn't just use one of those libraries in this article, and I will save that for another time. Instead, I want to discuss my learnings and recommendation after learnings from creating this date library.

Best way to create a new date

Like most things in JavaScript, JS Date is very flexible when parsing different date formats. Unfortunately, flexibility often comes with its issues and drawbacks. Let's first take a look at what is possible.

As you can see below, there are many different ways to create a new Date object.

const today = new Date();
const birthday = new Date("December 17, 1995 03:24:00"); // DISCOURAGED: may not work in all runtimes
const birthday2 = new Date("1995-12-17T03:24:00"); // This is ISO8601-compliant and will work reliably
const birthday3 = new Date(1995, 11, 17); // the month is 0-indexed
const birthday4 = new Date(1995, 11, 17, 3, 24, 0);
const birthday5 = new Date(628021800000); // passing epoch timestamp

// Borrowed from MDN docs

After some struggles, I found the best way to create a Date object is using the ISO8601-compliant method.

Why ISO8601-compliant method?

I find this method the most straightforward to work with:

It follows a strict format of YYYY-MM-DDTHH:mm:ss.sssZ, so there is no confusion around the date format, making it easy to read.
It is still flexible, so we can still create dates without specific data, e.g. new Date("1995-12-17") or with the hour and minutes new Date ("1992-12-17T12:50").
We can easily convert a date into UTC by adding Z suffix, e.g. new Date("1992-12-17T12:50Z").

However, I also noticed a few disadvantages of using this method.

First, it is not as flexible as creating a Date with integers, which allows parsing partial time formats. When passed just the hour,

new Date(1992, 11, 17, 12) // "1992-12-17T12:00:00.000Z" 
new Date('1992-12-17T12') // null

Second, it cannot infer an unknown date format. In the following example, it didn't know what to do when I passed 50 as the Date.

new Date(2000, 0, 50) // "2000-02-19T00:00:00.000Z" 
new Date('2000-01-50') // null

Avoid parsing integers

Although this method is the most flexible when it comes to creating new dates. Here's a list of examples:

new Date(2022, 10, 27) // "2022-11-27T00:00:00.000Z" 
new Date(2022, 10, 27, 15) // "2022-11-27T15:00:00.000Z" 
new Date(2022, 10, 27, 15, 40) // "2022-11-27T15:40:00.000Z" 
new Date(2022, 10, 27, 15, 40, 12) // "2022-11-27T15:40:12.000Z"
new Date(2022, 10, 27, 15, 40, 12, 333) // "2022-11-27T15:40:12.333Z"

The fact we always have to be mindful of month start from the index of 0 is a massive flaw. Many times, I needed to fix bugs around this when I forgot to subtract 1 when creating a new date. Here's a full explanation on StackOverflow if anyone is interested in knowing why this is the case.

This method's second and most significant problem is a lack of an easy way to create a UTC date. Whilst there is a workaround, it is long-winded and makes it difficult to read. Not to mention how easy it is to forget to add the Date.UTC() part when creating a new date.

new Date(Date.UTC(2022, 10, 27)) // "2022-11-27T00:00:00.000Z" 
new Date(Date.UTC(2022, 10, 27, 15)) // "2022-11-27T15:00:00.000Z"
new Date(Date.UTC(2022, 10, 27, 15, 40)) // "2022-11-27T15:40:00.000Z" 
new Date(Date.UTC(2022, 10, 27, 15, 40, 12)) // "2022-11-27T15:40:12.000Z" 
new Date(Date.UTC(2022, 10, 27, 15, 40, 12, 333)) // "2022-11-27T15:40:12.333Z"

Avoid Epoch Timestamps

Passing dates as Epoch Timestamps was common when I worked with PHP around 2015. However, I haven't seen this method used at all since I moved over to using Node in the backend.

First, one problem with this method is relying on the Timestamp to be in the correct time standard (e.g. UTC) to begin with. If we pass in a UTC timestamp, then we'll get a UTC date object, and vice-versa with non-UTC timestamps.

console.log(new Date(2000, 3, 0).getTime()) // 954457200000
console.log(new Date(Date.UTC(2000, 3, 0)).getTime()) // 954460800000

Secondly, timestamps are extremely difficult to comprehend for humans. There could be a bug in front of my eyes, and I wouldn't be able to spot it. So unless there is a solid reason to use this method of storing date, I would avoid it.

Avoid non-ISO8601-compliant string formats

Not sure I need to go into much detail regarding why new Date("December 17, 1995 03:24:00") is a bad idea. The lack of a strict standard makes creating new date inefficient at best and error-prone at worst.

UTC, GMT, ISO and local time

Time zones are confusing to work with and we are barely scratching the surfaces here with these acronyms.

What is the difference between UTC and GMT?

Coordinated Universal Time (UTC) is a non-location-based time standard. It means no country on Earth uses it as a local time. Instead, think of it as international time. Greenwich Mean Time (GMT) is a time zone in some European and African countries. In the UK, we change between GMT and British Summer Time (BST) depending on Daylight Saving Time (DST).

It is always better to use UTC when possible. For example, it is a headache to consider time zones and daylight savings when we want to add two hours to a JS Date.

What is ISO Date?

International Organization for Standardisation (ISO) is an organisation that develops and publishes international standards. Essentially, they developed the concept of UTC.

ISO is only referenced by JS Date as an output format, e.g. new Date().toISOString(), which returns a string like this "2022-11-27T15:40:12.333Z". Since the value is always zero UTC offset, you could also think of it as .toUTCString() (this is something I made up!).

Local time

By default, JS Date will use the local device to figure out the timezone and use it as a base for date manipulations. This can be desirable since it makes displaying Date on the client side much more straightforward as now, as developers, we don't need to consider time zone.

However, this approach has a massive problem if we ever need to work across different time zones. We also work across time zones more than you think. For example, clients and servers are often in different time zones. Sending a non-UTC date from a client to a server will usually result in bugs because the two sides have a very different understanding of the date's value.

Always use UTC

After running into several bugs, I finally decided to refactor my date library to always use UTC. One of the bugs I encountered was unit tests failing in the pipeline when they passed locally. This is because the pipeline's server was at a different timezone than my local machine.

Another reason why I'm so fixated on using UTC is that when we create a new date with non-UTC the output of getDate() and getUTCDate() is different in the example below.

// code to get the number of days in the current month

new Date(2022, 3 + 1, 0) // 2022-04-29T23:00:00.000Z
new Date(2022, 3 + 1, 0).getDate() // 30
new Date(2022, 3 + 1, 0).getUTCDate() // 29 <= wrong

new Date(Date.UTC(2022, 3 + 1, 0)) // 2022-04-30T00:00:00.000Z
new Date(Date.UTC(2022, 3 + 1, 0)).getDate() // 30
new Date(Date.UTC(2022, 3 + 1, 0)).getUTCDate() // 30 <= correct

It's also worth mentioning that using UTC doesn't mean we lose the ability to display the correct Date and time within our applications. On the client side, we can use toLocaleDateString() to convert the UTC Date object back to any time zone we would like.

Setting up ZSH with VS Code on Apple silicon Mac (M1 chip)

Sun, 10 Apr 2022 21:38:00 GMT

I upgraded to MacBook Pro 14" a few months back. I've been using it for personal development and commercial contracts (working inside large teams) with no problem. For anyone who's on the fence about buying a Mac with Apple silicon (AKA M1 chip), I would encourage them to go for it. I want to point out that the lack of issues could be because I'm running all codebases with Resetta 2, an emulator that could translate Intel chip commands so they could run on Apple Silicon. In this article, I'll share my setup in case it helps anyone out there.

A bit of history

You might be wondering why it is called Rosetta 2. It is because this is the second version of Rosetta, as the name indicates. Apple introduced the first version back in 2006 to help migrate users from PowerPC processors to Intel. So it is somewhat ironic that the first version of Rosetta brought Apple users to Intel, only to take them all away after 16 years with the second version.

Install Rosetta 2

If you have an Apple Silicon Mac, it should prompt you to install Rosetta when you try to open any app designed for Intel Macs.

If you don't want to go through the trouble of first finding an app that is incompatible with your mac first, you can do it in the terminal just as easily:

softwareupdate --install-rosetta --agree-to-license

Install VS Code

It is worth noting that VS Code now natively support Apple Silicon. So the install process is the same as before - download link.

Zsh

Zsh should be pre-installed on Macs now. So you shouldn't need to do anything here.

Setup Rosetta Terminal

We will duplicate the mac Terminal app at this stage and run one of them through Rosetta. This way, when we run any commands, it will be no different than running them on an Intel-based Mac.

First, let's check the CPU architecture by running arch command in the terminal. Unless you have configured it already, it should say arm64.

$ arch
arm64

Open the Finder app on your app, then use the cmd + shift + u shortcut to get into the Utilities folder.

Now, right-click on the Terminal app and click on Duplicate.

Then rename one of them to Rosetta Terminal or whatever you want to call it.
Right-click on the Rosetta Terminal app and click on Get Info

Toggle on the Open using Rosetta checkbox, and we are done!

Open "Rosetta Terminal" and rerun the arch command. We can see now it says i386.

$ arch
i386

Configure VS Code

Open up VS Code and go to settings.json, then paste in the following:

{
  ...  
  "terminal.integrated.profiles.osx": {
    "rosetta": {
      "path": "arch",
      "args": ["-x86_64", "zsh", "-l"],
      "overrideName": true
    }
  },
  "terminal.integrated.defaultProfile.osx": "rosetta",
  ...
}

Then make sure the selected default is the rosetta profile, as shown below:

Check everything is working

Finally, let's type arch command in the VS Code's integrated terminal. If you see i386 then it is working correctly. Congrats!

Setting up Vitest within SvelteKit repo

Sun, 10 Apr 2022 19:51:00 GMT

SvelteKit doesn't come with unit test tools out of the box, so we'll have to configure our own. Usually, I'd add in Jest because it's always been my favourite testing framework. However, I recently heard good things about Vitest and thought I'd give it a try. For those unfamiliar with Vitest, it is a drop-in replacement for Jest and is much faster. In addition, I've noticed another benefit of how simple it is to configure compared to Jest since it natively supports TypeScript. However, we still need to do a couple of things to get Vitest working properly with SvelteKit. Let's go through what these things are.

Install Vitest

First, we need to install it. So go ahead with one of the following commands depending on the configuration of your project.

// with npm
npm i -D vitest

// or with yarn
yarn add -D vitest

// or with pnpm
pnpm add -D vitest

If you need more information, maybe follow the official guide.

Configure Vite/Vitest

Once Vitest has been configured, we need to update vite.config.ts to look like the following:

/// <reference types="vitest" />
import { defineConfig } from 'vite';
import { svelte } from '@sveltejs/vite-plugin-svelte';

export default defineConfig({
	plugins: [svelte({ hot: !process.env.VITEST })],
	test: {
		globals: true,
		environment: 'happy-dom'
	}
});

Notice the /// <reference types="vitest" /> at the top of the file in order to define test inside defineConfig. If we don't add this, we'll get the following error message because the defineConfig type doesn't understand the test property.

Here's the full error message:

Argument of type '{ plugins: Plugin[]; test: { globals: boolean; environment: string; }; }' is not assignable to parameter of type 'UserConfigExport'.
  Object literal may only specify known properties, and 'test' does not exist in type 'UserConfigExport'.ts(2345)

Configure TypeScript

Next, we need to configure TypeScript, so it understands the globals such as describe, test, or it. The tsconfig.json should be just extending the default tsconfig in svelte-kit. We want to add "vitest/globals" type in compilerOptions.

{
	"extends": "./.svelte-kit/tsconfig.json",
	"compilerOptions": {
		"types": ["vitest/globals"]
	}
}

If we don't do this, when you create or open a test file, you'll see the following errors. It is because typescript doesn't know where to look up these globals.

Again, here's the full error message:

Cannot find name 'describe'. Do you need to install type definitions for a test runner? Try `npm i --save-dev @types/jest` or `npm i --save-dev @types/mocha`.ts(2582)

All done

If you make those two small config file changes, everything should work as expected.

I'm having a blast with Vitest, love the improved SPEED! More importantly, coming from Jest, I didn't need to re-learn any new APIs. In fact, I don't even remember I'm working with a different test framework most of the time.

How to add SCSS or SASS to SvelteKit

Sat, 09 Apr 2022 21:45:00 GMT

Although Svelte/SvelteKit support other CSS preprocessors, nothing comes baked-in so we need to manually install dependencies in order to use them in our project. In this blog post, we'll go over the simplest way to achieve this.

For people who are new to Svelte or SvelteKit, the syntax for using SCSS or SASS is simple, just need to add lang="sass" attribute to style tag.

// SASS

<style lang="sass">
	.page {
		width: 100%;
		max-width: var(--column-width);
		margin: var(--column-margin-top) auto 0 auto;
	}
</style>

// SCSS

<style lang="scss">
	.page {
		width: 100%;
		max-width: var(--column-width);
		margin: var(--column-margin-top) auto 0 auto;
	}
</style>

However, if I try to do this in a new SvelteKit repository, I'll see the following error message.

Here's the full error message:

Cannot find any of modules: sass,node-sass

Error: Cannot find module 'node-sass'
Require stack:
- /Users/hao/workspace/demo/node_modules/.pnpm/svelte-preprocess@4.10.5_svelte@3.46.6+typescript@4.6.3/node_modules/svelte-preprocess/dist/modules/utils.js
- /Users/hao/workspace/demo/node_modules/.pnpm/svelte-preprocess@4.10.5_svelte@3.46.6+typescript@4.6.3/node_modules/svelte-preprocess/dist/autoProcess.js
- /Users/hao/workspace/demo/node_modules/.pnpm/svelte-preprocess@4.10.5_svelte@3.46.6+typescript@4.6.3/node_modules/svelte-preprocess/dist/index.js

The file cannot be parsed because style requires a preprocessor that doesn't seem to be setup or failed during setup. Did you setup a `svelte.config.js`? If you use SCSS, it may be necessary to add the path to your NODE runtime to the setting `svelte.language-server.runtime`, or use `sass` instead of `node-sass`. 

See https://github.com/sveltejs/language-tools/tree/master/docs#using-with-preprocessors for more info.svelte(style)

Solution

All we have to do is add sass to our dev-dependencies, then SvelteKit will automatically pick it up.

There are a few packages we can use to achieve this:

node-sass is the one that is referenced the most, but since it has been deprecated I strongly recommend you avoid using this.
dart-sass is a more modern implementation of node-sass. But it has since been renamed to sass.
sass is the one you should use!

// NPM

npm i -D sass

// Yarn

yarn add -D sass

// PNPM

pnpm add -D sass

Then you should see the above error message disappear.

If the error message doesn't go away, try reloading your editor. If you use VS Code on a Mac, you can do this with cmd + p then search Developer: reload window.

Note: you may also need to restart the SvelteKit app for this to take effect.

The best programming language to use in coding interviews

Tue, 05 Apr 2022 23:34:00 GMT

As I've mentioned in a previous blog post Becoming an interview engineer, I have been an interview engineer at Karat for about six months now. As an interviewer, it is my job to observe different candidates attempt the same questions. Over time, I noticed something intriguing - the best performing candidates I interviewed all use the same programming language. To make it clear, Karat allows candidates to pick from 26 programming languages (excluding Markdown) when attempting coding interviews. Out of these languages, candidates tend to choose the same few languages. Out of these languages, there's a clear winner when it comes to the best language to use for coding interviews - Python. Of course, candidates can still perform well using other languages, but I strongly feel the interviewing "game" is rigged against them.

The best candidates I've interviewed all used Python 3. I don't think the version of Python matters here, but I thought I'd be accurate. So let's talk about why I think Python is the best language to use in coding interviews (if you are familiar with it already).

Disclaimer: before people assume I'm just a Python fanboy, I want to clarify that it isn't my language of choice, and it isn't something I use regularly. I can write basic scripts in Python, and it wouldn't be hard for me to pick it up again quickly. But I wouldn't put myself even close to the Python camp based on my skills and experience. So hopefully, that proves I'm not being biased here.

Indentation > curly-brace system

As the interviewer, I cannot say anything during interviews that directly helps the candidate, or they will get marked down for my comments. So imagine how painful it is for me to watch people spend a few minutes trying to fix a missing curly bracket.

It might sound stupid to suggest brackets are a big deal. But when interviewing, people tend to make more mistakes due to the time pressure and knowing someone is watching them code. Our brains focus on the interview question and don't pick up simple syntax issues straight away. Coding slowly isn't an option to get the best marks since companies want fast and accurate solutions. When people rush, minor issues like missing brackets become harder to spot. I once watched someone spend 10 minutes struggling with a bracketing issue. Based on comments from other interviewers, I wasn't the only one either.

If you are not convinced, I've included some basic code snippets below to illustrate how the brackets can be confusing when coding under pressure. It is also worth noting that the code block tends to be more complex and be much longer in interviews, which makes spotting missing brackets even harder.

# Python

def helloWorld:
	for i in range(len(some)):
		for j in range(len(random)):
			if falseCondition:
				for k in range(len(random)):
					...

// JavaScript

function helloWorld() {
	for (const i in some) {
		for (const j in random) {
			if (falseCondition) {
				for (const k in loop) {
					...	
				}
			}
                 }
           }
      }
}

Did you notice there is one too many bracket in the JavaScript code?

Dynamic type = less code = code faster

When it comes to writing code fast, it is tough for statically typed language to complete with dynamically typed languages. Things as basic as declaring a function or variable are faster to do in Python than in a language like Java.

Here is a comparison between declaring a function that takes in a 2D integer array and returns the same type in Python and Java, respectively.

# Python

def myFun(arr):

// Java

public static int[][] myFun(int[][] arr) {

Python developer has already started implementing the function by the time Java developer types out the word "static". Yes, this is just one line of code, but Python code is always shorter with the same approach. A few characters here and there don't sound much. But when there are tens or hundreds of lines of code, it all adds up because it gives a developer more time to be debugging code instead of typing things out.

Longer code also means more likely for mistakes to creep in. For example, people make mistakes with function data types or forget to type the "static" keyword when declaring the function. With dynamically typed languages, you simply don't have these concerns. Even compared to another dynamically typed language like JavaScript, I have to admit Python code is a lot faster to write because it requires fewer characters to implement the same logic.

"Statically typed languages have advantages too!" I hear you say. That's 100% true, and I agree. Personally, I went from extremely against statically typed language at the start of my career to starting every single project now using TypeScript. Unless I'm throwing something together quickly, TypeScript gets my vote every time. Types come with safety and guarantee, and IDE can leverage it to give better autocomplete, it can catch issues that dynamically typed language cannot, and plenty more advantages. But when shorter code beats good code in an interview, dynamically typed languages win hands down. Like many places, at Karat, we put much more emphasis on working solutions than nice-looking code.

Some languages are better for implementing algorithmic questions

Maybe it's because I'm a JavaScript developer, so this is a more significant pain point. For others languages, this might be irrelevant. But, as a JavaScript developer, I find the lack of useful util functions makes coding algorithm solutions significantly harder.

To demonstrate my point further, here is a comparison between Python and JavaScript for finding the intersections between two arrays. It is not just that the Python code is shorter; the JavaScript code is also harder to read and understand. All this adds friction to the interview process.

# Python

def intersection(a, b):
    return list(set(a).intersection(b))

// JavaScript

function getIntersection(a, b) {
    const set1 = new Set(a);
    const set2 = new Set(b);

    const intersection = [...set1].filter(
        element => set2.has(element)
    );

    return intersection;
}

Would I find an intersection like this in an interview when coding in JavaScript? Of course not. My first choice will be to implement the solution using forEach() loop because I'm more familiar with it. In interviews, familiarity is king. But this implementation isn't any quicker to write and could introduce unnecessary bugs than the Python approach. It is also inferior in terms of time complexity.

function getIntersection(a,b) {
	const arr = [];

	a.forEach(elem => {
		if (b.contains(elem)) {
			arr.push(elem);
		}
	}

	const arr;
}

To ensure this isn't a problem that only partially affects JavaScript developers, I researched online and found this question on stackoverflow, so it seems GoLang has a similar problem.

Conclusion

Now, should you drop everything and learn Python for all future interviews? No, that would be crazy. People can be good at any language they spend time mastering. The better thing to do is practice with more interview-style coding questions and find a solution faster. Finding the most straightforward solution will save you more time than switching to Python and typing less code. It is often best to go with something more familiar and comfortable than to min-max the benefits I explained above in time-sensitive environments. However, for people who already use multiple languages where Python is already in their toolbox, I'd strongly recommend using it for interviews unless the job requires a specific language.

Some candidates ask me how come I became an interview engineer. It always sounds like a cliche, but I enjoy seeing different people approach the same interview questions from different angles and coming up with unique solutions. Furthermore, I often learn something from the candidates I've interviewed. Believe it or not, an interview I recently conducted inspired this blog post. If this sounds like something you are interested in, you can apply to become an interview engineer using this link (if you are from the US) or this link (from any other country)!

Becoming an interview engineer

Wed, 30 Mar 2022 21:05:00 GMT

A few months ago, I was approached by a recruiter telling me about this fully remote job at Karat that offers decent pay. If you haven't heard of Karat before, it is the world's largest interviewing company. They use a global network of professional software to conduct technical interviews for companies, some of which you may have heard of before: Robinhood, Indeed, Roblox, Coinbase and more. Of course, at the time, I didn't know this, but the recruiter hooked me on the idea of setting my hours to work part-time or full time. After receiving several emails, I finally decided to reach out to hear more about the role. The phone conversation assured me that the role was an excellent way to earn extra money and get interviewer experience. I wanted to do it part-time, so I could always stop on my terms if it got too demanding.

When I spoke with the recruiter on the phone, he mentioned that I'd have to go through this mandatory boot camp (which lasts just over a week) before I could start conducting interviews. This training period is billable, which means they effectively pay people to learn. How brilliant is this! This boot camp aims to teach new joiners basic interviewing skills and learn the basics to provide a good Karat style interview experience for candidates.

Note: later, I found out the learning track I'm about to explain isn't the standard process at Karat. I joined around Christmas time, which meant a peak period for grad interviews. So this track is unsurprisingly called "uni track" (I can't remember the exact wording). We focused on learnings relevant to interviewing candidates for internships or grad jobs.

The boot camp

Getting setup was pretty easy, Karat sent me some links, and I registered. They also sent me Google Docs links that contained schedules and learning materials in various formats. The materials cover topics such as how to be a good interviewer, explained important principles we had to follow and what to do in various situations.

Most of the training was self-learning, which was perfect because I could plan it around my free time. Some events were mandatory to attend, but most of them were in the late afternoon for me due to timezone differences. Maybe I could have managed it without taking time off work, but I booked days off just in case.

One thing the recruiter didn't mention was that there'd be an assessment at the end of the boot camp to determine whether they keep me on or not, which makes complete sense. Just weird the recruiter didn't mention it to me. It's not like I won't find out since it was the first thing they said to us in the induction session. To pass the boot camp and get accepted, I had to conduct two mock interviews as the interviewer. These two interviews will be reviewed and marked, and then finally, a decision will be made.

Day 1

The first day was straightforward. I only had to join a mandatory induction meeting which lasted about an hour. After that, I was free to plan the rest of my day, most of which I spent reading the learning materials. I wanted to do well in this boot camp since this job opportunity had a lot to offer, and getting kicked out before I even got in the door will feel pretty crappy. As a result, I spent around 5 hours learning the materials on top of the induction session.

Day 2

On the second day, the plan was to finish any self-learning by the end of the day. Since Day 3 will primarily focus on practice for the assessment interviews, where we'll use everything in the materials. I finished the self-learning materials and then went back and noted areas I felt were important.

In addition, other bits and pieces happened on this day too. Such as preparing for practice interviews and submitting interview question answers to prove we are familiar with them. All in all, this took me somewhere between 4 - 5 hours complete.

Day 3

A Karat's mentor hosted a two-hour video call to explain the next steps and answer questions we had from the previous two days. Then everyone got randomly allocated into pairs, so we could each take turns practising interviewing each other to get familiar with the interview format and style. Craig and I got paired together, and it turns out we both wanted to do well in the upcoming assessments. So we practised everything we learned from each other, testing each other on mock questions and discussing them in great detail. We spent nearly 3 hours on this call, which was supposed to take less than two hours. But I felt more confident by the end of it.

Day 4

It is the day for us to complete the first assessment interview. Karat already scheduled time slots for two sessions. Like the practice, people in the boot camp will play interviewer in one session and candidate in the other. According to the schedule, I first play the candidate for someone else, then later take on the interviewer role.

I joined the call a few minutes early, but my "interviewer" never showed. After a few minutes, I messaged the person (my partner) to see if he had forgotten or wished to push the interview back for whatever reason. He responded and said he was joining. A couple more minutes later, he joined but dropped off almost immediately due to internet issues. After some back and forth, Karat's support person decided to postpone this until my partner's internet gets better, which is likely to be the next day. It sucked for me because I was all mentally prepared.

Whilst I waited for the assessment interview, I created my interview script to use during interviews to make sure I followed the structure and format of the interview correctly going forward. Maybe I should have thanked the person I was initially paired with because it has proven very handy later.

A couple of hours later, they found someone willing to play the candidate for me later during the day. This time it is an experienced interviewer and not someone from the boot camp. During the assessment interview, she played a tricky candidate on purpose and pushed me to my limits. I was surprised but handled the "unexpected" events relatively well during the interview. After the interview, I spent about an hour on the mandatory write up where I had to describe and mark the candidate's performance. Later, I attended an optional Group Q&A session, which lasted about an hour and a half. Only a handful of us attended, so we are got plenty of opportunities to ask questions.

Based on the schedule, this was supposed to be a short day. But in reality, at least for me, it was full of surprises due to my original assessment getting delayed. As a result, I spent around 10 hours on everything mentioned above. It was quite an exhausting day.

Day 5

The day's main activity was to conduct another practice interview, again once as a candidate and another as interviewer. Since Craig and I got on so well, we requested to be paired together again. Like last time, we went through the exercise without any drama. First, we went through our assessment interviews as part of this call, and then we took a step further by sharing our feedback results and analysing them together. I managed to get everything done in roughly 3 hours.

Day 6

It's time for the second assessment. I had a different partner this time, and fortunately, there was no drama. We went through the two interviews quickly (two hours in total). Like the previous assessment, I played candidate in one interview and then interviewer in another.

After the two interviews, I spent about an hour on the interview write-up. So, in total, I completed everything in 3 hours.

Day 7

We all waited around for news about the final result.

After boot camp

A few days later, I heard back from Karat that I had passed the assessment. I was relieved that I got through and slightly nervous about interviewing real candidates. Fast forward a few months; I've completed 40 interviews since then. It's been a unique educational experience. I learned how to conduct good interviews adapted to the candidate, and the real-world experience was also invaluable. But I won't go into more detail in this blog post.

Conclusion

"You never really know something until you teach it to someone else." - John Maxwell

When looking back at my experience at Karat, the above quote comes to mind. It couldn't be more true when it comes to technical interviews. I never realised before that interviews are more than just asking questions and checking if the candidate gave the correct answer. Conducting interviews at a pace that is best suited for the candidate, making the candidate feel comfortable to showcase the best versions of themselves, effectively presenting questions, and more are just some of the challenges I faced. I guess what I'm trying to say is interviewing someone is easy; providing a good interview experience for every candidate is hard.

I won't lie; the money caught my attention first and is still an excellent incentive. Working from anywhere in the world with a good internet connection and setting my hours feels like a dream. Honestly, I can't ask for a better part-time job. But through the boot camp and the interviews that followed, I realised I still had a lot to learn.

So, what's next?

As mentioned already, I've conducted 40 interviews so far. I was also lucky enough to get promoted to senior interview engineer a few months back. I will continue with this part-time role, and my next goal is to reach 100 interviews!

If this sounds interesting to you and it's something you want to try out, you can apply using this link (if you are from the US) or this link (from any other country)!

Software field salary guide for looking for a new job in 2022

Sun, 06 Feb 2022 18:27:00 GMT

Are you thinking of switching jobs or moving to a different role altogether in 2022? Knowing what salary to ask for is always tricky since there are a lot of factors at play. But having a rough figure might at least put you in the right ballpark and avoid that mistake of saying yes to an underpaid job. I came by this list of salaries for different roles in the software development domain. From what I can tell the figures in these tables are pretty accurate. Hope you find it helpful!

All these are UK salaries, so may not apply in other countries.

Software	Permanent	Contractor Day Rate (outside IR35)	Contractor Day Rate (inside IR35)
Principle Software Engineer	£110k - £150k	£750 - £900	£800 - £1000
Lead Software Engineer	£90k - £115k	£600 - £700	£750 - £900
Senior Java Engineer	£75k - £90k	£500 - £600	£700 - £850
Java Engineer	£60k - £75k	£450 - £500	£650 - £750
Senior Frontend Engineer	£75k - £95k	£550 - £650	£600 - £750
Frontend Engineer	£45k - £60k	£400 - £500	£500 - £600
Senior .NET Engineer	£70k - £100k	£500 - £600	£650 - £750
.NET Engineer	£45k - £60k	£450 - £500	£600 - £650
Senior Node.js Engineer	£70k - £95k	£550 - £750	£600 - £750
Node.js Engineer	£45k - £70k	£500 - £550	£500 - £600
Senior Scala Engineer	£80k - £95k	£600 - £700	£750 - £850
Scala Engineer	£60k - £80k	£500 - £600	£675 - £800
Senior Golang Engineer	£70k - £95k	£600 - £700	£700 - £750
Golang Engineer	£50k - £70k	£550 - £600	£600 - £700
Senior Kotlin Engineer	£85k - £95k	£700 - £850	£700 - £800
Kotlin Engineer	£70k - £80k	£650 - £750	£650 - £750
Senior PHP Engineer	£55k - £80k	£500 - £600	£600 - £750
PHP Engineer	£40k - £55k	£400 - £500	£550 - £650
Senior Python Engineer	£70k - £100k	£600 - £700	£700 - £800
Python Engineer	£50k - £70k	£500 - £600	£600 - £700
Lead Data Engineer	£90k - £120k	£700 - £800	£800 - £1000
Senior Data Engineer	£80k - £100k	£600 - £700	£750 - £900
Data Engineer	£50k - £75k	£550 - £650	£650 - £750
Senior iOS Engineer	£75k - £90k	£450 - £650	£600 - £800
iOS Engineer	£50k - £75k	£300 - £450	£500 - £650
Senior Android Engineer	£75k - £90k	£450 - £650	£600 - £800
Android Engineer	£50k - £75k	£300 - £450	£500 - £650
Senior React Native Engineer	£65k - £95k	£550 - £750	£700 - £850
React Native Engineer	£40k - £65k	£500 - £550	£550 - £650
Senior Flutter Engineer	£80k - £100k	£550 - £750	£700 - £850
Flutter Engineer	£50k - £80k	£500 - £550	£550 - £650
Senior C++ Engineer	£45k - £75k	£400 - £700	£750 - £900
C++ Engineer	£30k - £45k	£300 - £350	£550 - £700
Senior Ruby Engineer	£65k - £85k	£450 - £600	£650 - £800
Ruby Engineer	£45k - £65k	£350 - £450	£450 - £600
Senior Clojure Engineer	£75k - £95k	£600 - £700	£700 - £900
Clojure Engineer	£60k - £80k	£500 - £600	£600 - £750
Junior Engineer	£35k - £55k	£350 - £450	£450 - £550
Grad Engineer	£25k - £35k	-	-

DevOps & Infra	Permanent	Contractor Day Rate (outside IR35)	Contractor Day Rate (inside IR35)
DevOps Manager	£90k - £120k	£700 - £1000	£800 - £1200
Senior DevOps Engineer	£75k - £110k	£650 - £800	£750 - £1000
DevOps Engineer	£50k - £70k	£500 - £650	£600 - £850
Junior DevOps Engineer	£35k - £50k	£400 - £500	£500 - £700
Grad DevOps	£25k - £40k	£200 - £300	£300 - £500
Senior System Admin	£60k - £90k	£500 - £700	£600 - £900
System Admin	£40k - £50k	£350 - £500	£450 - £700
Junior System Admin	£30k - £40k	£300 - £350	£400 - £550
Grad System Admin	£25k - £40k	£200 - £300	£300 - £500
Application Support	£40k - £70k	£300 - £500	£400 - £700
Security & Network Enginner	£50k - £100k	£400 - £750	£500 - £950
SRE Engineer	£60k - £120k	£500 - £750	£600 - £950
Database Admin	£50k - £100k	£300 - £600	£450 - £800

QA & Testing	Permanent	Contractor Day Rate (outside IR35)	Contractor Day Rate (inside IR35)
Lead Tester Engineer	£80k - £95k	£600 - £750	£700 - £950
Senior Tester Engineer	£60k - £80k	£500 - £600	£600 - £800
Tester Engineer	£40k - £60k	£400 - £500	£500 - £700
Junior Tester Engineer	£30k - £40k	£300 - £400	£400 - £600
Grad Tester Engineer	£25k - £35k	£250 - £300	£350 - £500
Senior SDET	£70k - £90k	£550 - £700	£650 - £900
SDET	£50k - £70k	£450 - £550	£550 - £750
Junior SDET	£30k - £45k	£300 - £400	£400 - £600
Grad SDET	£25k - £35k	£250 - £300	£350 - £500

Here are also some salaries and rates for other roles related to software development.

Architects	Permanent	Contractor Day Rate (outside IR35)	Contractor Day Rate (inside IR35)
Solution Architect	£95k - £130k	£600 - £750	£750 - £1000
Technical Architect	£100k - £140k	£700 - £900	£900 - £1200

Product, UX & Design	Permanent	Contractor Day Rate (outside IR35)	Contractor Day Rate (inside IR35)
Senior Product Manager/Owner	£75k - £100k	£550 - £800	£600 - £900
Product Manager/Owner	£60k - £75k	£400 - £550	£500 - £650
Senior UX Designer	£65k - £85k	£500 - £650	£600 - £700
UX Designer	£45k - £65k	£300 - £500	£500 - £600
Senior UI Designer	£65k - £85k	£500 - £650	£550 - £650
UI Designer	£45k - £65k	£300 - £500	£450 - £550
Senior Product Designer	£65k - £85k	£500 - £650	£600 - £700
Product Designer	£45k - £65k	£300 - £500	£500 - £600
User Researcher	£45k - £65k	£400 - £600	£500 - £650

Delivery	Permanent	Contractor Day Rate (outside IR35)	Contractor Day Rate (inside IR35)
Delivery Manager	£70k - £110k	£450 - £750	£550 - £850
Programme Manager	£80k - £120k	£600 - £800	£700 - £1000
Senior Project Manager	£70k - £90k	£550 - £650	£650 - £750
Project Manager	£50k - £65k	£400 - £550	£500 - £650
Senior Business Analyst	£75k - £85k	£525 - £600	£575 - £675
Business Analyst	£50k - £70k	£300 - £550	£400 - £650
Agile Coach	£80k - £120k	£600 - £1000	£700 - £1200
Scrum Master	£60k - £80k	£400 - £600	£500 - £700
Project Coordinator	£25k - £40k	£200 - £350	£250 - £400

Senior Appointments	Permanent	Contractor Day Rate (outside IR35)	Contractor Day Rate (inside IR35)
CTO/CIO/CPO	£160k - £230k	£1000 - £1500	£1400 - £2000
Early Stage Startup CTO	£120k - £170k	£750 - £1000	£900 - £1400
VP/Head of Engineering	£120k - £175k	£800 - £1000	£1100 - £1600
Engineering Manager	£95k - £130k	£600 - £800	£900 - £1200
VP/Head of Product	£120k - £180k	£750 - £1000	£950 - £1400
Chief/Head of Architecture	£120k - £160k	£850 - £1000	£1000 - £1500
Head of Infra/DevOps/Cloud	£100k - £150k	£750 - £1000	£900 - £1300
Head of Project Management	£85k - £120k	£550 - £650	£700 - £900
Head of QA/Testing	£80k - £110k	£600 - £800	£800 - £1100

Talent Acquisition & HR	Permanent	Contractor Day Rate (outside IR35)	Contractor Day Rate (inside IR35)
VP People	£90k - £120	£600 - £800	£750 - £1000
Head of recruitment	£85k - £110k	£500 - £600	£600 - £850
Technical Sourcer	£35k - £60k	£250 - £350	£350 - £550
Lead Recruiter	£60k - £85k	£300 - £500	£450 - £700
Recruitment Coordinator Recruiter	£30k - £50k	£200 - £250	£300 - £400

How to run third party applications securely on the same domain?

Thu, 30 Dec 2021 16:12:00 GMT

Have you wondered how companies like Heroku, Netlify and Vercel host and run your applications on a single domain? Is that even safe? What if people try to set/get cookies in other people's sub-domain or root domain? I recently ran into this problem at work, and we ended up going down the rabbit hole of how it all works. I was pleasantly surprised by how it all works, so let's talk about it!

The answer to the above questions is simple, there isn't a good solution, at least not unless we manually tell each browser that we want our domain handled differently. Just relying on the browsers alone, there is no good way around the fact rogue developers could do harmful things. For example, they could deploy an app on bad-app.mysite.com and then try and set cookies on good-app.mysite.com created by another developer like you or me. Worse still, there is even nothing stopping rogue devs from setting cookies via JavaScript on the root domain mysite.com. So for companies like Netlify and Heroku, hosting different apps on the shared domain seems impossible due to the lack of separation and security concerns, right?

Luckily, we have this thing called Public Suffix List (PSL). It is a cross-browser initiative that's created and maintained by Mozilla volunteers. It is essentially a list of domains put together under its unique specification (more on this later). Different browsers would then retrieve this list at various stages and treat these domains differently from others when it comes to setting and getting cookies (Chromium browsers pulls this list during compilation and bake it into the installable app).

For example, say we want to start hosting third-party apps on subdomains of mysite.com. We could submit a change to the PSL Github repository. After the maintainers verified that we are the owner of a given domain, they will merge in our pull request, and then our domain will be added to the official PSL. Going forward, when browsers release new versions they use the PSL with our domain inside, and JavaScript can no longer set cookies on the root domain or other subdomains.

But why a manual list?

As explained already, there is no good way to determine which domain and sub-domain we should treat differently without help. Here is part of the explanation on the official PSL website:

There was and remains no algorithmic method of finding the highest level at which a domain may be registered for a particular top-level domain (the policies differ with each registry), the only method is to create a list.

PSL specification

Here is the official specification of the PSL:

The list is a set of rules, with one rule per line.
Each line is only read up to the first whitespace; entire lines can also be commented using //.
Each line which is not entirely whitespace or begins with a comment contains a rule.
Each rule lists a public suffix, with the subdomain portions separated by dots (.) as usual. There is no leading dot.
The wildcard character * (asterisk) matches any valid sequence of characters in a hostname part. (Note: the list uses Unicode, not Punycode forms, and is encoded using UTF-8.) Wildcards are not restricted to appear only in the leftmost position, but they must wildcard an entire label. (I.e. ..foo is a valid rule: *bar.foo is not.)
Wildcards may only be used to wildcard an entire level. That is, they must be surrounded by dots (or implicit dots, at the beginning of a line).
If a hostname matches more than one rule in the file, the longest matching rule (the one with the most levels) will be used.
An exclamation mark (!) at the start of a rule marks an exception to a previous wildcard rule. An exception rule takes priority over any other matching rule.

Given we have a basic PSL that looks something like this:

1. com
2. *.jp             // Hosts in .hokkaido.jp can't set cookies below level 4...
3. *.hokkaido.jp
4. *.tokyo.jp       // ...except hosts in pref.hokkaido.jp, which can set cookies at level 3.
5. !pref.hokkaido.jp
6. !metro.tokyo.jp

The browser will treat the above domains as follows:

Cookies may be set for foo.com.
Cookies may be set for foo.bar.jp. Cookies may not be set for bar.jp.
Cookies may be set for foo.bar.hokkaido.jp. Cookies may not be set for bar.hokkaido.jp.
Cookies may be set for foo.bar.tokyo.jp. Cookies may not be set for bar.tokyo.jp.
Cookies may be set for pref.hokkaido.jp because the exception overrides the previous rule.
Cookies may be set for metro.tokyo.jp, because the exception overrides the previous rule.

Real PSL examples

So what does submission in PSL looks like? Here's the official submission for Netlify and Heroku:

// Netlify : https://www.netlify.com
// Submitted by Jessica Parsons <jessica@netlify.com>
netlify.app

// Heroku : https://www.heroku.com/
// Submitted by Tom Maher <tmaher@heroku.com>
herokuapp.com
herokussl.com

If you are interested, you can find the full list on the PSL website.

Conclusion

In case you need to build a site that hosts apps from strangers, I hope now you know what to do!

I also want to say that when the web is evolving at a remarkable rate and so much innovation, it is crazy to think there is this list on the internet manually managed by a group of people that determines the behaviours of our browsers. It is nice to see some human power when we are surrounded by machines :)

Useful resources

How to fix Jest and Mock Service Worker integration errors

Wed, 15 Sep 2021 19:49:00 GMT

In a recent project, my team decided to use Mock Service Worker (MSW) in our unit tests. It was meant to be a very straightforward change since the documentation for MSW is pretty clear. I encountered a few errors along the way that had me scratching my head. I'm going to document them here, hopefully will help people in a similar situation.

This is what we had before the change:

// jest.config.js

module.exports = {
  preset: 'ts-jest',
  testEnvironment: 'jsdom',
  globals: {
    'ts-jest': {
      tsconfig: 'tsconfig.build.json',
    },
  },
};

Integrating with MSW

Following the official guide here, I created server.ts and jest.setup.ts files.

// src/mocks/server.ts

import { setupServer } from 'msw/node';
import { handlers } from './handlers';

// This configures a request mocking server with the given request handlers.
export const server = setupServer(...handlers);

// jest.setup.js

import { server } from './src/mocks/server';

beforeAll(() => {
  // Enable the mocking in tests.
  server.listen();
});

afterEach(() => {
  // Reset any runtime handlers tests may use.
  server.resetHandlers();
});

afterAll(() => {
  // Clean up once the tests are done.
  server.close();
});

Then I just had to reference the setup file in jest.config.js.

// jest.config.js
module.exports = {
  ...
  setupFilesAfterEnv: ['./jest.setup.ts'],
};

The handler is directly relevant to what is being explained here. But if you are interested, the handlers file looks something like this:

// src/mocks/handlers.ts

export const handlers = [
  rest.get('https://api.backend.dev/user', (req, res, ctx) => {
    return res(ctx.json({ firstName: 'John' }))
  }),
];

Issue 1 - can't find module 'typescript'

After the integration with MSW was done, I saw this error when I tried to run the jest tests. It is very strange because I changed nothing and it suddenly complains typescript doesn't exist.

Error: Cannot find module 'typescript'
Require stack:
- /Users/hao/node_modules/ts-jest/dist/config/config-set.js
- /Users/hao/node_modules/ts-jest/dist/ts-jest-transformer.js
- /Users/hao/node_modules/ts-jest/dist/index.js
- /Users/hao/demo-project/node_modules/jest-util/build/requireOrImportModule.js
- /Users/hao/demo-project/node_modules/jest-util/build/index.js
- /Users/hao/demo-project/node_modules/@jest/core/build/cli/index.js
- /Users/hao/demo-project/node_modules/@jest/core/build/jest.js
- /Users/hao/demo-project/node_modules/jest-cli/build/cli/index.js
- /Users/hao/demo-project/node_modules/jest-cli/bin/jest.js
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:902:15)
    at Function.Module._load (internal/modules/cjs/loader.js:746:27)
    at Module.require (internal/modules/cjs/loader.js:974:19)
    at require (internal/modules/cjs/helpers.js:92:18)
    at Object.<anonymous> (/Users/hao/node_modules/ts-jest/dist/config/config-set.js:58:20)
    at Module._compile (internal/modules/cjs/loader.js:1085:14)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1114:10)
    at Module.load (internal/modules/cjs/loader.js:950:32)
    at Function.Module._load (internal/modules/cjs/loader.js:790:14)
    at Module.require (internal/modules/cjs/loader.js:974:19)

error Command failed with exit code 1.

After some digging around, it appears this error is actually complaining about ts-jest is missing in the dependencies. See original stackoverflow question.

Issue 2 - non-standard JavaScript syntax

After installing ts-jest as a dev-dependency, I got yet another error.

● Test suite failed to run

    Jest encountered an unexpected token

    Jest failed to parse a file. This happens e.g. when your code or its dependencies use non-standard JavaScript syntax, or when Jest is not configured to support such syntax.

    Out of the box Jest supports Babel, which will be used to transform your files into valid JS based on your Babel configuration.

    By default "node_modules" folder is ignored by transformers.

    Here's what you can do:
     • If you are trying to use ECMAScript Modules, see https://jestjs.io/docs/ecmascript-modules for how to enable it.
     • To have some of your "node_modules" files transformed, you can specify a custom "transformIgnorePatterns" in your config.
     • If you need a custom transformation specify a "transform" option in your config.
     • If you simply want to mock your non-JS modules (e.g. binary assets) you can stub them out with the "moduleNameMapper" config option.

    You'll find more details and examples of these config options in the docs:
    https://jestjs.io/docs/configuration
    For information about custom transformations, see:
    https://jestjs.io/docs/code-transformation

    Details:

    /Users/hao/demo-project/jest.setup.js:1
    ({"Object.<anonymous>":function(module,exports,require,__dirname,__filename,jest){import { server } from './src/mocks/server';
                                                                                     ^^^^^^

    SyntaxError: Cannot use import statement outside a module

      at Runtime.createScriptFromCode (../../node_modules/jest-runtime/build/index.js:1517:14)

This time it is complaining that in jest.setup.js file above I used import. The fix for this was quite easy, I rename both jest files to .ts ( jest.setup.js → jest.setup.ts and jest.config.js → jest.config.ts).

DigitalOcean App Platform: how to redirect all requests to index.html for Single Page Applications

Mon, 09 Aug 2021 22:20:00 GMT

If you are not aware, DigitalOcean now offers their own App Platform which works like Netlify or Vercel if you used those platforms before. If not, essentially you could connect your code repository and then these platforms will take care of the rest e.g. install dependencies, building the project and deploying it somewhere, host it, manage SSL certificates, custom domains, etc.

Everything was quite straightforward, the dev experience is noticeably behind that of the competitors which is a little disappointing to see. The free tier is also more stingy in my opinion. But instead of restricting you to Lambda functions (serverless), you could deploy full blow Node.js (or other languages) backend applications. DigitalOcean charges for each "app" instead of by usage which could end up cheaper if you have a moderately popular application.

The problem

If you have used the competitor platforms and tried to deploy single page application, then you would know if a user tries to go to any path that isn't the root will result in a 404 error.

Why does this happen?

When a user tries to navigate directly (e.g. type into URL or refresh) to domain.com/some-url, the browser will request some-url.html from the server. However, Single Page Application (SPA) runs from index.html file, hence the name. Therefore it does not have the requested file and returns a 404 error.

How come the router library allows us to go to domain.com/some-url?

Routing libraries such as React Router manipulates the browser history to create the effect of multiple path names and pages are available.

The solution

Normally, if we were hosting the SPA ourselves then we'd have to configure Nginx to redirect all traffic to index.html. But what do we do with DigitalOcean?

DigitalOcean doesn't have very clear documentation around this, which could partly be due to the App Platform is still relatively new. I hope this will improve going forward, as this is a pretty fundamental config for any SPA project. After some digging around I found out this is normally done via submitting a config file, which is pretty well documented here. Fortunately, there's an even easier way to do that now. Navigate to your SPA application under Apps.

Then select the application you wish to make this change to.

Now go to the Settings tab, and scroll down to the bottom. There should be a section that says "Custom Pages". Edit it, input "index.html" and save. Now wait for the redeployment to happen and it should work when you navigate to any paths! 🥳

How to integrate Auth0 authenticated Frontend applications with API services

Tue, 03 Aug 2021 21:54:00 GMT

Following on from the previous blog post on How to setup Single Page Applications with Auth0 in the local environment without CORS issues, I noticed another area that is very vague in the official docs and thought I'd share my finding in this post. It turns out Auth0 provided everything we needed just didn't tell us how to use it.

This blog post assumes you have a working Frontend app already. If you are not quite there yet, be sure to check out How to setup Single Page Applications with Auth0 in local environment without CORS issues if you want some tips!

Auth0's user object

If you are not aware already, on the Frontend side it is possible to request a user profile from Auth0 and it will return something similar to the following. Assuming, of course, you've got the authentication working already.

{
    email: "hao@email.com"
    email_verified: true
    family_name: "Dev"
    given_name: "Hao"
    locale: "en-GB"
    name: "Hao Dev"
    nickname: "Nickname"
    picture: "https://image.com/fake-profile-pic.png"
    sub: "google-oauth2|1924810923701923"
    updated_at: "2021-07-29T17:40:14.223Z"
}

The important thing to note here is the sub property, this is the equivalent of something like user.id we typically use to refer to a unique user in our database. If we want to store custom data about this user or user inputted data then we would save them against the sub value and later we can retrieve it using the same thing.

Attach access token to API requests

According to Call Your API Using the Client Credentials Flow section of the doc, we need to attach the access token as a bearer token in all requests headers. Which makes sense as this will allow the Backend to authenticate and verify the user without blindly trusting what the Frontend says.

Here's the node example given in the above documentation:

const axios = require("axios").default;

const options = {
  method: 'GET',
  url: 'https://myapi.com/api',
  headers: {'content-type': 'application/json', authorization: 'Bearer ACCESS_TOKEN'}
};

axios.request(options).then(function (response) {
  console.log(response.data);
}).catch(function (error) {
  console.error(error);
});

Receiving access token on Backend

On the Backend, we will need to intercept all requests that require authentication. Luckily, Auth0 has made this part very easy to find (see screenshot below).

The code example given on the page is unique to each API, here is an example to illustrate to anyone who hasn't set up an API yet.

const express = require('express');
const app = express();
const jwt = require('express-jwt');
const jwks = require('jwks-rsa');

const port = process.env.PORT || 8080;

const jwtCheck = jwt({
      secret: jwks.expressJwtSecret({
          cache: true,
          rateLimit: true,
          jwksRequestsPerMinute: 5,
          jwksUri: 'https://example-api-dev.eu.auth0.com/.well-known/jwks.json'
    }),
    audience: 'https://an-example.com',
    issuer: 'https://example-api-dev.eu.auth0.com/',
    algorithms: ['RS256']
});

app.use(jwtCheck);

app.get('/authorized', function (req, res) {
    res.send('Secured Resource');
});

app.listen(port);

In this example, it shows us that we need to use two libraries here express-jwt and jwks-rsa to verify the access/bearer token passed from the Frontend's side. Then using jwtCheck() function as a middleware, the request will only reach app.get('/authorized') if the JWT token passes the authentication check.

Now we have used the access token that came as part of the request and verified it with Auth0 and made sure it is valid and haven't been tampered with.

Getting unique identifier on Backend

Apparently, jwtCheck verification automatically populates req.user if everything is successful. After checking out express-jwt documentation, it seems that it is doing all that work for us.

It is very confusing because that isn't made clear anyway in the Auth0 docs and I went through a lot of them. I was lost and confused until I saw this line in one of the example codebases which helped me to figure it out.

app.use(jwtCheck);

app.get('/authorized', function (req, res) {
        // if we can reach here, then req.user is already populated
        console.log(user.req);

    res.send('Secured Resource');
});

The console log will print out the same user object shown on the Frontend side.

{
    email: "hao@email.com"
    email_verified: true
    family_name: "Dev"
    given_name: "Hao"
    locale: "en-GB"
    name: "Hao Dev"
    nickname: "Nickname"
    picture: "https://image.com/fake-profile-pic.png"
    sub: "google-oauth2|1924810923701923"
    updated_at: "2021-07-29T17:40:14.223Z"
}

Keeping some endpoints open

Is it possible to not use app.use(jwtCheck); and lock down all of the endpoints? Yes, as far as I am aware there are a couple of ways to achieve this.

Authenticate each endpoint/route individually
Define app.use(jwtCheck); after the endpoints that don't require authentication

Although in many of Auth0's examples I read that it is possible to define JWT verification on each endpoint, I never actually got the first method working myself. The following snippet is supposed to work.

// ❌ didn't work for me

app.get('/authorized', jwtCheck, function (req, res) {
    res.send('Secured Resource');
});

app.get('/unauthorized', function (req, res) {
    res.send('Public Resource');
});

app.get('/another-authorized', jwtCheck, function (req, res) {
    res.send('Another Secured Resource');
});

In the end, I went with the second approach, which works for me with no problem. It is unfortunate since the first solution would have been a lot neater.

// ✅ worked for me

app.get('/unauthorized', function (req, res) {
    res.send('Public Resource');
});

app.use(jwtCheck);

app.get('/authorized', function (req, res) {
    res.send('Secured Resource');
});

Final words

I hope this information was useful, I was stuck on this for a long time because I couldn't find any information on it.

PS: If you want to know more about authorisation checks (e.g. use Auth0's permission system) check out Server + API: Auth0 Configuration and Validate Access Tokens. Although they don't give you the code you need, it is a good place to start.

How to setup Single Page Applications with Auth0 in local environment without CORS issues

Sat, 31 Jul 2021 14:52:00 GMT

I used Auth0 in a Single Page Application (SPA) side project recently, because I didn't want to spend the time to build all the auth related features e.g. password reset, sign up, etc. Surprisingly it was extremely painful to get it working locally due to HTTPS, CORS and localhost issues. In this blog post, I'll be walking my final setup with all of that working locally.

You might also want to check out How to integrate Auth0 authenticated Frontend applications with API services after are done setting up the Frontend app.

Simple app architecture

I am going for simple and very standard app architecture in my project. I wanted to separate my frontend from the backend because I didn't want Server Sider Rendering (SSR). When the user visits my web app, I want the frontend to call the backend for any data.

In this case, I'm using React (create-react-app) on the frontend and Node express server on the backend.

Simple app with Auth0

Here's an updated diagram with Auth0 added to the picture. Now the flow is a little more complex. When someone visits my web app, I now want to redirect them to Auth0 for sign up or sign in before returning to my app to continue with the user journey (steps 1, 2 and 3 in the diagram).

When the user wants to edit any app data, the frontend will attach the bearer token from Auth0 and send it with the request to the backend (steps 4 - 5). The backend will then verify the user is who they claim they are by calling Auth0 with that bearer token, then return the data to the frontend if all is good (steps 6 - 8). Finally, the front end will display the updated UI on the browser (step 9).

This flow is more complex compared to what we started with, but it is still pretty standard. Any sort of auth service or Single Sign On (SSO) will involve a similar process. Now on to the more complex part - local setup.

Auth0 in the local environment

While Auth0 has dedicated a Test Applications Locally section in their website that focuses on local app testing, unfortunately, it is pretty basic as far as documentation goes. It is tough to pinpoint which type of app architecture they are referring to. As far as I understand this section seems to be for SSR apps, not SPA. There's also a section on HTTPS in Development (I'll explain why we need HTTPS later), which I'm certain isn't for the architecture in our diagram above.

So long story short, I didn't find any good guidance on Auth0's website so I had to forge my own.

We need HTTPS everywhere

We cannot authenticate with Auth0 if our frontend is running in HTTP mode. If you do, then you'll get the following error message.

Error: auth0-spa-js must run on a secure origin.
See https://github.com/auth0/auth0-spa-js/blob/master/FAQ.md#why-do-i-get-auth0-spa-js-must-run-on-a-secure-origin for more information.

Luckily, it is pretty easy to get around this if you are using create react app, it is as simple as HTTPS=true to your start command e.g. HTTPS=true npm start. See more about it here.

Doing this fixes the issue with Auth0, but now Chrome will complain that our HTTPS frontend is now sending non-secure requests (HTTP) to the backend. So next we will need to run the backend in HTTPS mode. To do this with Express, we will need to generate some certificates, I recommend using mkcert. This will output two files a key and a certificate, which we will both need to run the Express server in HTTPS mode. You will then do something like the following.

import fs from "fs";
import https from "https";

...

const privateKey = fs.readFileSync("../../key.pem", "utf8");
const certificate = fs.readFileSync("../../cert.pem", "utf8");
const credentials = { key: privateKey, cert: certificate };

...

const httpsServer = https.createServer(credentials, app);

httpsServer.listen(process.env.PORT || "8000");

Well, now we have these certificates we might as well use them on the frontend. Create react app to the rescue again! All we need to do is include the following in our .env file and react-scripts will pick it up during development mode automatically.

SSL_CRT_FILE=../../cert.pem
SSL_KEY_FILE=../../key.pem

We can't be on localhost

To do step 5 in the second diagram, we need to use getAccessTokenSilently() (again I'm using React, your mileage may vary) to extract the bearer token we need to pass to our backend. This, however, will trigger a consent check (even though this is a first-party application!) and if we are running on [localhost](http://localhost) it will fail silently. You can read more about it on the User Consent and Third-Party Applications page.

If you are on mac or linux then edit /etc/host, if you are windows it is C:\Windows\System32\Drivers\etc\hosts. You could replace [localhost](http://localhost) with [example.site](http://example.site) but I prefer to keep both since I work on codebases that assume localhost as well.

127.0.0.1  localhost example.site

CORS, CORS, CORS

CORS on the local environment is so frustrating. If you follow the instructions above, then hopefully in your Express server you can just add the following and everything should work.

import cors from "cors";

...

app.use(cors());

Alternatively, consider adding some whitelist logic.

import cors from "cors";

...

const whitelist = process.env.ORIGIN_WHITELIST_URL || "https://production.site";
const corsOptions = {
  origin: function (origin: any, callback: any) {
    if (whitelist.includes(origin) || !origin) {
      callback(null, true);
    } else {
      callback(new Error("Not allowed by CORS"));
    }
  },
};

...

app.use(cors(corsOptions));

If it still doesn't work, try allowing any origin. Of course, this isn't secure, so come back to this later and bulletproof it for production.

import cors from "cors";

...

const corsOptions = {
  origin: '*',
};

...

app.use(cors(corsOptions));

...

Alternative service to auth0

If you are thinking of using Auth0, I came across another service called Magic which seems to offer a much better dev experience as a whole. Their local environment setup seems too good to be true, check it out here. However, the reason I went with Auth0 at the end is because of their generous free tier which allows up to 1000 active users. Magic in comparison charges based on the number of user logins and there is no free allowance.

State of Yarn 2 (Berry) in 2021

Tue, 20 Jul 2021 20:20:00 GMT

Yarn 2, aka Berry, was announced back in April 2019, it was a pretty big deal at the time. I've moved 2 clients since and still, no one is talking about it. I got curious recently and did some research about this topic. In this post, we'll see what is the deal with Yarn 2, and what the adoption is like after 2 years.

What was the biggest change in Yarn 2?

The creator and maintainers of Yarn identified an issue we have in the JavaScript community and found a brilliant solution to address it. The solution is called Plug'n'Play (PnP). Surprisingly, the PnP feature exists in Yarn 1. The feature was unveiled back in September 2018, almost a year before Yarn 2's announcement. However, it was turned off by default due to incompatibility issues with well known NPM modules. As you can imagine almost no one used this feature for this reason. Then when the Yarn team released Yarn 2, PnP was on by default and can't be turned off unless you use some highly experimental workaround. I guess they were hoping this will push up the adoption rate.

But why did the maintainers want everyone to use PnP so bad?

Yarn team discovered that the current way of maintaining JavaScript dependencies is extremely inefficient and there is a lot of room for improvement. For example, 70% of the Yarn install time is used to generate node_modules. Sadly, without drastic changes, there is no way for package managers to optimise anything. This means the problems are not going to get better over time without intervention.

If you are interested here are a few other ways why the node_modules design is wasteful and impractical see Yarn: The Node Module Problem.

So, how does PnP fix these issues?

PnP generates a .pnp.cjs file that points to packages in local disk location. Instead of Node looking for a package in a folder, Yarn will tell Node where to find the packages instead. Since now Yarn only needs to generate a single file instead of creating a node_modules folder of modules during install. It is much faster and has way fewer I/O operations, meaning a massive performance boost!

By committing .npn.cjs file it is even possible to avoid running yarn install (hence the name Plug'n'Play). Node apps will also launch much faster without Node iterating over files to find what it needs. Just remember back when Windows XP supported USB devices without the user installing any drivers, this was meant to be a game-changer.

If Yarn 2's PnP feature is so awesome then why so much resistance?

As mentioned above the main issue is around the breaking changes and requires existing packages to be updated by their respective maintainers to be compatible with PnP. This caused further discussions around if Yarn 2 should be released as a separate package manager altogether under a different name e.g. Berry. It is obvious now looking back, the suggestion wasn't adopted.

In the early stages, people who tried Yarn 2 reported a lot of errors when they tried to upgrade. A lot of tools and libraries people rely on stopped working or experienced issues after the upgrade. Yarn 2 offers workarounds such as adding pnpMode: loose into .yarnrc.yml to disable PnP. But even this was a highly experimental feature at the time.

Unfortunately, not everyone is on board with this change. Facebook and Twitter both came out and said they plan on staying with Yarn 1. There must be a lot of other smaller companies which also didn't want to spend ages upgrade to Yarn 2 and risk their product stability and product roadmap.

Summary

The Yarn maintainers wanted to push the brilliant PnP feature, which has been in Yarn 1 for ages, for wider adoption. To achieve that goal, they wanted to force developers to start building and using PnP (reminds me of how Apple forces people to move to newer I/O ports).

The fact it breaks compatibility with a lot of projects caused concerns within the community. Well-known names in the JavaScript community opened stated on Twitter that they or their team wouldn't be upgrading to Yarn 2. This probably further hindered the adoption rate. But the root cause is the lack of easy migration between the two versions since a lot of the issues lies with the NPM package maintainers who needs to spend time supporting Plug'n'Play. People just didn't want to run the risk of upgrading and having that one library not working for them.

If you are familiar with Python 2 and 3 drama over the past ten plus years then you'll find the story pretty similar here. Python 3 breaking changes were introduced without real ways for developers to bridge the gap. So for years, Python devs struggled with the two major versions. Unable to move to the new version completed and unable to use newer libraries that only support v3. From my own experience, it seems like that's been getting better.

State of Yarn 2 in 2021

In all the companies I've worked at over the past 3 years, only one company looked into adopting Yarn 2. As far as I'm aware, other than a presentation showcasing its benefits, that didn't go anywhere. I almost can't find any discussions online regarding Yarn 2. I don't think Yarn 2 is dead, but compared to Yarn 1 the adoption rate is much slower.

Should you use Docker for local development?

Mon, 12 Jul 2021 10:52:00 GMT

I've worked for many companies over the years since I'm in the field of consulting/contracting. It allowed me to see how teams do things differently and the pros/cons of each approach. In this blog, I want to share some of my observations on using Docker for local development.

Firstly, I just want to put it out there that I think Docker is great and helps to make deployments much more reliable by providing a consistent environment to build and run apps. The ability to use the same environment in CI and Production is amazing, it helps to catch issues that otherwise are missed.

So do I think teams should use Docker for local development?

Unfortunately, no, I don't believe most teams should be using Docker for development and they probably should move away from it as soon as possible.

1. Slow performance

When I used Docker to run an application, I never thought "oh yeah, this makes the app run much faster". Nope, it is always the opposite - a lot slower! Running a decent sized application inside docker will be noticeably slower since docker containers have limited resources than the Operating System on your machine. I don't doubt it is nice for someone else to write all the config/script to spin up everything, while you just kick back and relax as one command bring everything online.

As a developer, I think we all understand the benefits and drawbacks of too much presumption. What makes your life easier also hides aways some customisation options that are useful tools in our toolkit. As a software engineer, I believe the more important thing here is to be intentional. Using something for the sake of using it isn't a good reason, neither is using Docker when it isn't required. Why trade something like performance, slower feedback loop and increase complexity for the percieved benefit of Docker making things "easier"?

2. Additional friction and another point of failure

I find any codebase that uses Docker a lot less intuitive to see what's going on. It is the same as calling custom script as part of npm start or yarn start. It is another step in the process, and this step is abstracted away for better or for worse. For sure do it if your codebase is unique in that way and requires tens of lines of node scripts to set up and run (in case of custom scripts). But most of the time that just means something isn't right about the codebase. I'd look to solve the root problem instead of bolt-on something that just fixes the symptoms. Similarly, I think it's the same with using Docker for local development. Most of the time, the repo or app architecture shouldn't be that complex where Docker is necessary.

Docker normally stays out of the way, but like everything else, it also requires maintenance and sometimes causes problems too. I can recall times where I just wanted to run an app quickly to check something but instead had to go out of my and spend time on debugging docker issues. Maybe someone changed the Dockerfile and you just had to pull changes and rebuild the docker image. Or maybe it's something more complex, now the question becomes how far down the rabbit hole are you willing to go?

The fact all the Dockerfile used to spin up a basic node app is never the same is almost comical, somehow we all seem to want to use different base images, different library installations, different mount directories, etc than everyone else. The lack of consistency and standard, makes reading each Dockerfile is just as hard as the next one. Of course I am just as guilty and as anyone when it comes to this. But I'd like to argue we are never set up for success. How often do you find Dockerfile guideline documentation in a company? I don't even believe there is a consistent way agreed by the wider community.

Then there is the release process for Docker images, some teams have dedicated CI whilst others just let you release to Dockerhub (assuming you don't need to chase people for permission). I've yet to see this managed in a way that screamed consistency or good practice. All this is often decided by one person, often new to the company and want to set a "standard way of working" that ends up introduces more disagreements. Normally DevOps teams create the initial Dockerfiles, then they don't have the time to maintain everything and teams ignore the Dockerfiles until something breaks.

I'm a big fan of simple is better or KISS for those who like acronyms. I just think sometimes people having local issues isn't the way to introduce a solution that makes everyone's life a bit harder. Many of these issues can be resolved by being better at pairing and writing better docs.

3. Difficult to debug

Dockerfiles are often written to be used in the Production environment, so debugging is turned off by default. Maybe we can change the Dockerfile, build it locally then use that image to run the app with debugging turned on? Doesn't that sound massively exhausting? But yeah, I admit that's a solution. How about in situations where the repo doesn't include the Dockerfile it needs, instead it pulls Production docker images to run the app locally. It turns out if you want to modify the docker image, you need to go on a treasure hunt and find it in another repo. I've even seen developers host company docker images' code on their own Github account. Maybe these are extreme examples, but they all happened to me.

My point is sometimes we need to run the app slightly differently to normal production build, especially when debugging issues or trying to narrow down a problem. It isn't the best time to be messing around with Dockerfiles and docker images.

But how can I easily spin up the whole app without docker?

The right answer is we don't. Let's skip the argument of why monolith is bad, I assume most people agree with the fact that it should be avoided at all cost. If you are working on a tiny slice of the codebase, then the ideal scenario is to only spin up that slice (in a micro-frontend setup), run that slice locally and proxy everything else to your dev or staging environment. Since we don't care about the rest of the application in this scenario. If there is a concern that changing this slice might break another app, then perhaps there should be more integration in CI, have more decoupled apps or more contract testing between apps.

If the feature requires working on multiple apps to deliver. I'd argue each app should be worked on one at a time, changes deployed to Production under a feature flag or released in a way that it doesn't introduce a breaking change. Why? Assuming each app has separate CI/CD (if it doesn't then it should, it's 2021, there are no good reasons to group apps and do big bang releases), if there are inter-dependencies how can you guarantee the release to Production doesn't cause a problem if one app got released before another. Implement backwards compatible change and then deprecate later is a much better approach for situations like this. Unfortunately, many teams are either not working in codebases that allow them to do this, or the company culture is to do edit multiple apps and release in a big bang fashion.

Final words

I think the goal of local development is never to code in the Production runtime environment. We already have CI and CD pipeline to detect and spot any errors caused by the difference in setup or environment configuration. Not to mention Staging or QA environments dedicated to catching any issues that are otherwise missed by automated tests. Although it sounds useful, I'm not convinced we should run applications through Docker locally.

As developers, we need and are entitled to fast feedback cycles for either TDD or to write iterative code. It is an incredibly stressful feeling to be constantly taken out of the zone because the computer becomes the bottleneck and a liability. All the companies I work for say they want their developer to be happy. They understand happy developers are more productive. In my opinion, using Docker for local development isn't the path that leads to developer happiness.

Just a short disclaimer, I'm not saying don't ever use Docker for local development. If there are serious benefits and everything has been weighted and Docker came out on top, then sure go for it! I just believe it shouldn't be used unless there are really good reasons, and definitely shouldn't be the default option of running applications locally.

Interview notes on React Reconciliation (Virtual DOM)

Thu, 20 May 2021 14:36:00 GMT

It's been a while since I had to recall the reasons why React uses Virtual DOM. I was recently asked this in an interview, and I did an okay job explain it but not to the extent I wanted. So I spent some time re-reading the official documentation and wrote down some notes. I hope these will be helpful for anyone preparing for a React interview. Also, I have called out the vital part of each section to make it easier to see the key points.

What is it?

Virtual DOM is the virtual representation of all the elements on the page in Javascript.

Virtual DOM, like the name suggests, is a virtual JavaScript representation of the actual page DOM. All the elements which exist on the page also exist in this virtual representation. When React code detects a change, it will perform a differentiation (AKA diff) by comparing the before and after virtual DOM objects. The diff algorithm will compute a list of DOM manipulations that it deems necessary to bring the actual DOM in line with the virtual DOM. As a result, the user will see the page changing when they interact with it (or changing in the background automatically if that's the intended behaviour like a stock ticker site).

Why virtual DOM?

React uses Virtual DOM because it makes page updates more performant when there are many elements on the page. Since it could figure out which nodes need to be updated, then go and do that.

The virtual DOM makes it easier to handle changes on the page, displaying a basic page layout or handling complex DOM changes. It only re-renders the part of the DOM that changed (we'll cover how in the Diff Algorithm section below) whilst leaving the rest intact. Therefore it ensures page element update speed even when there are tens of thousands of elements on the page since only a tiny subset of the total number of elements in the DOM requires updating.

Virtual DOM allows developers to focus on what needs to happen rather than how it happens. By handling DOM changes for the developer automatically, it makes writing web applications much faster.

Virtual DOM isn't a silver bullet

Like many solutions, React Virtual DOM shines in some scenarios, specifically complex DOM tree structures that are deep and wide.

It's worth noting that React's virtual DOM only has an edge in performance if the DOM tree structure is both deep and wide. Virtual DOM isn't faster 100% of the time.

In small applications written in vanilla, JavaScript could find an element with getElementById and update it with much less time and resource. However, the vanilla way is an old school way of writing web applications, mainly because it isn't scalable in larger projects. For one, it is not a realistic expectation that developers will assign every DOM element an id when working in large applications. If we don't do this, then finding a part becomes more computationally expensive. Not to mention the development overhead with assigning and recalling id for each element on the page.

Angular also abstracted away from the complications of manual DOM updates but doesn't utilise Virtual DOM. In comparison, in smaller and more straightforward applications, React's Virtual DOM wouldn't contribute to much performance gain and might negatively impact it. Instead, it shines in more significant codebases.

Tradeoffs

The React team highlighted two main problems:

Need to ensure element types (e.g. div to a span ) only changes when necessary since React will blindly reconstruct all the nodes under that element.

Altering a component's type might be very costly, especially if they have identical nested nodes since React virtual DOM will reconstruct all the nodes under the changed element.

Must provide keys to all the list item to ensure the O(n) performance.

Using non-unique and reliable keys could result in poor performance where the O(n) performance isn't guaranteed. See in the next section on why this is the case.

Diff Algorithm

Imagine the virtual DOM as a tree structure. The algorithm will start from the root node then traverse through the whole tree. React checks for two things to see if an element has changed: Is the element type different? If the type has changed, then tear down and reconstructive all of the nodes under this element. If nothing has changed, then continue. If the element properties/attributes changed? When a component updates, the instances remain the same so that the state is maintained across renders.

Update new props on the underlying component instance, and call render() to regenerate the sub-tree

FAQ

Why are keys used in React lists?

Keys can reduce recursive elements lists to only require a single parsing O(n) to update the whole tree. Whereas without using keys, the diff and update process is more computationally expensive O(n^3).

By default, React will assign an item's list index as the key. However, this only partially solves the problem, it works when the list remains the same, and just one element's properties are updated. When elements get added/removed, their indexes change to end up with a similar issue.

<ul>
  <li>Duke</li>
  <li>Villanova</li>
</ul>

<ul>
  <li>Connecticut</li>
  <li>Duke</li>
  <li>Villanova</li>
</ul>

React asks for a manual key assigned to lists items to reduce the unnecessary computation of the indexes on the page when it changes.

My struggles with React Testing Library

Wed, 12 May 2021 19:35:00 GMT

Open any software development book, and there is probably a section on testing and why it is essential. Testing is a great feedback tool. If you think about it, it is incredible how we can write code and then write other code to check the initial bit of code. I can't think of many professions out there who can automatically validate and verify their work like us software engineers.

Testing JavaScript functions with tools like Jest is easy. Testing React or other rendering libraries/frameworks is a different beast. React Testing Library (RTL) overtook Enzyme in popularity a few years ago and became the "go-to tool" for testing React apps. I'll be covering some of the issues with RTL later, but I admit it is a massive improvement over the native test kit offered by React team.

In this post, we'll be mainly talking about React variant of the Testing Library. But it's worth pointing out that Testing Library doesn't just support React. It also supports other libraries/frameworks too (even Cypress).

Integration testing or unit testing?

To clarify, my use of the phrase "integration test" here means Cypress test that tests the UI in the browser with mocked responses. When I talk about "unit test", I'm referring to the sort of test you'd write to test a pure JavaScript function.

I struggle with RTL because I can never determine if I'm writing a unit test or integration test. When I write tests with RTL, I'm telling myself these are unit tests. But it always feels like I'm writing integration tests. Things are rendered in a virtual DOM; we then inspect the DOM to find the elements we need. Isn't this basically Cypress?

So naturally, teams start by writing RTL tests for smaller components. Then they write the same sort of test for a higher-level component, which wraps multiple nested components. Then there are page-level or complex components that make API requests or contains side effects. These tests will often render even more nested components in the virtual DOM, mixed with data fetching, side-effects that cause re-rendering.

You might say: "No, that's not right, we should be testing basic components and leave page-level test to E2E tools like Cypress".

Well, just about every team (different companies) I joined in the past 3+ years all decided Cypress tests are too slow, CI pipeline end up taking too long, so let's migrate them to Jest and write them in RTL. I was in those teams, I disagreed with the decision, but I don't blame them. It seems like the right thing to do. Especially when Cypress tests running in parallel were taking over 40 mins to complete.

So should we call the simpler component test "unit tests" and the more complex component "integration tests"? Although this distinction is vague, how could we get all the developers to agree on them? One of the team I worked in even added "integration" into the file name to help to distinguish. Those are the tests that no one wanted to touch since they were slow to run and hard to understand.

Maybe we are doing it all wrong, and we shouldn't be writing integration tests in RTL.

Actually, according to Kent C. Dodds, the creator of RTL, we really should. He challenges the traditional testing pyramid model and promotes the testing trophy in the following tweet.

"The Testing Trophy" 🏆

A general guide for the **return on investment** 🤑 of the different forms of testing with regards to testing JavaScript applications.

- End to end w/ @Cypress_io ⚫️
- Integration & Unit w/ @fbjest 🃏
- Static w/ @flowtype 𝙁 and @geteslint ⬣ pic.twitter.com/kPBC6yVxSA
— Kent C. Dodds (@kentcdodds) February 6, 2018

So we should be writing integration and unit tests in the same format, layout, and with the same tools? Let me know what you think in the comment section. I'm still very undecided about this 😆

Stop mocking children

Another problem with writing tests in RTL for complex components is the amount of mocking required. RTL tests render the tested component in the virtual DOM. Therefore we need to provide all the props expected by the component itself or any nested components. Otherwise, it will error, and your test is likely to fail. Mocking is manageable with smaller or simpler components that either doesn't have much data passed to it or are relatively flat in the tree structure.

I've worked on very data-heavy apps. Complex components often required mocked responses around a few hundred (extreme ones are thousands) lines of JSON. So the standard practice is to copy the real response from the browser Network tab and paste it into a "mock data file". It is probably possible to trim it down a little, but that will be more time-consuming. I've also seen and tried to write test helper functions that generate mocked data, but these often would grow in complexity to a point it is even less maintainable than the static data copied from the browser.

Maybe these are just extreme cases? Not all complex components consume that much data.

Rendering large or complex components in DOM is slow

If mocking isn't a problem you encounter every day, then perhaps slow tests annoy you enough to rethink this.

Complex components tend to have side-effects, e.g. request data with useEffect, when external data comes back, update component state with useState, then re-render to display that data.

One of these components can take up to a few hundred milliseconds to render. Now imagine a modest number of 20 test cases in a test suite; this already adds up to a few seconds. Let's face it, 20 test cases per page level component is a very conservative estimation. If you prefer Test-Driven Development (TDD) or simply want to modify a few tests, it can be painfully slow to wait for the tests to run each time you change anything. I often deal with individual test suites that take 20 - 60 seconds to run (I will cover why in the next section).

I have been utilising Jest's .skip and .only to limit the number the test cases I run when writing tests. I also only run Jest on one file at a time. Otherwise, waiting around for all the test to finish would drive me mad. And then there are the async tests.

Await, waitFor, what are we actually waiting for?

RTL supports async tests. As we mentioned already, it is effectively doing what Cypress is doing, the only difference is that it does it in the virtual DOM. So when the page re-renders after a user interaction or data request, it needs to wait for the page/component to finish re-rendering.

In my experience, when you are testing complex components, you may as well assume something needs waiting on. That's right, start all of the test cases assuming it is async unless proven otherwise.

I've battled with await and waitFor() (RTL's built-in API for waiting for stuff to happen) a lot recently. A few months ago, we increased the timeout to 10 times the default value because unit tests were too flaky. Yeah, those flaky integrations or E2E tests are in our unit tests now. Luckily, after increasing the timeout to 10s, tests seem to be more stable now. But it also means when a test fails, Jest will also wait for 10s before marking it as a failing test. Now we have an interesting scenario if the test takes too long to finish, then probably it is because there are failing tests.

RTL tests are hard to write

It shouldn't shock anyone that Cypress tests are laborious and slow to write because it is annoying to write the different selectors for elements on the page. Now, let's try to do that without seeing and inspecting the page at all. Because there is no way to do that in RTL, the best you can do is use RTL's debug(), which prints out content in the virtual DOM. If you want to be fancy, you can first select a sub-component that contains the element you need, and we can call it wrapperComponent. Then we can do debug(wrapperComponent), which will only print out the selected element and its nested elements.

Recently, I was so stuck with an async RTL test that keeps complaining about how it cannot find the element I'm looking at, but I can see it when I print it out with debug(). I went as far as installing jest-image-snapshot to see what RTL is rendering. I think this is enough evidence to justify why writing an RTL test for complex components is challenging.

RTL tests are harder to maintain

If you think writing RTL tests are complicated, now imagine opening up an RTL test suite someone else wrote and try to fix broken tests because you changed a small thing in a nested component. Hopefully, Jest's error message is helpful enough to spot the issue. Otherwise, there is no other way other than to use RTL debug() to check what is going on. Let's take it a step further, and say you made an app-wide change. Now several complex component test suites are failing. Well then, have fun!

Cost Driven Development

Thu, 04 Mar 2021 19:48:00 GMT

Are you a fellow software engineer who had great product or business ideas then abandoned them? Have you been put off the idea due to hosting and running costs? I think we can all agree that it would be a shame if a million pounds idea got buried due to early-stage costs. At the same time, we need to do our best to navigate the money-grabbing realities of SaSS and platforms out there. Sure, a few pounds for a domain here and a few pounds to spin up a server there might not sound much on their own. However, when combined, they could quickly deplete our savings, which is not what we want. Continue reading, and I'll be sharing some of my thoughts and tips around how to cheaply or even create side projects for free.

Developing software is cheap when it only requires our time

As developers, we are the perfect group of people to experiment and trial digital business/ideas with extremely little commitment, which essentially boils down to our time. Countless open source libraries are at our disposal to speed up prototyping and coming up with an MVP from our idea. Maybe one of these projects will become the opportunity for you to quit the day job, make a lot of money, take control, or whatever your dream may be. We should understand how lucky we are; business people with ideas who lack technical skills can only dream of our situation.

Of course, building any product can be complex and time-consuming. If we want to make a quality product loved by users, it is even more challenging and demanding. However, this doesn't change the fact we don't need capital to create Proof Of Concepts (POC) to try out business or product ideas. As we all know, hiring us developers is expensive.

Now, we've established the fact being software developers give us a massive edge over everyone else. What stands between us and the idea we have is simply time. So, what are the other costs we need to consider?

Hosting an application is expensive

Since it is so cheap for us to work on side projects, I tend to work on poorly considered projects. Despite this, it is still an excellent exercise because I'm learning in the process, and I can showcase these projects to people. I'm a big fan of the concept "succeed even when you fail", whereby mixing in some learning, you are guaranteed to walk away with a smile even when the business or product idea does work out. For example, creating a side project using Svelte may not reach the Facebook level of success. Still, by the end, the enhanced Svelte skill might be excellent if a Svelte job opportunity presents itself.

These "low cost" personal projects often lead to a large number of apps created. Naturally, we'd want to share them with the world considering we put in time and energy. However, hosting these applications often comes with some costs. It could be as essential as domain names, Continuous Integration (CI) pipeline, etc. I would categories these costs in two ways: 1) baseline fees, these are the charges that you cannot avoid, e.g. minimum requirement to ensure the application is running, 2) usage fees, cost that increase with usage.

The first cost category is easy to predict but hard to avoid. The good news is that these costs tend to be on the cheaper side. That is as long as you are flexible, e.g. you can probably pick a domain name that's as cheap as a few pounds or more than thousands. For an unproven side project, I'd personally lean on the cheaper side and get a domain that's "good enough".

The second category is harder to predict, but often there'll be a free-tier. Unless you are getting some decent traffic or load to the application, it is unlikely to cost too much. Just be careful because these tiers are there to draw you in, so it doesn't take much for the cost to rack up, e.g. someone might decide to crawl your site a few times or DDoS it for fun will send your bill rocketing. On the other hand, if the product is attracting legitimate users, which increases cost, this could be problematic if there isn't a business model to generate revenue.

This is where Cost-saving Driven Development comes in

What do I mean by Cost-saving Driven Development? As the name suggests, we are putting cost as the absolute centre when designing and writing software. In my opinion, projects that aren't sustainable on their own isn't worth having. Thus minimising cost shouldn't be one of the goals; it is the only goal. Is a portfolio of applications that costs hundreds of pounds a month to run worth having around, on the odd chances some mystical future employer stumbles on to it? I would argue a Github portfolio might be better in this case. Take some screenshots and videos to demo your work if the codebase by itself is too boring on its own.

It is too easy to create a technical design and solution to a convoluted and expensive problem in today's environment. There are so many SaSS products out there waiting to grab hold of our wallets, failing that there are the big boys in town. Yes, I'm looking at you AWS, Google Cloud, etc. Did you know that Amazon gets 70% of its profit from AWS alone (Where the money is really made at Amazon)? Guess where the money came from? That's right, businesses and individuals like us.

Unlike a profitable business, as individuals, we don't have lots of money to splash around. Personally, I have very little incentive to throw money at several projects that I don't know will succeed or not. I'd probably be better off to invest that money in the stock market. Therefore, the cost is always the first thing I think about when I work on a side project.

I have dedicated a whole section below on a list of my favourite services and platforms for creating side projects. But here are some general tips that might lead to some cost savings:

Instead of buying a domain for each product, use one domain name for all of them. E.g. I could use *.hao.dev to host different apps.
Consider bring in some revenue to support more popular projects. Get donation, sponsorship, advertising, or any other way you can think of and bring in some money to pay for the bills.
Researching all the tools and services can be tedious and time-consuming. Reading comparison articles can be a fast and effective way to get a rough idea of the best options out there.
Search around for promotions, free credits or sign up discount codes to maximise your allowances or benefits. Services and platforms often use these techniques to bring in new customers.

Another take on Cost-saving Driven Development from a professional angle

As software engineers, costs aren't something we consider in our jobs until the middle management level. Therefore, it is understandable for developers not to think about cost and cost-saving. I find there are always many options to achieve the same goal using various techniques. For example, I implemented search functionality on this blog utilising a SaSS service called algolia, which focuses on searching instead of building a search API myself. I could have implemented a serverless API that performs searches and returns relevant results. Taking a step further, there are even ways to generate a JSON file during the build stage, which then can be used as a "fake" search API (see https://github.com/sw-yx/netlify-plugin-search-index). All of these approaches have their benefits and drawbacks. We should always consider them all, then made an informed decision.

I once heard a client saying that their company AWS bill was £2 million a month. Why? Unused servers and other cloud resources are significant contributors. Allocating too much resource is a common problem as well. In my opinion, costs should be at the centre of cloud infrastructure. Each pound spent should be intentional and be reviewed after a while to see if it still makes sense. The more mature engineering teams have reporting setup to see where the costs are coming from, but too often, cost is a byproduct and not a constraint.

One of the reasons why companies and business have a massive "tech" bill is because people often blindly following approaches defined as the "common approach" or "best practices" by companies or industry experts. In these situations, teams would disassociate cost from the technical implementation details altogether. If option A saves the company thousands of pounds and option B is slightly more brutal to work with but free, shouldn't we consider both options before jumping on the first one and calling it a day?

As engineers, we are constantly told it is better to over-engineer and cater for possible unexpected developments later. Based on my personal experience, I'm not sure this is true. Over-engineering is expensive in terms of development time. Additional unknown factors and the extra architecture setup are reflected in the cost to accommodate potential future changes.

Part of me thinks it is unfair that we make fun of business people who always talk about costs, want things done yesterday and has no regard for software quality. Aren't we also a little hypocritical to value code quality over time and never consider the cost side?

I think many people jump on the latest trend without really considering the actual benefits for the company or business, whilst others refuse to move on from the tech stack they are familiar with. It isn't good to be in either of the two extremes. Instead, we should all aim to be somewhere in the middle.

I think putting cost at the centre will allow people to have more concrete debates about technical implementations instead of opinion battles. In dev chats, people often argue over different approaches and require a higher power to intervene when they reach a stalemate. For example, someone might say, "We should go with SSR because it performs better on low-end devices". Someone else will argue, "CSR can be optimised to work just as well". By considering cost, the conversation might look like this, "SSR will cost £x and add additional complexity to our approach, whereas CSR cost 1/10 that much but may need longer development time to optimise the performance on low-end devices". Placing a limitation such as cost will facilitate discussion and reduce the scope of possibilities makes sure people are no longer arguing about ideologies and talking about practical solutions.

Introducing limitation doesn't remove our ability to innovate. Rather, it pushes us to find ways around those constraints. There's a quote I love from the book How to Take Smart Notes: "the limitation of canvas does not reduce the artistic expression of artists".

Here are the best cheap or free services I'm using

The services I use are either free or has a basic free-tier. I have checked and made sure the allowances given as part of the free-tier is more than enough for side projects. If you get close to them, then maybe it is worth considering monetising these projects to cover the extra costs?

My workflow is straightforward, and I usually start working on a project idea in CodeSandbox. It is as easy as opening up my browser on any machine, and then I could start coding away. The path of least friction is critical for me at this stage of the process, where the most important thing is to visualise the idea. It is possible to create web apps like this, and CodeSandbox will host it for free. My advice is to move into local development as more files and computation will cause a slowdown.

Once the project has a few pages, this is when I'd migrate the code into a Github repository. Since they made private repository free a couple of years ago, there is no reason to go with anything else. Gitlab, Bitbucket, and other alternatives are all fine too but I always end up coming back to Github.

My go-to choice for CI and CD pipeline is Netlify. I am getting close to the CI build allowance. When I go over it, I might decide to pay for this as it has made my life so much easier. The only other real competitor to Netlify is Vercel. Of course, Github actions is also very competitive, but Netlify and Vercel are more straightforward, more powerful, and there's something nice about having a dedicated CI pipeline.

Netlify and Vercel also come with other added benefits such as On-Demand Environments (ODE), essentially creating a preview for each pull request you make, allowing you to test out the app before merging the code to master. They both host the static files (e.g. HTML, JavaScript, and more). They have built-in CDNs for custom domains, redirects, etc.

Netlify and Vercel support analytics, but both platforms have it under paid-tiers. I decided to stick with Google Analytics for now because it is... you guessed it, free! There are also a number of other alternatives with better privacy promises, e.g. Matomo. On the other hand, Segment seems to be a popular choice for businesses.

I use a headless CMS for storing my data if I'm creating an application that is content-heavy. My top choice right now is Prismic with Contentful at a close second. If the application needs to store user-generated data, my go-to database service is FaunaDB.

I have covered some of these services in more depths here: Awesome tools to use in personal projects.

Summary

As a software engineer, I love to experiment and work on personal projects, with the dream of one day creating a product that goes viral. Needless to say, this is both a luck-based exercise as it is through hard work. It will take many, many attempts at different ideas to be successful. Any failed experiments can be costly if we are not careful, not just our time and energy but may even leave our pockets empty. So it makes sense for us to make minimising cost our number one priority. Only by taking the financial burden off our shoulders can we enjoy the process of working on personal projects. Knowing that if the project isn't successful, we have nothing to lose, only valuable experience and learning to gain, is comforting and motivating.

Finally, based on my personal experience with cost-saving, there's a point where it'll give diminishing returns. Don't end up spending months researching the cheapest way to do something. Just be pragmatic and practical with this whole concept.

Hello 2021, I'm late to the party

Sun, 07 Feb 2021 22:52:00 GMT

Hello friends, it's been a while since I published anything on this blog, it's been so long since I last posted anything that it is a little embarrassing. I have gone off the grid because I was just a little occupied from a few personal projects in the past few months. With that said, the Xmas break was a perfect time to recharge my batteries.

Learning about note-taking and changes in workflow

One of my projects is probably going to sound bonkers - I was too busy tidying up my notes. Or using the big words, I was learning, adapting and improving my note-taking method. Why? To be more effective at managing what I take in and how I process them and convert them to content so others can consume, like this post.

I have many notes saved on the topic of "how to take notes effectively", but I intend to leave it for another day. This post was meant to be a quick welcome back after all.

Changes to this blog

If this isn't the first time you are here, another change you probably have noticed is the domain name. Yep, I've recently migrated my blog from haodong.io to this one blog.hao.dev . For a long time, I wanted to do this, ever since I purchased the current domain back in Jun 2020. At one point, I just decided enough was enough, and that I need to prioritise this over other tasks. As soon as I finished the note-taking migration project, I jumped on this.

There is a reason why the blog is on the blog subdomain, I may write about it at some point, but honestly, I am more motivated to turn my plan into reality. In short, I wanted to add a few sections to my site, and it felt weird, stitching them to this blog. The plan from now on is to split up this site into multiple applications by sub-domain. Then it would enable me to have an isolated place to publish blog posts, a place to deploy some side projects, and a proper welcome area which can link off to the different areas rather than presenting first-time visitors a list of blogs.

Finally, I have always been interested in writing about other topics that always fascinated me. I debated if I should start new blogs for each topic I want to publish. I had this feeling that mixing the different contents might not be the right thing to do, in case people who are interested in coding may not find the other topics very interesting. But in the end, I decided to keep things simple and publish it all on this blog. The other reason is that I have such a tiny audience it is not like anyone would be upset anyway 😂

Final words

Lastly, I would like to say welcome to 2021, even though it's already February. 2020 wasn't a brilliant year for many reasons, here's hoping this would be a better year for all of us! As well as looking forward to getting out properly, I am looking excited to see if the ways of working changes I've adopted will get me to publish more content or not!

How to use different git emails for personal and work repositories on the same machine

Tue, 06 Oct 2020 18:07:00 GMT

I recently started working on some side hustle projects. To keep things organised, I created a new git account with a different email. When I tried to push to the git repository within the new git account, I got the following error.

Enumerating objects: 26, done.
Counting objects: 100% (26/26), done.
Delta compression using up to 12 threads
Compressing objects: 100% (16/16), done.
Writing objects: 100% (21/21), 69.90 KiB | 4.37 MiB/s, done.
Total 21 (delta 2), reused 0 (delta 0)
remote: GitLab: Author 'personal@mail.com' is not a member of team
To gitlab.work.co.uk:project/repo.git
 ! [remote rejected] feature/a -> feature/a (pre-receive hook declined)
error: failed to push some refs to 'git@gitlab.work.co.uk:project/repo.git'

(I removed some sensitive data.)

I didn't want to change my default git email, because I still want to use it for other git repositories. After some research, I've found two ways to solve my problem.

Method 1: use different git email based on repository path

This method involves adding some conditions to global git config. Then within each git repo, it will automatically figure out which email to use based on the path.

By default, the global git config looked something like this for me.

# ~/.gitconfig

[user]
    email = personal@email.com
    name = Hao Dong

1. Rename ".gitconfig" to ".gitconfig-personal"

Rename the global config to ~/.gitconfig-personal using the following command. It will now be our new config for any personal projects.

mv .gitconfig .gitconfig-personal

Don't worry, we will add the global config back later.

2. Create ".gitconfig-work"

Create the new config file for work.

touch ~/.gitconfig-work

Then add the following into the config file.

# ~/.gitconfig-work

[user]
    email = work@email.com
    name = Hao Dong

3. Create a new ".gitconfig"

Add a new global .gitconfig file, this will point to the other two custom config files.

touch ~/.gitconfig

Then add the following into the global config file.

# ~/.gitconfig

[includeIf "gitdir:~/work/"]
    path = .gitconfig-work
[includeIf "gitdir:~/personal/"]
    path = .gitconfig-personal

Source: https://git-scm.com/docs/git-config#_includes

Method 2: setting git email per repository

Personally, I would suggest going with Method 1, no matter your situation. But if you are confident it is a one-off case, then this method might be appropriate.

Navigate to the work repository, then at the root folder run the following command to change the email.

$ git config --local user.email name@work.com

Note: this command only affects the current repository. Any other repositories will still use the default email specified in ~/.gitconfig.

Question: I did all that, but the commit still says the wrong email

If you already committed the code change like I have, then you will have to go through each commit made prior to the email change and then update the email in the commit using the following command.

git commit --amend --author="Full Name <email@address.com>" --no-edit

How to share a Roam page with others?

Wed, 30 Sep 2020 21:22:00 GMT

Roam is still in active development right now, which means some common features in other note-taking app is not quite as easy to use here. One of these features is the ability to share a single page of note with friends or family. In this blog post, I'll show you how to do this properly.

Firstly, I just to make it clear that there are two ways to share notes in Roam. The first way is easier to find in the menu, but it may expose your entire notes on Roam to the public. The second way is what we want, which only makes one specific page publicly accessible.

Go to the page you wish to share, then click on the three-dot menu. You should see similar options like the ones shown in the image below.

If you click on Share, then you should see the following pop up appearing. There is a warning message saying this is still an experimental feature, so use this with caution, there could be bugs where when you share a single page Roam may accidentally make other pages linked publicly accessible as well.

If you are happy to continue, then toggle "Page Permissions" on. The toggle button should now be active (blue).

Now close the pop-up, and click on the three-dot menu again. This time you should see "Share Page" Option similar to the one below.

Click on it, and a different pop-up will appear. By default, the page permission is set to Private, change it to Public Readonly then close the pop-up window.

Then click on the three-dot menu again, this time select Share link to this page.

Clicking on the menu option should automatically copy the page link to your clipboard. At this stage, you should see the following alert notification if everything is working as expected.

Now the link is ready to be shared with the right people!

Boost your productivity with Google Calendar shortcuts

Tue, 22 Sep 2020 19:00:00 GMT

I use Google Calendar, both personally and professionally. I have a shared calendar with a friend to keep track of different events and activities. I have always found it hard to navigate Google Calendar in the browser, I love the look and feel of the web app, though I find myself always looking for the right buttons, etc. I knew about the brilliant shortcuts that made Gmail fun to use and decided to see if Google Calendar offered something similar. Luckily, they do, and they are straightforward to remember.

Changing Views

I am always changing between Day, Week or Month views. I use Week view to plan and check upcoming events that I should be aware of for the week. I use Day view when I want to focus. Month view is convenient for a lot of other tasks, despite I only use this occasionally.

During the day I'm continually changing between Day and Week views, sometimes even Month view. Which means each time I open up the calendar, it is often not showing me what I want to see. I end up clicking around a whole bunch to get what I'm interested in to display.

As it turns out, there are shortcuts to switch straight to these views:

1 or D Day View
2 or W for Week View
3 or M for Month View

Quick Access To Calendar Search

When searching for events that you might not know the exact date range, it is useful to use the search feature built into Google Calendar.

Most people probably don't even notice the small search icon in the top menu bar. It is also quite fiddly to click because of how little it is.

/ will put your cursor directly in the search box, therefore making it easier to search for events quickly.

Navigate Date Ranges

I mainly plan in Week view because it is not possible to see everything clearly in Month view. However, I often need to go back and forth between the current week and the next couple of weeks. The chevron buttons ("< "and">" respectfully) at the top menu bar are for navigating back and forth. They are a pain to use because of how tiny they are, and the round shape makes the clickable area even smaller.

Luckily, we have a few easy shortcuts to get the job done.

N or J moves the calendar view to the next date range. e.g. go forward one week in Week view
P or K moves the calendar view to the previous date range. e.g. go back one week in Week view
T moves to the current day. Very useful for getting back to the current week.
G will open a popup allowing you to specify a date and then move to the date, which is useful for hopping around dates that are far from the current date range.

Additional shortcuts

Personally, these doesn't fit well into my workflow. However, each person works differently. So some may also find the following shortcuts useful to know.

4 or X for Custom View
5 or A for Agenda View
6 or Y for Year View
R to refresh when using Google Calendar on multiple devices/tabs
C to create a new event
Backspace or Delete to delete an event
Esc will return to calendar grid from an event details page

For more information, you can check out Google's official doc on this.

Time to go back to the fundamentals of software development?

Thu, 10 Sep 2020 06:00:00 GMT

With all the new toys coming out each day, libraries, frameworks and even architectures. Have we forgotten the fundamentals of software development? Every developer wants to write good code, but how often do we do what we say? Maybe, every so often we all need to take a step back and rethink our approaches to software development. If nothing else, we should find challenges and excitement in what we do, if only to make sure the passion that drove us to this career path doesn't die out.

When I was taking my regular walk in the park, I couldn't help but notice how people perform "heel strike" when they jog. It is when the heel of the foot lands first when running. It isn't the most effective way to run, since each leg carries more than twice our body weight when it hits the ground, and when the heel lands force travels up the led and ends near our knees and hips. Long term jogger may find themselves with injuries in these areas. The best form of running on flat pavement, arguably, is a midfoot strike. Where the middle of the foot land first then followed by the heel. It gives the leg muscles time to absorb the force like a spring before they travel up the body.

Considering our ancestors depended on running to survival in the majority of the past 200,000 years. There fore running is and should be instinctive to us, and yet we don't all seem to do it correctly for one reason or another. It just shows even the most basic and essential things in life there's a chance we might not be aware we are not doing it well.

Which made me think, what else could I be doing wrong without realising?

I've been working in the software development field for over six years. It's not a crazy long time compared to people who's been doing this for 30 years. But it is still long enough for me to develop many ineffective habits or misconceptions during this period without realising. Perhaps even those I practice daily at work?

Given how fast everything changes in the digital world, it is a constant battle to stay relevant, not to mention finding the time to perfect or enhance our existing skills. It might even logically make sense to tune down the perfectionist voice in our heads and focus only on getting what's right in front of us, our work, done. There is no time to be steady, methodical and think. But aren't these traits of a good software engineer?

Learn and priorities language specification changes

People are so busy looking at new shiny toys which flood our social feeds each day. Could it be possible that we forgot the more fundamental part of our skills - programming languages?

All the frameworks and libraries are cool, the tools and their plugins are fantastic, and the new theme for the terminal is pretty sick! However, as long as you develop software, programming languages are at the core of what we do. Just like everything else, languages change and evolve over time. I wonder how many people actually follow the new language spec changes and study it properly? It's shameful for me to say that I'm one of these people right now.

There could be many reasons for this, and I think one of the major contributors is the fact most of us don't work in an environment that promotes the importance of language specification changes enough. Instead, skillset and knowledge about frameworks and libraries are more heavily promoted.

Perhaps we should spend more time studying the language specification and understand the limits of the language we use every day. Often, learning the new features and design shift highlights the limitations of the older approach. It helps us to understand the why behind individual language quirks, but more importantly, we will know when "best practices" posted all over the internet stops being true.

I once heard a senior JavaScript developer telling junior devs to never use forEach() in JavaScript due to its "inefficiency". The person isn't wrong, and this performance analysis proves it. The crucial piece of information left out is that we often never loop over arrays with a length of 50 or more. The difference is minuscule with this kind of data set, I mean we are talking about a fraction of a millisecond!

Write our own libraries

Open-source libraries are amazing. They solve the most challenging problems and present them as a nice solution, allowing us to focus on other aspects of our job. JavaScript's package manager reach its one-millionth goal back in Jun 2019. That is a lot of potentials to leverage when it comes to creating applications.

Positives aside, I reckon it is also dangerous to always reach out to these 3rd party libraries whenever there's a problem. Maybe sometimes it is worth implementing a simple and even substandard piece of code instead of pulling down a library from NPM. In many ways, using software other people wrote is a dark path to go down, the more you rely on them, the less you'll learn, your mind gets lazy, and the cycle continues.

I worked in a team a few years back that decided to write a custom rendering library just because it was an interesting experiment. We knew we'd learn something along the way, and that we'll be able to get the job done no matter what. The result was fantastic; every member of the team had a deep understanding of the core rendering logic, fixing bugs and making optimisations couldn't have been easier. Needless to say, I feel like I learned more in that one project than ten projects using an off-the-shelf library.

What's new with TypeScript 4.0?

Fri, 21 Aug 2020 07:52:00 GMT

TypeScript 4.0 just came out, and people probably want to be upgrading to it soon. Here's a quick summary of what's new based on official release notes. This is only meant to be a quick summary, so I have left out edge cases and detailed examples. If you have time, I'd recommended giving the official release note a proper read. But for those who want to get a high-level gist of what changed, this might be useful.

Of course, all credits goes to TypeScript team.

Variadic Tuple Types

TypeScript 4.0 introduced two fundamental changes to improve typing:

Spreads in tuple type syntax can now be generic.
Rest elements can occur anywhere in a tuple, not just at the end.

It is now possible to assign better types to tuples, which validates lengths of the input and the order of the elements.

function concat(arr1, arr2) {
    return [...arr1, ...arr2];
}
`"

```jsx
type Arr = readonly any[];

function concat<T extends Arr, U extends Arr>(arr1: T, arr2: U): [...T, ...U] {
    return [...arr1, ...arr2];
}

See official release note for more info.

Labeled Tuple Elements

In order to improve readability, it is now possible to provide labels to tuples types.

// ealier versions
type Range = [number, number];

// TypeScript 4.0
type Range = [start: number, end: number];

This could be useful when using tuple in the following situation. In earlier versions, there are no parameter names which can make the code harder to understand.

function foo(...args: Range): void {
    // ...
}

See official release note for more info.

Class Property Inference from Constructors

TypeScript 4.0 now uses control flow analysis to determine the types of properties in classes when noImplicitAny is enabled.

In the following example, TypeScript can work out the type of area based on the assigned type of sideLength and the fact sideLength was used to calculate area.

class Square {
    // Previously: implicit any!
    // Now: inferred to `number`!
    area;
    sideLength;

    constructor(sideLength: number) {
        this.sideLength = sideLength;
        this.area = sideLength ** 2;
    }
}

See official release note for more info.

Short-Circuiting Assignment Operators

Compound assignment operators apply an operator to two arguments and then assign the result to the left side. This is very commonly used in JavaScript and other languages.

// Addition
a += b; // a = a + b

// Multiplication
a *= b; // a = a * b

// Exponentiation
a **= b; // a = a ** b

// Left Bit Shift
a <<= b; // a = a << b

TypeScript 4.0 has introduced three new assignment operators:

&&= that substitutes a = a && b
||= that substitutes a = a || b;
??= that substitutes a = a ?? b;

Suppose you haven't seen ?? before (like me). It is called a nullish coalescing operator. It is a logical operator that returns its right-hand side operand when its left-hand side operand is null or undefined and otherwise returns its left-hand side operand. Here are some examples from MDN.

const foo = null ?? 'default string';
console.log(foo);
// expected output: "default string"

const baz = 0 ?? 42;
console.log(baz);
// expected output: 0

See official release note for more info.

unknown on catch Clause Bindings

From the start TypeScript uses any type for catch variables. Although nothing has been done to the default catch variables. This new version of TypeScript has allowed assignments of unknown type instead of any since unknown is safer.

try {
    // ...
}
catch (x) {
    // x has type 'any' - have fun!
    console.log(x.message);
    console.log(x.toUpperCase());
    x++;
    x.yadda.yadda.yadda();
}

Probably don't need to worry too much about this for now. Just know that there might be more changes coming later which will build on top of this.

See official release note for more info.

Custom JSX Factories

In JSX Fragments allow a component to return several child elements without specifying a wrapper parent element.

const Component = () => (
    <React.Fragment>
        <img src="icon" alt="icon" />
        <button type="submit">Click me</button>
    <React.Fragment>
);

In Typescript 4.0, it is now possible to customise the fragment factory through the new jsxFragmentFactory option in tsconfig.json to apply this change globally.

{
  "compilerOptions": {
    ...
    "jsx": "react",
    "jsxFactory": "h",
    "jsxFragmentFactory": "Fragment"
  }
}

Alternatively, it is possible to only apply to single files using the new /** @jsxFrag */ pragma comment.

/** @jsx h */
/** @jsxFrag Fragment */

import { h, Fragment } from "preact";

let stuff = <>
    <div>Hello</div>
</>;

See official release note for more info.

Speed Improvements in build

TypeScript now cache previous compilation in .tsbuildinfo file, to speed up compilation with --incremental and --noEmitOnError option after a previous compile error.

It is now also possible to use --incremental with --noEmit options and still take advantage of the faster build.

See official release note for more info.

Editor changes

I won't go through them too much here, as they are not likely to affect most people. Here are one sentence explanations:

TypeScript's editing support now recognises when a declaration has been marked with a /** @deprecated * JSDoc comment.
Visual Studio Code supports selecting different versions of TypeScript.
Visual Studio 2017/2019 have the SDK installers above, and MSBuild installs.
Speed up Visual Studio Code startup time for large projects with a new mode for editors to provide a partial experience until the full language service experience has loaded up.
Smarter auto-imports.
There is now a new refactoring option which can convert code to utilise optional chaining.

Breaking Changes

See official release note for more info.

lib.d.ts Changes

Our lib.d.ts declarations have changed – most specifically, types for the DOM have changed.

Properties Overriding Accessors (and vice versa) is an Error

TypeScript now always issues an error when declaring a property in a derived class that would override a getter or setter in the base class.

Operands for delete must be optional

When using the delete operator in strictNullChecks, the operand must now be any, unknown, never, or be optional (in that it contains undefined in the type). Otherwise, use of the delete operator is an error.

interface Thing {
    prop: string;
}

function f(x: Thing) {
    delete x.prop;
    //     ~~~~~~
    // error! The operand of a 'delete' operator must be optional.
}

Usage of TypeScript's Node Factory is Deprecated

TypeScript 4.0 provides a new node factory API, the older "factory" functions for producing AST Nodes are now deprecated.

See official release note for more info.

Running client side only JavaScript with Gatsby.js

Wed, 12 Aug 2020 06:00:00 GMT

Gatsby is great for generating static sites, but sometimes we need to run some client-side only JavaScript as well. The question is how might we go about that? Well, that's what I'll be covering in this blog post. Actually, I've already written about something similar in a recent post Render client-side only component in Next.js. The approach is very similar, but I'll be walking through an example in this blog that is more specific to Gatsby.

What I tried to build

If you open up this blog on a mobile device or resize the window to a small enough width, you'd see the sandwich menu button appear. The idea is that when this button is clicked, an overlay would appear along with the navigation links.

The challenge is that, if I implement this in React, then Gatsby will try to render out the logic into HTML. Which means after the site has been generated, clicking on the menu button will not do anything.

There are two crucial factors to consider when trying to implement client-side only logic in Gatsby:

Delaying any logic during "on-component-mount" stage of the life cycle until we are on the client-side.
We need to be careful with any window or document references that only exist on the client-side.

The solution

I used useState to track the open/close state of the overlay. Since I wanted to disable the background scroll when the overlay is open as well, I ended up going with handler function handleMobileNavChange() to set the overflow value of <body> tag.

On clicking on the button, the value of mobileNavOpen will be set to true, and use can close the overlay by clicking on it which will set the state value to false.

const Header = () => {	
	const [mobileNavOpen, setMobileNavOpen] = useState(false);

	function handleMobileNavChange(value) {
        if (value) {
            document.body.style.overflow = "hidden";
        } else {
            document.body.style.overflow = originalStyle;
        }
        setMobileNavOpen(value);
    }

	return (
		<>
			<button type="button" onClick={() => handleMobileNavChange(true)}>Sandwich menu</button>
			<Overlay onClick={() => handleMobileNavChange(false)} />
		</>
	);
}

The value of originalStyle is set outside of the Header component by reading the value from DOM. It was to ensure the correct overflow value is fetched on the initial page load.

The typeof window !== 'undefined' check is to prevent Gatsby from erroring when generating the static site since window only exists on Browser and would error in Node environment.

const originalStyle = typeof window !== `undefined` && window.getComputedStyle(document.body).overflow;

const Header = () => { ... }

export default Header;

Final words

I have simplified the example above to make the explanation a bit simpler, if you are interested here is the code on Github.

Fixing Cypress errors part 3: miscellaneous and additional notes

Thu, 30 Jul 2020 18:00:00 GMT

This is the last part of this blog series on Cypress errors. There were some additional notes I made which didn't go into the previous posts. This blog post is less specific and more about general advice and some obscure errors I saw along the way.

Blog Series:

Are you using the latest of everything?

Cypress is being actively developed right now, in the span of 2 months, I saw six minor updates. It is very likely any issues that you see, assuming it was something Cypress team could address, then it'll get resolved very quickly.

We were using cypress/included:4.5.0 at the start, I updated it to cypress/included:4.10.0 after reading a lot of bug fixes in the changelog. Unfortunately, upgrading to the latest version didn't help me much with my issues. However, if you are debugging a new issue that occurred on an exiting project, this might be the first thing to try. Cypress team provides an official changelog. So give that a read instead of blindly upgrading. Don't forget to watch out for those breaking changes!

Rare errors

Test runner exited unexpectedly

I saw this error once or twice during my instigation (running over 100 builds). It wasn't obvious what caused the problem, and I wasn't able to reproduce it since. Maybe just something to be aware of that this kind of errors could occur.

Failed to connect to Chrome

This error seems to be caused by the fact Cypress wasn't able to connect to Chrome. I'm not sure how this could happen when everything was running inside a Docker container on the CI pipeline.

Once again, I've only seen this error once after switching over from Electron to Chrome, and the failure also went away after triggering a rerun.

Fixing Cypress errors part 2: cannot find element due to assertion timing out

Thu, 30 Jul 2020 14:00:00 GMT

Apart from the Cypress chromium crashes I explained in Part 1 of this blog series, I also experienced some assertion timeout errors due to element cannot be found. In this part, we will talk about ways you could fix it.

Blog Series:

Error: Random timeout issues due to flaky tests

This problem was almost as common as the chromium crashing error. Cypress E2E tests would randomly error and complain it cannot find certain elements. The weird thing is that most of the time, all tests pass, and when tests fail, it is different every time. Finally, rerunning the build often fixes them, to the point people who maintain the project told me this is fine.

Note: interestingly for me, the flaky tests stop after implementing the memory crashes mentioned in Part 1. So before you try any of these solutions, give that a try first.

Solution 1: Increase timeout time

Back in Selenium days, it was common to use commands like wait(10000) in between assertions to wait up to 10 seconds in case network was unstable or slow. It would reduce tests' flakiness in a lot of situations.

Cypress has a similar API cy.wait(500), but we don't want to use it because it is an anti-pattern as explained by Unnecessary Waiting - Cypress best practices.

Instead, we are looking at using a more elegant solution. Since Cypress already has a default timeout duration of 5000ms (5s), sometimes we may need to extend the duration slightly without interfering too much with what Cypress is doing.

We can do this by passing an extra argument to cy.get() API. Here is an example of how to do this:

cy.get('[data-testid="modal"]', { timeout: 10000 }).should('be.visible');

Of course, if you have timeouts all over the place as I mentioned above, then maybe, we need a more drastic approach and extend the global timeout.

The simplest way to do this is by using the defaultCommandTimeout flag or in this case, updating it in the cypress.json config file.

// cypress.json
{
  "baseUrl": "http://localhost:8080/",
  "defaultCommandTimeout": 10000,
}

Personally, this didn't help much with the errors I saw. They still appeared just as frequently after I made these changes.

Solution 2: Retrying failed tests

The alternative to increasing the timeout duration is to run the failed tests again. Based on what I saw as part of my investigation, a lot of people were recommending this approach as well.

The plugin is called cypress-plugin-retries, and it will do what the name suggests, rerun individual failed tests. After implementing this, the timeout errors all disappeared from the pipeline.

The plugin's README has clear instructions on how to implement it so that I won't repeat it here.

Final words

My conclusion is that solution 1 is worth trying if you always see the same tests failing due to assertion timeout. If the tests are failing randomly, I think solution 2 is a better bet. But as a colleague of mine correctly pointed out, these solution 2 doesn't remove the root problem in your application.

Lastly, if you are still having issues, maybe check out the Part 1 and Part 3 of this blog series.

Fixing Cypress errors part 1: chromium out of memory crashes

Thu, 30 Jul 2020 09:00:00 GMT

After recently working on an existing project that adopted Cypress, I was both amazed at how pleasant the local developer experience was and how badly it runs on the CI pipeline. No joke, at one point I actually missed Selenium. To debug the issue, I must have run the pipeline over 100 if not 200 times and read many many Github issue threads. Finally, I was able to problem resolved and noted down a few things along the way which may help someone else in the same shoes. I am pretty confident these solutions will fix your problems, if not leave an angry comment and let me know that I failed 😂

Blog Series:

Error: Out of memory, chromium renderer crashed

Chromium crashing is one of the issues I kept seeing in the pipeline. It would happen even when no code was changed and with all the tests passing locally.

Cypress team was kind enough to give a very detailed error message in the console and even a link pointing to a potential fix. If you follow the link provided, you will find the same error description on Cypress' official website and yet another link to click on which takes you on this Github issue 350. A few pieces of advice were offered here on this thread, let's take a look at them separately.

Solution 1: Use --ipc=host Docker flag

The was a recommended fix from one of the Cypress authors.

This is actually not indicative of a memory leak inside of Cypress (nor your application). This has to do with the way that browsers work under the hood. Luckily, we have a found a simple one line fix for this problem - simply add the flag --ipc=host.

IPC stands for inter-process communication, and it is related to Docker networking. Docker allows a few different options and host is one of them. With this option, it is meant to enable the Docker container to use the host system’s IPC namespace. It seems to solve insufficient shared memory (shm) issues within worker (child containers). To be completely honest, it is a little over my head what the underlying issue is here. My understanding is that this config allows the child container to utilise its parent's memory resources which would be more than its own, and thereby eliminating the problem.

If networking is something you are interested or knows a lot about, maybe you will have better luck with the Docker doc, which explains it this in more detail.

So did this fix my problem? Nope. When I checked this workaround was already in place, but I was still getting those crashes reasonably consistently. But this might be worth trying if you do not have this in place already.

Solution 2: Use --disable-dev-shm-usage Cypress flag

If you continue to read in the comments in Github issue 350 you will see people suggest using --disable-dev-shm-usage flag with Cypress.

When I decided to try this, I found out that the project already had this fix in place, but it was using the wrong config. It was using chrome browser family with Cypress v4 (see below for more description on why this is wrong). Unfortunately, even after correcting this mistake, the pipeline was still crashing. Which meant I had to continue my search.

Given the number of positive emojis on the Github comment, it must have solved the problem for a lot of people. So don't let the fact it didn't work for me put you off, give this a try by reading how I tested this solution. below

There are different ways to implement the same fix in Cypress, so we will talk about both here.

Using browser family

If you are using family property, make sure to use chromium if you are using Cypress v4 or later. If you are using v3 or earlier then use chrome.

As of Cypress v4, the property now supports two options: chromium or firefox.

// cypress/plugins/index.js

// v4
module.exports = (on, config) => {
  on('before:browser:launch', (browser = {}, launchOptions) => {
    if (browser.family === 'chromium') {
      launchOptions.args.push('--disable-dev-shm-usage');
    }

    return launchOptions;
  });
}

// v3
module.exports = (on, config) => {
  on('before:browser:launch', (browser = {}, args) => {
    if (browser.family === 'chrome') {
      args.push('--disable-dev-shm-usage');
    }

    return args;
  });
}

There was a breaking change introduced in Cypress v4.0.0.

We updated the Cypress browser objects of all Chromium-based browsers, including Electron, to have chromium set as their family field. Addresses #6243.

Source: https://docs.cypress.io/guides/references/changelog.html#4-0-0

See this Github comment on differences between v3 and v4.

Using browser name

Using browser name means you have to specify the exact browser name.

This property supports these options: chrome, electron, edge or firefox.

For example, if you wanted to use both chrome and electron then your snippet might look like this.

// cypress/plugins/index.js

module.exports = (on, config) => {
  on('before:browser:launch', (browser = {}, args) => {
    if (browser.name === 'chrome') {
      args.push('--disable-dev-shm-usage')
    } else if (browser.name === "electron") {
      args["disable-dev-shm-usage"] = true;
    }

    return args
  })
}

PS: notice how Electron sets the argument slightly differently. There is no better reason other than Electron takes in options different to other browsers. See this Github comment for more information.

Solution 3: Change over from Electron to Chrome

So where do we stand right now? The project I was working on already had the first two fixes in place (although I had to fix the browser family). Just before I was losing hope, I saw this Github comment asking when the memory issue will be fixed for Electron.

Could this be the issue?

Since the project I was working on was already using the cypress/included docker image, this meant the Docker image included Chrome browser already.

image: cypress/included:4.10.0

And I could simply use --headless --browser chrome flags to tell Cypress to use Chrome instead of Electron.

cypress run --headless --browser chrome

After pushing this change, I finally got some positive results!

This was the first time where I was able to get the pipeline to pass 5 times in a row. The previous best score was 3 out of 5 runs. Of course, all subsequent runs have all been running fine as well!

Solution 4: Reduce the number of tests kept in memory

As mentioned already, I was able to address all of the pipeline issues I was seeing using the above solutions. This solution was something I discovered, tried out and didn't have much luck with it. I thought I'd document it here anyway, in case it is the necessary fix for some people's problems.

In a more recent (and still open in time of writing) Github issue 4164 introduced me to this Cypress config called numTestsKeptInMemory.

You can set it in cypress.json file like this or pass it in as another flag.

// cypress.json
{
  "baseUrl": "http://localhost:8080/",
  "numTestsKeptInMemory": 0,
}

The default value of numTestsKeptInMemory is 50. For some people reducing this number seemed to have helped them. The recommendation is to try with a low value such as 1, and if that doesn't work then finally try 0.

Note: setting numTestsKeptInMemory to 0 will disable this feature altogether, and no tests will be kept in memory.

There is the official documentation for more information.

Final words

CI pipeline issues such as this are very annoying to fix because it is not reproducible locally. Which means it requires pushing changes and wait for the pipeline to finish to test anything. I hope you managed to resolve your issue using one of the solutions mentioned here, if not try why out Part 2 and Part 3 of this blog series, hopefully, one of them contains your answers!

Just a side point, as part of the research I found out about this open Github issue 349 which is looking at introducing auto-recovery to Cypress which may continue running even after a crash. However, since it was opened back in 16th Dec 2016, therefore I wouldn't hold my breath for it to be done any time soon.

How to approach a disagreement

Tue, 28 Jul 2020 17:59:00 GMT

As software engineers, there are a lot of things to disagree on: tech stack, syntax, code standard, testing, methodologies, etc. No matter your seniority, learning how to frame a disagreement is vital. It includes both disagreeing with what others have done and have others disagree with your approach. It happens a lot more often than we expect, so if we don't pay attention to our methods, then it is easy to cause stress and tension within the team or between teams. If this continues for a long time then well, then it will create a toxic work environment. As someone who has a consultancy background, which requires people to jump in a new team and start delivering value as well as navigate office politics, I thought this is an intriguing subject for me to share my thoughts on.

When dishing out disagreements

Are you raising a valid concern?

Sometimes, our emotional side can get to us, and we raise questions which aren't related or urgent. It is worth being mindful about the issues you are highlighting and reflect on it before speaking up. If it is already too late, then there is nothing wrong with a quick apology and retract your statement. It is as simple as saying something like "actually, on second thought I am not sure it is relevant. Please ignore me".

Having the gut to admit you are wrong is a good thing, and I, for one, appreciate this kind of behaviour over continue to argue even if you are not sure for argument's sake. It may depend on the company culture, but it is worth questioning if you should remain in a company that penalises you for making small mistakes like this.

It is always better to do the reflection in your head before speaking up, and you don't want to be the person always asking questions and continuously retracting their questions in meetings.

Are you offering solutions?

There is nothing wrong with raising a lot of questions or issues, and instead, it should be encouraged. It doesn't mean the raised questions or concerns must be addressed straightaway. If they are things which can be taken care of later, then create a ticket and add it to the backlog for a later date.

It is also essential to think about some potential solutions when raising issues. Not only does that show you have put thoughts into it, but the extra effort will make people less defensive since the conversation is more about picking the best solution at that point instead of the natural response to issues = who caused the problem.

Compare the following statements:

Statement 1: "Some bad practices are being used in the codebase".
Statement 2: "Some new practices are being used in the codebase deviating from the standard. Should we consider adopting a linting tool going forward? I think Eslint is quite good, but I'm sure there are others out there. Maybe we should have a spike on this?"

Are you motivated to get it resolved?

It is often easy to point out problems with something others had done. When you raise a concern, in people's eyes, you are getting yourself involved (unless you are the CEO of the company). Just saying this isn't good enough is not only irresponsible but creates a "you v.s. me" situation, and implies "see, I'm better than you" kind of message. Even if you didn't mean that it is easy for people to get that feeling, from a logical sense, if the issue you pointed out if actually that urgent, then maybe it makes sense for you to act on it as well and not just say it.

Again, compare these statements:

Statement 1: "I noticed the app error rate has increased by 200% over the last week".
Statement 2: "I noticed the app error rate has increased by 200% over the last week, who's the best person I should talk to about this? Happy to share my findings, support or pair on this as it is affecting the stability of the site."

Lead by example is always a great way to build team rapport, and everyone will remember it (including your managers 😁).

When dealing with disagreements

Be patient and professional

We all make mistakes, and we need to own up to them. Getting defensive is never a good look, nor is never admitting that you are wrong when the evidence is right in front of you. Being professional means being accountable to your actions, after all that is professional if everyone can go and do things and not worry about consequences? When people raise issues about your work or approach, don't cut people up mid-sentence. Allow everyone to express their feelings and thoughts, then have a healthy and calm discussion about it. It is possible that it was merely some misunderstanding, and after some talk, you both find you agree with each other. Some people might have different ways to express themselves, and others may even have language problems if they are not speaking in their mother-tone.

Give people benefit of the doubt, and assume people are curious instead of attacking you or your work personally. Some people might speak fast or have a loud voice; in both situations, they might not even be aware of it and don't mean harm. So don't react too quickly, and you'll avoid awkward apologies after.

If you are sure someone is rude or aggressive towards you, then consider giving them feedback or speaking with them in private before raising it with the right channels (e.g. manager, HR, etc.). It could just be that person is having an awful day and not being themselves.

Be able to take criticisms

Some criticisms might be misjudged, but it is always important to reflect on feedback and think about what may have triggered the other person to feel a certain way or say what they said. If someone was questioning code quality was it because how you structured your code or how you named everything which made it hard to understand? Just because you are not in the wrong, it doesn't mean there is no room for improvement.

We should welcome critiques, only by knowing what we are doing wrong can we improve the next time. Some even regard feedback, positive or constructive, as a gift. Also, if it is not something you value too much about, the fact the other person wants to open up to you and tell you what they think or how they feel is huge. It shows that they value you as a teammate. After all, would you go out of your way and give critiques to a stranger?

Giving good feedback is difficult, and many people avoid giving criticisms because they don't feel comfortable for one reason or another. As a result, it is probably worth valuing what critiques you do receive.

Finally, just want to say being able to take in criticism isn't the same as accepting abuse. I doubt this will happen to you, but just something to be aware of in the back of your mind. Again, always give people benefit of the doubt when unsure!

Admit mistakes

No one likes to be told they are wrong, and wouldn't it be wonderful never to make mistakes? Unfortunately, that is not possible or realistic. As a responsible professional, we need to stand by the decisions we've made and more importantly admit we are wrong when presented the facts.

Is it painful to do that? Yes, absolutely. But that is not the reason to shy away from it. Painful experiences are the ones we remember, and this is how we learn and improve the next time. The main take away is to remember not to get yourself into the same situation the next time. Did you over-promise something or were too sure of yourself? Maybe next time leave some room for manoeuvre when making a claim. Was your work not up to standard? Maybe next time get someone to check over it before putting it in front of everyone.

The best way to deflect the shame and regret that comes from mistakes is to be the first one to point out that you made a mistake. By doing this, not only do you avoid someone else wording the situation that's less in your favour, but also takes the worrying out of the equation. The worst part about making a mistake is thinking "Oh god, when will people find out, and what will they do or think?". By openly stating your mistake, you will eliminate that worry.

Final words

Unable to have a productive discussion due to fear of confrontations or lack some communication skills is something many people struggle with from my experience. It had made me appreciate even more the "soft-skills training" I received back when I was working for a consultancy.

In a lot of ways, it is all a mindset difference, but it is harder than it sounds because people have often built the mindset into their ways of working and styles of communication. These points I mentioned aren't tricky to do, just requires some self-reflection to identify how and where you need to improve. Or even better, ask for others for feedback!

How to render react components in markdown in Gatsby

Mon, 20 Jul 2020 18:37:00 GMT

In a recent post Stop trying, you can't multitask, I wanted to embed a few productivity books which I found useful. Since I write all the content in Markdown, I thought about embedding iframe components or insert image version of the book components in the Markdown, which are both terrible ideas. Luckily, there's a better way to do this with MDX, which allowed me to use React components inside Markdown. The solution is super clean, and in this article, I'll be explaining how to add React components to your Gatsby site.

Install Gatsby plugins

There are two things we need to get this to work:

Gatsby needs to understand the React component when parsing the Markdown document. I used gatsby-remark-component-parent2div to detect custom components.
Gatsby needs to hydrate the React component. rehype-react is required to transform HTML to React component.

Now install the following dependencies:

npm i gatsby-transformer-remark rehype-react gatsby-remark-component-parent2div

Why I use gatsby-remark-component-parent2div

Initially, I tried using gatsby-remark-component plugin. However, I found that it was throwing warnings in the console. It turns out Contentful, the headless CMS I am using, returns <div> inside <p>. It is against HTML specification so that some browsers will throw the warning. There are two ways to resolve this:

Change the component, avoid using <div> inside <p>
Use gatsby-remark-component-parent2div plugin instead, which changes the AST node parent of your custom component from <p> to <div>.

The first method doesn't work for me since I cannot control how Contentful save Markdown into HTML. In the end, I went with the second method, which arguably is the easier approach out of the two.

If you don't have this issue, I suggest you use use gatsby-remark-component plugin instead.

Update gatsby-config.js

If you want the ability to auto-detect custom components, then the config update is minimal.

plugins: [{
    resolve: "gatsby-transformer-remark",
    options: {
        plugins: ["gatsby-remark-component-parent2div"]
    }
}]

I wanted to be somewhat defensive with my approach, incase in the future, there's a use case for non-React custom components.

plugins: [{
    resolve: "gatsby-transformer-remark",
    options: {
        plugins: [{
            resolve: "gatsby-remark-component-parent2div",
            options: {
                components: ["book-affiliate-link"],
                verbose: true
            }
        }]
    }
}]

Update Markdown template

This step will compare the tag name of each element in the abstract syntax tree (AST), which is returned by GraphQL. If the element tag matches against the component map (in this case it's book-affiliate-link ) then it will to hydrate it with BookAffiliateLink React component.

import rehypeReact from "rehype-react"
import { MyComponent } from "../pages/my-component"

const renderAst = new rehypeReact({
    createElement: React.createElement,
    components: { "book-affiliate-link": BookAffiliateLink }
}).Compiler

Replace dangerouslySetInnerHTML with the renderAst function we introduced above because we want to render out the custom component in React instead of static HTML.

// <div dangerouslySetInnerHTML={{ __html: post.html }} />
<div>{renderAst(post.htmlAst)}</div>

Update GraphQL query

The final step is to update the GraphQL query, so we get the result in the correct (AST) form.

Previously, I was querying for the Markdown like this.

body {
  childMarkdownRemark {
    html
  }
}

You could replace html with htmlAst, however, since I need html will do a word count I now query both.

body {
  childMarkdownRemark {
    htmlAst
    html
  }
}

Final result

Now when I add the following into my Markdown document.

<book-affiliate-link title="Getting Things Done" subtitle="The Art of Stress-Free Productivity" author="David Allen" link="<https://amzn.to/30WuPNm>" imgsrc="<https://images-na.ssl-images-amazon.com/images/I/41iI4SMqzCL._SX318_BO1,204,203,200_.jpg>"></book-affiliate-link>

The following book component will appear.

Render client-side only component in Next.js

Tue, 30 Jun 2020 23:31:00 GMT

If you are familiar with Next.js then you will know it is the React SSR (server-side rendering) framework created by Vercel. There are a lot of headaches in trying to build SSR sites in React, Next.js makes it drastically simpler by doing many of the not-so-nice parts and hiding it from developers. The problem with frameworks, in general, is that it does what it is designed to do very well while sacrifices control and customisation to achieve it. Next.js is very good at handling components rendered on server-side and then sent to the client. But how would you render a client-side only component in Next.js? That is an issue I encountered and what I'll be discussing in this article.

How to make code run only on client-side with Next.js?

To my surprise, this is surprisingly easy. Next.js inherited the process.browser check from Webpack. So the following check is all we need to run client-only code:

if (process.browser) {
    // on browser
}

What about client-side components?

Since it is just React, surely we can do the same check before rendering the client-side only component right?

<div>
    { process.browser && <MyAwesomeComponent /> }
    <div>some other component</div>
</div>

Well, yes and no. If I try to run the code above, the app will function normally, but React will throw this error in the browser console Warning: Expected server HTML to contain a matching <div> in <div>.

Here's the complete warning:

Looking at the error itself, it is not too obvious what is going on at first. Don't worry, it is actually quite simple.

In the example above, I've told React not to render anything on the server-side, so it didn't. When the code got handed over to the browser, React suddenly recognised the fact something needs to be rendered, and it does just that on the client-side. Since React expects the rendered content to be identical on both the server-side and the client-side, I have now confused it by breaking that expectation. So it then returns the favour and displays a confusing error that had me scratching my head for hours.

The solutions

I managed to narrow down the problem to "something is causing React to go ape s**t during render and hydration". But I haven't gotten any further than that. Later, I spoke to a colleague and he pointed out how simple the solution was.

Solution 1 - suppressHydrationWarning

I felt pretty embarrassed, because all I had to do was add suppressHydrationWarning={true} attribute to the component with the issue.

<div suppressHydrationWarning={true}>
    { process.browser && <MyAwesomeComponent /> }
    <div>some other component</div>
</div>

It is worth noting that, this is merely hiding the error and the issue never went away. Make sure you are aware of the following if you decide to go with this approach:

It only works one level deep (not a problem in this case)
This should be avoided if at all possible, only to be used as last resort - me paraphrasing the React team
All the patching could impact site performance in some scenarios

Solution 2 - two-pass rendering

This is the recommended approach by the React team. As long as the DOM elements stay the same during the first pass render, it doesn't bug us with the error we saw above. To trigger a second pass, we will need to implement useEffect() React Hook and achieve a similar result as componentDidMount().

useEffect(() => {
    // update some client side state to say it is now safe to render the client-side only component
    this.state.renderClientSideComponent = true;
}, []);

Decision

In my team, we ended up going for the first solution since it is more straightforward, the client-side only component was a single level deep, and there wasn't any crazy logic which may cause performance issues.

Additional Reading

Here is the original React documentation if you want to read more about it.

Stop trying, you can't multitask

Mon, 29 Jun 2020 21:36:00 GMT

There are a lot of articles out there discussing why people need to focus instead of trying to multitask. At best, I have seen this being done from a biology point of view going into depth about how our brain works on a more fundamental level. I studied Computer Science at University, and this multitasking problem isn't new to me. Let's discuss how computer operating systems schedule tasks and what valuable lessons we can learn from it.

What is Process Scheduling?

A computer operating systems' fundamental role is to allow users to interact with the many applications on its platform. While it can offer computer programs many benefits such as a natural and consistent way to display something visually, but more importantly avoid headaches that come from interacting with hardware directly.

At the early days, there is only one core (which includes one thread) on a CPU. The operating system has no choice but to run one application at one time. This was fine when everyone was inputting into a terminal and looking at a black screen. When operating systems introduced the GUI (Graphical User Interface), this was no longer possible, people naturally people wanted to interact with several programs at the same time.

From a hardware point of view, it was not possible to achieve multitasking. So computer scientists decided to tackle it via the software side. Then process schedulers came and saved the day! Instead of the CPU working on a single program at one time until the user decided to quit it, the operating system would instruct it to split its processing power across all applications that are running. By jumping around and processing different programs for an allocated amount of time, it provided the illusion that we are running many programs at the same time.

However, it is not all rainbows and roses. The more operating system tries to switch between tasks, the less processing it could complete in the allocated time. In order to switch to another scheduled job, the current work progress has to be saved, and then new task data has to be loaded before it is possible to process anything. Both saving and loading take time, if the scheduled time for each task is too short, then the computer barely has enough time for anything else. Effectively, it will just be switching between tasks and doing zero processing, for a user it may feel like the computer has frozen and is unresponsive.

We are not so different from computers

The human brain works surprisingly, similar to a computer CPU. Our brain cannot process multiple pieces of data at the same time. The fact people think they could watching Netflix and doing work at the same time is also just an illusion. Behind the scenes, the brain is jumping between consuming video and audio from the TV screen and trying to think about the work problem they are trying to solve. It will not only exhaust you mentally very quickly, but it is also very ineffective.

Just like the process scheduling, our brain needs time to switch between different tasks. It is usually referred to as context switching overhead/cost. No matter how trivial another task is, breaking your current flow will require energy and time to bring the focus back to what you were doing. To put simply, this context switching creates processing overhead and is very expensive. If constant interrupted, you would probably find yourself getting very little done even though you have been staring at your work for a very long time. Also, if you do manage to get the job done, the quality of work produced is likely to be far from your best when you have to "mentally juggling" multiple things.

To be honest, I am not a biologist, so I won't be able to explain the intricacies of how the brain functions. But checkout this excellent video by BBC Ideas What multitasking does to your brain which explained it perfectly.

Here are a couple fun and easy to read books if you would like to learn more about the human brain.

(Please support me by using the affiliate book links 😁)

How to be mono-focused (one task at a time)

1. Record ideas that are bothering you

Write down all the ideas floating in your head on a piece paper and promise yourself that you will come back to it later. Knowing these ideas will be reviewed at some point shortly, this will be enough to calm your brain down, and it'll feel "safe" to stop thinking about them allowing you to concentrate on your task fully. If you want to learn more about this, it is a technique I learned from Getting Things Done book.

2. Avoid visual and audio distractions

I often listen to music when trying to focus, but you mustn't overshoot it. In general, I would stay away from music videos because the adverse effect is too obvious to ignore. Slower songs with more lucid lyrics are also bad for me because my brain will start picking up on the cues. Instrumental music is my goto music genre for better concentration.

Alternatively, if you don't want to listen to loud music, how about some peaceful rain noise? It keeps me calm and doesn't feed my brain too many triggers to go off on a tangent. I have been using rain.today a lot recently.

3. Avoid notifications

Silence every alerts on all devices when you are trying to focus. Don't let an unexpected message or notification from some random app break your previous focus and flow. If you struggle at the start, use a Pomodoro timer and follow its work and break schedule. Once you get comfortable with the idea, stop using a timer and just put your head down and work until you are mentally tired, then get up and walk around.

4. Noise cancellation headphones

Use a pair of noise cancellation headphones. I absolutely love using this type of headphones when I want to focus, and sometimes I would even use it for the noise cancellation without playing things. I find it that it could block most of the background hums, cracks, car driving past, distant chatter, etc.

5. Find people-free time

I am a night owl, so I focus very well late at night, this is probably because everyone else is asleep. It is a bit hard to do during the day or in a work setting, but try to find ways to find some quiet time by booking a small meeting room and work in there for a couple of hours if you have that luxury.

Given the current Covid-19 situation, the distraction probably comes from family and flatmates. If you are struggling, it might be beneficial to have a calm and friendly chat with them to see if they could give you some space and time during the day. It might involve not disturbing you during a specific time of the day unless it is very urgent.

Additional Reading

Scheduling is an interesting algorithmic problem, people have designed many different scheduling methods over the years, although none is best at everything. Some are better at bursts of computation, while others may consume less energy but translates to a sluggish performance. There are many schedulers out there and check it out here Process schedulers in operating system.

Set up domain forward from Cloudflare to Netlify

Mon, 15 Jun 2020 23:15:00 GMT

I recently migrated a few domains to Cloudflare. Sure enough, my domain set up is all over the place afterwards. I decided to clean everything up and set up everything again from scratch. This article explains how I was able to get Cloudflare to forward all request to Netlify.

There is a much easier way of achieving what I am trying to do here with Nameservers, however, with Cloudflare you need to be at least on a business subscription for a given domain before they open up that feature to you.

How much does it cost to be on their business subscription I hear you ask?

That's right, $200 per month!

Here I am with my blog that has no monetisation whatsoever. Clearly, it isn't the solution I can afford. Luckily, Cloudflare is "kind enough" to let you set up DNS records.

Add a custom domain on Netlify

Go to "Domain management" section on your Netlify app, then click on the big button that says "Add custom domain".

Next, it should take to you to the following page. Add in your domain name here. In this example, I'm using "beta.haodong.io", but I could be using "haodong.io" if I was setting it up for my main blog.

After clicking on "Verify" button, it will state that the domain already has an owner. This is expected since the domain is now Cloudflare. Just click on "Yes, add domain" and continue to the next step.

Now you can see "beta.haodong.io" has been added to the custom domain section, although it is greyed out.

Click on "Check DNS configuration" link, and then you should be presented with the following modal.

beta CNAME infallible-bhaskara-ddc8cd.netlify.app is the DNS record we need to add to our domain on Cloudflare.

Add DNS record on Cloudflare

Click on the "DNS" button in the top navigation bar.

Then you should see the current DNS records, I've greyed out mine just in case.

Click on "+ Add record" then you should see the following form dropping down. I have added the record Netlify told me to, and you should do the same since you use case might be slightly different.

Then wait for a few minutes (for me it was seconds), go to your domain and it should now work as expected.

Things to note

Remove domains from Netlify altogether if you previously used Netlify DNS for the domain. Otherwise, it will get very confusing when attempting to debug any problems. You will confuse Netlify as well since now it will try to generate SSL/TLS certificate for your domain.
As mentioned at the start of this article, you should try to use Servername and point to Netlify DNS preferably. The approach explained in this article is hackier, in my opinion. Netlify DNS has all the benefits of Cloudflare in terms of optimising assets, caching optimisation, post-processing, etc. If you are like me, who have different apps running on separate subdomains, then it is a lot easier to manage if Netlify manages the DNS.

Are you ready for Google Core Web Vitals?

Wed, 10 Jun 2020 18:00:00 GMT

Google recently announced the new Core Web Vitals, which will reshape the web soon. Although now they are only used as guidance metrics for websites to follow, it will massively impact how Google's Search Engine algorithm works and thus, how websites get ranked on Google search results next year. Is your company or your site ready for all this? In this article, we will go over what these metrics are and how it affects developers and businesses.

What is Core Web Vitals?

Google isn't happy the website standard today. It believes too many sites have poor user experience and performance. For good or for worse, Google has released the next step of their plan to drive the industry to up their game - Core Web Vitals.

I thought the name on its own is quite self-explanatory. It will act as the key vitals (metrics) in the future when deciding if a website is healthy or not by Google's standards.

Web Vitals is an initiative by Google to provide unified guidance for quality signals that are essential to delivering a great user experience on the web. They are a set of real-world, user-centred metrics that quantify key aspects of the user experience. They measure dimensions of web usability such as load time, interactivity, and the stability of content as it loads. - Google

It is worth mentioning that this isn't the first time Google has given websites a list of standards to follow. In the past, it has already punished sites that do not align with its vision in these areas:

Now, Google wants to add Core Web Vitals to the list.

To show they are serious; it even gave thresholds for metrics in terms of how performance will be measure:

Largest Contentful Paint (LCP): measures loading performance. To provide a good user experience, LCP should occur within 2.5 seconds of when the page first starts loading.
First Input Delay (FID): measures interactivity. To provide a good user experience, pages should have a FID of less than 100 milliseconds.
Cumulative Layout Shift (CLS): measures visual stability. To provide a good user experience, pages should maintain a CLS of less than 0.1.

The new list now looks like this:

When is it coming?

The metrics have already been released in May 2020. however, due to COVID-19, Google has decided to delay the rollout its Search ranking changes until next year.

There was no specific date given for the rollout next year. It did promise to send out notice at least six months before the release to give companies enough time to take actions.

Why should businesses care?

Businesses should be preparing for this now because Google will be incorporating Core Web Vitals into their search algorithm some time next year (could be as early as Jan 2021). As mentioned already, these metrics will contribute to how Google judge if a site should be ranked higher or lower on their search results page. Falling in ranking number will lead to businesses going over to competitors, or worse disappear from the customer entirely if the site drops off the first Google results page. If this is sounding familiar, that's because this is another way of saying these metrics will impact SEO in the future. If you would like to learn more, How far down the search engine results page will most people go is a good article explaining why.

For large sites, it will take time to bring improvements to all the pages. It may even require some significant amount of rework to make it happen, and it is hard to say as every site is different. But given the consequences, it is probably best to play it safe and get started on the work as soon as possible.

On a positive note, by following these metrics, it will provide users with a much more pleasant experience. It might even be possible that these changes will drive up traffic and increase conversion.

Why should developers care?

No doubt, these metrics will significantly impact how websites are designed and built once Google releases change that affects search rankings. As developers, we should be putting monitoring in place, auditing existing sites, coming up with plans to align with the metrics as soon as possible. The last thing we want to do is be forced to explain to our managers and their managers, why all of a sudden, the website ranking is tanking and the company is losing revenue.

As developers, it is our job to create high-quality websites anyway. It is a perfect opportunity for tech teams to explain to the business why it is worth spending time and energy on these aspects of the site that doesn't directly contribute to generating more revenue. I have seen many situations in the past where performance improvements labelled as "tech debts" get deprioritised when compared with new features that on the surface offers "more business value". Consider this as a rare chance for tech teams to speak up and have solid reasons behind your back.

The last benefit is that Google is baking in ways to measure these metrics in all of its tools. It should be easy for developers to identify problems and correct them with all the free tools available. These tools are straightforward to use, even for people who usually doesn't associate themselves with the performance and metrics side of web development, this should be relatively easy to pick up. It could even be a rare chance that open doors to a new career path or at least understand more about web performance.

How to prepare for the update?

Google has already provided a list of tools which can help you get started.

All of the above tools are useful for getting started with Core Web Vitals. I recommend taking a look at a free, open-source tool called Sitespeed. It offers a lot more monitoring and metrics out of the box than these tools. It also integrates well with some of Google's tools, such as Lighthouse and Google Page Speed Insight (GPSI). As a whole, it is a much more complete product to rely on to measure website performance. If you are interested in Sitespeed, check out this article I recently wrote Easily test your site for improvements with Sitespeed.io.

Summary

It is hard to argue against the list of metrics Google put out as Core Web Vitals, by enforcing these metrics it will massively benefit users. I am a little sad that a single company is driving all these changes, not to mention a business that gets most of its revenue from ads ($134.81 billion in 2019).

Additional Reading

Easily test your site for improvements with Sitespeed.io

Fri, 05 Jun 2020 18:23:00 GMT

Too often we forget to measure how our websites are doing, that is especially true for smaller sites like this blog. Not that it has never crossed my mind, but it just felt too much hassle to do. Recently, I found about Sitespeed.io, and it has changed everything! It is a collection of open-source tools are makes monitoring website performance super easy. In this article, I'll walk through how we can utilise it in our workflow.

Why do we care about performance metrics?

Think of the metrics as a way of quantifying a vague statement like "a good website". Achieving high scores in all the metrics doesn't mean your website is perfect (since there are ways to cheat). Instead, it should be the minimum baseline that we aim for.

It probably doesn't make sense to be concerned about performance metrics for all the websites. For example, if you have a personal project that you don't care about, then it is probably not worth spending time to improve the metrics. For a website that has more traffic or is trying to increase its user base, then these metrics must be taken seriously.

As mentioned already, achieving high scores doesn't mean a website will lead to success. If your goal is to do well in terms of SEO ranking (how Google search ranks your page on their results page) or just simply offer a better user experience, then it would be silly to ignore these metrics.

Sitespeed

I ran Sitespeed against this site, and here are the results.

As you can see, it gives you a lot of insights into what is happening behind the scenes. If this is still not enough, then see the section below on how to extend Sitespeed's functionality through plugins.

Collecting metrics

There are two ways to run Sitespeed on any page. It is possible to run it using Docker or just through NPM.

Note: I prefer not locking to any particular versions since I just want to get a one-time measurement of the metrics. So the code examples below are all using the latest version each time you run it. However, if you do wish to monitor the metrics over a more extended period, then it might be best to stick with a particular version to avoid future breaking changes.

Collect metrics via Docker

Here is the command to run Sitespeed on this blog:

docker run --rm -v "$(pwd)":/sitespeed.io sitespeedio/sitespeed.io:latest https://haodong.io

Collect metrics via NPM package

If you prefer the NPM way, then it is even simpler.

npx sitespeed.io https://haodong.io

Reading the results

After running Sitespeed it will create a folder of artefacts upon completion in the folder where you ran the command. In my case, the folder is called sitespeed-result. Then simply open index.html in the browser to see the results.

The most useful page is probably the Summary page. If you wish to see more details around a particular metric, then the other pages will offer both more information and in some cases suggestions on how to improve.

Note: It is possible to run Sitespeed on multiple pages, don't worry the folder sitespeed-result will not be overwritten. Instead, the results of each page will be inside the sitespeed-result/pages folder. It is also fine to run Sitespeed multiple times on the same page, results of each run is timestamped so nothing will be overwritten this way.

Sitespeed Plugins

It is even possible to add in plugins to get even more out of it. Here is an example of me collecting Lighthouse metrics.

The command to add Lighthouse plugin:

docker run --shm-size=1g --rm -v "$(pwd)":/sitespeed.io sitespeedio/sitespeed.io:latest-plus1 https://haodong.io --plugins.add analysisstorer --plugins.add /lighthouse

However, I prefer to use the pre-built Docker container that includes both Lighthouse and the Google PageSpeed Insights plugin inside already.

docker run --shm-size=1g --rm -v "$(pwd)":/sitespeed.io sitespeedio/sitespeed.io:latest-plus1 https://haodong.io

Note: Documentation doesn't say how to run plugins with NPM, so I'd recommend going with Docker instead.

Final words

Hopefully, this article showcased how easy it is to get some very detailed metrics on your site. The only downside is that we need to run this command on our pages every-so-often to monitor it, which could be inconvenient for some.

Luckily it is very straightforward to automate this with Netlify and Github webhooks. E.g. on merge to master, kick off a build. I will be posting a guide shortly, keep an eye on this space!

A non-technical introduction to git

Thu, 04 Jun 2020 05:10:00 GMT

Git is often explained in a way that is too complex for the non-technical people to grasp its usefulness or the purpose of adopting it in the business. In my opinion, it is such a simple (yet powerful) concept that anyone should be able to understand easily with some simple explaining.

Git is one of the pillars of software development today. It enables software engineers to build software one step at a time, and in times of need, it offers engineers the most powerful tool in the world - the ability to restart from a last known good state.

Git is a free and open-source distributed version control system designed to handle everything from small to very large projects with speed and efficiency. - Git

Ok, so Git is a version control system for software development. But what is version control, and why is this useful?

Why is version control useful

To answer these questions, let's look at a very well established product that most people probably have heard of - Google Doc. When you are working on a Google Doc, did you ever notice this link on any non-empty Google Doc?

If we click on this link, then we are presented with all the edits that we have done (see screenshot below).

Notice the right-hand panel that says "version history". Under it, it shows each edit we ever made in this document as a separate version. It is handy for a few different reasons:

All changes are recorded in a time series, from oldest to newest. With this feature, we can go back all the way to start of the document when it was blank. Think of it as the regular "undo" action on steroids.
We can browse and replay changes like it was a movie, which is extremely useful when we would like to investigate something at a later point. e.g. for understanding why a paragraph changed to the way it is.
It allows us to track who edited what and when. If there is an issue, then we will know who to walk to about it.

Git is doing the same thing for software engineers. It provides all of the above benefits to software development and more.

Setting up Gatsby incremental build on Netlify for existing project

Tue, 02 Jun 2020 18:28:00 GMT

Gatsby incremental build has been such a highly anticipated feature. Without it, using Gatsby in large projects is unwise due to the poor scaling that comes with the increasing amount of content. Even for this small blog site, I've been feeling the pain, every time i need to deploy a minor fix. The upgrade process of anything is never smooth, so here are the issues I encountered. Maybe this will help some of you out there!

Step 1 - Gatsby version bump

To take advantage of the new Gatsby feature, you need to be on v2.20.4 or higher. After checking the status of my repository with ncu (npm-check-updates), I decided to bring everything up-to-date.

It took less than five lines to address breaking changes in the entire repository. react-helmet was the only repo's breaking change that affected the code.

Step 2 - Enable Netlify Build Plugins

The Netlify plugins are currently in beta, but from my experience, everything we are trying to do here is very stable. Click here to enable Netlify Build Plugins.

Step 3 - Add gatsby-cache-netlify-build Plugin

The new incremental build feature is based on Gatsby's new ability to cache public and .cache folders for future builds. Luckily, we only need to add two lines of text into Netlify config to take advantage of this.

You could do this in two ways, one through the UI or use the Netlify config file. I recommend the config file because it is much simpler to do and more straightforward for me to explain.

If you haven't done it already, add netlify.toml to the root of your repository. Then paste in the following snippet, and then we are done!

[[plugins]]
  package = "netlify-plugin-gatsby-cache"

If you are using other plugins, make sure you still paste in the [[plugins]]. See the example below for more information.

[[plugins]]
  package = "an-awesome-plugin"

[[plugins]]
  package = "netlify-plugin-gatsby-cache"

Step 4 - Use the new gatsby build command

Use the GATSBY_EXPERIMENTAL_PAGE_BUILD_ON_DATA_CHANGES flag to let Gatsby know to build incrementally. Change the default build command to:

 # change this original command
"build": "gatsby build"

# to
"build": "GATSBY_EXPERIMENTAL_PAGE_BUILD_ON_DATA_CHANGES=true gatsby build"

Note: There is also an optional --log-pages flag for seeing extra logs.

Step 5 - Check Node version

One of the issues I experienced is that the new Gatsby version requires Node v10 LTS version. Building with Node v8.x.x will result in the following error on Netlify.

Node.js version 8 reached End-of-life status on December 31, 2019. Many of Gatsby’s dependencies are updating to Node.js version 10 and above. Gatsby must also update in order to deliver new features and bug fixes more quickly.

Add a .nvmrc file into root of the repository and paste in the following to ensure we always use Node v10 (LTS) to build on Netlify.

The end

That is it, push all your changes and see the incremental build on Netlify :)

Update

After updating all the dependencies and turning on incremental builds, I actually saw an increase in build time on average. This might be due to all the dependencies I updated at the same time, let me know in the comments if you experience the same problem. Or am I missing something here?

Build time before

Build time after

Tips on how to start and finish personal projects

Tue, 17 Mar 2020 00:04:00 GMT

2020 has been chaotic so far, I hope everything returns to normal soon and best wishes for everyone. On the bright side, now the UK government has recommended people to avoid going to mass gatherings and businesses have told their employees to work remote. It is a perfect time to hammer out some personal projects. I have always struggled to work on personal projects on the side over the years, however, this year I’ve managed to complete 2 projects in 2 months. I know this is a common problem, so I’d love to share my experiences in this post.

1. Find motivation

It is too tempting to watch something on TV or play games after a day's work. After all, these entertainments are distractions specially designed and engineered to hook us in. I find it genuinely difficult to master enough motivation to open up my laptop and code when it is so much easier to not be productive. From my own experience, only meaningful ideas will generate enough desire and motivation to work on it.

So what is a meaningful project?

I believe meaningful comes in two forms:

It is fun for you. Yep, even more fun than jumping into your favourite game or watch that TV show someone recommended to you.
There are consequences if you don’t finish. E.g. someone is paying you for it, or it is an app you desperately need for yourself (I'll talk more about it later).

I've tried to work on projects that I don't care much about, but started on them anyway because I felt other people might use it. Often than not, these projects are abandoned very quickly. Why? Because it is not meaningful enough for me. A promise of someone in the future may use it is all smoke and mirrors until it happens, and the unknown could drain away any motivation and interest when the project hits a blocker. These blockers could come in many forms, I won't go into many details here.

In my opinion, it is much better to finish something that probably no one will ever use, compared to building a “sound” project but never end up finishing it.

2. Focus

If you are anything like me, then you also have the "I have too many ideas" problem. This could be switching to work on a new project halfway, or just increasing scope in the current project like changing frameworks and libraries. I am completely aware that I’m my demon when it comes to getting something done. So I’m always in search of productivity tools, hoping it could be the thing I'm missing from my workflow and that it will get me to finish my side projects.

However, the real problem often isn’t the underlying technology or tool. Instead, it’s the mindset of taking the side project serious enough to see it through. Mindset is such a blur, vague and BS sounding word that is overused in self-help contents. I hate to say it, but it did capture my problem. I tried to use scrum methodology in my projects, or kanban boards, but it didn’t matter what I used the result was no different. I have been reading the Shape Up book by Ryan Singer, which talks about how to build the right product and ship it too.

My key takeaways from the book are:

Work on a personal project in 6 weeks cycles. This way you could technically build 8 projects in a year with time to spare. In 6 weeks, there is enough time to work on a project that is not too simplistic, but not long enough to get bored with it.
Must deliver the planned scope at the end of the 6 weeks. This assumes that you will do whatever it takes to finish the goal set. This means working late into the evenings or even whole weekends had to. It feels like a terrifying commitment, so make sure the scope is realistic and always go for less scope if unsure. After all, it is better to have something basic working than nothing at all.
Use both words and sketches to define the scope and the solution. Don’t waste time building wireframes, or worse, pixel-perfect designs for personal projects. Remember designs don’t bring value on their own if the goal is to release something at the end of 6 weeks. Sketches should be enough to give a vague clarification which is enough at this stage. Don’t forget to describe in words the constraints and scope of this period.

3. Discover Project Ideas

I have plenty of terrible and weird ideas, so I never really struggle with this. But many people I speak to don’t have anything they want to work on. So here are a few of my favourite questions which hopefully will help you to find some meaningful side project ideas:

Is there anything that bothers you? That you just wished there was a simpler or better way to do it?
Is there something you know people wanted but doesn’t exist? Who would be wanting to try it out on day one?
What would be something cool to build which you’d be proud to show off to others?
What could you create to make your personal or work life easier?

There are lots of resources online which can probably help you to find the right project idea better than my questions here. It doesn’t matter how or where you find the inspiration, it doesn’t even matter if it is not original. One of the two projects I completed this year is a Todo app, there are already countless of Todo apps out there. But that doesn’t matter, I wanted a Todo app that I could adjust to how I see fit, whereas other solutions are less flexible. So the important thing is to find something meaningful to you.

4. Improve ideas by adding notes over time

Try to record ideas that come to you in a central place. This way you could always look up the ideas when the time comes like browsing a catalogue, instead of pulling out hairs trying to think of something on the spot.

Another advantage of keeping everything documented is that even if you don't work on a project idea right now, it is always good to add in notes or thoughts over time. For example, I had lots of notes written up for the Todo app over time, ranging from sketches, features, milestones, etc.

5. Choose what to work on

Okay, let’s say you were able to generate many ideas and stored them in one place. Which one should you work on next? It is hard to give strict guidelines on how to pick the ideas, but I always remind myself to pick something I am motivated enough to work on for the next 6 weeks. It doesn’t matter if the idea is good or not, just make sure you have enough motivation to finish it.

My final advice here is to always finish a product. Always try to finish a project, it is a good mental boost, gives confidence and increases motivation to tackle the next project. No matter how pointless the idea may seem, there are always value to be found in doing the work. First, it proves you were able to successfully build something, perhaps something to add to your work portfolio? Secondly, don’t forget that you’ve learned, practised and enhanced your skills in the process. Maybe next time when you need to implement something similar, you’ll not only have more experience but also a source of reference. Lastly, you get to tick something off your ideas list instead. The feeling of marking something as complete is awesome!

6. Set yourself up for success

Firstly, no matter what you do, try not to work on multiple projects at the same time. Stick to one thing and get it done, then work on the next idea. Context switching is expensive and more importantly, it confuses when you finally decide to do something productive.

It is hard enough for me to find the motivation to open my laptop in the first place. It will just make the problem worse if I had to decide between multiple projects which one I want to tackle at the same time. The thought about justifying what to work on is enough to drain out my energy and turn me towards some kind of entertainment that requires less processing power e.g. watching TV.

The goal here is to make working on side project the path of least resistance. Prepare your desk where you normally work, clean it up, put your laptop in the middle ready to rock. This way all you have to do when you come back from work is sit down and then you are off.

7. Reduce Friction & Cost

When working on personal projects or side projects, you are likely a one-man-army. Where you wear multiple hats, Design, Developer, Quality Assurance and Product Manager. Try to make use of all the free (tier) tools out there, avoid reinventing the wheel where ever possible.

There are lots good CI/CD tools out there, take your pick, just make sure if something could be automated then do it. Of course don't spend weeks on setting up the best pipeline ever, just set up something simple and usable.

In recent years, there's a massive push for open source. Many tools and services will let people use their full product for free for open source projects. There are even more tools to pick from if consider paid services which gives a free tier. The free tier is often more than enough for side projects.

Some other tips:

Don't host your apps, stick with static web apps when possible, avoid node apps since they require a server to run.
Query database from client-side at the start, use something like FaunaDB or Firebase FireStore.
Use a headless CMS if there are lots of content.

Maybe some of these options don’t offer the best scaling, may lock you into a platform, or might have some security concerns. I’d argue it is best to worry about it later, have something working first will motivate you to come back and improve it. Whereas if you spend 4 weeks building a secure auth solution, it is highly probable that you’d give up, because let’s face it, that is not what most people find exciting. Of source, this doesn’t mean don’t build things properly, make sure things are well-rounded before a proper release.

For more information on tools and services, check out a previous post on Awesome tools to use in personal projects.

My two projects

For both inspirations' sake and also to prove I didn't just make up my two side projects in two months stats, here they are:

Slash-Dash: a basic word-based RPG game built with React, xState and Bootstrap CSS.
Organise: yet another todo app built with Svelte, xState, Firebase FireStore, Serina (one of my open-source projects), and some Bootstrap CSS. It also supports PWA.

Slash-Dash was meaningful to me since it was quite fun to build, I tried to recreate some aspects of a typical RPG game (e.g. kill monsters, get drops, change equipment, etc). I showed it off to a few people and it was nice to get feedback and boosted my motivation to do more work on it.

Organise todo app is something I had been planning to create in years. This must be the 6th attempts to create a todo app. This project is meaningful because I wanted something more customised to solve my challenges. To make sure it is clever enough, I even went out of my way and created a natural language parser for time and dates, see Serina on NPM.

So what did I do differently this time compared to the last 5 times?

The first thing I did was to create something extremely basic very quickly. Then I fully switched to this app, even before I implemented edit and remove functionality. Now I've locked myself into my app, if I wanted something, I'd have to go and build it otherwise I just have to deal with the pain point again and again. Needless to say, it was super frustrating at the start when the app was unusable, but it motivated me to add the missing features and fix annoying bugs.

How to configure user snippets in VS Code

Wed, 11 Mar 2020 22:45:00 GMT

VS Code often has you covered when it comes it auto-suggestions and adding in snippets of code. But there will be times where something that is often used by you but isn't provided by VS Code, this is a great opportunity to reach out to the User Snippets feature in VS Code. Where you could add code snippets templates that make it easier to enter repeating code patterns, such as loops or conditional-statements. It is super easy to set up, but there are a few simple things to be aware of to get the best experience with it. We'll talk about these in this blog post.

Create Your Snippets

There is a section on this on the official documentation.

Here's a snippet that I use all the time:

// in file 'Code/User/snippets/typescript.json'

{
		"Unit Test Describe": {
		    "prefix": "desc",
		    "body": [
		        "describe('$1', () => {",
		        "\\t",
		        "});"
		    ],
		    "description": "Write out describe block"
		}
}

The whole config is a JSON file, so make sure whatever the file remains as valid JSON for it to work correctly.

For each snipper you could give it a name, in this case I called this snippet "Unit Test Describe".

Then inside the block of each snippet you could define 3 properties:

prefix - keyword(s) you need to type to see it in the auto-suggest dropdown body - what will be output once you select the snippet from auto-suggest dropdown description - optional

I picked desc for my snippet since it is easier than typing out description.

As for body, each line in the array represents a separate line in the editor. Notice how I was even able to add in the correct indentation with \t. The other interesting thing is the use of $1 , this is where the cursor will go after the snippet is added.

So in summary, each time I type desc then selecting it in the auto-suggestions, I will get the following snippet added. the focus is automatically inserted in between the single quotes. This and a few other user snippets makes writing unit tests so much faster.

describe('', () => {
	
});

Improve Snippet Priority

Depending on what other VS Code plugins are installed, user snippet may not show up at the top of the suggestions as you code. If this is happening to you, then you could get user snippets to show up at the top of the list by adding the following settings option.

// in file 'Code/User/settings.json'

"editor.snippetSuggestions": "top"

Here's what the option looks like in the UI.

Snippet Scope

Again, this section is also covered by the documentation.

My suggestion here is to scope it to the language, as this will carry through different projects. If you use one project for each repository and works on many repositories at the same time, scoping to language makes much more sense.

Advanced Snippet Configs

I don't like using custom snippets heavily. I believe in However, if you prefer to explore the possibilities of user snippets, have a look at the Syntax section in the documentation. It supports placeholders, choice, variables, transforms, binding with shortcuts, etc.

Sharing Snippets

Sometimes team members find it useful to share snippets with each other. Textmate is the tool for that!

Why I would avoid Sequelize in the future

Sun, 08 Mar 2020 13:14:00 GMT

Following on from a previous post on Database Migration: Knex vs TypeORM vs Sequelize, I’d like to share my experience regarding Sequelize after using it in a commercial project. TLDR, it wasn't as good as I hoped and for the love of God don’t use it with Typescript! Read on to see why.

How work was carried out

Due to how our project work was broke down and prioritised, I ended up writing the migration scripts first and integrated with codebase later.

Here's the order of tasks:

Database table & data migration
Integrated with CI/CD pipeline
Creating data models
Linking up codebase to DB through Sequelize

In hindsight, if it might have worked better if we completed Database table & data migration last. Not that it would have made much difference in terms of implementation approach, but for sure it would have highlighted some issues much earlier on.

Writing migration scripts

When I wrote the initial migration scripts it was straight forward. The documentation wasn’t perfect, but with some light Googling, I was able to get it done. At this stage I haven’t spotted anything off-putting about Sequelize yet.

Side note: It might be good to clarify how to add static data (record with fixed id values) into the database. It could either be done as DB migration or DB data seed. Just want to mention it here as we debated the two approaches. Just FYI, we have gone with the DB migration approach at the end.

Migration pipeline

This part is not too relevant to the title of the blog, feel free to skip to the next section.

The initial idea is to run the DB migration scripts for each commit pushed to master. This way we can version control our DB changes which is the main reason for using a DB migration tool. Therefore we could have a simple rollback strategy in case we find a problem after a deployment. The project's pipeline is fairly complex, this is partly due to infrastructure is in the process of being migrated to a more modern setup.

Although Sequelize provides a rollback solution natively, we still experienced problems with how to actually trigger it. The environment setup is super strict and for good reasons e.g. security. However, this means developers can't easily access the Database (not even on QA environment), so don't even ask me about Production.

Given these strict environment configurations, how coule we rollback migration changes?

We had a few ideas as a team:

Ideal case: the CI pipeline would do Database migration first when something is pushed to master. We'd also have a final step of the CI to run rollback if the build isn't successful. Unfortunately, making such a change wouldn't be easy or straightforward as this will require a lot of CI rework. Due to the fact no such setup has been done previously and the pipeline inherits from shared Jenkins scripts.
Running rollback manually: because of strict envrionment setup, it wouldn't be easy to convince people to give the whole team access to Production. Personally, I think this is still useful to have in case of emergencies, even if there was a way to do auto rollback in CI.

Unfortunately, we are still trying to find a nice solution for this. Maybe I'll post an update if we find a good solution.

Creating Models

This is the first time I started using Sequelize with Typescript, and this is where the pain started.

Lack of Typescript related documentation

Right at the start, I was hit with an issue. The Sequelize documentation for Typescript is lacking to say the least. Here is the official documentation and see for yourself. There is barely enough content to fit on 1 page of A4.

Unclean approach

Even if I look past the lack of documentation issue, there are other problesm.

If you read the example provided on the doc, the way the models are declared and used isn't very clean in my opinion. The example shows that, to initialise a model Sequelize needs to establish a connection to the database and then passed into each model.

const sequelize = new Sequelize('mysql://root:asd123@localhost:3306/mydb');

class Project extends Model {
  public id!: number;
  public ownerId!: number;
  public name!: string;

  public readonly createdAt!: Date;
  public readonly updatedAt!: Date;
}

Project.init({
  id: {
    type: DataTypes.INTEGER.UNSIGNED,
    autoIncrement: true,
    primaryKey: true,
  },
  ownerId: {
    type: DataTypes.INTEGER.UNSIGNED,
    allowNull: false,
  },
  name: {
    type: new DataTypes.STRING(128),
    allowNull: false,
  }
}, {
  sequelize,
  tableName: 'projects',
});

Notice how sequelize is passed to the init() function after a connection has been established. The model itself is a Typescript interface, table column types and everything else is defined during the init() function, what?!

When I was working on this, I really struggled to see how this was a good approach to both declaring models and using them like this. The fact there is a dependency between model initilisation and sequelize having established a DB connect feels odd.

After I have spent more time with it, I can see how I could work around this issue. I also want to clarify, there isn't as much of a problem with the tool, instead, I feel like the documentation itself could be better. For example, if the example read something like the follow it would have made a lot more sense for first timers like me.

// server.ts
import Project, { projectModelConfig } from './models/project.model.ts';

const sequelize = new Sequelize('mysql://root:asd123@localhost:3306/mydb');
Project.init(projectModelConfig, {
  sequelize,
  tableName: 'projects',
});

// project.model.ts

export const projectModelConfig = {
  id: {
    type: DataTypes.INTEGER.UNSIGNED,
    autoIncrement: true,
    primaryKey: true,
  },
  ownerId: {
    type: DataTypes.INTEGER.UNSIGNED,
    allowNull: false,
  },
  name: {
    type: new DataTypes.STRING(128),
    allowNull: false,
  }
};

export default class Project extends Model {
  public id!: number;
  public ownerId!: number;
  public name!: string;

  public readonly createdAt!: Date;
  public readonly updatedAt!: Date;
}

In the end, I decided to use sequelize-typescript library instead, mainly because their syntax seemed a lot cleaner and more maintainable.

// server.ts
import Project, { projectModelConfig } from './models/project.model.ts';

const sequelize = new Sequelize({
    ...
    models: [__dirname + '/**/*.model.ts']
});

// project.model.ts

import { Table, Column, Model } from 'sequelize-typescript';
 
@Table
class Project extends Model<Project> {
 
  @Column({
    type: DataTypes.INTEGER.UNSIGNED,
    autoIncrement: true,
    primaryKey: true,
  })
	id: number;

  @ForeignKey(() => User)
  @Column
  ownerId: number;

	@Column({
		type: new DataTypes.STRING(128),
    allowNull: false,
	})
  name: string;
}

Issue 3 - Decorators

By choosing sequelize-typescript I created another problem for myself. It uses decorators by default as you can see from the example above, which means I also had to turn on experimental decorators feature in Typescript. It wasn't until later I found out that it caused other issues when we tried to write unit tests.

Linking up Sequelize to Codebase

Code

This process was painful, partly because we have a strict typing configuration in our project, but the fact Sequelize and Typescript doesn't work well together is definately a contributing factor as well.

I'll just come out and say it Sequelize with Typescript is just confusing, even after adopting sequelize-typescript this statment still holds true. I encountered several problems caused by Typescript types, but I'll just focus on the most memorable one here.

I wanted to inject Sequelize into a function to make testing a bit easier. But I simply couldn't get the type to work, I was really confused. After trial and error for ages, here is what I found out.

Have a look at the code example below. The fact Sequelize (the one we import) is not Sequelize type blows my mind. But an instance of Sequelize (connectedSequelize in the example) can be assigned the Sequelize type. Again, what?!

import { Sequelize } from 'sequelize-typescript';

const connectedSequelize = new Sequelize(...);

// "Sequelize" is NOT Sequelize type
// "connectedSequelize" is Sequelize type

So basically, I can't give Sequelize (the function param) a type because there is no type for it unless we create an instance of it. In the end, I had to leave the type to any and I feel like a horrible human being for doing it.

Unit Testing

There is litterally no documentation around how to do this properly nor are there good example shared online.

Unit testing, in general, is messy for Sequelize. Paired it with Typescript is almost asking for trouble. I won't bore you with all the problems I experienced in too much depth, so here are some key ones I experienced:

There are Decorators errors when trying to create a new instance of a Sequelize model in test. I ended up mocking all the models in functions that require them.
Get error complaining the model isn't initialised by Sequelize. So I added the models without any test, probably something for me to revisit now I have some ideas.
Sequelize query returns a list of data in its own prototype, so the type doesn't match with what you'd expect the results to be. I had to resolve to any type in many instances when trying to mock something.

As you can see I didn't have a great deal of fun testing Sequelize code, especially considering this is with dependency injection (just imagine without). Normally, this would make the unit test extremely easy to write, that is not the case here.

Summary

We picked Sequelize as our Database Migration tool for the same reasons highlighted in the post (see first paragragh). At the time, it seemed more mature than TypeORM and more modern than Knex. Unfortunately, as I found out the hard way that it isn’t modern enough.

If you plan to use Sequelize in a pure JavaScript project, it will probably serve you fine. However, if there is a chance to be adding Typescript into your backend stack, then I’d recommend trying out TypeORM instead.

Other Thoughts

When I was talking to our DevOps team, I was introduced to the concept of Event Sourcing. Now looking back, maybe we could have approached our data differently and then there might not be a need for relational data structure. Thus avoiding using database migration tools altogether.

Here is some top Google suggested reading materials on Event Sourcing:

Configuring custom CDN caching with AWS S3 and CloudFront

Wed, 29 Jan 2020 08:29:00 GMT

I recently encountered a problem where we wanted to move some assets to AWS S3 and have AWS CloudFront CDN in front of it. The great benefit of having a CDN is to reduce cost and improve performance. The get the most out of this configuration we want to have the following set up:

Client (Browser) → (no cache) → CloudFront → (cache) → S3

Note that this set up might not be right for everyone, it makes sense for us because:

Normally, it is common to use hashed file names to indicate to browsers that they now need to use the new file instead of loading it from browser cache. But we want to avoid this approach since it would require us releasing many apps that consumes this assets.
The benefit this brings is that we can control when to invoke cache (e.g. when we release a new version of the asset), as suppose to users caching files on their browser (e.g. if we release a new version since the browser believes it has the right version it won't try to download the latest version.

Key technology

S3 - is used for storing the files (AWS call them as Objects) in folders (AWS call them Buckets).
CloudFront - a CDN gateway (AWS calls it a Distribution) that we set up to just deal with caching, which would utilise all the different server farms Amazon has in different parts of the world. This is also a way to defend against DDOS attacks, since the requests never makes it past the CDN layer. It is also cheaper for this exact reason because there is no computation when requests are served by cache. Finally, this is also an opportunity to utilise AWS’s Lambda@Edge to implement logic on CDN level.

Creating a S3 bucket

First we created a standard S3 bucket with mostly default settings. This is relatively easy to do, and AWS already has a decent guide on how to do this.

Creating CloudFront Distribution

Then we set up a CloudFront Distribution with these changes:

Make sure that it is not possible to access Bucket files directly via URL, and to grant the Distribution access to the Bucket (so it can cache and serve it). Identify could be new or Existing depending what makes more sense.

Then change some of protocol to make sure we always to try serve with HTTPS. For Object Caching we select cusomize and give the TTL appropriate values.

For more information about the full set up, here is an official AWS YouTube video explaining in detail how to create a new distribution: https://www.youtube.com/watch?v=KIltfPRpTi4

Testing Cache is working correctly

This AWS documentation suggest that it is best to use TTL option in CloudFront Distribution (see above screenshot) to control the cache here: AWS CloudFront → (cache) → AWS S3. Then use HTTP headers to disable cache between: Client → (no cache) → CloudFront.

Here’s the quote taken from the link above:

For web distributions, you can use Cache-Control or Expires headers, and CloudFront minimum, maximum, and default TTL values to control the amount of time in seconds that CloudFront keeps an object in the cache before forwarding another request to the origin. Headers values also determine how long a browser keeps an object in the cache before forwarding another request to CloudFront.

Here explains how the headers and TTL works together. The thing to note here is that we need to use the minimum TTL in conjunction with Cache-Control: no-cache header.

With the above understanding, then we tested our assumptions with the following test:

Upload a file.js (v1) to S3 bucket and then set the minimum TTL on the Distribution to 1800 seconds (30 mins).
Then try to request the file via the browser (goes to CloudFront) which returned file.js (v1) as expected.
We then uploaded a file.js (v2) to the same S3 bucket which replaced file.js (v1).
We tried to request file via the browser and got back file.js (v1) still, this means the minimum TTL is working. So this part is correct: AWS CloudFront → (cache) → AWS S3.
We then busted the cache on the Distribution.
We request the file again via the browser and got back file.js (v2), this again proves that CloudFront is the one providing the cache and cache isn’t coming from the browser.

Here's a simpler version in the form of a sketch to illustrate the above:

The browser header is set to no-cache as well in this case, so we can be sure that we are just testing the CloudFront.

Invalidate Cache

As mentioned above, we want to be able to bust the cache after we re-release the asset. It turns out this is quite easy to do with the AWS console.

It is also possible to do it through AWS CLI, which we can use to invoke cache as part of the CI pipeline after we released a new version (see official doc).

Custom domain name

If you don't like using the default CloudFront url (e.g. https://wiebfpweufpwubfe.cloudfront.net), it is very simple to update the CloudFront Distribution to use a custom domain.

Database Migration: Knex vs TypeORM vs Sequelize

Tue, 28 Jan 2020 18:51:00 GMT

I recently joined a full-stack team, and we decided to shift a prototype app into something that scales better in the future. There were a few decisions made about how we might achieve this goal, one of which is to introduce database migration to our CI/CD pipeline. This blog post will cover the key findings and the final implementation around it.

It is worth noting that at this point, the decision has already been made to go with AWS's Aurora relational database. This is due to there being a lot of relations between data so picking a relational database made sense, and picked Aurora to reduce the amount of maintenance. Although it is possible to hack around and use a document based database instead, but it just means more data duplication and introduces more maintenance headaches.

What is Database Migration?

Database schema changes is typically a concern in relational DBMS (Database Management Systems), document based DBMS doesn’t have this problem. Why? Relational DBMS often rely on established schema to enforce the relationships between records, so it is crucial to maintain these relationships. Document based DBMS often has no schema (at least not enforced by the DBMS itself), so as developers we could store anything we like in any records e.g. I could have a record storing user data, and the next record could be about different types of potatoes.

Making changes to a database is probably unavoidable, sooner or later we will find ourselves in situations where we need to make some modifications. If we add a new field, then what should the default value be? If no default value is provided then the old records will have no data for that field, the application assuming the data might struggle to coup with this edge case since the app might always expect some data to be returned. Or worse, we might need to add or delete tables to a production database. What is the best way to make these changes, what should we do if something goes wrong?

These questions are also solved by a database migration tool. A database migration tool should be able to iteratively introduce changes by based on its versioning system e.g. it might decide to do nothing if it is already on the latest version, but if it find it self behind in version it could then run the versions that’s missing and introduce those changes to the relevant database.

Investigation

Having decided we wanted to implement DB migrations and settled on the Database schema as a team, the next step is to investigate what might be the best tool for database migrations. Our API is using Node, so the solution we are looking for here is a JavaScript based tool.

Expectations

In terms of expectations of this spike, we hoped to find something that:

Support adding, editing and deleting tables and column

e.g. we might want to add new tables in the future or modify fields in a given table

Support inserting, editing and deleting records

e.g. similarly, we have seeded data that we might want to modify later on

Support seeding data

e.g. put add in static data such as brands, cookie categories, cookies and etc

Support versioned incremental migration changes (automatically)

e.g. we should be able to add new changes into a version controlled file, this should then be picked up and only new changes should be picked up and ran by the migration tool

Support migration reverts (in the event of causing issues on production)

e.g. the tool should be able to take revert migration logic, which means we can safely and easily undo our migration if the changes causes issue on production

Different choices

After doing a bit of research online, these seems to be the top choices right now:

Knex - already using this in our app
TypeORM - is built for TypeScript and may end up being a better choice than Knex
Sequelize seems to be very popular as well

It is worth noting that all of the above supports TypeScript according to their documentation pages.

Knex v.s. TypeORM v.s. Sequelize

Expectations

For TypeORM, there’s this nice unofficial plugin we can use TypeORM Seeding

Codebase

All three libraries are actively maintained, with frequent commits with the last 30 days and before that.

Here’s a breakdown of the numbers of issues and PRs on each repo:

Documentation

I went through the documentation all 3 libraries. Knex’s documentation is detailed but very difficult to read, both in terms of how the content is structured and presented. I found that everything I needed are documented, just not in an ideal way. TypeORM is on the other side of the scale, it is very well presented and has plenty of examples. I was able to find what I needed very quickly, and everything seems intuitive enough. However, there seems to be a lack of explanation for some of the sections. Often the documentation involves a brief statement, then shows an example and that’s it. Finally, Sequelize is somewhere in between. It seems to have the best documentation in terms of detail, its presentation isn’t as good as TypeORM but definitely better than Knex.

Testability

All three options has similar levels of testability. It is possible to extract out migration logic to functions and then unit testing those functions. The only exception is that due to TypeORM doesn’t have official seeding functionality, unit testing it might be slightly complicated.

Typescript Support

On paper all three tools support Typescript.

Knex admits that Typescript isn’t first class citizen.

However it is to be noted that TypeScript support is currently best-effort. Knex has a very flexible API and not all usage patterns can be type-checked and in most such cases we err on the side of flexibility. In particular, lack of type errors doesn't currently guarantee that the generated queries will be correct and therefore writing tests for them is recommended even if you are using TypeScript.

Sequelize also mentions that Typescript probably will do very little when they are paired together.

As Sequelize heavily relies on runtime property assignments, TypeScript won't be very useful out of the box. A decent amount of manual type declarations are needed to make models workable.

TypeORM just mentions the fact they support Typescript. After some deeper digging around, I found the TypeORM Typescript example. I could really find cases where TypeORM really made use of Typescript, see the code snippets below:

// typescript-example/src/index.ts
  
import {createConnection} from "typeorm";
import {Post} from "./entity/Post";
import {Category} from "./entity/Category";

// connection settings are in the "ormconfig.json" file
createConnection().then(async connection => {

    const category1 = new Category();
    category1.name = "TypeScript";
    await connection.manager.save(category1);

    const category2 = new Category();
    category2.name = "Programming";
    await connection.manager.save(category2);

    const post = new Post();
    post.title = "Control flow based type analysis";
    post.text = `TypeScript 2.0 implements a control flow-based type analysis for local variables and parameters.`;
    post.categories = [category1, category2];

    await connection.manager.save(post);

    console.log("Post has been saved: ", post);

}).catch(error => console.log("Error: ", error));
// typescript-example/src/entity/Category.ts

import {Column, PrimaryGeneratedColumn, Entity} from "typeorm";

@Entity()
export class Category {
    @PrimaryGeneratedColumn()
    id: number;

    @Column()
    name: string;
}

Suggestion

Best to avoid Knex since it seems to be the worse choice in many areas, there’s very little that it outshines either TypeORM or Sequelize.

Out of TypeORM and Sequelize, Sequelize seems like the safer option with the added benefit of a bigger community. This means nice plugin and tools such as (sequelize-ui, sequelizer, etc).

Last but not least, TypeORM looks like to be the new kid of the block making use of decorators and etc. Compared to Sequelize’s philosophy of “getting migration done in a stable way”, TypeORM seems to be more focused on “doing migration in the right way”. This can be seen from its documentation, there’s a lot of emphasis on “reducing duplication”.

Update (8th March 2020)

After going with Sequelize and working with it for a few months, here are my latest update on this subject Why I would avoid Sequelize in the future.

VS Code, Svelte and Prettier set up

Sat, 04 Jan 2020 14:36:00 GMT

I recently tried to set up a new project to try out Svelte in more depth. I search everywhere but couldn't get VS Code (e.g. on file save) and Prettier CLI (e.g. running npm run lint) working at the same time. After hours of trial and error, I finally got it to a place I'm happy with. Hopefully, this will help anyone in a similar position.

Install Dependencies

npm i -D parcel-bundler parcel-plugin-svelte prettier prettier-plugin-svelte svelte

svelte because we want to develop apps with it, duh! parcel-bundler parcel-plugin-svelte Svelte works with all the popular bundler tools such as Webpack, Rollup and Parcel. Personally, I'm a big fan of Parcel so I've gone with it here. prettier prettier-plugin-svelte People tend to use a mixture of Eslint and Prettier for code formatting and linting. But I wanted to keep dev dependencies lite, so I'm only using Prettier in this case.

Project Structure & Files

/
|-- package.json
|-- package-lock.json
|-- .prettier
|-- .gitignore
|-- src
     |-- index.js
     |-- index.html
     |-- style.css
     |-- App.svelte
     |-- components
          |-- ...

package.json

When using Svelte with Parcel, one thing to note is to ensure the main property is set to the correct.

{
    "name": "my-svelte-template",
    "version": "0.1.0",
    "private": true,
    "description": "This is not a real repo.",
    "author": "Hao Dong",
    "main": "src/index.js",
    "scripts": {
        "dev": "parcel src/index.html",
        "build": "parcel build src/index.html",
        "lint": "prettier --check 'src/**/*.{html,js,svelte}'",
        "lint:fix": "prettier --write 'src/**/*.{html,js,svelte}'"
    },
    "devDependencies": {
        "parcel-bundler": "^1.12.4",
        "parcel-plugin-svelte": "^4.0.5",
        "prettier": "^1.19.1",
        "prettier-plugin-svelte": "^0.7.0",
        "sass": "^1.24.2",
        "svelte": "^3.16.7"
    }
}

Notice here we are using Prettier CLI to both check linting and auto-fix linting errors. This means we could both check for linting errors and auto fix them in the terminal by running npm run lint and npm run lint:fix respectively. It is a good practice to run at least the lint check in the CI pipeline to ensure code consistency.

What about VS Code, you might ask. We'll cover how to set up auto format each time we save a file in later sections. Unfortunately, the CLI auto format and VS Code auto format shares very little in common so will require two different setups. This is also one of the reasons it took me a while to get it working the way I wanted.

package-lock.json

I'm going to skip package-lock.json since it is a generated file.

.prettierrc

For Prettier I try to leave most of the formatting options as default, with the following exceptions.

{
    "tabWidth": 4,
    "printWidth": 120,
    "semi": true,
    "singleQuote": true,
    "trailingComma": "all"
}

.gitignore

A very standard git ignore file.

.DS_Store
node_modules

# svelte
.cache
dist

# dotenv environment variables file
.env

src/index.html

<!DOCTYPE html>
<html>
    <head>
        <meta charset="UTF-8" />
        <meta name="viewport" content="width=device-width, initial-scale=1.0" />
        <title>Svelte with Parcel</title>
        <link rel="stylesheet" href="./styles.css" />
    </head>

    <body>
        <script src="./index.js"></script>
    </body>
</html>

src/index.js

import App from './App.svelte';

const app = new App({
    target: document.body,
    props: {
        name: 'Svelte',
    },
});

export default app;

src/style.css

html {
  height: 100%;
  box-sizing: border-box;
}

*,
*:before,
*:after {
  box-sizing: inherit;
}

body {
  min-height: 100%;
  margin: 0;
  padding: 0;
  font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Oxygen,
    Ubuntu, Cantarell, "Open Sans", "Helvetica Neue", sans-serif;
  scroll-behavior: smooth;
  text-rendering: optimizeSpeed;
  line-height: 1.5;
  color: #212223;

  display: flex;
  flex-direction: column;
  justify-content: center;
}

src/App.svelte

<script>
    export let name;
</script>

<style>
    h1 {
        text-align: center;
    }
</style>

<h1>Hello {name}!</h1>

VS Code Set Up

Extensions

In order to get the best Svelte support, install the following VS Code extensions:

It might be useful to install these extensions as well, but they are optional.

Settings

Within src folder we have 3 main file types that requires linting and formatting:

*.html will be formatted by Prettier (extension installed above, based on .prettierrc config)
*.svelte will be formatted by Svelte (extension installed above)
*.js will be formatted by Prettier (extension installed above, based on .prettierrc config)

In order to get format on file save working, open Settings (JSON) then add the following options:

{
    ...
    "editor.formatOnSave": true,
    "[svelte]": {
        "editor.defaultFormatter": "JamesBirtles.svelte-vscode"
    }
}

Then open up the different file types and see what VS Code is showing on the bottom status bar. Notice how both JavaScript and HTML have Prettier ticked, and Svelte doesn't.

As mentioned above the npm run lint and npm run lint:fix will work with Svelte file types no problem. However, Prettier in VS Code for some reason just doesn't want to auto format Svelte files. So I left the auto format to VS Code's Svelte extension instead of the Prettier extension.

Additional Reading

https://www.rockyourcode.com/prettier-and-es-lint-setup-for-svelte-js/

Software developers' learning dilemma

Mon, 02 Dec 2019 01:18:00 GMT

The world of tech is moving fast, maybe too fast! Trying to keep up with everything is stressful, to say the least, might even call it impossible to be realistic. There are a countless number of frameworks and libraries being update each day and new ones being released as well. Yes, maybe is more of a JavaScript problem, but the same problem more or less applied to other programming languages as well. So how can we find time to enhance our existing skills to be a better developer, while the backlog of learning constantly growing? We are being bombarded with information each day, sooner or later we'd miss demotivated and lose the urge to learn. I've gone through all those stages and feel I have a pretty good balance on how to learn daily but not risk burn out.

I feel that I go through these phases at least every few months. It is probably quite obvious now I blog, from Sep to Nov 2019 has been a tough time for me. This is accurately reflected by the number of blog posts released in this period. At the end of Nov I finally managed to pull myself back out of this slump state, unfortunately, I know it is just a matter of time before it happens again. I'm sure this is not just a problem for me, so I want to talk about it in this blog post.

The dilemma

When I was still at University I didn't do much learning through blogs, etc. I found that lecture notes, recommended reading and course works were enough to keep me busy. There was one lecturer who recommended 10 books and what seemed like an infinite list of papers to read for her module. I am going to be honest and say that I read none of it. If you are wondering how I did in the exam, let's just say I deserved the exam marks 😂. Now looking back I think it was the amount of reading that scared me away, I kept pushing it back and then before I knew it exam day came.

Now I am no longer told I have to do this kind of reading, but I feel like the same problem persists in a developer's everyday work life. Let's face it, there is a lot to do and think about. The more curious and responsible you are as a developer, the more the mountain of things to do and learn grows. I am interested in many roles in a product team, I love to learn about UX (including UI), I want to know the business requirements from a product point of view and not just something to code against, I also want to make sure I'm playing a useful role in my team (help to add tickets to backlogs, etc). That is a lot of stuff to do if you break it down and slot it into everyday work time. Then there are meetings and doing the work I'm paid to do (duh!). So when is the time to learn about sveltejs or other frameworks and libraries? I'm ashamed to say that I even write down learning other languages as part of my learning goals.

Assume you somehow find the time to learn things at work, or you find the motivation to sit down and do self-study after work. The next question that comes to mind is "what should I learn"? Only to find all sorts of negative emotions flooding my brain. How can you identify if something is useful or a waste of time, when there are tens of thousands of pieces of information are thrown your way? "Deep dive in computer algorithms", "Machine Learning: an introduction", "AI: from zero to hero"... There is no way for you to consume faster than content being produced. There are YouTube videos, blog posts, podcasts, books, new libraries, new frameworks, new languages, tweets, Linkedin posts and what feels like a million other things to digest each day. They all have click-baity title, they all seem relevant and important to know. Just saving it to read later isn't helpful either, it just kicks starts a vicious cycle. The more we bookmark and save, the more negative we feel. The thought of "oh my god when will I ever find enough time to consume all these" becomes an anchor weighing us down more and more.

Just explaining this stuff makes me want to smash my computer, quit my job and go live in the woods somewhere. This information overload is not healthy. It brings anxiety and stress with it, which is why every-so-often I enter the slump. I should think of a cool phrase for this feeling and call it something like "Drowning in information, gasping for air".

The balance

It doesn't take a genius to see that we need a balance, better way of consuming knowledge coming our way. At least this is what we need to do before a magical AI comes along and feed us the right information to learn at a steady rate. My most recent slump just shows that I haven't figured everything out yet, but here are some of my thoughts: On one hand we have super chilled people, they never seem stressed about anything. They would deliver work at a slow pace, causing inconveniences for both their team and maybe other teams. They regularly take days off to recharge, and probably will never burn-out. I like their attitude to life, no doubt they will be very happy. But different people look for different things in life, I want to be reliable and bring value. On the other hand, some people spend at least one day of their weekend maybe longer coding and learning. People build cool libraries, maintain multiple open source projects, and then have side hustles that bring in some extra cash. I would look at them and think to myself "I could I could be this motivated", but I know realistically I won't be able to do what they do. So how should I find my balance? I started listening to podcasts and read self-help books. It is ironic, to answer the question "how should I learn things" I must first learn about it. Here are some useful materials: Getting Things Done by David Allen - Taught me to organise my life using todo lists. One of the key messages is that by writing things down and keeping it somewhere we know we will review regularly, it will reduce anxiety. Essentialism by Greg Mckeown - It talks about concentrating on things that are most essential in your life, only this way will there be peace and balance in your life (pretty deep stuff). Pareto principle (the 80:20 rule) - The Pareto principle states that roughly 80% of the effects come from 20% of the causes. These means don't be a perfectionist, one way consuming more stuff is to spend 20% of the time skim through an article getting 80% understanding of the content than to read it word for word.

Final words

These three points are by far some of my most important learnings in the past few years. I hope people find them useful. As mentioned already, even after adopting the above advice I'm still struggling to find the right balance. Maybe that is okay, life is short we should enjoy things other than improving our coding skills. Even for those wanting to speed up their career, sometimes losing yourself for a while helps to find the "real" direction.

Awesome tools to use in personal projects

Sun, 01 Dec 2019 22:13:00 GMT

A developer who doesn't work on personal projects isn't a true developer! Just kidding! But working on small side projects is an awesome way to solidify and improve our craft. It is both practice and exploration, this is the best opportunity for you to pick up on something others might not know about! In this blog post, I will walk through some of my favourite tools and what I feel is missing in my toolbox.

I often have crazy product or app ideas, most of which I don't carry through with. Why? It takes too much time to code up and then there's the maintenance later on. In the back of my mind, I'm always looking for awesome products that could solve these problems for me. Luckily over the years, I have found many of them, and each time I find a new tool that makes the process smoother I find myself finishing more products.

What I have found so far

The best part about all the following tools is that they all have a free tier. If you are planning to use them for personal use, it's extremely unlikely to go over the allowance. If you do end up going over, then your project must be super successful, in which case, congrats!

Best hosting & CI/CD tool - Netlify

I enjoy learning about DevOps, but I never intend to become a DevOps engineer. I enjoy creating and building products, DevOps is too far from the customer for my liking. I also found CI/CD of any personal projects to be a pain point, it takes way too much time to set up that's not even mentioning the cost of running and maintaining your CI/CD solution. A few years ago, I found out about Netlify. It opens the door to many of my projects. This blog is one of them, it makes many of the challenges around hosting and deploying so easy. A few clicks later I can start working on my new project.

Development tools - CodeSandbox, Gitpod, VS Code

Isn't it annoying when you just want to brain dump some code and you have to go through all of the project setups? I started using CodeSandbox for any small or app prototypes. All it takes is selecting a template project to get started, then I can just jump in and start coding right in my browser. It is amazing!

Then there are some established projects, making small changes requires clone projects from Github, etc. It is so much hassle and I'd just postpone it time and again. Recently I found this tool called Gitpod which solves this exact problem, it could check out any repository and allow you to work on it straight from your browser. You could run terminal commands, so it is possible to build your projects as if you are developing on your laptop. Finally for any serious projects I normally would git clone the repository on to my laptop and work on it with VS Code. I have a list of extensions which empowers my development experience. But I tend to only use VS Code for projects I know I'll be working on for a long time.

I have written about VS Code keyboard shortcuts since all three tools mentioned in this section is based on VS Code, so most of these keyboard shortcuts can be used with all three products!

CMS - Contentful

For a long time, I struggle to find an effective way of building my blog. Until I found out about Contentful which is a headless CMS. It is perfect for serving data for my blog! I have yet to find another use case to use it in my other personal projects. There are also many other headless CMS out there, I just happened to have picked Contentful when it was the most popular. It has been a couple of years, there could well have been newer and better headless CMS out there!

What is still missing - user data storage

All the above products have been a godsend, they have made my development process so much easier. Since I tend to be the only one working on my personal projects, so their tools make it easy to get started on an idea and release something in a few days.

However, I am still searching for one more thing. There are tones of apps I want to create that could do with some kind of storage in a server (instead of storing in Local Storage). I have created several apps that utilise Local Storage so I have nothing against it. I just find it annoying that all the data were unavailable when I switch between devices.

I would love for there to be a product that could:

Store individual user's data
Have some kind of authentication in place so only the user themselves could access their data (maybe me as the developer as well for tech support). It would be ideal if this product could allow client-side direct access, without the need for a backend server doing data validation and authentication.
Easy to set up, like have a data store ready with a few clicks
A free tier, yes I'm cheap like that! But seriously, I don't want to be paying for tens and hundreds of personal projects. They will have minimal usage anyway, so a free tier would be amazing.

I have look around and there isn't a product that could do the above well. The best one I have found so far is Google's Firebase. It almost nails all of the above points, with the only exception being authentication. I found it very difficult (and confusing might I add) to set data store rules.

I don't want to spin up a new database and create a bunch of APIs every time I have created a small app. This data storage problem is the last one to solve for me. If anyone knows any products I am not aware of, please get in touch I would love to know!

Linux is still Linux in 2019

Fri, 08 Nov 2019 17:36:00 GMT

The experience of installing Linux and getting it running on a machine is just as inconsistent as a year ago and years before that. There were visible improvements, but still plenty room for improvements compared to Windows and Mac OS. I encountered many issues during my little experiment, I ended up switching back to my old setup at the end.

Background

I had a few days off last week and didn't book any holidays to go anywhere. So the nerd in me thought it would be a great idea to get Windows and Linux dual boot setup on my desktop. I was looking forward to the new improvements which were added in the past year or so. The result is somewhat disappointing, and I ended up switching back to Windows (I use a MacBook Pro for most of my coding tasks). In this post I'll go over some of the issues I faced, hopefully, it will show why I decided to switch back.

Before I get started, I just want to say the following as disclaimer. I ended up struggling with the setup process for 3 evenings in a row, trying to get a good Linux distribution dual booting on my Windows desktop. I have experienced a lot of weird issues and it is frustrating, to say the least. So naturally, this post will contain quite a lot of negative feelings. But I don't want people to think this is me attacking Linux for being worse than Mac OS and Windows. I just had so much hope and expectations going in, it was quite a disappointing experience.

As part of my little experiment, I tried the following distributions:

Pop!_OS (19.10, released 22nd October 2019)
Deepin (15.11, released on 19 July 2019)
Elementary OS (5.0 "Juno", released on 16 October 2018)
Zorin (OS 15, released on 5 June 2019)

Every distribution has its problems

I tried the latest version of the above distributions, but as you can see from their release dates. They range from Oct 2018 to Oct 2019. These release dates nicely reflected the state of these distributions.

Elementary OS

Elementary OS was the most stable one out of all of them, but it also meant there was a lack of new features.

Elementary OS used to be the best-looking distribution a couple of years ago. Since the latest version is from a year ago, all other distributions look more modern and simply better.

Issues:

Lack of Chinese Input - This distribution is the oldest out of all the ones I tried. It isn't based on Ubuntu 19.10, therefore, it doesn't have built-in Chinese input. For others, they might not care, but for me, this is also a deal-breaker.

Apart from the fact it was missing all the new features from 19.04 and 19.10 releases, there is little to complain about it. I decided not to go with it because it didn't have native support for Chinese input. Which is a deal-breaker for switching to it completely? When the next version of Elementary OS comes out based on Ubuntu 19.10 or later, then it would be the best choice for me I think.

Note: there are a lot of activities on Github, hopefully, it won't be long now.

Pop!_OS

A more recent distribution compared to the likes of Elementary OS. But it is backed by a company called System76, who specialises in building and selling Linux laptops. It claims to have very good gaming support out of the box, now with the new Nvidia driver improvements brought in as part of Ubuntu 19.10 (which the latest version is based on), it should be one of the best choices for people.

Notice the keyword "should". I spent the most time trying to get this distribution working because I felt like it was "the one". It got so much right and has so much potential!

Issues:

Lack of fractional scaling - Since it uses Gnome desktop by default, this means it doesn't support fractional scaling without turning on beta flag via Terminal. Out of the box a user could choose between 100%, 200%, 300%, etc scaling options. 100% is too small on 4k monitors and 200% is way too big in my opinion.
Auto power off monitor resets fractional scaling back to 100% - After turning on beta fractional scaling option, I left the machine idling for a bit. After 5 min (system default option) it turned the monitor off when I woke the machine again it changed the scaling back to 100%. The only way for me to work around this is to disable auto turn off the monitor.
Login screen scaling cannot be changed - No matter what I changed the fractional scaling value to, I could never get the login screen to change from the default 100% scaling level.
Short scroll - I had really weird mouse scrolling speed. It would take me down 5 lines of text on a 4k monitor at 150% scaling when I'd normally expect the whole screen to be scrolled down. Maybe it could be fixed by changing scroll input values in the Terminal, but I have given up on this distribution at this point.
No dual boot with Windows - No matter how I configured my BIOS, I simply couldn't get Windows boot option to show up during boot. I even reinstalled both Windows and Pop!_OS multiple times as a result. Maybe this is due to I'm trying to dual boot Windows and Linux on different SSD hard disks. I didn't figure it out at the end.
Untrusted sources - This is another problem that just left a bad taste in my mouth. After booting into Pop!_OS, I wasn't able to add quite a few sources. Every time I try, I just get the error message cannot look up untrusted sources (or something like that). It seems that 19.10 is so new, there are many packages which haven’t been released yet. Long story short it is not possible to pull these packages down, or at least with everything I tried I was not able to get it working.

Deepin

Deepin UI looks very sleek, but after trying it out it seems a little too flashy for my taste. The settings menu on the side is not easy to use in my opinion, it should have just kept it simple like the other distributions, but it's just my preference.

A big plus for me is the fact the app store is the best out of all of the distributions I have tried. It had everything I needed, even tools such as VS Code which 9 times out of 10 I need to manually install in the Terminal.

The OS is customisable, maybe a little too much. Spending too much time and energy means a lot of areas in the OS itself feels very rough around the edges.

Issues:

Poor performance - The performance is one of the worst I have experienced on Linux. When opening, hiding and closing apps everything just feels a lot slower than other distributions.
Confusing UI - The problem of spending too much effort on making everything look too fancy is that as the user you'll sometimes find small UI issues when using the OS normally.
Inconsistent UX - There are really weird things embedded in the OS, it feels like lots people build different parts then glued it all together.
Bad Chinese input - this had Chinese input support, but it is so buggy they may as well excluded it. It got to a point where I couldn't switch back to English 🤦‍♂️

Zorin

Zorin by far is the best-looking distribution out there in my opinion. If you upgrade to the paid version, then there are even more type of layouts to pick from e.g. Mac dock layout. UI and UX just looking amazing.

Issues:

Terrible performance - This is the worst performing distribution out of everything I tried to-date. Even dragging windows around it would lag like hell. It felt like an animation problem, but when I checked GPU settings everything looked fine. It detected by GPU and everything.
Lack of Chinese Input - Similar to other distributions, this isn't based on Ubuntu 19.10, therefore, it doesn't have built-in Chinese input.

I didn't play around with this distribution enough to find other issues, because of how bad the performance was.

Gaming

Gaming has always been an issue on Linux. Considering this is a problem for all distributions I thought a dedicated section makes more sense.

Almost every other month I'd read someone writing that they have fully switched to Linux and the gaming experience is much better than before. I could see their point, with tools such as Lutris it is much easier to install games. Unfortunately, it is still no Windows. If you are lucky you might be able to get some game running, but major OS update might break the game and a game update might break the game. There is so much uncertainty when you are you in a mood for gaming you might find yourself doing debugging instead. This is why I wanted to dual boot Linux with Windows instead of switching over completely. The truth is I was right when I tried to get League of Legends (one of my most played game) working on Pop!_OS. I could install and boot into the game with Lutris using Wine no problem, but every time I start playing it would crash. I'm not saying this is not possible to fix, but it is too much for me when I just want to play a couple of games and relax.

The competitions are too strong

I got WSL 2 (Windows Subsystem for Linux) working on my desktop before I tried to do all this. Sure it is not perfect, which is why I wanted to get Linux running on my desktop. Though it is not perfect, and I can see myself running into problems in the future. It is a lot less maintenance than dual booting Linux with Windows.

My perfect development machine is still a MacBook Pro right now. I just find it superior in so many ways compared to Windows and Linux.

Final words

To summarise, compared to a year or two ago there have been huge improvements. I'm quite excited about new competitors such as Zorin, Deepin and Pop!_OS. Whilst Linux Mint, Ubuntu, etc are still leading the pack in terms of downloads and interests, I see a lot of potential in these new challengers.

Now, if you are thinking of installing or switching to Linux, my recommendation is to stay put with Mac OS and Windows. Unless you want to tinker with the OS, in my opinions these distributions are not quite ready to be used as Mac OS and Windows replacements for productivity. No matter what I did, I found myself more or less had to settle with some compromises.

I love the fact Linux is open and free. They aren't just Operating Systems on people's personal computers but runs most of the cloud too. Linux has a special place in my heart, it is a rare and positive thing in the world of technology and software development (with all the spying and privacy leaks).

I am still looking forward to switching to Linux one day. Maybe I'll give it another year and try again? We'll see...

Creating TV Apps With Web Technology

Mon, 26 Aug 2019 18:30:00 GMT

Ever wondered how TV apps are made? How it differs from the usual web app or mobile app development? I was fortunate enough to work on TV platforms for about year now, and there is some exciting stuff for me to share around this subject. I will first give an introduction on TV platforms, then explain why we decided to use web technology to build TV apps, finally I'll go through some of the challenges we faced that are specific to TV app development.

As mentioned already I've broken down the blog post into different sections, feel free to skip to the relevant sections if you are not interested in the background information.

Introduction to TV platforms

Many people own at least one TV. It is what we use to play console games, watch the latest shows and movies. We can even project our phone or computers on to them to show things to a bigger group. It is estimated that 259 million TV units will be solved by 2020 source: statista.

Here are just some of the major TV platforms that we support:

As shown in the image above, the platforms are made out of a mixture of different type of devices. Some platforms are built into TVs (e.g. Samsung's Tizen, LG's webOS, Android TV), there are also external boxes that you can plug into any TVs and turn them into a smart TV (e.g. Amazon Fire Stick, Apple TV, Android TV again, Chromecast, etc), then there are game consoles that can act as a TV box on top of its gaming functionalities. Finally, streaming platforms often come up with their own dedicated physical devices and operating systems (Roku, etc). It is probably safe save to say, the fragmentation within TV platforms are a lot worse than web or mobile.

The above platforms are also known as "smart-TVs" by the customer, why is it smart? Because they have apps.

Building TV Apps

It is quite unfortunate for companies who needs to release apps on TVs, because of the fragmentation issue mentioned. So how might one go about developing a TV app? It is a bit like Mobile app development actually, there are 3 general approaches:

Native
Web or PWA
Hybrid

Let's talk about each of them and see what are the pros and cons.

Native Apps

In order to go Native, we will need to build a separate package (app bundle) for each platform. Give the number of platforms we need to support, this is both time-consuming and costly for the business. Especially when most of the platforms use different languages with their unique frameworks and libraries. The development process involves writing redundant business logic in different languages which are not shareable. Needless to say, this is very inefficient and should be avoided if possible.

Mobile development has a strong argument to go Native because there are only 2 platforms for them to develop against. Imagine if there are 10 platforms, in that case I doubt many companies can afford to build their app in Native.

On the positive side, out of all the choices, building dedicated Native apps for each platform gives the most customizability and control. If would be possibel to adapt apps to their respective platforms, or even make them totally different from each other if we wish.

Web App or PWA

Surprisingly this is more viable than one might think. Most platforms support some kind of WebView which are used to render web applications. It is how Progressive Web Apps (PWA) work on Android (mobile). However, the main drawback with this approach is the lack of native API support. Say I want to detect when the TV loses internet connection then show an error notification as it happens. A pure Web App won't be able to achieve that as most TV platforms don't bake it into its WebView. These kind of limitations means it is difficult to build a high-quality app, and in some cases, a basic application.

The main advantage of this approach is how easy it would be to deploy apps to all the platforms since all we have to do is to deploy the web app and then register it on each platform. Unfortunately, this approach is not ready just yet.

Hybrid Solution

This solution effectively combines the benefits of the first two approaches. With this we create a basic shell application for each native platform, they all have limited functionalities and provide WebView with native APIs support. Inside the shell app, there is a web app that contains most of the app logic. This web app is shared across all platforms, which means less redundant code.

Here's a diagram to better illustrate how it all works:

In this example, the shell application is written in Java and uses Cordova to provide native API to WebView.

TV Platform Specific Challenges

Before we drive into the challenges, let me first show you what kind of UX we are working with. The image below shows an out-dated design because the new design is still sensitive information.

Note the following terminologies, we'll be referring back them a lot in this section.

Tile - a rectangle block
Rail - a row of Tiles
Grid - multiple rows of Rails

Display

TV devices are meant to be used from far away, this means we had to make all the Tiles large enough so that the text is still readable from a few metres away. This leads to less information is shown on the screen, in turn, it would require more user actions to browse around. This mean all of our UI had to be simple and concise.

TVs come in different screen sizes (luckily the same ratio). How can we make the UI elements the right size for all devices no matter the screen size? We ended up going with Viewport units to resolve this issue. But it is still challenging to use a percentage based unit, especially when we want to add animation to something often have to use values with long decimal points.

Navigation

Navigation is a core part of any application, without it, there isn't much the user could do with the app. Unlike web or mobile, we can't click or tap on stuff to register an action. Instead, we are limited to TV controller inputs, again due to the fragmentation problem identified already, we want to keep things simple here. We support these actions: UP, DOWN, LEFT, RIGHT, SELECT, and BACK. The whole app behaviour needs to be built around these basic interactions and this brings some serious challenges.

Focus is a big thing, so I've dedicated a whole section for it later on.

With browsing all controlled by arrow keys. Each button press equates to one user action. If there is a long Rail, it will take tens of button presses to browse through all of Tiles which is not ideal. To improve the user experience here, we created "fast scroll" functionality. When the user press and holds a button (e.g. LEFT), this allows faster navigation around Rails (they will see the tile shifting through) until they release the button.

With Grids, we have the same problem as long Rails. The "fast scroll" technique is not going to help us much here, because now we are working with 2 dimensions horizontal and vertical navigation. To help users find what they want on the grid faster, we introduced the "wrap-around" concept, which means when the user press LEFT on the most left Tile, the focus will wrap around to the Tile on the far right side. Using the screenshot example above, when user press LEFT on the current focus Tile "Milan v Atalanta" the focus will shift to "Roma v Sampdoria".

Layered Views

It is worth knowing the structure of the app before we proceed to Focus management. The app we are building has 4 layers:

Player layer - video player
Page layer - shows rails
Overlay layer - transparent and appears on top of Page layer
Notification layer - for error handling, acting as modals

The reason why we have this multiple-layer design is that the user must use the buttons specified in the Navigation section to move around the screen. User can navigate back and forth between the different layers using SELECT and BACK buttons. When the user change between different layers, the focus will update and refocus on something appropriate based on what layer and content are being displayed. Therefore the layered design, will give the user a sense of depth on an otherwise very flat navigation flow.

Focus Management

Focus is a crucial part of the UX since we need a way to indicate to the user what they are focusing on. This was by far one of the most challenging parts for us.

One of the reasons that make focus difficult to manage is that we require one and only one thing in focus at any point in time. Otherwise we either risk multiple elements in focus at the same time, or that focus was completely lost with the user stuck in their journey.

We had to ensure focus can be handed around the app correctly. This means between elements, pages, layers and even deep-linking. How do we know what to give focus to and focus on? To solve this problem, we create a focus map (JavaScript object) which we use to look up where the focus should go in different situations. E.g. given the page, currently focused element, keypress we can find out what to focus on next.

For deep-link journeys on TVs, when the user lands on a page with no prior focus, the app will assign focus to the default focusable element. This "default focus" is part of the focus map explained above.

Back Journeys

This is another topic that gave the whole team a lot of headaches. Put simply, the back journey is what happens when the user press the BACK button. I think the main challenges here comes from the UX and not the TV platforms. But though I'd mention it anyway, just because it caused us a lot of pain.

Back journey could mean any one of these things (knowing how complex it is, I probably missed some out):

It could be simply reset focus on the current page, often going back to the default focusable element
It could close or dismiss notification without affecting Browser History (no routing change)
Go between Overlay and Page layers (change in Browser History)
Go between different Pages (change in Browser History)

On any BACK button press, it could result in one of the above behaviours. The difficulty is working out which one, and it depends on several factors: where the user came from, where the user is right now, what is in focus or is it part of any specific business logic?

There are talks of change the journey completely.

Performance

The TV devices are such a mixed bag, some are quite powerful while others are very limited in resources. Performance optimisation is one of those conversations we have weekly. But I won't cover more of it here, if you are interested please check out my previous posts improve CSS performance of Cordova apps on Android TVs and ways to improve list loading performance.

Development

We didn't want the Native side of things slow us down when we develop the web application. So we made sure that we can spin up the web app in the browser and develop it the same (almost) way as a standard web application. We use hot-module-reloading and other common development strategies, so we can implement new features at roughly the same speed.

Once we finish developing a feature, we always have to test it on physical devices. This almost always means generating a new Native Shell app and side-load the web app inside, then test it on whatever device we are developing against.

Final words

The actual project I am working has a lot of complexities than the ones mentioned above, because we are using a pretty complex micro-frontend architecture and teams in different countries work on different part of the architecture. But I still enjoyed building TV apps with web technology, there are some interesting findings along the way which helped me to understand more about frontend development in general. If anyone gets an opportunity to work on TV platforms, based on my own experience I'd recommend people give it a try!

Micro Frontend: An Introduction

Mon, 26 Aug 2019 15:55:00 GMT

Micro-frontend is not a new concept, it is simply a rebranded idea which is now a new buzz word. In this blog post, we'll talk about what it is and why it is here, but more importantly, I hope to illustrate that people shouldn't shy away from micro-frontend because they probably know a lot about the subject already.

There are many ways to implement micro-frontend and depending on the nature of the project some may fare better than others, but we will cover all this in another blog Micro Frontend: 3 ways to implement it. If you are already familiar with it, feel free to skip to the next post.

Introduction

In the past few months, the phrase "micro-frontend" is gaining more traction by the day. After seeing how much people loved the idea "micro-services" in the backend community, I guess the frontend community felt a bit left out. The concept, like the name, borrows ideas from Micro-services, another buzz word that became popular a few years ago and people still use it as one of those things that can be thrown into a conversation to earn credibility. Today we'll talk about it and see what all the fuss is about.

Here's a quick summary of micro-services if you are not familiar with it:

The central idea behind microservices is that some types of applications become easier to build and maintain when they are broken down into smaller, composable pieces which work together. Each component is continuously developed and separately maintained, and the application is then simply the sum of its constituent components. This is in contrast to a traditional, "monolithic" application which is all developed all in one piece. - what are microservices.

In the tech sector, people are very good at labelling something with a new name and present it as a new concept. Just see how often Apple rename existing technology to better market new products, we all roll our eyes when we see/hear it but no one can argue its effectiveness. A new name comes with a new mindset, it will get people talking about it and at the same time become confused by it.

How does it work?

Just like micro-services, this new "micro-frontend" concept is about breaking down your application code into multiple parts so that they can be developed, maintained and work independently. But from the user's perspective, it works and feels just like a normal application.

Here's a diagram illustrating what a monolith approach looks like.

As you can see it the inner workings of the application is not straightforward by any means. In fact, it will probably take some time to understand what each part do and why they are there. Most likely they are the result of poor planning or specific business logic governing something, or both.

Now let's take a look at what a micro-frontend approach looks like.

Of course, this is oversimplified for demonstration purposes. But the idea is that, once we split a big app into smaller ones everything becomes exponentially easier to understand. This means we can implement new features faster, fix bugs easier, and etc. Of course, there are some drawbacks with micro-frontend too which we'll discuss another time.

I won't go any further with the "how" part here as that will be covered by the blog post mentioned above.

No, it is not a new concept

Wow, this is amazing, right? Well, not really.

I think most of us are already using micro-frontend one way or another. It is pretty unlikely to see a monoliths approach that contains all of the frontend logic in one place. Businesses often split up parts of the website such as blogs, forums and even parts of the main app to help with the separation of concerns or simply to divide work between all the development teams. In many cases, it often means the tech-stack is slightly different too (which may or may not be a pain point). Out of the client projects I worked on in the past few years, I have seen a trend of people starting to split up big apps into individual Single Page Applications (SPAs) without knowing they are doing micro-frontend. Teams have been going down the right path due to other new concepts in the industry such as componentisation and etc, in pursuit of a more maintainable codebase. The reason components are so popular is because it benefits both the business and developers. Componentised apps are easier to develop and maintain, due to isolation and separation of concern it offers. Thus it also makes it easier for business to divide work among teams or team members. I think micro-frontend will become popular for the same reasons.

If it is not a new concept, then why is it rising in popularity now?

The problems we are trying to solve with "micro-frontend" aren't all that new under the surface. The only reason why this is getting more popular is due to the overused SPA in projects, more specifically, projects that implement monolithic SPA. Some large and complex websites are cramped into single SPAs are pretty extreme examples, but we all end up doing something similar sooner or later. It could be lack of time to split code up when under pressure from the business. This leads to over-complicated logic for handling user journeys, possible app states, routing, etc. Anyone who's worked on a large React-Redux app should know how complicated the state gets, with all kinds of Actions and Events triggering from different components. The store could also get too bloated and become harder to maintain after some point. Here's a good article on the issues with SPA if you want more information: problem with single page apps.

Hopefully, now I've convinced you that you are not looking at a new concept whcih will take a lot of effort to learn. The new "micro-frontend" is just here to remind us to be logical when it comes to designing and architecting new web apps. If we know a project is going to be complex and we want to use the SPA approach, then we probably should break it down before it gets too big and scary.

Self Reflection: How To Run Coding Sessions As A Volunteer

Sat, 24 Aug 2019 13:48:00 GMT

After almost 4 months of volunteering, I've learned that running a weekly session for one volunteer is too much. Also don't try to teach big topics that span over a few sessions, because people will drop in and out. They will struggle after missing sessions and either stop attending or come in feeling very lost the next time. People love coding games and challenges instead of tasks with vague requirements like "make a website that uses HTML form". Going forward I'm going to try smaller bite-sized sessions so people can drop in and out more freely.

Mistake 1: Weekly session is too much for me

I didn't think it through properly when I signed up and agreed to weekly sessions. I thought 1 hour (if we overrun then 1.5 hours) a week is not a big deal, I am sure I can make up time for that. Oh man, was I wrong! I forgot to take into consideration the time for planning each session, and manage the communication side of things as well.

What ended up happening was I'd spend 2+ hours on preparing the slides for each session, creating exercises and solutions too (because not everyone can attend every week so they need good material to keep up). I'd then send out slides before the session (due to some people prefers to print out the slides for the session). I'd also remind people about these sessions and answer any questions. Sometimes the session will overrun and end up being close to 2 hours instead of the 1 hour I initially planned. I don't mean to blame anyone, these are realistic expectations, I just didn't consider them at the start. Anyhow, I was spending close to 5 hours each week. This was eating up quite a lot of my free time, especially given that I have a 9 am - 5:30 pm day job Monday to Friday.

If anyone is thinking of volunteering, as I've mentioned in my volunteering to teach people to code post I think it is a great thing to do. I strongly recommend it, not only because you are doing something to help other people in your community but also it will help you in so many ways too. My advice here is to try to give yourself some breather room by not committing to do it weekly. I think participating bi-weekly would be okay, but maybe monthly would be more sustainable. Of course, it very much depends on what it is that you do. If all you have to do is turn up on the day and have no other responsibilities then go for weekly if you can make the time. But since I am a one-man team, weekly is too challenging.

Mistake 2: Don't aim for the sky

Another mistake I made was to plan out 4 months worth of session and set out a big goal "to teach people with no coding experience the basics web development". This means teaching them about HTML, CSS and JavaScript, so they could make a simple web site at the end. I think for a small number of attendees, they managed to reach the original goal. But for most people, it wasn't the case.

Smaller and independent sessions are much better for this kind of free sessions. It sounds weird to say, but because it is free people feel less strongly about attending each week because it is free. If they paid for the course then they would push themselves more to attend I think. The lesson learned here is: don't expect people to turn up each week. It is probably not their fault, things come up sometimes. It probably also didn't help that I was running the sessions on Wed evenings.

Going forward, I'm going to plan bite-sized sessions with more realistic learning goals. My current plan is to only focus on teaching people JavaScript and show them how coding/programming works instead of teaching them about the whole web. This will allow people to drop in and out of these sessions. If they miss a few, it is no big deal they can still follow and learn. Whereas the old format means if people miss one session, it is very hard to follow the next session. This often means they'd drop out completely and stop attending future sessions. For those who are more motivated, they would struggle a lot more.

Mistake 3: Specific challenges are better

It shouldn't be a surprise really, but people loved the Flexbox Froggy exercise. I was so amazed at how many people managed to finish all 24 levels! I just want to remind you these are people who have very little coding experience and this is the second session on CSS. Everyone needed a little help, but I was still so happy almost everyone got through it all in less than 2 hours that the session after I bought snacks for everyone to reward them.

In comparison, the exercises I created myself often would confuse people or just see a much lower level of engagement in general. I half expected it, because it takes time to create a good exercise and I was already spending nearly 5 hours a week at this point. So, unfortunately, the quality of the exercises wasn't as high or as fun as something like Flexbox Froggy.

In the future, I will use one of the many code challenge websites (probably codewars) and in each session, we will do some of the challenges together. Then people can go home and do some in their spare time if they like. It is gamified, so people will be more engaged for sure.

Don't let this put you off

If you are thinking about volunteering, please don't let this put you off! This post is a personal reflection and how I want to improve the sessions. It is still an amazing experience for me and I would strongly recommend it! If that is not enough to convince you, there are the added benefits of something that will give you "moral high-ground" to talk about in interviews :D

Visual Studio Code: the most important 12 shortcuts to get started with

Fri, 23 Aug 2019 10:28:00 GMT

New to Visual Studio Code (VS Code)? This post will walk you through the most useful and commonly used VS code shortcuts. Some of them might take some getting used to, but once you are comfortable using them instead of clicking around with a mouse you should see your productivity skyrocket!

Here's a quick summary of all the commands for anyone who are already familiar with VS Code.

cmd + shift + P - to open up command palette
cmd + P - search for a file in workspace
cmd + T - search for a method in workspace
ctrl + tab - switch between opened tabs
cmd + option + left-arrow & cmd + option + right-arrow - go left or right of currently opened tabs
ctrl + ` - toggle integrated terminal
cmd + B - toggle
cmd + D - select the next occurrence of the currently highlighted text (multi-select)
cmd + shift + L - select all occurrences of currently highlighted text (multi-select)
cmd + option + up-arrow and cmd + option + down-arrow will add cursor above/below (multi-select)
option + up-arrow & option + down-arrow will move the line up or down respectively
option + shift + up-arrow & option + shift + down-arrow will duplicate the current line and insert it at line above or below

If you would want to learn the why behind each of the shortcuts, feel free to keep reading.

`cmd + shift + P` to open command palette

If you don't know already, command palette is built into many software nowadays. The idea of command palette is that it gives the user one central place that allows many possible actions. E.g. if I want to open a file, open up settings, change my theme, switch project, open new workspace you name it. If there is something you can change in VS Code, you can find it or get to it via the command palette.

And no, it is not just geeky developer software and tools. Mac's spotlight search is a pretty good example of this, but I personally use Alfred which is an enhanced version of it.

`cmd + P` to search for a file

In large projects, in order to keep things clean, we often use very nested folder structures. This makes it a nightmare trying to find the file we are looking for. Personally, I think the only time you are allowed to use the Explorer (the files size menu) is when you have no idea what the file is called but remember where it is located. VS Code does a great job at indexing everything, which means file searches are fast and with vague lookups, you don't even need to enter the right filename to get it to show in the results.

Here's a screenshot of me butcher the word "reducer" and it still managed to find it.

`ctrl + tab` switch between opened tabs

When coding I regularly open 3 - 5 files then constantly switch before them, in a way this is good because it means the codebase is more modular. Although it can be quite painful because you got to keep clicking on different tabs. ctrl + tab will let you cycle through all of your tabs. If that is not enough ctrl + shift + tab will cycle through them in the opposite direction, in case you missed a file this way you don't need to loop through everything again.

`cmd + option + left-arrow` or `cmd + option + right-arrow` switch to the left or right tab from current tab

Just like ctrl + tab this is another way to switch between opened tabs. This might be more easy to visualise and remember since it uses the arrow keys.

ctrl + ` toggle integrated terminal

Are you the type of person who uses an external terminal when you are coding in VS Code? Can I suggest that you give the integrated terminal a try? Rather than digging through the many opened apps trying to find the terminal every-so-often when you need it, simply hit `ctrl + `` to show and hide the integrated terminal. The integrated terminal can do everything your normal terminal does. Just as icy on the cake, why not try out ZSH shell which is a much better-improved Bash shell alternative. You can install it from here or if you are on latest Mac OS beta your machine already has it.

You can read more about how to configure it on my other post Use ZSH shell as an integrated terminal in Visual Studio Code.

`cmd + B` toggle side menu

I only develop on a laptop, which has limited screen space. This means when I need to open two tabs side-by-side, I still can't see much of either tab. I normally keep the side menu hidden and bring it up when I need something e.g. explorer, source control and etc. Rather than clicking on specific icons to show hide side menu, this shortcut is a much faster way of making room for the more important stuff on the screen.

There are even shortcuts to switch between the different side menu interfaces e.g. go from explorer to source control. But I'd recommend nailing the shortcuts I'm suggesting here first, otherwise, you might be taking on too much at once.

`cmd + D` select the next occurrence of highlighted text

This is a very handy multi-select tool, personally, I use it daily. Imagine you have the following snippet of code, and you would like to rename all of the matches variable to foundMatches.

const matches = matchPattern(text, pattern);
 
if (!matches) return null;

// for each match, get the parsed cases
return matches.map(match => {
    const dateTimeObj = this.convertMatchToDateObj(match);
    return parseMatches(text, match, dateTimeObj);
});

Normally this will involve you changing the first one then copy and paste it over the other occurrences. Too slow!

This is super helpful when refactoring code, or when you just decided you named something poorly. The other use case if when I'm trying to understand code other people written. For example, I can just highlight the function name then hit cmd + D this will take me to the next place the function is used in the same file. I could just keep hitting cmd + D until it cycled through all of the occurrences in the file, then it'll just stop.

`cmd + shift + L` select all occurrences of highlighted text

This is similar to cmd + D, but it will select all of the occurrences of the highlighted text in the same file. This saves you time from hitting cmd + D multiple times if the text occurs multiples times.

But notice how it also selected parseMatches since it contained match, it is faster but just be careful you are not accidentally changing another similarly named variable.

`cmd + option + up-arrow` or `cmd + option + down-arrow` will add cursor above/below

To continue on from the last two multi-select shortcuts, these two are also very useful when it comes to editing multiple lines at the same time. Instead of modifying the highlighted text that occurs multiple times in the same file, these shortcuts will insert another cursor at either the line above or below. When you start editing one line, the other cursor will also perform the same actions.

Here's an example where this shortcut might be useful.

const var1 = { name: "foo1", address: "bar4" };
const var2 = { name: "foo2", address: "bar5" };
const var3 = { name: "foo3", address: "bar6" };

Say I want to swap the value of name and address, this will not be easy using the other multi-select commands. But with cmd + option + up-arrow and cmd + option + down-arrow it will be a piece of cake.

PS: if the line context doesn't line up very well, you can also try to press and hold option and click to insert cursors.

`option + up-arrow` or `option + down-arrow` move current line up or down

I often see people copy or cut a line of code and then past it a few lines away from its original position. These shortcuts can achieve the same results in much fewer steps and time.

In the above screen recording, I have moved the first time to the bottom and then back up.

`option + shift + up-arrow` or `option + shift + down-arrow` duplicate current line

These shortcuts will duplicate the current line and insert it at the line above or below. There are many ways to make the best out of these shortcuts during your development workflow. I find it most useful when I'm playing around with code trying out a different solution.

Here's an example of me commenting out the original line of code and replacing it with something slightly different. The benefit of this is that I can quickly put back the original line if I wanted to try a different approach from the same starting point.

What should you care?

VS Code is the most popular Text Editor in web development right now. As you are reading this VS Code is expanding to other languages, recently there's been a lot of support for Java added. There are also many web applications which have online editors (Code Sandbox, Stackblitz and many more) that are based on VS Code, so learning it now and be good at it means this stone will hit many birds! Yes, they use the same shortcut as VS Code.

Just to finish off, learning these shortcuts will not only make you look like a god when pair programming or coding in front of other people. It honestly will improve your productivity by a mile. I am so used to them in fact, I can't tell you how many times I tried cmd + D on Chrome browser when I'm searching for something and accidentally bookmarked that page instead (who knew cmd + D is a shortcut for bookmarking on Chrome!).

Disclaimer: by no means are these the only VS Code shortcuts worth learning, they are only my suggestions. I use these shortcuts daily for a few years back when I was using Sublime Text and Atom.

Note: these shortcuts are not Mac-specific, it is just that I don't develop on Window or Linux enough to list them all out easily. You can find the above shortcuts for your platform in VS Code shortcut settings.

Trying out new blogging format

Thu, 22 Aug 2019 21:13:00 GMT

Going forward I am updating how blog posts are structured. My thinking is that it will help people to consume the content and makes it a fun challenge for me. So each blog post will have summary as its first paragraph, then go on to the main body of the content. If possible, I will keep the most relevant and important stuff at the top and the least at the bottom of the page. Hope everyone (the very few people who visit my blog) enjoys the new change.

Here's a more visual explanation:

An interesting background story

I recently watched a video on YouTube which inspired what I'm going to try out next and this post. I am currently looking to get a new iPad and I did what most people would do - search my question on YouTube. I was expecting the same old unboxing and reviewing type of videos from YouTubers whom I follow already because they tend to be big names on the platform. I'm very surprised to have found this video Which iPad should you buy (2019) - Analysis of iPad Pro vs iPad Air vs iPad Mini vs iPad by a YouTuber called Ali Abdaal. I was quite impressed with how he layout his video content.

His video structure:

Quick intro and summary of the video
Conclusion
The main content of the video

Compared to most (if not all) other similar videos, the content is backwards. But I found the video very refreshing. I can't remember all the time I watched half or most of a video only to find I knew most of the stuff anyway and got nothing out of it. The fast the conclusion goes first means I can check what is interesting then proceed if there is something I'd like to learn more about. Also, the YouTuber even added timestamps to allow people to jump to the different sections without wasting their time on things they weren't interested in.

I want to do this with my blogs going forward. I'm very much aware that I often give some background that might not interest most people, but it just feels odd and headless if I don't give an introduction first. Especially given the first paragraph of my blog posts are taken to show on the home page. It would be pretty useful for both people visiting my blog and myself (when I'm trying to remember what I wrote about) if everything can be summarised into that one paragraph.

Final words

I hope this format makes sense and ends up improving the content layout a bit. What very few readers I have, it would be good if I can get some feedback on this. Even if you are just a random stranger who just stumbled on to my blog, feel free to leave a line in the comments :)

I find Ali (the YouTuber mentioned above) quite inspiring, strongly recommend his channel!

My React Hook Notes

Thu, 08 Aug 2019 18:10:00 GMT

I have been doing a lot of reading on React Hook, there's quite the hype for it. The more I read, the more I understood why. It will breakdown existing complex app logic into simpler, flatter architecture. There were some key points I wrote down as part of my reading, which I think will be very helpful later to refresh my memories and decided to upload it on here for others to see as well. It is a digested version of the reading materials I listed in the source section below.

Hooks are a new addition in React 16.8. They let you use state and other React features without writing a class.

No breaking changes

Completely opt-in. You can try Hooks in a few components without rewriting any existing code. But you don’t have to learn or use Hooks right now if you don’t want to.
100% backwards-compatible. Hooks don’t contain any breaking changes.
Available now. Hooks are now available with the release of v16.8.0.
No plan of removing classes
Hooks builds on top of other React concepts - props, state, context, refs, and lifecycle

Must remember

Hooks are JavaScript functions, but they impose two additional rules:

Only call Hooks at the top level. Don’t call Hooks inside loops, conditions, or nested functions.
Only call Hooks from React function components. Don’t call Hooks from regular JavaScript functions. (There is just one other valid place to call Hooks — your custom Hooks. We’ll learn about them in a moment.)

Use the linter plugin to enforce these rules automatically.

React Hook V.S. Traditional React Class

There are many reasons to use Hook, but the most important might to reduce complexity for both machine and people.

People

Reuse stateful logic between components in React has always been a goal and a challenge. We can achieve it with render props and higher-order components. But these patterns require you to restructure your components when you use them, which can be cumbersome and make code harder to follow. If you look at a typical React application in React DevTools, you will likely find a “wrapper hell” of components surrounded by layers of providers, consumers, higher-order components, render props, and other abstractions.

Machines

Class components can encourage unintentional patterns that make these optimisations fall back to a slower path. Classes present issues for today’s tools, too. For example, classes don’t minify very well, and they make hot reloading flaky and unreliable.

// Component with Hook

import React, { useState } from 'react';

function Example() {
  // Declare a new state variable, which we'll call "count"
  const [count, setCount] = useState(0);

  return (
    <div>
      <p>You clicked {count} times</p>
      <button onClick={() => setCount(count + 1)}>
        Click me
      </button>
    </div>
  );
}

// React Class Component

class Example extends React.Component {
  constructor(props) {
    super(props);
    this.state = {
      count: 0
    };
  }

  render() {
    return (
      <div>
        <p>You clicked {this.state.count} times</p>
        <button onClick={() => this.setState({ count: this.state.count + 1 })}>
          Click me
        </button>
      </div>
    );
  }
}

Hook APIs

useState

Call it inside a function component to add some local state to it. React will preserve this state between re-renders.

import React, { useState } from 'react';

function Example() {
  // Declare a new state variable, which we'll call "count"
  const [count, setCount] = useState(0);

  return (
    <div>
      <p>You clicked {count} times</p>
      <button onClick={() => setCount(count + 1)}>
        Click me
      </button>
    </div>
  );
}

useEffect

You’ve likely performed data fetching, subscriptions, or manually changing the DOM from React components before. We call these operations “side effects” (or “effects” for short) because they can affect other components and can’t be done during rendering.

This hook adds the ability to perform side effects from a function component. It serves the same purpose as componentDidMount, componentDidUpdate, and componentWillUnmount in React classes, but unified into a single API.

import React, { useState, useEffect } from 'react';

function FriendStatus(props) {
  const [isOnline, setIsOnline] = useState(null);

  function handleStatusChange(status) {
    setIsOnline(status.isOnline);
  }

  useEffect(() => {
    ChatAPI.subscribeToFriendStatus(props.friend.id, handleStatusChange);

    return () => {
      ChatAPI.unsubscribeFromFriendStatus(props.friend.id, handleStatusChange);
    };
  });

  if (isOnline === null) {
    return 'Loading...';
  }
  return isOnline ? 'Online' : 'Offline';
}

The ChatAPI.subscribeToFriendStatus line is the effect that subscribes to a friend’s online status.

The return inside useEffect would clean up and unsubscribe from our ChatAPI when the component unmount, as well as before re-running the effect due to a subsequent render.

What does useEffect do? By using this Hook, you tell React that your component needs to do something after render. React will remember the function you passed (we’ll refer to it as our “effect”), and call it later after performing the DOM updates. In this effect, we set the document title, but we could also perform data fetching or call some other imperative API.

Why is useEffect called inside a component? Placing useEffect inside the component lets us access the count state variable (or any props) right from the effect. We don’t need a special API to read it — it’s already in the function scope. Hooks embrace JavaScript closures and avoid introducing React-specific APIs where JavaScript already provides a solution.

Does useEffect run after every render? Yes! By default, it runs both after the first render and after every update. (We will later talk about how to customize this.) Instead of thinking in terms of “mounting” and “updating”, you might find it easier to think that effects happen “after render”. React guarantees the DOM has been updated by the time it runs the effects.

Other Tips

Use Multiple Effects to Separate Concerns
Optimising Performance by Skipping Effects

useContext

Accepts a context object (the value returned from React.createContext) and returns the current context value for that context. The current context value is determined by the valueprop of the nearest <MyContext.Provider> above the calling component in the tree.

When the nearest <MyContext.Provider> above the component updates, this Hook will trigger a rerender with the latest context value passed to that MyContext provider.

const value = useContext(MyContext);

A component calling useContext will always re-render when the context value changes. If re-rendering the component is expensive, you can optimise it by using memoization.

Tip: Don’t forget that the argument to useContext must be the context object itself.

Other Hooks

Custom Hooks

A custom Hook is a JavaScript function whose name starts with ”use” and that may call other Hooks.

Let's extract the logic into a custom Hook called useFriendStatus.

import React, { useState, useEffect } from 'react';

function useFriendStatus(friendID) {
  const [isOnline, setIsOnline] = useState(null);

  function handleStatusChange(status) {
    setIsOnline(status.isOnline);
  }

  useEffect(() => {
    ChatAPI.subscribeToFriendStatus(friendID, handleStatusChange);
    return () => {
      ChatAPI.unsubscribeFromFriendStatus(friendID, handleStatusChange);
    };
  });

  return isOnline;
}

Then it can be used in different components:

function FriendStatus(props) {
  const isOnline = useFriendStatus(props.friend.id);

  if (isOnline === null) {
    return 'Loading...';
  }
  return isOnline ? 'Online' : 'Offline';
}

function FriendListItem(props) {
  const isOnline = useFriendStatus(props.friend.id);

  return (
    <li style={{ color: isOnline ? 'green' : 'black' }}>
      {props.friend.name}
    </li>
  );
}

Other Tips

Pass Information Between Hooks
Optimising Performance by Skipping Effects

Sources:

Write clean React code: starting with JSX

Fri, 21 Jun 2019 20:08:00 GMT

React is arguably the most popular JavaScript rending library/framework right now. With its rising popularity and now dominating the job market, many similar JavaScript rendering libraries followed suit and adopted JSX into their work. In case you are new to frontend development, JSX is a JavaScript Syntax introduced as part of React (open sourced by Facebook back in 2015). Now you can find JSX in not only projects that are meant to be similar to React (e.g. Preact) but also things like Nerv, infernojs, Deku, and etc.

If we just look at the downloads in the past 6 months, you can see these projects have a reason to believe React and JSX is loved by the community. Even with the rise of Vue, it is still destroying the competition at the moment.

I have been using React for almost 4 years now, in projects anything between personal side projects to large scale commercial products at work. So then why am I saying we need something that's not JSX? Before we answer this question let's take a look at the syntax itself first.

The JSX syntax

JSX is a syntax extension to JavaScript. We recommend using it with React to describe what the UI should look like. JSX may remind you of a template language, but it comes with the full power of JavaScript. - React Documentation

That might be a bit vague, so here's an example.

// a function component AKA dumb component
const myComponent = ({ title, description}) => {
    const title = 'my title';
    const description = 'a simple description';
    const list = [
        { name: 'item 1' },
        { name: 'item 2' },
        { name: 'item 3' },
        { name: 'item 4' },
    ];

    const listComponents = list.forEach(listItem => (
        <li>{listItem}</li>
    ));
    
    return (
        <div>
            <h1>{title}</h1>
            <p>{description}</p>
            {listComponents}
        </div>
    );
};

If you used React before then you'll know the above example is a simple Function component that displays a title, description as well as a short list. Of course, this is just a very simple example and there are other ways we could create components in React. In this example, it is simple enough that there isn't any opportunity to add in any ugly code. In the next part, we'll look at some common mistakes people make.

Common mistakes

Personally, I'm not a big fan of JSX or what I prefer to call "HTML-in-JS". Even if the tools support solved all the issues from a few years ago, such as weird linting errors, auto-format breaking indentation and etc. What makes it worse is the "flexibility" JSX offers, and unlike Vue or Angular you can seriously write some horrible code that is perfectly functional just impossible to read.

Passing Props

React is all about componentising the application, so everything is in chunk size files which means separation of logic and concern. With this componentisation, what we also have is the issue of passing data around, especially from parent component to their children. When done wrong can be a serious pain to look at.

Check out this example below. This is what happens when a child component requires a lot of prop data from its parent, and all these data are passed down as individual attributes on the component tag.

 return (
    <div>
        <h1>{title}</h1>
        <myUglyComponent
            user={user}
            date={date}
            isActive={isActive}
            shouldDisplay={shouldDisplay}
            selectedDate={selectedDate}
            options={options}
            activePeriod={activePeriod}
            propertyOne={propertyOne}
            propertyTwo={propertyTwo}
            propertyThree={propertyThree}
            propertyFour={propertyFour}
            propertyFive={propertyFive}
        />
    </div>
)

By the way, this is nowhere close to the worst I've seen people do. No joke, I've seen child component with as many as 20 props passed to them. Also, there are multiple of these children components in the same file. To me this is terrible, and we should avoid dumping so much unnecessary data into the return function. It is JSX's fault for allowing this in the first place, but we should know the difference between "okay amount of attributes" and "too damn many to see without scrolling".

Please just create a variable before the return or render and pass in as one prop value instead, see the example below.

const data = {
    user,
    date,
    isActive,
    shouldDisplay,
    selectedDate,
    options,
    activePeriod,
    ...
};

 return (
    <div>
        <h1>{title}</h1>
        <myUglyComponent data={data} />
    </div>
);

I hope everyone agrees that this is much easier to read. People might ask "what about the proptype checking?", well it is no different to proptype checking than other objects passed into a component. Below is an example of proptype checking for nested objects.

import PropTypes from 'prop-types';    

propTypes: {
    data: PropTypes.shape({
        user: PropTypes.shape({
            ...
        }),
        date: PropTypes.date.isRequired,
        isActive: PropTypes.bool,
        ...
    })
}

Conditional (ternary) operator

I love using ternaries, they are concise and to the point. But when used wrongly, they could make the code confusing to read even in plain JavaScript not even mentioning JSX. Now let's see how we can break the spirit of the next person looking at our code.

return (
  <div>
    <h1>{title}</h1>
    {
      isValidComponent
        ? <ComponentOne />
        : <Error />
    }
    { shouldDisplay && <ComponentTwo /> }
  </div>
)

Did you find the above example difficult to understand? Probably not, again this is a simple example. But imagine a component with a few of these in the return logic. I don't think many will argue for this kind of approach. Ternaries should only be used if it doesn't sacrifice readability if anything it should improve it.

Just for fun, here's something else I've seen before, that should only exist in nightmares.

return (
    <div>
        <h1>{title}</h1>
        {
            validData
            ? (
                displaytype
                    ? <ComponentOne />
                    : <ComponentTwo />
            )
            : <Error />
        }
        { shouldDisplay && <ComponentThree /> }
    </div>
)

It is not even difficult to fix. Just like the part, we talked about passing props, let's move some of the logic outside of the return and then see.

const renderComponentOne = if (validData && displaytype) <ComponentOne />;
const renderComponentTwo = if (validData && !displaytype) <ComponentTwo />;
const renderComponentThree =  shouldDisplay && <ComponentThree />;
const renderError = if (!validData) <ComponentThree />;

return (
    <div>
        <h1>{title}</h1>
        { renderComponentOne }
        { renderComponentTwo }
        { renderComponentThree}
        { renderError }
    </div>
)

Deal with complex business logic

Often due to product requirements, we need to add in logic to handle specific cases and scenarios. Over time, this logic would build on top of each other and can become difficult to maintain.

I think it is always important to remember, a component is meant to be isolated from other parts of the application. But this doesn't mean we cannot break a complex component down into multiple files.

Ultilities/Helper Files

There is nothing wrong with having multiple utilities or helper files in a project, not only do we abstract out common code. I have seen both projects maintaining a single file approach to avoid duplication of logic, but I also worked on projects where the team preferred many single logic helper files. It doesn't matter which of these approaches teams go with, the important thing here is that the logic is abstracted out from the over-complicated component.

These helpers will (or should) always be just JavaScript code. This means it is super easy to unit test these functions.

High-Order Components (HOC)

We could have a component that only handles basic presentational logic e.g. render a component. Then we can create a High-Order Function (HOC) component to handle the complex product requirements. This way we have cleaner components, and also this also makes testing a lot easier with this abstraction.

// Higher Order Component
const hoc = (WrappedComponent) => (props) => (
  <React.Fragment>
    <WrappedComponent {...props}>
        £{props.children}
    </WrappedComponent>
  </React.Fragment>
);


// Component
const Price = (props) => <span>{props.children}</span>;

// Call HOC
const ConvertedPrice = hoc(Price);

// App
const App = () => <ConvertedPrice>17.00</ConvertedPrice>;

ReactDOM.render(
  <App />,
  document.getElementById('root')
);

See CSS Tricks: What are HOC for more information.

Ways to improve list loading performance

Mon, 10 Jun 2019 17:41:00 GMT

As developers lists are one of the main things we work with, for no other reason than it is such a powerful data visualisation technique. We use it so much that it is second nature to dump a list of data on the screen. This is fine for users with good hardware devices and fast internet connection. But what about when we don't have the ideal scenario? In this article, we'll talk about some nice techniques we can use to improve user experience with lists on low-end devices or slow internet connection.

Pagination

Pagination in essentially divides up a long list of results in pages, then allow the user to see a subset of the results on each page. This is probably the most common way of dividing content into more manageable chunks so that only a small amount of data is requested, sent and displayed to the user. No doubt this will result in snappier user experience. This practice is easy to implement and easy to handle long lists on web pages. This technique became popular on desktop devices and is still on many sites today when a user might want to skip some content in the middle. For example, going from page 1 to page 5.

Lazy Loading

Lazy Loading is very similar to Pagination since you could achieve both with the same backend API. The difference here is how the user perceives the results and interact with it. With Lazy Loading, the user has to scroll to a particular position and then loading will be triggered and results loaded automatically. This allows the user to continue to browse without clicking on anything. Lazy Loading makes the most sense to be used in situations where the most relevant data is at the top and the user is not expected to want to skip any data in the middle. Or in some cases like twitter or facebook, these companies never want you to stop scrolling, so lazy loading will supply an endless amount of feeds as you continue to zombie-scroll.

(Screenshot taken from https://www.premierinn.com/gb/en/search.html)

Removal of content when it is outside of the viewport

Now we are getting to the optimised list rendering territory. Since on low-end devices, memory is a key limitation, by switching out items for fake ones we are holding fewer data in memory and thus reduces RAM shortages on these devices.

This approach is just as it sounds when the list item is outside of the viewport then replace it with a fake item. When the user scrolls and the item come back into the item, then we'll do the opposite and replace the fake item with the real item again.

It is also recommended to keep 1 or 2 extra tiles off-screen, so when the user scrolls off-screen tiles into the viewport they don't constantly see images being re-downloaded.

A small case study

I am working on a project where we are developing web apps that sits inside a Cordova shell which runs on TV devices. The first thing to know about TV UI is that since these devices are designed to be used from far away, this means bigger UI and Text on the screen, therefore less on-screen content at a given time.

So why is Pagination and Lazy Loading not enough on TVs? Firstly there is the hardware limitation, this kind of apps often face performance issues due to the devices have lower specifications compared to laptops or even mid to high-end mobile devices. When devices try to load a long list, without doing any garbage collection it can consume too much RAM and result in a laggy device. By garbage collecting tiles off-screen, we can effectively reduce the amount of data held in memory, and when the user scrolls the item into viewport again we can simply re-render the item on the screen again.

Pre-fetching Lazy Loading

This optimisation might be useful for users with a slower internet connection. This technique essentially keeps extra data in memory that is not rendered out, and each time the user reaches to the bottom of the page, it will render out the extra data and pre-fetch more data for another future lazy load. The extra data is acting as a buffer. The whole point of this optimisation is to deliver a scroll behaviour that seems to the user where everything is loading in lightning speed. Something that might not be an issue for most people, but might be a nice optimisation to improve people's user journey on slower internet speeds.

For example, we can have a site that initially loads 20 items, but we only show 10 of them. When the user gets to the bottom of the 1 - 10 item list, it will try to show the next 10 items (11 - 20) which we already have data for. Since we already have the data, so we just need to render it out. What the pre-fetching means is that we will load the next 10 items (21 - 30) as we are rending out the items 11 - 20 on the screen. So by the time the user scroll down to the bottom again, we already have 21 - 30 ready.

Final words

Web app optimisation has become an interesting topic for me recently. Partly because I'm working with more low-end devices at the moment where performance is a serious factor to consider. More importantly, these are interesting challenges to tackle, as it requires better design and clever implementations instead of just dump a bunch of data on the screen. I hope this blog post will inspire some more optimised sites!

How to configure Cordova app to filter TV devices by default

Tue, 21 May 2019 14:19:00 GMT

In this article, we'll be covering how to do configure Cordova to automatically filter supported devices. Thus saving you or someone else ticking and unticking hundreds if not thousands of boxes on Android app stores.

If you never used Cordova before, here's an official explanation.

Cordova wraps your HTML/JavaScript app into a native container which can access the device functions of several platforms. These functions are exposed via a unified JavaScript API, allowing you to easily write one set of code to target nearly every phone or tablet on the market today and publish to their app stores.

Basically, it creates a native application wrapper/shell for your web application. This is super neat and has many advantages, of course, here are just some of them on top of my head:

Faster updates - deploy changes without user installing another APK
Use same tech stack as web - instead of dedicated platform teams, you could just have one team that works on all platforms since they all use the same technology
Avoid weird proprietary technology - imagine trying to hire Play Station developers to create an app V.S. web developers.

No doubt this approach also comes with disadvantages too. We'll need to find ways to configure things which is very easy to do on these platforms. The issue we are taking a look today is trying to get our Android package (APK) to automatically filter the supported devices when we upload it to an app store. So far I've only tested this on Amazon's Appstore, so I'm not sure if it works 100% on Google's play store. But I'm quite confident it should work fine.

Instructions

Here's the snippet that we need to insert into config.xml file, in the root level of our Cordova app.

<config-file parent="/manifest" target="AndroidManifest.xml">
    <uses-feature android:name="android.software.leanback" android:required="true" />
    <uses-feature android:name="android.hardware.touchscreen" android:required="false" />
</config-file>

What does is it doing and why do we need it?

config-file tag

<config-file> tag Cordova official documentation states that it allows you to append new children to an XML document tree. The children are XML literals to be inserted in the target document, which in our case is the AndroidManifest.xml. The parent attribute means where about it inserts into the file, thus they will all be inserted inside the <manifest> tag.

<?xml version='1.0' encoding='utf-8'?>
<manifest ... xmlns:android="http://schemas.android.com/apk/res/android">
    ...
    <uses-feature android:name="android.software.leanback" android:required="true" />
    <uses-feature android:name="android.hardware.touchscreen" android:required="false" />
    ...
</manifest>

Notice how the <config-file> tag is gone in the AndroidManifest.xml file after the insertion.

android.software.leanback

This basically tells the app store that our app is designed for TV devices or the "lean back experience".

<uses-feature android:name="android.software.leanback" android:required="true" />

Declare that your app uses the Leanback user interface required by Android TV. If you are developing an app that runs on mobile (phones, wearables, tablets, etc.) as well as Android TV, set the required attribute value to false. If you set the required attribute value to true, your app will run only on devices that use the Leanback UI. - Android official documentation

android.hardware.touchscreen

This should be quite easy to guess, basically says our app is not designed for touch input devices. Which disables those devices by default when we upload our APK.

The app uses the device's touchscreen capabilities for gestures that are more interactive than basic touch events, such as a fling. This is a superset of the android.hardware.faketouch feature. - Android official documentaion

Final Product

Here is what we get at the end, of course, there are probably plenty other XML tags within <widget> I've left them out here just to make sure everything is easy to read.

`<?xml version='1.0' encoding='utf-8'?>
<widget ... xmlns:android="http://schemas.android.com/apk/res/android">
    ...
    <platform name="android">
        <config-file parent="/manifest" target="AndroidManifest.xml">
            <uses-feature android:name="android.software.leanback" android:required="true" />
            <uses-feature android:name="android.hardware.touchscreen" android:required="false" />
        </config-file>
    </platform>
    ...
</widget>

Volunteer to teach people to code

Wed, 08 May 2019 20:33:00 GMT

I volunteered a couple of weeks back and offered to teach whoever is interested to learn to code. It takes place in my local library, which fortunately for me, is very close to both where I live and work. Today, I just had my first session. This is the first time I'm doing something like this, though I'd share my experience and learnings. In case anyone is thinking of doing something similar.

How to get started?

I had this idea of setting something up and teaching people to code but never took it to the next stage. Luckily for me, someone asked on Nextdoor (a social network website for local communities), and I offered to help and volunteer.

Surprisingly, getting it all set up and running were so easy from my point of view. Fiona from the local council emailed me a simple form to fill in, then offered me a number of locations for me to choose from. Other people sorted out the location and finding people who were interested. I didn't have to a thing, apart from going to the library for a quick interview and hand in my form. Before I knew it, it was ready to go!

I am a bit nervous about speaking in front of a lot of people, so I was hoping for a small group. The smaller the better, even if it was just 1 person I would have been happy. Apparently, 20+ people signed up but 18 people turned up today, which is still a huge number of people in my opinion.

The conclusion is that there is so much support around volunteers, it can't be easier to get involved and give something back to the community. Everyone is super friendly, emailing to see how I was feeling after today's session and etc. To be honest, I don't think I worked anywhere where I felt so cared for, haha!

My advice is don't be like me, and wait for something to fall on my lap, which is not a healthy habit for many reasons. If you are interested in volunteering just go for it, I mean what's the worst that can happen?

Learning from each other

Yes, I volunteered to help others. But I know my public speaking ability is shocking, and lack a lot of experience. So I knew there'll be many learning opportunities for me when I signed up for this. Just in the span of 40 minutes, I already noticed things that I wasn't doing very well.

When we were waiting for people to come, I didn't immediately start a conversation, the room was silent and awkward. When I realised this, even before the session started, I went around the room and asked people some simple questions like "why they were here" and "what are their coding skills level" to get a conversation going.
I wrote down notes, fairly detailed notes as well. But I didn't even get to look at them, because there's 18 people staring (okay, just looking) at me at all time. I completely forgot to check my laptop. Stuff I planned to ask, I forgot. Not mentioning things I didn't even think of asking, like asking people for their email address so I could contact them about the next session! But luckily I had people supporting me, they were clearly way more experience at this.
At the start, I keep focusing on one side of the room, it must have been quite obvious for other people. But I only realised after 10 - 20 minutes, so for the rest of the session, I forced my self to look around a bit more and not look at any one place for too long.
Going back to my public speaking point, I noticed myself repeating myself and losing track of the overall direction of the conversation. At one point I had no idea what I need to say or ask, but some BS skills came into play and just kept on going.
The attendees asked a lot of good and useful questions, which not only reminded what I planned to say but also gave me an opportunity to pause and think. So I kept on asking them if they had more questions :D

Hold your ground?

Today, I went in thinking I'm going to run these sessions once every two weeks because I'm worried it will suck up too much of my personal time. But after seeing people's desire to learn, I felt I should try and do more, and ended up committing to weekly sessions. I don't know if this is good or bad, I think I'm happy with my decision. But if I knew I couldn't afford to do weekly sessions, then I hope in that situation I would have held my ground and stuck with what I planned.

Happiness

A final point, maybe even the most important point. I was so hyped after today's session, especially, with people coming to talk to me after asking me a question and saying thank you. Biologically, helping others makes us feel good. So if you want some more happiness, this isn't a bad way of getting some dopamine hits!

How to set up Site Preview with Contentful, Gatsby and Netlify

Wed, 01 May 2019 11:07:00 GMT

It has been a while since I wanted to get preview working for my blog. There is a lot of custom styling on my blog site, and often things don't look quite right when I publish an article for the first time. This often means it will take a few republishes before getting everything nailed. If someone were to visit my site during this stage, it wouldn't have been good user experience. So I was always curious about the Preview function on Contentful. Last night, I decided to take a stab at it after seeing this contentful doc. It looked easy enough, but as always, I encountered some issues. After getting everything working, I thought I'd share my solution on here, maybe it'll help someone else out.

The complete setup flow

This article assumes that you have all the pre-requisites set up and configured already, which includes:

Using Gatsby static site generator
Have a site hosted Netlify with the build passing
Content is hosted on Contentful
All the normal integration between Netlify and Contentful are done, only missing the preview functionality

1. Install Netlify App on Contentful

Click on Apps in the Contentful menu.

Then click on Netlify in the list of supported apps. This will bring up a modal, simply click on "Authorize" button. If you are logged into Netlify already, then it will self-close.

2. Configure sites on contentful

Use the dropdown to chose a site on Netlify you want to build. Usually, it makes sense to have two sites to build production (using the Content Delivery API) and preview (using the Content Preview API) versions of your site.

Once Netlify App has been authorised, the next step is to configure the Production and Preview sites.

I recommend go to Netlify and duplicated the site again, so we have one site for Production build and another for Preview build. This is due to for Preview build, we need to use a different accessToken compared to Production build. Having two sites on Netlify means we can rely on one of them for preview builds without affecting the production site.

If you haven't done it already, I also recommend moving accessToken and other sensitive API data into .env file and then add it to ignored file list (e.g. gitignore) so it doesn't end up in version control.

The image below shows where you can find the API keys on Contentful.

3. Update site code

In order to build preview, we need to modify our Gatsby code a little. More specifically, the configuration file gatsby-config.js.

Add host into the options we pass into gatsby-source-contentful. The value should be preview.contentful.com for Preview and cdn.contentful.com for Production.

See the snippet below for how I've done this.

// .env
CONTENTFUL_ACCESS_TOKEN=xxxxxxxxxxx
CONTENTFUL_SPACE_ID=xxxxx
CONTENTFUL_HOST=preview.contentful.com

// gatsby-config.js
require('dotenv').config()

module.exports = {
    plugins: [
        {
              resolve: 'gatsby-source-contentful',
              options: {
                  spaceId: process.env.CONTENTFUL_SPACE_ID || '',
                  accessToken: process.env.CONTENTFUL_ACCESS_TOKEN || '',
                  host: process.env.CONTENTFUL_HOST || 'cdn.contentful.com'
              }
          }
    ]
}

4. Netlify Content Preview

The way to use Preview feature is by opening up a piece of content, then on the right-hand side you can click on "Build" to trigger a build or "Open preview" to see the page in action.

Issues I encountered

Invalid Contenful spaceId/accessToken

When I try to build using the Preview API accessToken, I get the following error.

Accessing your Contentful space failed. Perhaps you're offline or the spaceId/accessToken is incorrect.
Try running setting GATSBY_CONTENTFUL_OFFLINE=true to see if we can serve from cache.

It is saying spaceId/accessToken is incorrect, but I know for sure both of these are correct because I copied from the API page on Contentful. I tried to check out any documentation on Contentful but found nothing. Surprisingly, there were very few questions asked about this online as well. Maybe because it is still quite new?

It turns out the solution is exactly what I explained in step 3 in the previous section. I had to add host into the options we pass into gatsby-source-contentful. It seems in order to retrieve draft documents from Contentful, you need to hit a completely different endpoint, hmmm interesting.

Failed to create a page

The second error I came across was thrown by gatsby-node.js, complaining that it couldn't create a page because there are a lot of null values.

Your site's "gatsby-node.js" must set the page path when creating a page

{ path: null,
  component: '/Users/hao/Documents/workspace/gatsby-blog/src/templates/blog-post.js',
  context: { slug: null } }
error See the documentation for createPage https://www.gatsbyjs.org/docs/actions/#createPage

After I did some investigation and printed out the code that failed, I realised this error was caused by the fact I had a blank draft saved. The missing mandatory fields were causing all these error when Gatsby tries to create a page for that record.

So if you encounter the same or similar errors, maybe check Contentful contents and ensure all the mandatory fields are filled out.

Final Words

The benefit of the preview feature is that I don't have to trigger a production build to see what the blog post looks sitting on the site. However, the build time is still quite long, so the "write something" and "see the changes" process can still be further improved. But that's a problem for another day!

Docker Cheat sheet

Fri, 26 Apr 2019 20:40:00 GMT

It seems like everyone has their own Docker Cheatsheet nowadays, and I thought I'd add one more into the mix. I've noted down some handy commands I use most often when I work on Docker-related tasks.

Cheat sheet

Status of docker

A quick and easy way to see an overview of docker on your machine. E.g. how many containers and images have been downloaded and their status.

docker info

Build Docker image

This is only required if we want to create a Docker image from a Dockerfile.

docker build .
docker build -t newImage . // tag the image

Run Docker container

Once we create our own Docker image or pull one from a registry, we'd want to run it on our machine.

docker run alpine:3.9
docker run -d alpine:3.9 // detached mode
docker run -it alpine:3.9 /bin/bash // interactive mode
docker run -v "$(pwd)/src:/app" alpine:3.9 // map volume
docker run -p 8080:8000 alpine:3.9 // map port

Stop and remove all Docker containers

Wipe all containers. This is especially handy after playing around with a bunch of ideas on the development machine.

docker stop $(docker ps -aq) // OR docker stop $(docker ps -a -q)
docker rm $(docker ps -aq)  // OR docker rm $(docker ps -a -q)

docker rm -f $(docker ps -aq) // doing it all in one line with force remove

Remove all Docker images

I personally prefer to have no Docker images on my machine unless they are in use. This is a brutal but easy way to get rid of all the random images that are taking up precious storage space.

docker rmi $(docker images -q)

Create personal email address with custom domain

Mon, 22 Apr 2019 20:48:00 GMT

Since Oct 2018, roughly 6 months ago, I have decided to utilise my own custom domain and create my own personal email address with this domain. In my mind, it was always going to be some complex process, and have been avoiding it until now. But recently, I found an easy way to do this and thought I'd share it on here.

Why you might want to do this

Personally, I wanted a way to separate my personal and work emails. It can be quite frustrating to get LinkedIn and other job emails spamming my personal inbox. It just means I have to constantly check my phone all the time. If I had them in separate email addresses, maybe then it is possible for me to turn off alerts for work email after working hours and turn it back on the next day when the workday starts (maybe with some tools it's even possible to automate this). If nothing else, it will help me to keep my personal email private.

My setup

This guide assumes you have a custom domain and it is in use somewhere. For me, I got this domain on Netlify DNS. Getting this done on their site was very simple, I can't say about other services. But hopefully it will be simple enough for you as well!

Step 1. Pick your solution

There are a few ways of getting this done. I've picked the easiest one improv.mx, but I'd recommend you look at the other options. My personal favourites are services such as:

Google Suit which costs money but arguably provides a better and more complete service. Since it includes 30GB Google Drive storage, video conferencing and etc.
Zoho Mail is free for "Up to five users,5GB/User, 25MB attachment limit. Web access only. Email hosting for a single domain". Yes, it is a lot of requirements, but for some people, it might be enough.

If you choose to go with another service Google Suit or Zoho Mail, then the rest of the steps won't make much sense to you.

Step 2. Set up DNS records

The improv.mx website has really good instructions on how to get this done. Just follow the following steps and it will start forwarding your emails.

improv.mx will take a custom domain and alias for that domain (e.g. hello@domain.com) then any email sent to that email domain will be forwarded to a user-defined email address (e.g. user-defined@gmail.com).

I have decided to use my personal gmail address for now, but the ideal setup would be to sign up to another random Google account like 93g93984@gmail.com and use that as the forwarded email address. This way you could have more control over the forwarded emails. An example could be what I mentioned above, turn off the notification for that account altogether when you go on holiday. Just write yourself a little note to turn it back on when you get back!

Step 3. Reply from forwarded email account

This step is pretty key considering there is no way for you to reply to any emails on improv.mx website. It would be pretty weird for people to send email to X email address, and get a reply from Y email address.

How to do this you might ask? Luckily, improv.mx got us covered once again. Follow this guide from improv.mx and get the reply from your custom email setup once and for all.

Notes: There are quite a few steps involved, so I won't include any images here.

Step 4. Set up a filter in forwarded email account

This step is to prevent emails forwarded to this account from going into spam folder. Gmail seems to be very suspicious of this, and placed a few of my test emails to spam folder. I personally have followed this tutorial to get this setup, it was quick and painless. With this done, you will never have to worry your emails getting moved to spam by mistake.

Be warned!

I tested this solution and it worked great! But it is worth noting on improv.mx website they only guarantee 99% uptime. This means when their services are down, and someone send you an email at the same time, you will never receive that email. Unfortunate, I was not able to test this out since it would require improv.mx service to be offline. But my expectation is that, the sender's email will failed to send and they would know about it, and hopefully could contact you through another method.

Final Words

I'm personally looking forward to start giving out my own custom email address to people. It will also be a lot easier to give to people on the phone as well. Hope this was useful!

Web API: appendChild V.S. createDocumentFragment

Tue, 16 Apr 2019 23:05:00 GMT

All the JavaScript rendering libraries and frameworks we use, all boils down to the fundamental web APIs at the end of the day. People are so used to work with the high-level React APIs or Vue syntaxes (me included), we forget the raw and interesting layer underneath it all. But not today!

Today I'm going to show you something that you'll probably never use again in your life as a JavaScript developer, something that will not earn you any new titles. All I can say is that it is interesting and it is fun!

Now let's get straight into it.

How does the DOM work?

The Document Object Model (DOM) is a programming interface for HTML and XML documents. It represents the page so that programs can change the document structure, style, and content. The DOM represents the document as nodes and objects. That way, programming languages can connect to the page. - MDN

Everything that the user sees or doesn't see all exist on the DOM, to add or remove something new on to the DOM we can do something like this:

// create node
const node = document.createElement('div');
// append node to DOM
document.body.appendChild(node);

This is easy enough, but what about createDocumentFragment?

MDN documentation says that createDocumentFragment API (most likely) has better performance than appendChild API. Since a document fragment only exists in memory and is not part of the main DOM tree. This means appending children to it does not cause page reflow, thus gives better performance.

So how much more performant are we talking about here? 10%, 20% or like 100%?! Let's do some experiments.

const totalNumOfNodes = 100000;

const parent = document.createElement("div");
document.body.appendChild(parent);

const t0 = performance.now();

for (let i = 0; i < totalNumOfNodes; i++) {
  const child = document.createElement("div");
  child.textContent = `test 1 - ${i}`;
  parent.appendChild(child);
}

const t1 = performance.now();

const fragments = document.createDocumentFragment();

for (let i = 0; i < totalNumOfNodes; i++) {
  const child = document.createElement("div");
  fragments.appendChild(child);
}
parent.appendChild(fragments);

const t2 = performance.now();

console.log("Rendering took " + (t1 - t0) + " milliseconds.");
console.log("Rendering took " + (t2 - t1) + " milliseconds.");

The results are shown in the graph below, I didn't want to try with even more nodes but it is easy to see the more nodes we add the more advantage createDocumentFragment will have.

Final words

I haven't checked what the popular rendering libraries or frameworks are doing to optimise the addition of a massive number of nodes. But I can't imagine createDocumentFragment isn't utilised to improve performance since people do care and research which has the best performance.

Learning about createDocumentFragment is probably not a very practical piece of knowledge. Most of us would never come across opportunities where this becomes an issue, but hey, if this comes up in an interview you will nail it!

Two practical Webpack performance optimisation

Tue, 09 Apr 2019 18:27:00 GMT

If you have seen my previous blog on Parcel bundler then you'd know my thoughts on Webpack. It is big, complicated and slow. It's big and complex because there are a lot of plugins for it and many ways to set it up as well as customising it. It is slow often due to the enterprise applications it is used in, simply because there are a lot of things going on in these projects than someone's SPA (single page application) personal project. Nobody wants to wait more than 1 min for the build process, here I'll be talking about a couple of practical hacks to boost the Webpack build performance used in a recent project.

Parallelise When Possible

The project is a bit obscure, instead of building and bundling the application into one single minified JS file. Due to the number of region and device supported, we need to build multiple times to satisfy these requirements.

Here is a simple example of the output (we are ignoring device for now for simplicity):

.
+-- index.html
+-- some-other-file.xx
+-- region1
|   +-- bundle.js
|   +-- some-other-file.xx
+-- region2
|   +-- bundle.js
|   +-- some-other-file.xx
+-- region3
|   +-- bundle.js
|   +-- some-other-file.xx

There's a loop that returns an array of configs which Webpack will take and start building all of them sequentially. This is bad, and we need to parallelise this!

If you are not too sure about parallel computing. Imagine someone drops 100 pens on the floor, two people will always pick up these pens faster than one person (assume everyone picks up pens at the same rate). So the more people we can add to this job, the faster it will get done (until you hit the parallel slowdown phenomenon, but that's for another time).

There is a tool called parallel-webpack which accepts the same input as Webpack but runs them in parallel utilising all the CPU threads. Since the input is basically the same, as long as you are providing an array of Webpack configs it will work out of the box with only the following changes:

// instead of doing
webpack --config=webpack.config.dev.js

// just change it to
parallel-webpack --config=webpack.config.dev.js

This alone took our build time from more than 65 seconds to around 35 seconds which is about 46% performance uplift!

If you want to go a step further, according to the parallel-webpack official documentation, turning off statistics also improves performance (version 1.3.0+).

// just change it to
parallel-webpack --no-stats --config=webpack.config.dev.js

Loader Caching

babel-loader caches its results in node_modules/.cache/babel-loader by default. After enabling caching with babel-loader we improved the build performance by another 26%. Although configurations vary among locales, most of the modules are the same. By enabling caching, expensive babel recompilations are avoided for subsequent locales being processed.

I read the above from this blog that claims by caching Babel they were able to achieve a considerable performance gain. I did some simple benchmark tests and here are my results:

Type of build	1st test	2nd test	3rd test	4th test	Average
Without cache build times	35.295 sec	30.551 sec	32.12 sec	30.867 sec	32.208 sec
With cache build times	26.944 sec	21.766 sec	25.93 sec	20.175 sec	23.703 sec

The overall performance increase was 26.3%, the blog above was spot on!

Note: In this project, we only use babel-loader, but there is no reason why this technique cannot be applied to other loaders.

Final words

This was just me scratching the surface of Webpack optimisation, I'm sure there are hundreds of other things I could be doing to reduce the build time further. But since now on average the build time took less than 30 seconds, it wasn't worth me spending more time tweaking the setup further.

For anyone who's Webpack build time is frustratingly slow, I'd recommend them trying out these two hacks. If these don't apply then be sure to checkout Webpack's own build performance documentation.

My blogging workflow

Thu, 21 Mar 2019 19:21:00 GMT

I started to blog around the end of 2015, 3 years later it has been an interesting journey. I think I benefited quite a lot from it and thought I'd share my experience and some of my blogging practices with people who are looking to start blogging or simply want some inspirations.

Why blog

One main reason for me to start blogging at the time was to start building my own "brand". Just like Github account with a lot of open source contributions, this blog was meant to help me along in my career. 3 years later, I can confirm it was a good decision.

For people who are already in a comfortable job that they'll never see themselves leave, there are other reasons to blog too. I didn't have these as goals when I decided to blog, instead, I discovered them along the way:

Improves writing

Surprise, surprise, writing blogs helps to improve written communication skills. Simply blogging a couple of times a month, I can feel my writing getting better over time. At work, I also found writing up documentation or meeting notes a lot easier than before.

Learning

Setting up the blog requires time and effort, but maintaining it is more stressful. I initially tried to blog using an open source software called Ghost Blog. But it turns out it is a lot of hassle to go in and update the software every few weeks. The update process (at the time) required SSH into the virtual server and enter terminal commands. Then I move on to Hexo static site generator and hosting it on Github Pages, I also tried Hugo and finally settled on the current solution with Gatsby.

Would I have learned all this if I didn't blog? Maybe, but that would be very unlikely since there's very little reason to touch these kinds of tech stack at work. Setting up a blog is a personal project in a way that requires nurturing. Thinking of ways to improve it will test and improve your technical abilities along the way.

Some other things I learned as part of the blog set up:

Google Analytics (GA)
Search Engine Optimisation (SEO)
Preserve URL after site migration through redirects
JAMstack and benefits of static sites
And etc

I hope the above reasons are enough to convince people that they should start blogging. Don't worry if you haven't done it before. If you've seen my previous blog posts then you'd know my blog at the start felt more like tweets than actual posts because they were so short. At the start just do the best you can, because with practice you can do it as well as anyone.

Coming up a blog idea

You probably have seen blogs on Medium being shared around like crazy and getting thousands of likes. How can they think of good topics to blog about? What if no one likes what you write about?

My blog heading is called "Hao's learning log" for a reason. I intended it to be a place where I write down my own thoughts, in another word it is for me. If I write something that I care about and no one else finds it interesting, then that's fine. Until today most of my posts don't get any views. But when I look back I can see what I was thinking at a time and how my views have changed over time.

If you really find it hard to think of a cool topic, here is where I get my ideas from:

What are you working on at the moment? Anything cool? Anything different? Challenges? Did you learn anything new? All these could be a blog post, if you start paying attention you might find that you have too many ideas to blog about.
Check out the trending topics online. What is the most popular framework? What is becoming popular? In my experience, the blog posts I got most of my views on are Vue related. That was only because I started blogging about it at the early stages and it scored high on search engine ranking.
If you find a popular topic or discussion and find yourself with a different perspective on it. Rather than writing a long message that boosts other people's popularity consider writing a blog post about it.
Watch YouTube videos, listen to Podcasts, read books and etc. There are plenty of places to find inspirations from.

Write it

Most of the time I do too much thinking and not enough writing, so I recommend others not to be like me. If you have an idea start jotting down points and if you find there's enough content then just write up a draft. If you really don't like the content after that, then you don't have to post it. But most of the time, getting it out of your head is the important first step.

You'll want to do this in a way that is quick and clean. I personally prefer to use Markdown because I will then focus on the writing instead of wasting time tweaking font size, colour and etc. Another reason for using Markdown is because Contentful and Gatsby both support Markdown, so works even better for me.

If you never used Markdown before here are a few nice tools to write in Markdown:

VS Code is a text editor, so it is just as good for writing Markdown content
StackEdit is an online-only Markdown Editor, so great for people switch between machines all the time.
typora is a desktop application available on Window, Linux and Mac (beta).
Notion is more than just a markdown editor, it might be an overkill for some. The only downside is that copying text out of Notion has always been a pain for me. Since their format messes up Markdown links.

Checks

Sometimes it is hard to find someone to proofread your content for you. I am always posting blogs at weird times, and I don't like to bother people. I used to just post whatever I write online, and often I'd find mistakes later. If you are like me, I'd recommend using a tool called Grammarly which could correct some simple grammar and spelling mistakes.

It means I often need to copy what I write around, I am looking to find a good way around this. But so far, this is my workflow:

Write up content in Markdown
Copy it into Grammarly for simple Checks
When I'm happy, copy it into Contentful which is what I'm using to store my blog posts

One thing I'd like to mention which can be stressful is working with images. Because I need to copy my content around a few times, it means I cannot insert the images into the article as I write. I normally add in placeholders for images and then add them in when the content is in Contentful. So I always find there is some assembling work in the end.

With this workflow, I normally find I prefer to write shorter posts that can be done in a few hours instead of a longer article which may span over a few days. It is not impossible, just I prefer to sit down and have everything done in one session.

Deploy

Deployment is very easy with my current setup it is one of the reasons why I settled with it. I won't go in too much detail here but get in touch if you'd like to know more. I've also written a blog post on this already and explained why I decided to change to it.

The tech stack includes:

Github
Netlify
Contentful
Gatsby

When I add a blog post into Contentful and press publish. It will trigger a Netlify build via webhook, which takes the Gatsby blog code and generates a static site using the data coming from Contentful. So apart from the initial setup and some bug fixes here and there. There is very little technical maintenance work.

Netlify build error: vue-cli-service not found

Sat, 16 Mar 2019 11:37:00 GMT

I recently created a project using Vue CLI 3, and after got the project set up to a basic state I decided to add it onto Netlify and get the builds going for future reference (Netlify keeps all previous builds and if they are static sites then it also serves them at generated URLs). I started doing this late at night, and after battling it for a few hours I still didn't get anywhere. It was a pretty stressful situation, I just couldn't work out why I kept getting vue-cli-service: not found errors when Netlify runs the build. There was surprising little help online for this, maybe not many people have started using Vue CLI 3 with Netlify yet? I only managed to find out the reason after more digging and thought I'd share my solution so no one else needs to go through the same amount of pain.

The problem

Here is the full error log from Netlify:

PS: As you can see it is 2:18 am

2:21:14 AM: Build ready to start
2:21:16 AM: build-image version: 324ec043422499a87b63cac1f1dabeefe6dca19d
2:21:16 AM: build-image tag: v3.0.2
2:21:16 AM: buildbot version: 867b17c4ea5ef955db899d19b431b26c9d57e87d
2:21:16 AM: Fetching cached dependencies
2:21:16 AM: Starting to download cache of 254.8KB
2:21:16 AM: Finished downloading cache in 94.089461ms
2:21:16 AM: Starting to extract cache
2:21:16 AM: Failed to fetch cache, continuing with build
2:21:16 AM: Starting to prepare the repo for build
2:21:17 AM: No cached dependencies found. Cloning fresh repo
2:21:17 AM: git clone https://github.com/h-dong/clean-slate
2:21:18 AM: Preparing Git Reference refs/heads/master
2:21:18 AM: Starting build script
2:21:18 AM: Installing dependencies
2:21:20 AM: Downloading and installing node v8.15.1...
2:21:20 AM: Downloading https://nodejs.org/dist/v8.15.1/node-v8.15.1-linux-x64.tar.xz...
2:21:20 AM: 
                                                                           0.0%
2:21:21 AM: 
##                                                                         3.3%
2:21:21 AM: 
######################################################
2:21:21 AM:  75.9%
2:21:21 AM: 
######################################################################## 100.0%
2:21:21 AM: Computing checksum with sha256sum
2:21:21 AM: Checksums matched!
2:21:23 AM: Now using node v8.15.1 (npm v6.4.1)
2:21:24 AM: Attempting ruby version 2.3.6, read from environment
2:21:25 AM: 
2:21:25 AM: ** WARNING **
2:21:25 AM: Using custom ruby version 2.3.6, this will slow down the build.
2:21:25 AM: To ensure fast builds, set the RUBY_VERSION environment variable, or .ruby-version file, to an included ruby version.
2:21:25 AM: Included versions: 2.6.2
2:21:25 AM: 
2:21:25 AM: Required ruby-2.3.6 is not installed - installing.
2:21:26 AM: Searching for binary rubies, this might take some time.
2:21:26 AM: Found remote file https://rvm_io.global.ssl.fastly.net/binaries/ubuntu/16.04/x86_64/ruby-2.3.6.tar.bz2
2:21:26 AM: Checking requirements for ubuntu.
2:21:27 AM: Requirements installation successful.
2:21:27 AM: ruby-2.3.6 - #configure
2:21:27 AM: ruby-2.3.6 - #download
2:21:28 AM: ruby-2.3.6 - #validate archive
2:21:40 AM: ruby-2.3.6 - #extract
2:21:46 AM: ruby-2.3.6 - #validate binary
2:21:47 AM: ruby-2.3.6 - #setup
2:21:47 AM: ruby-2.3.6 - #gemset created /opt/buildhome/.rvm/gems/ruby-2.3.6@global
2:21:47 AM: ruby-2.3.6 - #importing gemset /opt/buildhome/.rvm/gemsets/global.gems
2:21:49 AM: ............................................................
2:21:49 AM: ruby-2.3.6 - #generating global wrappers
2:21:50 AM: .......
2:21:50 AM: ruby-2.3.6 - #gemset created /opt/buildhome/.rvm/gems/ruby-2.3.6
2:21:50 AM: ruby-2.3.6 - #importing gemsetfile /opt/buildhome/.rvm/gemsets/default.gems evaluated to empty gem list
2:21:50 AM: ruby-2.3.6 - #generating default wrappers
2:21:50 AM: .......
2:21:51 AM: Using /opt/buildhome/.rvm/gems/ruby-2.3.6
2:21:51 AM: Using ruby version 2.3.6
2:21:51 AM: Using PHP version 5.6
2:21:51 AM: Started restoring cached node modules
2:21:51 AM: Finished restoring cached node modules
2:21:52 AM: Installing NPM modules using NPM version 6.4.1
2:22:04 AM: > grpc@1.18.0 install /opt/build/repo/node_modules/grpc
2:22:04 AM: > node-pre-gyp install --fallback-to-build --library=static_library
2:22:05 AM: node-pre-gyp
2:22:05 AM:  WARN
2:22:05 AM:  Using needle for node-pre-gyp https download
2:22:05 AM: [grpc] Success: "/opt/build/repo/node_modules/grpc/src/node/extension_binary/node-v57-linux-x64-glibc/grpc_node.node" is installed via remote
2:22:07 AM: added 154 packages from 65 contributors and audited 41523 packages in 13.961s
2:22:07 AM: found 64 low severity vulnerabilities
2:22:07 AM:   run `npm audit fix` to fix them, or `npm audit` for details
2:22:07 AM: NPM modules installed
2:22:07 AM: Started restoring cached go cache
2:22:07 AM: Finished restoring cached go cache
2:22:07 AM: Installing Go version 1.10
2:22:12 AM: unset GOOS;
2:22:12 AM: unset GOARCH;
2:22:12 AM: export GOROOT='/opt/buildhome/.gimme_cache/versions/go1.10.linux.amd64';
2:22:12 AM: export PATH="/opt/buildhome/.gimme_cache/versions/go1.10.linux.amd64/bin:${PATH}";
2:22:12 AM: go version >&2;
2:22:12 AM: export GIMME_ENV="/opt/buildhome/.gimme_cache/env/go1.10.linux.amd64.env"
2:22:12 AM: go version go1.10 linux/amd64
2:22:12 AM: Installing missing commands
2:22:12 AM: Verify run directory
2:22:12 AM: Executing user command: npm run build
2:22:14 AM: > clean-slate@0.1.0 build /opt/build/repo
2:22:14 AM: > vue-cli-service build
2:22:14 AM: sh: 1: vue-cli-service: not found
2:22:14 AM: npm
2:22:14 AM: ERR! file sh
2:22:14 AM: npm ERR!
2:22:14 AM: code
2:22:14 AM:  ELIFECYCLE
2:22:14 AM: npm
2:22:14 AM:  ERR!
2:22:14 AM: errno ENOENT
2:22:14 AM: npm ERR!
2:22:14 AM:  syscall
2:22:14 AM:  spawn
2:22:14 AM: npm ERR! clean-slate@0.1.0 build: `vue-cli-service build`
2:22:14 AM: npm ERR! spawn ENOENT
2:22:14 AM: npm ERR!
2:22:14 AM: npm
2:22:14 AM: ERR!
2:22:14 AM:  Failed at the clean-slate@0.1.0 build script.
2:22:14 AM: npm
2:22:14 AM:  ERR!
2:22:14 AM:  This is probably not a problem with npm. There is likely additional logging output above.
2:22:15 AM: npm ERR!
2:22:15 AM:  A complete log of this run can be found in:
2:22:15 AM: npm ERR!     /opt/buildhome/.npm/_logs/2019-03-17T02_22_14_316Z-debug.log
2:22:15 AM: Caching artifacts
2:22:15 AM: Started saving node modules
2:22:15 AM: Finished saving node modules
2:22:15 AM: Started saving pip cache
2:22:15 AM: Finished saving pip cache
2:22:15 AM: Started saving emacs cask dependencies
2:22:15 AM: Finished saving emacs cask dependencies
2:22:15 AM: Started saving maven dependencies
2:22:15 AM: Finished saving maven dependencies
2:22:15 AM: Started saving boot dependencies
2:22:15 AM: Finished saving boot dependencies
2:22:15 AM: Started saving go dependencies
2:22:16 AM: Finished saving go dependencies
2:22:17 AM: Cached node version v8.15.1
2:22:17 AM: Cached ruby version 2.3.6
2:22:18 AM: Error running command: Build script returned non-zero exit code: 1
2:22:18 AM: Failing build: Failed to build site
2:22:18 AM: failed during stage 'building site': Build script returned non-zero exit code: 1
2:22:18 AM: Finished processing build request in 1m1.487616832s
2:22:18 AM: Shutting down logging, 0 messages pending

I ran the build for at least 5 times, but each time I get the same result. I tried updating dependencies, I tried adding @vue/cli as a dependency, and etc. It was late and I tried a whole bunch other "solutions" which I can't recall now. The moral of the story is that nothing worked!

I tried building locally, it works.

$ npm run build                           

⠋  Building for production...

 WARNING  Compiled with 3 warnings                                                                                                                                 03:09:34

 warning  

asset size limit: The following asset(s) exceed the recommended size limit (244 KiB).
This can impact web performance.
Assets: 
  js/chunk-vendors.29d7585c.js (716 KiB)

 warning  

entrypoint size limit: The following entrypoint(s) combined asset size exceeds the recommended limit (244 KiB). This can impact web performance.
Entrypoints:
  app (734 KiB)
      js/chunk-vendors.29d7585c.js
      css/app.8a47e692.css
      js/app.77cc03b0.js


 warning  

webpack performance recommendations: 
You can limit the size of your bundles by using import() or require.ensure to lazy load some parts of your application.
For more info visit https://webpack.js.org/guides/code-splitting/

  File                                      Size             Gzipped

  dist/js/chunk-vendors.29d7585c.js         716.30 KiB       198.37 KiB
  dist/js/app.77cc03b0.js                   11.87 KiB        4.36 KiB
  dist/service-worker.js                    0.95 KiB         0.53 KiB
  dist/precache-manifest.0eba1e4600aba3b    0.45 KiB         0.24 KiB
  67c18f1ca50ecba92.js
  dist/css/app.8a47e692.css                 6.15 KiB         1.52 KiB

  Images and other types of assets omitted.

 DONE  Build complete. The dist directory is ready to be deployed.
 INFO  Check out deployment instructions at https://cli.vuejs.org/guide/deployment.html

Hmmm... I tried building it using Travis, guess what it works!

So what is the issue then? It must be Netlify! If it weren't late in the night I totally would have spammed Netlify with some (possibly passible aggressive) messages. But I left it and went to bed instead.

The solution

The next day, after some most needed sleep and more digging around I found the issue! As expected, it was simple and stupid.

I asked Vue CLI to generate a service worker for PWA support, inside registerServiceWorker.js it uses process.env.NODE_ENV === "production" to determine if service worker should get into action (do something). Similarly, in Vuex store I am also checking process.env.NODE_ENV === "production" to decide if I should turn on strict mode or not. Naturally, I felt NODE_ENV was important and should be declared as an environment variable on Netlify to ensure the app behaves correctly.

Little did I know Netlify does something special when you add a process.env.NODE_ENV build environment variable.

If you manually set the NODE_ENV to production, any devDependencies in your package.json will NOT be installed when we automatically run yarn install or npm install at the beginning of the build.

So basically since none of my devDependencies were even getting installed, therefore the build failing should be no surprise. I just wish I found and read Netlify's documentation a bit sooner.

Source: Netlify build gochas devDependeices

Web Application Rapid Prototyping

Mon, 11 Mar 2019 02:03:00 GMT

First of all, I just want to say that I love the concept of rapid prototyping!

Why? There are so many reasons, but here are just a few which I value deeply:

Different to normal development - It gives you a whole new perspective to software development, normal expectations and assumptions are not valid here. Testing? Hmmm, maybe later. Team chat about coding standard? No need, we will use someone else's lint rules.
Learning opportunities - It really tests your knowledge of different toolsets, libraries, and services. If you are working with other people, then there is a high chance of you picking up something you didn't know before.
Achievement - Literally build something from nothing in a matter of hours or days.

It might not get talked about a lot, but the use cases are probably more common than you think. It could be creating a proof of concept for a client about a potential project, doing a 24h or 48h hackathon or just get out some crazy personal projects like "implement the next Facebook app over a weekend". Either way, it takes a great number of trial and error before it's possible to narrow down to a set of tools and libraries which will ensure the "rapid" part of prototyping. In this blog, I'll be sharing my favourite tools and set up for such scenarios.

What it means to be rapid?

To be rapid, we need to cut down as much work as possible (duh!). This means anything that has nothing to do with building the actual product (look, feel and functionality) is dead weight. Even with the aforementioned three things, if we can avoid reinventing the wheel when we should. Here are some things to consider:

Don't spend ages setting up a custom webpack build script, project configs or install NPM dependencies manually. Use a CLI or something similar to generate a workable project in minutes, then build the app and deliver value. Don't be afraid of standing on the shoulders of giants!
Use a UI library instead of writing custom styling, or take it to the next level using a component library. Which will even save you time from adding styling to components such as buttons and form elements.
Don't build your own CI/CD pipeline, often it takes time to put together or configure. Time is too precious to be used for these kinds of activities, use an off-the-shelf product. For prototypes, I think 99% of the time they will be more than enough.
Don't build custom APIs, and don't spin up your own Databases. There is also the off-the-shelf solution that are simply too brilliant to ignore.
At this stage often there aren't enough people working on the same codebase, so there should not be a lot of friction. Keep the releases regular and iterate quickly. I recommend using the Github flow for any projects.

Then the interesting question is since there are so many libraries, tools and SaaS(s?), what is the easiest, fastest and most scalable way to create a web app? That is not only easy to set up but also easy to iterate on top of and scale up (if we hit the jackpot!) to be able to serve to more users. Some may question why scalability is a concern here, I would return the question and ask "what if you can get it with this set up for free?" :)

Steps we can't bypass

No matter how much we simplify the process there will always be a few areas we cannot remove completely. These are:

Creating the application - implementing features that differentiates itself from other products on the market, otherwise, there is no need of creating this prototype right?
Production environment setup - the code must be hosted somewhere in one form or another. So that when the user enters an URL they can visit the application.
Release new changes - there must be a way to have one version on the production environment, whilst developing new features. Then when the new feature is ready, release the new feature to the production environment.
Dynamic functionalities - be able to save data and provide a complex and dynamic user experience, most likely through using external APIs and SaaS.

We will cover each of the above in more depth in the following sections.

Easiest way to creating the application

The simplest way to creating web apps is using something like create-react-app repo as a starting point or simply generate a get started project using Vue CLI . I am not an Angular developer, but there is also the Angular CLI tool as well.

In this case, I would use something like Vue CLI. I won't cover too much detail here, because I've already covered it in another blog post. It's a simple 10 steps even if you want to configure everything yourself.

If getting the ideas out there is a top priority, then even consider not including things such as E2E testing. Although, I'd recommend against that if there's a chance of building on top of this in the future. Maybe the best option is to include it in the setup but not add any test until a decision has been made, as it is always easier to remove than to add later.

I have deliberately left out UI and component libraries because there are so many out there, so just pick one that works best for you.

App Hosting

Developers no longer need to pay for app hosting. There are so many free or free-tier services on the market now, so don't waste money on this.

There are a few that comes to mind, but the one I recommend is Netlify. Check out the link for more info, but just to mention a few key features:

Automate build & deployment based on webhook
Each push to preview your changes and a Netlify URL for all your branches
Built-in SSL/TLS and certification (Let's Encrypt)
And many more

The setup process is also very straight forward. Login with Github, then select the repository you'd like to set up.

Enter in the build command (e.g. npm run build) and folder where the app is generated (e.g. dist folder). Voila!

Of course, there are lot more settings for you to custom (like previously mentioned SSL/TLS certificates, custom domain and etc), but this is all it takes to get everything started.

Release new versions

Not much to say here. With Netlify, it automatically creates webhook with Github so that when the code is pushed (to any branch) it could trigger a build. This means when you want to merge a PR, it will also tell you if the code is working correctly.

Dynamic functionalities

With any web application, we will need to use APIs or other means to communicate with a database in order to read and write user data. This is to ensure when the user return, they can resume from where they left off. In other cases, it could be used to provide the user information to improve their experience. E.g. typeahead with a list of countries in a country form field instead of asking the user typing it in manually.

The service I like the most is Google's Firebase. It has so much features baked in, things like OAuth, sign up, forget password and etc. But most importantly, its real-time database. This will allow us to build the application that can change on user's device in real time (hence the name real-time database). For example, if I have a chat app open on two devices. If I send a message on one of these devices then the other device will simultaneously show up without the user having to refresh the page or interact with the app. This is seriously powerful stuff, these kinds of features used to be a headache to build with socket.io.

The setup process for Firestore is very well documented by Google. Check out the official doc. If you are too lazy to click open the link, then TL;DR is:

Install NPM dependency
Initialize Cloud Firestore by adding in API keys and called firebase.firestore() in the app

It worth noting the API keys should be version controlled because it will then forever be on Github or other source control tools. I'll cover the best way to set up Firebase in a different blog post, as it is not quite as relevant to rapid prototyping.

Summary

To conclude, the best way to rapidly set up a CI/CD pipeline, codebase and production environment is by using the following tools:

Vue CLI
Netlify
Firebase

Here's a diagram illustrating how everything fits together.

If you run through the above steps a couple of times, I'm sure it's possible to get a prototype project set up in less than 5 mins (excluding the Vue CLI dependencies build time of course).

Alternative tools

As mentioned already, there are simply too many tools out there for me to include in one set up. Most of them have overlapping use cases or is not as powerful as the tools mentioned above. But I am aware depending on the project idea, the requirement for tools can be very different so here are a few other tools which I've been very impressed in the past:

Contentful - if what you are building requires a CMS, then this is a strong contender in this field. I am personally using this for this blog, and I can say a lot of great things about it.
Apiary - If you do need to create your own custom API for a project but don't want to actually waste time building it. This is a cool way to mock some responses then build the real ones later.
Cloudnary - creating an image-heavy app? Definitely check out this service, told simply it hosts media files for you e.g. images and videos. But they allow dynamically change image size, rotations, cropping and etc via URL param or using their SDK.

I tried to keep the explanation short, so check out their websites for more information. They will definitely sell themselves than I ever could :D

Final words

I'm always looking for new cool tools, libraries and services. If you think I've missed something, then I'd love to know! Get hold of me either by leaving comments here or ping me on Twitter.

I hope this article was helpful, thanks for reading!

A quick overview of creating a new project with vue CLI

Sun, 10 Mar 2019 22:30:00 GMT

Vue CLI is a powerful tool, designed to help developers to set up a base project on which they could start building on top of with extreme ease. It offers a bunch of configuration options which caters for different people's preferences and needs. However, as someone who's new to Vue or web development, it could be scary trying to go through the manual setup. This blog post attempts to change that, by explaining each step and give my personal recommendation for each configuration.

Disclaimer: this setup is just what I prefer, it might not fit everyone's needs. Hopefully, it can act as a guideline for anyone who wants to hear a second opinion.

Note: This blog assumes the CLI v3 has already been installed if you would like to find out more info on how to do that check out the official site.

Setup process

Navigate to the ideal location on your development machine, then run the following command in the terminal:

vue create awesome-app

Then the rest is just selecting how you'd like this project to be set up. Here are the steps and my choices.

1. Select a preset

I have decided to stick with my custom setup, so I selected "Manually select features"

2. Check features needed for your project

Here are my justification for my choices and why:

Babel - transpiles JavaScript for better backwards support. I would even urge people to consider not selecting this, especially if the older browsers support is not a requirement.
Typescript - I did not pick this because this is a small prototype app, and I did not want to spend time defining types and etc. But for a large project, I would recommend selecting this option.
Progressive Web App (PWA) Support - this is a no brainer, as more browser adopt this standard more and more people will start using this. It is important to have the shell in place even if you don't need to work on it right this minute.
Router - is your application going to be a single page application? Or would it benefit from defining different views? The choice is yours, but having this would make the app scale much better in the future and saves troubles of re-introducing this later.
Vuex - similar to the above point, depends on the complexity of the app. But because I tend to link applications to a real-time database so I decided to include this.
CSS Pre-processors - I hate writing in CSS because it forces you to style the application in a repetitive manner. So I would always suggest to turn this on unless the application requires no styling. E.g. if you are mostly using a components library
Linter / Formatter - Linter helps to keep code consistent throughout the project, so I'd always recommend this no matter the size of the project. In many cases the linter will be able to auto-fix linting errors, so including it has almost no drawbacks.
Unit Testing - unless the application is for demo only, otherwise I'd strongly recommend having unit test in place.
E2E Testing - similar to the point above, depends on the use case but strongly recommended.

3. Use history mode for router?

I prefer history mode because it doesn't use the hash in the URL hack e.g. domain.com/#/. But be aware you may need to tweak some file path on production.

4. Pick a CSS pre-processor

I have gone with Stylus because it is pure node.js and doesn't require Ruby to run node-sass for compile SASS/SCSS. But this is just personal preference.

5. Pick a linter / formatter config

I have gone with "ESLint + Prettier", because I can install the Prettier plugin and it let take care of everything. I've tried to go with something like "ESLint + Airbnb config" in the past. But the IDE/Text editor config always clashed with the ESLint config which can be annoying.

6. Pick additional lint features

I selected Lint on save here, but selecting the other choice or not selecting any will also work fine for me.

7. Pick a unit testing solution

Jest wins here hands down, no competition. Having used both, I think Jest is better in every way.

8. Pick a E2E tesing solution

Either of the two options is fine, but I've gone with Cypress because I've used it before.

9. Where do you prefer placing config for Babel, PostCSS, ESLint, etc.?

I'd say always keep your configs separate. It is just simpler, cleaner and easier to find.

10. Save this as a preset for future projects?

This doesn't really matter, but I selected no.

All Done

Once the above steps are done, which will take a while the first time because it needs to pull down all the dependencies, you will be able to serve the application using npm run serve.

Final words

The purpose of this blog post is to give an overview of the manual configs when using Vue CLI. I tried to keep everything simple without going into too many details. If you disagree or have any questions, feel free to get in touch in the comments section below or on hit me up on Twitter!

Which Node.js package manager tools to use?

Tue, 29 Jan 2019 19:00:00 GMT

Everything in JavaScript world moves fast in comparison. Libraries and frameworks often fighting hard to dominate one another, in order to win the hearts of developers. Today let's discuss the positions of npm, yarn and a challenger called pnpm. Both yarn and pnpm were announced around the same time, only difference being Yarn was backed by Facebook and it gained traction much faster.

Performance

It is probably safe to say that pnpm > yarn > npm based on the stats released by pnpm official site.

Of course there could be biases here, but the source code for the benchmark was released on Github so anyone can go and take a look.

If you are working on a personal project that has a lot of dependencies, and build time is painfully long. Then pnpm could be a valid choice here.

Update (3rd Apr 2019): It seems like Yarn has also added the ability to do package security auditing, see offical doc. So there is even less reason to pick npm over yarn.

Features

pnpm has been compared to npm and yarn in these articles written by the author of pnpm , the articles are pnpm vs Yarn: monorepo node_modules and Why should we use pnpm? respectively.

It seems all three solutions offer similar key features, such as generating a lock file to ensure consistent dependency creation across different machines and environments. They are ultilise

Security

NPM security is a much bigger concern nowadays especially with the incidents such as "event-stream@3.3.6 (Sep - November)". So I have seperated it out from the Features section, to discuss this on its own.

There are currently two ways of getting notified of package vulnerabilities:

npm has some basic checks baked into npm audit (free)
Snyk is a open source tool that does what npm audit does plus more (free only for open source)

Even though npm hasn't got the best performance, but it is the default node package manager (it's even in the name) used across platforms. For example, if you use npm then Github can alert you when they detect vulnerabilities in our package dependencies. This feature is not avialable for yarn or pnpm. One of the main reasons I'd consider npm would be the ability to quickly and easily audit dependencies.

Of course the same could be achieved if Snyk is used somewhere in the pipeline, one might even argue this provides better audit than npm's solution. If, however, a project is private (not open source) then this would become more costly ($599+) depending on the team size. For a company that's able to make revenue off the product this might not be a very high cost, but for individual developers creating private personal projects this could be too much to consider. Then npm would be the only viable solution.

Maybe it is worth using npm on the side of yarn or pnpm just for its npm audit feature? It is worth mentioning the main reason to move to another package manager is for its speed improvements. Since npm audit requires package-lock.json to function and building it is very time intensive. If npm was added to a build pipeline purely for its security audit feature then the overall build time would be increased instead of decreased. At this point, it might be worth just taking a hit and say it is okay for the build to take 10 - 30s more, by getting some better security check in return.

Final Thoughts

Out of these three options, my personal recommendation is to use npm . Not only because it comes with Node by default, which saves an additional step when you are setting up your machine but shortens CI pipeline builds. Since it is widely used and seen as the default package manager, so you do not get some of the benefits e.g. npm audit and Github's vulnerabilities alerts.

When there are so many moving parts in the mordern frontend application already, it feels nice to not stress about another thing when developing software.

Let's embrace modern browsers in 2019

Wed, 09 Jan 2019 23:07:00 GMT

Anyone who's been working with frontend web development should know the pain of browser support. I never understood why people stick with older browsers and I might never find the answer. I'm sure many fellow developers have the same confusion. Apart from corporate machines, there is very little reason to stay on older browser versions. Modern browsers are both more secure and faster! Need more reason? You get all that for free and there are plenty to choose from, what more can we ask for? For us, developers, nothing about this makes any sense.

It is frustrating, but there is nothing we can do about it. Because our manager or Product Owner told us that we need to support "that version of x browser" (normally IE) because that is what the business wants. So here we go again, spending 20% of our effort developing a new feature then spend the other 80% making it work on browsers which even their own creator decided to drop support for a few years back.

I know plenty of people have been pushing for this online for years, but this year is better than any to actually make a difference (how little it may be).

Better developer experience

Let's get the bias out of the way first. Yes, I'm a developer, of course, I want to develop against fewer browsers and have an easy life. But how many people actually enjoy spending hours, days or even weeks debugging an issue on older browsers that may affect 0.1% of the customer?

Even though this is a valid point, maybe don't go to your boss with the only explanation. Let's look at a few more reasons that don't sound like a selfish ass.

Build a better product

I believe in full-stack development, so plenty of people have tried to convince me why I should focus on one thing. When I go on explaining my reasons, they always come back with the following phrase. I think it works much better here.

Jack of all trades, master of none

I have worked on projects where they wanted to support as many browsers as possible, sometimes up to 30 browsers & devices combinations. It is easy to make that decision to say yes we will support everything. Let's put development and maintenance effort aside, and just imagine the amount of testing required to ensure the quality of those browsers/devices.

I hear there are plenty of automation tools which can make it very easy to do these kinds of tests. There are, and I have used quite a few in the past. They are better than nothing, but they are not perfect either. Not only are UI tests a lot slower to run on the cloud, but they also only support a core list of browsers/devices.

Before any production releases, some teams prefer to sanity check everything. Do you know how long it would take to manually sanity check 30 browser/device combinations?

Spreading the team's effort this thin, and causing a lot of frustration in a lot of situations is plain unnecessary. If the business makes 50% of the revenue on IE10, then I say go ahead and support it. But sometimes it is worth taking a look at the big picture. Spending 10% or even 20% of the resource to support 1% of the customer is not a rare thing business decides to do. If this is happening in your team, it is worth bringing this up and at least discuss it.

On the other side, now imagine all you have to do is support 5 browsers (latest versions). The team would have so much more focus to build a better product. Instead of having a backlog full of bugs on different browsers, and creating a half-baked product. Wouldn't it feel better to see a backlog of features and no bugs?

Honestly it is time

Most applications written nowadays are still converted and polyfill to ES5 standard. Do you know when ES5 spec was released? 2009! That is 10 years ago!

Okay, I need to calm down. I'm not saying today onwards everyone should be publishing app written in ES2018, that is a tad extreme. But is ES2015 (ES6) really that much to ask? Don't trust me when I say it is fine to write and publish apps in ES2015? Let's look at the facts!

Check out this awesome tool created by kangax. It shows a full breakdown of the supported of different browsers. Let's assume that our intended user is using the modern browsers, the lowest support % is Edge at 96%.

Yes, we are not at 100% yet. You really want to use those missing features, then release your code in ES6 and polyfill that 4 % of ES2015 features. I personally think it is just easier not to use those features, but hey, the choice is yours.

Handle old browser visits

What if a user on an older browser comes to my site? Would they just see a broken page? Surely that can't be good, there's a brand name to live up to! Good, let's see what we can do here.

There is this site called outdatedbrowser.com. Go and take a look, it is a beautiful site. This is free, responsive and available in many languages. It only appears when people browse your site on the older browser and tell the user that they should consider download one of the listed modern browsers to continue using your site.

With this as a fallback, your modern browser users will experience a faster site and the older browser users will be greeted with a nice looking fallback site as well. I think the first step towards a more modern web is to have as many sites showing this as possible. When a user keeps seeing this page, they will understand how fast behind they are and will want to download one of those nice looking modern browsers. If not, then sooner or later they will be forced to if they want to see the content on these sites.

If we keep tailoring for the small percentage of people, we might be still using ES5 after another 10 years. For some people, some tough love is needed. But remember, by upgrading they are also more secure and better protected online as well.

Fight for a better web

I also think the first thing any developer should say after seeing the list of supported browsers (and maybe devices) is "can I see the analytics of the site which decided this list"? Yes, analytics is key. It might not be 100% accurate, but it should be the first thing to look at when considering to support older browsers. As developers, people should be coming to us and try to convince us if a browser is needed e.g. over 10,000 users use it to visit our site. It shouldn't be us convincing the business. But since we don't live in a perfect world, at least ask the above question. It is not just making your own life easier, it is also saving the business tones of resources which we will get to later.

After you've seen the latest analytics numbers, the next step to make some the tough decisions or try to convince some difficult people in the business. Explain to them the different benefits and fallbacks that can be put in place to handle everything gracefully even if someone's grandparents load the site on IE6. Yes, the outdatedbrowser.com tool works on IE6! How many other sites out there works on browsers that old? With this tool, you are technically improving the customer experience for people with old browsers (say there is someone crazy enough to be using IE6 visiting your site).

A faster web

We all know small JavaScript bundles are good, it means the site will load faster on low-end devices. ES2015 lets us write much better code to achieve the same things in ES5, there are also plenty of new features of ES2015 which doesn't exist in ES5 which is why we add in those huge polyfills.

But how much small is ES2015 bundle compared to ES5?

This is a tricky question to answer because it honestly depends. But just as an example, let's take a look at the source code of two implementations of todomvc. One is written in ES2015 and the other in ES5.

Things to note:

It was not implemented by the same person, so the code quality may vary
This is just for fun, don't quote me based on this example
Try doing a comparison on the project you are working on, it will be much more relevant than this example

I combined all the source code of the ES2015 vanilla and then minified it, the uncompressed size is 12,074 bytes, gzip compressed size is 3,190 bytes.

I then did the same for ES5 vanilla and the uncompressed size is 16,085 bytes, gzip compressed size is 4,104 bytes.

The actual difference here is pretty tiny especially after gzip, but the percentage difference (ES2015 is about 75% smaller than ES5) is nice to see. It is worth remembering this is only several hundred lines of code. In a big problem, 25% of bundle size is quite considerable. Especially considering todomvc doesn't use any polyfills, if you use a few big polyfills then that's going to add a lot more to the bundle size.

We are professionals

If you are walking past a stranger and see their bag is open, wouldn't you tap them on the shoulder and tell them? You are all nice people, of course, you would. Now, you see your company wasting time and effort on some poor decisions made a while back, the solution is simple and makes everyone's life easier. Should you not raise your hand and make everyone aware of the issue?

Often the people making the decisions doesn't know they are wasting resources on things that don't matter. As professionals, we should stand our ground and say what we think is right. Maybe do it in a polite and assertive way, so people don't think you are crazy or you are attacking them personally. I'm sure most bosses would love to cut 20% cost and give up 1% of the customers, it is simple maths. But remember we are not "giving up" on those customers either, we have discussed fallbacks already which will help them to "find their way".

As professionals, we should and I'm sure most of us want to create the best product. It is also a much better ego boost when people say "oh I love your product!" compared to "can you fix x, y and z issues on your site? I keep getting them".

Final words

I hope the above is enough to convince you that this is something that is worth doing, not just to make our developers' lives easier. It will also benefit our companies and our customers. But if you can't get this point across to anyone at your workplace no matter how hard you tried. Then at least do this on your personal sites :)

I hope 2019 is the year ES2015 will become the default standard when creating new projects, and that ES5 polyfill becomes a thing in the past!

Adding code syntax highlights to gatsby blog

Mon, 24 Dec 2018 16:52:00 GMT

I was recently learning about Mobx and came across a blog with nice code syntax highlighting, and thought the code snippets on my blog are both hard to read and pretty boring to look at. So it inspired me to upgrade my blog!

Here's the blog post.

A quick comparison

The blog I saw

The blog post is a very nice read, but what caught my eye and made me save it into my notes is the awesome looking code syntax highlighting on this site. I've attached a screenshot below.

Code syntax highlighting is of course not a new thing, it is seen on a lot of blog site. Some even go as far as including the jsfiddle (or equivalent) code snippet which lets users click through and run them when reading the content.

For me, the colours really caught my eye. The bold and standout colour makes identifying code and normal text that much easier. I also really like the green padding on the left of the code snippet.

My blog - Before

Just as a reference, this is how my blog looked before I made any changes.

The theme was based off Cho's work. It heavily uses different shades of grey, which meant the site looked clean and minimalistic, but code in the text is not so easy to read.

My blog - After

After some tweaking, I came up with this.

The result is a combination of the blog I mentioned above, Prism's "tomorrow" theme and some of my own custom modifications. It definitely made everything pop a bit more, but it could be a bit too much. I'll test it out for a while, and may come back and dial it down later.

How it was done

Using gatsby the whole process is pretty simple, with a few caveats.

I mainly used gatsby-remark-prismjs plugin to do all of this. I considered of integrating everything myself manually. But I think it would lead to a messier codebase. So at the end, I decided to stick with this plugin.

Installation

This is quite straightforward:

npm install --save gatsby-transformer-remark gatsby-remark-prismjs prismjs

Then just add the following plugin config into the Gatsby config:

// gatsby-config.js

plugins: [{
  resolve: `gatsby-transformer-remark`,
  options: {
    plugins: [{
      resolve: `gatsby-remark-prismjs`,
      options: {
        classPrefix: "language-",
        inlineCodeMarker: null,
        aliases: {},
        showLineNumbers: false,
        noInlineHighlight: false,
      },
    }],
  },
}]

Understand the configs

First of all, let's run through a quick explanation of the above configs which wasn't clearly explained by the documentation:

classPrefix will default to "language-text" if not set which doesn't add any code syntax highlighting, so just leave as it is
inlineCodeMarker still not sure what this does, but you can still use the inline code with single backticks
aliases in case you don't want to keep writing long language names e.g. "js" instead of "javascript"
showLineNumbers I recommend keeping this to false. I tried setting this global setting to TRUE, then override it to FALSE in specific situations. But it did not work. So it seems it is only possible to override to TRUE, for now, so leave this as FALSE unless you want line numbers in every code snippet.
noInlineHighlight This allows you to highlight certain lines in a code snippet which might be useful, I have turned this on thinking I may use it in the future. Note: this will conflict with showLineNumbers in terms of CSS styles, so if you want to use this, make sure CSS works in both situations.

Where to include the CSS

The documentation wasn't too clear on this, I tried to add Prism theme into gatsby-browser.js but it did nothing for me. The file I needed was in fact src/layouts/index.js, after importing

import 'prismjs/themes/prism-tomorrow.css';

at the top code syntax highlighting was showing up correctly.

If you wish to use line numbers then you'll need to import the following CSS file as well.

import 'prismjs/plugins/line-numbers/prism-line-numbers.css'

Wait, we are not done just yet. If you want to use showLineNumbers and noInlineHighlight then you'll have to include a custom CSS file which holds the following overrides as well. For me, since I'm using Prism's "tomorrow" theme, I've included the following (depending on your theme you probably want to change this) and I've also decided to use SCSS here.

import  '../styles/prism-override.scss'

/**
* Add back the container background-color, border-radius, padding, margin
* and overflow that we removed from <pre>.
*/
.gatsby-highlight {
  background-color: #2d2d2d;
  border-radius: 0.3em;
  margin: 0.5em 0;
  padding: 1em;
  overflow: auto;
  border-left: 0.5em solid #FF9800;

  /**
  * Remove the default PrismJS theme background-color, border-radius, margin,
  * padding and overflow.
  * 1. Make the element just wide enough to fit its content.
  * 2. Always fill the visible space in .gatsby-highlight.
  * 3. Adjust the position of the line numbers
  */
  pre[class*="language-"] {
    background-color: transparent;
    margin: 0;
    padding: 0;
    overflow: initial;
    float: left;
    min-width: calc(100% - 3em);

    &.line-numbers {
      padding-left: 3em;
      padding-right: 1em;

      .line-numbers-rows {
        right: calc(100% - 25px);
        left: unset !important;
      }

      .gatsby-highlight-code-line {
        background-color: #455A64;
        display: block;
        margin-right: -1em;
        margin-left: -1em;
        padding-right: 1em;
        padding-left: 0.75em;
      }
    }
  }
}

A quick Parcel bundler review

Tue, 04 Dec 2018 19:20:00 GMT

I heard about Parcel about a year ago, back then I didn't feel like it was stable enough to be used in my personal projects. And that I was just getting into RollupJS, compared to Webpack that was already a massive step up in terms of ease of use and simplified configuration. A year later, now I am finally ready to try out Parcel and see exactly how simple it is compared to the established alternatives like RollupJS and Webpack.

My Experience With Parcel

Parcel is one of the newest contenders in this field, it aims to shake off all the dead weight that comes with a bundler and hopes to enable developers to jump on a new personal project with ease. For me personally, it is stressful to spend a couple of hours or even half a day configuring bundler and build tool when I think of a new personal project. Yes, I could just do it once and then copy and paste a generic setup. But it means using out-dated dependencies, have generic code that does nothing and inefficient setup, to me that sounds like a horrible idea and I wouldn't want to start a greenfield project on top of a brownfield setup. So when I created the software-developer-career-roadmap project, knowing it will be a simple Single Page Application (SPA) I decided to try out Parcel.

Set up

The initial experience was amazing, everything was set up and ready to go with one command npm install parcel-bundler typescript shiva. Of course, only parcel-bundler is relevant to this blog.

Hello World

The next step is to add an index.html file and an app.js file just so we have something to serve. Feel free to take a look at the repository mentioned above, but I'm going to add an example here too just to keep things simple.

// index.html

<!DOCTYPE html>
<html lang="en">
    <head>
        <meta charset="UTF-8">
        <meta name="viewport" content="width=device-width, initial-scale=1.0">
        <meta http-equiv="X-UA-Compatible" content="ie=edge">
    </head>
    <body>
        <script src="./app.js"></script>
    </body>
</html>

// app.js

const helloWorld = document.createTextNode('hello world')
document.body.appendChild(helloWorld)

Serve

With Parcel it is super easy to serve the simple web app we just created, simply run parcel serve src/index.html or alternatively you could also consider using parcel watch and parcel build depending on the desired outcome.

Challenges

During my first few hours with Parcel I encountered 2 issues, which could be small or massive depending on what kind of projects you are working on.

I couldn't get Hot Module Reload working. The official site mentioned that Parcel has Hot Module Reload built-in, however, I wasn't able to get it working at all. In fact, I couldn't even get the browser to reload on file change. To see the changes I've made, I always had to refresh on the browser which did get annoying after a while.
No easy way to use a JSON file as a mocked service call. For the SPA that I was working on, I didn't want or need to create a real backend so I decided to mock it by requesting a local JSON file directly. This way I can have the code logic to behave as if a proper API call was made, but still, keep the data with the repo (where I want other people's contribution in as well). The challenge here is that when I serve the app, a dist folder gets generated and all the files within it created from the source code. But I didn't have an easy to move the JSON file into dist folder. I tried a few different methods e.g. include JSON in HTML and etc, but nothing worked well enough. In the end, I didn't have a choice but to change the JSON file into a normal JavaScript object and remove the fake API call entirely.

Final Words

Going forward, I'm definitely going to be using Parcel as much as I can. But where does this leave Webpack, RollupJS and other bundle tools?First of all, stop using other bundle tools in new projects right now! Not that they are not good, but Webpack, RollupJS and Parcel are just that much better in their own unique ways.

Webpack is really good for an enterprise application that is complex, requires a high level of customisation and aiming to scale. There's a massive community behind it, with popular projects like VueJS and React all utilising it in their starter projects. There are a countless number of Webpack plugins to pick from, if there's something you want to do, the chances are there are multiple plugins to pick from. Another benefit is that, since there are so many people using and creating plugins for Webpack, you can be assured there are plenty of resources online.

RollupJS is still a solid choice in my opinion, mostly used for creating libraries instead of an application. "Tree-shaking" was one of its main competitive advantages when it was introduced, at that time Webpack did not support this feature or it was very difficult to achieve even if it was possible. Rollup is still very simple and easy to use compared to Webpack, but still, have the right amount of packages to support it. When creating a library the most important things are tree-shaking, code splitting, minification and creating code bundles. By allowing multiple entry points, and utilising its plugins Rollup can all of those things with ease. Webpack is a bit of an overkill, in this case, adding additional unnecessary complexity. Whereas, Parcel lacks some of those plugins to achieve the same outcome.

Parcel out of the box requires no setup, and I mean no setup. There is no need to create configuration files for each environment, not even a single line of configuration. But at the same time, it only will let you do the basics, and there isn't a long list of plugins as of today. My experience with it is that it does the core part of bundler's job very well. Getting started with a new project cannot be easier. However, as soon as you need more customisation (the one I explained above) you'll have doubts about choosing it over Webpack or Rollup. I would definitely keep using Parcel, it is amazing at what it intends to do. I would happily live with the challenges it brings then to add 5 - 10 additional NPM dependencies and create a build process that is both slow and complex. My recommendation is to only use Parcel is simple or small projects, there's a high chance of migrating to another bundler tool if you do decide to use Parcel on a medium or large project.

Additional notes

Parcel usage is still tiny compared to the alternatives listed in this blog. Check out the trends on NPM: https://www.npmtrends.com/parcel-bundler-vs-webpack-vs-rollup

Or if you are too lazy to click here are the diagrams:

Improving CSS performance of Cordova apps on Android TVs

Sat, 17 Nov 2018 20:16:00 GMT

I have recently been working in a team that creates web apps that sits inside a webview wrapper application on Android TVs. Coming from more of a web background, even though this is still using React and other common web technologies, there are still some interesting challenges.

If you have read the title of this blog, then you know we'll be talking about CSS performances on Android TVs. This is not a new topic, since performance is an issue on all low end spec devices, but the ones comes with TV are a bit different.

App architecture & Webview

To understand the problem, let's first take a look at how everything what we are dealing with. Essencially, there is a shell app which was created using Cordova which loads the web application into the view. Cordova, if you are new to it, connects up JavaScript events to native Android events and sensors. This means with Cordova it is now possible to detect things like "if the device has internet connection and etc".

We are not too concerned with the Cordova shell app, but it is worth knowing it uses whatever version of Webview that comes with the OS. Different versions of Android has different version of Webview, which makes sense since Android will just include the latest Webview when a new version of Android is released.

Since Android 4.4 (KitKat), the WebView component is based on the Chromium open source project. WebViews now include an updated version of the V8 JavaScript engine and support for modern web standards previously missing in old WebViews. New Webviews also share the same rendering engine as Chrome for Android, so rendering should be much more consistent between the WebView and Chrome. In Android 5.0 (Lollipop), the WebView has moved to an APK so it can be updated seperately to the Android platform. To see what version of Chrome is currently used on a Lollipop device, simply go to Settings < Apps < Android System WebView and look at the version. -- Chrome Team

Based on the above description by the Chrome team, this means the user could actually update the version of Webview. But in reality it is much safer to build for the default version of Webview since not many users won't be updating to newer versions. Luckily, all the TVs my team support are Android 5.0 (Lollipop) or later.

So why am I talking about Webviews, isn't this a blog on CSS performance? To understand the root problem and the solution, it is important to understand how Webview fits into all this. Next we'll be talking about the actual issue and solutions, and hopefully this will make more sense by then.

What is the problem?

I was working on a new feature which allowed user to browse and find the right content by using a grid navigation. The grid will scroll vertically depending on if the user wishes to see more content (no horizontal scroll). When the user tries to scroll to a row that is outside of the current viewport, there is a nice shifting transition that brings all the tiles upwards or downwards. Then the focus maintains on the new tile and the new row will be within the viewport.

After the implementation has been done, the feature was tested on one of the highest spec TVs and I noticed some weird behaviors.

Verticle navigation is way more jumpy and laggier than horizontal navigation
This issue seems to be only occuring on the TVs, on our development machines (Mac) everything works fine

It didn't take long for me to conclude that this is a performance issue.

The solutions

After some research and investigation using the Chrome browser tools, I couldn't find anything suspicious. However, I came across some discussions around how some people have tried to trigger GPU rendering instead of CPU for better CSS animation performance.

There are two ways of achieving this:

TranslateZ(0) hack: It is normally used in CSS to translate a component on the Z-axies as the name suggests. A less well-known functionality is that even by declaring 0 translate (no change) it will force hardware acceleration and thus render with the GPU instead of CPU.
will-change: prepares the browser to complete optimizations before an element is actually changed. As a result, the transition will appear much smoother to the end user even on low end devices.

After testing both of the ideas above, everything looked the way I expected. So which of the two solutions should we pick? The normal thinking might be that since will-change is designed for this and TranslateZ() is a hack, it only makes sense to use will-change right?

Well, I actually went with TranslateZ() due to the following reasons:

Using will-change too many times in the code will actually cause enough overhead to decrease the performance of the web app. The recommended approach is to add will-change on the element right before the animation, then take it off the DOM again. Which requires a lot of additional logic in place to add and remove it from the elements, which in this case there quite a few.
The other reason being will-change was released not too long ago, so the support is a bit lacking in older browser version. If there happens to be a TV with older Webview then this optimisation apply.

It is worth to point out that over using TranslateZ() would also lead to performance issues. Each time it is used, it will move the applied component to its own composited layer. So even if you do decide to use this hack, try to keep it at a minimum. Overall, based on my research I felt that this would be less of an issue compared to will-change.

To learn more about will-change checkout css will-change property by Opera and re-rastering composite layer by Google.

BUT WAIT! What about the fact this issue only occured on TVs and not on the development machines? My conclusion is that newer version of Chrome/Chromium must be doing some kind of optimisation by default for 2D translates, because when I checked the Mac's Chrome dev tools (without the above fix) the GPU was already being used to do the rendering for those transitions. This explains why I never saw the issue on Mac and the TVs had this issue because they were on lower version of Webview that didn't have this optmisation built-in.

Final Thoughts

The TranslateZ() hack is a good way of improving CSS performance for sites with a lot of transitions. However, since hardware acceleration is forced and GPU render drains more battery, one thing to bear in mind is that we may want to avoid doing this on mobile devices or anything that runes on battery (e.g. unplugged laptops). For example, in the future we could use navigator.getBattery() and user-agent to detect the user's device and bettery state. We may then wish to disable transitions to best optimise for mobile browsing experience. Obviously, this is not a concern for plugged in devices and TVs which are plugged in all the time.

There are a few other things I'd like to mention:

I joined this project late after everything was already setup. So I am not sure why React-native or other ways of developing for cross-platform wasn't picked instead
Not all smart TVs are based off Android, so the performance issue in some TVs could be a lot worse than what I explained (and may not even apply). On the other hand, not all systems demand as much resources as Android does. So your mileage may vary.
Amazon's FireTV stick is also based on Android, and with the FireTV 4k's recent release, the market share of Android TVs is very likely to further increase.

Triad Learning: a new way to train and learn

Wed, 22 Aug 2018 09:08:00 GMT

This is my fifth year as a software developer, it certainly has been a long and fulfilling learning journey until now. There are simply too much to learn no matter how we plan it out, not even mentioning there are new tech coming out every day. When I started my career I felt like I knew so little about the world of software development, and that feeling never really went away. This game of catch-up is tiring to say the least, some even gave this overwhelming feeling a name - development fatigue.

The struggle is real

When new joiners ask me what's the best way to learn something, I had often struggled to find a good way to explain my way of learning. I want to tell them to learn everything, because everything is connected on some levels in the world of software development. To be a good developer, it takes a lot more than knowing a framework (e.g. React) very well. Other questions comes to mind too: "how did you structure your project and why?", "which build tool did you use, how do you plan to publish the code?", "have you considered the other tool/lib/framework?", "what is your testing strategy?" and etc. All of these are common and practical questions people ask when they join a project, a good developer should be able to explain them. But it takes a lot more than reading a few article to truly grasp answers for these questions. So I often default to saying "depending on what you are trying to do...".

I personally also struggled a lot in this area, because I'm interested in a lot of areas: frontend, backend, mobile, and etc. It becomes increasingly difficult to keep track of everything as I learn more. Knowledge, like a spider web, the area increase exponentially as it grows. When I talk to people about this, the more senior developers often say that I should pick one area and focus on it. And they would go on to quote the following line:

Jack of all trades, master of none

On some levels I do agree with it, I only have finite amount of time like everyone else. So even if I work twice as hard to learn I still won't be able to learn everything I want. But at the same time, I don't just want to give up. I have experienced the benefits of full stack, the ability to see a problem from different perspective and being able to find a solution that make sense overall.

So what then?

I wanted to find a way of learning that is practical and easy to follow. It needs to be something that can be followed by an individual as well as adopted by multi people team.

Path to the solution

I analysed my situation and what is going on at my work place. Unsurprisingly, I found quite a lot of problem.

I decided to first write down my way of learning to see where the problem are:

Read articles and find out about something new.
Keep track of it, so can back to it at a later point.
If something came up a few times, I then spend more time doing some quick research on that topic. This could be reading the official documentation, and checking out a few blog posts online. I don't spend too much time at this point, the aim is to understand the high level gist.
If I find it interest and I have the opportunity, then I go on to get some hands on experience through personal project or other means.
If I get stuck I'd google for some tutorials in that specific area

Surprisingly enough, this process is actually okay I think. The area that I struggle the most with, turns out to be keeping track of things to learn and how to best management them. I had done an exercise to tackle that separately (I'll cover it another time). But this is really good news to me, since this means my learning process itself is fine.

So how about my work place? Here are just some of the problems I identified:

Our internal training mostly focuses on topics are a deep dive in certain areas, which can seem overwhelming for people who are new to them. It is also difficult to get on these training, due to the limit slots.
There is a lack of structured learning in general within the company, what you learn and what you get to work on is mostly luck based. There isn't much support in terms of learning either, for graduates just came out of university it is easy to feel lost.
Senior developers and coaches are too busy with client work and aren't too interested with helping junior people. Even if they wanted to, this can be quite difficult because they are based in different locations.
There is little focus on training our non-technical people about the technical side of things. So very senior Business Analyst might not understand concept which a junior developer work with daily. Which makes the analyst's job difficult when they have to understand the challenges or gather valuable from stakeholders.

Based on my own learning process, I came up with a new model of learning which can applied to teams. I call it Triad Learning.

Solution: Triad Learning

I decided to give it this name because it is made up of three parts: Presentation, Workshops, and 1:1 paring. Each part has its target audience and goal. Any of the individual parts on their own aren't groundbreaking, the fundamental idea here is that by combining them we create reinforced learning. Rather than just telling people about high level ideas, or do a deep dive on a topic people aren't familiar with. The goal of Triad training is to take some topic from the basic high level concept all the way down to practical hands on experiences.

The follow sections will be explained in a team environment, please refer to the five points I stated above if you need more information around applying this for your own individual learning.

Presentation

Talk or presentation is probably the most common verbal way to share knowledge. This type of training/learning aims to give large group of people a high level understanding about a topic.

Example: if I want to train people around automation testing. I might want to give a presentation around why we test and how we might go about it. The target audience could be technical and non-technical people.

Workshops

Workshops are quite useful due to its discussion nature. These kind of sessions will offer two things: 1.) give people an opportunity to try something simple out with guidance and support. 2.) the other advantage is allowing people to listen to what others are talking about so more introverted people can also benefit from it.

Example: using the automation testing as an example, this could be doing some TDD with a smaller group of people. The intended target audience is more developers in this case.

1:1 Paring

It is extremely frustrating when you can't get something to work. It is very helpful to have someone there who you can bounce ideas off. Even if they don't say much, explaining the problem to someone forces you to think about it more. Often, I end up solving my own problem half way through the explanation. The idea of 1:1 paring is for just that. To give other people the opportunities to approach the trainers and ask questions and bounce ideas off.

Example: back to our testing example, this could someone asking how to mocking in Jest. Something that is very specific which doesn't interest everyone, but it will speed up the learning of the person if they had someone who could answer their questions or point them to the right direction.

Final words

The main focus on Triad Learning is around reinforced learning. The first layer of the pyramid is to understand the high level concept. Then you go on to understand how it works through more hands on and a deeper dive. Lastly, you'll learn through more practical and encountering challenging problems.

Typescript: the key to a maintainable JavaScript codebase

Wed, 15 Aug 2018 19:20:00 GMT

So here's a question, if someone told you to draw a sketch would you like to be rulers and grids OR do you prefer to just have a blank canvas? One gives you structure, guidance and assistance, while the other gives you freedom. This is exactly the same argument with statically typed language VS dynamically typed language. This question has been around for a long time, and no one has managed to settled the argument yet.

So what is the difference between static and dynamic typing, I hear you ask. Static type, like the name suggests, restricts a developer's ability to assign different data to data types by having rules in place. For example, static type means I won't be able to assign a number to a string typed variable. If I wanted I can still do the assignment, it is just I will have to convert that variable from string type to number type first. But why can't we just assign it, isn't the computer clever enough to work it out? That's exactly what a dynamically typed language does. It will allow flexible assignments between different types.

Without going into much a deeper argument between static and dynamic typing, the easier high level comparison is that statically typed languages gives more confidence to developers when they work on a code base by catching errors early, whereas dynamically typed languages' freedom means less interruptions during development and end up writing code faster.

So what's Typescript?

With the above long explanation of static and dynamic types, now let's talk about Typescript. Typescript was designed to allow people write typed JavaScript code. It is a open-source programming language developed and maintained by Microsoft. It was introduced back in Oct 2012, but it didn't pick up pace until recent years (see diagram below).

We briefly talked about the differences between static and dynamic typed languages, so Typescript is meant to be the perfect language that allows the best of both worlds. In reality, that is not quite 100% true but it is still extremely useful in my opinion.

Features

There are a lot of reasons to use Typescript in a project, I have listed a few which were most impactful for me. I recommend checking out the official Typescript documentation and have a read yourself about all of its features.

Flexible & optional to use

Some people might think Linters are more than enough to ensure the maintainability of their code, and this could be true. Just like any other tools, Typescript is not a one solution fits all language and it is not meant to be. What it does offer is choice, the ability to choose when to use static type in a dynamically typed language is very powerful. At the same time, it doesn't force you to use static typing everywhere like other static typed languages. This feature makes it very useful, because it means it can be added to any JavaScript project at any given time. Since all it takes is to renamed the file from .js to .ts then Typescript will take over. After the renaming I could literally leave the file as it is, and make no change at all and it would work since Typescript is an extension of JavaScript. So it will understand all the JavaScript syntax out of the box, then it is up to the developer to utilise the TypeScript features if they wish.

No more Babel

On the Frontend of web apps, People used to use Babel to compile latest JavaScript syntax to older JavaScript syntax so that the code can be understood by older browsers such as Internet Explorer (IE). The other good thing about Typescript is that you won't need to use Babel anymore, Typescript will also do this backwards compilation for you out of the box (configurable in the tsconfig.json).

Enhanced IDE support

After using Typescript in a few projects, the thing I noticed was I didn't need to do inline documentation in the code nearly as much. With text-editors such as VS Code, when people get so much information when they cmd + hover on everything, there is really no need to document what parameters functions take or what kind of data each variable holds. Following code logic is also a lot easier, everything (e.g. function, class, variable) is just a Cmd + click away.

The help doesn't just stop there, when writing code the most text-editors or IDEs will also use the information provided by Typescript and auto suggest things as developers code. Not only does this speed time development, it also means people make less mistakes with typos. Amazing stuff!

Good support in the wider community

Most common libraries today comes with or has a Typescript definition file (often in the form of index.d.ts). These definition files are generated in order to preserve the static typing used during development of those libraries. Still confused? Let's walk through an example. Assume that I'm developing a Frontend app, in order to implement a new feature I want to use a library created by someone else so I don't reinvent the wheel. I install the library, then my text-editor automatically picks up the Typescript definition file that came with it. Now when I use the library I get all the useful intellisense information and features mentioned in the previous section on the new library. If there's something i'm not sure about, for example I don't know how to call a function in this library. I don't even need to go to their official website to read the documentation, I could simply look at what Typescript tells me and get on with what I'm trying to do.

When to use Typescript?

Javascript is a beautify language and its dynamic nature is amazing for prototyping. Try not to use Typescript in a small project where speed of delivery is key e.g. hackathons, small personal projects or product prototypes. Make a judgement call before start using it in all the projects. Don't give yourself unnecessary headaches or worse give your team unnecessary headaches for no reason.

Typescript will require setup and config, and actively type everything to get maximum usage out of it. There are some configurations which might cause more pain points than others, so try out the configuration options and turn them off when they become a hassle to deal with. For example, I just recently worked on a project where I had to use Typescript with Redux, the experience wasn't great. I had to hack around the declaration of my Redux store to make Typescript happy. It would be wise at this point to either disable Typescript for those lines or tweak the Typescript config.

Final thoughts

Maintainability should always in the minds of responsible developers. At the end of the day, it is not just about writing code. Instead, as developers we should be able to defend our decisions and work when they are being questioned. In fact, we should also be proud to show off our work to other developers. Creating a maintainable clean codebase is a great place to start, and Typescript is a great tool for achieving that.

Of course, Typescript is not the only solution to this problem. It might not even be the best tool, since I didn't have enough time to try out all of the alternative tools (such as Flow, Elm, and etc) and do a complete comparison. The best way to choose the best tool for your team or project is to spend a bit more time with each if possible, then make the call for yourself. If time is also a limiting factor, then Typescript is a solid tool which I can sensorily recommend.

Redux testing - Invariant violation error

Fri, 03 Aug 2018 18:21:00 GMT

I came across this error "Invariant Violation: You must pass a component to the function returned by connect. Instead received undefined" when trying to debug React/Redux app. This article walks through how I went about to solve it, and what was the issue.

I came across this weird error when trying to test some React components.

Test suite failed to run

Invariant Violation: You must pass a component to the function returned by connect. Instead received undefined

The structure is very simple, there is a Redux container wrapping a React presentational component with local state.

// Container component

const mapStateToProps = ...

const mapDispatchToProps = ...

export default connect (
	mapStateToProps,
	mapDispatchToProps
)(Container);

// Component

export default class Component extends React.Component {
    constructor(props) {
        super(props);
        
        this.state = {
            foo: null
        };
        
        ...
    }
        
    render() {
		return (
            <div>Component</div>
		);
   	}
}

I wrote tests for both of the files above, the container tests ran fine. However, when I try to run my test in Jest I'd get this error.

Test suite failed to run

Invariant Violation: You must pass a component to the function returned by connect. Instead received undefined

I checked my container's connect() function, but everything looks fine. After doing a little research, other people mentioned their issues were caused by one of the following issues:

Circular dependencies between components
Wrong usage of export and export default then imported the wrong way
Used the connect function wrongly, passed the wrong parameters

I checked against all these points, but none of it applies to the code I was working with.

What's even more strange is that I could build and run the web app no problem. So this problem seems to only exist in Jest.

So I decided to do more debugging around it, "when stuck, trial and error" right? I commented out the line which does the rendering, which looked something like this:

it(('should not fail') => {
    // const wrapper = shallow(<Component />);

	expect(true).toBe(false);
});

This test obviously failed, and it actually ran the test suite. Which is a good news, because I was never getting this far before. The fact the rendering is failing must mean my export was done right, or something related to it. So I went back and checked again, and same as before I found nothing.

I tried commenting back all the shallow render and commented the import at the top of the file (see below). This also ran the test suite giving me the error that Component is undefined.

// import Component from './Component';

it(('should not fail') => {
    const wrapper = shallow(<Component />);
});

The issue

It turns out the problem was still circular dependency, but the shared dependency was something really stupid. Let me explain.

I am using Typescript, I declared a type a few levels higher and then imported it in Component. This made that entire high level component dependent on the Component, thus causing the circular dependency.

As suspected it was a simple fix, but I had to go through so many loops. For anyone getting this error, check the 3 points mentioned above because I think they will cover 99.99% of the cases.

Also note to self:

Circular dependency doesn't get picked up by Typescript in React/Redux project. This could be a limitation of webpack or my webpack config, as when I used Typescript with rollup.js it did pick up circular dependency.
Typescript will build React app with Circular dependency, but it won't run the test suite

The real life example

Here is the exact error I got so people can compare to see if they have the same problem. SpendPointsMenu in this case is the Component in my example above, and SpendPointsMenuContainer is obviously the container. Spend is the high level component where I was importing the type from. Don't worry too much about SpendPoints it just sits in between Spend and SpendPointsMenuContainer.

Test suite failed to run

Invariant Violation: You must pass a component to the function returned by connect. Instead received undefined

      26 | )(SpendPointsMenu);
      27 |

      at invariant (node_modules/invariant/invariant.js:40:15)
      at wrapWithConnect (node_modules/react-redux/lib/components/connectAdvanced.js:101:29)
      at Object.<anonymous> (src/views/spend/spendPoints/SpendPointsMenuContainer.tsx:28:141)
      at Object.<anonymous> (src/views/spend/spendPoints/SpendPoints.tsx:13:33)
      at Object.<anonymous> (src/views/spend/Spend.tsx:16:46)
      at Object.<anonymous> (src/views/spend/spendPoints/SpendPointsMenu.tsx:20:40)
      at Object.<anonymous> (src/views/spend/spendPoints/SpendPointsMenu.test.tsx:7:461)

My first clue should have been:

SpendPointsMenu.tsx is causing an error in SpendPointsMenu.test.tsx, that's fine because that's the file I trying to test. But why is Spend.tsx causing an error in SpendPointsMenu.tsx.

I over looked this clue, and ended up spending hours trying to fix something very simple.

Why do projects always end up with tech debt?

Fri, 13 Jul 2018 17:26:00 GMT

As a developer, I have both explained and heard others explain to less technical people about how the team need to slow down adding new features/functionalities and take care of some tech debt first. Often, the management asks why. Sometimes it is hard to communicate the idea across to some people. In fact, it is quite nature for tech debt to occur and it is no one's fault that this happens. It has always bothered me when tech debt become a point of tension between developers and management. Because I think it happens so often that everyone, developers and management, should understand why projects end up with tech debts, what they are and how to manage them.

Tech debts is always talked about in development teams. It is a common questions in job interviews and it is often the hot topic of debate in team discussion. It is always hard to decide if time should be spend on building new features or reducing the amount of tech debts within a given project.

When a team gets stuck in a loop of "why should we reduce tech debts" and "why shouldn't we", it is extremely frustrating for everyone. This has always bothered me, because I don't think tech debt should become a point of tension in any project. It is completely natural to end up with tech debts in any project, as long as everyone has a good understanding of it, what matters the most is how to address it. I'm not saying this kind of conversation isn't health, in fact, quite the opposite it is very health for the Product Owner (PO) to challenge the devs and vice versa. What isn't healthy to going around in circles, and debating about it week after week.

This blog will explain what tech debts are from a developer's perspective since it is a technical problem at the end of the day. But I'll also explain why it might also be useful for devs to bear in mind of some the business concerns too instead of just pushing for reducing tech debts.

Why do we have tech debts?

Let's use library as an example, to illustrate why projects might end up tech debts.

Imagine there's are a librarian, who works hard to keep 1,000 books tidy and organised in the right order on all of the book shelves. The above is what maintaining a project looks like. Sounds easy enough right?

Now, let's say customers comes to the library everyday and sometimes books gets misplaced, books can also be borrowed and returned. On top of all that, each week 100 new books are also added to the library. That is a bit more work now right? As time goes on, it gets increasingly more challenging to achieve the original goal of keeping books tidy and organised. Simply because of how many books there are in the library. This is how it feels for a developer when they have to deliver value in an agile methodology such as scrum. There are a lot to balance out, e.g. time vs quality, off-the-shelf approach or use own implementation, performance, security and etc.

As the number of books increase, sometimes books gets misplaced by customers but the librarian has too much to do and doesn't notice it. Books might be lost because something went wrong with borrowing or returning processes. Librarian might not have enough time to shelf all the books properly, and they just do their best and only put the books in the rough areas. All the these problems can occur in a project. As the project grows, same "amount" of work will take longer because there are more things to consider when delivering a new feature. It is like playing a game of Jenga, even with the same number of pieces, as the game goes on it comes harder to stay in the game. Not just because there are more variables to consider, but all the previous actions has a massive impact on the future actions, as well as limits what can and cannot be done.

In some projects, it may have been worse. Assume, now the CEO of company decided they want to track the performance of each library by measure how many new books they can add on to the shelf each week. It seems crazy and illogic here right? But it does happen and maybe happens too often. This forces libraries to import more and more books to add on to the shelfs, and has a knock on effect on all of the problems mentioned above. This is why scrum recommends teams to maintain a constant velocity instead of strike for a higher number each sprint.

What are the tech debts and how to address them?

Tech debts comes in all sorts of shapes and forms. Here are just some of the common tech debts that I have seen first hand.

Duck tapped business logic

Now, we all work using some kind of agile methodology to ensure the product development can quickly pivot if and when the business requirement changes. This means requirements are not fed through until when they need to be pulled in. This means development teams are often unaware of business logics that's further than a few sprints away. This is where people having to build on top of any existing business logic. The lack of big picture at a given sprint and the fact these logic could have been the worked on by different people means that, the end result might look messy and is difficult to understand. Naturally, it will make any future work in this area more difficult, because people need to first understand what is going on before implementing anything new. In bad code bases, it can feel like walking around in a massive maze, where many routes will end up to be an dead end. It's a very time consuming and frustrating process.

Not mentioning the fact, unless the existing code is fully understood, it is very easy to introduce new bugs. Unfortunately, the areas where this kind of duck tapping happens is also where the core business logic sits. So this could be financial calculations, complex user journey mapper and etc. Basically, things you don't want tripping over on production environment. This is why it is worth tackling them ASAP.

When dealing with these issues, the core focus is to rewrite the code so that the logic is easier to understand. This kind of actions could massively speed up future development, because rather than looking at something like this:

face North
turn Left
turn Left
face South
turn Right

It will be as simple as:

face West

Pretty straight forward to see which is easier to understand.

Code cleaning

Code cleaning doesn't sounds like a necessity, but it can be hugely helpful. Unclean code makes it difficult to read and understand existing code base. Often a small amount of messy code and spiral out of control very quickly as the team loses grip on the "right way" of doing things.

The common tasks of code cleaning includes:

Remove redundant code
Remove badly formatted/organised code
Remove inconsistent coding styles
Adopting better coding practices

All of the above should be fairly simple to understand. If you are still unclear, then think of it like tidying up a box of tens of thousands of Lego brick. The best way is to separate each type into its own compartment, then organise them in a logical way.

Performance improvements

When features are implementing quickly, most of the time it is probably NOT done in the most efficient way. Going back to previous point about the lack of big picture, which can definitely be challenging on one person is focusing on a very small part of a product or system. It might be that one aspect of the performance was missed during development. It could even be that, a while later someone has found a better way of doing the same thing.

Depending on the situation, the performance improvements will be different. But a simple example might be that, instead of serving a default sized high quality images on a website, the team could export different sizes of the same image, so we can server different image sizes depending on the user's internet connection and device.

Other tech debts

Improving security: this is probably a bit rare for a product, since most likely everything has been thought of at the early stages of product live cycle. Since nowadays most products uses open source software and the open source nature means it is easy for hackers to find security flaws. Thus it is important for the team to review the libraries and frameworks used constantly, and update to the latest version ASAP when bugs/security flaws has been revealed.

Improving SEO performance: overtime changes made to the code base might have negatively impacted the website/app ranking on search engines.

Improving accessibility: accessibility is often missed by development teams, but we should aim to build products that can be used easily by everyone.

Better practices: as a team and product grows, it should be encouraged for the team to explore for better tools and practices to improve the ways of working, code quality, efficiency and etc.

Why might teams want to delay fixing tech debts?

For the business, tech debts is this vague and mysterious work that only the developers knows about. It is not hard to understand why management gets worried when the development teams says they need a month to work on tech debts without much justification. Unless people are being difficult and is being unreasonable, as long as they get the justification the management needs there is no reason why they would say no.

Understand priorities within the project

Tech debts is horrible, and teams should aim to reduce it as soon as possible. But it is worth to be mindful about the priorities in the current project. For example, any of the following would be good reasons to postpone tech debt for while, unless the tech debts is a massive blocker when it comes to delivering those goals.

A big release is about to go out
High priority feature addition/changes
High priority bug fixes

Basically, use the best judgement to decided if it is the most appropriate time to be working on tech debts. For example, between releasing the next version of the product which contain important bug fixes and fixing some code formatting, it would be wise to focus on the release first then come back to the code format issues post release.

Summary

Hope this blog post has helped to clear up any questions about tech debts, and in some cases how to fix them as well. If you have any question feel free to get in touch!

Migrated blog to Gatsby & Contentful

Thu, 14 Jun 2018 22:19:00 GMT

One of the reason why there wasn't any blog post recently was because I was migrating my blog to a new platform...again. I can't remember how many times it has been in the past few years, and this is excluding the times where I investigated in other blogging platform but didn't decide to commit. But I'm happy to announce that this new platform should be much easier for me to create content going forward. In this post, I wanted to explain how it works now and why I've spend month migrating again.

Previous challenge

One of the reason why I haven't been pushing much content to my blog was because I kept having problems with my previous blogging solution - Hexo. For a long time, my local just wouldn't build due to some npm dependency issues. For months I couldn't publish anything because I simply can't get my local working to push them (If you are interested in how it worked you can find out more in a previous post). It is also a lot of hassel to publish anything even when there were no problems, for example I always had to do it on the same machine or I will need to install Node, Hexo and etc. It is more work than it sounds, especially on some platforms.

When I finally got everything fixed, I was so fed up anyway and I wanted a better publishing experience. For a while I even considered ditching this site and going with Medium. Yep, I was that annoyed!

A new hope

So I have been using Netlify for a while to host my static blog, I was loving it. Then one day I was told there is something called Netlify CMS and I got really excited. All I'm missing from my Hexo set up was literally a place to create and publish content. I also didn't want to make a code change, manage my own image insert every time. It is a pain to say the least.

I tried out Netlify CMS, but it turns out too basic for what I need. I mean there isn't even a draft option, it is either publish blog or no blog. So unfortunate I had to pass on this solution. But this gave me more motivation to deep looking for other alternatives which worked better.

Errr... another new hope

Don't worry, this time it is for real. Someone at work was really impressed by it, so I was curious to find out what the fuss is all about. Then when I did look it up, I realised it is perfect for my blogging solution. If you haven't heard about it already, it is an amazing tool go check it out. But on a high level it is a CMS site, it provides APIs for anyone with access key to request data. The cherry on top is that it support graphQL, amazing right?

So now the CMS site and connect is sorted out. The next step is trying to find a library to do the static site generation based on the data in Contenful. I have always wanted to use Gatsby in one of my projects, and this turns out to be a perfect opportunity. Gatsby uses React, so even less learning curve there, perfect!

So in the past couple of months, I've been finding time here and there to rebuild my blog from scratch using Gatsby. Then I re-implemented the more advanced features I had such as pagination, tag pages and etc. Now it is about 80% done, with some features still missing such as commenting.

The new work flow

With this new set up, which is amazing btw, everything becomes so simple compared to before.

If I want to publish a new blog post, I would write it anywhere using markdown. Then copy and past into Contentful and publish it. I couldn't write on Contentful because when I tried I hit the max request limit due to its auto-save feature. So now I write it using Typora, but since it is markdown you can essencially write it anywhere.

Once the blog is published, it will trigger a webhook on Netlify which will pull the Gatsby source code from my Github repo. Then the entire blogging site will get re-generated based on the lastest data from Contentful. No need to make code change, no need to open an IDE. I can do this on any machine that has a internet connection.

If I ever want to add new features to the Gatsby app, then I just develop it as normal. When the source code is merged to master, it will also trigger a Netlify webhook which also triggars the build. About 30 seconds later I'll get the new feature on the live site.

Final Words

If you are looking to do the same, then apart from Contentful and Gatsby I'd also recommend taking a look at gatsby-source-contentful library which saved me a lot of time.

Create a JavaScript library using Typescript, Jest and Rollup

Mon, 27 Nov 2017 22:23:00 GMT

Recently I was working on a personal JS library Serina. It is a open source library, I wanted to use Typescript (TS) to ensure better stability due to it is static typed. It should make the code base more readable and therefore easier to contribution.

There are many ways to build projects, and Rollup is just one of them. I decided to go with Rollup because it is much lighter than something like Webpack, but it is still very well supported in terms of plugins. Most importantly, I've used it in other projects and I'm lazy :P All jokes aside, I have to say Rollup is my go to build tool right now.

In this blog, I have broken down each part of the project into different sections, so it is easily to find alternative options if prefered. E.g. rather than using Typescript, you can simple skip it.

Setup core library

Like all JavaScript projects, first create project folder and within the project folder run npm init to initialise it.

Create a src folder and then create an empty file inside it. It should now look something like this:

yourFolder
    |-- package.json
    |-- src
        |-- myLib.ts

Inside the myLib.ts file is where the library logic will live, but let's do something simple for now. In the follow example, our library will simple take in a string then return it back. It is pretty useless, but for the purpose of this blog I'm going to keep it like this so it is easier to follow.

export default myLib = (text: string): string {
    return text;
}

Typescript

Install Typescript npm package by running npm i typescript (or use npm install).

It is always a good idea to configure the compiler options of Typescript to fit your coding pattern, and pick up on any potential issues with the code when it compiles. Create a file called tsconfig.json at root folder level with the following config:

{
  "version": "2.5.0",
  "compilerOptions": {
    "allowSyntheticDefaultImports": true,
    "declaration": false,
    "diagnostics": true,
    "target": "ES5",
    "downlevelIteration": true,
    "forceConsistentCasingInFileNames": true,
    "noFallthroughCasesInSwitch": true,
    "noImplicitReturns": true,
    "noUnusedLocals": true,
    "noUnusedParameters": true,
    "removeComments": true,
    "noLib": false,
    "moduleResolution": "node",
    "sourceMap": true,
    "typeRoots": [
      "/.node_modules/@types"
    ]
  },
  "include": [
    "src/**/*"
  ],
  "exclude": [
    "src/**/*.test.ts"
  ]
}

The folder should now look like this:

yourFolder
    |-- package.json
    |-- tsconfig.json
    |-- src
        |-- myLib.ts

Code linting

It is always important to have linting setup for any project to ensure code standards and catch silly errors when coding. Since I'm using Typescript, therefore TSlint is my choice here, but feel free to use ESlint if you wish to use pure JavaScript.

Let's install tslint and the extra eslint rules by running npm i tslint tslint-eslint-rules.

Now create tslint.json file in the root project directory.

yourFolder
    |-- package.json
    |-- tsconfig.json
    |-- tslint.json
    |-- src
        |-- myLib.ts

Inside the file, paste in something like the following. This is my prefered setup, but feel free to tweak it however you see fit.

{
  "extends": [
    "tslint:recommended",
    "tslint-eslint-rules"
  ],
    "defaultSeverity": "warning",
    "jsRules": {},
    "rules": {
      "curly": [true, "ignore-same-line"],
      "interface-name": [true, "never-prefix"],
      "max-line-length": [true, 180],
      "object-literal-sort-keys": false,
      "trailing-comma": [true, {"multiline": "never", "singleline": "never"}],
      "arrow-parens": false,
      "member-access": [true, "no-public"],
      "eofline": false,
      "whitespace": [false, "check-module"],
      "no-angle-bracket-type-assertion": false,
      "ordered-imports": false,
      "quotemark": [true, "single", "avoid-escape"],
      "object-curly-spacing":[true, "always"]
    },
    "rulesDirectory": []
}

For more information on the above rules, refer to the official documentation.

Inside package.json I would recommend adding 2 npm scripts related to linting. npm run lint will check for all the TS files within src folder for mistakes, and npm run lint:fix will fix all the mistakes it can automatically (e.g. it is nice not having to go in different files and remove a whole bunch of white spaces).

"lint": "tslint -c tslint.json \"src/**/*.ts\"",
"lint:fix": "tslint --fix -c tslint.json \"src/**/*.ts\""

Setup Test

In order to get test set up, we will need to install Jest and the related packages by running npm i jest ts-jest @types/jest.

Then add the following Jest config inside package.json. This will be used to pick up test files and etc.

{
  ...

  "jest": {
    "transformIgnorePatterns": [
      "node_modules/(?!shiva/)"
    ],
    "globals": {
      "ts-jest": {
        "tsConfigFile": "tsconfig.test.json"
      },
      "google": {}
    },
    "collectCoverageFrom": [
      "src/**/*.ts",
      "!src/**/*.schema.ts",
      "!src/schema/*.ts"
    ],
    "testMatch": [
      "<rootDir>/src/**/?(*.)test.ts",
      "<rootDir>/tests/**/?(*.)test.ts"
    ],
    "coverageThreshold": {
      "global": {
        "branches": 0,
        "functions": 0,
        "lines": 0,
        "statements": 0
      }
    },
    "moduleFileExtensions": [
      "ts",
      "js"
    ],
    "transform": {
      "\\.(ts)$": "<rootDir>/node_modules/ts-jest/preprocessor.js",
      "^.+\\.js$": "babel-jest"
    }
  }

  ...
}

Setup Build

To set up Rollup, first install Rollup and TS plugin by running npm i rollup rollup-plugin-typescript2.

Now create a rollup.config.js file in the root project directory. Inside this file create a simple rollup config like the following:

import typescript from 'rollup-plugin-typescript2';

export default {
  input: 'src/myLib.ts',
  output: [{
    name: 'MyLib',
    file: 'dist/myLib.js',
    format: 'umd'
  }],
  plugins: [
    typescript()
  ]
}

The project structure should now look like this:

yourFolder
    |-- package.json
    |-- rollup.config.js
    |-- tsconfig.json
    |-- tslint.json
    |-- src
        |-- myLib.ts

Now, again inside package.json and add in some npm scripts for building the project. npm run build will call other npm scripts and then at the end compile the TS into JS and place it in the dist folder.

"build": "npm run lint && npm run test && npm run clean && rollup --config",
"clean": "rm -rf ./dist && mkdir dist"

Icing on the cake

Since it is going to be an open source library, it is also worth considering doing the following to give people more confidence in the project:

Add a README.md file
Add a LICENSE file so people know how they can use your code
Set up CI/CD, and then place a badge in the README to show the code bade status (e.g. passing, failing)
Consider adding an .editorconfig file

That's it!

Hope this was relatively straight forward to follow, I tried to cover all the aspects in this blog and answer the common questions. Please let me know in the comments if there's anything I missed or hit me up on Twitter!

Blogging with Hugo & Netlify & Forestry.io

Tue, 21 Nov 2017 18:23:00 GMT

About 10 months ago I switched from Ghost blog to Hexo. I decided Hexo because it would be generating static site, which is great for SEO. It can be hosted on Github pages, which saves some £££. Then there is the most important reason: so I don't have to spend energy and time on maintaining the blog server and constantly migrating to newer Ghost versions.

So why am I considering to change to another blog tool less than a year later? The most important reason is the complexity involved just to get something published. I found myself not wanting to write any blogs not because I don't have anything I want to write, but the publish process is so complex. Before when I was using Ghost, I could go on any computer type in an URL and start blogging. With Hexo, I need to install node, hexo and etc if I am not using my main computer. Then there's the manual work such as tagging blogs, adding in images and sorting out URL path names. If that is not bad enough, recently I have been getting all sorts of errors when I try to spin up Hexo locally, then after I updated the version it now no longer serves the local development site. So I decided enough is enough and researched around for a better way.

My initial thought was maybe server Ghost blog locally then convert it into static site using buster. This would solve some of the problem, but I'm basically back to square one minus the hosting and server mantainance problem. Then I read about a CMS tool called forestry.io which sounded very interesting. It will essentially act as the CMS of the blog site, which means no more hexo new --draft "yet-another-blog-post"! I could do it with a simple click on any computer browser like I used to be able to with Ghost (I've also checked it works on mobile too). By no means is this a perfect solution, because the few pages which I had visitors will be lost after the site migration. And then there's the migration itself. But given this is the best solution I can find, here goes nothing...

Set up

Some prerequisites before we get started:

A computer to set everything up initially e.g. Hugo, source code repo
A Github/Bitbucket/Gitlab account to keep the source code of the site
Netlify, Github pages or any alternatives supported by forestry.io
And finally forestry.io

There are different ways of setting this up, by no means this is the best way. But I think this is the easiest way to follow.

Create Repository

Create a repo on any source code hosting site of your choice. I'm using Bitbucket because it offers private repo hosting for free.

Then git clone the repo into a directory on your computer.

Installing Hugo

The installation is pretty straight forward. Since I'm using a Mac, so the installation commands will be for Mac OS. If you are using Windows or Linux, please refer to the offical guide.

brew install hugo

cd ~/yourFolder

hugo new site blog

This will create a blog folder within the repo folder. It should look something like this right now.

yourFolder
    |-- blog
        |-- config.toml
        |-- archetypes
        |-- content
        |-- data
        |-- layouts
        |-- static

The next thing to do is to move everything out of the blog to the level above. It should look something like this after you are done:

yourFolder
    |-- config.toml
    |-- archetypes
    |-- content
    |-- data
    |-- layouts
    |-- static

Then follow instructions on installing theme, and update the config. There should be plenty of documentation in the Hugo offical guide.

Once everything is done push it to the master branch.

Setting up hosting

I am personally using Netlify, because it is awesome and easy to set up. But you will get everything you need using Github pages too.

This step is as easy as:

Login into Netlify using your Github/Bitbucket/Gitlab OAuth.
Select the repository
Then fill in these values in the relevant fields
```
Build command: hugo
Publish directory: public
```

I have added a netlify.toml file to the repo (next to config.toml) to state the production hugo version as well. Not sure how relevant this will be, but I just wanted to be safe.

For more information, Hugo has a detailed documentation for this step as well.

Setting up CMS

This step is even simpler! Go to forestry.io and then login with your relevant Github/Bitbucket/Gitlab OAuth. Go through the simple wizard, then you'll be taken to the CMS page. It is so simple, there's no hassle at all!

All done!

That's it, now you can blog where ever and when ever! Like I mentioned already, forestry.io even support mobile devices, so you can do that too!

Migration

The last step is off course migrating from which ever platform you are using right now.

Final thoughts

This was very cool, but I personally had some issues with theme styling and decided to stick with Hexo. But for anyone who is setting up a new blog, since this is so easy it is almost silly not to give this a go first!

JavaScript Events

Mon, 02 Oct 2017 19:00:00 GMT

We are always using JavaScript events when developing web application, it is what makes it interactive at the end of the day. In this blog, I'll go over what it is and the different ways to intercepted a JavaScript event.

JS events in HTML

HTML events are “things" that happen to HTML elements. We can use JavaScript to “react” to these events. Events can be:

An HTML web page has finished loading
An HTML input field was changed
An HTML button was clicked

Here are some common HTML events:

onchange - An HTML element has been changed
onclick - The user clicks an HTML element
onmouseover - The user moves the mouse over an HTML element
onmouseout - The user moves the mouse away from an HTML element
onkeydown - The user pushes a keyboard key
unload - The browser has finished loading the page

<button onclick="displayDate()">The time is?</button>

In summary, this section explained how to use events in HTML elements.

Source: https://www.w3schools.com/js/js_events.asp

JS events in JS

Events is the fundamentals to how modern framework achieves data bindings. Angular 1.x is probably the best example, its two way data binding on the lowest level is events.

How to use it

event = new Event(typeArg, eventInit);

typeArg is a DOMString representing the name of the event.

evenInit is optional object type, that can take in the following:

bubbles - A boolean indicating whether the event bubbles. Default value is false.
cancelable - A boolean indicating whether the event can be canceled. Default value is false.
composed - A boolean indicating whether the event will trigger listeners outside of a shadow root. Default value is false.

// create a look event that bubbles up and cannot be canceled
var evt = new Event("look", {
  "bubbles":true,
  "cancelable":false
});
document.dispatchEvent(evt);

Intercepting an event

There are times when you want to stop an event from doing what it. My most recent experience was, closing an menu when the user clicks anywhere else on the screen. So the code logic was:

Click anywhere on the screen and hide menu
When click on the menu, override the above logic and do not hide menu

So now the question is how do we override an event? There are a number of ways to achieve this, we will try to cover most of them here.

1. Event.preventDefault()

preventDefault() method tells the user agent that if the event does not get explicitly handled, its default action should not be taken as it normally would be. The event continues to propagate as usual, unless one of its event listeners calls stopPropagation() or stopImmediatePropagation(), either of which terminates propagation at once.

Usage: Prevent a user from checking a checkbox.

Tip: If an event is not cancelable, calling preventDefault() will throw errors. It is recommended to check on cancelable before invoking preventDefault().

2. Event.stopPropagation()

Prevents further propagation of the current event in the capturing and bubbling phases.

3. Event.stopImmediatePropagation()

Prevents other listeners of the same event from being called. It is the same as event.stopPropagation() plus preventing other registered event handlers on the same element to be executed. So it does not prevent the default action for an event, such as following a clicked link.

4. Return false

Return false as an event is basically the same as calling both, event.stopPropagation() and event.preventDefault().

Tip: return false does not prevent the event from bubbling up in “normal" (non-jQuery) event handlers but still prevents the default action.

Source: https://developer.mozilla.org/en-US/docs/Web/API/Event/Event

Never Add People to a Project Just Before the Deadline

Sat, 22 Jul 2017 15:24:00 GMT

For all the project manager, delivery manager, program manager and anyone else who manages a project’s delivery out there, seriously, stop assigning people on projects right before the deadline thinking that it will help you to deliver on time. Because, surprise, it doesn’t!

I’m not going to give you any lecture based on business book (which anyone who manages a project should already know about), how about some real life experiences? I have been on both sides of this kind of situations, as someone who are thrown into an unknown project and has someone joining the existing team. So believe me when I say putting more people on a project last minute doesn’t work.

The problems

Last minute joiners lacks the commitment and motivation

Let’s start with something as simple as this, imagine you just came off a project. You are thinking, what a great time to do some self study. But guess what, you have been told to join another project tomorrow (or in some cases on the same day). So not only does it throws all your plan out of the window, but also get ready for some intense delivery roller coasters.

The next day, you come to office not knowing where you need to sit or what you should be doing. If you are lucky you get a good introduction to the project background, the state of the project, what needs to be done before the deadline. If you are not lucky, then there won’t be much introduction at all. Either way you will have a lot of questions. When you try to understand more about the project to do your job properly, you will end up pulling people away from their work and they will most likely be stressed out. Which is not a fun experience for either of you.

As a new joiner it is normal to think that they are only here to help, if the project doesn’t deliver on time they are not the ones to blame. Even if the project delivers on time, they won’t get much credit from it either. At the end of the first day, they will be sitting there feeling confused, stressed out, miserable and want out from the project. How can they be engage with project, and be motivated to delivery on time if there is no commitment?

People asking why not this, or this, or this…

The biggest benefit of having a consistent team is that everyone understands why things are done the way it was. People are also on the same page in terms of ways of working.

Bringing in new people will change this. Normally, asking questions and challenging the current ways of working is health since it ensures continuous improvements over time. However, when there’s 2 weeks left in the project and you have people challenging the fundamentals of what was done already, then it can be a problem.

Sure, someone could explain to the new project joiners and say now is not the time and we will log those suggestions and tackle them post release. But it won’t be the last time this type of conversations come up until the project finishes. This type of conversations are very expensive since it is wasting at least 2 people’s time (from the delivery point of view).

Lack of project understand/knowledge

How can someone do a good job, they don’t have all the information they need. It is like asking a school kid to do University assignments. They are simple not ready to tackle those problems, as a result they will need a lot of support and help from existing team members. It will be hard to say no in this kind of situations, because the task is too challenging for people who are new to the project.

In this type of situations, the team could leave the new joiners to fend for themselves, or have an existing team member to help them and tackle the problem together.

If the team decides not to allocate another person to help the new joiner, then they will take a long time to finish the task. Even after they finish, the review stage will take up a lot of time to ensure work quality, accuracy, and consistency.

If an existing team member is allocated to help out, then the reality is that normally the task will take longer to complete because answering questions, explaining and etc will eat up time. So a task that might take a day for the existing team member to finish individually, now it will take up 2 days (4 man days because there are 2 people working on it).

Lack the technical skills set

This is the worse kind of people put on a project. If there is a project that requires a specific skill set, putting someone who doesn’t have that skill does nothing to the project but slow the existing team down.

What should teams do

Extend the deadline

I think if a project is being delayed, extending the deadline is the best option. This will guarantee the best quality and probably use the least amount of man days compared to bring in new joiners. Knowing there are enough time, will make people less stressed out and might even make them be more efficient.

Bring in people early

There is nothing wrong with bringing in people to help the project, the key thing is not to do it last minute. Bring people in before reaching half of the project duration, and give the team to adapt this change. Adding new joiners is like throwing a rock into a stable lake surface. It will take time for the ripple to stop and settle back to the original state.

Have a big team at the start, then take people off

This seems stupid but, it is always easier to take people off a project than to add someone in later on. So if there is a risk, then just put in some additional people at the start. If later the team feels there’s no point to keep everyone on, then take some people off it.

Final words

The above problems are very general and will apply to most projects, but it is especially the case for software development. These problems are not new concepts, but for some reason projects always end up in a situation where more people need to be added to a project a few weeks before the deadline to meet the deadline. There is no excuses, it is just bad management and shows the person in charge doesn’t know what they are doing.

Use ZSH Shell as Integrated Terminal in Visual Studio Code

Fri, 23 Jun 2017 18:18:00 GMT

I have migrated to ZSH shell long ago, and never looked back. It was weird when I opened up and saw the normal shell in VS Code.

Update: 10th April 2022

As mentioned by Jeffrey Thomas in the comments, this post is now out of date. The correct configuration is now:

  "terminal.integrated.defaultProfile.osx": "zsh",

For anyone who's trying to get this working on Mac with M1 chip, please check out Setting up ZSH with VS Code on Apple silicon Mac (M1 chip).

It took me a while to figure it out, but it was as simple as this one line change in the settings.

"terminal.integrated.shell.osx": "/bin/zsh"

Instructions

1. Find the setting

Open up VS Code Settings, then search for terminal.integrated.shell.osx, terminal.integrated.shell.linux or terminal.integrated.shell.windows depending on your platform.

2. Copy it over to User Settings

Note: Make sure you have User Settings selected. Workspace Settings only apply to the current folder or project. Whereas User Settings will change the default behaviour everywhere.

After searching the above string, you should see the filtered list showing the relevant settings. If you hover over the line then there’s a pencil (edit) button. Click on the button and select copy to settings.

3. Update the setting

After copying it over, change /bin/sh to /bin/zsh or whatever shell you prefer to use.

The simplest answer to - should I use Redux in my React app?

Sun, 11 Jun 2017 21:12:00 GMT

There are a lot of good answers on the internet for this question already, I have no doubt that they all provide a much more complete explanation. But sometimes a high level understanding is enough to make an decision, instead of wasting hours of research then come to the decision that it is not needed. In this article, I’ll give the shortest answer I can come up with to answer this question.

And here it is:

The diagram is too vague? Need more explanation? Ok, assuming you have an application that is made up with components Root, A, B, C, D, E and F. Root component is the top level, and then state is passed to A and D respectively, these components then have child components of B and E respectively and etc. If the components of share states then using Redux in your application would make development a lot easier.

Compare the above examples, which would be easier to sync state between B and E? In the React example, state will flow like this B => A => Root => D => E. Whereas in the Redux example, state will flow from B => store (not shown in the diagram) => E.

I hope this article was useful to some of you. For a more detailed explanation of what Redux is and how it works I recommend the official documentation for Redux. If you want to understand more about React states, here is the official documentation as well.

What it takes to learning a new skill

Sun, 07 May 2017 19:05:00 GMT

Recently, I have been helping out with designing the new competency model for the my current employer. It has been a great learning experience, and as a result I have been trying to apply some of the learning to my personal development. This blog describes how I am planning to structure my own learning in the future.

The different stages of learning

When we describe how good someone is at something, we often compare them to ourselves and rated them based on how good we are. If we are just a beginner then someone who is proficient will be rated as "amazing" when in fact they still have a lot of room to improve. This kind of inconsistency will cause a lot of problem in companies where rating and review needs to be consistent to ensure fairness.

So how should we rate people's competency level?

No knowledge - never encounter this before
Awareness - can talk a little bit about the given skill e.g. framework, library or tool
Learn - currently developing this skill, can complete some basic tasks but still requires supervision and support
Use - able to start from scratch and complete a complex task without any support
Teach - not only do they have the skill, they understand the best practices and can coach others in this area.
Expert - leading the exploration, producing work that influences the development and progress of this field

An example

The descriptions above might be somewhat abstract and is too high level. Let me take you through an example I always use when explaining how to learn CSS.

Stage 1 - no knowledge

Someone who is not technical at all, and has no idea how CSS works on a website.

Stage 2 - awareness

Someone started creating simple static web pages, and they want to be able to changing colours and layouts on the page. After some research they understand they need to use CSS to do so.

Stage 3 - learn

Someone who has started using simple CSS to change the look and feel of a web page, but still lack the ability to implement complex designs. This person also doesn't fully understand the reason behind how everything works.

Stage 4 - use

A web developer with a couple years of experience, will be able to implement a web site based on designed wireframes.

Stage 5 - teach

A more experienced web developer who has read in the field about CSS methodologies and best practices in organising code to ensure consistency and maintainability.

Stage 6 - expert

For this stage, I think of someone who has created a CSS framework that is well received by the community.

My biggest take aways

Create a learning catalogue

When learning something new it is easy to get lost in all the different opinions and conversations around the field. It is important to do some research and list out all the important topics to cover before diving in head first. What I always find myself doing before is getting lost in the minor topics such as "how to optimise x", "comparison between y and x" and etc. These type of knowledge can be very useful when I get to the "use" stage, because knowing about all these would help me to understand the best practices and thus teach others about it. But being exposed to it too early will distract me away from the more important things.

Say I'm learning about a new JavaScript framework, by creating a learning catalogue, I can be confident that if I follow it through I will complete the "learn" stage and get to the "use" stage. However, if I'm still at the "learn" stage and I'm already looking at minor topics I mentioned above then it doesn't help me to move on to the "use" stage. The most obvious problem is that, I find myself in situations where I know some of the stuff other do not. But at the same time I wish I had a better understanding of the more fundamentals.

Personal projects helps to learn

Some might argue that different people learn differently, but here I'm going to recommend no matter what your learning process is, be sure to create something as you learn.

I have 3 reasons:

Getting your hands dirty will help your brain to strengthen the neural pathways created by reading/watching. So you remembered things better and will be able to recall things later.
Nowadays, it is important to show evidence of your work when applying for jobs. The ability to demonstrate you skills is essential for getting a senior role. For example, if I'm applying for a tech lead role. I'm going to need more than a few line of text on a CV to convince them that I have the right skills for the job.
The last reason is actually the most important one for me. When trying to do something, no doubt I'll encounter problems. This forces me to think about digest the spoon fed information better.

Learn intelligently

In the world of software engineering, there are a lot of old technology and even more new technology. Learning new things is always more exciting and probably will provide better job security. But it is not always the case, for example languages such as C is 45 years old, it was introduced back in 1972. But it is still widely used today.

When people ask me where they should start when learning JavaScript framework. Instead of telling them go and learn React .jsor Vue.js (which in my opinion are the 2 best frameworks to learn right now), I would recommend them to go back and learn Angular.js 1. Why? Because it is easy!

The syntax is user friendly
When working on simple single page applications the set up is straight forward
There are a lot of resources online, in terms of plugins, questions and answers

jQuery is my first Javascript framework, but creating a single page application with it is messy and confusing for new beginners. Creating an application with Angular 1 people will learn the basics of web development. Even though the philosophy is not as up-to-date as the component based development, it creates a much better learning process in my opinion. This reminds me in school textbooks how sometimes children are taught the wrong information in Science, because the correct explanation is too advanced for them to understand (if you are interested I'm referring to the atomic structure diagrams).

After knowing the fundamentals, learning the other JavaScript frameworks will be easier because the person would understand why things happen certain ways. However, it is not easy to understand what is the best way to learn. Having a good coach to support you is extremely useful.

Find a role that creates opportunity to learn

When working as the same role for too long, it could prevent a person to reaching the next step. Not because the person doesn't want to learn, instead the role simply does not provide the opportunities to do so. For example, code maintenance can teach me a lot about how other people work and this would be extremely useful I was at the "learn" stage. However, I could never reach "expert" stage by maintaining code everyday. This might not be a problem if being an expert is not one of your career goals, but if it was then I would recommend changing the role or job.

I often find a person can learn 80% of the skills related to his/her work within 3 months. Continuing doing the same thing will have little benefit after that. Of course this is just an timeframe I came up with based on my experience, so take it with a pinch of salt.

Final words

This blog was more or less a brain dump of the ideas I have been having recently after helping to develop the new competency model. I hope it was a useful read, and please leave comments to let me know how you found it.

My Recent Assertiveness Training

Fri, 27 Jan 2017 18:00:00 GMT

I am always trying to be polite, friendly and calm at work. It is called being professional, and it is especially important for people working in professional services. The word professional is in the name kind of gave it away, so the last thing I want is saying something I regret to a client or teammate.

Being professional all the time can be challenging. Everyone has bad days, there are days I feel tired or impatient, and I am always worried that my actions might appear aggressive to others. In my head, I might be saying something reasonable, but other people might see it as I'm snapping at them when they asked a legitimate question. Being in professional services, the first thing to learn is saying no. The client always wants much more than what the team is capable of delivering. Not knowing when to say no is very harmful both to the customer and the team. The client will be disappointed later on when the project can't be delivered, and the team will feel stressed when the amount of work just keeps piling up.

What is assertiveness?

I searched "assertiveness" on Google, and this is one of the better definition I got back.

"Assertiveness means standing up for my personal rights - expressing thoughts, feelings and beliefs in direct, honest and appropriate ways." - skillsyouneed.com

Before attending the training, I saw assertive as being more confident and less passive. It turns out that being less aggressive is just as important.

My definition of assertiveness now would be "controlling emotions, focusing more on the goal. Achieve it using fair, friendly and neutral language while sounding confident".

Some key points I learned

Do not let the other person dictate your behaviour or thoughts

I often find myself distracted by other people's comment or action. When other gets angry with me, I would also lose patience because I think I'm right. When I'm having an argument with others, they could quickly change the topic to something else by raising some points which I am eager to challenge. For example, when people accuse me of things I quickly become defensive and start questioning them. This leads to the fact I stop focusing on the goal and will easily get tricked into talking about something else.

Being assertive means that I know what I want to achieve and focus on the goal. When people try to draw me away from the conversation, the best thing is to pause and use repetition assertion to highlight the points you are trying to make. No matter what they are saying, that can wait. Solve one problem at a time is always better than mixing more issues into the ball of mess.

Focus on the positive

To be assertive, having a clear head is the key. For example, before attending a meeting, I might already be thinking some people might be speaking a lot and complaining about everything. With this kind of negative inner dialogue in my head, it is easy to say something rude without realising. What I should do here is take all these negative dialogues and try to think about the positives. Instead of seeing the person complaining as annoying, I could see it as the person caring about what they are doing, and they have a lot of ideas on how to improve it. With this kind of positive inner dialogue, when things occur in the meeting my use of words will be positive and supportive.

Being assertive is tough to do, and everyone will find it difficult at first. It suggests I should disagree with myself, even if I were able to change the thought into a positive thought the whole process is still very tiring. Have you wondered why meditation helps people to remain calm and have a more positive view of life? Well, it could just be that when people meditate, they think about the negative thoughts. During the process, they can take a step back and see it from another perspective without making biased assumptions and judgements. Maybe it is time for all of us to start doing some meditation in this busy and stressful time of our lives.

Mirror, match, pace & lead

When approaching difficult people, this technique is one of the ways to start building the relationship. The method can be especially useful when the other person is being difficult with you (e.g. not showing much respect).

If the someone is sitting down having a conversation with me, the first step mirrors the person by slowly adapting to their posture.
When they change, try to match their position and then pace them as they change. Be careful to maintain the respect for the other people throughout the process.
After a period, start taking the initiative by start changing my position.
If the mirror/matching is strong enough, the other person starts copying my position changes. Now, I can lead the other person to the direction I wish.

Maintain the relationship

Being assertive is not just about speaking my mind and coming out directly with how I feel. Even though that is what people should do if they find themselves very passive. The important thing here is finding the line between honest and rude. Because one will strengthen a relationship and the other will chip away the foundation.

When speaking to people, it is important to get my points across. But at the same time, I should be asking the other person about what they want and seek clarification. If a person is trying to tell me something, there are the reason behind it, and without understanding the actual cause, the real problem will never be resolved.

One of the most important things I learned about being assertive is that while being assertive, I can still be respectful and polite.

Build up confidence

The first thing is to understand rights belongs to everyone, if someone can say or act a certain way to me, then I should have the same right to do the same with them, and vice versa. If the right only works unidirectional, then something is wrong here. If I think I have more right than other people, then I'm probably not being fair to them and acting in a not assertive way towards them. On the other hand, if I believe someone (e.g. my manager) has more right than me. Then my behaviour will change around him/her, most likely acting passively in conflicting situations.

One way to build up confidence when in a passive situation is by taking pauses, and let the other person speak. Pausing makes the other person feel respected, and give me to take a step back as well as get a chance to think about the conversation.

If I am in a position I think the other person is being too passive; I could empathise more with them when they speak. Asking open questions such as "what are your thoughts on…?" should also help to boost their confidence.

Techniques

Challenging beliefs

I have often heard people say things like "I NEVER want to..." or "ALL of them are...". When I want to challenge people speaking in this kind of certainty, this is the technique to use.

For example, what if someone said to me "ALL cats are lazy"?

Is the statement accurate today?
Does it contain generalisation?
Is it the truth or did the person just stated their assumption/interpretation as fact?
Is there any counter example where the statement does not apply?
If I can't think of any counterexample, can the statement be addressed with development?

No one knows everything; it is best to avoid saying things that are impossible to backtrack.

Say NO

The biggest problem with passive people is how much they want to avoid saying no to someone. When someone asks them to do something, they will agree to do it no matter how much they didn't want to do it.

The biggest lesson is to know it is ok to say no. Just come out directly and state that you won't be doing it. Try to avoid using nice padding words such as "I'd love to but..." or "I'm so sorry, but...".
The most respectful thing to do here is to state the reason and make it concise. People will understand if it is a valid reason.
It is also useful to empathise, to ensure the relationship isn't broken by showing understanding toward the other person.
The final useful thing to do trying to give suggestions, maybe there are other people or ways to achieve the same end goal.

If the other person doesn't take the NO very well, just repeat your reasons and empathise with them. Don't get dictated by their behaviours and stay assertive.

Dealing with unwanted behaviour

If someone is doing something that is offensive to me. The assertive thing to do is try to talk to them in private and raise the issue. Use the DEESC model to deal with this kind of behaviour.

Describe the situation – state the facts which cannot be challenged, let them know the situation using neutral language.
Express feelings - let the person know how much the issue affected my feeling.
Empathise - show some understanding, so they don't feel suffocated and turns aggressive by acting defensive. The goal here is to give them some room to continue the conversation in a gentle manner.
Specify a solution – explain what should happen in the future, so the person can easily understand what to avoid next time and never do the same thing again.
Clarify consequences - if the same issue happens again in the future, there are two ways forward. Using the carrot/stick method, either let them know how much you value the relationship so that they will stop. The other way is to threaten with some undesirable consequences which should hopefully prevent the same behaviour appearing again.

Constructive disagreement

When speaking with an unreasonable stakeholder, here is how to raise a disagreement. This could be making statements about your team's "poor efficiency".

Pause - Let the person speak their mind, it is important not to jump to conclusions before fully understanding the problem.
Ask questions - find the people believe and values, then it will be easy to understand why he/she is acting this way.
Summarise - extract the key points from their explanation and try to word it into a more assertive description.
Empathise - show understanding towards what was said and find common grounds which both people agrees.
State reservations/concerns/issues/points - lead the conversation, so the key points are covered. It could just be that the stakeholder lacked the full picture and misunderstood the situation.
Make suggestions to move the conversation on and find solutions to solve the client's problem.

Some final thoughts

Like most soft skills, this takes the time to build up the experience. But this training helped me to see everything from a different perspective. Even though I don't agree everyone should be assertive all the time, it is a useful skill to have when faced with challenging situations.

Migrating blog to Hexo

Mon, 23 Jan 2017 17:02:00 GMT

After using Ghost blog for over a year, I decided to switch to Hexo. Here's a high level comparison framework the two very different blog framework.

Ghost V.S. Hexo

Installation

Hexo is a static website generator, whereas Ghost is a fully functional blog website that allows creating drafts directly from the blog website. Since it is a fully functional website, I expected Ghost to be more complicated to install, however, I was surprised to find that the installation was very similar. With Ghost, all it takes is downloading it from the Ghost repo on Github and then extract the content from the zip file. Hexo is a bit more technical and requires running a few NPM commands in the terminal. Since I am using NPM for most of my projects, it hardly makes any difference for me.

Configuration and running

There is no clear winner when it comes to configuration and running the blog. Both are good for different things and not great at both. Ghost was super easy to set up due to the straightforward configurations, however, since it requires a server to host the web content the setup takes more time (e.g. spin up new server and installing software on it). Just remember it is not always necessary to install Ghost yourself, for example sites like DigitalOcean could deploy a Ghost blog with just a few clicks. The reason why I decided to migrate to Hexo is the nice compatibility with Github Pages. It means I can essentially host my blog on Github for free. So getting it up and running was as easy as running one command in the terminal. But the downside is that there were a lot of things to configure, and most of useful functionalities doesn't even have a default value e.g. Language.

UI and themes

Both frameworks support 3rd party themes and have similar number of resources to choose from. I personally don't have a preference when it comes to this, and even if none of the themes makes you happy there's always the option to create one yourself.

Functionalities

When it comes to functionalities, I think Ghost wins hands down. Not only the fact it supports multiple users, but the blog writing and publishing features feels fantastic to use as well. The split view of Markdown and rendered view is an amazing idea that doesn't exist in other frameworks I came across. It strikes the perfect balance between "focus on writing" and "focus on the end result". Image upload, draft auto-save and other functionalities makes writing blog posts fun to write. With Hexo, there's only one thing to do which is write it in a IDE or TextEditor. Uploading images and videos also involves more manual steps, definitely less enjoyable compared to Ghost.

For more advanced features such adding search, comments or changing themes I actually find Hexo much easier to customize. Not only because the configuration is in the same place, but since there's no server involved deployment is also easier. With Ghost, most of the time I had to SSH into the server and modify a bunch of files to get the same result.

Publishing blogs can't be easier on Ghost, with a click of a button it can be published or unpublished. For Hexo, it takes a few more steps (commit changes & deploy to Github Pages) and often requires the author to be using one machine due to the dependency on NPM set up and etc.

Long term support and upgrade

Both frameworks has a community behind it. As of now, Ghost has 21,720 Stars and 5,160 forks on Github whereas Hexo only has 14,282 Stars and 2,206 forks. This makes Ghost the clear winner. So why did I switch to a less "popular" framework? During the time I have been using Ghost, there has been a few updates and the only way to install them is by SSH into the server and download/unzip the newer version of Ghost and migrate the data across. This is not a good experience and definitely gets annoying after a few times. The constant notification reminder in the admin panel also doesn't help. For Hexo, it would be as easy as running npm install in the terminal.

Final thoughts

It has only been a few days with Hexo, and already I am enjoying it more than Ghost. Hexo is probably not suitable for non-technical people out there. But if you are a developer or comes from a technical background (basically knows how NPM and Github works) then I encourage you to give this a try. After the switch over, I don't ever have to worry about my server being taken offline and the hosting will always be free :)

Scrum development from a developer's point of view

Thu, 29 Dec 2016 22:45:00 GMT

Scrum development is great, it allows iterative development and much faster feedback loops. The concept is nothing new to a lot of people, and the adoption rate has been increasing. But I have yet to work in a team where I think we have achieved the maximum efficiency, in fact, most of the time we are far from it. The concept of Scrum seems easy enough, so why it is so hard to do it right?

I have used Scrum in more than 6 projects, some are over 13 months long and others could be as short as a few weeks. I even have the Scrum Mater certificate from Scrum.org, so the concept is not new to me. People always talk about a project from a business or a Product Owner's point of view, here I want to bring a new perspective and talk about some of the issues I have experienced as a developer.

All about the product and NOT the project

I think that people need to focus on the product, and not the work itself. Waterfall development methodology is all about the project, the goal is to produce something that is closest to the requirement spec as possible. Once the project is finished then that is it, the team built it is no longer responsible for the product. On the other hand, agile development focuses on the product. The development team must care about the product, suggesting ways to improve it over time. A good development team will fix issues that are not captured in the business requirement, they will also actively find and report issues about the product.

Less can be more

A team consists of fewer people might have a clearer focus and can self-manage much better than a bigger team. I have worked on a project where the two of us managed to rewrite the front end of a web application is a few days. For a different project, the team of around 5 developers achieved almost nothing for weeks. Skills and experience obviously come into play here, but I think the desire to do something and do it well is also a big factor. With a smaller team, it is much easier for the developers to come to the same understanding than a big team.

More isn't always more

A development team is always better to be kept consistent, adding or removing people is going to impact the velocity, always! Just because the burndown chart shows the team is working just as well, the code quality and the overall maintainability of the product will say otherwise. It is a common problem to have a set deadline for projects, this means rather than pushing back deadline business will decide to increase the team's size to increase velocity. But business doesn't understand that if a team isn't delivering enough, allocating 5 more developers is not going to help. If anything, changing the team is going to decrease velocity due to the extra training time spent bringing the new joiners up-to-speed. Sure, it might be a velocity increase a few months down the line but it can't be used to fix a short term goal.

Code quality

When working in a team of 3 - 15 people, it is hard to make sure everyone has the right skills, experience and same ways of working. This is one of the problems I have seen with some of the teams I have been in. As a developer, when there are one or more people producing crappy code, it drags down the quality of the product overall. It introduces bugs, makes the product more difficult to maintain and just simply makes developing the product less enjoyable. Nobody enjoys working on a product that they don't like.

Good Product Owner and a good backlog

A good product owner (PO) is someone who is always accessible to the team, he/she should be there to answers questions as they come up. Waiting till the next day is a sign that the PO should spend more time with the development team. The PO should be actively shouldering any blames made to the product, as he/she is the sole person responsible for the product. In order to avoid blades, the PO should clearly understand the business requirements and break them down into tasks and user stories. This leads on to my next point, PO is also responsible for creating the product backlog.

All future changes must be captured and stored in the backlog, tasks and user stories should be clear to the development team. It sounds straightforward, but the backlog needs to be managed, maintained and updated just like the product's code base. This is always something that is missed and causes dragged out arguments in planning or refinement sessions.

Testing or no testing?

A good development team will always have automated testing, this includes unit, integration and end-to-end tests. The team should find a balance between the different type of tests, and focus on writing the least number of quality tests as possible. In a recent project, we have gone overboard with regression tests, not only the tests takes hours to run locally but also means write tests takes 2/3 of the total development time. I can say for sure that is a terrible place to be, so definitely try to avoid the mistakes we had made.

Why Scrum, why not Kanban?

I personally prefer Kanban over Scrum, because Scrum has all these scrum ceremonies to follow whereas Kanban is extremely simple. With Scrum, new teams often miss important points and use it wrongly, and for a more experienced team, it is easy to forget about the big picture and get caught up in scrum ceremonies like Sprint Retrospective. It is extremely difficult to find a good balance, and all the ceremonies could become a distraction for the team and take away the more important focus on the product they are trying to build. I am certainly not the expert in this field, but I think for a new development team picking up Scrum or Kanban will take the same amount of time. With Kanban, there's a much clearer focus on the product.

Google Sheets as a service

Sun, 25 Dec 2016 22:44:00 GMT

The most difficult choice to make when creating a simple web application is not choosing the right framework or colour scheme (for any backend devs out there). Instead, the struggle comes from deciding if a backend is required or necessary. A lot of the time, it is hard to justify setting up a backend service for a simple web application. More times than not, the only thing I really need is one or two simple APIs to bounce some data around. For that, I don't really want to spin up a new EC2 instance on AWS, create a backend, setting up a database and etc.

There are plenty of services out there which already do this kind of stuff I know, AWS has Lambda for serverless services. Even with Lambda functions, I still need a database to store any data. The closest service I have found is called Orchestrate. It essentially lets me run a database then using different API calls I can query different data from it. The downside of both of these solutions is that they are paid services, sure it might not cost a whole lot but no doubt it is a bad way to implement open source projects. With things tied to my bank account, it then can't be shared publicly and the open source community won't be able to contribute to it. So this is where Google Sheets comes into play!

I have been a massive fan of the entire Google Doc suite, but I only found out recently that there are publics APIs for querying data from any published Google Sheet documents. Sounds amazing right? If I need an API to return some data, I can just create a Google Sheet document then populate it with whatever data I need. Compared to my previous method, which involved storing data as a JavaScript object in the application, using Google Sheets means data and source code can be decoupled. Another thing to point out is using Google APIs, all the request balancing and performance are passed to Google there is no additional work on my end.

Note: For the purpose of this blog, I'm going to use the non-authenticated API which required the document to be publicly viewable in order to request data from it (bypass authentication). So if the project you are working on has sensitive data, I would recommend that you use the authenticated implementation as suggested here by google.

How to request data from Google Sheet document

1. Publish the document
On the Google Sheet document, click on File then Publish to the web.... Once the popup opens, just click on the blue Publish button. Here is my example Google Sheet document.

2. Find out the document ID and worksheet ID
Once published, copy the shareable link from the same popup. Now open it in a new tab, you can see it is an HTML version of the same document. There are a bunch of formats you can get it in. I personally recommend JSON, which is not on the list. Don't worry this is completely fine, just leave the data type as Web page.

3. Return data as JSON
Use the following API to request data back as JSON, just replace DOCUMENT_ID and SHEET_ID respectively with your own.

https://spreadsheets.google.com/feeds/cells/DOCUMENT_ID/SHEET_ID/public/basic?alt=json

DOCUMENT_ID is the key of the spreadsheet you want to retrieve, it can be found in the URL. In my case, it is docs.google.com/spreadsheets/d/==1MYzZjtsujLXH43F_iQl242OV045UahkENWkv0nF83Qo==/edit#gid=327263911.

SHEET_ID is the positional or unique identifier of the worksheet, the default ID is od6 for default sheet name Sheet1. If you don't know what your SHEET_ID is, then it is worth using the following API to dig through and find it.

https://spreadsheets.google.com/feeds/worksheets/DOCUMENT_ID/private/full

The returned data is a massive JSON object, unfortunately, I had to write my own logic to extract the relevant data from it. Depending on how the sheet is laid out the code implementation to extract it will obviously differ. For this reason, possibly it is best to only consider Google Sheet as short term or temporary backend service.

4. Additional APIs to insert, update, delete data from the document
For anything else more advanced, please refer to the official Google API documentation.

Using vue-cli (vuejs) with Github Pages

Sat, 24 Dec 2016 22:43:00 GMT

Vue.js is a fantastic JavaScript framework, I have been using it to build small projects in its very early stages. The fact you can set up your projects in so many different ways is amazing, there are ways to set up a quick hack project or to build something is maintenance ready. One of the best ways to set up Vue properly is using Vue-CLI.

Recently, I have found some issues with the Vue-CLI when I tried to use it with Github Pages (an amazing way to set up a quick prototype, I'll prob write more about it in another blog).

Problem 1: Builds to `/dist` folder instead Github prefered `/docs`

This turns out to be super easy to fix, simply open /config/index.js file and find the following lines.

...
index: path.resolve(__dirname, '../dist/index.html'),
assetsRoot: path.resolve(__dirname, '../dist'),
...

Then change them to the following:

...
index: path.resolve(__dirname, '../docs/index.html'),
assetsRoot: path.resolve(__dirname, '../docs'),
...

Problem 2: File path is broken and shows blank page on Github Pages

After fixing the first problem, pushing the changes to master. Then load up the relative URL, then I realised there is still nothing being displayed. Here are the console errors:

Looking at the build info in the terminal, it also says the build directory should be served over an HTTP server.

)

After looking around at the config more, I also found a relatively easy fix. Again open up /config/index.js file and find the following line.

...
assetsPublicPath: '/',
...

Now just change the absolute path to a relative path, then voila, everything should work on Github Pages as expected.

...
assetsPublicPath: './',
...

Also feel free to remove or comment out the warning in the terminal show above, since we have now fixed the problem. Within build/build.js, the following snippet is responsible for showing the warning.

console.log(
  '  Tip:\n' +
  '  Built files are meant to be served over an HTTP server.\n' +
  '  Opening index.html over file:// won\'t work.\n'
)

Web security - OWASP top 10 vulnerabilities

Thu, 22 Dec 2016 22:41:00 GMT

Open Web Application Security Project (OWASP) is a non-profit organisation that was driven by a community which wishes to make the world wide web a safer place. The latest top 10 document can found here. As you can see it is created in 2013, so many of the issues would have been addressed already by new technologies. In this blog, we will be explaining the top 10 vulnerabilities and how they affect the technology today.

Injection

This is something we hear about all the time, mostly known for websites written in PHP that uses MySQL as its database. Since the combination was fairly old technology and it did not have built-in defences against such simple attack. By typing SQL queries into inputs and URL parameters, attackers are able to perform unauthorised actions such as return other user's data or even drop (delete) production databases.

The fix for this issue wasn't difficult. Any website can prevent this by adding type checks on form inputs and URL parameters to ensure injection was not possible. An even easier fix would be to choose or a database management system that has a built-in defence for SQL injection e.g. PostgreSQL. The problem was never with the difficulty of the issue, instead, it was just developers lacked the understanding of this vulnerability or was too lazy to implement checks for 100% of the code base. As a result, a single point of failure could put the entire website risk.

This issue is still an important problem today, this is because there are still many legacy systems out there. If a web was built on today's frameworks, I feel this wouldn't be a problem at all. Modern frameworks are very good at preventing the developer from write code that is prone to this attack.

Authentication/Session management

Many websites require a user to login in order to access more features on a website. To avoid constantly asking the user to type their username and password, a session is used to tell the website the user is already logged in and who he/she is. This vulnerability is when attackers steal user sessions and pretend that they are authorised and have access to information they shouldn't have.

This is a much broader issue since not all website is implemented the same and depending on how it is created the vulnerability could be different. In some cases, the problem could be that a session ID is sent as part of a link that allows anyone to login as the user. There are also ways to steals people's authentication cookie by sniffing over insecure networks. The best way to prevent this is to always ensure session cookie has a timeout period and is regenerated upon every authentication.

This issue has not changed since 2013 since the front end session management and authentication has hardly changed. Sure, there are ways to ensure the websites' security. But the fundamental issue never went away. If the attacker is clever enough and the user is silly enough, there are plenty of ways for the attacker to steal their session e.g. Cross Site Scripting.

Cross Site Scripting (XSS)

Used to be a popular method to embed snippets of code on a legitimate website that could take a user to fake websites and trick them into typing in their credentials so it can be captured by the attacker. There are also many other reasons to perform this type of attack. A simple example of this could be the attacker noticed that a segment of the URL parameter is mapped to a DOM element unconditionally. Thus, by changing the URL parameter to a snippet of JavaScript code could essentially manipulate the DOM element to do as the heart desires.

In modern frameworks, especially front end, there are very little ways to deploy this kind of attack. Frameworks such as Angular, React and etc does not rely on URL parameters as much as jQuery websites do. These frameworks are also much tighter on URL parameters and routeing, it doesn't leave any room for error. On the other hand, modern frameworks also utilise two-way data binding, so gone are the days where values have to be passed as a URL parameter to a DOM element.

Direct Object References

This kind of attack is to do with authorisation, in older websites database record IDs are referred to in the URL parameter (again, yes). If security is not in place, user or an attacker could simply increment the ID value and get someone else's data. Ideally, there should be a check which only allows the user to request their own data. Surprisingly, a few month back a colleague of mine actually found this kind of issue on a live production website for a well-known company. It is pretty much common sense, but yes companies often make this kind of mistakes.

Unlike older websites, newer sites use RESTful APIs to make requests to the server and the URL parameters are hidden from the browser address bar. But the problem is still there, an attacker could still fake a request (a tiny bit more work than altering the URL) to the server asking for someone else's data. So correct security checks must be in place, this is extremely important for public facing APIs since everyone on the internet could access them.

Security Misconfiguration

There are new system and software vulnerabilities are found all the time, thus it is very important to keep everything up-to-date. Companies, especially larger companies, are always reluctant to upgrade because new changes could break their software and cause incompatible issues. I mean look how many times companies paid Microsoft to not drop support for Windows XP when windows 8 was already out. Imagine the vulnerabilities on a system that is almost 13 years old. Other ways to reduce risks of vulnerabilities are to disable ports, services and accounts if they are not being utilised. Speak of accounts, ensure all default passwords are changed to something more secure, e.g. router or database passwords.

There are probably a million other things to do and your system might still be vulnerable. But here are 3 areas to start securing:

Turn off features that are not needed and apply the "principle of least privilege"
Change default settings, and ensure production-ready (turn off debugging, and limit admin access)
Always try to stay up-to-date in terms of tools, software and systems

Sensitive Data Exposure

This issue covers the lack of SSL (HTTPS) implementation of websites, using secure connection prevents a lot of issues such as "Man in the middle" attack. The password is obviously one of the most important user data, so the lack of encryptions could lead to database table being decrypted and leaked after one point of failure e.g. Inject. If the encryption is strong enough, even if the data is stolen the attacker can't do anything with it (a strong encryption could take hundreds of years to decrypt). It is almost silly not to use strong algorithms to ensure the security.

Companies often try to capture as much of user information as possible, what businesses don't understand is that unless the data adds value recording it simples puts the company more at risk. By capturing fewer data, it might mean a much better end result if the system was compromised and user data was stolen. As a user, I definitely prefer to only have my email address leaked, compared to all of my personal information e.g. date of birth, address and etc. Having more user data might attract more attackers in the first place, since they know they can get all the data they need just by hacking your company. So to both reduce potential damage and cost, only record data that is absolutely necessary.

HTTPS is currently being pushed by a lot of companies, Google being one of them. Soon Chrome will start marking websites as insecure if they do not have HTTPS connection. Let's Encrpty also provides free SSL certificates for any website for free, again to push for a more secure online experience.

Function Level Access Control

This issue is similar to the "Direct Object References" section, but just on a more general. It is important to prevent the UI from showing unauthorised functionalities. If a user should not have access to a button don't just disable it, hide it completely. It is also worth noting just because a page is not navigatable doesn't mean the attacker can't find it by looking through the code, so prevent user going to that page altogether in the routeing.

As mentioned already, it is crucial to perform all the security checks on each and every public facing APIs. This includes checking if the user is authenticated, authorised, has the correct privileges, and etc. It is important to have validation on the front end, but API level testing is more important. Since it is extremely easy to bypass front end validations and checks.

Cross-Site Request Forgery (CSRF)

This attack is very clever, almost like a puppet master controlling the user to do their biddings. Often, the attacker will trick the user from making requests to important requests (e.g. doing a bank transfer to the attacker). This could be a fake website, this could also be part of an XSS attack.

To prevent this type of attacks, there are a few things we can do:

Anti-forgery tokens, that is generated randomly instead of sending the same token every time
Valid requests only from known origins and referrers, fraud detection patterns
Some browsers already have built-in defences for this e.g. Chrome prevents cross-origin requests

Using Components with Known Vulnerabilities

Component and version could be discovered by studying the request headers, file name patterns and etc. We already know that the components have vulnerabilities, so it is a lost battle right? Kind of, it is pretty difficult to defend against this kind of issues. The best thing to do is to update to a newer version with a fix as soon as possible or to move away from unsafe components.

Unvalidated Redirects and Forwards

You probably already noticed as we move down the list, the amount of description and text is decreasing. This is because attacks such as this one have a much less impact than issues such as SQL injection. The concept of this attack is when attacker changes redirect links on a website, so they are redirected to other malicious websites. At this point we are assuming, the original website has already been compromised and this attack is simply a way to spread malware to more users. This is so low in terms of importance, companies like Google doesn't even offer bug bounties for reporting this kind of issue on their sites.

Conclusion

Security is a constant battle, it is also an on-going issue in terms of support and maintenance. As explained already some of the issues are already ageing, and might disappear in a few years when all the frameworks contain built-in support for these type of attacks. At the same time, more issues will appear, for frameworks like Angular its implementation is very predictable thus may allow attackers to think of new ways of attack. To conclude, I feel some of the issues are definitely worth bearing in mind when developing new products. A lot of the issues mentioned above are to do with the backend system and backend development has not changed that much in the past few years. Although new database technology such as MongoDB does not have any vulnerability towards query injection and more websites will be adopting HTTPS, so I guess somethings are changing and for the better too.

Using NPM scripts correctly

Sun, 18 Dec 2016 22:39:00 GMT

Nowadays, most frontend projects and JavaScript projects use NPM (Node Package Manager) to manage dependencies. This is often done through the file package.json. This file, as the name extension suggests, contains a JSON object about the project. The follow snippet shows a very simple example of what a package.json file might contain, for more information about what each section means browsenpm will offer a much better explanation. If you have not seen a package.json file before, it might be best if you do some quick reading before continuing reading.

A little bit of background info on package.json

{
    "name": "package-json-sample",
    "version": "0.1.0",
    "description": "A sample package.json file",
    "main": "index.js",
    "scripts": {
        "start": "node index.js",
        "test": "echo \"Error: no test specified\" && exit 1"
    },
    "dependencies": {
        "express": "^4.13.3"
    },
    "repository": {
        "type": "git",
        "url": "https://github.com/some-url"
    },
    "keywords": [
        "package", "json", "example"
    ],
    "author": "Hao Dong",
    "license": "MIT"
}

Now we got that out of the way, let's talk about why in my opinion how to set up projects correctly by using NPM scripts.

What are NPM scripts?

NPM scripts are actually super customisable and powerful, it is a shame that they are not always being utilised fully/correctly. You can run simple commands to do something like cleaning up projects or more complex chained commands to combine, compile and build projects.

So how do they work? Within scripts object in the example above, there are two NPM commands start and test. When declared in package.json like this, it means if you navigate to the directory in the terminal and you can execute them by typing npm run start and npm run test (npm start or npm test also works because start and test are reserved keywords). In our case, npm start will start a node app using a file called index.js and npm test will print out "Error: no test specified" in the terminal.

Why should we use it?

1. Using local rather than global NPM packages

This is probably the most important reason in my opinion. Tools such as Bower, Gulp and etc requires to be installed as a global NPM package, like most cases in software development global is never good. This causes a number of problems, just imagine:

You are working on two different projects but they require different versions of the same NPM package (e.g. web driver). If you install it globally, how much of a pain will this be to constantly change between them?
What if you are on a newer version of a package than your co-workers? Or even worse, what if everyone is a different version to the production environment? When things start failing, it will be hard to track down issues and creates confusion.

You might ask why do people keep installing NPM packages as global then? Well, if you want to run something like gulp test in the terminal Gulp must be installed as a global package, otherwise, gulp is undefined.

How do NPM scripts fix this? Well, we don't have this issue because NPM script will try to find the relevant packages within node_modules directory. This means that I can run npm test (assuming the script is "test": "gulp test"), and it will use the locally installed version of Gulp. The local package version should always be defined in the dependencies or devDependencies objects within package.json file. Therefore, everyone will be using the same version of Gulp this way.

2. Easy to find

Since package.json is used by a lot of tools and it appears in almost every JavaScript project, therefore it is the first place developers will look to understand a project (still important to have a good README file!). So why not put the most important information in the place people will look first?

3. Centralise commands

Different projects use different JavaScript package managers, bundlers, and task runners. E.g. NPM, Bower, Browserify, Gulp, Grunt, Webpack and many others. Rather than having each of these tools doing something different (e.g. Bower as bundler and Gulp as task runner), why not put the commands in one place and centralise them? NPM scripts can execute commands for any of these tools. If I want to run something in gulp I can just have a script like "test": "gulp test", then run npm run test in the terminal to execute it. So when developing I can just run one command, instead of a whole bunch just to get it to build my project.

4. Complex/chained commands

As mentioned before NPM will allow chaining commands, if you want to something quite complicated this is super useful. The snippet below shows what a chained command looks like.

"scripts": {
    "start": "npm run command1 && npm run command2 && npm run command3",
    "command1": "echo \"Running command 1\" && exit 1"
    "command2": "echo \"Running command 2\" && exit 1"
    "command3": "echo \"Running command 3\" && exit 1"
}

After looking at the chained commands you might think all the npm run is a bit ugly when chaining scripts. Well, it happens someone already thought of this. There's a nice package called npm-run-all which lets you write much nicer chained commands. Using this package, you can write nice chained commands in the next section.

The recommended way of using NPM scripts

As of now, the follow snippest shows my recommended way of using NPM sripts and structuring projects.

"scripts": {
    "dev": "npm-run-all build:clean build:html watch:html build:css watch:css watch:js",
    "build": "npm-run-all build:clean build:html build:css build:js",
    "build:clean": "rm -rf ./docs/*",
    "build:html": "cp -r ./src/*.html ./docs/",
    "watch:html": "onchange './src/*.html' -v -- npm run build:html &",
    "build:css": "cp -r ./src/*.css ./docs/",
    "watch:css": "onchange './src/*.css' -v -- npm run build:css &",
    "build:js": "browserify ./src/app.js -o ./docs/bundle.js",
    "watch:js": "watchify ./src/app.js -o ./docs/bundle.js -v",
    "test": "mocha ./test/**/*.js"
}

The above example is taken from one of my recent project Finacial Calc if you are interested in looking at the folder structures and etc.

Easy ways to mange Email Inbox

Mon, 12 Dec 2016 22:37:00 GMT

Keep a clean Inbox is not an easy task for even the most organised people because there are simply too many spams. There is only so many filters you can set up before it gets really annoying and tedious. Here are just some simple, but useful ways to keep a clean Email Inbox.

Unsubscribe from all marketing emails

It can get overwhelming very quickly, after signing up to a few online stores. The best thing to do is to change the email settings as soon as the account is created. This way you won't ever need to worry about it later on, and you won't get into the awkward situation where you need to reset your password first before unsubscribe if you end up forgetting it.

There is a nice tool which can help you if it is already too late to start from the beginning. It is called Unroll Me, it will go through all your emails and find the marketing emails. Then show them in a nice elegant list, which will allow you to unsubscribe from them in one click. I would recommend using this if you have no sensitive information in your Inbox because you are giving the company to parse all of your emails. There's no harm in being careful right?

Delete emails you don't need

If there is an email you clearly won't read and never will, then why not delete it rather than letting it pile up in your Inbox? Sure, mail clients such as Gmail has very good search functionality. But they will still slow you down more or less. If you are unsure about an email, definitely keep it. But for some useless marketing material sending it straight to the bin is not a bad idea.

Use Gmail's Inbox App

If organising email is not your thing, then Google's new (well, sort of) Inbox app on both mobile and the web app will make things really easy. It is going for the philosophy that no one needs to delete emails or manage them, the important thing is for their search to understand what you want and always return the emails you wish to see. You can pin important emails and set timers on emails reminders, but there is no need to do anything more. The app is meant to handle the rest for you.

Use Gmail's support for different alias

For easy Inbox management, use the infinite number of Email addresses Gmail gives you. This is a cool tip I learned recently, and you can read more about it here.

One trick you may or may not have picked up about Gmail is that you can add in periods anywhere in the front part of your address and it makes no difference whatsoever: john.smith@gmail.com works just the same as johnsmith@gmail.com. What's more, you can add a plus sign and any word before the @ sign (e.g. johnsmith+hello@gmail.com) and messages will still reach you. If these tweaks make no difference, then why use them? One major reason: filters.

So assume my Gmail address is name@gmail.com. Rather than using this email address to sign up to technology news website, I can actually sign up to newsletters using name.newsletter@gmail.com. With some simple filters, I can easily manage all of my subscriptions from here. It makes it easier to search and maintain later on as well.

Fixed Windows 10 cannot shutdown issue

Mon, 12 Dec 2016 18:23:00 GMT

I normally bounce around between Linux, Mac OS and Windows for different tasks. I mainly use Windows for gaming or consuming media, so there isn't much software installed on my machine. Not too sure when it started, but I have been having this weird issue where Windows doesn't turn off my machine when I click on Shutdown. It would just Log me out rather than do a full shutdown. What's even weirder is that Restart works just fine. I was doing a bit of searching around, I managed to resolve it. But I didn't find any of the instructions that was easy to follow. Hence why I'm creating this post.

Step 1

Go to Control Panel and search for 'power'. As soon as you stop typing (wait a couple of seconds) then you should see the results coming back.

Step 2

Click on Change what the power buttons do. It should take you to the follow screen.

Step 3

Ensure the Turn on fast start-up (recommended) option is unchecked like shown in the picture above.

For some reason, this functionality is causing problems for a lot of people. I guess this just proves that I am correct not choosing Windows for development.

Stopping ng-click from being fired in Angular (1.x) directive

Fri, 18 Nov 2016 22:34:00 GMT

In the project I'm working on right now, we needed a way to disable some input fields based on a condition which is pretty common. I ended up writing a directive to reduce the amount of duplicated code and keeping it clean and simple. However, there was one problem, we didn't use the default checkboxes. Instead, we ended up using the icons from fonts-awesome library (Checked Icon, Unchecked Icon).

Here is an example of how we implemented the checkbox:

<i aria-hidden="true" ng-click="doStuff()" ng-class="checked ? 'fa fa-check-square-o' : 'fa fa-square-o'"></i>

<input type="checkbox" style="display: none" ng-model="form.input1">

Applying the directive on the input field doesn't really work since it is hidden. The right thing to do is to apply the directive on the <i> tag itself. However, disabling <i> doesn't do much, it will still allow clicking and thus firing off the function within ng-click.

The Solution

I spend hours looking for a simple solution that will do something along the lines of event.stopPropagation() when the checkbox icon is clicked. This should be enough to prevent ng-click being triggered.

At the end, I managed to find the solution (see below).

app.directive('stopNgClick', [function () {
    return {
        restrict: 'A',
        link: {
            pre: function (scope, element, attrs) {
                element.bind('click touchstart', function (e) {
                    e.stopImmediatePropagation();
                    e.preventDefault();
                });
            }
        }
    };
}]);

The problem I was facing was due to the directive is doing default postLink, and after changing it to preLink everything worked as expected.

Additional Reading

Compile, Pre, and Post Linking in AngularJS is a good article that focuses on the execution order of Linking and Compile.

Use brain or password manager

Mon, 31 Oct 2016 22:33:00 GMT

I struggled with this question for a long time. It is contradictory in a way that using a password manager should make everything more secure, but using a master password introduces a single point of failure. So is it more or less secure? Given the number of accounts I have created over the years, I would say going with a password manager is the better choice and stop repeating same passwords on different sites. This way hackers can only hack into one of your accounts if they hack into a site.

In terms of password managers, there are quite a few choices to choose from currently. I have tried few of them and want to share my experience here. I will start by listing some of the problems I have experienced, some personal tips, and finish off with a few recommendations.

Avoid password managers that store your data on their servers

The idea of having all my security details stored on a company's server makes me very uncomfortable. Not only can hackers crack open their server and have access to all of my details, it makes my insanely long master password essentially useless. Putting security aside for a minute, imagine you put all your passwords on a service like Lastpass. One day their server goes down, then you cannot even access your accounts. I think the best way is for the password manager act as a client, the storage of the passwords should not be their concern. I prefer saving the password database on cloud storage services such as Google Drive, One Drive and etc. This means I don't have to worry about people stealing my data from external servers. Sure, they could try to steal the password database, but since it is encrypted it will take them maybe 100 years to decrypt the content. Alternatively, they can steal my master password. However, without access to my cloud storage, the master password is useless on its own. To summarise, cloud storage beats service provider's servers.

Bad platform support

Do not choose a service that does not offer both mobile and desktop support. This was a tricky problem because I need it to work with Android, Mac, Windows and Linux. This was often the deal breaker, just think it through since migrating to another service later will be time-consuming and expensive.

Fingerprint support is a must for mobile Apps

I have tried a few password managers on Android and I was surprised how they did not include support for fingerprint access. Instead, they ask the user to enter an extremely secure master password. Can you imagine every time before you access your amazon account you have to spend about 10 seconds login into the password manager app and find your Amazon password? Compare this to fingerprint access which will take maybe half of a second? So another thing to look out for is fingerprint support on mobile.

Subscription VS one-time payment

Personally, I much prefer one-time payment services because it is generally cheaper in the long run. However, something to consider here is that subscription-based service may have better long term support due to the fact they will have more money to invest back into the product.

My recommendations

I have talked a lot about what to look for and things to avoid. Here are a few products I would recommend, they are intended for different user groups and hopefully at least one of them will be useful for you.

There are quite a few open source projects out there, there is no reason not considering them because they are secure and is free to use. The most popular one is KeePass Password Safe by far, it supports both desktop and mobile. It also lets user choose between local password database or cloud storage, amazing! The only drawback is the user experience, compared to other services its desktop and mobile apps are not as well polished.
SafeInCloud is my current service. It is well polished, has browser support, and is available on most platforms. It can be purchased with a one-time payment. Since this is not a subscription based service, I am hoping the project doesn't get abandoned over time. There is no Linux support right now which might turn some people away.
My last recommendation is LastPass. I have used it for more than a year, and it is all you can hope to get for a password manager. It has auto-fill on desktop browsers and mobile apps. It supports all platforms (even Linux since it is browser based). Desktop features are free to use, however, mobile apps require paid subscription. Even the paid subscription is very reasonable, right now it is $14.40 for a year. Lastpass is almost perfection, my biggest problem with it is how it stores all user details on their server. I personally have moved away from them for this exact reason. However, if you only want to use it for non-critical account info like random website login then it is definitely the go-to choice. Otherwise, I recommend looking at the two alternatives above.

What I love and hate about JavaScript

Sat, 09 Jul 2016 22:32:00 GMT

JavaScript is a dynamically typed language, and that gives it a lot of edge in my opinion. It is one of the most used programming language right now. So definitely worth learning even if people do not want to head towards this direction in their careers.

I really enjoyed coding in JavaScript when I first started working with it. Some languages such as Java can be dry and boring to learn, due to the strictness of its syntax. For a new learner this reduces work efficiency, and makes the coding experience less enjoyable. With JavaScript it is a different story, the language is flexible and allow programmers to code the logic of their applications without having to be constantly interrupted by errors messages.

Using undeclared variable in JavaScript is allowed and everything will work just fine. Whereas other languages such as Java will throw about hundreds of errors (okay, I might have exaggerated) when compiled. Here's an example with undeclared variables.

y = 8;
x = 5;
z = y + x; // this will give you 13

This is amazing, because when I am trying to code something there's no need to think about the type or memory allocation when compared to languages such as C. It might not seem like a big thing, but it takes an extra step from the process of converting your thoughts into code.

Now, on to the problems. Even for experienced developer it is still a terrible idea to code like this. The lack of strictness makes it very hard for people to understand the code. JavaScript does some weird interpretations of undeclared variables.

If there is a bug, it will be very difficult to spot even for the person who wrote the code. It doesn't take too many badly managed code base to convince someone to write good code. Let's go through some examples.

Undeclared variable within a function becomes a global variable:

// can use x here outside of the function

function foo() {
    x = 5;
}

Look at this stackoverflow question and see how confusing it is to just check if a variable has been set
Types are not what you expect

var x = "Hello";             
var y = new String("Hello");
(x === y) // is false because x is a string and y is an object

Type conversion when you don't expect it

var x = "Hello";     // x is a string
x = 5;               // now x is an integer

If the above is not enough to convince you dynamic type is not enough there are plenty of other examples on the web. In general this makes large web applications extremely unstable and difficult to maintain. Through experience I think everyone will slowly feel the pain. I was such a strong supporter of dynamic languages, but looking at code bases with terrible code quality makes it really hard to appreciate the benefits it brings.

Luckily, we can stop this by doing the following:

Using 'use strict'; within functions/scripts, this will enforce some strict nature in the code
Follow a JavaScript Style Guide within your team to enforce the coding standard
Using a JavaScript Linter with your IDE or Text-editor
Add a linter check in the CI process as well to doubly ensure code quality

Learning React and Flux - Introduction & Comparison with AngularJS 1.x

Wed, 08 Jun 2016 22:27:00 GMT

Recently, the community of React has been exploding. AngularJS 2.0 has not convinced me that it is worth investing time in. So I have decided to start learning React with Flux Architecture. This blog is the summary of my recent learning of React and Flux. I am just writing about my thinking/learning process, so apologies if the structure is not very easy to understand.

I just want to clarify I am using AngularJS 1.x for comparison just to show some of the key differences I have learned with React, not to say AngularJS is a bad framework. I am also aware that AngularJS 2.0 is out and Angular team also fixed a lot of the problem mentioned in the blog. But I'll be using AngularJS 1.x for comparisons, because I am not familiar AngularJS 2.0.

React is very different to Angular, both in terms of philosophy and architecture. The first thing to know is that, React tries to add HTML in JavaScript code and AngularJS tries to add JavaScript code to HTML.

An example of Angular Code:

<div ng-controller="my-ctrl">
  <div ng-class="commentBox">
    Hello, world! I am a CommentBox.
  </div>
</div>

An example of React Code:

var CommentBox = React.createClass({
  render: function() {
    return (
      <div className="commentBox">
        Hello, world! I am a CommentBox.
      </div>
    );
  }
});
ReactDOM.render(
  <CommentBox />,
  document.getElementById('content')
);

So what else did I learn?

when AngularJS was originally developed back in 2009, its two way binding shocked the web development world. It is miles more elegant and developer friendly compared to jQuery which dominated the JavaScript world before it. Two way binding is great (for small/single-page applications).

AngularJS is designed with MVC architecture. HTML is the View and it is what the user sees (the DOM), JavaScript is the Controller (the business logic behind views), finally the Model is the data shown to the user in the view and that the user interacts with. The main benefit of MVC architecture is that, now it is also possible to separate out the HTML and JS code into different files (No longer will you see PHP, CSS, JS and HTML in the same file, massive win!).

AngularJS' problem arrises when you try to build large or complex applications, due to the inefficient way it renders the page. AngularJS tried to address the issue in later releases, and people have found plenty to "tips" and hacks to improve its performance. But the problem hasn't gone away, and as the application scale it will remain to be the blocking factor. At least, this is what I experienced with it. For me the reason to look towards React is the performance gain when compared to AngularJS.

React is component based architecture, it wants you to forget the current structure of your page and re-imagine it in different component blocks. Menu header can be a block, since it is a sufficient element and is independent to other elements on the page. By separating its logic away from the rest of the page will make it easier to develop, test and maintain over time. React wants you to do this for all elements on the page. As a result this means every page is made up of these components and together they form the entire site. It also allows refactoring of repetitive code, again another big plus.

This makes maintenance a lot easier in large applications. Imagine we have to update the look and feel of all the checkbox on the site, ideally we have already made checkbox into its own component. So it is as simple as going into the component file and making the changes we need, upon refresh we should see our changes on every single checkbox on this site. In AngularJS, we need to go to each page and update it. If you miss one, well, no one will know until someone reports it as a bug.

If you are interested the differences between these two frameworks, there are plenty of comparisons so feel free to read up on it. But this is as much as I'll cover in this blog.

Why is React's performance better

To me the main differences is not the components implementation, for example you can use the React architecture in an AngularJS app if you wanted to. In fact, I have personally worked on this kind of projects before. I have to say it does help the problem a little. But this does not solve the performance issue we mentioned above. So how does React solve it?

React is super clever when it comes to rendering. Like a person, it thinking about something first, decide what to do, then go and do it. What do I mean? Let's look as an example:

With AngularJS, it tries to do exactly what it is told. If you have a list of 100 items, 1 is updated, then it tries to render all 100 items again. Sure there are ways to fix/optimise it, but not everyone will know how. With React however, it has this concept of Virtual Dom, here it does exactly what AngularJS is doing. But since it is virtual and exists only in memory, thus it is super fast to compute. Then React will use its 'diff' algorithm to compare the virtual Dom and actual Dom, where ever it finds a difference it will update it and render the differences. In reality, this means a very small amount of re-rendering has been done on the actual Dom. Hence why React's performance is so much better.

Why Flux is the way forward for large applications

Angular's two way binding is great, it is easy very developer friendly. But as more and more functionalities are added to the app, things will get a little hard to keep track of. In the diagram below, we are displaying the same information in 3 places. If one is changed, all needs to be updated. This is a very simple example, but imagine this 10 or even 100 times more complex. This problem is only worse with React, since developers need to keep track of all the components than just a single page.

Facebook has certainly had a lot of issues in this area, this is why they created the Flux architecture. It restricts the web application to one way data flow (see diagram below).

Wait, isn't this taking a step back and going back to how jQuery does things? Well, not exactly, one way data flow is not the same as one way binding. The core concept is this:

Every time user takes any action within the app, the action is recorded and then broadcasted to every component. It is up to each component to decide what to do with this action. So rather than calling 3 functions updating the value in 3 different places, everything will be updated automatically if they receive the relevant actions. This massively reduces complexity within large applications. Going forward, I definitely want to be using Flux in my React applications.

A final comment, I find React a little difficult to read compared to AngularJS. Maybe because I am just too used to writing Javascript code into HTML and not the other way around. Luckily, React created JSX to help developer to write code more efficiently, but in my opinion this makes readability of the code even worse.

Sources:

"React and Flux in the real world" tutorial by Anthony P. Alicea.

Hidden skills of a good developer

Sat, 09 Apr 2016 22:25:00 GMT

Programming is a skill like any other, it is easy to get something done but difficult to do it well. Everyone can learn to code, there are so many resources teaching anyone to write their mobile app or website. So what defines a good developer from the crowed? Here's what I believe.

Design focused (UX)

In meeting, developers often expects people to understand the technical side of things. For example, it is so frustrating when the person you are talking to doesn't get what seems so elementary to you. I have been in similar situation, I get that. I think everyone should try to learn technical knowledge, in a scrum team that includes the Analysts and Product Owner.

On the other hand, developer should not take anything granted either. Learn how to design a beautiful product, that has been thought from a UX point of view too. Sure technical side is important, but a good developer will suggest cool ways for the user to interact with the product since they know what is possible to achieve. I have been in teams where the art team will create all UI design and hand them over to the development team. To me, this sucks for the developers. Design is the soul of the product, it is where creativity shines. A good developer should be part of the design process, not copying design someone else made correct to the pixel.

Be responsible

Great powers comes with great responsibilities

Don't abuse your power as a developer. No body likes reading crappy code and fixing bugs. If you are not sure, then check before committing your work to Github (or other Source Code Version Control tools).

Don't choose the easy way out. People often write awful code due to time constraints. This leads to terrible and horrible code. Either rethink your logic and change the solution, or split up the logic into different methods/functions. So it is easier to understand and also makes it more reusable. Honestly, think about the poor guy picking up this code.

No limitation

Digital and Technology are changing all the time, try not focus on a single field. It is bad enough when developer separate themselves from designers, but splitting again between Mobile, Front-end web, Back-end web and Dev-ops? This goes back to work style, but for me I think work should be split into features so a developer will have to design the database schema, create relevant APIs, design the UI then link everything together.

Knowledgeable

Spend time learning about a language. Getting familiar with what is the best way to implement something in the language you are using. Which is the best loop to use? How to effectively manage memory? What are the constrains of the language? A good developer will know how to get most of the language, because you are using the language for a reason right? Otherwise, it probably makes more sense to use another language.

Understand possible solutions and how to choose the best one. Developer often have 2 or more ways to implement something, they either uses different technology all together or just implementation logic within the code. Learn the different libraries, frameworks, tools or technologies. Know the difference between them, when is one of them is better than the rest, and the drawbacks as well.

Be the go to guy for your team. If people always come up to you asking questions (how to do something) even those unrelated to your work, don't get annoyed instead be happy. That can only mean one thing, that you are a great developer and knows more than a lot of the people you work with. This is awesome, embrace it!

How to implement correct REST API & RESTful Web Services?

Tue, 22 Mar 2016 22:24:00 GMT

It is always easy to assume that you've understood something before someone showed you the correct solution. When I tried to follow someone's tutorial on how to create REST APIs using node.js and express.js, I found out I didn't get REST APIs completely right. I have also found out a few more things on the side, which I will also be sharing in this blog.

What is REST and RESTful?

While REST stands for Representational State Transfer, which is an architectural style for networked hypermedia applications, it is primarily used to build Web services that are lightweight, maintainable, and salable. A service based on REST is called a RESTful service.

So REST is a method for client side to communicating with the server side. The requests are structured in such a way that they often can be seen as black-box APIs, where anyone can make requests to the server with simple HTTP requests. This is how people can create their own clients for Twitter, Github and etc. Because all they need are the APIs released by companies like Twitter.

A more traditional method is to use AJAX call to fetch XML formatted data from server. But XML is replaced by JSON in REST APIs, due to better readability and usability.

The different request methods

GET and POST are the most common ones used, but there are many others which are not fully and correctly utilized. I have see project using GET when they should be using POST, or using POST when they should be using DELETE.

The HTTP verbs comprise a major portion of our “uniform interface” constraint and provide us the action counterpart to the noun-based resource. The primary or most-commonly-used HTTP verbs (or methods, as they are properly called) are POST, GET, PUT, PATCH, and DELETE. These correspond to create, read, update, and delete (or CRUD) operations, respectively. There are a number of other verbs, too, but are utilized less frequently. Of those less-frequent methods, OPTIONS and HEAD are used more often than others.

I found the above explanation REST API Tutorial web site. Head over there for more detailed descriptions.

HTTP codes

The other thing I realized from the tutorial is that I really didn't know what I was doing with HTTP codes. There is a whole list which can be found on official W3 site. But following on from HTTP request methods, REST API Tutorial site also listed out the common HTTP codes as well.

200 The request has completed successfully and results is returned.
201 The request has been fulfilled and resulted in a new resource being created.
204 The request has completed successfully, but nothing is returned.
404 The server has not found anything matching the Request.
409 The request could not be completed due to a conflict with the current state of the resource. e.g. resource already exist.
500 Internal server error, which can be used if the user is misusing the APIs.

Consider to use these HTTP codes for the following requests

POST Method

201 404 409

GET Method

200 404

PUT Method

200 204 404

PATCH Method

200 204 404

DELETEMethod

200 404

Hypermedia API

This opened up a brand new door for me in terms of how API should be implemented for the sake of other developers. APIs can be messy, especially if it is not well documented. I have mentioned above that black-box API exists, where the developer using it does not need to know what is going on behind the scenes. "But how do you know how the API works?" you may ask. Well, hypermedia API is mention to help with that.

The example below shows what was returned from a GET request. Many book results has been returned. Normally, the developer using this will need to go read the documentation in order to figure out how to query the book by ID. But in this case, within links attribute of the object, you can find a link. This is just to show a point, but in a production environment, this API should return a list of links, providing key insight into what you can actually do with the products returned. This will help people to understand the API without reading any additional docs, which means the usability is massively improved.

[
  {
    "_id": "56f061f4364dd9282ead6e9e",
    "title": "My New Book",
    "genre": "Fiction",
    "author": "Hao",
    "__v": 0,
    "read": false,
    "links": {
      "self": "http://localhost:8000/api/books/56f061f4364dd9282ead6e9e"
    }
  },
  ...
]

The trend of 'clever' software

Mon, 21 Mar 2016 22:23:00 GMT

There are plenty of companies researching into Artificial Intelligence (AI) and Machine Learning. Products such as Google Now, Siri and Cortana are all examples of combinations these. Apart from these big players, the trend is moving toward "vague" software usages, where the user specify something without being very exact. From these non-deterministic instructions, the software will be able to understand and execute and product the correct results. The following are just two examples I came across recently, both are pretty cool.

numi.io

There is no much I want to say about numi, it is a calculator for mac. Click on the image below to watch the video.

[

Datejs

Datejs is a JavaScript date picker as suggested by its name. Its unique feature is that, rather than a date picker it allows user to type in plain english and based on that it can work out the date. It can understand inputs such as:

today tomorrow July 2008 next friday 3 weeks from now last April 2004.08.07 6/4/2005 8:15 PM 22:30:45 +5years

I think a lot of this is hard coded, since Datejs still can't understand something like last day of march.

How to create models in Mongoose

Fri, 18 Mar 2016 19:34:00 GMT

Started working on a existing project built using MEAN.IO, and Mongoose framework was used as well. I found its documentation a little confusing, so here's what I found out after doing some research.

How to structure your model

Here's a very simple example of how to structure a model.

var mongoose = require('mongoose');
var Schema = mongoose.Schema;

var userSchema = new Schema({
	name: {
	    type: String,
	    required: true,
	    trim: true
	  },
	team: {
	    type: String,
	    required: true,
	    trim: true
	  }
});

userSchema.methods.testFunction = function(params, callback) {
	// do something
}

userSchema.statics.defaultFunction= function(params, callback) {
	// do something
}

module.exports = mongoose.model('User', userSchema);

Difference between static and methods functions in Model

I got the following information from the Mongoose doc.

To put it simply, Method lets you replace the original .find() function with custom find query (equivalent of SELECT in SQL).

user
.myOwnFind()
.where(something)
.limit(20)
.exec(function (err, data) {
  // etc
})

Statics are pretty much the same as methods but allow for defining functions that exist directly on your Model.

userSchema.statics.search = function(){} would overwrite default user.search(){}.

How to make calls your model from controller

Once the model is set up, it can then be called from controllers.

var User= require('../models/User');

Exports.create = function (params) {
	User.testFunc(params, function(error, data) {
		// do something
	});
}

How to make Zsh shell look nice on Mac Terminal

Wed, 16 Mar 2016 22:20:00 GMT

I have already blogged about bash alternatives, Fish shell and Zsh in my previous blog here. I have only started using Zsh recently due to recommendations from a colleague, so thought I'd give it a try. This blog is mainly focusing on how to make Zsh look nice, so no tutorial on installing it.

With Zsh there are two main popular framework, oh-my-zsh and prezto. From my research, it seems that oh-my-zsh is more complete in terms of resources and plugin, but prezto seems to have better performance. I went for performance myself, so prezto.

There are already plenty of themes for prezoto:

agnoster cloud damoekri giddie kylewest minimal nicoulaj paradox peepcode powerline pure skwp smiley sorin steeef adam1 adam2 bart bigfade clint elite2 elite fade fire off oliver pws redhat suse walters zefram

For some visuals refer to this blog where Mike Buss listed all the themes.

I personally quite like the default theme sorin, so I kept it like that. But the colours looked really bad on all the different Terminal profiles on Mac. So I went ahead and add my own colours for it. I ended up using material design colours for this.

(screenshot updated on 20/08/2018, old link was broken)

You can download the Terminal profile here.

Why testing isn't enough

Thu, 10 Mar 2016 22:18:00 GMT

A lot of people talks about the benefits of Test-Driven Development, how great it is when you compare it with non-test driven development. People would give a massive list of advantages of using TDD. For me these are the core reasons:

Writing tests first require you to really consider what you want from the code
Makes writing unit test easy
Act as documentation for the code

There is so much praise around it, to the point where people would believe it solves problem it is not meant to solve. For example, just because you use TDD does not mean your code has no bug. In fact, depending on developer it might not make any difference at all. If a good developer is writing the code, who has very clear head and knows exactly what he is doing. On top of that it takes a lot of brain power to think about all the different scenarios and design the possible inputs/outputs before hand. So it is kind of a moo point if it doesn't add too much value.

The reason I wanted to write about this today is because I read a blog from Github introducing their Scientist 1.0 product. They highlighted a very good point, that "tests aren't enough". The scenario they explained is when you have a legacy system, but wish to replace a component or part of the system to improve performance or whatever the reason is. Then how can you be confident that the new changes won't cause issues and trip over on production? The solution they gave was using this idea that if you want to replace a function in the system, then run the old and new function are the same time side-by-side for a period of time and compare their results. Record any difference in results and then remove the old function when you have enough confidence in the new function.

Here's some sudo code to explain the concept:

function expriment() {
  old = old()
  new = new()
  if (old != new) {
    record data
  }
}
function old() {
  do x
}
function new() {
  do x
}

Here's the original text from the article. Find Original Article here.

Why tests aren't enough

If you want to test correctness, you just write some tests for your new system, right? Well, not quite. Tests are a good place to start verifying the correctness of a new system as you write it, but they aren't enough. For sufficiently complicated systems, it is unlikely you will be able to cover all possible cases in your test suite. If you do, it will be a large, slow test suite that slows down development considerably.

There's also a more concerning reason not to rely solely on tests to verify correctness: Since software has bugs, given enough time and volume, your data will have bugs, too. Data quality is the measure of how buggy your data is. Data quality problems may cause your system to behave in unexpected ways that are not tested or explicitly part of the specifications. Your users will encounter this bad data, and whatever behavior they see will be what they come to rely on and consider correct. If you don't know how your system works when it encounters this sort of bad data, it's unlikely that you will design and test the new system to behave in the way that matches the legacy behavior. So, while test coverage of a rewritten system is hugely important, how the system behaves with production data as the input is the only true test of its correctness compared to the legacy system's behavior.

Mobile and hybrid app testing tool - appium.io

Wed, 09 Mar 2016 21:00:00 GMT

Came across this free app testing tool today, currently not much use for me as I mainly work on web related projects. But hopefully will be working on mobile projects in the future and this will be very useful. Just want to mention that this is a automation test framework from what I have seen so far, so not meant to be used to do manual testing.

The official site is appium.io.

How to use tabs in Vue-strap

Tue, 08 Mar 2016 21:59:00 GMT

Vue-strap is a repository that contains a set of native Vue.js components based on Bootstrap's markup and CSS. This is all very well and good, however, the second I tried to use this I came across issues.

1. I didn't know how to import the components

I was using Webpack, and this is the way to import a component. The other notation did not work for me, probably for good reasons but I never found out why.

import alert from 'vue-strap/src/alert'

2. Requires Bootstrap CSS on the HTML page

This sounds stupid now I managed to work it out, but Vue CLI by default does not load CSS. So the simplest way is to include this in the <head> tag of your HTML page.

<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.6/css/bootstrap.min.css">

3. Tabs simply refuses to work

It took me a while to finally work this out, the documentation is simply rubbish when it came to explaining how Tabs works. The example code (listed below) looks pretty simple.

<tabs>
  <tab header="one">
    ...
  </tab>
  <tab header="two">
    ...
  </tab>
  <tab header="three" disabled>
    ...
  </tab>
</tabs>

import tab from 'vue-strap/src/tab'

If I import the tab component surely this should work, right?

No! This is only half of the story. Import tab is correct, but you know <tabs>? Yep, that needs to be imported as well. The logical thing to call it would be 'Tabs.vue', but that would be way too easy. It is actually called 'Tabset.vue', so you got to also import the following to get tabs working.

import tabs from 'vue-strap/src/tabset'

Not sure why they thought it is a good idea to leave out this pretty important information, especially for someone who is new to Vue. But here you go, hopefully, saved you some time and pain.

Joking aside, this is actually a pretty cool project, you can find it at yuche.github.io.

Regex for getting the host of a website

Wed, 17 Feb 2016 21:56:00 GMT

I ran into this problem where we wanted to strip away the url of different sites to its root url. On top of that I also wanted to remove anything before the domain name, for example for staging.domain.com I only want domain.com from it.

I ended up solving this issue using regular expression:

/[^.]*\.([^.]{2,}|[^.]{2,3}\.[^.]{2})$/

Explaining from the last character of the regex to the first one:

$ is the end of string e.g. 'www.domain.co.uk'<-(here)
Within the () bracket there are two different matches e.g. (a|b) matches to a or b
- [^.]{2,}: matches to anything like ‘com’, ‘uk’, ‘london’ and etc
- [^.]{2,3}\.[^.]{2}: matches to anything like ‘co.uk’ or ‘com.us’ and etc
[^.]*\. this matches against ‘domain.’ or ‘google.’

Some test domain urls are listed below, you can try it out on regex101.com (make sure you click on JavaScript on the right hand side instead of pcre):

www.domain.com
www.domain.co.uk
www.domain.us
www.url2s.com
www.domain.london
www.url2s.travel
www.bbc.com
andigital.com
bbc.co.uk
www.domain.com.us
staging.su.ik.il.domain.org.uk
www.domain.xn----china

Other shell alternatives to Bash

Mon, 18 Jan 2016 20:46:00 GMT

Came across something called Fish Shell today, and after Google search took me to this page on unix-shells I realised how many different variation there are I can choose from. Each having their benefits, so far it seems ZSH has very good customisability and Fish Shell has very good out of the box experience with no configurations.

I personally don’t feel confident enough to switch to something that is not Bash, just because how little I use it. But this could be very handy for someone who knows Bash like back of his hand and wants a bit more horse power.

I ended up switching to ZSH a few weeks after this original post, and after a year I thought it would be a good chance to give an update. The setting up process is not as straight forward as I would like, but after everything has been set up my god it is amazing! It makes the whole terminal comes to life, awesome predictions and etc. I have even spent the time and added Material colour pallets to it. Here's a screenshot of my current set up.

A useful tool when working with Regular Expression

Sun, 03 Jan 2016 21:38:00 GMT

I was working with replacing strings using Regular Expression today and I found this amazing tools which saved me a lot of trouble. It allow custom text string to perform the regular expression on, and you can toggle between 3 different regular expression syntaxes: PHP, JavaScript and Python. There are also useful tips on the right hand side as well as a regex syntax look up table. Overall, a very handy free tool.

You can find it at regex101.com.

Convert PDF to tables

Sun, 03 Jan 2016 20:41:00 GMT

I recently needed to convert PDF documents which contains tables. From what I could find, there are only two good solutions out there.

https://pdftables.com - A web app that could convert a PDF document to CSV file. Simply upload the file and wait for it do its magic. However, this is no longer free for everyone to use. After the free trial ends, a fee is needed to keep using its service.
http://tabula.technology - Tabula is a software which are available on Windows and Mac, it opens as a web application on your browser. The automatic mode is not as accurate as pdftables.com web app, but using manual selection.

How to uninstall from Debian Linux

Tue, 08 Dec 2015 20:35:00 GMT

Linux is still something that's new to me, I know the basics like to install some software I need to run sudo apt-get update and sudo apt-get install in the terminal. But I never really knew how to 'undo' these actions, today when I was trying to fix another bug, I came across the following ways to uninstall the software are installed in Linux terminal.

sudo apt-get remove - Packages installed are removed (Does NOT include configuration files).

sudo apt-get purge - Purge is identical to remove except that packages are removed and purged. Purge meaning that any configuration files are deleted too.

This is going to so useful!

Moltin, a great service to consider for creating E-Commerce

Mon, 07 Dec 2015 20:28:00 GMT

I have not personally used Moltin, however, based on my intraction with it so far it looks like a good service to consider when building an E-commerce platform. It is designed to take stress away from creating E-Commerce sites by letting the programmer focus on the creative and fun parts.

So what service do they actually provide? Well they allow you to configure everything, e.g. add categories to databases via UI on their site. Then later you can access it as an API call, so any data stored can be retrieved using a provided API. Database management and API creation is no longer your concern. Neither is the security and compliances for that matter, free account has 30, 000 API per month which is more than enough to get started with.

Creates beautiful timelines with Labella.js

Thu, 03 Dec 2015 20:12:00 GMT

I came across this library today, and from a brief look it is both elegant and beautiful. I have seen a lot of timeline implementation before, but nothing as simple and nicely designed as this one. It is created and supported by Twitter, based on my experience with Bootstrap CSS framework by Twitter I think this will be an excellent framework as well.

This is not something I personally need right now, but I'd be glad to have in the back of my pocket. Click here to go to Labella's official site.

Time to drop CSS and learn SASS/SCSS

Mon, 30 Nov 2015 20:02:00 GMT

SASS/SCSS is the compiled version of CSS, it allows CSS to be written in a more dynamic and reusable way. The philosophy is make CSS, a very static language, into something that is more similar to a scripting language, then the SCSS file is then compiled back into a normal CSS file to be deployed. y introducing variables and splitting the single CSS file into different SASS/SCSS files (will be combined into one when releasing), this makes the CSS much easier to read and more maintainable. This is especially true when more than one developer is working on the same code base.

I personally prefer SCSS over SASS, to be honest there's not much difference between the two. I like SCSS because of it uses brackets to show the scope instead of just indentation. Curly brackets makes nested code much easier to read, this is very important since that is how to write CSS or SASS/SCSS.

Here is the official site, which will explain everything in more depth with examples.

Awesome packages for atom

Mon, 23 Nov 2015 21:23:00 GMT

I have been using Atom for more than a year, since it was in beta. There are many things I love about it, the most important thing is customizability. It allows anyone to create an Text Editor that is designed from the ground up for themselves. In this blog, I will be listing some of my favourite packages.

Update: Tweaked content on 30th Jan 2017 Update: Fixed some broken image links on 7th Jan 2019

Must Haves

Atom Beautify

The most annoying thing about reading other people's code is that they always seems to be using a different indentation and formatting to you. With this package, you can convert almost any code (supports stupid amount of programming languages and frameworks) to a clean and nice to read format. If there is something you don't like about the default settings then there's option to customise each language and how it converts them.

Files Icon

There is no reason not to install this package, no matter what you use Atom for this will make it easier for you by changing the default file icon to these nice looking icons. This will make files standout from each other, and in general improves the look and feel of the whole editor. Trust me when I say once you start using this, there is no going back. I mean take a look at these nice icons :D

For more advances users, there are also setting to resize, change colours and etc.

Project Manager

I have quite a few projects going on at one point, switching between them is really annoying. Now with this package, you can switch between any project with a short cut.

Sync Settings

Switching between computers means setting up everything, and it is a pain to re-configure IDE/Text Editors. With this package you will be able to backup everything on Github Gist.

It could sync almost everything:

Sync Atom's and package settings
Sync installed packages
Sync user keymaps
Sync user styles
Sync user init script
Sync snippets
Sync user defined text files
Manual backup/restore to a gist

Pigments

Pigments will scan source files in your project directories looking for colors and will build a palette with all of them. Then for each opened file, it will use the palette to evaluate the value of a given color. This works with CSS or even SASS/SCSS, and makes it a lot easier to read other people's code as well.

This package has a lot of settings, the image below shows all the different visual ways to show the colour indicator. I personally prefer the coloured dot.

Minimap

This is a package ported over from Sublime Text I believe. It replaces the scroll bar of the opened document with a shrinked version, so when you won't ever need to scroll blindly anymore. If you are familiar with the page layout, it might even be possible to go to the correct section by simply clicking on it.

If you do decide to use Minimap, here's a few other packages to be used with this one minimap-highlight-selected and minimap-pigments.

Team Tools

Linter

Code linting is amazing, it forces consistency and ensures the whole development team is following the same format. I don't know how many versions of indentation I saw in previous projects before, the answer is too damn many. With linter, you won't be the "guy" that breaks the style everyone agreed on. In my previous work place, we went as far as making passing linter a process in the build process. So for each of the wrong styling added, it will throw and error/warning.

Obviously, this is the base package for language specific linters e.g. eslint for Javascript.

Linter Write Good

If you have heard of or used Grammarly, this is basically a simpler version of that in an Atom package. It will give writing suggestions and makes sure you write in proper English. I don't think the suggestions are as good, but it can help to identify some simple grammatical errors.

Editor Config

Following on from the Linter package, this is also designed to keep the code conssitent in a dev team. Unlike the linter, this will not throw any errors/warnings. Instead it will update the current editor's config to whatever configuration is set. Let's the use the indentation example again, if someone sets the indentation to 4 spaces in the config I will automatically get that setting when I load the project.

Highlight Selected

Double click on a word to highlight it throughout the open file. There's not much else to say really!

Indent Guide Improved

The default indentation on Atom is not very good, this package will show the indentation line depending when you click. See the Gif below to see it in action.

Web Development focused

Auto Close HTML

Note: Used for web development

In a lot of IDEs and TextEditors this would be a built-in feature, but since Atom is designed to be modular. The community has taken up the responsibility to add in these kind of useful features. This package will automatically add closing tags when you complete the opening tag. Sometimes it can be annoying in some scenario and I wish it can be a bit cleverer. But compared to the default behaviour, this is fantastic and makes writing HTML way easier.

Random And Fun

Atom Clock

Sometimes I want to cut all distractions out, and maximizes the Atom window to prevent myself from seeing notifications and etc. Hiding everything can be annoying when I want to check the time. With this package you will see a little clock on the bottom status bar, neat right? This package also allowing many variations of date/time formats and also available in different languages.

What Else? Well, Just Be Creative

The best functionality of Atom is not any of its packages, the ability to combine x number of packages to achieve one desired purpose. Here is a quick example of how I used different packages together to create a new feature.

When I'm working on web development projects, I often finds it annoying how I have to manually navigate to a HTML and double clicking on it to open it in a browser. I wanted an easy to do this when playing around with HTML, CSS and JS on my machine. The first package I installed is Tool Bar, it creates a empty toolbar in Atom. To populate the empty toolbar, I installed another package Flex Tool Bar. This adds the necessary buttons. The final package I needed was Open In Browser, which does what the name suggests. I to add a button to the Flex Tool Bar, I added the following cson into the config file.

{
  type: "button"
  icon: 'chrome'
  callback: 'open-in-browser:open'
  tooltip: 'Open in Chrome'
  iconset: 'icomoon'
}

Some may think this is stupid, why don't they just create a package that does by default? Well, by splitting up the modular package like this, it makes it a lot easier to build on top of other people's work rather than keep reinventing the wheel.

Final Thoughts

I personally have way more packages installed. Since a lot of it is focused on the work I am working on right now and might not apply to a lot of people, so I have decided to leave them out of this post. But there is a page on Atom's website which lists the popular packages and themes, it is worth checking it out every so often and see if there's anything new that's been added. Finally, I hope this has been useful and leave me any package suggestions in the comments below!

Mouse software for gaming on Mac

Sun, 22 Nov 2015 23:48:00 GMT

When using normal Windows mouse on a Mac, there were some weird acceleration applied to the cursor. After some quick research apparently there is this amazing free tool Smoothmouse which can simulate the mouse experience on Windows.

No offence to the Mac lovers, this is way better for gaming. So if you are used to gaming on Windows, then you will love this software.

Display Spreadsheet Tables On Website

Sun, 22 Nov 2015 21:44:00 GMT

Came across this good library to display a large table on web pages. It is called handsontable, it is open source and seems very handy in some situations e.g. creating reports or needs to show large amount of data for some reason.

Is Docker the new Vagrant

Thu, 19 Nov 2015 18:23:00 GMT

I have been a massive Vagrant fan since I was introduced to it in 2013/2014. To me, it is the first time I stepped into the world of DevOps and see the huge difference it made to a team. Before Vagrant, ensuring team member's machines and the server environments software versions is a huge headache. We always ran into issues where code works fine on one person's computer and ends up breaking on another machine. This is terrible news when the whole deployment process is meant to be automated. My Team had no choice but to increase the amount of manual testing done on the website to reduce these kind of issues getting through to production.

With the introduction of Vagrant, we were able to easily manage everyone's software version since everyone build their VM (virtual machine) based on the same configuration. This means software updates can be managed easily, everyone just need to rebuild their local VM using the updated config.

So, what is Docker? Docker is not really an update to Vagrant, but instead of concept of VM. Since most of the time we don't need a full Linux VM to run our software. So Docker allows the ability to run a Docker container that is a stripped Linux OS, supporting its core functionalities but giving the user ability to install everything back if they wish. Docker doesn't quite replace Vagrant, since Vagrant was designed to be used for local development. Whereas Docker is aimed at server level optimisation. Of course, there is nothing stopping you from using Docker to do local development. Since the two products are meant to solve completely different issues, so I don't think they are in direct competition. But over time, the popularity of Docker may impact Vagrant.

Personally, after moving away from PHP backend and more into Node.js I have also moved away from Vagrant. Maybe Docker will be more popular than Vagrant in the future, but for now neither is really my top choice.

International postal address format

Wed, 18 Nov 2015 23:39:00 GMT

I am currently working on a personal project which involves the user entering their address to be searched. I wanted to ensure this application is designed properly, so it can easily be scaled to work with international postal address. However, after some research it is still far from clear what format I should follow.

Many answers online make suggestions based on US format, which is not very helpful for other countries with slightly different formats. I have included the address fields for my project below, I am sure for some countries this might not be 100% correct, but this is the best one I can come up with so far. Just FYI, I'm mainly focusing on UK addresses here.

Address Line 1 (Room 15)
Address Line 2 (Flat A)
Address Line 3 (15 Street)
City (London)
Postcode (XX00 0XX)
County (Middlesex)
Country (United Kingdom)

The different font types

Wed, 11 Nov 2015 23:11:00 GMT

I have been working with different fonts for ages, but never understood the difference. Recently, I have been working on a project where we need to decide a list of fonts to support. After some very basic research, here is what I found out.

There are 3 different type of fonts, each with their unique properties and thus advantages:

Serif fonts - Good for long paragraphs and text heavy pages Sans-Serif fonts - Good for page headings Monospaced fonts - Good for displaying code

This will be useful to know when designing a poster or creating a web page in the future.

Development Technologies Landscape (2015)

Wed, 11 Nov 2015 23:03:00 GMT

For a developer sometimes it can be difficult to know what to focus on in terms of languages, frameworks and libraries. I came across this diagram created by devnetwork. It shows the top technology to focus on in different fields. Pretty useful way to measure yourself against, see what you already know and pick something new to learn in the future.

Project tracking tool - wll.space

Fri, 30 Oct 2015 20:09:00 GMT

Today, I discovered a new tool which is very similar to Jira and Trello. I feel Jira is designed mostly for complex projects but can be a bit overkill for smaller/personal projects. Trello is very simple and straightforward to use, however, it lacks the ability to scale as the project unfolds. From a brief look at wll.space is very similar to Trello in this regard, simple but lacks scalability. Its design is what differentiates from the other two well-known applications.

If you want a more visual looking project tracking tool, I recommend giving wll.space a try.

Set up MEAN.js in Vagrant

Sun, 11 Oct 2015 22:50:00 GMT

If you have used Vagrant already then, you must know how powerful it can be when it comes to sharing local development environments. If you have not used it already, the whole idea is one person from the development team sets up a VM on their local machine then configure it with all the required libraries, settings and correct software versions. This VM can then be packaged up and shared with anyone.

Why is this powerful? Well, you can ensure that everyone on your team has the exactly same development environment no matter which OS they are using. You can control and maintain the exact version of all the software. Rather than the whole team spend weeks trying to resolve issues on individual machines, it is done once and for all.

It all sounds well and good, but it has its disadvantages too. Since it is a VM box, not all the setting up is straight forward. It is a VM after all, and sometimes the commands are little tricky. You are faced with issues which otherwise does not exist on real machines, and this happened to me during the setup process with my team. After many trial and error, I finally found the solution and want to share it here.

Installing MEAN.js

This section is for one person from the team to complete, because everything from here can be shared with the team. There are already excellent sources out there; I recommend this one: http://thejackalofjavascript.com/vagrant-mean-box

I normally have a repository structure like this:

- root_folder
   - Vagrantfile
   - myApp

Issues and how to resolve them

The blog by arvindr21 has everything right, but the only problem I found was when anyone else on my team tries to pull from the repo and run the site they get a lot of errors.

It turns out; it is not as simple as npm install. First of all, you have to run it as an admin user, so you have to do sudo npm install. Even if you run it as an admin user, you will still see errors along the lines of 'sys-link broken'. This is a vagrant issue, to solve this you have to run sudo npm install --no-bin-links. Then you got to install git and update bower. I know all this sounds confusing, so please follow the instructions shown below.

Note: This is for members who are consuming the VM, and not the creator.

Pull the project from Github (or another repository). I am assuming the vagrant file is set up correctly e.g. private_network IP is "11.12.13.14", and port 3000 forward is added. If not then add these two lines in your Vagrant file.

config.vm.network "private_network", ip: "11.12.13.14"
config.vm.network "forwarded_port", guest: 3000, host: 3000

Run vagrant up; vagrant ssh
Then go to the project folder in the VM, by running cd /vagrant/myApp
Now set up dependencies by sudo npm install --no-bin-links. This process could take a few minutes, so take a quick tea break.
Then install git, because you need it before the next step sudo apt-get install git. Type y and then enter to accept the confirmation.
Once git is installed, do bower install
Run grunt and wait for everything to be loaded and go to http://11.12.13.14:3000 to see the site working.

Awesome Text Editor - Atom

Sat, 10 Oct 2015 13:34:00 GMT

Since I started using Sublime Text 2, I have fallen in love with the simplicity design. Compared to other more "robust" integrated development environment (IDE), it was faster to run, more customisable, had a large community and some of the functionalities are simply awesome. Since Sublime Text requires a payment of $70, it is a little pricey for me. After I had enough with pop ups telling me to pay the licence fee, I decided to look for an alternative.

I have tried many IDEs and text editors, they all had problems:

Not available on all OS platforms
Expensive licence/subscription fee
Too bulky, and not as light as Sublime Text
Too limiting, does not have all the key functionalities
Lack the subtle touches of Sublime Text e.g. multi-select
Does not have the massive community behind it to create plugins and improve the product in the long run

There were probably more issues, but those are the main problems which prevented me from using them. Then I found out about Atom, an open source project backed by Github. At the time I first used it, it was slow, buggy and lacking many of the plugins from Sublime Text (hell it wasn't available for Windows). After a year, everything changed! Now I can recommend it to anyone, for me it doesn't have any problem, and I can be confident to say it is now a better Text Editor compared to Sublime Text. So if you haven't tried it already, I think you should spend a few days and play around with it!

Installing linux on GUID Partition Table (GPT) disk

Fri, 09 Oct 2015 21:53:00 GMT

There are many advantages of GPT disks compared to Master Boot Record (MBR). MBR was introduced back in 1983, any technology being that old has a restriction in modern day computing, and MBR is no other. If you are interested, feel free to read up on the comparison between MBR and GPT, but I won't be explaining the differences here. Instead, I want to explain the issues I had with GPT and how to resolve it.

I tried to dual boot my Windows ten desktop computer with a new Linux OS (Debian based). I started off with Elementary OS, but then found some of the bugs annoying and moved to Linux Mint. I had the same issues on both distribution, and I wasted a lot of time trying to fix it. So I would recommend not convert your hard disk to GPT and install Linux on it unless you know what you are doing.

Simplify SSH connectivity with a config file

Thu, 08 Oct 2015 19:40:00 GMT

Managing a large number of servers can be annoying, but a config file will make it much easier. Here I talk about how a config file can make all the difference.

Creating a config file

In order to SSH into a server, the IP address is required and needs to be specified in the terminal command.

$ SSH user@1.2.3.4

The above format is okay if you are only working on a single server, if you have to work with many servers, then managing all the IP address can quickly get out of hand. The beauty of a config file is that it allows you to map an IP address to your domain name. After the mapping, you can just do the following.

$ SSH mydomain.com

If you don't have this file already, then create it first.

touch ~/.ssh/config

Then copy the following template into the file, don't forget to update the relevant values (e.g. domain, IP and user).

Host yourdomain.com  
HostName 1.2.3.4
User root
IdentityFile ~/.ssh/id_rsa