The 2025 curriculum added the objective "Use Helm and Kustomize to install cluster components". The verb is use. Nothing in it asks you to author a chart. What it asks, in practice:

• install a chart someone else wrote, at the version you are told
• change its values, and know what your change did to the values that were already there
• undo a bad upgrade
• adapt one set of plain YAML to two environments without copying it
• see what any of this will produce before it reaches the cluster

This post does all of that on the two-node cluster from part 1.

Two tools, two ideas

They get mentioned together, but they solve the problem from opposite ends.

| | Helm | Kustomize | |

A Service routes by port. It works at layer 4 and knows nothing about HTTP. An Ingress routes by host and path, at layer 7. That lets one entry point send /app1 to one Service and /app2 to another, where you would otherwise need a load balancer per Service.

Gateway API is the successor, and it has its own post next in the roadmap. But Ingress is still in the curriculum, and still in most clusters you will meet. And the lesson it teaches best carries straight over to Gateway API: a routing rule is only data until something reads it.

The one idea to hold

Three objects, three jobs:

| Object | What it is | What it does on its own | |

Every troubleshooting task on the exam assumes you know what a healthy cluster looks like. The fastest way to learn that is to build one by hand, break it, and put it back — which is why this comes before anything else in the roadmap.

A browser sandbox or minikube gets you a working cluster in a minute, and hides exactly the parts worth learning: which machine you are on, what kubeadm writes to disk, and why a pod network can look fine while no packet crosses between nodes. I broke this cluster twice in two days. Both breakages are below, because they taught me more than the build did.

The task

Two fresh Ubuntu 24.04 VMs, cp and worker. Bring up Kubernetes 1.33 with kubeadm and a pod network that works across both nodes. Prove it: a pod running on worker must reach a pod IP on cp.

What you are building

| | | |

This is the smallest task in the roadmap and the one that pays back fastest.

Over a two-hour exam you type kubectl something like 150 times. At roughly two seconds each for the long form, that is five minutes spent on spelling a command name. Completion matters more still: resource names in the exam are long and easy to mistype, and one wrong character means the task is done against the wrong object.

So the goal here is double. Set up your practice environment today — and be able to retype the whole setup from memory, in under 60 seconds, at the start of the exam.

The task

Before reading the first question: make k a working alias for kubectl with tab completion, add a shortcut that turns any kubectl command into a YAML skeleton, add one for deleting a pod without waiting, and make vim safe for editing YAML. Then retype all of it without notes in under a minute.

Check before you configure

The exam environment usually has alias k=kubectl and completion already set up. Do not bet on it. The first thing to type is:

k version

If that says command not found, paste the setup below before you read anything.

Remember too that aliases live in the current shell on the current machine. The moment you ssh to another node, they are gone.

The solution

An alias that completes

source <(kubectl completion bash)
alias k=kubectl
complete -o default -F __start_kubectl k

The complete line is the one people leave out. Without it, kubectl get po<TAB> completes and k get po<TAB> does not — the alias works but you have thrown away the half that matters.

Order matters too. __start_kubectl is a function defined by kubectl completion bash, so the source line has to run before the complete line.

Completion also needs the bash-completion package on the machine. Check for it, and install it if it is missing:

type _init_completion >/dev/null 2>&1 && echo OK || sudo apt install -y bash-completion

Completion is worth more than the alias. It fills in resource names — k get pod ngin<TAB> — and flag names, which is where typos actually happen.

$do — a YAML skeleton from any command

export do='--dry-run=client -o yaml'

Use it to generate manifests instead of writing them:

k run nginx --image=nginx $do > pod.yaml
k create deploy web --image=nginx --replicas=3 $do > deploy.yaml
k create svc clusterip web --tcp=80:80 $do > svc.yaml

This is how you handle any task along the lines of "create a pod with …" where kubectl run has no flag for what is asked: generate the skeleton, edit it, then k apply -f.

$now — delete without waiting

export now='--force --grace-period=0'

k delete pod nginx $now

A normal pod deletion waits out terminationGracePeriodSeconds, which defaults to 30 seconds. In the exam you delete and recreate things constantly — thirty seconds, ten times over, is five minutes of watching a terminal.

vim that does not break YAML

YAML refuses tabs, and vim uses them by default. Paste a block into a manifest and the indentation is wrong in a way that takes longer to fix than to retype.

cat > ~/.vimrc <<'VIM'
set expandtab
set tabstop=2
set shiftwidth=2
set number
VIM

Three keystrokes worth knowing by heart when editing long YAML:

• :set paste — turns off auto-indent before you paste, so each line stops drifting further right
• yyp — duplicate the current line, for adding another container or env entry
• dd and u — delete a line, undo

Two more aliases worth having

| Add | Why | |

Operators came into the CKA with the 2025 curriculum revision. The objective reads "Understand CRDs, install and configure operators" — and nothing in that line asks you to write a controller or design a CRD. What it does ask for, taken at its word:

• install an operator someone else wrote
• find out what new resource types it added
• write a custom resource for a schema you have never seen, without a template
• work out why a custom resource you created is not doing anything

This post does all four on the two-node cluster from part 1.

The one idea to hold

An operator is a CRD plus a controller. Everything else in this post follows from which half you are looking at.

| Piece | What it is | What it does | |

Lập cho tôi kế hoạch ra mắt khóa học trực tuyến. — Make me a launch plan for an online course.

It returns a plan. Week one, week two, week three, on to week eight, then a post-launch section. Nothing in it is wrong, and none of it could be acted on tomorrow morning, because nothing in it knows anything about you.

What follows is the same prompt, seven times, each version adding one block to the last. All eight prompts are here in full — the seven intermediate ones and the master prompt they add up to.

The run itself is on one model — an Anthropic one; the name in the auto-captions is too garbled to reproduce. The claim made alongside it is that the seven upgrades port to any of them, ChatGPT and Gemini included. That is plausible, because none of the seven depends on a feature: they are all things you write in the prompt box. It is still a claim, though. Nothing here was run twice on two models to check it.

On Google's launch page, Gemini 3.7 Flash sits at or near the top of every category that matters for agent work: professional code quality, long-horizon software engineering, web development, PDF handling, enterprise automation. Occasionally a competitor edges it out. Mostly it leads.

On arena.ai, an independent leaderboard built from real sessions rather than a vendor's own evaluation, the same model ranks like this:

| Category | Gemini 3.7 Flash | |

And yet, at the level of the whole delivery pipeline, throughput has barely moved.

That gap is the entire subject of Session 1. Not "can AI write code" — that question is settled — but a harder one: if AI already accelerates nearly every individual activity, why isn't the overall process meaningfully faster?

What AI is actually good at

Before deciding where AI belongs in a process, the session grounds things in what AI contributes across any phase. Five capability groups:

| Capability | What it looks like in practice | |

So why do projects still rework their requirements, over and over?

Session 2 answers that, and the answer reframes how you use the tool: the highest-value thing AI does in requirement engineering is not answering — it is asking.

Part 1 — Requirement engineering with AI

Make it ask, not answer

The course states the reframe plainly: AI that asks the right question is more useful than AI that answers immediately.

The reason connects straight back to Session 1. Most missing requirements come from missing context. If you ask for user stories before the context is complete, you get fluent, confident, well-formatted stories built on unstated assumptions — and every one of those assumptions becomes rework later.

There is a hard boundary here that the session keeps returning to:

AI can propose business rules. It cannot confirm them. Business rules and the final decision belong to a person.

The six moves

Requirement work with AI breaks into six activities, each producing an artifact that feeds the next:

1. Elicitation — AI helps mine the existing material for what is actually there
2. Clarification — AI generates the questions that expose what is not there
3. User stories — AI turns clarified requirements into structured stories; a human evaluates
4. Acceptance criteria — so that AI, developer and tester share a definition of done
5. Review — AI re-reads the whole package for gaps, ambiguity and contradiction
6. Prioritisation — AI proposes, a person decides scope

AI is only strong when the specification is good enough.

Sessions 1 and 2 were an argument for building context. Session 3 is where you find out whether that argument paid. The AI-Ready Specification Package assembled on Day 2 stops being a deliverable and becomes an input — the only input — for writing code, writing tests, and deciding whether any of it ships.

Where this sits

That is the failure Session 4 is about. The team moved quickly and left no trail to come back along.

Context debt

The course gives the condition a name:

Context debt is the state where technical and business knowledge is no longer held in engineering artifacts, and survives only in people's memory or in chat history with an AI.

Technical debt you can see in the code. Context debt is invisible until someone asks a question and nobody can answer it.

Three questions that expose it:

"Why is this premium?" — "The PO said so back in March." — "Is it documented?" — "No." When business logic lives only in the heads of people who were there, or inside the code, the team is vibe coding whether or not it thinks of itself that way.

"What does verified mean?" The rule is Verified = OCR AND valid ID AND face match AND blacklist check. The unit tests check the implementation, not that rule. Behaviour tests protect the business; implementation tests only protect the current shape of the code.

"The PO changed the rule two months in." Originally: disburse only after the customer is verified. Now: VIP customers get disbursed first and complete verification within 24 hours. With good traceability you know exactly which use cases, criteria, tests and modules that touches. Without it, you go looking.

The failure taxonomy

Failures get sorted into groups, because each group needs a different fix.

Context loss

| Failure | What it looks like | |

AI helps us execute faster. Humans help us decide better.

And the first content slide sets up why the previous four days were necessary:

The final challenge is not using AI better. It is designing an engineering system that AI can take part in safely.

AI-assisted is a loop; AI-driven is a system

The distinction the session opens with is sharper than the usual buzzword pair.

AI-assisted activities are a loop one person runs: prompt → generate → review → test → fix, and back around. It works. It just does not accumulate anything.

An AI-driven engineering system is defined by five properties instead of five verbs:

| Property | What it means | |

Auto-Configuration: Automatically configures your Spring application based on the dependencies you have added. Embedded Servers: Comes with embedded Tomcat, Jetty, or Undertow servers, eliminating the need for deploying WAR files. Production-Ready Features: Provides metrics, health checks, and externalized configuration to help in managing the application in production environments. Spring CLI: A command-line tool that helps in quickly developing Spring applications using Groovy scripts. Spring Boot is widely used for microservices architecture due to its simplicity and efficiency in setting up and managing independent services.

1. Introduction

Many applications have a front end and a back end. The front end is the user interface—the part that users interact with. The back end runs behind the scenes on a server and is responsible for overall application functionality.

Whenever you submit form data, upload files, access databases, or run other complex operations, these commands are executed on the back-end and results are posted back to the front end.

In essence, back-end development is about building the core of the application. Backend engineers use various frameworks, open-source programming languages, libraries, and APIs to accomplish their tasks.

2. What is a Backend Developer?

A backend developer is a type of software developer who focuses on the server-side of web applications. They are responsible for creating, managing, and maintaining the technology that powers the backend of a website or web application. This includes server logic, databases, and APIs (Application Programming Interfaces) that connect the frontend (what users interact with) to the backend (the server and database).

3. Popular Technologies and Languages for Backend Development

Backend development involves a variety of technologies and languages. Here are some of the most popular ones:

Programming Languages:

1. JavaScript (Node.js):

   • Node.js: Allows for server-side scripting using JavaScript, making it possible to use a single programming language for both frontend and backend.

2. Python:

   • Django: A high-level framework that encourages rapid development and clean, pragmatic design.

   • Flask: A lightweight framework for building web applications with Python.

3. Java:

   • Spring: A comprehensive framework for building enterprise-grade applications.

4. Ruby:

   • Ruby on Rails: A robust framework that makes it easy to build database-backed web applications.

5. PHP:

   • Laravel: A web application framework with expressive, elegant syntax.

   • Symfony: A set of reusable PHP components and a PHP framework for web projects.

6. C#:

   • ASP.NET: A framework for building web applications and services with .NET and C#.

7. Go:

   • Known for its simplicity and performance, especially for concurrent applications.

8. Rust:

   • Gaining popularity for its performance and safety features.

Database:

1. SQL Databases:

   • MySQL: An open-source relational database management system.

   • PostgreSQL: A powerful, open-source object-relational database system.

   • SQLite: A C-language library that implements a small, fast, self-contained SQL database engine.

2. NoSQL Databases:

   • MongoDB: A document-oriented database that stores data in JSON-like documents.

   • Redis: An in-memory data structure store used as a database, cache, and message broker.

   • Cassandra: A distributed NoSQL database designed to handle large amounts of data across many commodity servers.

Web Servers:

1. Nginx:

   • A high-performance web server, reverse proxy, and load balancer.

2. Apache:

   • An open-source web server that provides a secure, efficient, and extensible server.

API Technologies:

1. RESTful APIs:

   • REST (Representational State Transfer) is an architectural style for designing networked applications.

2. GraphQL:

   • A query language for your API, and a server-side runtime for executing queries by using a type system you define for your data.

DevOps and Containerization:

1. Docker:

   • A platform for developing, shipping, and running applications in containers.

2. Kubernetes:

   • An open-source system for automating the deployment, scaling, and management of containerized applications.

3. CI/CD Tools:

   • Jenkins, CircleCI, Travis CI: Tools for continuous integration and continuous deployment.

Version Control:

1. Git:

   • A distributed version control system for tracking changes in source code during software development.

2. GitHub, GitLab, Bitbucket:

   • Platforms for hosting and managing Git repositories.

These technologies and languages form the backbone of backend development, providing the tools needed to build and maintain robust, scalable, and efficient web applications.

4. The Role of a Backend Developer

Backend developers are essential to the development of web applications, focusing on the server-side logic, databases, and integration of various services. Here’s a detailed look at their role:

Key Responsibilities:

1. Server-Side Development:

   • Logic Implementation: Develop and maintain the core logic that powers web applications

   • Performance Optimization: Ensure the server responds quickly and efficiently to requests.

   • Scalability: Design systems that can handle increased load by optimizing code and database queries.

2. Database Management:

   • Database Design: Create and manage databases that store and organize data efficiently.

   • Data Integrity: Implement measures to ensure data is accurate and consistent across the database.

   • Backup and Recovery: Develop strategies for data backup and restoration to prevent data loss.

3. API Development and Integration:

   • API Design: Create APIs to allow the frontend to communicate with the backend and other services.

   • Third-Party Integration: Integrate external services and APIs to enhance functionality.

4. Security and Data Protection:

   • Authentication and Authorization: Implement robust authentication and authorization mechanisms to protect user data.

   • Data Encryption: Ensure data is encrypted during transmission and storage to prevent unauthorized access.

5. Collaboration and Communication:

   • Team Collaboration: Work closely with frontend developers, designers, and project managers to align on project goals.

   • Documentation: Write clear documentation for code, APIs, and system architecture to facilitate collaboration and future maintenance.

6. Testing and Debugging:

   • Unit Testing: Write unit tests to ensure individual components function correctly.

   • Integration Testing: Test the interaction between different parts of the application.

   • Debugging: Identify and fix bugs that arise during development and production.

7. Version Control and Deployment:

   • Version Control Systems: Use tools like Git to manage code changes and collaborate with other developers.

   • Continuous Integration/Continuous Deployment (CI/CD): Set up CI/CD pipelines to automate testing and deployment processes.

Skills Required:

1. Programming Languages:

   • Proficiency in backend languages such as JavaScript (Node.js), Python, Java, Ruby, PHP, C#, and Go.

2. Database Management:

   • Knowledge of SQL and NoSQL databases, including MySQL, PostgreSQL, MongoDB, and Redis.

3. API Development:

   • Experience with RESTful and GraphQL APIs.

4. Server Management:

   • Understanding of server architecture and experience with tools like Docker and Kubernetes.

5. Security:

   • Knowledge of security best practices, including encryption, authentication, and authorization.

6. Problem-Solving:

   • Strong analytical skills to identify and resolve issues efficiently.

7. Version Control:

   • Proficiency with Git and understanding of version control principles.

Tools and Technologies:

1. Frameworks and Libraries:

   • Node.js for JavaScript, Django and Flask for Python, Spring for Java, Ruby on Rails for Ruby, Laravel and Symfony for PHP, and ASP.NET for C#.

2. Databases:

   • SQL databases like MySQL, PostgreSQL, SQLite, and NoSQL databases like MongoDB, Redis, Cassandra.

3. Web Servers:

   • Nginx and Apache.

4. DevOps Tools:

   • Docker for containerization, Kubernetes for orchestration, and CI/CD tools like Jenkins, CircleCI, and Travis CI.

5. Version Control:

   • Git and platforms like GitHub, GitLab, and Bitbucket.

Backend developers are vital to the creation and maintenance of functional, secure, and scalable web applications. Their role encompasses a wide range of responsibilities, from server-side logic and database management to API development and security implementation. Mastery of various programming languages, tools, and technologies is essential for backend developers to succeed in their roles.

5. Essential Skills for a Backend Developer

Backend development requires a diverse set of skills to handle the complexities of server-side logic, database management, and system integrations. Here are the essential skills every backend developer should possess:

1. Programming Languages:

   • Java: A robust, object-oriented language often used in enterprise-level applications with the Spring framework.

   • Go: Known for its performance and efficiency in concurrent applications.

   • Python: Widely used for its simplicity and readability, with popular frameworks like Django and Flask.

2. Database Management:

   • SQL Databases: Proficiency in relational databases like MySQL, PostgreSQL, and SQLite.

   • NoSQL Databases: Experience with document-oriented databases like MongoDB, in-memory data stores like Redis, and distributed databases like Cassandra.

   • Database Design: Skills in designing efficient database schemas and understanding normalization and denormalization.

   • Query Optimization: Ability to write efficient queries and understand indexing, query planning, and performance tuning.

3. API Development and Integration:

   • RESTful APIs: Understanding of REST principles to build scalable and maintainable APIs.

   • GraphQL: Knowledge of GraphQL for more flexible and efficient data querying.

   • API Security: Implementation of authentication and authorization methods (e.g., OAuth, JWT).

4. Server Management:

   • Server Architecture: Understanding of server setup, configuration, and management.

   • Web Servers: Experience with Nginx and Apache for serving web applications.

   • Containerization: Proficiency with Docker for containerizing applications.

   • Orchestration: Knowledge of Kubernetes for managing containerized applications.

5. Security:

   • Data Protection: Implementing encryption for data at rest and in transit.

   • Secure Coding Practices: Understanding common security vulnerabilities (e.g., SQL injection, XSS) and how to prevent them.

   • Authentication and Authorization: Implementing secure user authentication and role-based access control.

6. Version Control:

   • Git: Proficiency with Git for version control and collaboration.

   • Platforms: Experience with GitHub, GitLab, or Bitbucket for code repository management.

7. Performance Optimization:

   • Caching Strategies: Using caching mechanisms (e.g., Redis, Memcached) to improve performance.

   • Load Balancing: Implementing load balancers to distribute traffic across multiple servers.

   • Scalability: Designing applications that can scale horizontally and vertically.

8. Testing and Debugging:

   • Unit Testing: Writing unit tests to verify individual components.

   • Integration Testing: Testing the interaction between different parts of the application.

   • Debugging Tools: Proficiency with debugging tools and techniques to identify and fix issues.

9. DevOps and CI/CD:

   • CI/CD Pipelines: Setting up continuous integration and continuous deployment pipelines using tools like Jenkins, CircleCI, or Travis CI.

   • Infrastructure as Code: Using tools like Terraform or Ansible for infrastructure management.

10. Soft Skills:

    • Problem-Solving: Strong analytical skills to troubleshoot and resolve issues.

    • Communication: Effective communication skills to collaborate with frontend developers, designers, and stakeholders.

    • Time Management: Ability to prioritize tasks and manage time effectively.

Backend development requires a blend of technical and soft skills to manage the complexities of server-side operations, database interactions, and application security. Mastery of these essential skills enables backend developers to build robust, scalable, and efficient web applications.

6. Benefits and Challenges of Being a Backend Developer

Benefits of Being a Backend Developer

1. High Demand and Job Security:

   • Strong Job Market: The demand for skilled backend developers is consistently high, providing numerous job opportunities.

   • Job Stability: Given the essential role backend developers play in web and software development, they enjoy a level of job security.

2. Competitive Salaries:

   • Attractive Compensation: Backend developers often receive competitive salaries, reflecting their specialized skills and importance in the tech industry.

3. Opportunities for Advancement:

   • Career Growth: There are clear paths for career advancement, from senior backend developer roles to technical lead or engineering manager positions.

   • Skill Development: Continuous learning and skill development can lead to further opportunities in various tech domains.

Challenges of Being a Backend Developer

1. Complex Problem-Solving:

   • Technical Complexity: Backend development involves complex problem-solving and requires a deep understanding of various technologies and systems.

   • Debugging Issues: Identifying and fixing bugs, especially in large codebases, can be challenging and time-consuming.

2. High Responsibility:

   • System Reliability: Backend developers are responsible for the reliability and performance of applications, which can be a significant pressure.

   • Security Concerns: Ensuring data security and protecting against breaches is a critical and ongoing challenge.

3. Keeping Up with Technology:

   • Rapid Technological Changes: The fast pace of technological advancements requires continuous learning and adaptation.

   • Learning Curve: Mastering new tools, languages, and frameworks can be demanding

4. Workload Management:

   • Deadlines: Managing tight deadlines and delivering high-quality work under pressure is a common challenge.

   • Task Prioritization: Balancing multiple tasks and projects requires excellent time management and prioritization skills.

7. Career Opportunities and Development Path

Backend development offers a wide range of career opportunities and a clear development path for those interested in progressing within the field. Here’s an overview of the potential career trajectory and the opportunities available:

Entry-Level Positions

1. Junior Backend Developer:

   • Responsibilities: Assist in developing and maintaining backend components, write simple scripts, and handle minor bug fixes.

   • Skills Needed: Basic knowledge of programming languages (e.g., JavaScript, Python, Java), understanding of databases, and familiarity with version control systems like Git.

Mid-Level Positions

2. Backend Developer:

   • Responsibilities: Develop server-side logic, manage databases, create APIs, ensure application performance, and implement security measures.

   • Skills Needed: Proficiency in backend programming languages and frameworks, experience with SQL and NoSQL databases, knowledge of RESTful and GraphQL APIs, and understanding of server management.

3. DevOps Engineer:

   • Responsibilities: Oversee the deployment, automation, and management of applications, work with CI/CD pipelines, and ensure infrastructure scalability and reliability.

   • Skills Needed: Knowledge of cloud platforms (AWS, Azure, GCP), containerization tools (Docker, Kubernetes), and automation scripts (Ansible, Terraform).

Senior-Level Positions

4. Senior Backend Developer:

   • Responsibilities: Lead backend development projects, mentor junior developers, design complex systems, optimize application performance, and ensure robust security practices.

   • Skills Needed: Advanced knowledge of multiple backend languages and frameworks, extensive experience with databases, strong problem-solving skills, and leadership abilities.

5. Technical Lead / Team Lead:

   • Responsibilities: Oversee the technical direction of projects, coordinate between different development teams, ensure code quality, and guide the team in best practices.

   • Skills Needed: Strong technical expertise, excellent leadership and communication skills, ability to manage multiple projects, and experience in project management.

Development Path

1. Education and Learning:

   • Formal Education: Obtain a degree in Computer Science, Software Engineering, or a related field.

   • Online Courses and Bootcamps: Participate in coding bootcamps and online courses (e.g., Coursera, Udemy, edX) to gain practical skills.

2. Skill Development:

   • Programming Languages: Master backend languages such as JavaScript (Node.js), Python, Java, Ruby, PHP, C#, and Go.

   • Frameworks and Tools: Gain proficiency in frameworks (e.g., Django, Spring, Ruby on Rails) and tools (e.g., Docker, Kubernetes).

3. Practical Experience:

   • Projects and Internships: Work on real-world projects, contribute to open-source projects, and pursue internships to gain hands-on experience.

   • Build a Portfolio: Create a portfolio showcasing your projects, code samples, and contributions to open-source communities.

4. Certifications:

   • Certifications: Obtain relevant certifications (e.g., AWS Certified Solutions Architect, Microsoft Certified: Azure Developer Associate) to validate your skills and knowledge.

5. Networking and Community Involvement:

   • Networking: Attend industry conferences, join professional networks (e.g., LinkedIn), and participate in local tech meetups.

   • Community: Engage with online communities (e.g., Stack Overflow, GitHub), contribute to discussions, and share your knowledge.

8. Conclusion

Society is constantly changing and developing, especially with the rapid advancements in technology. Backend developers must continually enhance their professional skills and stay updated with the latest technology trends. Additionally, participating in programming communities can help you hone your skills and learn from others in the field.

Why Spring Boot if There is Already Spring!

Spring is widely used for creating scalable applications. In the case of web applications, Spring provides Spring MVC which is widely used to create scalable web applications. But the major drawback of spring projects is that configuration is really time-taking and can be a bit tedious for the new developers. To make the application production-ready takes some time in spring. And the solution for this is Spring Boot.

Spring Boot is created on the top of the spring and includes all the features of spring. And is becoming a favorite of developers these days because of its rapid production-ready environment which allows the developers to directly concentrate on the logic instead of struggling with the configuration and setup.

What is Spring Boot?

Spring Boot is an open-source Java framework designed to simplify the development of Java-based web applications. It is built on top of the popular Spring framework and provides a convention-over-configuration approach, which allows developers to quickly create production-ready applications with little effort.

The primary goal of Spring Boot is to remove boilerplate code and complex configurations that developers often encounter when building web applications with the Spring framework. It comes with a set of opinionated defaults and auto-configuration capabilities, which reduces the time and effort needed to set up a Spring web application.

It is very easy to create stand-alone, production-grade Spring-based Applications that you can “just run“

Features of Spring Boot

1. Auto-Configuration: Automatically configures your Spring application based on the dependencies you have added. For example, if H2 database is in the classpath, Spring Boot automatically sets up an in-memory database.

2. Standalone: Spring Boot applications can run independently without needing an external application server. They can be run from the command line using java -jar.

3. Embedded Servers: Spring Boot comes with embedded servers such as Tomcat, Jetty, and Undertow, which makes it easy to develop and test web applications.

4. Opinionated Defaults: It provides sensible defaults out-of-the-box, reducing the need for boilerplate configuration.

5. Production-Ready Features: Includes a range of non-functional features, such as metrics, health checks, and externalized configuration, essential for running applications in production.

6. Spring CLI: Command-line tool that helps to bootstrap a new project with Spring Boot.

7. Starter Dependencies: Spring Boot provides a wide range of "starter" dependencies. These are pre-configured dependency bundles. These starters simplify dependency management by providing a set of dependencies for specific functionalities such as data access, security, testing, etc.

Core Concepts of Spring Boot

1. Spring Boot Starters: Starters are a set of convenient dependency descriptors you can include in your application. For example, spring-boot-starter-web includes everything to start a web application.

2. Spring Boot AutoConfigurator: Auto-Configuration is a mechanism that automatically configures Spring applications based on the dependencies in the classpath.

3. Spring Boot Actuator: Provides production-ready features such as monitoring, metrics, and health checks.

Why You Should Learn Spring Boot

1. Simplified Development

   • Auto-Configuration: Automatically configures Spring applications based on the dependencies present in the classpath, reducing the need for extensive XML configuration.

   • Starter POMs: Provides a variety of starter dependencies that simplify adding necessary libraries and frameworks.

2. Enhanced Productivity

   • Reduced Boilerplate Code: Minimizes the amount of boilerplate code developers need to write, enabling faster development cycles.

   • Embedded Servers: Built-in support for embedded servers like Tomcat, Jetty, and Undertow means you can run your applications directly without needing an external server setup.

3. Microservices Architecture

   • Modular Development: Ideal for building microservices, which are small, modular services that can be developed, deployed, and scaled independently.

   • Spring Cloud Integration: Seamlessly integrates with Spring Cloud for building robust, scalable microservices architectures.

4. Production-Ready Features

   • Spring Boot Actuator: Provides out-of-the-box monitoring and management features, including health checks, metrics, and environment information.

   • Externalized Configuration: Supports externalized configuration to easily manage different environments (development, staging, production) using properties or YAML files.

5. Strong Community Support

   • Extensive Documentation: Well-documented and supported by a vibrant community, making it easier to find resources, tutorials, and troubleshooting tips.

   • Ecosystem Integration: Part of the larger Spring ecosystem, ensuring compatibility and integration with other Spring projects and third-party libraries.

6. Learning Curve

   • Ease of Learning: Compared to other Java frameworks, Spring Boot has a gentle learning curve due to its opinionated defaults and extensive documentation.

   • Educational Resources: A wealth of learning resources, including books, online courses, and community forums, are available to help you get started.

7. Career Advancement

   • Industry Demand: Knowledge of Spring Boot is highly valued in the job market, especially for roles in backend development and enterprise application development.

   • Versatile Applications: Enables you to develop a wide range of applications, from simple web applications to complex enterprise solutions.

8. Modern Development Practices

• RESTful APIs: Simplifies the development of RESTful web services, a common requirement in modern web applications.

• Reactive Programming: Supports reactive programming with Spring WebFlux, allowing you to build responsive and resilient applications.

9. DevOps and CI/CD Integration

   • Containerization: Easily containerize Spring Boot applications using Docker, facilitating smoother deployments and scaling.

   • CI/CD Support: Compatible with continuous integration and continuous deployment (CI/CD) pipelines, streamlining the development workflow.

10. Scalability and Performance

• Efficient Resource Management: Designed to efficiently manage resources, ensuring that applications can scale seamlessly with increasing load.

• Optimized for Performance: Fine-tuned for performance, making it suitable for high-traffic applications.

Best Way to Master Spring Boot – A Roadmap to Learn

Spring Boot uses the hierarchical architecture where each of the layers is stacked on top of each other and communicate is also in the same fashion.

Spring Boot Architecture

Now let’s explore the main layers and architecture of Spring Boot to understand the inner working of the framework. The layers a divided into 4 main categories:

• Presentation Layer

• Business Layer

• Persistence Layer

• Database Layer

Nevertheless, soon you will discover that the business layer is the most important layer for a backend engineer as it is where all the logic of your application is written. Now let's explore each of these layers and learn what function they provide.

Presentation Layer

The presentation layer is what the user sees from the outside or what is exposed to the outside world. It can be a JSON structure that is made up of viewpoints. It can be split into the front layer which is what is seen from the outside and a back layer which is where the structure is built from viewpoints. The back layer also interprets JSON and handles authentication and HTTP requests before entering into the business layer.

Business Layer

The business layer houses all the logic that is specific to the application under development. It interacts with both the presentational layer and the persistence to perform its responsibilities. It is responsible for performing validations, performing authorization, and handling business logic and rules and it consists of service classes.

Persistence Layer

The persistence layer is responsible for all the storage logic including databases and any other form of data persistence. It is responsible for containing storage logic, Fetching objects, and translating them into database rows (and vice versa).

This layer is the equivalent of the Repository interface and database queries are written inside this interface. The Persistence layer is the only layer that communicates with the Business layer and the Database layer.

Database Layer

The database layer contains all the databases such as MySQL, MongoDB, Postgres, etc as the name suggests. It is responsible for performing database operations (mainly CRUD operations) and it is the actual database that you decide to use to build your application.

Spring Boot Workflow

There’s no doubt that Spring Boot relies heavily on the Spring framework, which means that it integrates almost all of the features and modules of Spring such as Spring MVC, Spring Core, etc.

Therefore, the Spring Boot framework is structured using the popular MVC (Model, View, Controller) pattern that is slightly modified.

Here is a theoretical example of the workflow of Spring Boot from Request to Response:

• First, the Client makes an HTTP request using GET, PUT, POST, etc.

• The Controller class receives the HTTP request.

• The Controller understands what type of request will process, and then it deals with it.

• If it is needed, it calls the Service class.

• The Service Class is going to handle the business logic. It does this on the data from the database.

• If everything goes well, it returns a response as a JSP page.

Modified MVC Architecture

When you are building a Spring Boot Web API project, typically, you will want your code to be maintainable, modular, and have a separation of concerns. Therefore, the 4 main layers above are further broken down into different smaller components which make up the modified MVC pattern used in Spring Boot.

Below are some of the components that make up the Spring Boot framework and you will encounter these components always as you continue to work with the framework.

• Controllers: Controllers handle incoming HTTP requests and return the appropriate responses. It receives requests from clients, invokes validations and business logic using Services, and returns the response to the client. Controller classes are usually annotated with @RestController or @Controller annotations and also have methods that are annotated with request mapping annotations like @RequestMapping, @GetMapping, @PostMapping, and so on.

• Services: Services house the business logic of your application. This is where you write all the complex business rules, and coordinate data access, and transformation operations. They process data, perform validations, apply business rules, and invoke data access operations through repositories or DAOs(Data Access Objects). The service class is an abstraction layer between the controller and the data access layer.

• Data Access: Data Access interacts with the underlying data storage systems, such as databases or external Web APIs. It carries out CRUD (Create, Read, Update, Delete) operations on data. It consists of repositories or DAOs (Data Access Objects) that provide an abstraction for database operations. Spring Boot provides support for data access and manipulation through libraries like Spring Data JPA.

• Models: Models contain the entities or domain objects that represent the data and the relationships between them. These domain objects are mapped to database tables. Models provide a definition for the structure and behavior of the data used in the application. It may also include DTOs (Data Transfer Objects) and/or View Models for transferring data between layers or to external clients.

• Exception Handling: Exception handling deals with capturing and handling exceptions that occur during the execution of the API. It provides a mechanism to catch exceptions, transform them into meaningful error responses, and return them to the client. Spring Boot offers various exception-handling techniques, such as using @ExceptionHandler annotations or implementing custom exception handlers.

• Configurations: Configurations contain configuration files and classes required for setting up the Spring Boot web API. It contains properties files (application.properties or application.yml) which are used for configuring various aspects of the application, as well as custom configuration classes that define beans, security settings, or other application-specific configurations.

Spring Boot Architecture Patterns

Spring Boot applications often follow certain architectural patterns to achieve modularity, scalability, and maintainability. Here are some common architectural patterns used in Spring Boot development:

MVC (Model-View-Controller):

• Spring Boot follows the MVC pattern for building web applications.

• Model: Represents the application’s data and business logic.

• View: Responsible for rendering the user interface.

• Controller: Handles user requests, interacts with the model, and updates the view.

Microservices Architecture:

• Spring Boot is commonly used in microservices architectures where the application is decomposed into small, independently deployable services.

• Each microservice is responsible for a specific business capability and communicates with other services through APIs.

Restful API Design:

• Spring Boot promotes the design of RESTful APIs for communication between components.

• Controllers expose RESTful endpoints, and data is typically exchanged in JSON format.

Layered Architecture:

• Organising the application into layers like presentation, service, repository, and entity helps maintain a separation of concerns.

• This layered architecture promotes modularity and testability.

Dependency Injection (DI) / Inversion of Control (IoC):

• Spring Boot relies heavily on the principle of dependency injection, where dependencies are injected into components rather than being created by the components themselves.

• This reduces tight coupling and makes the application more maintainable.

Event-Driven Architecture:

• Spring Boot supports event-driven architecture through the use of Spring Events.

• Components can publish and subscribe to events, allowing for loosely coupled communication between different parts of the application.

CQRS (Command Query Responsibility Segregation):

• CQRS separates read and write operations, allowing for independent scaling of the query (read) and command (write) sides.

• Spring Boot can be used to implement CQRS patterns using different components for command and query responsibilities.

Saga Pattern:

• In microservices architectures, the saga pattern is often used to manage distributed transactions.

• Spring Boot applications can implement sagas by coordinating a series of transactions across multiple microservices.

Gateway Pattern:

• API gateways are used to manage and route requests in microservices architectures.

• Spring Cloud provides tools like Spring Cloud Gateway for building API gateways.

These patterns can be combined and adapted based on the specific requirements of the application. The flexibility of Spring Boot allows developers to choose and implement the patterns that best suit their needs.

Architecture of Spring Boot Microservices

The architecture of Spring Boot microservices typically follows principles that align with the characteristics of microservices, such as modularity, independence, and distributed systems. Here’s a conceptual overview of the architecture for a Spring Boot microservices application:

Microservices:

• The application is decomposed into small, independent, and loosely coupled services, each focused on a specific business capability.

• Microservices communicate with each other through well-defined APIs, often exposed as RESTful endpoints.

Service Registry and Discovery:

• Services register themselves with a service registry (e.g., Spring Cloud Netflix Eureka).

• Service discovery allows services to locate and communicate with each other dynamically.

API Gateway:

• An API Gateway (e.g., Spring Cloud Gateway, Netflix Zuul) is used to manage and route external requests to the appropriate microservices.

• It can handle authentication, load balancing, and routing.

Configuration Management:

• Centralised configuration management (e.g., Spring Cloud Config) allows microservices to retrieve configuration properties dynamically.

• Configuration can be stored in a version-controlled repository and updated without redeploying services.

Distributed Data Management:

• Microservices often have their own databases, and the choice of database depends on the specific requirements of each service.

• Eventual consistency and distributed data management strategies may be employed.

Event-Driven Architecture:

• Events and messaging systems (e.g., Spring Cloud Stream, Apache Kafka) are used for communication between microservices.

• Event-driven architecture supports loose coupling and scalability.

Polyglot Architecture:

• Each microservice can be implemented in a different programming language or use a different technology stack, known as a polyglot architecture.

• The choice of technology is based on the specific requirements of each service.

Testing and Continuous Integration/Continuous Deployment (CI/CD):

• Microservices are independently testable, allowing for easier unit testing and integration testing.

• CI/CD pipelines enable automated testing and deployment of microservices.

Domain-Driven Design (DDD):

• Microservices are often designed based on domain-driven design principles, emphasising a clear understanding of the business domain and its interactions.

This architecture allows for flexibility, scalability, and easy maintenance of a microservices-based system. Spring Cloud provides a set of tools and libraries to implement many of these patterns and streamline the development of microservices in a Spring Boot environment.

1. General introduction

In this article, we will create a simple Spring Boot API application for managing a club. Through the implementation of this application, we will explore the basic components necessary to build an API application.

2. Set Up Your Development Environment

• Java Development Kit (JDK): Spring Boot requires JDK 8, 11, or later.

• Integrated Development Environment (IDE): IntelliJ IDEA, Eclipse, or VS Code are popular choices.

• Build Tool: Maven or Gradle.

2.1 Install JDK

• Windows/Mac/Linux: Download and install the JDK from the Oracle website or use OpenJDK.

Verify the installation by running:

java -version
javac -version

2.2 Install an IDE

• IntelliJ IDEA: Download and install from JetBrains.

• Eclipse: Download and install from Eclipse.

• VS Code: Download and install from Microsoft.

2.3 Install Maven or Gradle

• Maven: Download and install from Apache Maven.

• Gradle: Download and install from Gradle.

Verify the installation:

mvn -v  # for Maven
gradle -v  # for Gradle

3. Generate a Spring Boot Project with Spring Initializr

Spring Initializr is a web-based tool that helps you quickly bootstrap Spring Boot projects. Follow these steps to quickly generate one:

Open your web browser and go to https://start.spring.io/.

| Project | Value | |

Cloud Services are something that none of us can stay away from. If you are into tech then it is kind of impossible that you did not hear about the Cloud.

And those who are familiar to this industry, AWS EC2 is not something new.

In this article, I will demonstrate step by step, how to deploy a spring boot application to the AWS EC2 instance.

Source Code — Sample application that I am going to use for this can be downloaded from here — #

1. General introduction

In this article, we will deploy a Spring Boot application using Docker on EC2 with the following contents:

• Packaging the Spring Boot application with Docker container
• Creating and deploying a new container on EC2.

2. Set Up Your Development Environment

• AWS Account

• Docker

• Docker HUB Account

• Docker desktop installed on your local machine

3. AWS Setup

3.1 Create VPC

Access to VPC console and select Create VPC

Configuration Setup:

• Resources to create : VPC only

• Name tag - optional: vpc-app-southeast-1-dev-web-app-stack

• IPv4 CIDR block: IPv4 CIDR manual input

• IPv4 CIDR: 10.30.0.0/16

• IPv6 CIDR block: No IPv6 CIDR block

• Tenancy: Default

The VPC was successfully created and has the VPC ID: vpc-0ce2993ccf77969a5

3.2 Create a Subnet within a VPC

Access to VPC console and select tab Subnets -> Create Subnet

The subnets you are seeing are the default of AWS. In AWS, a default subnet is automatically created in each Availability Zone (AZ) within a Virtual Private Cloud (VPC) when the VPC is created. This default subnet has a route table that routes traffic to the default Internet Gateway (IGW), allowing the virtual machines in the subnet to communicate with the Internet.

The main purpose of a default subnet is to provide a convenient way to deploy virtual machines in an AWS region without having to create a subnet each time you want to deploy.

Configuration Setup:

Here we will create 1 public subnets (Note: In the VPC ID section, select the VPC we just created above).

• VPC ID: vpc-0ce2993ccf77969a5(vpc-app-southeast-1-dev-web-app-stack)

• Name: app-southeast-1a-public-dev-web-app-stack

• Availability Zone: ap-southeast-1a

• IPv4 VPC CIDR block: 10.30.0.0/16

• IPv4 subnet CIDR block: 10.30.1.0/24

After creating, we can see that

1 public subnet have been successfully created.

3.3 Create Internet Gateways within a VPC

Access VPC console and select tab Internet Gateways -> Create internet gateway

Configuration Setup:

• Name tag: web-app-stack-gw

The IGW was successfully created and has the IGW ID: igw-0f97e3e6b347f9e1f

Here we can see that the State of the IGW is Detached (the IGW is not attached to any VPC and cannot provide Internet connectivity to any resources in the VPC). Therefore, we will Attach this IGW to the VPC (vpc-app-southeast-1-dev-web-app-stack) that was created earlier.

Select IGW(web-app-stack-gw) created -> Action -> Attach to VPC

Continue by selecting the VPC ID(vpc-0ce2993ccf77969a5) created earlier -> Choose Attach internet gateway

Successfully created and the State is now Attached.

3.4 Create Route Tables within a VPC

Access VPC console and select tab Route tables -> Route route table

The Route Tables you are seeing are the default of AWS. In AWS, when you create a Virtual Private Cloud (VPC) in Amazon Web Services (AWS), AWS will automatically create a default Route Table for that VPC.

Configuration Setup:

• Name optional: web-app-stack-route-table-public

• VPC: vpc-0ce2993ccf77969a5(vpc-app-southeast-1-dev-web-app-stack)

Select Create route table

Route Table was successfully created and has the Route Table ID: rtb-0ef6d58f29c45b44a

Check the Subnet associations tab and see that it is not currently assigned to any Subnet. Next, I will assign the Route Table: web-app-stack-route-table-public to the Public Subnets. Assigning the Route Table to Subnets has the following purposes for routing packets from resources within the Subnets:

1. Internet Access: Public subnets typically contain resources (e.g., web servers, web applications) that you want to be accessible from the Internet.

2. Load Balancer: If you are using an Elastic Load Balancer (ELB) to distribute traffic to servers within the public subnets, you need to assign a Route Table containing routes to the ELB. This allows requests from the Internet to be routed to the ELB and then to the servers.

3. Connecting to AWS Services: Sometimes, you may want resources in the public subnets to access AWS services such as S3 or DynamoDB without going through the Internet.

4. Security and Management: By routing requests from the Internet into public subnets through a Route Table, you can control and manage access to the resources in the subnets. This can be done using Security Groups or Network Access Control Lists (NACLs).

To assign the Route Table: web-app-stack-route-table-public to the Public Subnets:

1. Select the Subnet associations tab.

2. Click on Edit subnet associations.

3. Check the Public subnet options. Specifically, select the Public subnets: app-southeast-1a-public-dev-web-app-stack

4. Click on Save associations.

When successfully assigned, the screen will display the updated Subnet associations tab.

The next step is to assign the IGW ID: igw-0f97e3e6b347f9e1f to the Route Table: web-app-stack-route-table-public to enable access Internet.

When you enter the screen, you'll see a default Destination, which is a special route used to route packets to resources within the same VPC. This route targets the internal IP addresses of the resources within the VPC and is used to enable communication between resources within the VPC without going through the Internet.

Next, select Add Route.

Configuration Setup:

• Destination: 0.0.0.0/0.

• Target: Internet Gateway(igw-0f97e3e6b347f9e1f).

Next, select Save changes. If you want to preview the changes, you can choose Review.

When completed, it will display Status: Status: Active.

3.5 Create Security Group

Access the VPC console and select the Security groups -> Create security group.

The Security Groups you are seeing are the default of AWS: a special security group automatically created for each new VPC (Virtual Private Cloud). This security group can be customized to adjust security rules for the resources within the VPC.

Configuration Setup:

• Security group name: app-web-stack-sg

• Description: Allows SSH access to developers

• VPC: vpc-0ce2993ccf77969a5(vpc-app-southeast-1-dev-web-app-stack)

• Inbound rules: IPv4/All traffic/All/0.0.0.0/0

• Outbound rules: IPv4/All traffic/All/0.0.0.0/0

Select Create security group.

The Security group was successfully created and has the Security group ID: sg-04837c0728cd79e89.

In this section, we have successfully created a virtual network environment (VPC) on AWS, including VPC, Subnets, Internet Gateway, Route Tables, and Security Groups. These are the basic steps to begin deploying an application on AWS. In the next lesson, we will continue to deploy Instances (EC2) to run the application.

3.6 Create EC2 Instances

Access the Amazone EC2 and select Launch instance

Configuration Setup:

• Name : app-web-backend

• Quick Start: Ubuntu

• Amazon Machine Image (AMI): Ubuntu Server 22.04 LTS

• Instance type: t2.micro

• Key pair name - required: app-web-public-keypair

• VPC - required: vpc-0ce2993ccf77969a5(vpc-app-southeast-1-dev-web-app-stack)

• Subnet: app-southeast-1a-public-dev-web-app-stack

• Auto-assign public IP: Enable

• Firewall (security groups): Select existing security group

• Common security groups: app-web-stack-sg(sg-04837c0728cd79e89)

After creating, wait for a while until the Instance state: Running. Note the Public IPv4 address (this will be the IP address you will use to SSH into the instance).

4. Dockerize Spring Boot Application

First, we need to dockerize the application we’ve built, hence we need to do some modifications.

Open src/main/resources/application.yml and add the following modification to the server section as shown below:

server:
  port: ${SERVER_PORT:8018}

spring:
  h2:
    console:
      enabled: true
  datasource:
    url: ${H2_URL:jdbc:h2:mem:premier-league}
    driverClassName: org.h2.Driver
  jpa:
    database-platform: org.hibernate.dialect.H2Dialect
    show-sql: true
    hibernate:
      ddl-auto: create-drop

We’ll be passing the PORT and url via environment variables on the server, especially the port. If we manually enter the port for our application, there is every tendency that there going to be a port conflict. If the SERVER_PORT environment variable is not set, the default value 8018 will be used. By using environment variables in this way, you can maintain a single configuration file that works across different environments (development, staging, production) without needing to hard-code values.

4.1 Define Dockerfile

Next, create a Dockerfile and add the following:

# Define build arguments
ARG MVN_VERSION=3.9.5
ARG JDK_VERSION=21
# Build stage
FROM jelastic/maven:${MVN_VERSION}-openjdk-${JDK_VERSION} AS build
LABEL maintainer="taleedevops.today"
WORKDIR /app

# Copy the application code to the container
COPY . /app

# Build the application
RUN mvn clean install -Dmaven.test.skip=true

# Package stage
FROM openjdk:21-jdk-slim
COPY --from=build /app/target/*.jar /app/app.jar
EXPOSE 8018
ENTRYPOINT ["java", "-jar", "/app/app.jar"]

• ARG MVN_VERSION=3.9.5: Defines a build argument for the Maven version, which can be overridden at build time.

• ARG JDK_VERSION=21: Defines a build argument for the JDK version, which can also be overridden at build time.

• FROM jelastic/maven:${MVN_VERSION}-openjdk-${JDK_VERSION} AS build: Starts a new build stage using a Maven image that includes the specified JDK version. This image will be used to build the application.

• LABEL maintainer="taleedevops.today": Adds a label to the image with the maintainer's information.

• WORKDIR /app: Sets the working directory to /app in the container.

• COPY . /app: Copies the entire content of the current directory (where the Dockerfile is located) into the /app directory in the container.

• RUN mvn clean install -Dmaven.test.skip=true: Runs the Maven build command inside the container to clean and install the project dependencies and compile the project, skipping tests.

• FROM openjdk:21-jdk-slim: Starts a new stage for the final image using a slim OpenJDK 21 base image.

• COPY --from=build /app/target/*.jar /app/app.jar: Copies the JAR file built in the previous stage from the /app/target directory to the /app directory in the new image.\

• EXPOSE 8018: Informs Docker that the container will listen on port 8018 at runtime.

• ENTRYPOINT ["java", "-jar", "/app/app.jar"]: Specifies the command to run when the container starts, which is to execute the JAR file using Java.

Summary:

1. Build Stage:

   • Uses a Maven image with the specified JDK to build the application.

   • Copies the source code into the container.

   • Runs Maven to build the application, skipping tests.

2. Package Stage

   • Uses a slim JDK image for the runtime environment.

   • Copies the built JAR file from the build stage.

   • Exposes port 8018.

   • Defines the entry point to run the application.

4.2 Generate Docker Image

You can create a docker image with the following command:

docker build -t <IMAGE_NAME>:<IMAGE_TAG> -f Dockerfile <PATH_TO_Dockerfile>

Applies to projects:

#Build docker images
docker build -t tunganhle/premier-league:latest .
#Process
=> [internal] load build definition from Dockerfile                                                               0.1s
 => => transferring dockerfile: 530B                                                                               0.0s
 => [internal] load .dockerignore                                                                                  0.1s
 => => transferring context: 2B                                                                                    0.0s
 => [internal] load metadata for docker.io/library/openjdk:21-jdk-slim                                             3.8s
 => [internal] load metadata for docker.io/jelastic/maven:3.9.5-openjdk-21                                         3.6s
 => [auth] library/openjdk:pull token for registry-1.docker.io                                                     0.0s
 => [auth] jelastic/maven:pull token for registry-1.docker.io                                                      0.0s
 => [build 1/4] FROM docker.io/jelastic/maven:3.9.5-openjdk-21@sha256:c43af75dc23700d0687ad4c8773f2ab441af5fb8792  0.0s
 => CACHED [stage-1 1/2] FROM docker.io/library/openjdk:21-jdk-slim@sha256:7072053847a8a05d7f3a14ebc778a90b38c50c  0.0s
 => [internal] load build context                                                                                  0.1s
 => => transferring context: 9.46kB                                                                                0.0s
 => CACHED [build 2/4] WORKDIR /app                                                                                0.0s
 => [build 3/4] COPY . /app                                                                                        0.1s
 => [build 4/4] RUN mvn clean install -Dmaven.test.skip=true                                                     223.7s
 => [stage-1 2/2] COPY --from=build /app/target/*.jar /app/app.jar                                                 0.2s
 => exporting to image                                                                                             0.3s
 => => exporting layers                                                                                            0.3s
 => => writing image sha256:43644c5f7e399cf6fe9a946dc77f936c6eb525db19c52f98f6df4ae98e5c7404                       0.0s
 => => naming to docker.io/tunganhle/premier-league:latest
 #Done

Check the docker image with the command:

docker images

REPOSITORY                                  TAG       IMAGE ID       CREATED         SIZE
tunganhle/premier-league                    latest    43644c5f7e39   2 minutes ago   487MB

4.3 Push Docker Image to Docker HUB

For this step make sure you have a docker hub account.

Push your the image to your account.

docker push tunganhle/premier-league:latest

docker tag your-image:tag tunganhle/premier-league:latest

# Tag the image
docker tag premier-league:1.0 tunganhle/premier-league:latest
# Log in to Docker Hub
docker login
# Push the image
docker push tunganhle/premier-league:latest

Once completed successfully, You will be able to see your image listed under repositories on your docker hub account.

5. Launch Application in AWS EC2

We are done with generating our artifacts in previous and step. Now its time to launch the docker image in the EC2 instance that we created earlier.

• Go to your terminal and run the following command to SSH into the instance.

# Navigate to the key pair and modify the permissions if not done already.
# If the permissions is set to 644 then you will be denied to ssh into the instance.
chmod 600 <key-pair.pem>
# SSH into the instance. Username changes on the basis of the image you are using. 
ssh -i /path/keypair.pem ec2-user@<Instance-Public-IP>

• RUN the following commands to update the instance.

# Install the updates to the instance. Depending on the image being used the update instruction changes. Refer to the documentation of your image provider to get the correct information.
sudo apt update -y
# This command installs Docker, an open-source platform for automating the deployment, scaling, and management of containerized applications.
sudo apt install docker.io -y
# Adding the user to the 'docker' group allows them to run Docker commands without needing to use 'sudo'.
sudo usermod -aG docker $USER
newgrp docker
# Start Docker Service
sudo service docker start

• Check Docker Version

docker --version

• Run docker container to test the application

# Start the application on Port 80
# In you modified application.yml of the downloaded spring boot application and changed the port then replace the source port with the value that is specified in your application.yml file.
docker run -d -p 8018:8018 <your-docker-hub-user-name>/premier-league:latest
docker ps
#Copy the docker container Id
docker logs <containerId>
# You will be able to see that your spring boot application is up and running on port 8018

• Open up a REST client. I will be using POSTMAN but you can also use terminal to CURL the POST request to the server.

• Make sure you use http instead of https.

POST http://<your-instance-ip>:8018/api/club/
#Request Body
{
"name": "Manchester United",
"captain": "Bruno Fernandes",
"manager": "Erik ten Hag",
"stadium": "Old Trafford"
}

• Similarly you can execute the GET request to test if the student was created successfully or no.

GET http://<your-instance-ip>:8018/api/club/

That's it. Your application is up and running on the EC2 instance in AWS.

I hope you really enjoyed the article. Thanks for reading. Feel free to share. 😃

Improved Readability: A standardized coding style enhances the readability of the code. When code looks the same regardless of who wrote it, developers can quickly grasp the structure and logic without being distracted by varying styles.

Ease of Collaboration: When multiple developers work on the same codebase, having a common style guide minimizes conflicts and misunderstandings. It streamlines the review process and facilitates smoother collaboration.

Reduced Cognitive Load: Developers spend less time thinking about how to format their code and more time focusing on solving problems. A common style guide provides clear rules, reducing the mental overhead of making formatting decisions.

Better Tooling Support: Many development tools, including linters and formatters, support Google's Java coding style. Using these tools helps automate code formatting, ensuring adherence to the style guide with minimal manual effort.

Why should we follow Google's Java coding style standards?

Because I desire and aspire to work in an environment like Google, but that seems far-fetched for someone with my modest skills. So, I might as well follow the standards they set. 😀

Google's Java Coding Standards: https://google.github.io/styleguide/javaguide.html

How to Install the Necessary Tools:

1. PLUGIN CHECKSTYLE

• You can find the plugin in: Settings > Plugins > Marketplace –> enter “CheckStyle”

• Then install it and restart the IDE.

After setting up, you need to configure:

2. PLUGIN: GOOGLE JAVA FORMAT

• You can do the same as with the Checkstyle plugin, but this time search for “google-java-format”:

After setting up and restarting the IDE, you need to activate the plugin:

3. Import the intellij-java-google-style.xml into IntelliJ IDEA

Create file fsales-cs-formatter.xml in project

After import to IntelliJ:

How to use

Now, your IntelliJ IDEA is configured to follow Google's Java coding standards, and your code will be automatically formatted accordingly.

You can use the "Reformat Code" option or the shortcut Ctrl+Alt+L to format your code according to the imported Google Java style.

--add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED
--add-exports=jdk.compiler/com.sun.tools.javac.code=ALL-UNNAMED
--add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED
--add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED
--add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED
--add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED

Spring Security acts as a vigilant guardian for your web applications. It oversees who can access your site, verifies user identities, and enforces rules to safeguard your data from unauthorized access.

Authentication And Authorization

1. Authentication in web applications

Authentication is the process of verifying the identity of a user accessing a system or application.

2. How Spring Security handles authentication

Spring Security handles authentication by employing various authentication providers, such as in-memory authentication, JDBC-based authentication, and LDAP authentication. Each provider validates user credentials and establishes the user’s identity within the application.

3. Concept of authorization

Authorization determines what actions a user is allowed to perform within an application. It establishes rules and permissions for accessing protected resources based on the user’s identity and assigned roles.

How DispatcherServlet works with Spring Security

In Spring MVC, all incoming HTTP requests are channeled through a single servlet known as the DispatcherServlet. This servlet directs these requests to your controller classes, where you define the endpoints for your application. Spring Security steps in at this point by introducing filter classes before the HTTP requests reach the DispatcherServlet.

This means that every incoming request will pass through these filter classes one by one. This approach allows us to verify authentication and authorization states before the request reaches the DispatcherServlet and subsequently the controllers. In essence, this is the core function of Spring Security.

Spring Security Architecture

Security Filter Chain

The Security Filter Chain is the backbone of Spring Security. It’s a series of filters that processes incoming HTTP requests and communicates with the Authentication Manager for the validation of requests.

These filters work together to ensure that your web application is secure and responsible for tasks like authentication, authorization, session management and more. These filters work together to ensure that your web application is secure and responsible for tasks like authentication, authorization, session management and more

List of default Filters in Security Filter Chain

• ChannelProcessingFilter — ensures protocol such as HTTP or HTTPS.

• SecurityContextPersistenceFilter — ensures the user’s authentication details persist during their session.

• UsernamePasswordAuthenticationFilter — checks the provided credentials and creates an Authentication Object if they are valid.

• ConcurrentSessionFilter — control how many sessions a user can have and handle scenarios where a user exceeds the session limit.

• LogoutFilter — intercepts logout requests and terminates the user’s session

• RememberMeAuthenticationFilter — allow user to be remembered across sessions, even after logout.

• AnonymousAuthenticationFilter — responsible for creating an anonymous user who is not authenticated but has limited access to some parts of the application.

• SessionManagementFilter — handles invalidating sessions, session fixation protection, and other session-related tasks.

• ExceptionTranslationFilter — responsible for translating the exception into a unauthenticated response.

• FilterSecurityInterceptor — ensures the user have the necessary roles and permissions to access specific resources.

You can read more about filters here.

@Bean
public SecurityFilterChain securityFilterChain(HttpSecurity httpSecurity) {
    httpSecurity.authorizeHttpRequests(authorize ->
          authorize.requestMatchers("/signup/").permitAll()
                   .requestMatchers("/users").authenticated()
    ).httpBasic(Customizer.withDefaults());

    return httpSecurity.build();
}

This code defines a custom Security Filter Chain with specific security rules. It permits unrestricted access to the “/signup/” URL while requiring authentication for the “/users” URL. Additionally, it configures HTTP Basic Authentication as the authentication method.

Authentication Manager

The Authentication Manager is the central component that verifies user identities. It coordinates the authentication process and delegates the actual authentication to Authentication Providers.

In more complex systems, you might have multiple Authentication Managers. Each manager can be associated with a specific set of Authentication Providers, allowing you to handle different types of authentication for different parts of your application.

The AuthenticationManager is an interface which processes the Authentication request. It has an authenticate() method which takes Authentication object as a parameter.

Authentication authenticate(Authentication authentication) throws AuthenticationException;

The implementation class of AuthenticationManager is the ProviderManager class which provides the logic for authenticate() method. We can provide our own implementation class of AuthenticationProvider or can use the default implementation.

Authentication Providers

The Authentication Providers are the workers of the Authentication Manager. They are responsible for actually performing the authentication. AuthenticationProvider is an interface that defines the contract for authenticating users. It is responsible for taking an Authentication object, which represents the user’s credentials, and returning an authenticated Authentication object if the credentials are valid. If the credentials are invalid, the AuthenticationProvider should throw an AuthenticationException.

The AuthenticationProvider interface has two methods:

• authenticate() : This method takes an Authentication object as input and returns an authenticated Authentication object if the credentials are valid. If the credentials are invalid, the AuthenticationProvider should throw an AuthenticationException.

• supports() : This method takes an Authentication object as input and returns true if the AuthenticationProvider can authenticate the object. If the AuthenticationProvider cannot authenticate the object, it should return false .

Spring Security supports various types of Authentication Providers

• DaoAuthenticationProvider — uses a UserDetailsService to retrieve user details from database and compare credentials.

• LdapAuthenticationProvider — used for authenticating against LDAP servers.

• JwtAuthenticationProvider — used for validating JWT tokens of the user.

• Custom Authentication Providers — handle authentication using specific logic or external systems.

@Bean
public DaoAuthenticationProvider daoAuthenticationProvider() {
    DaoAuthenticationProvider daoAuthenticationProvider = new DaoAuthenticationProvider();
    daoAuthenticationProvider.setUserDetailsService(userDetailsService());
    daoAuthenticationProvider.setPasswordEncoder(passwordEncoder());
    return daoAuthenticationProvider;
}

• In this snippet, a custom DaoAuthenticationProvider is defined, configured with a UserDetailsService and a PasswordEncoder.

UserDetailsService

The UserDetailsService is an interface which retrieves user details during the authentication process. It is often used in conjunction with Authentication Providers to obtain user details from the database for authentication.

public class UserDetailsServiceImpl implements UserDetailsService {
    @Autowired
    public UserRepository userRepository;
    
    @Override
    public UserDetails loadUserByUsername(String username) throws UsernameNotFoundException {
        Optional<User> user = userRepository.findById(username);
        if (user.isEmpty()) {
            throw new UsernameNotFoundException("user not found with this username");
        }

        return new UserDetailsImpl(user.get());
    }
}

The UserDetailsServiceImpl class is implemented as a custom UserDetailsService, and it overrides the loadUserByUsername method to retrieve user details from a database via a UserRepository.

@Bean
public UserDetailsService userDetailsService() {
    return new UserDetailsServiceImpl();
}

In the above code, we are defining UserDetailsServiceImpl as custom UserDetailsService.

Password Encoder

The Password Encoder is used to hash and verify passwords securely as storing and comparing passwords is crucial for user authentication.

• BCryptPasswordEncoder — this is widely recommended choice for securely hashing passwords in Spring Security. It handles the generation of random salts for each password.

• NoOpPasswordEncoder — this encoder does not perform any hashing or encoding of passwords and stores passwords in plain text which makes them highly vulnerable.

• StandardPasswordEncoder — this encoder uses one-way hashing algorithm which is less secure and it is not recommended.

• MessageDigestPasswordEncoder — this encoder uses a specified message digest algorithm (e.g., SHA-256) to hash passwords. While it’s more secure than plain text, it’s not as strong as BCrypt and is considered less secure in modern applications.

• SCryptPasswordEncoder — SCrypt is another secure password hashing algorithm, similar to BCrypt. It’s designed to be memory-intensive, making it resistant to certain types of attacks. SCryptPasswordEncoder is a good choice for secure password hashing.

@Bean
public PasswordEncoder passwordEncoder() {
    return new BCryptPasswordEncoder();
}

In the above code, a BCryptPasswordEncoder is defined as the custom PasswordEncoder .

SecurityContextHolder and Principal

The SecurityContextHolder class manages the user’s security context throughout the request-response lifecycle. It stores information about the currently authenticated user such as roles, and other security-related data.

Once the user is successfully authenticated, Spring Security creates an Authentication Object within the security context managed by SecurityContextHolder.

The Principal represents the current user’s username, authorities, and other user-specific data.

Authentication authentication = SecurityContextHolder.getContext().getAuthentication();
String username = authentication.getName();

The above code snippet is an example of how to retrieve the current user’s username from the Principal.

Request & Response lifecycle

Now, let us combine all the components we have learned so far and understand the whole flow of the architecture.

Authentication and UsernamePasswordAuthenticationToken

1. Authentication

It is an interface in Spring Security which represents token for incoming authentication request or an authenticated Principal (an interface which represents an entity like an individual) AuthenticationManager.authenticate() method.

Some methods provided:

Collection<? extends GrantedAuthority> getAuthorities();
 Object getCredentials();
 boolean isAuthenticated();
 void setAuthenticated(boolean isAuthenticated) throws IllegalArgumentException;

2. UsernamePasswordAuthenticationToken

This class extends the AbstractAuthenticationToken class (base class of authentication objects) and can be used with Username/Password authentication requests.

This class has two constructors:

public UsernamePasswordAuthenticationToken(Object principal, Object credentials) {
    super((Collection)null);
    this.principal = principal;
    this.credentials = credentials;
    this.setAuthenticated(false);
}

public UsernamePasswordAuthenticationToken(Object principal, Object credentials, Collection<? extends GrantedAuthority> authorities) {
    super(authorities);
    this.principal = principal;
    this.credentials = credentials;
    super.setAuthenticated(true);
}

The first constructor can be used for incoming requests to create unauthenticated Authentication object.

Authentication authentication = new UsernamePasswordAuthenticationToken(username,password);

The second constructor can be used to create a fully authenticated Authentication object.

Authentication authToken = new UsernamePasswordAuthenticationToken(username, password, userAuthorities)

This fully authenticated Authentication object is then returned from AuthenticationProvider/AuthenticationManager and represents an authenticated user. This authenticated object is then set in the SecurityContext. The SecurityContext in Spring Security is a representation of the security context of the current thread of execution. It contains information about the currently authenticated user, such as their username, authorities, and session information.

1. The request is intercepted by the Security Filter Chain. The Security Filter Chain consists of a series of filters, each with a specific security-related task.

2. If the user is not yet authenticated (i.e., not logged in), Spring Security’s authentication filters will trigger the Authentication Manager. If the credentials match, the Authentication Manager generates an Authentication Object indicating a successful authentication.

3. The Authentication Manager uses the configured Authentication Providers to verify the user’s credentials.

4. Authentication Providers will use the PasswordEncoder to store and compare passwords.

5. Authentication Providers may use the UserDetailsService to fetch user details. The user’s credentials are compared to the stored or provided credentials

6. UserDetailsService will fetch the data from the database.

7. The status of the authentication process will be sent to the user as a success or unauthorized response.

8. This Authentication object is stored within the security context managed by SecurityContextHolder. The security context now represents the authenticated user.

Conclusion

This article provides a brief description of authentication flow and different components which are needed to set up authentication. Anyone who wants to use Spring Security must have a clear understanding of these concepts before moving to more customizations and implementation.

Thank you for reading, and happy coding! 😊

What is JWT?

JWT, or JSON Web Tokens, are like digital passes that help keep web applications secure. When someone logs into an application, the server gives them a JWT.

It’s like getting a badge that proves who you are every time you come back. Each time the user wants to access a secure part of the application, they show this badge, and the server knows it’s safe to let them in.

Here’s why JWTs are so great:

1. Easy to Manage: They are simple strings of text, making them easy to handle and send between computers and devices.

2. All-in-One: Everything the server needs to verify the user is packed into the JWT. This means the server doesn’t have to keep asking a database for information, speeding things up.

3. Safe: They can be encrypted and signed, making it hard for unauthorized people to mess with them.

A JWT token typically appears as shown on the left side in the provided image below:

You can see more here: jwt.io

Parts of a JWT token:

A JWT consists of three main parts, each separated by a dot (.). Here's an easy breakdown:

1. Header

   • Purpose: The header typically tells us the type of the token, which is JWT, and the algorithm that is used for signing the token, like HMAC SHA256 or RSA.

   • Example: {"alg": "HS256", "typ": "JWT"}

   • Explanation: This is a simple JSON object that states the JWT is using the HS256 algorithm for encryption. It's encoded in Base64Url format to make it URL-safe.

2. Payload

   • Purpose: The payload contains the claims. Claims are statements about an entity (typically, the user) and additional data.

   • Example: {"sub": "1516239022", "name": "John Doe", "admin": true}

   • Explanation: This JSON object contains information about the user and other metadata. It says that the subject (user) of the JWT has the ID 1516239022, the name John Doe, and has administrative rights.

3. Signature

   • Purpose: The signature is used to verify that the sender of the JWT is who it says it is and to ensure that the message wasn't changed along the way.

   • How It's Made: To create the signature, you take the encoded header, the encoded payload, a secret, the algorithm specified in the header (like HMAC SHA256), and sign that.

   • Explanation: For example, if you are using the HMAC SHA256 algorithm, the signature will be created by applying the HMAC SHA256 algorithm to the combined string of Base64Url encoded header and payload, using a secret key.

These three parts together form the JWT: header.payload.signature. When the JWT is sent, it appears as a string of three Base64Url-encoded parts, separated by dots, looking something like this: eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c

That's essentially what a JWT is made up of and how each part functions.

Why JWT?

One might wonder: if we can simply use a username and password to access secure endpoints in a web application, why then do we opt for using JWT with Spring Security?

Choosing JWT (JSON Web Tokens) with Spring Security instead of just using usernames and passwords has several benefits, especially for modern web applications:

1. No Need for Constant Check-ins: JWTs contain all the user information right inside them, so servers can verify who you are without having to ask the database every time you make a request. This is great for handling lots of users because it cuts down on database traffic.

2. Extra Security: While you still use a username and password to log in, JWT provides a more secure way to keep checking who you are after you're logged in. The tokens are protected and have a built-in expiration time, which helps prevent them from being misused.

3. Works Everywhere: JWTs are good at working across different systems and devices. This means that once you're logged in to one part of a system, you can use other parts too without having to log in again.

4. Faster: Because your server isn't always asking the database who you are, things can run faster. With JWT, your server just needs to check the token you send with your requests, which speeds things up.

5. Carries More Info: JWTs can also include extra details like what permissions you have, which helps the server know what you're allowed to do without having to look it up all the time.

In simple terms, JWT makes things more secure, quicker, and easier to manage, especially when you have lots of users or need to work across different parts of a system. This makes them a strong choice for modern web applications.

In the case of microservices, where there are multiple servers involved:

In a microservices setup, where multiple small services work together, JWT (JSON Web Tokens) offer some clear benefits for managing security efficiently:

• Independent Verification: Each microservice can check who you are on its own using the information in the JWT. There’s no need to keep asking a central server to verify your identity, which cuts down on delays.

• Less Network Traffic: Because JWTs carry all needed user info right inside them, there’s no need for services to constantly talk to each other or to a central database to check user details. This reduces network traffic and speeds things up.

• Works Across Services: JWTs are great for setups that stretch over different areas or systems. Once you’re logged in and have a JWT, any service within the system can recognize and trust it.

• No Memory Needed: JWTs help keep the system simple because services don’t need to remember user sessions. Everything needed is in the JWT, which supports the stateless operation of microservices.

Overall, JWT makes security smoother, quicker, and more scalable in a microservices architecture, fitting well with the need for each service to operate independently yet securely.

Integrating JWT with Spring Security

In the previous post, we implemented security using basic user and password authentication. In this post, we will replace it with JWT. You can refer to my GitHub link for more details.

To implement JSON Web Tokens (JWT) for authorization in a Spring Boot Maven project, we need to include specific dependencies in our pom.xml file.

<dependencies>
    <dependency>
			<groupId>io.jsonwebtoken</groupId>
			<artifactId>jjwt-api</artifactId>
			<version>0.11.5</version>
		</dependency>
		<dependency>
			<groupId>io.jsonwebtoken</groupId>
			<artifactId>jjwt-impl</artifactId>
			<version>0.11.5</version>
		</dependency>
		<dependency>
			<groupId>io.jsonwebtoken</groupId>
			<artifactId>jjwt-jackson</artifactId>
			<version>0.11.5</version>
		</dependency>
</dependencies>

Explanation for the above dependencies:

1. jjwt-jackson

• What it does: Integrates the Jackson library with JWT operations, optimizing how your Java application encodes and decodes JSON data within JWTs.

2. jjwt-api

• What it does: Supplies the fundamental classes and interfaces for constructing and validating JWTs, providing the building blocks for JWT functionality in Java.

3. jjwt-impl

• What it does: Provides the actual implementation for the interfaces from jjwt-api, enabling the generation, parsing, and management of JWTs according to your application's security protocols.

Why Redis and ThreadLocal?

Redis is renowned for its blazing-fast performance as an in-memory data store, making it an excellent choice for session storage where speed and reliability are paramount.

ThreadLocal, on the other hand, provides thread-local variables in Java, allowing each thread to maintain its own session state. This is particularly useful in multi-threaded environments like web servers where each request is handled by a separate thread.

Setting Up the Environment

Before diving into the code, ensure you have Redis installed and running on your machine. If you haven't set up Redis yet, you can download and install it from the official website.

Prerequisites

• Spring Boot 3
• JDK 21
• Redis

Step-by-Step Implementation

1. Set up Redis

Ensure Redis is installed and running on your machine.

2. Add Dependencies

Add the necessary dependencies to your build.gradle if you are using Gradle.

dependencies {
    implementation 'org.springframework.boot:spring-boot-starter-data-redis'
    implementation 'redis.clients:jedis:4.3.1'
    implementation 'org.springframework.boot:spring-boot-starter-web'
}

If you are using Maven, add these dependencies to your pom.xml.

<dependencies>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-data-redis</artifactId>
    </dependency>
    <dependency>
        <groupId>redis.clients</groupId>
        <artifactId>jedis</artifactId>
        <version>4.3.1</version>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-web</artifactId>
    </dependency>
</dependencies>

3. Configure Redis

Create a Redis configuration class to set up the Redis connection.

package com.example.demo.config;

import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.data.redis.connection.RedisConnectionFactory;
import org.springframework.data.redis.connection.jedis.JedisConnectionFactory;
import org.springframework.data.redis.core.RedisTemplate;
import org.springframework.data.redis.serializer.StringRedisSerializer;
import redis.clients.jedis.JedisPoolConfig;

@Configuration
public class RedisConfig {

    @Bean
    public RedisConnectionFactory redisConnectionFactory() {
        JedisPoolConfig poolConfig = new JedisPoolConfig();
        poolConfig.setMaxTotal(128);
        return new JedisConnectionFactory(poolConfig);
    }

    @Bean
    public RedisTemplate<String, Object> redisTemplate() {
        RedisTemplate<String, Object> template = new RedisTemplate<>();
        template.setConnectionFactory(redisConnectionFactory());
        template.setKeySerializer(new StringRedisSerializer());
        template.setValueSerializer(new StringRedisSerializer());
        return template;
    }
}

4. Create a Session Manager Using ThreadLocal

Create a SessionManager class that uses ThreadLocal to manage the session ID.

package com.example.demo.session;

public class SessionManager {
    private static final ThreadLocal<String> sessionThreadLocal = new ThreadLocal<>();

    public static void setSessionId(String sessionId) {
        sessionThreadLocal.set(sessionId);
    }

    public static String getSessionId() {
        return sessionThreadLocal.get();
    }

    public static void removeSessionId() {
        sessionThreadLocal.remove();
    }
}

5. Create a Session Service to Interact with Redis

Create a service class that interacts with Redis to manage session data.

package com.example.demo.service;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.data.redis.core.RedisTemplate;
import org.springframework.stereotype.Service;

@Service
public class SessionService {

    @Autowired
    private RedisTemplate<String, Object> redisTemplate;

    public void setSessionData(String sessionId, String key, String value) {
        redisTemplate.opsForHash().put(sessionId, key, value);
    }

    public String getSessionData(String sessionId, String key) {
        return (String) redisTemplate.opsForHash().get(sessionId, key);
    }

    public void invalidateSession(String sessionId) {
        redisTemplate.delete(sessionId);
    }
}

6. Implement a Filter to Manage Session Lifecycle

Create a filter that manages the session lifecycle, setting and removing the session ID in ThreadLocal.

package com.example.demo.filter;

import com.example.demo.session.SessionManager;
import com.example.demo.service.SessionService;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;

import javax.servlet.*;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import java.io.IOException;

@Component
public class SessionFilter implements Filter {

    @Autowired
    private SessionService sessionService;

    @Override
    public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain) throws IOException, ServletException {
        HttpServletRequest httpRequest = (HttpServletRequest) request;
        HttpServletResponse httpResponse = (HttpServletResponse) response;

        String sessionId = httpRequest.getSession().getId();
        SessionManager.setSessionId(sessionId);

        // Example: Set some session data
        sessionService.setSessionData(sessionId, "user", "JohnDoe");

        try {
            chain.doFilter(request, response);
        } finally {
            SessionManager.removeSessionId();
        }
    }

    @Override
    public void init(FilterConfig filterConfig) throws ServletException {
    }

    @Override
    public void destroy() {
    }
}

7. Register the Filter

Register the filter in your Spring Boot application.

package com.example.demo;

import com.example.demo.filter.SessionFilter;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.web.servlet.FilterRegistrationBean;
import org.springframework.context.annotation.Bean;

@SpringBootApplication
public class DemoApplication {

    public static void main(String[] args) {
        SpringApplication.run(DemoApplication.class, args);
    }

    @Bean
    public FilterRegistrationBean<SessionFilter> loggingFilter() {
        FilterRegistrationBean<SessionFilter> registrationBean = new FilterRegistrationBean<>();
        registrationBean.setFilter(new SessionFilter());
        registrationBean.addUrlPatterns("/*");

        return registrationBean;
    }
}

Conclusion

By leveraging Redis and ThreadLocal in a Spring Boot 3 application, you can efficiently manage user sessions in a multi-threaded environment. Redis provides fast and reliable session storage, while ThreadLocal ensures thread-safe handling of session data. This setup enhances performance and scalability, making it ideal for modern web applications.

Implementing this approach in your Spring Boot application not only simplifies session management but also ensures a seamless user experience. Happy coding!

Prerequisites

You must need these tools installed on your computer to follow this tutorial.

• JDK 11 or higher

• Maven 3.8 or higher

• PostgreSQL running on Docker

• An HTTP client such as Postman, Insomnia, cURL, etc.

Set up the database

We need Docker to run a container for Postgres 16; you can skip it if Postgres is installed on your computer. Run the command below to start the Docker container from the Postgres image:

docker run --name some-postgres -e POSTGRES_PASSWORD=mysecretpassword -e POSTGRES_DB=management-employee -p 5431:5432 -d postgres:16-alpine

Set up the project

We won't go into the details of building a CRUD application since the focus of this article is on securing the application. However, a basic overview is provided for better understanding.

We have included the following dependencies in our project:

1. Spring Web: Enables building web applications with Spring, including RESTful services.

2. PostgreSQL Driver: Connects your application to a PostgreSQL database for data storage.

3. Spring Data JPA: Simplifies data access and manipulation through JPA repositories.

4. Lombok: Reduces boilerplate code by automatically generating getters, setters, and other common methods.

5. Spring Boot DevTools: Provides fast application restarts, live reload, and configuration options for a smoother development process.

The Spring Boot online project starter helps us create the project with these dependencies; go to the URL start.spring.io to generate a new project.

I chose Java 21 and Maven as the dependency manager, but you can use whatever you want.

Click on the button Generate to download the project, open it in your IDE, and install the Maven dependencies.

Developing a simple employee-based management system

Create the Employee Entity

This Java class defines an Employee entity with attributes such as employeeId, name, email, department, and company, utilizing JPA annotations for ORM and Lombok annotations for boilerplate code like getters, setters, and constructors.

package com.talee.employee.management.model;

import jakarta.persistence.Entity;
import jakarta.persistence.GeneratedValue;
import jakarta.persistence.GenerationType;
import jakarta.persistence.Id;
import lombok.AllArgsConstructor;
import lombok.Data;
import lombok.NoArgsConstructor;

@Entity
@Data
@NoArgsConstructor
@AllArgsConstructor
public class Employee {

  @Id
  @GeneratedValue(strategy = GenerationType.AUTO)
  private Long employeeId;
  private String name;
  private String email;
  private String nationality;
  private Integer old;

}

Create the Employee Repository

This interface defines a repository for Employee entities, extending Spring Data JPA to facilitate database operations.

package com.talee.employee.management.repo;

import com.talee.employee.management.model.Employee;
import org.springframework.data.jpa.repository.JpaRepository;

public interface EmployeeRepository extends JpaRepository<Employee, Integer> {
}

Create the Employee Service

The EmployeeService class interact with the EmployeeRepository to perform CRUD operations on Employee entities, allowing for the creation, retrieval, updating, and deletion of employee records in the database.

package com.talee.employee.management.service;

import com.talee.employee.management.model.Employee;
import java.util.List;
import java.util.Optional;
import org.springframework.stereotype.Service;

@Service
public class EmployeeService {

  private final EmployeeRepository repository;

  public EmployeeService(EmployeeRepository repository) {
    this.repository = repository;
  }

  public List<Employee> findAll() {
    return repository.findAll();
  }

  public Optional<Employee> findById(Integer id) {
    return repository.findById(id);
  }

  public Employee save(Employee employee) {
    return repository.save(employee);
  }

  public void deleteById(Integer id) {
    repository.deleteById(id);
  }

  public Optional<Employee> updateEmployee(Integer id, Employee employeeDetails) {
    return repository.findById(id).map(employee -> {
      employee.setName(employeeDetails.getName());
      employee.setEmail(employeeDetails.getEmail());
      employee.setNationality(employeeDetails.getNationality());
      employee.setOld(employeeDetails.getOld());
      return Optional.of(repository.save(employee));
    }).orElse(Optional.empty());
  }
}

Create the Employee Controller

package com.talee.employee.management.controller;

import com.talee.employee.management.model.Employee;
import com.talee.employee.management.service.EmployeeService;
import java.util.List;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.DeleteMapping;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.PutMapping;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

@RestController
@RequestMapping("/api/employees")
public class EmployeeController {

  private final EmployeeService service;

  public EmployeeController(EmployeeService service) {
    this.service = service;
  }

  @GetMapping
  public List<Employee> getAllEmployees() {
    return service.findAll();
  }

  @GetMapping("/{id}")
  public ResponseEntity<Employee> getEmployeeById(@PathVariable Integer id) {
    return service.findById(id)
        .map(ResponseEntity::ok)
        .orElseGet(() -> ResponseEntity.notFound().build());
  }

  @PostMapping
  public Employee createEmployee(@RequestBody Employee employee) {
    return service.save(employee);
  }


  @PutMapping("/{id}")
  public ResponseEntity<Employee> ulspdateEmployee(@PathVariable Integer id,
      @RequestBody Employee employeeDetails) {
    return service.updateEmployee(id, employeeDetails)
        .map(ResponseEntity::ok)
        .orElseGet(() -> ResponseEntity.notFound().build());
  }

  @DeleteMapping("/{id}")
  public ResponseEntity<?> deleteEmployee(@PathVariable Integer id) {
    return service.findById(id)
        .map(employee -> {
          service.deleteById(id);
          return ResponseEntity.ok().build();
        })
        .orElseGet(() -> ResponseEntity.notFound().build());
  }
}

Import data to Database

Our application is configured to automatically populate the database with initial data at startup, utilizing SQL files named schema.sql and data.sql.`

schema.sql

CREATE TABLE IF NOT EXISTS employee (
                          employee_id BIGSERIAL PRIMARY KEY,
                          name VARCHAR(255),
                          email VARCHAR(255),
                          nationality VARCHAR(255),
                          old INTEGER
);

data.sql

INSERT INTO Employee (name, email, nationality, old) VALUES ('Eric Cantona', 'eric.cantona@manutd.com', 'French', 57);
INSERT INTO Employee (name, email, nationality, old) VALUES ('Ryan Giggs', 'ryan.giggs@manutd.com', 'Welsh', 50);
INSERT INTO Employee (name, email, nationality, old) VALUES ('Paul Scholes', 'paul.scholes@manutd.com', 'English', 49);
INSERT INTO Employee (name, email, nationality, old) VALUES ('Roy Keane', 'roy.keane@manutd.com', 'Irish', 52);
INSERT INTO Employee (name, email, nationality, old) VALUES ('David Beckham', 'david.beckham@manutd.com', 'English', 48);

Configure the database connection

Let's configure the application to connect to the database and perform database table creation (using Hibernate under the hood). Open the application configuration file src/resources/application.yml and add the code below:

spring:
  application:
    name: employee-management
  datasource:
    url: jdbc:postgresql://localhost:5431/management-employee
    username: postgres
    password: 3+0M1606Pc_4
  jpa:
    hibernate:
      ddl-auto: update
    show-sql: true
    properties:
      hibernate:
        dialect: org.hibernate.dialect.PostgreSQLDialect
  sql:
    init:
      mode: always

server:
  port: 8018

Run the application with the command mvn spring-boot:run; it will start at port 8018.

If we pay attention to the console when starting the application, we can see the message displaying a security password generated because Spring Security has the HTTP Basic authentication enabled by default.

Protecting our Web Application using Default Spring Security Configuration

How It Works

Once you include Spring Security in your project, it instantly sets up some security features for you. This means your application will have a basic level of security without you needing to do anything extra.

For securing our web application, we need another dependency i.e, Spring Security:

Spring Security: Adds authentication and authorization features to secure your application.

Given that our project is built with Maven and Spring Boot, the dependency for Spring Security would appear in the pom.xml file as follows:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-security</artifactId>
</dependency>

Our final pom.xml file would be structured as follows to incorporate the specified dependencies:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">

	<!-- Basic POM Information -->
	<modelVersion>4.0.0</modelVersion>

	<!-- Parent Project Information -->
	<parent>
		<groupId>org.springframework.boot</groupId>
		<artifactId>spring-boot-starter-parent</artifactId>
		<version>3.3.2</version>
		<relativePath/> <!-- lookup parent from repository -->
	</parent>

	<!-- Project Coordinates -->
	<groupId>com.talee</groupId>
	<artifactId>employee.management</artifactId>
	<version>0.0.1-SNAPSHOT</version>
	<name>employee.management</name>
	<description>Implement authentication on a Web Api</description>
	<url/> <!-- URL for the project, can be filled as needed -->

	<!-- Licensing Information -->
	<licenses>
		<license/>
	</licenses>

	<!-- Developer Information -->
	<developers>
		<developer/>
	</developers>

	<!-- Source Control Management Information -->
	<scm>
		<connection/>
		<developerConnection/>
		<tag/>
		<url/>
	</scm>

	<!-- Project Properties -->
	<properties>
		<java.version>21</java.version>
	</properties>

	<!-- Project Dependencies -->
	<dependencies>
		<!-- Spring Boot Starter for JPA -->
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-data-jpa</artifactId>
		</dependency>

		<!-- Spring Boot Starter for Security -->
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-security</artifactId>
		</dependency>

		<!-- Spring Boot Starter for Web -->
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-web</artifactId>
		</dependency>

		<!-- Spring Boot DevTools for Development -->
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-devtools</artifactId>
			<scope>runtime</scope>
			<optional>true</optional>
		</dependency>

		<!-- PostgreSQL JDBC Driver -->
		<dependency>
			<groupId>org.postgresql</groupId>
			<artifactId>postgresql</artifactId>
			<scope>runtime</scope>
		</dependency>

		<!-- Lombok for Boilerplate Code Reduction -->
		<dependency>
			<groupId>org.projectlombok</groupId>
			<artifactId>lombok</artifactId>
			<optional>true</optional>
		</dependency>

		<!-- Spring Boot Starter for Testing -->
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-test</artifactId>
			<scope>test</scope>
		</dependency>

		<!-- Spring Security Test Dependency -->
		<dependency>
			<groupId>org.springframework.security</groupId>
			<artifactId>spring-security-test</artifactId>
			<scope>test</scope>
		</dependency>
	</dependencies>

	<!-- Build Configuration -->
	<build>
		<plugins>
			<plugin>
				<groupId>org.springframework.boot</groupId>
				<artifactId>spring-boot-maven-plugin</artifactId>
				<configuration>
					<excludes>
						<exclude>
							<groupId>org.projectlombok</groupId>
							<artifactId>lombok</artifactId>
						</exclude>
					</excludes>
				</configuration>
			</plugin>
		</plugins>
	</build>

</project>

When attempting to access any API via the browser, the default login form will be presented like the one given below:

The default username provided by spring security is user, while the password is auto-generated and can be found in the console.

But this username password should not be used in real-time production scenarios

Here, we are trying to access an API i.e. to get the list of all employees. Attached is a small video to get a clearer picture.

In this article, we will delve into the components of Kafka, explore how Kafka works, and understand its advantages and disadvantages, as well as its applications in real life.

What is Apache Kafka?

Apache Kafka is an open-source stream-processing software platform developed by LinkedIn and donated to the Apache Software Foundation, written in Scala and Java.

“Apache Kafka is an open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.”

Source: Kafka home page

The project aims to provide a unified, high-throughput, low-latency platform for handling real-time data feeds. Its core architectural concept is an immutable log of messages that can be organized into topics for consumption by multiple users or applications. A file system or database commit log keeps a permanent record of all messages so Kafka can replay them to maintain a consistent system state.

Kafka stores data durably, in serialized fashion. It distributes data across a cluster of nodes, providing performance at scale that’s resilient to failures.

Kafka is available as open source code or from vendors who offer it as a cloud managed service, including AWS (MKS – Managed Kafka Service), Azure, Google and Confluent (Confluent Cloud), the company founded by the creators of Kafka.

What is Apache Kafka Used For?

Kafka is flexible and widely-used, but there are several use cases for which it stands out:

• Log/Metricsk, aggregation: Kafka provides log or event data as a stream of messages. It removes any dependency on file details by gathering physical log files from servers and storing them in a central location. Kafka also supports multiple data sources and distributed data consumption.

• Stream processing: Kafka is valuable in scenarios in which real-time data is collected and processed. This includes raw data that’s consumed from Kafka topics and then enriched or processed into new Kafka topics for further consumption as part of a multi-step pipeline.

• Commit logs: Any large-scale distributed system can use Kafka to represent external commit logs. Replicated logs across a Kafka cluster help data recovery when nodes fail.

• Activity tracking: Data from user click stream activities, such as page views, searches, and so on, are published to central topics, with one topic per activity type.

• Event Sourcing: Event-driven architectures benefit from Kafka's ability to capture and store events that represent state changes, enabling the implementation of reliable and scalable event sourcing.

• Data Integration: Kafka acts as a bridge between different systems, enabling seamless data integration between applications and databases.

Understanding Event-driven Architecture in Apache Kafka

Kafka receives and sends data as events called messages which are organized into topics which are “published” by data producers and “subscribed” to by data consumers, which is why Kafka is sometimes called a “pub/sub” system. An event is simply a statement that something occurred in the real world. An event may also be referred to as a record or a message.

What is a Kafka cluster?

A cluster is a distributed computing concept in which a group of devices works together to achieve a specific purpose. Multiple Kafka brokers form a Kafka cluster.

The main goal of a Kafka cluster is to spread workloads evenly across replicas and partitions. Kafka clusters can scale without interruption. They also manage the persistence and replication of data messages. If one broker fails, other Kafka brokers step in to offer similar services without data loss or degraded latency.

Kafka Cluster Architecture

Multiple components comprise the core Kafka cluster architecture:

• Kafka Events

• Broker

• Controller

• Partitions

• Consumer

• Producer

• Topic

• Zookeeper (phasing out)

• Schema registry

About the Apache Kafka Events

A Kafka event records the fact that "something has happened" in the world or within your business. It is also known as a record or message. In many Kafka documents, the terms event, record, and message are used interchangeably.

Reading and writing data in Kafka are done through events. Each event contains a key, value, and metadata (if any).

Example of an event containing a key, value, and metadata (timestamp):

{
  "key": "user123",
  "value": {
    "action": "login",
    "status": "success"
  },
  "metadata": {
    "timestamp": "2024-07-20T10:15:30Z"
  }
}

About the Apache Kafka broker

A broker is a single Kafka server. Kafka brokers receive messages from producers, assign them offsets, and commit the messages to disk storage. An offset is a unique integer value that Kafka increments and adds to each message as it’s generated. Offsets are critical for maintaining data consistency in the event of a failure or outage, as consumers use offsets to return to the last-consumed message after a failure. Brokers respond to partition call requests from consumers and return messages that have been committed to disk. A single broker operates based on the specific hardware and its functional properties.

About the Apache Controller

Kafka brokers form a cluster by directly or indirectly sharing information. In a Kafka cluster, one broker serves as the Apache controller. The controller is responsible for managing the states of partitions and replicas and for performing administrative tasks such as reassigning partitions and registering handlers to be notified about changes.

Although the Controller service runs on every broker in a Kafka cluster, only one broker can be active (elected) at any point in time. The Kafka Controller is created and starts up as soon as the Kafka server starts up.

About partitions in Apache Kafka

Partitioning is a foundational principle in most distributed systems. In Kafka, a topic is split into multiple partitions. A partition is a discrete log file. Kafka writes records to each partition in append-only fashion. In other words, all of the records belonging to a particular topic are divided and stored in partitions.

Kafka distributes the partitions of a particular topic across multiple brokers. This distributed layout improves scalability through parallelism, as you’re not limited to one specific broker’s I/O. Kafka also replicates partitions and spreads the replicas across the cluster. This provides robust fault tolerance; if one broker fails other brokers can take over and pick up where the failed machine left off.

When a new message is written to a topic, Kafka adds it to one of the topic’s partitions. Messages with the same key (for example, an enrollment number or customer ID) are published to the same partition. Kafka assures that any reader of a given topic/partition always consumes messages in their published sequence.

Each event in a partition is assigned a unique offset, starting from 0 for the first event in the partition and incrementing by one for each subsequent event. The offset is used to determine the position of the event within a partition.

Each partition has a broker designated as the leader, which owns that partition, while the remaining brokers store replicas of the partition, known as followers.

If the leader broker fails, one of the followers with updated data will be selected as the new leader. This process is called leader failover, and it ensures the availability of the data.

About consumers in Apache Kafka

Consumers are applications or machines that subscribe to topics and process published message feeds. Sometimes called “subscribers” or “readers” consumers read the messages in the order in which they were generated.

A consumer uses the offset to track which messages it has already consumed. A consumer stores the offset of the last consumed message for each partition so that it can stop and restart without losing its place.

Consumers interact with a topic as a group (although a group can consist of only one consumer). This enables scalable processing. The group ensures that one member only consumes each partition. if a single consumer fails, the group’s remaining members reorganize the consumed partitions to compensate for the absent member. Groups enable Kafka to consume topics with a massive amount of messages horizontally.

Consumers connect to Kafka Brokers via the TCP network protocol. This is a bi-directional connection.

When a consumer group reads events from partitions, there are three possible scenarios:

• Scenario 1: The consumer group has fewer consumers than the number of partitions in a topic.

Consumers 1 and 2 read events from 4 partitions of Topic T1 sequentially.

• Scenario 2: The consumer group has the same number of consumers as the number of partitions in a topic.

If we add 2 consumers to Consumer Group 1, making the number of consumers equal to the number of partitions, each consumer will read events from one corresponding partition. In cases where consumers perform high-latency operations such as writing to a database or performing time-consuming computations on the data, increasing the number of consumers will distribute the load, thereby speeding up the data reading process from a topic.

• Scenario 3: The consumer group has more consumers than the number of partitions in a topic.

We should not have more consumers than partitions in a topic because some consumers may become idle as all partitions are occupied, leading to events being missed or not read.

About producers in Apache Kafka

Kafka Producer is a client application that publishes events to a specific topic in Kafka and always writes to the leader broker. By default, producers do not care which partition an event is written to and will publish events evenly across all partitions of a topic. In some cases, a producer will send events directly to specific partitions.

Producers connect to Kafka Brokers via the TCP network protocol. This is a bi-directional connection.

The diagram below provides an overview of the components in Kafka producers.

The process of sending events from Kafka producers to Kafka brokers involves 4 steps.

• Step 1: Create a ProducerRecord

A producer publishes an event to Kafka by creating a ProducerRecord, which must include a topic and value, and optionally a partition and key.

• Step 2: Serializer

Before sending the ProducerRecord over the network, the producer will serialize the key and value into ByteArrays (byte arrays).

• Step 3: Determine the number of partitions

After the Serializer step, the data is sent to a Partitioner. If a partition is specified, the Partitioner returns the specified partition; otherwise, the Partitioner selects a partition based on the ProducerRecord key.

• Step 4: Brokers process the events and return results to the producers

Once the producer knows which topic and partition the event needs to be sent to, it adds the event to a batch of records. These records will then be sent together to the same topic and partition. An independent thread is responsible for sending these batches of records to the appropriate Kafka brokers.

The broker sends back a response upon receiving the events. If the event is successfully written to Kafka, the broker returns an object containing RecordMetadata, which includes the topic, partition, and offset of the record within the partition. If the write is unsuccessful, the broker returns an error. When the producer receives an error, the event will be retried several times before giving up and returning an error.

About Apache Kafka topics

Topics classify messages in Kafka. A topic is roughly analogous to a database table or folder. Topics are further subdivided into several partitions as described above

A single topic can be scaled horizontally across various servers to deliver performance well beyond the capabilities of a single server. A Kafka cluster sustains the partitioned log for each topic. Note that while a topic generally has multiple partitions, there is no assurance of message time-ordering throughout the whole topic – only within a single partition.

About Apache Kafka Zookeeper

Zookeeper stores metadata for the Kafka brokers. It acts as a liaison between the broker and the consumers, enabling distributed processes to communicate with one another via a common centralized namespace of data registers called znodes.

“ZooKeeper is a distributed, open-source coordination service for distributed applications.”

Source: ZooKeeper home page

With the introduction of Apache Kafka 3.0, Zookeeper is in the process of being removed. Many users had complained about having to manage a separate system and the single point of failure that Zookeeper created. Going forward, Kafka brokers will essentially assume Zookeeper’s functions, storing metadata locally in a file. The Controller takes over registering brokers and removing failed brokers from the cluster, and upon startup the brokers just read from the Controller what has changed, not the full state. This enables Kafka to support more partitions with less CPU consumption.

About queues in Apache Kafka

Initially Kafka was developed as a messaging queue. But it took on a life of its own, and as a result, was donated to Apache for further development. Kafka still operates like a traditional messaging queue such as RabbitMQ, in that it enables you to publish and subscribe to streams of messages. But comparing RabbitMQ vs Kafka, there are three core differences.

1. Kafka operates as a modern distributed system that runs as a cluster and can scale to handle any number of applications.

2. Kafka is designed to serve as a storage system and can store data as long as necessary – most message queues (like RabbitMQ) remove messages immediately after the consumer confirms receipt.

3. Kafka is designed to handle stream processing, computing derived streams and datasets dynamically, rather than just in batches of messages.

About the Apache Kafka schema registry

A schema registry handles message schemas and routes them to topics. The schema registry verifies and maintains schemas implemented for Kafka messages. It also imposes compatibility before message addition. In this way publishers know which topics receive which forms (schemas) of events, and subscribers know how to interpret and extract information from events in a topic. The schema registry sends data serialized per schema ID so the consumer can map a schema to a message type.

Kafka stores the schema’s namespace in the record it receives from publishers. Then the reader uses the namespace to retrieve records from the schema registry and deserialize the data.

The schema registry saves Kafka from enumerating the same schema details repeatedly into every message. Producers communicate with the registry and write new message schema to it. The producer obtains the schema ID from the registry and includes that in the message.

About Core Kafka Enhancements

Beyond the core Kafka architecture are the following enhancements that extend the capabilities of Kafka. Kafka Streams integrates stream processing with Kafka and Kafka Connect facilitates connecting Kafka to external data sources and sinks.

What is Kafka Streams?

Kafka Streams is an API-driven client library for building applications and microservices, where the input and output data are stored in Kafka clusters. Kafka Streams jobs do not execute on Kafka brokers; rather, they run inside your application or microservice.

Kafka Streams’ approach to parallel processing resembles that of Kafka:

• Each stream partition is a fully-ordered sequence of data records that maps to a Kafka topic partition.

• A data record in the stream maps to a Kafka message from that topic.

• The keys of data records determine the partitioning of data in both Kafka and Kafka Streams – that is, how data is routed to specific partitions within topics.

Kafka Streams processes messages in real-time (that is, not in microbatches), with millisecond latency. It supports stateless and stateful processing and window operations.

Kafka Streams includes two special processors: a Source Processor and a Sink Processor.

• The source processor produces an input stream by consuming records from one or more Kafka topics and forwarding them to its down-stream processors.

• Sink Processor: A sink processor sends any received records from its up-stream processors to a specified Kafka topic.

The processed results can either be streamed back into Kafka or written to an external system.

2. What are Kafka Connect and Kafka Connectors?

Kafka Connect provides a scalable means of moving data between Kafka and other repositories. It provides APIs and a runtime for designing and operating connector plugins, which are frameworks that Kafka Connect executes and are essential for data movement.

The connector controls:

• The definition and number of tasks that operate for the connector

• How to divide the data-copying work among tasks.

• Getting task configurations from workers and passing them on to the target.

3. What are Kafka APIs?

Kafka provides five key APIs for Java, Scala, and command-line tools for management and administrative tasks. In addition, these APIs enable you to communicate with Kafka programmatically:

• The Admin API enables you to manage and analyze Kafka topics, brokers, Access Control Lists, and other objects.

• The Producer API enables applications to submit (write) data streams to Kafka cluster topics.

• The Consumer API enables programs to access (read) data streams from topics in the Kafka cluster.

• The Streams API provides stream processes that convert a data stream into an output stream.

The Connect API creates connections that continuously draw data from a source data system into Kafka or push data from Kafka into a target data system. (The Connect API is not often needed, as you can instead use pre-built connections without writing any code.)

Advantages of Kafka

• Scalability: The distributed architecture of Kafka enables seamless scalability. It can handle large volumes of data and increase data throughput by adding more brokers and partitions.

• Durability: Kafka ensures data durability through replication. Data is replicated across multiple brokers, preventing data loss even if brokers fail. Additionally, by adhering to the Append-only commit log principle, Kafka's commit log provides data durability, fault tolerance, and efficient message processing.

• Real-time processing: Kafka enables real-time data transmission and processing, making it suitable for applications that require low-latency data analysis and transmission.

• Open-source: Kafka is open-source software, providing the freedom to modify, customize, and extend its functionality to meet specific requirements. It can be used without paying licensing fees, helping to reduce costs associated with proprietary software. Kafka is cross-platform and can integrate with various tools and libraries, creating a large ecosystem.

• Long polling:: Although long polling is not a built-in feature of Kafka, the concept can be applied to how Kafka consumers interact with Kafka brokers to achieve efficient data consumption.

Disadvantages of Kafka

• ZooKeepers dependency: In older versions of Kafka, there was a dependency on Apache ZooKeeper for cluster coordination, which added complexity and contained potential points of failure.

• Complexity: The distributed nature of Kafka can introduce complexity in setup, configuration, and management, requiring expertise for effective deployment and maintenance. Additionally, learning to use Kafka and its ecosystem components effectively can take time and effort, especially for users new to distributed systems.

• Resource requirement: Managing Kafka clusters requires significant hardware resources, including memory, storage capacity, and network bandwidth.

Conclusion

Apache Kafka has redefined how organizations handle data streaming and event processing. Its distributed architecture, fault tolerance, and scalability make Kafka a powerful tool for managing the challenges of real-time data. From collecting and analyzing user interactions to building event-driven applications, Kafka's capabilities are vast and impactful.

Oracle Container Registry (OCR) is a fully-managed, enterprise-grade registry for storing and distributing container images on Oracle. Users can use OCR to store their own custom images, as well as pull down official images provided by Oracle. OCR supports both the Docker and OCI image formats and also provides features such as image scanning, image promotion and versioning, and image access control. Users can use the Oracle Cloud Infrastructure Console, Oracle Cloud Infrastructure CLI, or the Oracle Cloud Infrastructure SDK to interact with OCR.

Why to run Oracle database in a Docker Container?

There are several reasons why one might choose to run an Oracle database in a Docker container:

• Portability: Docker containers can be easily moved between environments, making it easy to deploy the database in different environments such as development, testing, and production.

• Isolation: Running the database in a container can help to isolate it from the host system, reducing the risk of conflicts with other software running on the same machine.

• Consistency: By packaging the database and its dependencies in a container, it ensures that the database will always run the same way, regardless of the environment in which it is deployed.

• Scalability: Containers can be easily scaled up or down as needed, making it easy to handle changes in load.

• Cost Savings: Docker containers can be run on-premises or in the cloud, allowing you to take advantage of cloud-based services while avoiding the costs of virtualization.

In this blog, you will see how you can use Docker to run Oracle Database in a Docker container

Getting Started

Pre-requisite

• Download and Install Docker

1. Visit Container Registry Oracle and create your new account

2. Select Enterprise

3. Click Sign In

4. Authenticate by entering your registered Email ID and Password

5. After successful authentication, navigate to the Database section and select the desired Oracle Database version.

6. Accept the license agreement to proceed with downloading the Oracle Database Docker image.

7. Accept the License and test if the docker login is working fine or not.

docker login container-registry.oracle.com
Username: xxx
Password: xxx
Login Succeeded

Oracle Database can be run in a Docker container, using the official Oracle Database Docker image. To deploy an Oracle Database using Docker, you will need to:

• Install Docker on your machine.

• Pull the Oracle Database Docker image using the command:

8. Open a terminal on your local machine and execute the Docker pull command to download the Oracle Database image. For example:

docker pull container-registry.oracle.com/database/enterprise:21.3.0.0

9. Verify the image is downloaded successfully by running:

docker images

You should see the Oracle Database image listed among the downloaded images.

10. Create a new container using the command:

docker run -d --name oracle-db -p 1521:1521 container-registry.oracle.com/database/enterprise:21.3.0.0

Once the container is up and running, you can connect to the Oracle Database using any SQL client, such as SQL*Plus, with the connection string hostname:1521/orclpdb1

The above command creates a new container with the name oracle-db and maps the host’s port 1521 to the container’s port 1521. The orclpdb1 is the default service name for the Oracle Database.

You can also customize the container by providing environment variables and volumes. For example, to set the SYS and SYSTEM password, you can use the following command

 docker run -d --name oracle-db -p 1521:1521 -e ORACLE_PWD=<password> container-registry.oracle.com/database/enterprise:21.3.0.0

Using Docker Compose

Create a docker-compose.yml file to define the Oracle Database service. Here is an example:

version: '3.8'
services:
  oracle-db:
    image: container-registry.oracle.com/database/enterprise:21.3.0.0
    environment:
      - ORACLE_SID=ORCLCDB
      - ORACLE_PDB=ORCLPDB1
      - ORACLE_PWD=Oracle_123
    ports:
      - 1521:1521
    volumes:
      - oracle-data:/opt/oracle/oradata
      - oracle-backup:/opt/oracle/backup
    healthcheck:
      test: ["CMD", "sqlplus", "-L", "sys/Oracle_123@//localhost:1521/ORCLCDB as sysdba", "@healthcheck.sql"]
      interval: 30s
      timeout: 10s
      retries: 5

volumes:
  oracle-data:
  oracle-backup:

This docker-compose.yml file uses the official Oracle database image from the Oracle Container registry and sets several environment variables for the container, including the ORACLE_SID, ORACLE_PDB, and ORACLE_PWD.

It also maps port 1521 on the host to port 1521 in the container, so that the database can be accessed from outside of the container.

It also creates two volumes oracle-data and oracle-backup to store the data and backup files respectively.

It also has a health check to check for the availability of the DB.

Don’t forget to replace the values of ORACLE_PWD with your desired password. Also, you should adjust the volumes and ports to match your specific requirements. The above example uses SQLPlus to check the health of the database, you can customise it accordingly.

Conclusion

You have successfully set up and run an Oracle Database in a Docker container using Docker Compose. This setup allows for easy management, portability, and scalability of your Oracle Database instances. Remember to follow best practices for security and resource management in a production environment.

In a monolithic architecture, all software components are interconnected and interdependent. If any component or single application’s function fails, the complete application could go down. Imagine you have a web application including functions like payment, history, and log in. If “history” function consumes more memory, the entire application would have to experience the same issue.

Unlike monolithic architecture, microservices break the large software projects into independent, loosely coupled, and smaller modules. In Microservices, the small programs are combined to deliver the functionalities of a large monolithic app

Read further to understand what exactly microservices are, its benefits and challenges associated with it and what can be done to overcome the challenges

As you know Docker is a tool for packaging, deploying, and running applications in lightweight containers. If you want to learn about the basics of Docker, refer to the Docker explained blog.

If you don’t have Docker installation, check out the docker installation guide.

Dockerfile Explained

A Dockerfile is a script that automatically creates containers on the Docker platform. A Dockerfile is basically a text document that contains all the commands a user could call on the command line to assemble an image.

The Docker platform runs natively on Linux and allows developers to build and run containers, self-contained applications or systems with no dependencies on the underlying infrastructure. Built upon the resource-isolation features of the Linux kernel, Docker helps developers and system administrators port applications across different systems and machines by running them inside containers.

Dockerfiles enable greater flexibility and portability of business applications. IT organizations use Dockerfiles to package applications and their dependencies in a virtual container that can run on premises. In public or private clouds, or on bare metal. Such containers allow multiple applications, worker tasks and other processes to run autonomously on a single physical machine or across multiple virtual machines. Kubernetes is an open source system for automatically managing and orchestrating containerized applications created by Dockerfiles.

In a Dockerfile everything on the left is INSTRUCTION, and on the right is an ARGUMENT to those instructions. Remember that the file name is "Dockerfile" without any extension.

The following table contains the important Dockerfile instructions and their explanation.

| Dockerfile Instruction | Explanation | |

Before installing docker on ubuntu, if you want to learn docker basics, head over to this article – What is docker

Docker supports almost all Unix and Linux distributions. Getting started with Docker is very easy. There is no complicated configuration for Docker.

Install Docker on Ubuntu

We will look at two methods to install Docker.

1. Install Docker from apt repository

2. Install Docker using one line command.

Install Docker From apt Repository

Docker package is available in the native apt repository. The installation package available in the repository will not be the latest version. If you want to install the latest release of Docker, you need to install it from the source.

Follow the instructions given below to install docker from the apt repository.

1. Set up Docker's apt repository.

# Add Docker's official GPG key:
sudo apt-get update
sudo apt-get install ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

# Add the repository to Apt sources:
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
  $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update

2. Install the Docker packages.

3. Verify that the Docker Engine installation is successful by running the hello-world image.

sudo docker run hello-world

This command downloads a test image and runs it in a container. When the container runs, it prints a confirmation message and exits.

Installing the latest Docker release [One Liner]

If you are trying out Docker or using it for test purposes, you can use a one-liner command to install Docker.

To install the latest docker release just execute the following curl command.

curl -sSL https://get.docker.com/ | sudo sh

Run Docker Commands Without Sudo

If you try to run Docker without sudo as a normal user, you will get the following error.

Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock:

To docker commands without using sudo, you need to add the system user to the Docker group.

Here is the command to add the current user to the docker group.

sudo usermod -aG docker $USER
newgrp docker

Exit the current terminal and log in again to use the docker commands without sudo.

Working with Docker on Ubuntu

To test the docker installation on Ubuntu, let’s go through some of the basic docker operations.

Pull a Docker Base Image

Now we have docker installed on the host. Let’s try to pull the latest ubuntu base image from the docker hub using the following docker command.

sudo docker pull ubuntu:latest

Verify the downloaded image using the following docker command. It will list all the downloaded and created images in the local repository.

sudo docker images

To check if Docker is enabled on boot, execute the following command.

sudo systemctl list-unit-files --type=service | grep docker.service

Run a Docker Container

Now we have the latest ubuntu docker image in our host. Let’s create an interactive container named “test” from the ubuntu image using the following command.

sudo docker run -i -t --name test ubuntu:latest

The above command will create an interactive container with a bash shell. You can try installing some packages in that container.

Type “exit” to exit the container. When you exit, the container will stop. You can view all the stopped and running containers using the following command.

sudo docker ps -a

Uninstallaing Docker from Ubuntu

To uninstall Docker, execute the following command.

sudo apt-get purge docker-ce docker-ce-cli containerd.io

Important Docker Configurations on Ubuntu

• To check all the important configurations of Docker on host, execute the following command.

docker info

• All the containers are stored in the following location

/var/lib/docker/containers

• If you want to add custom DNS servers for Docker, create the following file.

vi /etc/docker/daemon.json

Add the DNS servers in the following format.

}
  "dns": ["8.8.8.8", "8.8.4.4"]
}

Enable Docker Remote API Access

By default, the docker client communicates with the docker daemon using the Unix socket docker.sock. If you want to accept connections from a remote docker client, you need to start the docker daemon on a specific port.

Follow the instructions given below to start the docker daemon on a particular port.

Step 1: Stop the docker service using the following command.

sudo service docker stop

Step 2: Replace the following line in /lib/systemd/system/docker.service file to start the docker daemon on the port 5000 for client connections from all IP addresses. If you to establish a connection from a particular host, then you can mention that IP address instead of "0.0.0.0"

ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:5000 -H unix:///var/run/docker.sock

Step 3: Reload and restart docker.

sudo systemctl daemon-reload
sudo service docker restart

Now you will be able to run docker commands from the host as well as from a remote docker client. To run the docker commands from a remote client, use the following syntax.

docker -H tcp://<ip-address-of-host-running-docker-daemon>:5000 <docker-command>

Conclusion