The Titanic dataset on Kaggle is a classic machine-learning playground. My notebook largely follows Wissam S.’s excellent step-by-step XGBoost tutorial (Kaggle link).

This time, I leveraged AI tools for a bit of fun coding, quickly adding a few useful features:

• ROC/AUC analysis and threshold tuning for probability-based evaluation
• Saving and loading the trained model for reuse
• A browser-based CSV EDA tool, inspired by AWS Data Wrangler and DataBrew, that generates instant dataset insights — what took me weeks to code manually a year ago can now be done in minutes.

Full source code is available here: GitHub repository.

ROC Curve and Threshold Optimization

Instead of using the default threshold of 0.5, I can evaluate model probabilities using ROC curves and select a better threshold with Youden’s J statistic:

from sklearn.metrics import roc_curve, roc_auc_score
import numpy as np

fpr, tpr, thresholds = roc_curve(y_train, oof_probs)
auc_score = roc_auc_score(y_train, oof_probs)
best_threshold = thresholds[np.argmax(tpr - fpr)]

print("ROC AUC:", auc_score)
print("Best threshold:", best_threshold)

Typical Titanic XGBoost models achieve AUC ≈ 0.85–0.88, with an optimal threshold slightly below 0.5.

Saving and Loading the Model

To avoid retraining, the model can be saved and loaded easily:

import joblib

# Save model
joblib.dump(xgb_model, "titanic_xgb_model.pkl")

# Load model later
xgb_model = joblib.load("titanic_xgb_model.pkl")

Quick Browser-Based CSV EDA Tool (Inspired by AWS Data Wrangler/DataBrew)

Using AI assistants, I built a lightweight browser-based EDA tool in just a few minutes — a task that previously took weeks. The design and features are inspired by AWS Data Wrangler/DataBrew, but fully client-side and server-free.

see the eda_implementation_plan.md for the plan

These observations guide feature engineering and modeling decisions, just like AWS DataBrew would, but in a lightweight, fully local environment.

Takeaways

• AI-assisted coding makes development fun and fast.
• EDA inspired by AWS Data Wrangler/DataBrew accelerates dataset understanding.
• Quick ROC/AUC and threshold tuning improves probability evaluation.
• Saving/loading models streamlines experiments and reuse.

Even small enhancements show how AI can speed up development and exploration, letting you focus on learning and experimentation rather than boilerplate code.

Happy Coding!

How to utilize a no-code or low-code environment to create machine learning models without needing to write code

Tools for training model

There are several options for creating and training machine learning models. While Jupiter Notebook is primarily an interactive coding environment, AWS Sagemaker autopilot can automate the process for users who may have some coding knowledge but want to simplify the modeling process. It automatically prepossesses data, selects algorithms, and tunes hyperparameters.

Now AWS Sagemaker provides a more advanced platform — canvas, which allows users to build, train, and deploy machine learning models through a no-code or low-code environment. In this article, I will give a step-by-step example of how to utilize a no-code or low-code environment to play kaggle titanic competition without needing to write code.

canvas

environment

Download the titanic data from https://www.kaggle.com/competitions/titanic/data , there is train and test data.

Open AWS Sagemaker canvas workspace (assume you already have AWS account)

Charge: free tier(first two months) or charge ~1.6$ per hour

Import dataset

a quick look of dataset

Model configuration

create model, titanic is a predictive — binary task

select survived as target column

configure model

Object metric => “Accuracy”

Train method and algorithm: choose XGBoost, Linear Models,Random Forest, CatBoost

Data split => “80/20”

Max candidate and runtime => 1 hour (to save cost)

Build

choose standard build

It can’t configure the instance type when training model, if the dataset is big(e.g. train ~5GB data), I got this bill. (use m5.12xlarge type)

For titanic model training, the bill is rather small.

Canvas automatically selects appropriate instance types, such as ml.m5.12xlarge, ml.c5.18xlarge, and ml.m5.4xlarge, based on the dataset size, performance, and availability. Refer to the SageMaker instance pricing page for more information.

model is ready after ~20 minutes, we can see some model information

and

and metrics

Predict

Now we will try to predict on titanic test data.

Import titanic test data, then click predict, select the titanic-test.csv data

download the predict result(saved to aws-sagemaker-predict.csv)

convert to kaggle format (notebook or any IDE)

import pandas as pd 
prediction = pd.read_csv('./aws-sagemaker-predict.csv')
test_data = pd.read_csv('./test.csv')
output = pd.DataFrame({'PassengerId': test_data.PassengerId, 'Survived': prediction.Survived})
output.to_csv("aws_sagemaker_submission.csv",index=False)

login to kaggle and submit the result

It is not bad vs my previous submission(0.79186)

another highly optimized one is 0.79904 (https://www.kaggle.com/code/wissams/titanic-competition-step-by-step-using-xgboost)

canvas status

we can check canvas status, e.g what processing jobs it has run.

Multiple models

let’s create a second model

select hyperparameter optimization

set version 2

model status

model leaderboard — select the best one

and compare the model version

Deploy model

it can select the deployment instance type. e.g. ml.m5.xlarge4 has 16 GiB, and charge $0.23 per hour. it is included in free-tier sagemaker.

deployment url

https://runtime.sagemaker.ap-southeast-1.amazonaws.com/endpoints/canvas-m5xlarge/invocations

test client

Sagemaker canvas provides a UI to test the endpoint, it can also use API gateway and PythonSDK to access the model deployment endpoint.

• Invoke models for real-time inference
• Creating a machine learning-powered REST API with Amazon API Gateway mapping templates and Amazon SageMaker | Amazon Web Services

Reference

Build and evaluate machine learning models with advanced configurations using the SageMaker Canvas model leaderboard | Amazon Web Services

Happy coding!

Everyone is talking about API first strategy nowadays, In this article I will demo how we exactly implement API first strategy with OpenAPI generator plugin in a real-world solution.

Strategy

API-first, also called the API-first approach, prioritizes APIs at the beginning of the software development process, positioning APIs as the building blocks of software. API-first organizations develop APIs before writing other code, instead of treating them as afterthoughts. This lets teams construct applications with internal and external services that are delivered through APIs.

Advantages of the API-first approach with a common OpenAPI Specification:

• Ensures the use of the same contracts by all the parts of the system.
• Facilitates cooperation between the backend and the frontend, facilitating parallel development.
• Automates generation of repeatable elements like models, services and controllers, reducing the amount of boilerplate code, organizing the structure and saving developers’ time.
• Automatically generates and enables API documentation.

Diagram

As seen from this picture, before developing the application, at the API design stage, we will create OpenAPI spec(e.g. a yaml file) and store it as a single source in the git repo. At this stage, the API designer, such as architect, BA, developer, they need to contribute to this API spec document and do a proper review, then push it to git repo and trigger the CICD process.

CICD will validate the API spec, if the spec is incompatible with the previous version then we need to create a newer version and commit again. otherwise, the spec will be uploaded to an artifact store such as Jfrog Artifactory or Nexus. In this demo, I don’t have artifactory, hence I use bitbucket as the storage instead.

At the microservice development stage, developer will use the openapi generator plugin(maven/gradle), to import the API spec from remote url(e.g. bitbucket or jfrog artifactort), and generate server/client code from the API spec.

We can also customize the openapi generator plugin, parse the api spec configuration to export API dependency information. such as:

• microservice A has dependency of microservice B v1 api, microservice C v2 api
• microservice B has dependency of microservice C v3 api

with these information we can create application catalog, inventory, … etc.

Generate code with the OpenAPI generator plugin

When a microservice wants to create an API server/client, just needs to have this configuration in its maven POM.xml:

<build>        
    <plugins>        
        <plugin>
            <groupId>org.openapitools</groupId>
            <artifactId>openapi-generator-maven-plugin</artifactId>
            <version>${openapi-generator.version}</version>
            <executions>
                <execution>
                    <id>petstore-client</id>
                    <goals>
                        <goal>generate</goal>
                    </goals>
                    <configuration>
                        <auth></auth>
                        <!--                            <inputSpec>${project.basedir}/src/main/resources/api.yaml</inputSpec>-->
                        <inputSpec>https://raw.githubusercontent.com/softarts/apifirst/main/apifirst-contracts/retail/petstore/v1.yaml</inputSpec>
                        <generatorName>java</generatorName>
                        <skipIfSpecIsUnchanged>true</skipIfSpecIsUnchanged>
                        <generateApiTests>false</generateApiTests>
                        <generateModelTests>false</generateModelTests>
<!--                            <generateSupportingFiles>false</generateSupportingFiles>-->
                        <configOptions>
<!--                                <sourceFolder>src/gen/java/main</sourceFolder>-->
                            <interfaceOnly>true</interfaceOnly>
                            <dateLibrary>java8</dateLibrary>
                            <delegatePattern>true</delegatePattern>
                            <useTags>true</useTags>
                            <requestMappingMode>api_interface</requestMappingMode>
                            <library>resttemplate</library>
                            <generateClientAsBean>true</generateClientAsBean>
                            <useSpringBoot3>true></useSpringBoot3>
                            <useJakartaEe>true</useJakartaEe>
                            <apiPackage>com.zhourui.retail.petstore.consumer.api.v1</apiPackage>
                            <modelPackage>com.zhourui.retail.petstore.model.v1</modelPackage>
                            <serializableModel>true</serializableModel>
                        </configOptions>
                    </configuration>
                </execution>
                <execution>
                    <id>petstore-server</id>
                    <goals>
                        <goal>generate</goal>
                    </goals>
                    <configuration>
                        <!--                            <inputSpec>${project.basedir}/src/main/resources/api.yaml</inputSpec>-->
                        <auth></auth>
                        <inputSpec>https://raw.githubusercontent.com/softarts/apifirst/main/apifirst-contracts/retail/petstore/v1.yaml</inputSpec>
                        <generatorName>spring</generatorName>
                        <skipIfSpecIsUnchanged>true</skipIfSpecIsUnchanged>
                        <configOptions>
                            <!--                                <sourceFolder>src/gen/java/main</sourceFolder>-->
                            <validateSpec>false</validateSpec>
                            <interfaceOnly>true</interfaceOnly>
                            <dateLibrary>java8</dateLibrary>
                            <delegatePattern>false</delegatePattern>
                            <useTags>true</useTags>
                            <requestMappingMode>api_interface</requestMappingMode>
                            <generateClientAsBean>true</generateClientAsBean>
                            <useSpringBoot3>true></useSpringBoot3>
                            <useJakartaEe>true</useJakartaEe>
                            <apiPackage>com.zhourui.retail.petstore.producer.api.v1</apiPackage>
                            <modelPackage>com.zhourui.retail.petstore.model.v1</modelPackage>
                        </configOptions>
                    </configuration>
                </execution>
            </executions>
        </plugin>
    </plugins>
</build>

The full parameters list are at the openapi repo. here I just highlight some important stuff:

API spec

it use github repo to store the API spec(see here), and this is the single source of API spec. all participants need to create their code from this spec.

<inputSpec>https://raw.githubusercontent.com/softarts/apifirst/main/apifirst-contracts/retail/petstore/v1.yaml</inputSpec>

<auth>

This is the auth header when need to access API spec file which resides at remote url(e.g. different bitbucket repo). First of all, create a personal token at bitbucket account management; then encode it in base64.

bitbucket_username:bitbucket_token
=> base64 encode
Yml0YnVja2V0X3VzZXJuYW1lOmJpdGJ1Y2tldF90b2tlbg==

the <auth> will be:

<auth>Authorization:Basic Yml0YnVja2V0X3VzZXJuYW1lOmJpdGJ1Y2tldF90b2tlbg==</auth>

multi languages

openapi generator supports multiple languages, usually I will use

• java(generator) resttemplate(library) => java-client, use resttemplate as http client
• kotlin-spring(generator) => kotlin-spring server

Implement service with few lines of code

OpenAPI generator maven plugin will generate model, consumer, producer API files under the target folder.

controller(server)

Just implements from the generated API(StoreApi ) and copy the function declaration:

import com.zhourui.retail.petstore.producer.api.v1.StoreApi;

@RestController
public class StoreController implements StoreApi {
    @Override
    public ResponseEntity<Map<String, Integer>> getInventory() {
        // your code
        Map<String, Integer> m = Map.of(
                "sugar", 1000,
                "wheat", 250
        );
        return new ResponseEntity<>(m, HttpStatus.OK);
    }
}

client

use bean configuration to set server url

@Configuration
@Import(
        value = {StoreApi.class, com.zhourui.retail.petstore.consumer.api.ApiClient.class}
)
public class BaseConfiguration {
    @Value("${ApiFirstDemo.API.PetStoreUrl}")
    private String petStoreUrl;

    @Bean
    public RestTemplate restTemplate() {
        return new RestTemplate();
    }

    @Bean
    public StoreApi storeApi() {
        StoreApi storeApi = new StoreApi();
        storeApi.getApiClient().setBasePath(petStoreUrl);
        return storeApi;
    }
}

the URL is in application.yaml (use default value, in production it can be rendered by CICD)

ApiFirstDemo:
  API:
    PetStoreUrl: ${PetStoreUrlV1:http://localhost:8080/api/v3}

and usage

@RestController
public class DemoController {
    @Autowired
    private StoreApi storeApi;

    @GetMapping(value = "/clientdemo")
    public ResponseEntity<Map<String, Integer>> clientDemo() {
        Map<String, Integer> m = storeApi.getInventory();
        return new ResponseEntity<>(m,OK);
    }
}

Testing

curl -X GET "http://localhost:8080/clientdemo"
# => will hit its own endpoint "storeapi"
{"wheat":250,"sugar":1000}

Reference

• https://www.j-labs.pl/en/tech-blog/api-first-with-open-api-generator/
• https://www.baeldung.com/spring-boot-openapi-api-first-development
• source code is at github

Happy Coding!

Implement API first strategy with OpenAPI generator plugin was originally published in Javarevisited on Medium, where people are continuing the conversation by highlighting and responding to this story.

In this article, I will give an example of how to use Spring OAuth2 client to integrate with social login with just a few lines of kotlin code and configuration.

See the below diagram, Google/Facebook account service is the Authorization Server. When they receive Auth Code request from a client application, they will redirect them to login page, end user will complete the social login on the page and will be redirected back to the original login page.

Setup Google OAuth2 Client

First of all, goto https://console.cloud.google.com/apis/dashboard?pli=1, we need to create new project, e.g. spring-oauth2-client

then we add Credentials and OAuth consent screen

Remember that We also have to configure an authorized redirect URI in the Google Console, which is the path that users will be redirected to after they successfully log in with Google.

By default, Spring Boot configures this redirect URI as /login/oauth2/code/{registrationId}.

So, for Google we’ll add this URI:

http://localhost:8081/login/oauth2/code/google

Spring Boot Configuration

dependency in build.gradle

implementation("org.springframework.boot:spring-boot-starter")
implementation("org.springframework.boot:spring-boot-starter-web")
implementation("org.springframework.boot:spring-boot-starter-security")
implementation("org.springframework.boot:spring-boot-starter-oauth2-client")

See org/springframework/security/config/oauth2/client/CommonOAuth2Provider.java, Spring-boot-starter-oauth2-client has built-in support for OAuth2 client Google, Github, Facebook and Okta.

public enum CommonOAuth2Provider {

 GOOGLE {

  @Override
  public Builder getBuilder(String registrationId) {
   ClientRegistration.Builder builder = getBuilder(registrationId,
     ClientAuthenticationMethod.CLIENT_SECRET_BASIC, DEFAULT_REDIRECT_URL);
   builder.scope("openid", "profile", "email");
   builder.authorizationUri("https://accounts.google.com/o/oauth2/v2/auth");
   builder.tokenUri("https://www.googleapis.com/oauth2/v4/token");
   builder.jwkSetUri("https://www.googleapis.com/oauth2/v3/certs");
   builder.issuerUri("https://accounts.google.com");
   builder.userInfoUri("https://www.googleapis.com/oauth2/v3/userinfo");
   builder.userNameAttributeName(IdTokenClaimNames.SUB);
   builder.clientName("Google");
   return builder;
  }

 }
// xxxx

application.yaml

spring:
  security:
    oauth2:
      client:
        registration:          
          google:
            client-id: your_client_id
            client-secret: your_client_secret

          mykeycloak:
            client-id: login-app
            authorization-grant-type: authorization_code
            scope: openid
        provider:
          mykeycloak:
            issuer-uri: http://localhost:8080/auth/realms/SpringBootKeycloak
            user-name-attribute: preferred_username

Add the above configurations under oauth2.client.registration in spring boot profile will enable the Oauth2ClientAutoConfiguration and create the necessary beans. We can also add Facebook or our customized client mykeycloak

to setup keycloak authorization server, refer to this https://medium.com/@wirelesser/oauth2-write-a-resource-server-with-keycloak-and-spring-security-c447bbca363c

Testing

Spring OAuth2 client will automatically set /login as the default login page, all other requests need to be authenticated.

Start the application at 8081 port, and input http://localhost:8081/ in the browser, it will be routed to the default login page because it is not authenticated yet.

=>

after login successfully it will be routed to the default home page /

logout is

Start independently

Note: It must start keycloak server first because spring boot oauth2 client will try to fetch oauth2 client configuration information from keycloak server.

Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'clientRegistrationRepository' defined in class path resource [org/springframework/boot/autoconfigure/security/oauth2/client/servlet/OAuth2ClientRegistrationRepositoryConfiguration.class]: Failed to instantiate [org.springframework.security.oauth2.client.registration.InMemoryClientRegistrationRepository]: Factory method 'clientRegistrationRepository' threw exception with message: Unable to resolve Configuration with the provided Issuer of "http://localhost:8080/auth/realms/SpringBootKeycloak"

If we need to start it independently, then we can supply the jwk-set-uri property instead to point to the authorization server’s endpoint exposing public keys:

spring:
  security:
    oauth2:
      client:
        registration:
          google:
            client-id: 1043125932604-sq96atrmiserm85fds8apsimbpgg40dv.apps.googleusercontent.com
            client-secret: your_google_secret

          facebook:
            client-id: your_facebook_id
            client-secret: your_facebook_secret

          mykeycloak:
            client-id: login-app
            authorization-grant-type: authorization_code
            scope: openid
            redirectUri: http://localhost:8081/login/oauth2/code/mykeycloak

        provider:
          mykeycloak:  
            user-name-attribute: preferred_username
            jwk-set-uri: http://localhost:8080/auth/realms/SpringBootKeycloak/protocol/openid-connect/certs
            authorization-uri: http://localhost:8080/auth/realms/SpringBootKeycloak/protocol/openid-connect/auth
            token-uri: http://localhost:8080/auth/realms/SpringBootKeycloak/protocol/openid-connect/token
            user-info-uri: http://localhost:8080/auth/realms/SpringBootKeycloak/protocol/openid-connect/userinfo

Reference

• https://www.baeldung.com/spring-security-5-oauth2-login

How-to: Authenticate using Social Login

• source code at github

Happy Coding!

Spring OAuth2 client— integrate with social login was originally published in Javarevisited on Medium, where people are continuing the conversation by highlighting and responding to this story.

Nowadays in a Cloud-native application system, a microservice is a kind of resource server that needs to be protected. In this article, I will give an example resource server protected by Spring Security 6(Spring Boot 3) and Authorization Server(KeyCloak)

Authorization Server

There was an Authorization Server class in Spring Security OAuth2, but now it is deprecated.

https://stackoverflow.com/questions/59273338/what-is-the-replacement-for-the-deprecated-authorizationserver-in-spring-securit

Then they announced a new Authorization Server at https://github.com/spring-projects-experimental/spring-authorization-server

Announcing the Spring Authorization Server

If you want to write your own Authorization Server, ensure you use the correct package and dependency.

KeyCloak

We can also leverage some open-source projects, such as Keycloak and Forgerock to run as authorization server. For more details refer to here. Both of them provide similar IAM features, such as authentication, authorization, single sign-on (SSO), social media logins, multi-factor authentication (MFA), and user self-service. However, the specific implementations and capabilities may differ between the two solutions. ForgeRock is known for its comprehensive and enterprise-ready IAM platform with extensive customization and integration options, while KeyCloak is known for its simplicity and ease of use, making it popular for smaller-scale deployments and developer-friendly use cases. Keycloak has an active community of users and contributors but does not offer commercial support options.

In this case, we use keycloak as an authorization server

docker-compose.yaml file

version: '3.8'
services:
  keycloak:
    image: jboss/keycloak
    restart: always
    environment:
      KEYCLOAK_USER: admin
      KEYCLOAK_PASSWORD: admin
    ports:
      - "8080:8080"
    networks:
      backend:
        aliases:
          - "keycloak"
    volumes:
      #- keycloak_data:/data
      - keycloak_standalone_data:/opt/jboss/keycloak/standalone/data/
      - keycloak_data:/opt/keycloak/data/

volumes:
  db:
    driver: local
  keycloak_data:
  keycloak_standalone_data:


networks:
  backend:
    driver: bridge

volume mapping will store the change in local volume.

start:

docker-compose up

Configuration

http://localhost:8080/auth/

click administration console and username/password is admin/admin

• Add Realm

test the realm

curl http://localhost:8080/auth/realms/SpringBootKeycloak
{"realm":"SpringBootKeycloak","public_key":"MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAqbGRyZ+LkEykTBzKzqCwLxYYlkzipAGfGzlm58L2rhAlMBgBYakwHyZGMXhzA3mNO6oSlKb2NziWC5pHjFwyrt5rOIxPJK5M0zGSORPVfeQ7okOjX0ja6ozSjXXdLr8Zg3Xo/TRlYS5IzZCtyX/jmtR92wRwSTJnWhFcaenqs22A0Cle94/mK6nyMJvLkb/YWdpeeEQZekFVQ7QYhKmAWD91YkVQxh+tLkapAHh9yTvBI3fRfpzYTJKFpV5wBBUy6qbBksHWBw8EHdaOrJZ6QXcnChCNb/VNLyYtKoclS/JmW5tkLj2slPy4CA20P2qY87qykzuq+8BSC/5FDv33FwIDAQAB","token-service":"http://localhost:8080/auth/realms/SpringBootKeycloak/protocol/openid-connect","account-service":"http://localhost:8080/auth/realms/SpringBootKeycloak/account","tokens-not-before":0}

switch to the newly created realm

• Create Client

will show =>

• Create user user1

=>

set password to pass1

remove required user action update password

• Create Role

I created a role name “user”

assign the role

• get configuration

http://localhost:8080/auth/realms/SpringBootKeycloak/.well-known/openid-configuration

in this example we mainly use the token_endpoint

Resource Server

I use SpringBoot to implement a Resource Server application.

Security Configuration

The KeyCloak adapter is not used in Spring boot 3, see this =>

Use Keycloak Spring Adapter with Spring Boot 3

and this =>

Update on deprecation of Keycloak adapters

enable OAuth2 resource server in SecurityConfig file:

http.oauth2ResourceServer {
    oauth2: OAuth2ResourceServerConfigurer<HttpSecurity?> ->
    oauth2.jwt(
        Customizer { jwt ->
            jwt.jwtAuthenticationConverter(
                keycloakJwtTokenConverter // use custom converter
            )
        })
}

add dependency to the project:

implementation("org.springframework.boot:spring-boot-starter")
implementation("org.springframework.boot:spring-boot-starter-web")
implementation("org.springframework.boot:spring-boot-starter-security")
implementation("org.springframework.boot:spring-boot-starter-oauth2-resource-server")
testImplementation("org.springframework.boot:spring-boot-starter-test")

KeyCloak converter

KeyCloak stores the role information in this structure. We need a KeyCloak converter to extract the roles information from JWTheader:

code

override fun convert(@NonNull jwt: Jwt): JwtAuthenticationToken {
    val accesses = Optional.of<Jwt>(jwt)
        .map { token: Jwt -> token.getClaimAsMap(KEYCLOAK_RESOURCE_ACCESS) } // return Map<String, Any>
        .map { claimMap: Map<String, Any> -> claimMap[RESOURCE_ID] as Map<String?, Any?>? } // return Map<String?, Any?>?
        .map { resourceData: Map<String?, Any?>? -> resourceData!![KEYCLOAK_ROLES] as Collection<String?>? } // return  Collection<String?>?
        // it is arrayListOf("manage-account","manage-account-links","view-profile")
        .stream().flatMap { x->x?.stream() } // flatMap the child:ArrayList<String>
        .map { role ->
            SimpleGrantedAuthority(
                KEYCLOAK_ROLE_PREFIX + role
            )
        }

    // extract realm_access as well
    val realm = Optional.of<Jwt>(jwt)
        .map<Map<String?, Any?>> { token: Jwt -> token.getClaimAsMap( KEYCLOAK_REALM_ACCESS) as Map<String?, Any?>?}
        .map<Collection<String?>?> { resourceData: Map<String?, Any?>? -> resourceData!![KEYCLOAK_ROLES] as Collection<String?>? } // arrayListOf("manage-account","manage-account-links","view-profile")
        .stream().flatMap { x->x.stream() }
        .map { role ->
            SimpleGrantedAuthority(
                KEYCLOAK_ROLE_PREFIX + role
            )
        }

    // concat the 3 streams
    val authorities = Stream
        .concat(
            Stream.concat(jwtGrantedAuthoritiesConverter.convert(jwt).stream(), accesses),
            realm
        )
        .collect(Collectors.toSet())

    val principalClaimName: String = jwt.getClaimAsString(PRINCIPAL_ATTR) ?:jwt.getClaimAsString(JwtClaimNames.SUB)

    return JwtAuthenticationToken(jwt, authorities, principalClaimName)
}

At the end we should get these, “ROLE_user” is the role we assigned for the client/user in the KeyCloak console. ROLE_ is the prefix and user is the role assigned in the KeyCloak console.

and this is a JWT claims example

// an example
"sub" -> "dcbfb403-75cf-4388-8c6a-97d642964239"
"resource_access" -> {LinkedTreeMap@8450}  size = 1
"email_verified" -> {Boolean@8452} false
"iss" -> "http://localhost:8080/auth/realms/SpringBootKeycloak"
"typ" -> "Bearer"
"preferred_username" -> "user1"
"sid" -> "ff8bfcdd-1d91-4773-9c84-9b0a12dcd5a2"
"aud" -> {ArrayList@8462}  size = 1
"acr" -> "1"
"realm_access" -> {LinkedTreeMap@8466}  size = 1
"azp" -> "login-app"
"scope" -> "email profile"
"exp" -> {Instant@8426} "2024-01-28T15:04:17Z"
"session_state" -> "ff8bfcdd-1d91-4773-9c84-9b0a12dcd5a2"
"iat" -> {Instant@8425} "2024-01-28T14:59:17Z"
"jti" -> "1d73943b-cc38-4a8d-989f-f120caa59ce0"

and in the Security Configuration:

http.oauth2ResourceServer {
    oauth2: OAuth2ResourceServerConfigurer<HttpSecurity?> ->
    oauth2.jwt(
        Customizer { jwt ->
            jwt.jwtAuthenticationConverter(
                keycloakJwtTokenConverter // use custom converter
            )
        })
}

Configured Authorization server url:

spring:
  security:
    oauth2:
      resourceserver:
        jwt:
          issuer-uri: http://localhost:8080/auth/realms/SpringBootKeycloak

If we need to start it independently, then we can supply the jwk-set-uri property instead to point to the authorization server’s endpoint exposing public keys:

jwk-set-uri: http://localhost:8080/auth/realms/SpringBootKeycloak/protocol/openid-connect/certs

Testing

• get access token from the KeyCloak server:

or

curl --request POST --url http://localhost:8080/auth/realms/SpringBootKeycloak/protocol/openid-connect/token?= --header "Content-Type: application/x-www-form-urlencoded" --data client_id=login-app  --data username=user1  --data password=pass1  --data grant_type=password
# return token =>
{
 "access_token": "eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJEQzA3ZTZGSVF5NzVqZFdXOGdNNmM5M1BDNjk5OEpNZVJwS0dfaE1JZU1vIn0.eyJleHAiOjE3MDY5NzAyODAsImlhdCI6MTcwNjk2OTk4MCwianRpIjoiZTlkOGQ2M2QtYTljMy00NTBmLThiMGYtMTI0YmYxMWI3YWYxIiwiaXNzIjoiaHR0cDovL2xvY2FsaG9zdDo4MDgwL2F1dGgvcmVhbG1zL1NwcmluZ0Jvb3RLZXljbG9hayIsImF1ZCI6ImFjY291bnQiLCJzdWIiOiJkY2JmYjQwMy03NWNmLTQzODgtOGM2YS05N2Q2NDI5NjQyMzkiLCJ0eXAiOiJCZWFyZXIiLCJhenAiOiJsb2dpbi1hcHAiLCJzZXNzaW9uX3N0YXRlIjoiM2RmYjEzNDEtNzJlMS00ZTkzLWIyM2MtYWRkYWNjZDAwM2FlIiwiYWNyIjoiMSIsInJlYWxtX2FjY2VzcyI6eyJyb2xlcyI6WyJvZmZsaW5lX2FjY2VzcyIsImRlZmF1bHQtcm9sZXMtc3ByaW5nYm9vdGtleWNsb2FrIiwidW1hX2F1dGhvcml6YXRpb24iLCJ1c2VyIl19LCJyZXNvdXJjZV9hY2Nlc3MiOnsiYWNjb3VudCI6eyJyb2xlcyI6WyJtYW5hZ2UtYWNjb3VudCIsIm1hbmFnZS1hY2NvdW50LWxpbmtzIiwidmlldy1wcm9maWxlIl19fSwic2NvcGUiOiJlbWFpbCBwcm9maWxlIiwic2lkIjoiM2RmYjEzNDEtNzJlMS00ZTkzLWIyM2MtYWRkYWNjZDAwM2FlIiwiZW1haWxfdmVyaWZpZWQiOmZhbHNlLCJwcmVmZXJyZWRfdXNlcm5hbWUiOiJ1c2VyMSJ9.OT3do4T0vJo70oGK42Cw8TLej6y_wlCV2VsdlZlKsIZTw0ql5WGigq0YktMP80pUBc14dXbsd10g7PPGUoVqLDTZk-lCaskGsETjUwihyHU1pUbQE9-OARkUfeyiG__JDXOwe-KepyHL9ZKZ_8UL9VYVvm63PtWQfj3_CTW3jfgpynMSa3mnB8QBhDv23fW-VLvXpK7Ab1EcPaYW75J2pFtYb7W7dx73SCLKzEr7cxPJyEpb9D1iz0XuKW2F8l-0LwrIWyGvEaf81bQ9dg-RC_CmhjcGvDHDcP1JFG5G9ad-cZDlgBgDTM2wM8Pw8uR7ZwPnn5fJ9bFq3Iz9zcd_sQ",
 "expires_in": 299,
 "refresh_expires_in": 1799,
 "refresh_token": "eyJhbGciOiJIUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICIzMzM0YTg3YS0xZTRiLTQxOTQtYTVhZi00NjRmMTNhZDBhOGUifQ.eyJleHAiOjE3MDY5NzE3ODAsImlhdCI6MTcwNjk2OTk4MCwianRpIjoiMzhiZGJjZTgtMTEzOC00MDVlLWEyMzUtOGY0YTM0NmZiNGNmIiwiaXNzIjoiaHR0cDovL2xvY2FsaG9zdDo4MDgwL2F1dGgvcmVhbG1zL1NwcmluZ0Jvb3RLZXljbG9hayIsImF1ZCI6Imh0dHA6Ly9sb2NhbGhvc3Q6ODA4MC9hdXRoL3JlYWxtcy9TcHJpbmdCb290S2V5Y2xvYWsiLCJzdWIiOiJkY2JmYjQwMy03NWNmLTQzODgtOGM2YS05N2Q2NDI5NjQyMzkiLCJ0eXAiOiJSZWZyZXNoIiwiYXpwIjoibG9naW4tYXBwIiwic2Vzc2lvbl9zdGF0ZSI6IjNkZmIxMzQxLTcyZTEtNGU5My1iMjNjLWFkZGFjY2QwMDNhZSIsInNjb3BlIjoiZW1haWwgcHJvZmlsZSIsInNpZCI6IjNkZmIxMzQxLTcyZTEtNGU5My1iMjNjLWFkZGFjY2QwMDNhZSJ9.wkYEhoi9bSlTT6HQ4R8eIOoj8L2BJc1f4oKHWAXkt8E",
 "token_type": "Bearer",
 "not-before-policy": 0,
 "session_state": "3dfb1341-72e1-4e93-b23c-addaccd003ae",
 "scope": "email profile"
}

• access the public endpoint without any token

curl http://localhost:8081/public/api1
# =>
public api1

• accesse private endpoint with token

# put your actual token return from keycloak
curl --request GET --url http://localhost:8081/private/api1 --header "Authorization: Bearer you_token_from_keycloak"
# =>
return private api1

• accesse the private endpoint without token

curl -i http://localhost:8081/private/api1
# => return
HTTP/1.1 401
Set-Cookie: JSESSIONID=DF74D9FAD79621451E27977506C0F0F7; Path=/; HttpOnly
WWW-Authenticate: Bearer
X-Content-Type-Options: nosniff
X-XSS-Protection: 0
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Pragma: no-cache
Expires: 0
X-Frame-Options: DENY
Content-Length: 0
Date: Sat, 03 Feb 2024 14:33:25 GMT

or with insufficient role permission (/private/admin need “admin” role)

>curl -i --request GET --url http://localhost:8081/private/admin --header "Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJEQzA3ZTZGSVF5NzVqZFdXOGdNNmM5M1BDNjk5OEpNZVJwS0dfaE1JZU1vIn0.eyJleHAiOjE3MDY5NzQ2MjEsImlhdCI6MTcwNjk3NDMyMSwianRpIjoiMzliYzIwZDQtOWM1Yi00YjVjLWFkYzctY2RmNTc5MmUxMDRmIiwiaXNzIjoiaHR0cDovL2xvY2FsaG9zdDo4MDgwL2F1dGgvcmVhbG1zL1NwcmluZ0Jvb3RLZXljbG9hayIsImF1ZCI6ImFjY291bnQiLCJzdWIiOiJkY2JmYjQwMy03NWNmLTQzODgtOGM2YS05N2Q2NDI5NjQyMzkiLCJ0eXAiOiJCZWFyZXIiLCJhenAiOiJsb2dpbi1hcHAiLCJzZXNzaW9uX3N0YXRlIjoiNDZkNWRlNmMtMjZiYy00MjI4LWEyZDQtM2MxM2U3NzFiNGIyIiwiYWNyIjoiMSIsInJlYWxtX2FjY2VzcyI6eyJyb2xlcyI6WyJvZmZsaW5lX2FjY2VzcyIsImRlZmF1bHQtcm9sZXMtc3ByaW5nYm9vdGtleWNsb2FrIiwidW1hX2F1dGhvcml6YXRpb24iLCJ1c2VyIl19LCJyZXNvdXJjZV9hY2Nlc3MiOnsiYWNjb3VudCI6eyJyb2xlcyI6WyJtYW5hZ2UtYWNjb3VudCIsIm1hbmFnZS1hY2NvdW50LWxpbmtzIiwidmlldy1wcm9maWxlIl19fSwic2NvcGUiOiJlbWFpbCBwcm9maWxlIiwic2lkIjoiNDZkNWRlNmMtMjZiYy00MjI4LWEyZDQtM2MxM2U3NzFiNGIyIiwiZW1haWxfdmVyaWZpZWQiOmZhbHNlLCJwcmVmZXJyZWRfdXNlcm5hbWUiOiJ1c2VyMSJ9.IrZaC_IqmXxT9GV6mAMw0YXInN6Dge9iRaF7NUiErNt5zKk_7tPGVr5s4Z1RdctSk-Xv7S9XW9JmSpSnttmlxGuE_G3bhED1rCEpuVp5674nOghoq-80gbVmQC4rl9Uk0JIgk6JDFB03uC4Lg6tDuAUYfnJ3QbuPUsVc97Dml-5dPKoVfP7Q-D6NvzvO87EtBm5Io6evZkDE30bccmpBRLUtb4dA5DexCV4xEtFxDwSJmoCBwb8Bs_0CuxT7ZFw1C7QZJvFEbppNMtAf7YiUwVPW_pMNAKxCJP47GZrSvhzz41PCE7tYrUEoXiPKRPp1WRN55de5p8VJE2l8_TMGCA"
# return =>
HTTP/1.1 403
WWW-Authenticate: Bearer error="insufficient_scope", error_description="The request requires higher privileges than provided by the access token.", error_uri="https://tools.ietf.org/html/rfc6750#section-3.1"
X-Content-Type-Options: nosniff
X-XSS-Protection: 0
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Pragma: no-cache
Expires: 0
X-Frame-Options: DENY
Content-Length: 0
Date: Sat, 03 Feb 2024 15:32:30 GMT

Reference

• https://dzone.com/articles/spring-boot-3-keycloak (src code: https://github.com/Pask423/keycloak-springboot/tree/master/base-integration-spring-boot-3)
• https://docs.spring.io/spring-security/reference/servlet/oauth2/index.html
• https://www.baeldung.com/spring-boot-keycloak
• https://www.keycloak.org/2023/03/adapter-deprecation-update
• https://www.keycloak.org/2022/02/adapter-deprecation.html
• https://www.baeldung.com/spring-security-method-security
• https://github.com/softarts/resource-server (source code)

Happy Coding!

There are already lots of excellent commercial APM/Observability Tools can help us to trace the HTTP calls chain among distributed applications. But none of them can understand PlantUml/sequence diagram and draw a nice sequence diagram yet.

Demo

The source code is here.

I prepare a sequence diagram written in plantuml

@startuml
title Gateway flow
participant gateway
participant appserver
participant dataserver
gateway -> appserver: /test1
note right
url: v1/token/generate
header: api-key
param: card-no
end note

appserver -> dataserver: /test1
dataserver -> appserver: /test1
appserver -> gateway: /test1
@enduml

the expected sequence:

Then I started gateway, appserver, dataserver the 3 applications via docker-compose, and Initial a call /test1from insomnia with x-bdd-corrid = 1234abcd in header.

The sequencetracing will monitor the docker logs and draw the actual call sequence in the image.

Happy flow

in appserver

@GetMapping("/test1")
fun test1(): String  {
    val url = "http://data:8083/test1"
    val restTemplate = RestTemplate()
    val responseEntity: ResponseEntity<String> = restTemplate.getForEntity(url, String::class.java)
    val resp: String? = responseEntity.body
    println("test1 resp: $resp")
    return "app test1$resp"
}

everything is hit and returned successfully =>

Broken flow

put some bad stuff in the appserver controller, hence appserver won’t send HTTP call to the dataserver

@GetMapping("/test2")
fun test2(): String  {
    throw RuntimeException("app test2")
}

The sequencetracing application detects the HTTP call appserver->dataserver is missing, so it draws it in RED color; While gateway->appserver call is seen but the HTTP status code is 500 (internal server error because appserver throws an exception), hence it draws it as YELLOW and with an X symbol.

Step by step

• prepare, download the source code, install the docker environment, build the docker image for gateway, appserver, and dataserver

git clone https://github.com/softarts/sequencetracing.git
cd gateway
gradlew build # or gradlew.bat build 
docker build -t gateway .

# Do the same for appserver and dataserver

• start docker container

cd sequencetracing
docker-compose up

• Send HTTP request with postman, insomnia or curl

curl =>

curl -H "x-bdd-corrid:qwert" http://localhost:8080/test2
# 'qwert' is the corrid
# =>
{"timestamp":"2023-12-29T13:15:09.410+00:00","status":500,"error":"Internal Server Error","path":"/test2"}

• Run the sequencetracing application(I put the code in unit test so run through gradle test, and the parameters are passed through system property)

#corrHeader=x_bdd_corrid (the header name we will put in the HTTP request)
#corrid=qwert (the header value we will put in the HTTP request)
#uml=src/test/resources/seqtrace/gateway-test1.puml (input uml file) 
#output=src/test/resources/seqtrace/gateway-test1.png (output image)

cd sequencetracing/sequencetracing

# use the corrid set in the previous step(curl/postman/insomnia)
# /test1
gradlew.bat test --tests SeqTraceTest -DcorrHeader=x_bdd_corrid -Doutput=src/test/resources/seqtrace/gateway-test1.png -Dcorrid=qwert -Duml=src/test/resources/seqtrace/gateway-test1.puml --no-daemon --rerun

# /test2
# -info will print all the system output
gradlew.bat test --tests SeqTraceTest -DcorrHeader=x_bdd_corrid -Doutput=src/test/resources/seqtrace/gateway-test2.png -Dcorrid=qwert -Duml=src/test/resources/seqtrace/gateway-test2.puml --no-daemon -info

=> debug output

net.host.name                            --> appserver
user_agent.original                      --> Java/17.0.7
http.target                              --> /test2
net.sock.peer.addr                       --> 172.21.0.4
thread.name                              --> http-nio-8082-exec-1
http.status_code                         --> 500
net.sock.host.addr                       --> 172.21.0.2
http.route                               --> /test2
net.host.port                            --> 8082
net.protocol.name                        --> http
net.sock.peer.port                       --> 47786
http.method                              --> GET
http.scheme                              --> http
thread.id                                --> 25
net.protocol.version                     --> 1.1
net.sock.host.port                       --> 8082
container appserver is server

correlation output =>

CONTAINER                                          TRACE                                              SPAN                                               RAW
/gateway                                           e5ec972f3eb5d53975877a6d78d6d9b2                   0e4fd362145f56c2                                   [otel.javaagent 2023-12-29 11:06:55:492 +0000] [ht
/gateway                                           e5ec972f3eb5d53975877a6d78d6d9b2                   71e6163c32e369fd                                   [otel.javaagent 2023-12-29 11:06:55:506 +0000] [ht
/gateway                                           e5ec972f3eb5d53975877a6d78d6d9b2                   5054e2e10738ab48                                   [otel.javaagent 2023-12-29 11:06:55:674 +0000] [ht
/gateway                                           e5ec972f3eb5d53975877a6d78d6d9b2                   59bad2f1c117827e                                   [otel.javaagent 2023-12-29 11:06:55:676 +0000] [ht
/appserver                                         e5ec972f3eb5d53975877a6d78d6d9b2                   fa8be9d35d35db86                                   [otel.javaagent 2023-12-29 11:06:55:352 +0000] [ht
/appserver                                         e5ec972f3eb5d53975877a6d78d6d9b2                   f63c534155e0b969                                   [otel.javaagent 2023-12-29 11:06:55:486 +0000] [ht
/appserver                                         e5ec972f3eb5d53975877a6d78d6d9b2                   58bd47e93aec25f7                                   [otel.javaagent 2023-12-29 11:06:55:488 +0000] [ht

URI                                                SRC                                                DST                                                REQ        RESP       STATUS_CODE
http://localhost:8080/test2                                                                           gateway                                            Y          N          500
http://appserver:8082/test2                        gateway                                            appserver                                          Y          Y          500

SRC                                                DST                                                LABEL                                              CONNECTION      HTTP_SUCC
gateway                                            appserver                                          /test2                                             <=>             false
appserver                                          dataserver                                         /test2                                             -/-             false
dataserver                                         appserver                                          /test2                                             -/-             false
appserver                                          gateway                                            /test2                                             <=>             false
# <=> means the call is observed
# -/- means call is NOT seen.

the output image(generated in src/test/resources/seqtrace/gateway-test2.png) =>

How does it work

Open Telemetry

Most of the APM tools, such as Datadog, sumologic, and AppDynamic, are running on top of open-telemetry.

the open-telemetry agent runs within the same JVM space as the application, intercepts all the API calls from the application, and either produces to logs or send back to the Collector.

Use JAVA_TOOL_OPTIONS to load Open Telemetry agent in each JVM application. see their Dockerfiles, also need to configure it to capture the HTTP header x-bdd-corrid

FROM azul/zulu-openjdk:17.0.7-17.42.19-jre

COPY build/libs/*SNAPSHOT.jar /app/app.jar

ENV JAVA_TOOL_OPTIONS="-javaagent:/app/lib/opentelemetry-javaagent.jar"
ENV OTEL_TRACES_EXPORTER=logging
ENV OTEL_METRICS_EXPORTER=logging
ENV OTEL_LOGS_EXPORTER=logging
ENV OTEL_INSTRUMENTATION_HTTP_SERVER_CAPTURE_REQUEST_HEADERS=x-bdd-corrid

EXPOSE 8082

ENTRYPOINT ["java",  \
            "-server", \
            "-cp", "/app:/app/lib/*", \
            "-XX:MaxRAMPercentage=60.0", \
            "-XX:+UnlockDiagnosticVMOptions", \
            "-XX:NativeMemoryTracking=summary", \
            "-XX:+PrintNMTStatistics", \
            "-jar", "/app/app.jar"]

and I prepared a docker-compose.yaml to start all three applications, opentelemetry-javaagent will be mapped to each container.

version: "3"

services:
  gateway:
    image: gateway:latest
    container_name: gateway
    hostname: gateway
    restart: always
    ports:
      - 8080:8080
    volumes:
      - ./lib:/app/lib
  appserver:
    image: appserver:latest
    container_name: appserver
    hostname: appserver
    restart: always
    ports:
      - 8082:8082
    volumes:
      - ./lib:/app/lib      
  dataserver:
    image: dataserver:latest
    container_name: dataserver
    hostname: dataserver
    restart: always
    ports:
      - 8083:8083
    volumes:
      - ./lib:/app/lib

I uploaded an opentelemetry-javaagentto here, you can also download the latest version from its GitHub release page

Search docker container log with regex

The next step is to monitor the docker container logs and parse them. the log fetching part I wrote mainly refers to testcontainers , then I wrote a quick and dirty part myself to parse these logs and correlate the src/dst container and HTTP call.

docker-compose up
...xxx
gateway    | Picked up JAVA_TOOL_OPTIONS: -javaagent:/app/opentelemetry-javaagent.jar
gateway    | OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended
...
gateway    | [otel.javaagent 2023-12-25 14:47:54:467 +0000] [http-nio-8080-exec-1] INFO io.opentelemetry.exporter.logging.LoggingSpanExporter - 'GET /test1' : 42f38e33ab2f2ebb4f2d8715de482e89 ec52189d7f097cab SERVER [tracer: io.opentelemetry.tomcat-10.0:1.32.0-alpha] AttributesMap{data={http.route=/test1, net.sock.host.addr=172.19.0.2, net.protocol.name=http, net.protocol.version=1.1, http.scheme=http, http.method=GET, net.host.port=8080, http.request.header.x_bdd_corrid=[1234abcd], net.host.name=localhost, user_agent.original=insomnia/2023.5.8, http.response_content_length=29, http.target=/test1, http.status_code=200, net.sock.peer.addr=172.19.0.1, net.sock.host.port=8080, thread.id=25, net.sock.peer.port=43320, thread.name=http-nio-8080-exec-1}, capacity=128, totalAddedValues=18}

The last line is what the open telemetry has intercepted when there is an HTTP request. The sequencetracing application just needs to parse these logs and correlate (by trace, span, corr Id) them later on, which is similar to what the APM tools do at the backend.

I created the sequencetracing project based on plantuml intellij plugin , to parse and generate the PlantUml sequence diagram.

See this PR for the change I made.

Happy Coding!

It is just a Proof-of-concept project and NOT robust yet. It is probably unable to handle the case where that async call is involved, e.g. Kafka events.

It is quite a common requirement to ingest the data from the microservice cluster to the big data platform for further analytics. Depending on the data platform architecture, it can ingest data to either the object store(s3) or Hadoop/HDFS

Data Architecture

Hadoop/HDFS based

This tech stack is a bit outdated, but lots of companies are still using it nowadays.

Object Store(S3) based

object store/s3 as data lake is more popular when the data platform is deployed on the public cloud. No matter which deployment, there is not much difference in the data-sink(ingestion to data platform) part. Both of them will use Kafka-Connect to ingest data to data platform. I will give an example based on the Hadoop solution.

Start the environment

I created a docker-compose file run Hadoop and Kafka. refer to

cp-all-in-one/cp-all-in-one-community at 7.5.0-post · confluentinc/cp-all-in-one

and https://github.com/big-data-europe/docker-hadoop

docker-compose.yaml

version: "3"

services:
  namenode:
    image: bde2020/hadoop-namenode:2.0.0-hadoop3.2.1-java8
    container_name: namenode
    restart: always
    ports:
      - 9870:9870
      - 9000:9000
    volumes:
      - hadoop_namenode:/hadoop/dfs/name
    environment:
      - CLUSTER_NAME=test
    env_file:
      - ./hadoop.env

  datanode:
    image: bde2020/hadoop-datanode:2.0.0-hadoop3.2.1-java8
    container_name: datanode
    restart: always
    volumes:
      - hadoop_datanode:/hadoop/dfs/data
    environment:
      SERVICE_PRECONDITION: "namenode:9870"
    env_file:
      - ./hadoop.env
  
  resourcemanager:
    image: bde2020/hadoop-resourcemanager:2.0.0-hadoop3.2.1-java8
    container_name: resourcemanager
    restart: always
    environment:
      SERVICE_PRECONDITION: "namenode:9000 namenode:9870 datanode:9864"
    env_file:
      - ./hadoop.env

  nodemanager1:
    image: bde2020/hadoop-nodemanager:2.0.0-hadoop3.2.1-java8
    container_name: nodemanager
    restart: always
    environment:
      SERVICE_PRECONDITION: "namenode:9000 namenode:9870 datanode:9864 resourcemanager:8088"
    env_file:
      - ./hadoop.env
  
  historyserver:
    image: bde2020/hadoop-historyserver:2.0.0-hadoop3.2.1-java8
    container_name: historyserver
    restart: always
    environment:
      SERVICE_PRECONDITION: "namenode:9000 namenode:9870 datanode:9864 resourcemanager:8088"
    volumes:
      - hadoop_historyserver:/hadoop/yarn/timeline
    env_file:
      - ./hadoop.env
  
  zookeeper:
    image: confluentinc/cp-zookeeper:7.5.0
    hostname: zookeeper
    container_name: zookeeper
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  broker:
    image: confluentinc/cp-server:7.5.0
    hostname: broker
    container_name: broker
    depends_on:
      - zookeeper
    ports:
      - "9092:9092"
      - "9101:9101"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_METRIC_REPORTERS: io.confluent.metrics.reporter.ConfluentMetricsReporter
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_CONFLUENT_LICENSE_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_CONFLUENT_BALANCER_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_JMX_PORT: 9101
      KAFKA_JMX_HOSTNAME: localhost
      KAFKA_CONFLUENT_SCHEMA_REGISTRY_URL: http://schema-registry:8081
      CONFLUENT_METRICS_REPORTER_BOOTSTRAP_SERVERS: broker:29092
      CONFLUENT_METRICS_REPORTER_TOPIC_REPLICAS: 1
      CONFLUENT_METRICS_ENABLE: 'true'
      CONFLUENT_SUPPORT_CUSTOMER_ID: 'anonymous'

  schema-registry:
    image: confluentinc/cp-schema-registry:7.5.0
    hostname: schema-registry
    container_name: schema-registry
    depends_on:
      - broker
    ports:
      - "8081:8081"
    environment:
      SCHEMA_REGISTRY_HOST_NAME: schema-registry
      SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: 'broker:29092'
      SCHEMA_REGISTRY_LISTENERS: http://0.0.0.0:8081  

  connect:
    image: confluentinc/cp-kafka-connect:latest    
    hostname: connect
    container_name: connect
    depends_on:
      - zookeeper
      - broker
      - schema-registry
    ports:
      - 8083:8083
    environment:
      CONNECT_BOOTSTRAP_SERVERS: 'broker:29092'
      CONNECT_REST_ADVERTISED_HOST_NAME: connect
      CONNECT_REST_PORT: 8083

      CONNECT_GROUP_ID: compose-connect-group
      CONNECT_CONFIG_STORAGE_TOPIC: docker-connect-configs
      CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR: 1
      CONNECT_OFFSET_FLUSH_INTERVAL_MS: 10000
      CONNECT_OFFSET_STORAGE_TOPIC: docker-connect-offsets
      CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR: 1
      CONNECT_STATUS_STORAGE_TOPIC: docker-connect-status
      CONNECT_STATUS_STORAGE_REPLICATION_FACTOR: 1


      CONNECT_KEY_CONVERTER: io.confluent.connect.avro.AvroConverter
      CONNECT_KEY_CONVERTER_SCHEMA_REGISTRY_URL: http://schema-registry:8081
      
      CONNECT_VALUE_CONVERTER: io.confluent.connect.avro.AvroConverter
      CONNECT_VALUE_CONVERTER_SCHEMA_REGISTRY_URL: http://schema-registry:8081
      
  
      CLASSPATH: /usr/share/java/monitoring-interceptors/monitoring-interceptors-7.5.0.jar
      CONNECT_PRODUCER_INTERCEPTOR_CLASSES: "io.confluent.monitoring.clients.interceptor.MonitoringProducerInterceptor"
      CONNECT_CONSUMER_INTERCEPTOR_CLASSES: "io.confluent.monitoring.clients.interceptor.MonitoringConsumerInterceptor"

      CONNECT_LOG4J_ROOT_LOGLEVEL: "INFO"
      CONNECT_LOG4J_LOGGERS: "org.apache.kafka.connect.runtime.rest=WARN,org.reflections=ERROR"      
      CONNECT_PLUGIN_PATH: '/usr/share/java,/usr/share/confluent-hub-components/,/connectors/'
    
    command:
      - sh
      - -exc
      - |
        confluent-hub install --no-prompt --component-dir /usr/share/confluent-hub-components/ confluentinc/kafka-connect-hdfs:latest
        confluent-hub install --no-prompt --component-dir /usr/share/confluent-hub-components/ confluentinc/kafka-connect-datagen:latest
        exec /etc/confluent/docker/run

  control-center:
    image: confluentinc/cp-enterprise-control-center:7.5.0
    hostname: control-center
    container_name: control-center
    depends_on:
      - broker
      - schema-registry
      - connect
      - ksqldb-server
    ports:
      - "9021:9021"
    environment:
      CONTROL_CENTER_BOOTSTRAP_SERVERS: 'broker:29092'
      CONTROL_CENTER_CONNECT_CONNECT-DEFAULT_CLUSTER: 'connect:8083'
      CONTROL_CENTER_KSQL_KSQLDB1_URL: "http://ksqldb-server:8088"
      CONTROL_CENTER_KSQL_KSQLDB1_ADVERTISED_URL: "http://localhost:8088"
      CONTROL_CENTER_SCHEMA_REGISTRY_URL: "http://schema-registry:8081"
      CONTROL_CENTER_REPLICATION_FACTOR: 1
      CONTROL_CENTER_INTERNAL_TOPICS_PARTITIONS: 1
      CONTROL_CENTER_MONITORING_INTERCEPTOR_TOPIC_PARTITIONS: 1
      CONFLUENT_METRICS_TOPIC_REPLICATION: 1      
      PORT: 9021
      CONTROL_CENTER_CONNECT_HEALTHCHECK_ENDPOINT: '/connectors'

  ksqldb-server:
    image: confluentinc/cp-ksqldb-server:7.5.0
    hostname: ksqldb-server
    container_name: ksqldb-server
    depends_on:
      - broker
      - connect
    ports:
      - "8088:8088"
    environment:
      KSQL_CONFIG_DIR: "/etc/ksql"
      KSQL_BOOTSTRAP_SERVERS: "broker:29092"
      KSQL_HOST_NAME: ksqldb-server
      KSQL_LISTENERS: "http://0.0.0.0:8088"
      KSQL_CACHE_MAX_BYTES_BUFFERING: 0
      KSQL_KSQL_SCHEMA_REGISTRY_URL: "http://schema-registry:8081"
      KSQL_PRODUCER_INTERCEPTOR_CLASSES: "io.confluent.monitoring.clients.interceptor.MonitoringProducerInterceptor"
      KSQL_CONSUMER_INTERCEPTOR_CLASSES: "io.confluent.monitoring.clients.interceptor.MonitoringConsumerInterceptor"
      KSQL_KSQL_CONNECT_URL: "http://connect:8083"
      KSQL_KSQL_LOGGING_PROCESSING_TOPIC_REPLICATION_FACTOR: 1
      KSQL_KSQL_LOGGING_PROCESSING_TOPIC_AUTO_CREATE: 'true'
      KSQL_KSQL_LOGGING_PROCESSING_STREAM_AUTO_CREATE: 'true'

  ksqldb-cli:
    image: confluentinc/cp-ksqldb-cli:7.5.0
    container_name: ksqldb-cli
    depends_on:
      - broker
      - connect
      - ksqldb-server
    entrypoint: /bin/sh
    tty: true

  ksql-datagen:
    image: confluentinc/ksqldb-examples:7.5.0
    hostname: ksql-datagen
    container_name: ksql-datagen
    depends_on:
      - ksqldb-server
      - broker
      - schema-registry
      - connect
    command: "bash -c 'echo Waiting for Kafka to be ready... && \
                       cub kafka-ready -b broker:29092 1 40 && \
                       echo Waiting for Confluent Schema Registry to be ready... && \
                       cub sr-ready schema-registry 8081 40 && \
                       echo Waiting a few seconds for topic creation to finish... && \
                       sleep 11 && \
                       tail -f /dev/null'"
    environment:
      KSQL_CONFIG_DIR: "/etc/ksql"
      STREAMS_BOOTSTRAP_SERVERS: broker:29092
      STREAMS_SCHEMA_REGISTRY_HOST: schema-registry
      STREAMS_SCHEMA_REGISTRY_PORT: 8081

  rest-proxy:
    image: confluentinc/cp-kafka-rest:7.5.0
    depends_on:
      - broker
      - schema-registry
    ports:
      - 8082:8082
    hostname: rest-proxy
    container_name: rest-proxy
    environment:
      KAFKA_REST_HOST_NAME: rest-proxy
      KAFKA_REST_BOOTSTRAP_SERVERS: 'broker:29092'
      KAFKA_REST_LISTENERS: "http://0.0.0.0:8082"
      KAFKA_REST_SCHEMA_REGISTRY_URL: 'http://schema-registry:8081'
volumes:
  hadoop_namenode:
  hadoop_datanode:
  hadoop_historyserver:

hadoop.env

CORE_CONF_fs_defaultFS=hdfs://namenode:9000
CORE_CONF_hadoop_http_staticuser_user=root
CORE_CONF_hadoop_proxyuser_hue_hosts=*
CORE_CONF_hadoop_proxyuser_hue_groups=*
CORE_CONF_io_compression_codecs=org.apache.hadoop.io.compress.SnappyCodec

HDFS_CONF_dfs_webhdfs_enabled=true
HDFS_CONF_dfs_permissions_enabled=false
HDFS_CONF_dfs_namenode_datanode_registration_ip___hostname___check=false

YARN_CONF_yarn_log___aggregation___enable=true
YARN_CONF_yarn_log_server_url=http://historyserver:8188/applicationhistory/logs/
YARN_CONF_yarn_resourcemanager_recovery_enabled=true
YARN_CONF_yarn_resourcemanager_store_class=org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore
YARN_CONF_yarn_resourcemanager_scheduler_class=org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler
YARN_CONF_yarn_scheduler_capacity_root_default_maximum___allocation___mb=8192
YARN_CONF_yarn_scheduler_capacity_root_default_maximum___allocation___vcores=4
YARN_CONF_yarn_resourcemanager_fs_state___store_uri=/rmstate
YARN_CONF_yarn_resourcemanager_system___metrics___publisher_enabled=true
YARN_CONF_yarn_resourcemanager_hostname=resourcemanager
YARN_CONF_yarn_resourcemanager_address=resourcemanager:8032
YARN_CONF_yarn_resourcemanager_scheduler_address=resourcemanager:8030
YARN_CONF_yarn_resourcemanager_resource__tracker_address=resourcemanager:8031
YARN_CONF_yarn_timeline___service_enabled=true
YARN_CONF_yarn_timeline___service_generic___application___history_enabled=true
YARN_CONF_yarn_timeline___service_hostname=historyserver
YARN_CONF_mapreduce_map_output_compress=true
YARN_CONF_mapred_map_output_compress_codec=org.apache.hadoop.io.compress.SnappyCodec
YARN_CONF_yarn_nodemanager_resource_memory___mb=16384
YARN_CONF_yarn_nodemanager_resource_cpu___vcores=8
YARN_CONF_yarn_nodemanager_disk___health___checker_max___disk___utilization___per___disk___percentage=98.5
YARN_CONF_yarn_nodemanager_remote___app___log___dir=/app-logs
YARN_CONF_yarn_nodemanager_aux___services=mapreduce_shuffle

MAPRED_CONF_mapreduce_framework_name=yarn
MAPRED_CONF_mapred_child_java_opts=-Xmx4096m
MAPRED_CONF_mapreduce_map_memory_mb=4096
MAPRED_CONF_mapreduce_reduce_memory_mb=8192
MAPRED_CONF_mapreduce_map_java_opts=-Xmx3072m
MAPRED_CONF_mapreduce_reduce_java_opts=-Xmx6144m
MAPRED_CONF_yarn_app_mapreduce_am_env=HADOOP_MAPRED_HOME=/opt/hadoop-3.2.1/
MAPRED_CONF_mapreduce_map_env=HADOOP_MAPRED_HOME=/opt/hadoop-3.2.1/
MAPRED_CONF_mapreduce_reduce_env=HADOOP_MAPRED_HOME=/opt/hadoop-3.2.1/

Configuration steps

Create a Kafka topic

open localhost:9021 , Click Add topic

create topic data-sink-hdfs

Add a connector

open localhost:9021 then goto HOME/Connect-Cluster/Connectors

add a new connector

in this case we choose HdfsSinkConnector, and configure the connector as below. “key.converter.schemas.enable” and “value.converter.schemas.enable” are additional property that needs to be added at the bottom of the page

{
  "name": "HdfsSinkConnectorConnector_0",
  "config": {
    "key.converter.schemas.enable": "false",
    "value.converter.schemas.enable": "false",
    "name": "HdfsSinkConnectorConnector_0",
    "connector.class": "io.confluent.connect.hdfs.HdfsSinkConnector",
    "key.converter": "org.apache.kafka.connect.json.JsonConverter",
    "value.converter": "org.apache.kafka.connect.json.JsonConverter",
    "topics": "data-sink-hdfs",
    "hdfs.url": "hdfs://namenode:9000",  # hostname, see docker-compose.yaml
    "flush.size": "1"
  }
}

Test

produce a message

docker attach to namenode then check

docker exec -it namenode bash
root@f3ab6c37360e:/# hadoop fs -ls /
Found 3 items
drwxr-xr-x   - appuser supergroup          0 2023-10-17 14:51 /logs
drwxr-xr-x   - root    supergroup          0 2023-10-16 15:42 /rmstate
drwxr-xr-x   - appuser supergroup          0 2023-10-17 14:51 /topics

# /topics is the default destination
hadoop fs -ls /topics/data-sink-hdfs/partition=0

Found 3 items
-rw-r--r--   3 appuser supergroup       1048 2023-10-17 14:51 /topics/data-sink-hdfs/partition=0/data-sink-hdfs+0+0000000000+0000000000.avro
-rw-r--r--   3 appuser supergroup       1048 2023-10-17 14:51 /topics/data-sink-hdfs/partition=0/data-sink-hdfs+0+0000000001+0000000001.avro
-rw-r--r--   3 appuser supergroup       1046 2023-10-17 15:06 /topics/data-sink-hdfs/partition=0/data-sink-hdfs+0+0000000002+0000000002.avro

# we produce 3 messages

In the next article, I will explore how to save the message in parquet format and use data visualization tools to query on parquet file.

reference

• for hadoop docker, refer to this https://github.com/big-data-europe/docker-hadoop
• for kafka docker yaml, refer to https://github.com/confluentinc/cp-all-in-one/tree/7.5.0-post/cp-all-in-one-community
• All the files I used are here

Happy Coding!

Real-Time Message Ingestion to Big Data Platform was originally published in Better Programming on Medium, where people are continuing the conversation by highlighting and responding to this story.

Problem

A Kubernetes POD is configured with max 1000 MB memory and 1 CPU. Recently high memory usage has been observed(90%+) and results in POD being killed by OOM. In this article, I will explain how I solve this issue.

Tools

The first thing is we need to understand the application’s current memory usage. We can use a range of tools to track the memory and heap usage. Refer to https://www.baeldung.com/java-heap-dump-capture. However, there are some restrictions.

GUI

jconsole and visualvm are powerful tools to monitor memory usage, but in a Kubernetes cluster environment, usually we don’t have permission to connect to remote JVM/JMX port.

Command line tool

In case we can SSH to POD and have JDK tools installed, we can use jstat to print the memory usage.

# pid is 1 in POD, print every 10000 ms
jstat -gc -t 1 10000
S0C    S1C    S0U   S1U      EC          EU           OC          OU        MC        MU     CCSC  CCSU  YGC   YGCT  FGC  FGCT   GCT
512.0  512.0   0.0    32.0  31232.0  18740.0   175104.0    304.1    4864.0  2435.9  512.0   266.9     297    0.591   0      0.000    0.591
512.0  512.0   0.0    64.0  31232.0  19989.0   175104.0    304.1    4864.0  2435.9  512.0   266.9     301    0.591   0      0.000    0.591
512.0  512.0   32.0   0.0   31232.0   4372.6    175104.0    304.1    4864.0  2435.9  512.0   266.9     310    0.603   0      0.000    0.603
512.0  512.0   32.0   0.0   31232.0  13117.7   175104.0    304.1    4864.0  2435.9  512.0   266.9     314    0.609   0      0.000    0.609

and use jcmd to retrieve heap dump:

./jcmd 1 GC.heap_dump /path/to/heapdump-file
# run the below command at local machine, to copy heapdump fiel to local machine
kubectl cp mypod-name:/path/to/heapdump-file ./local-file-name -n namespace 

# or print heap info
./jcmd 1 GC.heap_info

# or print native memory statistic(must enable native memory summary in java command line)
./jcmd 1 VM.native_memory summary

Observability

In Cloud native applications, we often use Spring actuator, Jolokia, micrometer, and Prometheus to produce the application JVM metrics, On the other side we use collectors such as Telegraf, Prometheus and inject metrics into sumologic, datadog, etc.

GC log

we can also add this to the Java application command line to enable GC activities log at runtime.

"-Xlog:gc*=info"
# or mode detail
"-Xlog:gc*=debug"

example =>

2021-10-07T14:13:10.847+0800: [Full GC (Metadata GC Threshold) [PSYoungGen: 384K->0K(93696K)] [ParOldGen: 982K->1360K(1172480K)] 1366K->1360K(1266176K), [Metaspace: 410659K->410659K(456704K)], 0.0418110 secs] [Times: user=0.04 sys=0.00, real=0.04 secs]

Use Code

or we can use the below code to get more pool details. e.g. direct pool CodeCache Compressed Class Space etc. it will print the statistic every 10 seconds. The different memory categories will be explained next.

MemoryMonitor.kt

JVM memory configuration

Dockerfile

FROM azul/zulu-openjdk:17.0.7-17.42.19-jre

ARG DEPENDENCY=build/dependency
COPY --chown=userXX:userXX ${DEPENDENCY}/BOOT-INF/lib /app/lib
COPY --chown=userXX:userXX ${DEPENDENCY}/META-INF /app/META-INF/
COPY --chown=userXX:userXX ${DEPENDENCY}/BOOT-INF/classes /app/

ENTRYPOINT ["java", \
    "-server", \
    "-cp", "/app:/app/lib/*", \
    "-XX:MaxRAMPercentage=60.0", \
    "-XX:+UnlockDiagnosticVMOptions", \
    "-XX:NativeMemoryTracking=summary", \
    "-XX:+PrintNMTStatistics", \
    "com.company.myapp.ApplicationKt", \
    "--spring.config.location=classpath:/application.yaml"]

The above Dockerfile uses a max of 60% RAM(e.g. we configured a total of 1 GB for POD) and enables native memory tracking. which is equal to 600 MB for heap usage, and around the remaining 400 MB for other usage(e.g. non-heap, JVM itself, and OS), look good? actually, 400 MB is far less than enough in nowadays modern cloud-native applications.

memory usage category

This table lists several important memory categories.

Metaspace, formerly known as PermGen (Permanent Generation), is a non-heap memory area that stores class metadata, constant pool information, and method bytecode. It was introduced in Java 8 as a replacement for PermGen, which was removed due to memory management issues.

It is unlimited if not specified. my observation is it can easily hit 200–300MB in spring boot application

Direct Buffer, is also unlimited, used in socket, nio. In general Spring application usually occupies just a small amount of memory(20~40MB), but in some Big data applications such as Spark, it may use a huge amount and cause OOM!

with the above Dockerfile setting, CodeCache and CompressedClassSpace limits are 256 MB and 64 MB respectively, the actual used will be <50 MB together.

Solution

We need to be aware that JVM and OS itself also have memory overhead. I would say it is about 25~30% of total memory. So the total memory consumption :

heap(600 MB) + metaspace(250 MB) + codecache and compressed_class_space(50 MB) + OS&JVM(300 MB,30% of total) = 1200 MB, which exceeds the POD 1 GB limit!

In this case, it is not caused by a memory leak, hence we need to calculate and allocate memory accordingly.

1. increase the POD limit to 2 GB, this is a safer and easiest way
2. lower the MaxRAMPercentage to 40.0%, and/or limit the other non-heap usage, you can tweak the number according to what you observe in the log:

"-XX:MaxRAMPercentage=40.0", \
"-XX:MaxMetaspaceSize=176m", \
"-XX:CompressedClassSpaceSize=32m", \
"-XX:ReservedCodeCacheSize=48m", \
"-XX:MaxDirectMemorySize=24m",

Hence we get:

heap(400 MB) + non-heap(280 MB: metaspace + codecache+compressed class…) => we left 320 MB room for OS/JVM, barely enough :)

One more thing, Thread dump analytics

Recently I encountered another thread leak issue: https://github.com/harness/ff-java-server-sdk/pull/107

If the thread count(from the above code) increases dramatically, we may want to analyze the thread dump.

jstack -l 1 > thread-dump-1.txt
# copy to local machine
# wait until the thread count changes significantly
jstack -l 1 > thread-dump-2.txt

I use this online tool to compare the dump: https://jstack.review/

refer to https://www.baeldung.com/java-analyze-thread-dumps

Happy Coding!

JVM OOM in Kubernetes POD with small memory allocated was originally published in Javarevisited on Medium, where people are continuing the conversation by highlighting and responding to this story.

Buildpacks can transform your application source code into modern container standards images that can run on any cloud without Dockerfile. In this article, I will explain a couple of examples.

Buildpacks provide framework and runtime support for applications. Buildpacks examine your apps to determine all the dependencies it needs and configure them appropriately to run on any cloud.

Example

No need to write any Dockerfile or install any tool, just start the docker environment and build it

gradlew.bat bootBuildImage

> Task :bootBuildImage
Building image 'docker.io/library/template-service:1.0'

 > Pulling builder image 'docker.io/paketobuildpacks/builder:base' ..................................................
 > Pulled builder image 'paketobuildpacks/builder@sha256:17ea21162ba8c7717d3ead3ee3836a368aced7f02f2e59658e52029bd6d149e7'
 > Pulling run image 'docker.io/paketobuildpacks/run:base-cnb' ..................................................
 > Pulled run image 'paketobuildpacks/run@sha256:1af9935d8987fd52b2266d288200c9482d1dd5529860bbf5bc2d248de1cb1a38'
 > Executing lifecycle version v0.16.5
 > Using build cache volume 'pack-cache-c0553dfeb43b.build'

 > Running creator
    [creator]     ===> ANALYZING
    [creator]     Restoring data for SBOM from previous image
    [creator]     ===> DETECTING
    [creator]     6 of 26 buildpacks participating
    [creator]     paketo-buildpacks/ca-certificates   3.6.3
    [creator]     paketo-buildpacks/bellsoft-liberica 10.2.6
    [creator]     paketo-buildpacks/syft              1.32.1
    [creator]     paketo-buildpacks/executable-jar    6.7.4
    [creator]     paketo-buildpacks/dist-zip          5.6.4
    [creator]     paketo-buildpacks/spring-boot       5.26.1
    [creator]     ===> RESTORING
    [creator]     Restoring metadata for "paketo-buildpacks/ca-certificates:helper" from app image
    [creator]     Restoring metadata for "paketo-buildpacks/bellsoft-liberica:helper" from app image
    [creator]     Restoring metadata for "paketo-buildpacks/bellsoft-liberica:java-security-properties" from app image
    [creator]     Restoring metadata for "paketo-buildpacks/bellsoft-liberica:jre" from app image
    [creator]     Restoring metadata for "paketo-buildpacks/syft:syft" from cache
    [creator]     Restoring metadata for "paketo-buildpacks/spring-boot:helper" from app image
    [creator]     Restoring metadata for "paketo-buildpacks/spring-boot:spring-cloud-bindings" from app image
    [creator]     Restoring metadata for "paketo-buildpacks/spring-boot:web-application-type" from app image
    [creator]     Restoring data for "paketo-buildpacks/syft:syft" from cache
    [creator]     Restoring data for SBOM from cache
    [creator]     ===> BUILDING
    [creator]
    [creator]     Paketo Buildpack for CA Certificates 3.6.3
    [creator]       https://github.com/paketo-buildpacks/ca-certificates
    [creator]       Launch Helper: Reusing cached layer
    [creator]
    [creator]     Paketo Buildpack for BellSoft Liberica 10.2.6
    [creator]       https://github.com/paketo-buildpacks/bellsoft-liberica
    [creator]       Build Configuration:
    [creator]         $BP_JVM_JLINK_ARGS           --no-man-pages --no-header-files --strip-debug --compress=1  configure custom link arguments (--output must be omitted)
    [creator]         $BP_JVM_JLINK_ENABLED        false                                                        enables running jlink tool to generate custom JRE
    [creator]         $BP_JVM_TYPE                 JRE                                                          the JVM type - JDK or JRE
    [creator]         $BP_JVM_VERSION              17                                                           the Java version
    [creator]       Launch Configuration:
    [creator]         $BPL_DEBUG_ENABLED           false                                                        enables Java remote debugging support
    [creator]         $BPL_DEBUG_PORT              8000                                                         configure the remote debugging port
    [creator]         $BPL_DEBUG_SUSPEND           false                                                        configure whether to suspend execution until a debugger has attached
    [creator]         $BPL_HEAP_DUMP_PATH                                                                       write heap dumps on error to this path
    [creator]         $BPL_JAVA_NMT_ENABLED        true                                                         enables Java Native Memory Tracking (NMT)
    [creator]         $BPL_JAVA_NMT_LEVEL          summary                                                      configure level of NMT, summary or detail
    [creator]         $BPL_JFR_ARGS                                                                             configure custom Java Flight Recording (JFR) arguments
    [creator]         $BPL_JFR_ENABLED             false                                                        enables Java Flight Recording (JFR)
    [creator]         $BPL_JMX_ENABLED             false                                                        enables Java Management Extensions (JMX)
    [creator]         $BPL_JMX_PORT                5000                                                         configure the JMX port
    [creator]         $BPL_JVM_HEAD_ROOM           0                                                            the headroom in memory calculation
    [creator]         $BPL_JVM_LOADED_CLASS_COUNT  35% of classes                                               the number of loaded classes in memory calculation
    [creator]         $BPL_JVM_THREAD_COUNT        250                                                          the number of threads in memory calculation
    [creator]         $JAVA_TOOL_OPTIONS                                                                        the JVM launch flags
    [creator]         Using Java version 17 extracted from MANIFEST.MF
    [creator]       BellSoft Liberica JRE 17.0.7: Reusing cached layer
    [creator]       Launch Helper: Reusing cached layer
    [creator]       Java Security Properties: Reusing cached layer
    [creator]
    [creator]     Paketo Buildpack for Syft 1.32.1
    [creator]       https://github.com/paketo-buildpacks/syft
    [creator]         Downloading from https://github.com/anchore/syft/releases/download/v0.84.0/syft_0.84.0_linux_amd64.tar.gz
    [creator]         Verifying checksum
    [creator]         Writing env.build/SYFT_CHECK_FOR_APP_UPDATE.default
    [creator]
    [creator]     Paketo Buildpack for Executable JAR 6.7.4
    [creator]       https://github.com/paketo-buildpacks/executable-jar
    [creator]       Class Path: Contributing to layer
    [creator]         Writing env/CLASSPATH.delim
    [creator]         Writing env/CLASSPATH.prepend
    [creator]       Process types:
    [creator]         executable-jar: java org.springframework.boot.loader.JarLauncher (direct)
    [creator]         task:           java org.springframework.boot.loader.JarLauncher (direct)
    [creator]         web:            java org.springframework.boot.loader.JarLauncher (direct)
    [creator]
    [creator]     Paketo Buildpack for Spring Boot 5.26.1
    [creator]       https://github.com/paketo-buildpacks/spring-boot
    [creator]       Build Configuration:
    [creator]         $BP_SPRING_CLOUD_BINDINGS_DISABLED   false  whether to contribute Spring Boot cloud bindings support
    [creator]       Launch Configuration:
    [creator]         $BPL_SPRING_CLOUD_BINDINGS_DISABLED  false  whether to auto-configure Spring Boot environment properties from bindings
    [creator]         $BPL_SPRING_CLOUD_BINDINGS_ENABLED   true   Deprecated - whether to auto-configure Spring Boot environment properties from bindings
    [creator]       Creating slices from layers index
    [creator]         dependencies (69.7 MB)
    [creator]         spring-boot-loader (269.4 KB)
    [creator]         snapshot-dependencies (0.0 B)
    [creator]         application (54.9 KB)
    [creator]       Launch Helper: Reusing cached layer
    [creator]       Spring Cloud Bindings 1.13.0: Contributing to layer
    [creator]         Downloading from https://repo1.maven.org/maven2/org/springframework/cloud/spring-cloud-bindings/1.13.0/spring-cloud-bindings-1.13.0.jar
    [creator]         Verifying checksum
    [creator]         Copying to /layers/paketo-buildpacks_spring-boot/spring-cloud-bindings
    [creator]       Web Application Type: Reusing cached layer
    [creator]       4 application slices
    [creator]       Image labels:
    [creator]         org.opencontainers.image.title
    [creator]         org.opencontainers.image.version
    [creator]         org.springframework.boot.version
    [creator]     ===> EXPORTING
    [creator]     Reusing layer 'paketo-buildpacks/ca-certificates:helper'
    [creator]     Reusing layer 'paketo-buildpacks/bellsoft-liberica:helper'
    [creator]     Reusing layer 'paketo-buildpacks/bellsoft-liberica:java-security-properties'
    [creator]     Reusing layer 'paketo-buildpacks/bellsoft-liberica:jre'
    [creator]     Reusing layer 'paketo-buildpacks/executable-jar:classpath'
    [creator]     Reusing layer 'paketo-buildpacks/spring-boot:helper'
    [creator]     Reusing layer 'paketo-buildpacks/spring-boot:spring-cloud-bindings'
    [creator]     Reusing layer 'paketo-buildpacks/spring-boot:web-application-type'
    [creator]     Reusing layer 'buildpacksio/lifecycle:launch.sbom'
    [creator]     Reusing 5/5 app layer(s)
    [creator]     Reusing layer 'buildpacksio/lifecycle:launcher'
    [creator]     Reusing layer 'buildpacksio/lifecycle:config'
    [creator]     Reusing layer 'buildpacksio/lifecycle:process-types'
    [creator]     Adding label 'io.buildpacks.lifecycle.metadata'
    [creator]     Adding label 'io.buildpacks.build.metadata'
    [creator]     Adding label 'io.buildpacks.project.metadata'
    [creator]     Adding label 'org.opencontainers.image.title'
    [creator]     Adding label 'org.opencontainers.image.version'
    [creator]     Adding label 'org.springframework.boot.version'
    [creator]     Setting default process type 'web'
    [creator]     Saving docker.io/library/template-service:1.0...
    [creator]     *** Images (b2ff10a10f98):
    [creator]           docker.io/library/template-service:1.0
    [creator]     Reusing cache layer 'paketo-buildpacks/syft:syft'
    [creator]     Reusing cache layer 'buildpacksio/lifecycle:cache.sbom'

Successfully built image 'docker.io/library/template-service:1.0'

image created

docker images
REPOSITORY                 TAG                   IMAGE ID       CREATED        SIZE
template-service           1.0                   b2ff10a10f98   43 years ago   330MB
paketobuildpacks/builder   base                  050ed48532b2   43 years ago   1.31GB

Run

docker run -it --env POSTGRES_DB=jdbc:postgresql://host.docker.internal:5432/template-service template-service:1.0
Setting Active Processor Count to 12
Calculating JVM memory based on 22731180K available memory
For more information on this calculation, see https://paketo.io/docs/reference/java-reference/#memory-calculator
Calculated JVM Memory Configuration: -XX:MaxDirectMemorySize=10M -Xmx22090408K -XX:MaxMetaspaceSize=128771K -XX:ReservedCodeCacheSize=240M -Xss1M (Total Memory: 22731180K, Thread Count: 250, Loaded Class Count: 20321, Headroom: 0%)
Enabling Java Native Memory Tracking
Adding 137 container CA certificates to JVM truststore
Spring Cloud Bindings Enabled
Picked up JAVA_TOOL_OPTIONS: -Djava.security.properties=/layers/paketo-buildpacks_bellsoft-liberica/java-security-properties/java-security.properties -XX:+ExitOnOutOfMemoryError -XX:ActiveProcessorCount=12 -XX:MaxDirectMemorySize=10M -Xmx22090408K -XX:MaxMetaspaceSize=128771K -XX:ReservedCodeCacheSize=240M -Xss1M -XX:+UnlockDiagnosticVMOptions -XX:NativeMemoryTracking=summary -XX:+PrintNMTStatistics -Dorg.springframework.cloud.bindings.boot.enable=true
2023-09-17T15:36:27.814Z  INFO 1 --- [           main] c.tower.templateservice.ApplicationKt    : Starting ApplicationKt v1.0 using Java 17.0.7 with PID 1 (/workspace/BOOT-INF/classes started by cnb in /workspace)
2023-09-17T15:36:27.820Z  INFO 1 --- [           main] c.tower.templateservice.ApplicationKt    : The following 1 profile is active: "message-debug-logging"
2023-09-17T15:36:28.395Z  INFO 1 --- [           main] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data JPA repositories in DEFAULT mode.
2023-09-17T15:36:28.456Z  INFO 1 --- [           main] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 44 ms. Found 1 JPA repository interfaces.
2023-09-17T15:36:29.272Z  INFO 1 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port(s): 8080 (http)
2023-09-17T15:36:29.284Z  INFO 1 --- [           main] o.apache.catalina.core.StandardService   : Starting service [Tomcat]
2023-09-17T15:36:29.284Z  INFO 1 --- [           main] o.apache.catalina.core.StandardEngine    : Starting Servlet engine: [Apache Tomcat/10.1.12]
2023-09-17T15:36:29.389Z  INFO 1 --- [           main] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring embedded WebApplicationContext
2023-09-17T15:36:29.391Z  INFO 1 --- [           main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 1480 ms
2023-09-17T15:36:29.566Z  INFO 1 --- [           main] o.hibernate.jpa.internal.util.LogHelper  : HHH000204: Processing PersistenceUnitInfo [name: default]
2023-09-17T15:36:29.642Z  INFO 1 --- [           main] org.hibernate.Version                    : HHH000412: Hibernate ORM core version 6.2.7.Final
2023-09-17T15:36:29.645Z  INFO 1 --- [           main] org.hibernate.cfg.Environment            : HHH000406: Using bytecode reflection optimizer
2023-09-17T15:36:29.894Z  INFO 1 --- [           main] o.h.b.i.BytecodeProviderInitiator        : HHH000021: Bytecode provider name : bytebuddy
2023-09-17T15:36:30.038Z  INFO 1 --- [           main] o.s.o.j.p.SpringPersistenceUnitInfo      : No LoadTimeWeaver setup: ignoring JPA class transformer
2023-09-17T15:36:30.260Z  INFO 1 --- [           main] o.h.b.i.BytecodeProviderInitiator        : HHH000021: Bytecode provider name : bytebuddy
2023-09-17T15:36:30.841Z  INFO 1 --- [           main] o.h.e.t.j.p.i.JtaPlatformInitiator       : HHH000490: Using JtaPlatform implementation: [org.hibernate.engine.transaction.jta.platform.internal.NoJtaPlatform]
2023-09-17T15:36:30.860Z  INFO 1 --- [           main] com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Starting...
2023-09-17T15:36:31.029Z  INFO 1 --- [           main] com.zaxxer.hikari.pool.HikariPool        : HikariPool-1 - Added connection org.postgresql.jdbc.PgConnection@625db0e0
2023-09-17T15:36:31.032Z  INFO 1 --- [           main] com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Start completed.
2023-09-17T15:36:31.143Z  INFO 1 --- [           main] j.LocalContainerEntityManagerFactoryBean : Initialized JPA EntityManagerFactory for persistence unit 'default'
2023-09-17T15:36:32.091Z  INFO 1 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port(s): 8080 (http) with context path ''
2023-09-17T15:36:32.107Z  INFO 1 --- [           main] c.tower.templateservice.ApplicationKt    : Started ApplicationKt in 4.914 seconds (process running for 5.465)

Build behind firewall

In case it can’t be built behind the firewall:

BellSoft Liberica JRE 17.0.7: Contributing to layer
    [creator]         Downloading from https://github.com/bell-sw/Liberica/releases/download/17.0.7+7/bellsoft-jre17.0.7+7-linux-amd64.tar.gz
unable to invoke layer
unable to get dependency jre
unable to download https://github.com/bell-sw/Liberica/releases/download/17.0.7+7/bellsoft-jre17.0.7+7-linux-amd64.tar.gz
unable to request xxx
Get "https://xxxx"
ERROR: failed to build: exit status 1

add these to build.gradle :

tasks.named("bootBuildImage") {
    bindings = ["${projectDir}/bindings/ca-certificates:/platform/bindings/ca-certificates"]
}

and

mkdir -p bindings/ca-certificates
cp path/to/your-cert.pem bindings/ca-certificates
echo "ca-certificates" > bindings/ca-certificates/type

will see an additional cert added;

[creator]     Paketo Buildpack for CA Certificates 3.6.3
[creator]       https://github.com/paketo-buildpacks/ca-certificates
[creator]       Launch Helper: Reusing cached layer
[creator]       CA Certificates: Contributing to layer
[creator]         Added 1 additional CA certificate(s) to system truststore
[creator]         Writing env.build/SSL_CERT_DIR.append
[creator]         Writing env.build/SSL_CERT_DIR.delim
[creator]         Writing env.build/SSL_CERT_FILE.default

When a dependent package is blocked by firewall

If a package is not accessible(blocked by firewall), we can tell buildpack to look for it from a self-hosted artifactory

e.g. spring-cloud-bindings@1.13.0 , find this package’s sha256 in https://github.com/paketo-buildpacks/spring-boot/blob/main/buildpack.toml

  [[metadata.dependencies]]
    cpes = ["cpe:2.3:a:vmware:spring_cloud_bindings:1.13.0:*:*:*:*:*:*:*"]
    id = "spring-cloud-bindings"
    name = "Spring Cloud Bindings"
    purl = "pkg:generic/springframework/spring-cloud-bindings@1.13.0"
    sha256 = "70a448cd45d1dbc117770f934961cd9577c0c4404d34986824f8f593cae4aada"
    stacks = ["io.buildpacks.stacks.bionic", "io.paketo.stacks.tiny", "*"]
    uri = "https://repo1.maven.org/maven2/org/springframework/cloud/spring-cloud-bindings/1.13.0/spring-cloud-bindings-1.13.0.jar"
    version = "1.13.0"

Then we do this:

echo "https://${jfrog_user}:{jfrpg_pass}@my-artifactory-hostname/artifactory/jcenter-cache/org/spring/packagename/filename"    >> bindings/spring/spring-cloud-config/70a448cd45d1dbc117770f934961cd9577c0c4404d34986824f8f593cae4aada
echo "dependency-mapping" >> bindings/spring/spring-cloud-config/type

the folder structure will be:

refer to https://github.com/paketo-buildpacks/gradle

Choose Java version

change BP_JVM_VERSION in build.gradle,

tasks.named("bootBuildImage") {
    bindings = ["${projectDir}/bindings/ca-certificates:/platform/bindings/ca-certificates"]
    environment = ["BP_JVM_VERSION": "20"]
}

We can also specify image name in command line

gradlew.bat bootBuildImage --imageName=template-service-name:local

Refer to spring gradle plugin reference for More information

Build with PACK CLI

an example:

pack build template-service --builder paketobuildpacks/builder-jammy-base  --volume H:\work\template-service\bindings\ca-certificates:/platform/bindings/ca-certificates --env "BP_JVM_VERSION=17"

but seems it doesn’t support mavenLocal very well

 Repositories:
      Name: MavenLocal; url: file:/home/cnb/.m2/repository

> Task :compileKotlin FAILED

      FAILURE: Build failed with an exception.

      * What went wrong:
      Execution failed for task ':compileKotlin'.
      > Could not resolve all files for configuration ':compileClasspath'.
         > Could not find com.company:mylib:1.0.0.  (supposed in mavenLocal)
           Required by:
               project :

      * Try:
      > Run with --stacktrace option to get the stack trace.
      > Run with --info or --debug option to get more log output.
      > Run with --scan to get full insights.

I prefer to gradle-plugin , which is better platform-independent wise.

Rebase

Rebase allows app developers or operators to rapidly update an app image when its stack’s run image has changed. By using image layer rebasing, this command avoids the need to fully rebuild the app.

Rebase · Cloud Native Buildpacks

example：

pack inspect-image 
# =>
pack inspect-image template-service:1.0
Inspecting image: template-service:1.0

REMOTE:
(not present)

LOCAL:

Stack: io.buildpacks.stacks.bionic

Base Image:
  Reference: f2e5000af0cb0db7d16a2939fc88c2aa522e91826c0e87ef2a4ec2825b213585
  Top Layer: sha256:1eb5983d73014e2f11902c90f9af0f0dd6b4b1cb128a57b1cebee6ef54252199

Run Images:
  index.docker.io/paketobuildpacks/run:base-cnb
  gcr.io/paketo-buildpacks/run:base-cnb

Rebasable: true

Buildpacks:
  ID                                         VERSION        HOMEPAGE
  paketo-buildpacks/ca-certificates          3.6.3          https://github.com/paketo-buildpacks/ca-certificates
  paketo-buildpacks/bellsoft-liberica        10.2.6         https://github.com/paketo-buildpacks/bellsoft-liberica
  paketo-buildpacks/syft                     1.32.1         https://github.com/paketo-buildpacks/syft
  paketo-buildpacks/executable-jar           6.7.4          https://github.com/paketo-buildpacks/executable-jar
  paketo-buildpacks/dist-zip                 5.6.4          https://github.com/paketo-buildpacks/dist-zip
  paketo-buildpacks/spring-boot              5.26.1         https://github.com/paketo-buildpacks/spring-boot

Processes:
  TYPE                  SHELL        COMMAND        ARGS                                               WORK DIR
  web (default)                      java           org.springframework.boot.loader.JarLauncher        /workspace
  executable-jar                     java           org.springframework.boot.loader.JarLauncher        /workspace
  task                               java           org.springframework.boot.loader.JarLauncher        /workspace

rebase:

# need admin privilege on windows
pack rebase template-service:1.0 
#=>

base-cnb: Pulling from paketobuildpacks/run
Digest: sha256:1af9935d8987fd52b2266d288200c9482d1dd5529860bbf5bc2d248de1cb1a38
Status: Image is up to date for paketobuildpacks/run:base-cnb
Rebasing template-service:1.0 on run image index.docker.io/paketobuildpacks/run:base-cnb
Timer: Rebaser started at 2023-09-19T00:43:54+08:00
Timer: Saving template-service:1.0... started at 2023-09-19T00:43:59+08:00
*** Images (82c2cca9d640):
      template-service:1.0
Timer: Saving template-service:1.0... ran for 1.8416996s and ended at 2023-09-19T00:44:01+08:00
Timer: Rebaser ran for 6.9762501s and ended at 2023-09-19T00:44:01+08:00
Rebased Image: 82c2cca9d640d8401f45bf651b11f5e6ca689e476adf10f1fa1c59faba1f5f19
Successfully rebased image template-service:1.0

Reference

https://buildpacks.io/docs/

Getting Started

Happy Coding!

I wrote a story about getting the spring boot version, I want to check the JDK version at compile time as well

# A quick shell script to check the spring-boot version
# only work for gradlw project!!
# read repo-name from services.csv
# cat services.csv 
# camel-spring-demo,team,production

set -e
set -x
mkdir -p tmp

servicelist=$(pwd)/services.csv # reponame,squad,env(prod/dev)
output=$(pwd)/spring-ver.csv
repo_base="https://github.com/myaccount/"  # my repo root path
kt=/path/to/kubectl
JDK17=/path/to/jdk17/home
JDK11=/path/to/jdk11/home
JDK8=/path/to/jdk8/home

# "https://$username:$pass@github.com/myaccount/"  # it may need username/pass

# pull all code under ./tmp/
cd ./tmp/

while IFS="," read -r varservice varsquad varenv
do
  # skip line start with '#' 
  [[ $line =~ ^#.* ]] && continue
  repo="$repo_base$varservice.git"
  echo "$varenv : $varsquad :$varservice : $repo"

  if [ -d $varservice ];then
    echo "Directory $varservice exists."
    cd $varservice
    git pull
    cd ..
  else
    git clone --depth=1 $repo
  fi

  if [ -f "$varservice/gradlew" ];then
    echo "running gradlew"
    jdkbuildenv=$(grep sourceCompatibility $varservice/build.gradle |head -n 1|awk -F "=" '{ print $2}')
    if [ -z "$jdkbuildenv" ]
    then
      jdkbuildenv=$(grep jvmTarget $varservice/build.gradle |head -n 1|awk -F "=" '{ print $2}')
    fi

    jdkbuildenv="$(echo -e "${jdkbuildenv}" | tr -d '[:space:]')"
    
    case "$jdkbuildenv" in
    "1.8"|"JavaVersion.VERSION_1_8"|"JvmTarget.JVM_1_8")
      export JAVA_HOME=$JDK8
      jdkbuildenv="jdk8"
      ;;
    "11"|"JavaVersion.VERSION_11"|"JvmTarget.JVM_11")
      export JAVA_HOME=$JDK11
      jdkbuildenv="jdk11"
      ;;      
    "17"|"JavaVersion.VERSION_17"|"JvmTarget.JVM_17")
      export JAVA_HOME=$JDK17
      jdkbuildenv="jdk17"
      ;;
    *)
      echo "Unknown JDK version! Specify 8,11,17!"
      export JAVA_HOME=$JDK8
      jdkbuildenv="unknown,use jdk8"
      ;;
    esac

    # has sub project
    subprj=$($varservice/gradlew -q -b $varservice/build.gradle projects</dev/null|grep "Project ':"|head -n 1|cut -d ' ' -f 3)
    if [ -z "$subprj" ]    
    then
      springver=$($varservice/gradlew -b $varvalue/build.gradle -q dependencyInsight --dependency org.springframework.boot:spring-boot</dev/null|grep "spring-boot"|head -n 1 |awk -F":" '{ print $3}')||true
    else
      subprj="$(echo -e "${subprj})" | tr -d "'")"
      springver=$($varservice/gradlew -b $varvalue/build.gradle -q $subprj:dependencyInsight --dependency org.springframework.boot:spring-boot</dev/null|grep "spring-boot"|head -n 1 |awk -F":" '{ print $3}')||true
    fi


  else
    echo "not-gradle-project"
    jdkbuildenv="unknown,use jdk8"
    springver="not-gradle-project"
  fi

  # connect POD to fetch JDK runtime env
  podname=$($kt get pods -o name -n $varservice|head -n 1)
  if [ -z "$podname" ]
  then
    echo "No POD"
    jdkver="No POD"
  else
    jdkver=$($kt exec $podname -n $varservice -- java "-version" 2>&1|grep jdk)||true
  fi


  if [ -z "$jdkver" ]
  then
    echo "use buildenv"
    jdkver="unable to connect to POD, buildenv=$jdkbuildenv"
  fi

  if [ -z "$springver" ]
  then    
    springver="unknown-spring"
  fi


  echo "$varenv:$varsquad:$varservice:$springver:$jdkbuildenv:$jdkver"
  echo "$varenv:$varsquad:$varservice:$springver:$jdkbuildenv:$jdkver">> $output 
  
done < $servicelist

and this is my services.csv file

cat services.csv 
camel-spring-demo,team1,production

Output is:

cat spring-ver.csv # it is spring-boot 3.0.2
production:team1:camel-spring-demo:3.0.2 (selected by rule):jdk17:openjdk version "17.0.7" 2023-04-18 LTS

Happy Coding!