Jekyll2023-09-20T15:58:35+00:00https://zhenik.github.io/https://zhenik.github.io/feed.xmldev & just a guy2019-06-24T00:00:00+00:002019-06-24T00:00:00+00:00https://zhenik.github.io/https://zhenik.github.io/testing-kafka-based-applications<p>Blog post about <a href="https://medium.com/test-kafka-based-applications/https-medium-com-testing-kafka-based-applications-85d8951cec43?sk=b5a54ca1cedcaeac9ece529436874230">Testing kafka based applications</a> Different approaches how to make Unit, integration and end-to-end tests.<br /> —- <a href="https://github.com/sysco-middleware/kafka-testing">https://github.com/sysco-middleware/kafka-testing</a></p> <hr />

Blog post about Testing kafka based applications Different approaches how to make Unit, integration and end-to-end tests. —- https://github.com/sysco-middleware/kafka-testing

2019-04-08T00:00:00+00:002019-04-08T00:00:00+00:00https://zhenik.github.io/https://zhenik.github.io/kafka-metrics-prometheus<p>Proof of concept: <a href="https://github.com/prometheus/jmx_exporter">jmx-exporter</a> and kafka clients. Main idea: expose <code class="language-plaintext highlighter-rouge">JMX metrics</code> of applications which use Apache Kafka APIs. Jmx exporter requires <a href="https://github.com/prometheus/jmx_exporter/tree/master/example_configs">config patterns</a>.</p> <hr /> <p><a href="https://github.com/zhenik/kafka-clients_metrics-tracing">https://github.com/zhenik/kafka-clients_metrics-tracing</a></p> <hr /> <h2 id="approaches">Approaches:</h2> <h3 id="1-application-coupled-with-java-agent-this-approach-sort-of-standalone-mode-reference">1. Application coupled with java agent. This approach sort of standalone mode. <a href="https://github.com/prometheus/jmx_exporter#jmx-exporter">Reference</a></h3> <p><img src="https://raw.githubusercontent.com/zhenik/kafka-clients_metrics-tracing/master/docs/jmx-exporter-standalone.png" alt="img" /></p> <h3 id="2-application--exporterside-car">2. Application + Exporter(side-car).</h3> <p><img src="https://raw.githubusercontent.com/zhenik/kafka-clients_metrics-tracing/master/docs/jmx-exporter-decoupled.png" alt="img" /><br /> There is need to run application with several parameters to be able expose JMX metrics from application. Here I provide <code class="language-plaintext highlighter-rouge">Dockerfile</code> with <code class="language-plaintext highlighter-rouge">env</code> variables:</p> <div class="language-dockerfile highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">FROM</span><span class="s"> openjdk:8-jdk-stretch</span> <span class="k">COPY</span><span class="s"> ./target/producer-metrics-1.0-SNAPSHOT.jar .</span> <span class="k">CMD</span><span class="s"> java -Dcom.sun.management.jmxremote \</span> -Dcom.sun.management.jmxremote=true \ -Dcom.sun.management.jmxremote.local.only=false \ -Dcom.sun.management.jmxremote.authenticate=false \ -Dcom.sun.management.jmxremote.ssl=false \ -Djava.rmi.server.hostname=$HOST \ -Dcom.sun.management.jmxremote.host=$HOST \ -Dcom.sun.management.jmxremote.port=$JMX_PORT \ -Dcom.sun.management.jmxremote.rmi.port=$JMX_PORT \ -jar producer-metrics-1.0-SNAPSHOT.jar kafka:9092 </code></pre></div></div> <h3 id="3-application-on-dropwizard-with-custom-metrics--exporterside-car">3. Application on Dropwizard with custom metrics + Exporter(side-car)</h3> <p><img src="https://raw.githubusercontent.com/zhenik/kafka-clients_metrics-tracing/master/docs/jmx-exporter-custom-metrics.png" alt="img" /> To expand metrics I use <a href="https://www.dropwizard.io/1.3.9/docs/">Dropwizard</a> framework and <a href="https://github.com/prometheus/client_java/tree/master/simpleclient_dropwizard">prometheus client for dropwizard</a>.</p> <div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// Create registry for Dropwizard metrics.</span> <span class="nc">MetricRegistry</span> <span class="n">metrics</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">MetricRegistry</span><span class="o">();</span> <span class="c1">// Create a Dropwizard counter.</span> <span class="nc">Counter</span> <span class="n">counter</span> <span class="o">=</span> <span class="n">metrics</span><span class="o">.</span><span class="na">counter</span><span class="o">(</span><span class="s">"my_custom_counter_total"</span><span class="o">);</span> <span class="nc">CollectorRegistry</span><span class="o">.</span><span class="na">defaultRegistry</span> <span class="o">.</span><span class="na">register</span><span class="o">(</span><span class="k">new</span> <span class="nc">DropwizardExports</span><span class="o">(</span><span class="n">metrics</span><span class="o">));</span> <span class="cm">/** prometheus + dropwizard metrics are available * on localhost:8081/prometheus/metrics */</span> <span class="n">environment</span><span class="o">.</span><span class="na">getAdminContext</span><span class="o">()</span> <span class="o">.</span><span class="na">addServlet</span><span class="o">(</span><span class="k">new</span> <span class="nc">ServletHolder</span><span class="o">(</span> <span class="k">new</span> <span class="nf">MetricsServlet</span><span class="o">()),</span> <span class="s">"/prometheus/metrics"</span><span class="o">);</span> <span class="c1">// Add metrics about CPU, JVM memory etc.</span> <span class="nc">DefaultExports</span><span class="o">.</span><span class="na">initialize</span><span class="o">();</span> <span class="c1">// use </span> <span class="n">counter</span><span class="o">.</span><span class="na">inc</span><span class="o">();</span> </code></pre></div></div>

Proof of concept: jmx-exporter and kafka clients. Main idea: expose JMX metrics of applications which use Apache Kafka APIs. Jmx exporter requires config patterns.