Nicola Ferraro

println("Hello World!")

Nicola Ferraro — Sun, 14 Feb 2016 00:00:00 GMT

Welcome to my new blog. Cool things are coming up...

Exception Handling in Apache Spark

Nicola Ferraro — Thu, 18 Feb 2016 00:00:00 GMT

Apache Spark is a fantastic framework for writing highly scalable applications. Data and execution code are spread from the driver to tons of worker machines for parallel processing. But debugging this kind of applications is often a really hard task.

Exceptions need to be treated carefully, because a simple runtime exception caused by dirty source data can easily lead to the termination of the whole process. Let’s see an example.

// source contains dirty data
val transformed = source
  .map(e => myCustomFunction(e))
  // do some other actions
 
transformed.saveAsTextFile("xxx")

The code above is quite common in a Spark application. Data gets transformed in order to be joined and matched with other data and the transformation algorithms are often provided by the application coder into a map function. It is clear that, when you need to transform a RDD into another, the map function is the best option, as it changes every element of the RDD, without changing its size. But an exception thrown by the myCustomFunction transformation algorithm causes the job to terminate with error.

In the real world, a RDD is composed of millions or billions of simple records coming from different sources. The probability of having wrong/dirty data in such RDDs is really high. In these cases, instead of letting the process terminate, it is more desirable to continue processing the other data and analyze, at the end of the process, what has been left behind, and then decide if it is worth spending some time to find the root causes of the problem.

How should the code above change to support this behaviour? A first trial:

val transformed = source
  .flatMap(e => Try{myCustomFunction(e)}.toOption)
  // other actions

Here the function myCustomFunction is executed within a Scala Try block, then converted into an Option. The code is put in the context of a flatMap, so the result is that all the elements that can be converted using the custom function will be present in the resulting RDD. Elements whose transformation function throws an exception will be automatically discarded. Pretty good, but we have lost information about the exceptions. Can we do better?

Why don’t we collect all exceptions, alongside the input data that caused them? If the exception are (as the word suggests) not the default case, they could all be collected by the driver and then printed out to the console for debugging. What I mean is explained by the following code excerpt:

// define an accumulable collection for exceptions
val accumulable = sc.accumulableCollection(mutable.HashSet[(Any, Throwable)]())
 
val transformed = source.flatMap(e => {
  val fe = Try{myCustomFunction(e)}
  val trial = fe match {
    case Failure(t) =>
      // push to an accumulable collection 
      // both the input data and the throwable
      accumulable += (e, t)
      fe
    case t: Try[U] => t
  }
  trial.toOption
})
 
// call at least one action on 'transformed' (eg. count)
transformed.count

Probably it is more verbose than a simple map call. I will simplify it at the end. Now that you have collected all the exceptions, you can print them as follows:

// at the end of the process, print the exceptions
accumulable.value.foreach{case (i, e) => {
  println(s"--- Exception on input: ($i)")
  // using org.apache.commons.lang3.exception.ExceptionUtils
  println(ExceptionUtils.getStackTrace(e))
}}

So far, so good. Now you can generalize the behaviour and put it in a library. Or... you'd better use mine: https://github.com/nerdammer/spark-additions.

The Spark Additions way is a lot easier:

// import all implicit conversions
import it.nerdammer.spark.additions._
 
// ...
val transformed = source
  .tryMap(e => myCustomFunction(e))
 
// call at least one action on 'transformed' (eg. count)
transformed.count

The tryMap method does everything for you.

What you need to write is the code that gets the exceptions on the driver and prints them. Very easy:

// sc is the SparkContext: now with a new method
sc.accumulatedExceptions()
  .foreach{case (i, e) => {
    println(s"--- Exception on input: ($i)")
    println(ExceptionUtils.getStackTrace(e))
  }}

More usage examples and tests here (BasicTryFunctionsIT). Look also at the package implementing the Try-Functions (there is also a tryFlatMap function).

Contributions are always appreciated.

Using Non-Serializable Objects in Apache Spark

Nicola Ferraro — Mon, 22 Feb 2016 00:00:00 GMT

Anyone who starts writing applications for Apache Spark encounters immediately an infamous exception...

org.apache.spark.SparkException: Task not serializable
	at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:304)
	at org.apache.spark.util.ClosureCleaner$.org$apache$spark$util$ClosureCleaner$$clean(ClosureCleaner.scala:294)
	at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:122)
	at org.apache.spark.SparkContext.clean(SparkContext.scala:2055)
	at org.apache.spark.rdd.RDD$$anonfun$map$1.apply(RDD.scala:324)
	at org.apache.spark.rdd.RDD$$anonfun$map$1.apply(RDD.scala:323)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:111)
	at org.apache.spark.rdd.RDD.withScope(RDD.scala:316)
	at org.apache.spark.rdd.RDD.map(RDD.scala:323)

This happens whenever Spark tries to transmit the scheduled tasks to remote machines. Tasks are just pieces of application code that are sent from the driver to the workers.

Given the frequency of that exception, one may think that any piece of code that is executed by a worker node must be serializable. Fortunately, that is not true.

The classpath of the driver and worker nodes are controlled by the user that is launching the application. This is the case when the application is launched in a standalone Spark cluster as well when an external cluster manager, like YARN, is used. If the deployment solution is an uber-jar, the jar will be shared among all machines. When the application is composed of multiple separate jars, the –jars option (supported in YARN mode) can be used to include them all in the classpaths of the worker JVMs.

In this context, it is quite easy to use non-serializable objects in tasks. The only requirement is that they have just a serializable initialization code.

Let's see an example. Suppose you want to connect from the remote worker machines to a JDBC data source (hope that connections are issued towards a NoSQL – a.k.a. NowSequel – database). Connection factories are often not serializable.

// A pooled: org.apache.commons.dbcp2.BasicDataSource
val ds = new BasicDataSource() // Not serializable
ds.setDriverClassName("org.driver.Classname")
ds.setUrl("...")
// set username, options
 
sc.parallelize(1 to 10)
  .map(i => {
    val conn = ds.getConnection
    // do something
    conn.close()
  })
  .count
 
// does it work?

Of course, it will not work.

org.apache.spark.SparkException: Task not serializable
...
...
Caused by: java.io.NotSerializableException: org.apache.commons.dbcp2.BasicDataSource

It throws the infamous “Task not serializable” exception. But you can just wrap it in an object to make it available at the worker side.

// Holds a reference to the datasource factory
object Holder {
 
  def ds() = {
    val ds = new BasicDataSource()
    ds.setDriverClassName("org.driver.Classname")
    ds.setUrl("...")
    // set username, options
    ds
  }
 
}
 
object SparkApplication {
 
  def main(args: Array[String]) {
    // init Spark Context sc
    sc.parallelize(1 to 10)
        .map(i => {
          val conn = Holder.ds.getConnection
          // do something
          conn.close()
        })
        .count
  }
}

Why does it work? Scala functions declared inside objects are equivalent to static Java methods. In order to call a static method, you don’t need to serialize the class, you need the declaring class to be reachable by the classloader (and it is the case, as the jar archives can be shared among driver and workers).

Creating external objects can be tedious. I’ve included some utilities in the https://github.com/nerdammer/spark-additions library.

For example, to share the datasource one might write:

// Declare it as a shared variable
val dsv = SharedVariable {
  val ds = new BasicDataSource() // not serializable
  ds.setDriverClassName("org.driver.Classname")
  ds.setUrl("...")
  // set username, options
  ds
}
 
sc.parallelize(1 to 10)
  .map(i => {
    val ds = dsv.get
    // do something
  })
  .count

Any call to dsv.get that is executed at the worker side will create a new instance of the datasource. Alternatively the library allows you to define a SharedSingleton. There will be at most one instance of the SharedSingleton in each JVM of the Spark cluster.

// An object that acts as a singleton in each JVM of the cluster
val connectionPool = SharedSingleton {
  // initialization code
}
 
sc.parallelize(1 to 10)
  .map(i => {
    val cp = connectionPool.get
    // use the local-JVM connection pool
  })
  .count

Both SharedVariable and SharedSingleton objects can be used to share non-serializable objects that need to be used in task closures. There are other solutions that work in these circumstances, but boilerplate is reduced when using shared variables. More info here.

Spark-HBase-Connector 1.0.2 is out

Nicola Ferraro — Sun, 28 Feb 2016 00:00:00 GMT

The Spark-HBase-Connector project started as a 3-days programming marathon I made last year. At home, with the flu. Now it is becoming one of the most popular drivers to read/write data to Apache HBase from Spark.

The project is hosted on github: https://github.com/nerdammer/spark-hbase-connector. The library is also available on Maven. You can find more information on the github page.

Probably, its future will be limited by the advent of Apache Phoenix but, for now, Phoenix is running slowly and the connector’s popularity is increasing.

Some small improvements have been applied to the code base for version 1.0.2. Now Spark-HBase-Connector supports Spark 1.6.0+ and HBase 1.0.1.1+. Support for tables with up to 22 columns has been added.

Thanks for the contributions arrived so far. Pull requests are always appreciated.

A Brief History of Big Data

Nicola Ferraro — Fri, 04 Mar 2016 00:00:00 GMT

Why everybody talks about Big Data? Where does Hadoop come from? Which steps led to the diffusion of Spark? What’s next?

This presentation will drive you through all the steps that brought to this “big” change.

Logging to a NoSQL DB from Spark

Nicola Ferraro — Mon, 07 Mar 2016 00:00:00 GMT

Logging effectively is often a hard task in standard applications. But when the application runs in a distributed environment, for instance, a Spark job in a big YARN cluster, it becomes ten times harder.

Jobs are split into thousands of tasks that run inside multiple worker machines, so the classic console logging is not a good option, because the logs get written to the standard output of several remote machines, making it impossible to find useful information.

One of the best options available in all modern data platforms is logging to a NoSQL database. Many data platforms support HBase and Phoenix as NoSQL layer, so why don’t using a Phoenix table to store the logs?

First of all, the table must be created inside Phoenix, and it must be optimized for efficiently writing the log data. For example:

CREATE TABLE LOG 
(
  LOG_DATE  TIMESTAMP  NOT NULL,
  LOG_ID    BIGINT     NOT NULL,
  LOGGER    VARCHAR(150),
  LEVEL     VARCHAR(10),
  MESSAGE   VARCHAR(8192)
  CONSTRAINT LOG_PK PRIMARY KEY (LOG_DATE, LOG_ID)
) SALT_BUCKETS = 50;
 
CREATE SEQUENCE SEQ_LOG_ID MINVALUE 1;

The LOG table is defined as a salted table with 50 buckets (the number can be increased/decreased, depending on the size of the cluster). Bucketing is needed to spread the data across multiple region servers, in order to balance the load across all the machines. The LOG table has a timestamp as first column of the primary key (that is a monotonically increasing field) so, if salting buckets were not in place, only one machine at a time would be used to store the logs. The LOG_ID column is part of the primary key. It is useful to prevent collisions among log messages.

Once the table is defined, we need to configure log4j to store log messages inside it. Phoenix is compliant with the JDBC API. This allows using the JDBCAppender. Unfortunately, the standard JDBCAppender is not perfect for being used with Phoenix, just because it does not commit the transactions. Of course, I’m not saying that Phoenix supports transactions (not now). Phoenix requires that you commit the UPSERT statements, otherwise they will remain stuck in the driver’s cache. So, we need to write a custom extension of the JDBCAppender:

package it.nerdammer.log4j;
 
import org.apache.log4j.jdbc.JDBCAppender;
 
import java.sql.Connection;
import java.sql.SQLException;
 
public class PhoenixAppender extends JDBCAppender {
 
    protected Connection getConnection() throws SQLException {
        Connection connection = super.getConnection();
        connection.setAutoCommit(true);
 
        return connection;
    }
 
}

The class above just sets the autocommit=true property on every connection created by the JDBCAppender.

Now that we have an appender compatible with Phoenix, the next step is configuring it in the log4j.properties file:

# Root logger option
log4j.rootLogger=INFO, stdout
 
log4j.logger.com.enterprise=INFO, phoenix
log4j.additivity.com.enterprise=true
 
# Direct log messages to stdout
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.Target=System.out
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n
 
 
log4j.appender.phoenix=it.nerdammer.log4j.PhoenixAppender
log4j.appender.phoenix.URL=jdbc:phoenix:phoenixhost
log4j.appender.phoenix.user=anyuser
log4j.appender.phoenix.password=anypassword
log4j.appender.phoenix.driver=org.apache.phoenix.jdbc.PhoenixDriver
log4j.appender.phoenix.sql=UPSERT INTO LOG (LOG_DATE, LOG_ID, LOGGER, LEVEL, MESSAGE) VALUES ('%d', NEXT VALUE FOR SEQ_LOG_ID, '%C', '%p', '%m')
log4j.appender.phoenix.layout=org.apache.log4j.PatternLayout

An appender named phoenix has been created. It has been associated with the logger named “com.enterprise”. You can change it to the base package of your application, but note: you cannot associate the phoenix appender to the root logger. The reason is that Phoenix itself uses log4j while initializing the connection to the database. If you allow the logger org.apache.phoenix to append logs to the Phoenix table, you will get soon a stackoverflow error. One way of breaking the loop it is limiting the usage of the “phoenix” appender to just few packages of your application (in the example com.enterprise and all its sub-packages).

In order to test locally, you can just run one docker container with everything preinstalled, for example, my dockmob container for Phoenix https://hub.docker.com/r/dockmob/phoenix/, using the following script:

#!/bin/bash
# 
# Before executing the script, add to /etc/hosts the following entry
# <docker-machine-ip> phoenixhost
#
# Where <docker-machine-ip> is the IP assigned to the docker machine on OSX (usually 192.168.99.100), 
# or 127.0.0.1 on Linux
 
MYPHOENIX_ID=$(docker run -d -p 2181:2181 -p 60000:60000 -p 60010:60010 -p 60020:60020 -p 60030:60030 -h phoenixhost dockmob/phoenix:4.5.2-1.0.1 -t pseudodistributed)
docker exec -it $MYPHOENIX_ID /usr/lib/phoenix/bin/sqlline.py localhost

The last command starts a SQL console on the Phoenix instance. You need to paste the previous SQL commands to create the LOG table.

Ensure you have put all the required libraries for writing to Phoenix in the Spark application classpath. Here is a simple test for verifying if it works:

package com.enterprise.test
 
import java.sql.{DriverManager}
 
import it.nerdammer.log4j.PhoenixAppender
import org.apache.log4j.Logger
import org.apache.spark.{SparkConf, SparkContext}
import org.scalatest.{BeforeAndAfterAll, FlatSpec, Matchers}
 
class SparkJobTest extends FlatSpec with Matchers with BeforeAndAfterAll {
 
  lazy val sc: SparkContext = {
    val conf = new SparkConf()
      .setAppName("Console Test")
      .setMaster("local")
 
    new SparkContext(conf)
  }
 
  override protected def beforeAll(): Unit = sc
 
  override protected def afterAll(): Unit = sc.stop()
 
 
  "the console logger " should "work" in {
 
    val databaseURL = Logger.getLogger("com.enterprise").getAppender("phoenix").asInstanceOf[PhoenixAppender].getURL
    val conn = DriverManager.getConnection(databaseURL, "any", "any")
    conn.setAutoCommit(true)
    val pstm = conn.prepareStatement("delete from log")
    pstm.execute()
    pstm.close()
 
    sc.parallelize(1 to 10)
      .map(e => {
        Logger.getLogger(classOf[SparkJobTest]).info("This is a log message")
        e
      })
      .count();
 
    val pstm2 = conn.prepareStatement("select count(*) from log")
    val rs = pstm2.executeQuery()
    rs.next()
    val count = rs.getInt(1)
    rs.close()
    pstm.close()
    conn.close()
 
    assert(count == 10)
  }
}

Now the logs appear if you query the LOG table using the SQL console. You can find the full code here: https://github.com/nerdammer/spark-additions/tree/master/log4j-phoenix.

Spash - Organizing your Big Data with SSH and Bash

Nicola Ferraro — Sat, 16 Apr 2016 00:00:00 GMT

Spash is a command line tool for Big Data platforms that simulates a real Unix environment, providing most of the commands of a typical Bash shell on top of YARN, HDFS and Apache Spark.

Spash uses the HDFS APIs to execute simple file operations and Apache Spark to perform parallel computations on big datasets. With Spash, managing a Big Data cluster becomes as natural as writing bash commands.

Spash is an open source project from an idea of Nicola Barbieri. The code is here: https://github.com/nerdammer/spash.

Architecture

The Spash daemon runs on an edge node of a Big Data cluster and listens for incoming SSH connections on port 2222. Clients can connect using the OS native terminal application, or Putty on Windows.

The Spash daemon will emulate a Unix OS and leverage the power of Spark to perform efficient computations on distributed data.

The World Before Spash

For those who don’t remember the classic way of doing simple operations on HDFS, here’s a reminder:

bash$ hdfs dfs -ls /
#
##
### [wait for the JVM to load and execute]
...
##########################################
bash$ hdfs dfs -copyFromLocal myFile /
#
##
### [wait for the JVM to load and execute]
...
##########################################
bash$

Spash Makes It Easier

Just run the Spash daemon on a node of your Big Data cluster, you can connect to it using ssh user@hdfshost -p 2222 (the password is user) and then run all your favourite bash commands to manipulate data.

user@spash:/$ echo "Maybe I can provide a real life example..."
Maybe I can provide a real life example...
 
user@spash:/$ echo "My content" > myFile
 
user@spash:/$ ls
myFile
 
user@spash:/$ cat myFile
My content
 
user@spash:/$ echo text2 > myFile2
 
user@spash:/$ ls -l
drwxr-xr-x 4 user supergroup 0 30 mar 18:34 myFile 
drwxr-xr-x 4 user supergroup 0 30 mar 18:35 myFile2
 
user@spash:/$ cat myFile myFile2 > myFile3
 
user@spash:/$ cat myFile3
My content
text2
 
user@spash:/$ cat myFile3 | grep -v 2
My content
 
user@spash:/$ exit

And this is just the tip of the iceberg. From your host, you can use scp (scp myfile -P 2222 user@hdfshost:/) to copy a file into hdfs. You can even use FileZilla or WinSCP to browse HDFS.

Spash works on Cloudera CDH 5 and many other platforms.

Spash is a proof of concept and needs contributions. Look at how to contribute and get involved on the project home page: https://github.com/nerdammer/spash.

Creating a Telegram Bot in 5 minutes with Apache Camel

Nicola Ferraro — Fri, 27 May 2016 00:00:00 GMT

Recently, I started contributing to open source software of the Apache Software Foundation and I developed camel-telegram, a component that allows camel based applications to exchange data using the Telegram messaging network. It has been released as of Apache Camel 2.18.0.

The component is targeted to enterprise applications, to let them route chat messages within their ESB and communicate with users in a novel way. Camel is a widely used integration layer, that forms the basis of production-grade enterprise platforms such as JBoss Fuse. But there exists also a simpler and more ludic way of using it, to create a reactive messaging Bot. I’ll explain how to create such a bot with a 5-mins step by step guide. The full documentation is available in the Camel manual.

Creating the Telegram Bot

Before getting into the code, you first need to create the bot. Telegram lets you create bots by chatting with the @BotFather, the father of all Bots. The procedure is simple and takes just a minute. Open the Telegram app with your smartphone (it is better using the web version at https://web.telegram.org/, because you’ll have to copy/paste an access token) and start a chat with the @BotFather. So, just press the “write” button and type “@BotFather” (without quotes) as recipient, even if it does not appear at first in your contact list.

Here is what you need to tell the @BotFather:

/newbot
the-bot-name
the-bot-id

Just replace “the-bot-name” with a friendly name for your bot and “the-bot-id” with the unique name (use your fantasy, a lot of names are already taken). Here I attach two screenshots to show the creation of a bot.

| |

You need to annotate the authorization token given by the @BotFather in the last message. It is the key that lets your application act as the Bot (don’t try using the one in the picture, I revoked it :D).

Looking at the Code

Now the programming part. Just clone the skeleton of the bot using the following command:

git clone https://github.com/nicolaferraro/telegram-quickstart.git

Git will create the structure of the quickstart project. You need just to customize 2 files in order to create your own chat bot.

I'm now going to explain all the relevant parts, you can find the full code in the github repo https://github.com/nicolaferraro/telegram-quickstart. First, the Maven pom.xml, that imports the camel-telegram-starter component and all the related libraries.

    <!-- ... -->
    <properties>
        <camel.version>2.18.0</camel.version>
        <spring-boot.version>1.4.1.RELEASE</spring-boot.version>
    </properties>

    <!-- Some dependency management -->
    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>org.apache.camel</groupId>
                <artifactId>camel-spring-boot-dependencies</artifactId>
                <version>${camel.version}</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
            <dependency>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-dependencies</artifactId>
                <version>${spring-boot.version}</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>

    <dependencies>

        <!-- Import the camel-telegram dependency -->
        <dependency>
            <groupId>org.apache.camel</groupId>
            <artifactId>camel-telegram-starter</artifactId>
        </dependency>

    </dependencies>
    <!-- ... -->

Second, we define a camel route to describe how messages should flow inside the application. It is defined using the Java syntax in the CamelRoute class.

package com.example;

import org.apache.camel.builder.RouteBuilder;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;

/**
 * This class is responsible for routing the messages from and to the Telegram chat.
 */
@Component
public class CamelRouter extends RouteBuilder {

    @Autowired
    private Bot bot;

    @Override
    public void configure() throws Exception {

        from("telegram:bots")
        .bean(bot)
        .to("telegram:bots");

    }
}

As you see, all incoming messages are routed to a bean named Bot and the replies generated by the Bot are routed again to the source chat. Note that the Telegram component (starter version) requires that you specify the Telegram authorization token inside the application.properties file.

# To keep the application alive
camel.springboot.main-run-controller=true

# Create a new bot using the BotFather, then put here your token
camel.component.telegram.authorization-token=replace-me-with-your-telegram-token

You must change the file to put your own authorization token, the one given by the @BotFather.

To complete the picture we just need to write down the Bot logic, by implementing one method in the Bot class.

package com.example;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.stereotype.Component;

/**
 * This class contains the chat-bot logic: use your fantasy to implement your own Bot.
 */
@Component
public class Bot {

    private Logger log = LoggerFactory.getLogger(getClass());

    /**
     * This method processes incoming messages and return responses.
     *
     * @param message a message coming from a human user in a chat
     * @return the reply of the bot. Return null if you don't want to answer
     */
    public String process(String message) {
        if (message == null) {
            return null; // skip non-text messages
        }

        log.info("Received message: {}", message);

        return "Why did you say \"" + message.replace("\"", "-") + "\"?";
    }

}

You can leave the current implementation as it is and the Bot will reply to every message with a question (just like a 3-years old baby!!), but I’d better write something else :)

You can change the signature of the method to be able to receive and send also media content (like photo and video) to create a more interesting Bot. Look at the documentation for more info.

Running the Bot

You just need Maven and Java 8 to run the Bot. From the root of the project (where the pom.xml is located), type the following command in a shell to run the application:

mvn spring-boot:run

Maven will download all the dependencies and run your code.

Leave the application running and contact your bot with your smartphone (just write a message to @your-bot-id, using your Telegram app), it will reply to all your messages.

If you want to package the application in a single jar, you can run the following command:

mvn package

It will create a single jar with all the dependent jars nested inside it (note: this is different from a classic uber-jar, it is closer to a .war file). You’ll find the packaged jar into the target directory and you can just run it using:

java -jar target/telegram-quickstart-1.0.0-SNAPSHOT.jar

Have fun!

Apache Camel meets Spring-Boot

Nicola Ferraro — Sun, 25 Sep 2016 00:00:00 GMT

Next version of Apache Camel (2.18.0) is about to come with a long list of new features and components.

Even if previous release of Camel was 2.17.3, the new version is actually a major release, containing a lot of improvements over previous version.

Now Camel compiles against Java 8 and most part of the code has been allowed to use constructs like lambdas, but also the API interfaces have been adapted to support lambdas in the user code, eg. in transformations.

Camel 2.18.0 comes with a new set of components ideal for building microservices. The list contains the new circuit breakers in camel-hystrix, load balancing with camel-ribbon, distributed tracing with camel-zipkin. These are just some of the new components of a long list that include also my pet component camel-telegram, to quickly create applications integrated with the popular messaging app.

A Camel Spring-Boot application now automatically provides reliable health checks at the endpoint /health, leveraging the Spring-Boot actuator module. This is a fundamental requirement for any microservice living in a cloud environment, because it allows the cloud platform (eg. Kubernetes) to detect any anomaly and take corrective actions.

Indeed, one of the big news in the context of Microservices is that much effort has been made to improve Camel's compatibility with Spring-Boot.

Spring-Boot is one of the top trending frameworks in the Microservices era, one of the fundamental building blocks for creating cloud enabled applications, at least in the Java universe. Camel integration with Spring has ancient roots and recent releases included some features that simplified the usage of Camel into Spring-Boot applications, but this time we have done much more.

Camel Spring-Boot Starters

We have done a major refactoring of the code, separating the pure components from platform related features. In the case of Spring-Boot, this led to the creation of Component Starters.

A component starter is just like a normal Camel component, but it's optimized for the Spring-Boot framework. Maven users should just add a -starter suffix to the name of the component to switch to its starter version:

...
  <dependency>
    <groupId>org.apache.camel</groupId>
    <artifactId>camel-http-starter</artifactId>
    <version>2.18.0</version>
  </dependency>
...

But, what does it mean being optimized for Spring-Boot?

Spring-Boot is an opinionated framework, meaning that developers using it are choosing a specific set of technologies, products and patterns to develop their applications. One of the implications is that Spring-Boot "requires" you to use specific versions of some third party artifacts to play well in the ecosystem.

Starters bring with them a set of transitive dependencies perfectly compatible with the ones expected by the Spring-Boot framework. Versions are aligned with latest version of Spring-Boot (currently 1.4.1.RELEASE), logging is configured to match the Spring-Boot standard logging framework, and so on. Starters are meant to be included in a pom.xml and used, without hassle.

A new BOM (Bill of Material) is provided with the new version of Camel: camel-spring-boot-dependencies. It has no conflicts with the Spring framework's spring-boot-dependencies BOM and it replace the classical Camel BOM (camel-parent) in Spring-Boot applications.

Configuration of Starters

One of the most important features included in the starters is the possibility to configure Camel components using the new Spring-Boot configuration mechanism.

Once a starter is included in the pom.xml file, the IDE will suggest all the configuration options available for that starter.

This helps configuring the application easily, using the application.properties file.

A related feature that spring-boot offers out of the box is the possibility to override such configuration using environment variables or Java options. This is a great feature when you plan to run the application in a cloud platform like Kubernetes or Openshift, since you can exploit it to override some environment related settings using environment variables set up by the cloud platform automatically (eg. external services). A Kunernetes' ConfigMap can also be bound to environment variables, providing a way to inject external configuration into the application.

Trends are changing but camels are creatures that can adapt well to every environment!

Hot Reconfiguration of Microservices on Kubernetes

Nicola Ferraro — Sun, 23 Oct 2016 00:00:00 GMT

Spring Cloud Kubernetes is a fantastic project from the Fabric8 team that contains a lot of useful tools for building spring-boot based microservices. Version 0.1.3 includes a new feature that allows changing the configuration of a microservice at runtime using Kubernetes config maps.

Spring-boot applications are much appreciated for their simple configuration mechanism, based on properties or yaml files. There is plenty of ways to override the default settings of an application, usually provided in the bundled "application.properties" file. Environment variables take precedence over the provided configuration and this is particularly useful to automatically configure hosts and ports of external services inside Kubernetes. Other ways, like command line options or external configuration files, are supported out of the box.

By just including Spring Cloud Kubernetes into the microservice, the application configuration can be also provided using Kubernetes config maps. What's new is that the application can be configured to listen for changes in the Kubernetes namespace and apply the new configuration at runtime as soon as the config map is updated. Reconfiguration is applied immediately and only the configurable beans are reloaded to reflect the new settings. Respect to other solutions, hot reload does not require downtime because the application is not restarted (nor the application context).

I'll show the usage of this new feature with a quickstart. Let's create a Camel microservice (the feature works also without Camel) with a simple route:

@Component
public class Route extends RouteBuilder {

    @Autowired
    private RouteConfig config;

    @Override
    public void configure() throws Exception {

        from("timer://tick?period=3000")
                .id("generation")
                .transform().simple("message-${header.CamelTimerCounter}")
                .recipientList().method(config, "getRecipients");

        from("direct:async-queue")
                .id("async-queue")
                .log("${body} pushed to the async queue");

        from("direct:mail")
                .id("mail")
                .log("${body} sent via mail");

        from("direct:file")
                .id("file")
                .log("${body} written to a file");

    }
}

I'll show here only the relevant parts, but you can just clone or fork my spring-boot-configmap-quickstart repository and run it.

git clone git@github.com:nicolaferraro/spring-boot-configmap-quickstart.git

The route generates every 3 seconds a dummy message and pushes it to a list of recipents (this EIP is indeed called "recipent list"). Here there are 3 available endpoints where the message can be pushed to: direct:async-queue, direct:mail and direct:file. They are fake endpoints for the purposes of this example, as they just log every message with a different surrounding text.

The list of recipents is chosen using the getRecipients method of an external bean. Here's the bean:

@Configuration
@ConfigurationProperties(prefix = "route")
public class RouteConfig {

    /**
     * The recipient endpoint.
     */
    private String recipients;

    public RouteConfig() {
    }

    public String getRecipients() {
        return recipients;
    }

    public void setRecipients(String recipients) {
        this.recipients = recipients;
    }
}

The RouteConfig bean is a typical spring-boot configuration bean with a property of type String named recipients that contains the list of target endpoints for every message.

I've included the configuration in a bean annotated with @ConfigurationProperties because all the beans with such annotation will be refreshed automatically.

The list of recipients is given as comma separated value string, as shown in the application.properties file below.

# Application data
spring.application.name=configmap-example

# Enable auto-reload
spring.cloud.kubernetes.reload.enabled=true

# Using an internal port for health checks. 
# Always a good choice to avoid exposing sensitive paths.
management.port=8000

# Camel recipients default configuration.
# Can be overridden using configmaps.
route.recipients=direct:async-queue,direct:mail

The spring.application.name property serves to give a name to the application: the name is configmap-example. The reason why this is important is because the application is automatically configured to override the default configuration if a Kubernetes config map named configmap-example is present. A simple flag named spring.cloud.kubernetes.reload.enabled enables the auto-reload feature on configmap change. With this property enabled, the config map can be added at a later time and the application will refresh its status.

The route.recipients property is the one we want to override with a configmap. The default value includes two target endpoints for the messages: direct:async-queue and direct:mail.

Running the example in Kubernetes

You need a running Kubernetes or Openshift cluster to run the example. If you don't already have one, I suggest you using Minikube or Minishift, to startup a development Kubernetes or Openshift cluster in 2 minutes.

Note for Openshift:

If you're using Openshift (any kind of cluster, also Minishift), you need to give permissions to the running application to listen for changes in the project. This can be obtained by giving the view permission to the default service account. It is the service account responsible to run all the pods by default. Just login, switch to the project where you want to deploy the application and type:

oc policy add-role-to-user view --serviceaccount=default

The command above applies to the service account named default in the current project only.

Now just execute the following Maven goal to run the quickstart:

mvn clean install fabric8:run

The fabric8:run goal will deploy the quickstart to your local Kubernetes/Openshift instance and attach the console to the logs of the created pod. The output should be something like this (after the initialization):

[           main] m.nicolaferraro.quickstarts.Application  : Started Application in 5.902 seconds (JVM running for 9.15)
[ - timer://tick] async-queue                              : message-1 pushed to the async queue
[ - timer://tick] mail                                     : message-1 sent via mail
[ - timer://tick] async-queue                              : message-2 pushed to the async queue
[ - timer://tick] mail                                     : message-2 sent via mail
[ - timer://tick] async-queue                              : message-3 pushed to the async queue
[ - timer://tick] mail                                     : message-3 sent via mail

Leave the application running. We are going to change the routes dynamically.

Changing the configuration

Currently an application named configmap-example is running in our Kubernetes and it is listening for config maps with the same name in the current namespace. Let's create a config map with that name to see what happens.

Create a file named configmap.yml with the following content:

kind: ConfigMap
apiVersion: v1
metadata:
  # Must match the 'spring.application.name' property of the application
  name: configmap-example
data:
  application.properties: |
    # Override the configuration properties here
    route.recipients=direct:async-queue,direct:file,direct:mail

As you see, we are going to change the route.recipients property to add the direct:file recipient. The config map is named after the spring application. To create it, open a new terminal and type the following command if you're using Kubernetes:

kubectl create -f configmap.yml

# For Openshift
# oc create -f configmap.yml

The config map should be created without problems. Now look at the program running in the other terminal. You have triggered a configuration change.

[ - timer://tick] async-queue                              : message-22 pushed to the async queue
[ - timer://tick] mail                                     : message-22 sent via mail
[ - timer://tick] async-queue                              : message-23 pushed to the async queue
[ - timer://tick] mail                                     : message-23 sent via mail
[default.svc/...] .r.EventBasedConfigurationChangeDetector : Detected change in config maps
[default.svc/...] .r.EventBasedConfigurationChangeDetector : Reloading using strategy: REFRESH
[default.svc/...] s.c.a.AnnotationConfigApplicationContext : Refreshing org.springframework.context.annotation.AnnotationConfigApplicationContext@71cadf38: startup date [Sun Oct 23 08:44:57 UTC 2016]; root of context hierarchy
[default.svc/...] trationDelegate$BeanPostProcessorChecker : Bean 'configurationPropertiesRebinderAutoConfiguration' of type [class org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration$$EnhancerBySpringCGLIB$$bf2575c9] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
[default.svc/...] b.c.PropertySourceBootstrapConfiguration : Located property source: ConfigMapPropertySource [name='configmap.configmap-example.bbb']
[default.svc/...] b.c.PropertySourceBootstrapConfiguration : Located property source: SecretsPropertySource [name='secrets.configmap-example.bbb']
[default.svc/...] o.s.boot.SpringApplication               : The following profiles are active: kubernetes
[default.svc/...] s.c.a.AnnotationConfigApplicationContext : Refreshing org.springframework.context.annotation.AnnotationConfigApplicationContext@18d995cf: startup date [Sun Oct 23 08:44:57 UTC 2016]; parent: org.springframework.context.annotation.AnnotationConfigApplicationContext@71cadf38
[default.svc/...] o.s.boot.SpringApplication               : Started application in 0.469 seconds (JVM running for 92.617)
[default.svc/...] s.c.a.AnnotationConfigApplicationContext : Closing org.springframework.context.annotation.AnnotationConfigApplicationContext@18d995cf: startup date [Sun Oct 23 08:44:57 UTC 2016]; parent: org.springframework.context.annotation.AnnotationConfigApplicationContext@71cadf38
[ - timer://tick] async-queue                              : message-24 pushed to the async queue
[ - timer://tick] file                                     : message-24 written to a file
[ - timer://tick] mail                                     : message-24 sent via mail
[ - timer://tick] async-queue                              : message-25 pushed to the async queue
[ - timer://tick] file                                     : message-25 written to a file
[ - timer://tick] mail                                     : message-25 sent via mail

Now each message is also pushed to the direct:file endpoint, as required by the config map. And the change happens on the fly as soon as you create the config map and without any downtime.

You can also change again the config map from the Openshift console or the command line:

kubectl edit configmap configmap-example

# For Openshift
# oc edit configmap configmap-example

A vi-like editing screen will appear and let you change the contents.

In a real scenario

A common use case for this feature is creating a per-environment configuration, allowing you to use eg. different settings in the testing, staging and production environments. Plus, the feature allows you to change the configuration at runtime.

It's unlikely that you want to change the configuration of an application using vi in production (altough I've done that multiple times in the past XD).

Configuration management need a special care. For example, one might want to:

Track all the changes done to the configuration to be able to revert them;
Track the users that changed the configuration;
Authorize only certain users to change the configuration;
Peer review all the changes to avoid mistakes.

You can obtain this level of control by including the configmap in a git repository. There can be a single repository for all the configmaps, or one per configmap, you decide.

For the current example, I've provided an example of git repository with a config map ready to be deployed by the Fabric8 Maven Plugin. Instead of creating the config map file by yourself, you can just have cloned my repo and installed it:

git clone git@github.com:nicolaferraro/spring-boot-configmap-settings.git
cd spring-boot-configmap-settings
mvn clean install fabric8:deploy

Of course, this is not what you're going to do in production. You should instruct Jenkins to deploy the config map to your Kubernetes as soon as it changes. There are multiple ways to do so. Fabric8 has a fully featured Jenkins pipeline system that can be configured to accomplish this task with 2 clicks. Openshift Enterprise 3.3 is also able to use Jenkins pipelines.

Settings and limitations

The reload feature can also be configured to completely restart the application context for the cases where a refresh is not enough. It can also shutdown completely the pod to let the replication controller restart a new one (for the extreme cases). Check the documentation for the other available options.

When using the refresh mode, you should ensure no properties are deleted between two subsequent versions of a config map. Properties can be added, changed but not removed. This issue is related to the refresh mechanism of Spring Cloud and prevents you from using lists in configuration (elements cannot be removed from a list). If you want to use lists or to be able to remove properties, you need to change the reload level in the application.properties file:

spring.cloud.kubernetes.reload.strategy=restart_context

This option causes a short downtime because the application context is restarted whenever the config map changes, but it's still better than restarting the JVM.

The feature can be also used with Kubernetes secrets. Take a look at spring-cloud-kubernetes on Github.

Extending DevOps to Big Data Applications with Kubernetes

Nicola Ferraro — Fri, 10 Mar 2017 00:00:00 GMT

I've talked at Voxxed Days Buchares 2017 about how to apply some DevOps practices to Big Data applications by leveraging the power of Kubernetes and Openshift.

DevOps, continuous delivery and modern architectural trends can incredibly speed up the software development process. Big Data applications cannot be an exception and need to keep the same pace. In this talk, Nicola Ferraro (Red Hat) will show an overview of the next generation cloud-native Big Data systems based on Docker and Kubernetes. Switching from static Big Data platforms to infrastructure as code allows releasing robust applications in minutes by minimizing the gap between all the environments (development, testing, staging and production), a typical problem in Big Data application development. The presentation goes through the full release cycle of a Spark application alongside a fleet of microservices exchanging data in near real time. A fully featured continuous delivery pipeline, provided by the Fabric8 platform, will automate every step of the build and release process. Spark clusters will come to life from nowhere and disappear when they are no longer needed...

The slides are available here:

And here's the recording:

Cloud Native Applications on Kubernetes: a DevOps Approach

Nicola Ferraro — Sat, 06 May 2017 00:00:00 GMT

I've talked at Voxxed Days Ticino 2017 about building Cloud Native applications on Kubernetes and Openshift.

The slides are available here:

The slides are in English, but the talk's abstract is in Italian (that is also the language used in the talk).

Kubernetes è la piattaforma open source per l’orchestrazione di container Docker, nata dall’esperienza Google, che sta diventando lo standard “de facto” per applicazioni cloud native. Lo sviluppo di applicazioni su Kubernetes offre dei vantaggi notevoli, poiché è la stessa piattaforma ad occuparsi di aspetti cruciali della vita del software: deployment, service discovery, load balancing, self-healing, auto-scaling, rolling upgrades e molto altro.

Kubernetes, inoltre, è in grado di fornire questi servizi a qualsiasi tipo di applicazione: dal front-end web Node.js alle applicazioni Big Data sviluppate con Apache Spark!

In questo talk vedrete gli aspetti più importanti dell’architettura di Kubernetes e le principali risorse che la piattaforma mette a disposizione. Vi saranno mostrati alcuni metodi pratici per sviluppare e distribuire applicazioni su Kubernetes e Openshift Origin (“Kubernetes on steroids” di Red Hat). Attraverso gli strumenti di sviluppo del progetto Fabric8, vedrete inoltre come sia possibile applicare delle pratiche DevOps, quali continuous integration e continuous delivery, per creare applicazioni cloud sempre più robuste e automatizzarne il processo di rilascio.

And here's the recording (Italian):

Integrating Applications: The Reactive Way

Nicola Ferraro — Tue, 20 Jun 2017 00:00:00 GMT

It has been a pleasure to give a talk at JBCNConf, one of the biggest conferences about Java in Europe.

I've presented the work that we've been doing recently on reactive streams in Apache Camel, with a fancy introduction on the reactor pattern and a lot of reasoning on end-to-end backpressure.

I've been really happy to see my name in the "top 10" of speakers at the end of the conference. I feel that people liked the way I presented things, but this is also a confirmation that the reactive topic is hot and Camel is going in the right direction.

The slides are available here:

And here's the recording:

Creating Clustered Singleton Services on Kubernetes

Nicola Ferraro — Tue, 17 Oct 2017 00:00:00 GMT

Creating a cluster of related containers is really easy with Openshift and Kubernetes. Resources such as Deployment support scaling to multiple instances natively. Services and load balancers are provided for free. But any time you create a cluster of containers, as soon as your application becomes more complex than a "Hello World!", there's a question that often arises: "How can I run a single instance of a component in the whole cluster?"

Why do you need a singleton service?

Think to a common scenario where you have an application running on Openshift with n replicas. It's not difficult to imagine cases where you need a singleton service.

Scenario 1

Suppose that you want to do scheduled operations on a database. For sure, you don't want all the pods of the same application do the same batch operation together.

For this task there are Kubernetes CronJob resources, but if you don't want to start a different pod for every task you create, they are not the perfect solution. Think to the case where you need to schedule 10 different jobs. CronJob resources require some additional effort from a management point of view that you may not want to spend.

There are also many cases where you need to access your core business resources in the job, so if you package the scheduled operation into a external resource, you then need a way to call your business logic from outside.

Separating your application into multiple services is a core principle of the microservices architecture, but you should split your business logic according to domain rules, not technical reasons.

You may want to schedule tasks in your application. E.g. in spring-boot you can use the @Scheduled annotation:

@Component
public class ScheduledTasks {

    @Scheduled(fixedRate = 5000)
    public void doTask() {
        // do something
    }
}

But if you do this, each running pod will call it's doTask method. And you don't want this.

Scenario 2

Suppose that you need to poll data from a remote REST API and process them. But you cannot do it concurrently, because the API supports a single consumer. For example, Telegram (the popular messaging app) has a polling api that does not support multiple consumers on the same chat id.

The first solution you may think about is running another pod on a deployment with replicas=1. It will consume data from Telegram and push messages to a queue/topic.

But is replicas=1 equivalent to singleton in the Kubernetes world? NO!

Kubernetes will try to reach the goal of always having a single instance of your application, but that is a target, not an invariant. There are tons of situations where you can have multiple instances of your application, even if you asked for a single one. For example, in case of network partition between a node that run your pod and the Kubernetes API server, Kubernetes can decide (after a timeout) that the node is no more ready and reschedule the pod on another node: the result is 2 pods running concurrently, even if you asked for one.

These situations are common in case of failure of some pieces of the infrastructure. If having a singleton instance of a service is a requirement, you cannot rely on the replicas=1 setting.

You now may think that StatefulSets solve the situation, but (first) you end up running a stateless application on a stateful container model, with its limitations (e.g. missing rolling upgrades, limited auto-scalability, limited...). And (second) you cannot scale out the StatefulSet to multiple replicas, e.g. in order to speedup parts of your application while keeping only the REST-polling part singleton.

Scenario 3

Suppose that you want to expose multiple istances of your REST services, but you want that certain operations are executed in serial order by a single piece of code.

Setting replicas=1 does not solve the problem, as you've seen in scenario 2. So, how will you do?

You can use a clustered singleton.

Now use your imagination. There are a lot of other cases...

How can you create it?

Camel 2.20 has been released few days ago. If you have time, you can take a look at the huge list of release notes. Or, for something more human friendly, there's this post by Claus Ibsen.

One of the improvement introduced in Camel 2.20 is the new Camel Cluster Service (thanks to Luca Burgazzoli). We designed some general purpose interfaces that can be used to receive simple notifications when your specific instance of a service becomes the leader/master of the whole cluster. The cluster here is an abstract term: you define it. But often in Kubernetes/Openshift, a cluster is made by all the pods that compose the same deployment.

Camel supports multiple cluster services: atomix, consul, file, kubernetes and zookeeper (and also jgroups, added at the last minute by Andrea Tarocci). The kubernetes cluster service is just one of them and we'll use it to create a clustered singleton easily.

I've seen this before...

In case you were asking: this is the successor of the Camel Master Route concept available in JBoss Fuse. We'll see a new Camel master route shortly. But the new service is much more powerful than the classic master component.

Yes, it will be available in next version of Fuse (Fuse 7.0, most probably).

And yes: the initial code for the Kubernetes leader election has been donated to Apache Camel by the Fabric8 team from the camel-master project.

And the original idea comes from the Kubernetes code.

Ok, now show me the code!

The complete code is present on my Github repo nicolaferraro/camel-leader-election.

You need just 3 dependencies in the pom.xml file:

    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter</artifactId>
        </dependency>
        <dependency>
            <groupId>org.apache.camel</groupId>
            <artifactId>camel-master-starter</artifactId>
        </dependency>
        <dependency>
            <groupId>org.apache.camel</groupId>
            <artifactId>camel-kubernetes-starter</artifactId>
        </dependency>
    </dependencies>

We are running Camel on spring-boot and we also need the camel-master-starter to run Camel master routes and also camel-kubernetes-starter to tell Camel that we want to use the Kubernetes cluster service (that obviously works also on Openshift).

A bit of configuration is necessary. Here's the application.properties file:

# ...
camel.component.kubernetes.cluster.service.enabled=true

camel.component.kubernetes.cluster.service.cluster-labels[group]=${project.groupId}
camel.component.kubernetes.cluster.service.cluster-labels[app]=${project.artifactId}

Note: remember to enable resource filtering in the pom.xml.

The first line enables the Kubernetes cluster service. The other lines tell the cluster service that the cluster is composed of all pods having a Pod Label named group equals to the current Maven project groupId and a label named app equals to the Maven artifactId. This is the default mapping used by the Fabric8 Maven Plugin when creating Kubernetes/Openshift resources.

Now you can just write a Camel master route:

from("master:lock1:timer:clock")
  .log("Hello World!");

Note that this is a simple Camel route prefixed with master:lock1. Here lock1 is the lock name used for synchronization among all pods: only the pod that is able to get the lock on lock1 will start the route (implementation details later).

Singleton services are not limited to Camel routes: you can also create a generic singleton service!

For example, you can declare a service like:

public class CustomService {

    public void start() {
        // called on the leader pod to *start* the service
    }

    public void stop() {
        // called on the leader pod to *stop* the service
    }

    // ...
}

You can bind the service to the global CamelClusterService instance on your application:

@Configuration
class BeanConfiguration {

    @Bean
    public CustomService customService(CamelClusterService clusterService) throws Exception {
        CustomService service = new CustomService();

        clusterService.getView("lock2").addEventListener((CamelClusterEventListener.Leadership) (view, leader) -> {
            // here we get a notification of a change in the leadership
            boolean weAreLeaders = leader.isPresent() && leader.get().isLocal();
            
            if (weAreLeaders && !service.isStarted()) {
                service.start();
            } else if (!weAreLeaders && service.isStarted()) {
                service.stop();
            }
        });
        return service;
    }

}

And that's all. The service will start only on one pod at time.

Note for Openshift users: the lock mechanism assumes that you're able to modify Kubernetes resources from whithin your pod. This is normally forbidden, unless you give special permissions to the pod. The quickstart is already configured with a special ServiceAccount and RoleBinding (see the fragments here), so you don't need to add them manually.

Running the quickstart

Connect to a Openshift instance. You can use Minishift for local development.
Clone the repo:

git clone git@github.com:nicolaferraro/camel-leader-election.git

Deploy:

# Enter the project dir
cd camel-leader-election
mvn fabric8:deploy

The deployment is already configured to run 3 istances of the application pod. If you enter the log (run minishift console to se the logs in the UI), you'll see that "Hello World!" is printed continuously in only one pod. If you kill that pod, after some seconds, another pod will start to print "Hello World!".

Where's the magic (implementation details)?

Leader Election is a special kind of consensus in distributed systems and you know that consensus is hard to achieve. Fortunately Openshift and Kubernetes run a instance of Etcd and Etcd implements Raft and takes consistency very seriously. So the idea: why don't we use Etcd to do leader election?

Not so easy. We cannot use Etcd directly inside Kubernetes, it's not there for application purposes. But we can use Kubernetes API to create/change resources such as ConfigMaps... and ConfigMaps are stored in Etcd...

There's a kind of optimistic locking mechanism in Kubernetes resources. Every resource has a metadata.resourceVersion field that can be used to ensure that only one pod changes a ConfigMap at time. If two pods try to change the same ConfigMap, only one of them can complete the operation correctly.

So, how can we leverage this feature?

The camel-kubernetes cluster service creates a ConfigMap named leaders (by default) and stores inside it the pod name of the current owner of each lock declared in the application (we used lock1 for the master route and lock2 for the custom service).

The protocol is (approximately) this:

Every pod tries continuously to acquire the lock
A pod can acquire the lock if the lock has not been created yet, or the current owner is not in the set of pods alive in the cluster (filtered with the labels we've set above)
When a pod acquires the lock, it must wait for x seconds before starting the clustered services
The pod that owns a specific lock must continuously check if he's still the owner of the lock
When a pod loses the lock because another pod acquired it, it must shutdown the clustered services immediately
When a pod cannot check if it's still the owner of a lock (because of a network partition or any other error), it must shutdown the clustered services after a timeout y < x from last successful check

The protocol outlined above ensures that only one master is present in the cluster at time. Even in case of network partition (e.g. a physical node is completely isolated from the rest of the cluster) of a Kubernetes node that was hosting the leader, the application will shut down the clustered services before a new election can take effect.

For more details, look at the code!

The Saga Pattern in Apache Camel

Nicola Ferraro — Wed, 25 Apr 2018 00:00:00 GMT

A new enterprise integration pattern has been added to Apache Camel (2.21.0): the "Saga" pattern. This article will show you why, when and how to use it in order to build robust and consistent applications in the cloud.

What is a Saga?

Although the name "Saga" has been widely misused recently, especially in the field of front-end development, in the context of distributed systems the term "Saga" always refers to a pattern for coordinating actions in remote services in order to obtain a consistent outcome. Achieving consistency is something really useful in practice but also difficult, especially in microservice architectures that tend to split the processing logic into multiple autonomous services, usually communicating over HTTP.

What do you mean by "Consistency"?

There's not a unique definition of the term "consistency", but here I refer to the widely accepted notion of "keeping the system as a whole in a valid state" (and by "valid" I mean: "respecting all business invariants").

Let's try to make it more concrete with an example. Suppose you have designed a system for a travel agency that allows you to buy a trip from A to B, using two external (or internal) services to buy tickets for train (service 1) or plane (service 2).

From a business perspective, your main invariant here is simply stated:

For any trip from A to B, the agency should buy tickets for all sub-routes (train or flight) that take the customer from A to B, or none of them

It means, in simple words, that when a user asks to go from London to Florence, the agency should buy a flight from London to Rome and a train from Rome to Florence. But it should never buy a train from Rome to Florence if the flight from London to Rome is full (or too expensive). We should buy the full trip or tell the user that the trip cannot be reserved.

But the problem is that we have two distinct systems for buying train tickets and reserving flights. How do we coordinate them?

Now, use your imagination, this is a very common problem and happens in many contexts!

How did people use to solve this problem? (Transactions)

This is a classic problem in distributed systems and the traditional way to solve it is... with transactions, of course!

I've been a consultant for many years and I can tell you that the most common architecture used by people is the big monolith with a gigantic relational database where multiple application modules store data (and sometimes also communicate by writing and reading data from the DB).

So if that is your architecture, the problem is simply solved by wrapping the calls to the flight module and the train module inside a transaction, so that if one of the two calls fails, the whole transaction is rolled back and your system preserves the invariant (transactions are indeed ACID, where the C means "consistency").

And what if the system is distributed? Well, there are distributed transactions. One of the most widely used specification for distributed transaction is XA and it allows applications and resources to execute actions in the context of a globally defined transaction that preserve the ACID properties. So you can have a ACID transaction that can span multiple databases or even multiple distributed services (the transactional context can be propagated across services).

So, why don't we just use distributed transactions?

There are many drawbacks to using distributed transactions.

The most evident one is that protocols and specifications for propagating a transaction context between two remote services are missing or under-developed when we want to connect services developed using different languages. Java is probably the language that has the best support for local and distributed transactions, but many other languages completely lack support for the most basic features.

But also in Java, if you use a NoSQL database instead of a good old RDBMS, chances that you can use (ACID) transactions are pretty low.

And what if you want to use an asynchronous toolkit like Rx-Java on Vert.x or Project Reactor on Spring-Boot 2? Now chances that you can use transactions are close to zero (although there's some work going on...).

But there are other reasons why one would avoid using distributed transactions in the context of microservice architectures or distributed systems in general. One reason is that a transaction often causes locks to be created on resources and when you have something unreliable between parts of your system, like the network, it may be the case that the locks are kept for a time longer than expected, creating also issues to other parts of the application.

This problem becomes more important when the two services that want to participate in a transaction belong to two distinct organizations. If you ask an architect to connect two services in a way that a problem in one service may also propagate to the other, that architect would probably think twice before doing such choice (or better three times). And distributed transactions are a kind of heavyweight link between services that one would like to avoid. A service running slowly increases the duration of global transactions also in the other services. A failure of one service may leave locks in the database of another service for too long.

In summary, if you want to use distributed transactions, you also need to trust the other side..

That's probably the main reason why people prefer to keep the boundaries of distributed transactions very narrow (and make use of distributed transactions only when necessary).

And now we have Sagas

You may have heard of sagas in a talk about domain-driven design (DDD) or event-sourcing. In fact sagas are a central part of both approaches. But a saga is not necessarily linked to that context, it's a generic pattern that can be used to coordinate remote services.

In fact, since version 2.21.0 of Apache Camel, it has become a enterprise integration pattern (EIP).

A Saga can be defined as:

A series of actions that belong to a business activity that should be all executed correctly by (remote) participants or otherwise compensated

Sagas fit more naturally into the way the natural world works (at least, our understanding of it). Let's take the previous example of the travel agency and suppose a user wants to reserve a trip that includes buying a train ticket and a flight.

A system developed with transactions would try to reserve both the flight and the train ticket at the same time. If it doesn't succeed, none of them will be booked.
A system using the saga pattern will try to reserve the train and the flight independently. In case of failure in one of the two reservations, the other one will be canceled.

For this reason, a saga does exactly what a human would do in this scenario: check if the full trip can be reserved, try to book, then cancel in the event of issues.

On the other side:

There is no such thing as a transaction in the real world

Yes, ask Walter White if you don't believe me...

Sagas in Apache Camel

Designing a saga is fairly easy in Apache Camel. Let's see an example.

I've designed a sample quickstart system with the following (microservice) architecture.

The full example is available here: https://github.com/nicolaferraro/camel-saga-quickstart.

You can see the following services:

API Gateway: a sample camel app that is the main entry point (and will continuously start sagas simulating real users)
Flight Service: a service that sells flights
Train Service: a service that sells train tickets
Payment Service: a service that allows both services to request payments
The big "C" in the middle is a LRA Coordinator (see below!)

The basic workflow is:

The saga starts
The gateway will reserve a flight (include payment)
The gateway will buy a train ticket (include payment)
Saga is completed (or compensated in case of issues)

But since I am evil, I've made the payment service to fail with 15% probability. This means that e.g. if the payment service fails during the flight reservation process, we should cancel the reservation. But in any case (succeeded or not), we should also cancel the train reservation if it has happened in the meantime.

It sounds complex to maintain all services (train, flight and payment) in a consistent state, but I'll show you it's fairly easy with the Saga EIP in Apache Camel.

So, show me the code!

Camel API gateway

Writing the main gateway route is straightforward:

from("timer:clock?period=5s") // <-- replace it with rest() definition to create a real gateway
  .saga() // <-- start a new saga
    .setHeader("id", header(Exchange.TIMER_COUNTER))
    .setHeader(Exchange.HTTP_METHOD, constant("POST"))
    .log("Executing saga #${header.id}")
    .to("http4://camel-saga-train-service:8080/api/train/buy/seat") // <-- action 1
    .to("http4://camel-saga-flight-service:8080/api/flight/buy"); // <-- action 2

// you can also .multicast() the two calls

And this completes the saga definition.

Ok, we need also to write the services, but writing them is also easy.

Camel Saga-aware Service

Let's take the train service as an example. A Camel saga-aware service can be implemented like this:

rest().post("/train/buy/seat")
    .param().type(RestParamType.header).name("id").required(true).endParam() // <- from caller
    .route()
    .saga() // <-- join the saga with "supports" propagation
        .propagation(SagaPropagation.SUPPORTS)
        .option("id", header("id"))
        .compensation("direct:cancelPurchase") // <-- the compensation endpoint
    .log("Buying train seat #${header.id}")
    .to("http4://camel-saga-payment-service:8080/api/pay?bridgeEndpoint=true&type=train") // <-- propagate saga to payment service
    .log("Payment for train #${header.id} done");

from("direct:cancelPurchase") // <-- The compensation route
    .log("Train purchase #${header.id} has been cancelled");

And that's it.

The compensation endpoint is just the endpoint that must be called in order to cancel a reservation. It's declared in the main route and invoked by Camel when it's necessary to compensate (Camel detects failures in any point of the Saga and reacts accordingly).

Look at the Camel Saga EIP documentation. There are many other options and features you can use. E.g.

Adding timeouts for saga completion
Receiving saga completion callbacks
Asynchronous saga execution

Running the example

The example can be run on Openshift. Just install Minishift, connect to it and use the following commands to start everything.

git clone git@github.com:nicolaferraro/camel-saga-quickstart.git
cd camel-saga-quickstart
oc create -f lra-coordinator.yaml
mvn clean fabric8:deploy

It leverages the fabric8 maven plugin.

How do Camel Saga and "Long Running Actions" work

If you arrived here you may be wondering how this saga machinery works under the hood.

Saga is a Camel EIP and can have different implementations. The base implementation keeps all data about the status of every saga in memory, so it's not fault-tolerant. If the application crashes, everything is lost. Also, propagation across services cannot be used with the base implementation.

But Camel 2.21.0 ships also a new module called camel-lra and a spring-boot starter (camel-lra-starter).

LRA stands for "Long Running Action", that is the name of a Microprofile specification under-development (see microprofile-lra). Its main implementation is already available and developed by the Narayana team in https://github.com/jbosstm/narayana/tree/master/rts/lra.

I've provided Openshift resources to install a basic LRA coordinator in the quickstart example (file lra-coordinator.yaml).

In spring-boot, the camel-lra service can be enabled by adding the camel-lra-starter module to the pom.xml file and the standard Spring-Boot application.yml file:


camel:
  service:
    lra:
      enabled: true
      coordinator-url: http://lra-coordinator:8080
      local-participant-url: http://my-url-as-seen-by-coordinator:8080/context-path

You need to set the camel.service.lra.enabled=true flag (so it will be the backing implementation of the .saga() EIP) and provide:

The coordinator base URL
The participant (this service) base url in order to receive callbacks from the coordinator

Note that these two settings are overridden when running the quickstart inside Openshift.

Yes, in case you're wondering, the coordinator and the participant services communicate over REST. This allows to easily extend support for LRA to other languages.

A LRA coordinator is a stateful component. Indeed it's the only stateful piece of the quickstart but you don't need to customize it. It's a generic component that your application will just use through the camel-lra module.

Being stateful and persistent, it adds fault tolerance to your application: your business invariants are eventually respected even in the case of failure.

A Brief Overview of the Protocol

Nothing magic happens under the hood. The protocol is fairly simple and is explained briefly by the following diagram:

<p style="text-align: center"> <img src="/images/lra-sequence-diagram.png" alt="LRA Sequence Diagram"/> <caption align="bottom"><i>Sequence diagram of a failed LRA saga</i></caption> </p>

Here service is the application starting the saga (the API Gateway in the previous example). Before doing any operation, it first creates a saga (startLRA operation) by communicating with the coordinator (REST).

Then, it can talk with other services: Service1 and Service2 in the picture. The Long-Running-Action HTTP header is used to propagate the LRA context to the downstream services.

Before Service1 and Service2 do any operation they join the saga by registering a compensating action (Camel URI) in the coordinator (addCompensator operation, another REST call).

Then, after the main actions are executed, the whole saga can complete normally (everything fine) or exceptionally (like in the diagram). In case of abnormal termination of the saga, the LRA coordinator will ensure that all registered compensating actions are called.

And what if a compensating action fails? Of course, the coordinator will retry again and again. This means that compensating actions must be idempotent and assume they might be called more than once.

Caveats

Idempotency and ... "Commutativity"

We have seen that a compensating action must be idempotent, because the LRA coordinator can call it multiple times, especially in case of network error or application unavailability.

But there's a more severe restriction that you need to respect in order to write correct services: a compensating action should be commutative w.r.t. the main action.

It means that, since we are in a distributed environment, sometimes the compensating action may be called by the LRA coordinator before the main action has completed (or has even started).

So, for example, your train service must be able to cancel a reservation even if such reservation is still not present in the system, and the reservation must be considered already canceled when (and if) it's created by a late running main action in the future.

Sometimes it can be hard to satisfy the commutativity restriction... but there can be alternative solutions...

A Bit of Q&A

Is the LRA Coordinator a single point of failure?

Not necessarily, e.g. the Narayana team is working to provide scalability and failover for the coordinator.

Isn't a Saga just a kind of distributed transaction?

No, a saga is composed of independent actions that are executed in different services during a long timespan (a transaction completes within few seconds, usually).
It's true that in some cases you need to register a completion-callback in the downstream service to finalize the action and this is similar with what happens with 2-phase-commit transactions. But this is not always necessary. E.g. the train service above has not registered any completion endpoint because once a seat is reserved by one customer, it cannot be reserved by another one, and it does not matter if the reservation is confirmed (saga completed) or not (saga in progress).

Resources

Have Fun!

Introducing Camel K

Nicola Ferraro — Mon, 15 Oct 2018 00:00:00 GMT

Just few months ago, we were discussing about a new project that we could start as part of Apache Camel. A project with the potential to change the way people deal with integration. That project is now here and it's called "Apache Camel K".

The "K" in the title is an obvious reference to Kubernetes, you may think. But there's also a less-obvious reference to Knative: a community project with the target of creating a common set of building blocks for serverless applications. Yes, going "serverless" is the base idea that inspired many architectural decisions for Camel K.

But, let's take this one step at time...

What is "Camel K"?

Apache Camel K is a lightweight cloud integration platform based on the Apache Camel framework. It runs natively on Kubernetes and Openshift and it's specifically designed for serverless and microservice architectures. When I say "it runs", I mean "it runs, now, you can try it!". Just visit the homepage of the project on Github and follow the instructions: https://github.com/apache/camel-k.

It's based on the "operator pattern" and leverages the Operator SDK to perform operations on Kubernetes resources (we define some custom resources beside the standard ones). The operator is written in Go while the runtime is JVM based and leverages all the 200+ components already available in Apache Camel.

Like Kubernetes and OpenShift, also Knative will be a target platform in the near future. Specifically, we're following the development of Knative Eventing and Knative Serving building blocks to provide a full support for them, once they reach an adequate level of maturity.

Camel K brings integration to the next level, but at the same time is a return back to the roots for the Camel project: the Enterprise Integration Patterns (EIP). Camel has been shaped around enterprise integration patterns since its inception and developers have created a DSL that often maps patterns in a 1:1 relationship.

I'm not exaggerating if I state that now: the Camel DSL is the language of EIP. It's (at least in my opinion) the language that expresses better most of the patterns that were present in the original "book of integration", but also other patterns that have been added by the community during all these years. And the community keeps adding patterns and new components in every release.

The idea of Camel K is simply stated: let people use those Enterprise Integration Patterns natively on Kubernetes, expressing them using the poweful Camel DSL.

If I should provide a architectural overview on Camel K, I'd draw the following diagrams:

Camel K is what you get if you take the integration DSL distilled from the rest of the framework and offer a way to write integration code that is executed directly on a cloud platform: it may be a "modern" cloud platform like Kubernetes or Openshift, or a "futuristic" cloud platform like Knative for serverless workloads (Knative can run on both OpenShift and Kubernetes and it's powered by Istio).

How does it work?

Speaking technically, the starting point is writing the integration code that we want to run. For example:

File: integrate.groovy

// expose a rest endpoint that routes messages to a Kafka topic
rest().post("/resources")
  .route()
    .to("kafka:messages")
    
// transform all messages and publish them on a HTTP endpoint
from("kafka:messages")
  .transform()... // any kind of transformation
  .to("http://myendpoint/messages")

Integrations can range from simple timer-to-log dummy examples to complex processing workflows connecting several external systems, but you write them using the same Camel DSL.

Actually, I talked about "a" Camel DSL, but many of you already know that the Camel DSL is not a proper standalone language, but a set of primitives that can be used in multiple programming languages. So far we support the following languages:

Groovy: it is probably the best language for scripting and it is currently the preferred language for writing Camel K integration code
Kotlin: yes, we support also Kotlin that offers a similar experience to Groovy
Java: it's the classic Camel DSL that many of you already know
XML: it's also a classic DSL adaptation and give its best when you plan to use one of the visual editing tools that already exist
JavaScript: yes, we support it as well!

Let's not complicate things too much and consider one of the simplest integration for the moment. The classic "Camel Hello World":

File: hello.groovy

from("timer:tick?period=3s")
  .setBody().constant("Hello World from Camel K!!!")
  .to("log:message")

A user that wants to run this integration on a cloud platform needs to download a small binary file that is available in the release page on the Camel K Github repository. It's called kamel.

The kamel binary contains also a command to prepare your Kubernetes cluster for running integrations. Just run:

kamel install

Will take care of installing the Camel K CRDs, setting up privileges and create the operator (see next) in the current namespace.

Important: in some cluster configurations, you need to be a cluster admin to install a CRD (it's a operation that should be done only once for the entire cluster). The kamel binary will help you troubleshoot. If you want to work on a development cluster like Minishift or Minikube, you can easily follow the dev cluster setup guide.

Once the cluster is prepared and the operator installed in the current namespace:

kamel run hello.groovy

And your done!

This is what happens under the hood:

The kamel tool will sync your code with a Kubernetes custom resource of Kind Integration named hello (after the file name) in the current namespace.

The Camel K Operator is the component that makes all this possible by configuring all Kubernetes resources needed for running your integration. I'll talk more about it later.

There exists also a dev mode that allow users to create integration incrementally and with immediate "buildless" redeploys. I think a demo is worth 1000 words.

Demo

The following video shows an example of what you can do with Camel K. It starts from the installation on a Minishift dev cluster, showing how to run a basic quickstart. Then it proceeds with a more complex example.

The second integration shown in the video will connect a Telegram bot (you can interact with it through the Telegram app on your mobile phone) to a Kafka topic that will be used to buffer messages and to throttle them. Messages will be received from Kafka again, filtered through one of the basic enterprise integration patterns available out-of-the box in Apache Camel, then forwarded to a external HTTPS endpoint.

All this is done in few minutes of video, and all integrations are created incrementally, leveraging the new build engine behind Camel K that requires just 1 second to redeploy the integration after each change.

Take a look:

Bringing "operators" to the next level

Operator SDK is the framework that makes it possible to create all Kubernetes resources needed for running the "Camel DSL script".

Operators are commonly used to install and configure applications or platforms on Kubernetes and Openshift. They are the digital version of the "human operator" that once installed the application in legacy environments, making sure that everything is in place for the application to run.

We brought this concept to the next level in Camel K. The operator is "intelligent" and knows what you want to run. It can understand the Camel DSL.

So, for example, if you define a REST endpoint using the Camel REST DSL, the operator will make sure that your integration is exposed to the outside and it will create a Service and a Route on OpenShift to expose it, or a Ingress on vanilla Kubernetes.

And in the future, it will have more administrative responsibilities:

It will expose webhooks on Kubernetes and activate the webhook registration with the external provider to receive data
It will subscribe to feeds that you need to manage in your integration
It will convert your "polling" routes into Kubernetes Cronjobs for optimizing resource utilization
It will use a optimized Camel runtime platform under the hood if your routes support it
It will... do a lot of useful things!

The operator will make sure that all you should do is writing your integration in a "Camel DSL Script", without caring about any administrative operation. That's what we mean with "intelligent operator".

What's next

There are a lot of things coming. We keep updated the projects section in the github repository with the current areas we're working on. Those include the already mentioned work on Knative and also a Web UI for Camel K that will really rock!

We love contributions! If you're interested in the project, there are a lot of ways to contribute.

Meet us in our dedicated Gitter room for more information.

Camel K on Knative: Agile Integration becoming Serverless

Nicola Ferraro — Mon, 10 Dec 2018 00:00:00 GMT

Knative is an open source project for adding serverless building blocks on Kubernetes and it's constantly gaining traction among developers. In Apache Camel K, we've been working hard to leverage all the new possibilities that it provides and this article will show the results we've achieved so far.

If you're not familiar with Camel K, you can read the introductory blog post. Camel K provides a lot of new features when running on Knative, but it also runs on plain "vanilla" Kubernetes and OpenShift.

Let's start with the demo, then you can read the rest of the article to understand better what we're doing.

So, what is Knative? (and what is not)

Knative is not a complete serverless platform. Nor it's going to become one.

Knative provides a collection of Kubernetes "Custom Resource Definitions" (CRD) together with related controllers that make it easier to build and deploy certain kind of applications
on Kubernetes.

Knative building blocks can be roughly divided into 3 major areas.

The Knative Build area provides custom resources for building applications from source code and producing container images.

The CRDs provided in Knative Serving area allow to define services that that scale automatically based on the load. Services can expose generic HTTP endpoints (such as REST): they are not necessarily "functions" (as in FaaS). These services can scale up when the load increases, but also scale down to zero when the load is absent for a certain amount of time. Services that are scaled down to zero don't consume physical resources and they are brought up again as soon as they need to serve a new request.

The last part of Knative I'm going to describe (and the most important one for our purposes) is the Knative Eventing area.

The Eventing Model

Knative Eventing provides a set of building blocks (CRD) for developing event-based applications.

The main block is the Channel. A channel is the abstraction of a publish-subscribe resource: you can push data into the channel, other services can subscribe to it to receive your data. The channel also decouples producers and consumers, so that producers can always push data into them and consumers can do processing when they are available.

Channels can be backed by different implementations. You can use a "in-memory" implementation, but also a complex one: currently Kafka, GCP PubSub or Nats. Those are referred as provisioners in the Knative area.

You may ask now: why don't we just use Kafka or Nats directly, what's the value added by Knative?

Just look at this schema.

<p style="text-align: center"> <img src="/images/post-knative-channel-model.png" alt="Knative Eventing Model"/> <caption align="bottom"><i>The Knative Eventing Model</i></caption> </p>

You can notice that the relationship between the channel and the service is somewhat reversed respect to a classical scenario.

Normally, you define a service so that when it starts it connects to a messaging broker and start pulling data from within that connection. In Knative eventing, you subscribe to a channel (using a specific Subscription CRD), then it's the channel that pushes events towards your service.

The nice thing of all this, is that your service just receives messages through incoming Cloudevents, without having to actively connect to the broker. Your service becomes passive. And since Knative provides other building blocks for autoscaling, your service scales up and down with the number of events in the channel to which it's subscribed. Scaling to zero is included.

Another important building block in the Knative Eventing space is the concept of EventSource. A event source is a resource with the role of pushing data into a Channel. E.g. you can use a GithubSource to foward GitHub generated webhook events into a channel.

Current plan in Knative is to start adding as many EventSources as possible and this is one of the places where Camel and Camel K can do the difference. Camel is already able to connect to 250+ different systems. And Camel is also famous for the variety of enterprise integration patterns (EIP) it implements: EIPs are really important in scenarios enabled by Knative Eventing.

How Camel K works

I've already described some of the internals of Camel K in the introductory blog post and there will be more blog posts from the Camel K developers in the next days (so, stay tuned!).

The basic idea behind Camel K is explained in the following diagram.

<p style="text-align: center"> <img src="/images/post-camel-k-operator.png" alt="Camel K CRD and Operator"/> <caption align="bottom"><i>Camel K CRD and Operator</i></caption> </p>

Camel K users write their integration code (the Camel DSL) in a script file, using Groovy, Kotlin, Java or even JavaScript or XML, and run it directly in the cloud platform.

What happens under the hood is that the script is wrapped into a "Integration" Custom Resource and added to a Kubernetes namespace. The Camel K Operator (based on operator SDK) will then detect the new Integration and materialize it into running containers. Normally Camel K materializes an Integration into a Kubernetes Deployment, but when running on Knative, it uses auto-scaling services.

What is really cool about Camel K is that it's able to materialize and startup integrations in few seconds. This helps a lot during the development phase, because you have immediate feedback on the code you're writing. The video accompanying the first blog post emphasizes this feature.

Camel K and Knative

Camel K fully supports Knative since version v0.1.0. I often tend to remind (so you don't forget it) that Camel K can also run on plain (= without Knative) OpenShift and "vanilla" Kubernetes, but without "serverless features".

An important thing of the Knative model is that, since Knative is not a serverless platform, but a set of building blocks, you can run on top of Knative even multiple serverless platforms. Knative does not only provide the building blocks for creating auto-scaling services, but also the building blocks for those platforms to communicate with each other (eventing).

Camel K, thus, is not intended to run as exclusive serverless platform on top of Knative, it's rather a serverless integration layer.

You are not expected to build generic functions with Camel, you can just use your FaaS platform to do it (provided that it works on top of Knative). But there are at least three places where using Camel K is the best choice you can do.

1. Camel K for creating Event Sources

Camel can easily push data into Knative channels, acting as event source.

We've added a new component in Camel K named "knative", that allows publishing and subscribing to knative channels. You can use a Knative endpoint at the end of any route to create a event source. And you can use any combination of the 250+ Camel components as starting point.

For example, in the demo you're going to see the following route:

from('telegram:bots/<put-here-your-botfather-authorization>')
  .convertBodyTo(String.class)
  .to('knative:channel/messages')

You can run it by simply executing kamel run telegram-feed.groovy. As already mentioned, kamel is used to simply wrap the code in a Kubernetes custom resource, while the materialization work is always accomplished inside Kubernetes by the Camel K Operator.

The simple script above generates a "event source": an integration that forwards all messages sent to a specific bot on telegram to the "messages" Knative channel.

2. Camel K for Enterprise Integration Patterns (EIP)

Camel K can also be used within Knative for its powerful enterprise integration patterns. You will see in the demo the following integration:

from('knative:channel/messages')
  .split().tokenize(" ")
  .to('knative:channel/words')

It is an example of one of the simplest EIP available in Camel, the splitter EIP.

You run this integration as usual: kamel run splitter.groovy.

In Camel you have tons of EIP that you can use out-of-the-box, even within Knative: content-based router, dynamic router, message filter, message transformation, recipient list... and many others.

3. Camel K for defining Integration Functions

Many times you need to notify an external system that an event occurred in the serverless space. In these cases, it's likely that you'll write an integration function that gets executed when a specific event is received from a channel.

In the demo, you'll se something like:

from('knative:channel/words')
  .to('slack:#camel-k-tests')

This integration snippet is simply forwarding text messages to a Slack channel, but it could be also be doing transformations or enriching content with external data.

Future Work

Camel K runs really well on Knative, but we want to provide a even better experience in the next months.

The value that Camel K adds to Knative is a easy way for people to write EventSources, EIP and Integration Functions. We need to enable people to just do that, but in a way that fits more closely the ideas that are being developed within the Knative space.

The roadmaps of Camel K and Knative are closely related.

Apache Camel K is already here. Grab it while it's hot!

Camel K 1.0 is here

Nicola Ferraro — Mon, 08 Jun 2020 00:00:00 GMT

Apache Camel K has made a lot of progress since its inception and we're now proud to announce the 1.0 release. We've been working hard in the past months to add more awesome features to Camel K, but also to improve stability and performance. This post contains a list of cool stuff that you'll find in the 1.0 GA release.

NOTE: this article has been first published on the Apache Camel's Blog.

First of all, if you're living under a rock and it's the first time you hear about Camel K, you can read some introductory blog posts here (1 - introducing camel k, 2 - camel k on knative) or look at the Apache Camel website that contains a Camel K section with a lot of material that is automatically generated from the Github repository.

User experience

Camel K development style is minimalistic: you need just to write a single file with your integration routes and you can immediately run them on any Kubernetes cluster. This way of defining things is common to many FaaS platforms (although Camel K is not a proper FaaS platform, but a lightweight integration platform) and it's technically difficult to provide IDE support, such as code completion and other utilities, to developers.

But now we've done it. The integration tooling team has created some cool extensions for VS Code that make the development experience with Camel K even more exciting. You don't need to remember the Camel DSL syntax, the IDE will give you suggestions and error highlighting.

<p style="text-align: center"> <img src="/images/ide-autocompletion.gif" alt="IDE Autocompletion"/> <caption align="bottom"><i>IDE Autocompletion</i></caption> </p>

Code completion works with Java code, but it's not only limited to it: you also have suggestions and documentation out of the box when writing the Camel URIs and property files. And you also have many options to run integrations and interact with them, all integrated in the IDE.

Just install the VS Code Extension Pack for Apache Camel to have all these features available.

Getting started tutorials

Good tools are fundamental to have a great development experience with Camel K, but then you need to learn what you can do with such a great power. We've created a new repository in the Apache organization that hosts getting started examples: the camel-k-examples repository.

So far we've added guides that drive you through:

01 Basic: Learn the basics of Camel K and some interesting use cases
02 Serverless APIs: How to design a serverless (i.e. auto-scaling, scaling to zero) API and run it in a few minutes

The basic quickstart is also available online, so you can have a look at how camel k works without installing anything on your laptop.

More tutorials are expected to come in the following months. You are also welcome if you want to help us by contributing your own. They are based on the VSCode Didact project, that provides an awesome user experience.

If you are looking for Camel K code samples that you can just pick and run using the CLI, the examples directory of the Camel K main repository contains a lot of them. You can also run them directly from Github:

kamel run https://raw.githubusercontent.com/apache/camel-k/master/examples/Sample.java

You can find ready-to-use examples written in different languages (e.g. XML, JavaScript and others).

Serverless

Serverless is the most important area where we're focusing the new developments in Apache Camel K, although, you should remember, you can have a wonderful Camel K experience even without serverless features. To enable the serverless profile in Camel K, you just need to have Knative installed.

In recent releases, we have added support for the most recent advancements in Knative, for example, Camel K is very well integrated with the Knative event broker and you can easily produce or consume events from it.

With 2 lines of code you can transfer events (e.g. generated by IoT devices) from your MQTT broker to the mesh:

bridge.groovy

from('paho:mytopic?brokerUrl=tcp://broker-address:1883&clientId=knative-bridge')
  .to('knative:event/device-event')

No kidding, you just need to write those two lines of code in a file and run it with kamel run bridge.groovy to push data into the Knative broker.

And you can also scale the Integration out (Integration is a Kubernetes custom resource, kubectl get integrations to see all of them) to have a higher throughput. Scaling here is manual because the source of events is a MQTT broker (but we've plans to put auto-scaling also in this scenario).

The Camel K embedded auto-scaling feature works really well when you want to react to some Knative events:

listener.groovy

from('knative:event/device-event')
  .to('http://myhost/webhook/random-id')

This integration is configured to receive all events with type=device-event and scales automatically with the load because it is materialized into a Knative Serving Service and automatically subscribed to the Eventing Broker via a Trigger.

It then receives a CloudEvent when your IoT devices produce something and scales down to zero if there's no data coming. You just need to create it (as before, just kamel run listener.groovy), all the remaining configuration is done automatically by the Camel K operator.

We've added much more features for having a better integration with the Knative ecosystem and we've also fixed some compatibility and performance issues that were present in previous versions. The user experience is now much smoother.

If you are a Knative YAML developer (!), instead of using Camel K directly, you also have the option to use Knative Camel Sources which are part of the Knative release. They are wrappers for Camel K integrations that are compatible with all the tools used by Knative developers (such as the kn CLI or the OpenShift serverless console). Sources in Knative can only push data into the various Knative endpoints, but not the other way around (i.e. they cannot be used to publish data from Knative to the outside). In Camel K you don't have this limitation: the Route is the fundamental building block of a Camel integration and you can do whatever you want with it.

Fast startup and low memory

We cannot say we're serverless without mentioning the work that we've been doing in improving the performance of Camel K integrations.

Starting from Camel 3.3.0 which is the default version used by Camel K 1.0.0, you can benefit from all improvements that have been made directly in the Camel core to make it much more lightweight. More in depth details of the Camel core improvements can be found the following blog series that highlights what has been changed in the 3.x Camel timeline to reduce memory footprint and speedup the startup time, which is foundamental when running integrations in a serverless environment: part 1, part 2, part 3, part 4.

But improvements are not only limited to the Camel core: we're doing much more. Several months ago we've started a new subproject of Apache Camel named "Camel Quarkus" with the goal of seamlessly running integrations on top of the Quarkus framework. As you probably know, Quarkus is able to reduce the memory footprint of Java applications and improve the startup time, because it moves much startup logic to the build phase. And Quarkus applications can also be compiled to a native binary, allowing a dramatic improvements in startup performance and very low memory footprint.

In Camel K 1.0.0 we support Camel Quarkus in JVM mode. A goal is to have also the in-cluster native compilation soon (for some DSL languages, such as YAML), in one of next releases!

To use Quarkus as underlying runtime, you just need to enable the Quarkus trait when running an integration:

kamel run myintegration.groovy -t quarkus.enabled=true

Quarkus is expected to be the default underlying runtime in the next release, and support for Standalone mode (via camel-main) will be deprecated and removed. This means that you won't need to enable Quarkus manually in the next releases, but you still need to do it in 1.0.

Fast build time

Every application running on Kubernetes needs to be packaged in a container image, but in Camel K you only provide the integration DSL and the operator does what it takes to run it, including building images directly in the cluster.

The operator manages a pool of reusable container images and if you redeploy your integration code, it does try to reuse existing images from the pool rather than building a new one at each change, because it takes some time to build a new one. It was 1 minute at the beginning...

But Kubernetes is moving so fast that you cannot solve a problem once and forget about it, you need to take care of it continuously. It happened that some of our third party dependencies that we used for doing builds in "vanilla Kube" has slowly degraded in performance up to a point where Camel K user experience was highly affected.

We decided to work harder on the build system in order to dramatically improve (again!) the build phase of Camel K integrations.

Build time can be be now measured in seconds in dev environments such as Minikube. A bunch of seconds, most of the times. This is more than a simple improvement!

Better CLI

The 'kamel' CLI is the main tool we provide to developers to run integrations. It's not a mandatory requirement: at the end, an Integration is a Kubernetes custom resources and you can manage it with any Kubernetes standard tool (e.g. kubectl). But the kamel CLI adds a lot of value for integration developers.

For example, if you're a Camel Java developer it's not super easy to remember the boilerplate that you have to write in order to instantiate a Camel route builder. Now you don't have to remember that:

kamel init Handler.java

You get a Java file with all the boilerplate written for you and you just have to write your integration routes.

It works also with all other languages: Groovy, XML, YAML, Kotlin and JavaScript. For example you can write:

kamel init foo.js

This way you get a simple route written in JavaScript.

It's not just that. Often Camel K developers need to add a lot of command line options to configure the final behavior of their integration. For example, you may want to add a custom library with the -d option or configure a trait with -t. E.g.:

kamel run -d mvn:org.my:lib:1.0.0 -d mvn:org.my:otherlib:2.0.0 -t quarkus.enabled=true Handler.java

Sometimes the number of command line parameters you've to add can become too many. For this reason we've added the possibility to specify them as modeline options in the integration file (done by adding a comment line with camel-k: as prefix).

Handler.java

// camel-k: dependency=mvn:org.my:lib:1.0.0 dependency=mvn:org.my:otherlib:2.0.0 trait=quarkus.enabled=true

// ...
// your routes here

Once the options are written in the file, you can run the routes with just:

// simply this, additional args are read from the file
kamel run Handler.java

The other options are taken automatically from the file modeline. The CLI also displays the full command to let you know what's running.

This kind of configuration is extremely useful in CI/CD scenarios because it allows you to have self-contained integration files and you don't need to change the pipeline to setup additional options. If you're curious about the CI/CD configurations, you can follow the tutorial about Tekton pipelines to have more information.

Monitoring and Tracing

Ok, you've finished level 1 of Camel K development and you want to make serious things. You're in a very good position because Camel K provides a lot of useful tools to add visibility on what your integration routes are doing.

Let's suppose you've a Prometheus instance in your namespace and you want to publish your integration metrics:

kamel run Routes.java -t prometheus.enabled=true

That's it. No need to setup services and labels to enable scraping. A default prometheus configuration file is also provided for the integration, with sensible defaults. Of course you also have the option to provide your own configuration for advanced use cases.

Now, let's suppose you want to see what your routes are doing and trace the execution flow of an integration. What you need to do is to install an opentracing compatible application in the namespace, such as Jaeger, and run the integration as:

kamel run Routes.java -t prometheus.enabled=true -t tracing.enabled=true

That's it again. The Camel K operator will add the camel-opentracing library and connect it to the Jaeger collector that is available in the namespace. Here again, advanced use cases are supported.

Master routes

Good old Camel users know why and when master routes are useful, but for those who are not familiar with the term, I'm going to provide a brief explanation.

Whenever you have an integration route that must be running, at any point in time, in at most one single Camel instance, you need to use a master route. Master routes can be declared by simply prefixing the consumer endpoint by the 'master' keyword and a name that will be used to create a named lock, e.g.

from('master:mylock:telegram:bots')
  .to('log:info')

It can be used to print all messages that are sent to your Telegram bot. Since the Telegram API support a single consumer only, you can guard the route with a master prefix to have the guarantee that there will be at most only one consumer at any given time.

If you're wondering how there can be two instances running of you deploy one, well, think just to when you change your code and need to do a rolling update: for some time there'll be two pods running in parallel. In some cases, you may decide to scale your service out but keep only one instance of a particular route among all the pods of your service. Or you may want to embed a master route in a Knative autoscaling service: in this case, the service can scale autonomously based on the load, but there'll be only one telegram consumer at any time.

Master routes work out of the box in Camel K, you just need to put a prefix in your endpoint uri. A leader election protocol based on Kubernetes APIs resource locks will be automatically configured for you!

CronJobs

All complex enough systems contain several scheduled jobs. This is especially true for that part of the system that handles integration with the outside.

Ideally, if you need to execute a quick periodic task, say, every two seconds, you would startup an integration with a route based on timer to execute the periodic task. E.g.

from("timer:task?period=2000")
  .to(this, "businessLogic")

But if the period between two executions, instead of 2 seconds ("2000" in the Camel URI, which is measured in milliseconds) is 2 minutes ("120000") or 2 hours ("7200000")?

You can see that keeping a container with a JVM running for a task that should be executed once every two minutes may be overkill (it is overkill for sure when the period is 2 hours). We live in a time where resources such as memory and CPU are really valuable.

So the Camel K operator automatically handles this situation by deploying your integration not as a Kubernetes deployment, but as a Kubernetes CronJob. This saves a lot of resources, especially when the period between executions is high. When it's time to run your integration code, a container starts, triggers the execution and then gracefully terminates. Everything is handled automatically by Camel K and Kubernetes.

There are cases when you don't want this feature to be enabled, for example, when your code makes use of in memory caches that is better to keep between executions. In these cases, you can safely turn off the feature by passing the flag -t cron.enabled=false to the kamel run command.

The Cron feature does not only work with the timer component. We've also added a cron component since Camel 3.1 that works really well in combination with the cron trait.

So you can also write the cron expression in the route directly:

from("cron:job?schedule=0/5+*+*+*+?")
  .to(this, "businessLogic")

In this case, a new pod with a JVM is started every 5 minutes to execute your scheduled task. For the remaining 4+ minutes you don't use any resource.

Transparency

Camel K does a lot of work for you when you run your integration code in the cluster and it's possible that you put some errors in the code that can block the deployment process. We've added a lot of visibility on the deployment process that now communicates with the users via Kubernetes events that are printed to the console when you use the CLI.

This way you're always notified of problems in the code and you can better understand what to fix to make your integration run.

How to try Camel K 1.0

The first step is to go to the Camel K release page on Github (or the official Apache release repository), download the kamel CLI for your OS and put it in your system path.

Installation is done usually using the kamel install command, but, depending on the kind of Kubernetes cluster you're using, you may need to execute additional configuration steps. The Camel K documentation contains a section about installing it on various types of Kubernetes clusters.

If you have trouble or you need to install it on a particular cluster that is not listed, just reach out in the Gitter chat and we'll do our best to help you.

Future

We've reached version 1.0.0 and this is a great milestone for us. But we are not going to stop now: we've big plans for the future and we'll continue to develop awesome new features.

We need your help to improve Camel K and we love contributions!

Join us on:

Gitter: https://gitter.im/apache/camel-k
GitHub: https://github.com/apache/camel-k

Event-driven serverless applications with Camel K

Nicola Ferraro — Thu, 09 Jul 2020 00:00:00 GMT

One of the talks I've most enjoyed doing in 2020. A DevNation Tech Talk explaining where do we see Camel K in the context of event-driven applications.

Serverless Integration on Kubernetes with Apache Camel K

Nicola Ferraro — Tue, 08 Sep 2020 00:00:00 GMT

I've presented Apache Camel at the last Kubecon EU conference that took place on August 19th, 2020.

As for many conferences this year, Kubecon EU was held online, not in the sunny Amsterdam as it was planned. No time to meet in person the developers that are making really good things in the Knative and Kubernetes space, but the talks were pretty awesome, and now everybody can watch them for free on YouTube.

I'm proud that Apache Camel was there, thanks to the hard work that all the team has been doing over the last two years, with the new Camel 3.x releases, Camel Quarkus and especially Camel K that bring all those features to Kubernetes.

The serverless space is an area where Camel K can make the difference. Camel K integration with Knative really rocks and we're working every day on improving it.

If you want to know what I'm talking about, why don't you listen it from myself?

As you'll learn from the video, many cool features are cooking in the pot. So, stay tuned!

Kamelets

Nicola Ferraro — Mon, 12 Oct 2020 00:00:00 GMT

Kamelets are the most important feature released with Apache Camel K 1.2.0. Apart from their cool name, Kamelets represent a significant change in the whole Camel ecosystem, because they introduce new ways of using Apache Camel in the cloud and a novel approach for contributing new connectors.

What is a Kamelet?

A Kamelet is a "Kamel Route Snippet". Before going into the details of what this actually means, let's make a step backward to add some background context.

Traditionally, the building blocks of Apache Camel have always been the components. Camel users can write complex routes by leveraging the 350+ components that are available in Apache Camel. It's not a fixed pool: Camel developers and contributors are constantly increasing the collection of supported components at each new release.

This model has worked very well for all these years and it will continue to work. But it misses an important feature that we want to cover with Kamelets: the ability to abstract.

In Camel, if I want to publish a tweet in response to a Knative event, I'd do something like this:

from('knative:event/public.post')
  .to('twitter-timeline://user')

Every time some application sends an event of type "public.post" to the Knative broker, that event is published in my Twitter timeline.

As a user, I don't care about what the "twitter-timeline" component is doing under the covers: it may contact a single API, or more than one, or it may establish several connections to various systems using strange protocols. I'm only interested in the result.

Now, being an enterprise user, I would like to do a similar thing with my own systems and create a new component to add an item to the inventory of my e-commerce application. I decide to create a new component named "company-inventory" that provides an "add" endpoint to implement such feature. So:

from('knative:event/new.item')
  .to('company-inventory://add') // easy, but now try to implement it

Now it's time to implement the "company-inventory" component. When adding an item to the inventory, we need to call an HTTP API to get the ID of the actual item type, call another API if the ID is not already present, in order to create a new one, then a third API to add the item, then add another event to a third-party Kafka topic to synchronize downstream services.

Easy? Of course, with Apache Camel. But guess what? You can't use Apache Camel when writing a new component.

True. Developers who contribute code to Apache Camel usually adapt existing libraries to the Camel APIs, but they can't leverage existing components. There's been an attempt in Camel 2.x to bring such possibility with the Routebox component, but it was not intuitive in some parts and missed a good delivery model, so it wasn't widely used and it was finally removed in 3.x in favor of this new idea of Kamelets.

Kamelets come to the rescue. With Kamelets you can encapsulate the logic to connect to a specific system into route templates (new feature of Camel 3.5.0). And guess what? Kamelets are made of pure Camel DSL.

Kamelets are Kubernetes resources. We'll see shortly how to write them. As any Kubernetes resource, you can write a Kamelet into a file and install it on a cluster using Kubectl:

company-inventory-add.kamelet.yaml


apiVersion: camel.apache.org/v1alpha1
kind: Kamelet
metadata:
  name: company-inventory-add
spec:
  # ...
  # The Kamelet will declare all accepted parameters
  # in JSON-schema format.
  # 
  # Skipping the details here.
  # ...
  flow: # Here's the route
    from:
      uri: "kamelet:source"
      steps:
        - to: "http://first-endpoint"
        - to: "https://second-endpoint/{{itemType}}"
        - choice:
          # ...
          - to: "kafka:downstream"

I'm not going into the details of how to write a Kamelet, that is covered in the Camel K user guide about Kamelets. We need just to know that:

A Kamelet exposes a well defined JSON-schema interface, that documents its purpose and defines the accepted parameters.
A Kamelet can be a source of data or a sink (consumer or producer, in the Camel jargon). The "company-inventory-add" Kamelet is a sink.
A Kamelet defines what Camel code should be executed when the Kamelet is used inside an integration: the code is expressed as route template that can use properties (e.g. "{{itemType}}")

You can install the Kamelet on a namespace by simply executing:

kubectl apply -f company-inventory-add.kamelet.yaml

Once done so, the Kamelet becomes available in all integrations that are deployed in the same Kubernetes namespace. So you can write a route like this to use the Kamelet:

example.groovy

from('knative:event/new.item')
  .to('kamelet://company-inventory-add?itemType=grocery') // implementing this is much simpler

And run it with:

kamel run example.groovy

The logic about what it means to add an inventory item is encapsulated in the Kamelet (multiple HTTP calls, enterprise integration patterns and sync with Kafka), that can be shared among all other integrations. And it's all Camel DSL.

This is only a simple example of what a Kamelet can do, there are many more interesting use cases out there.

What about Knative sources?

The first time we thought about Kamelets was in the context of Knative Eventing sources. It was the end of 2018, believe it or not. We thought that they could really make the difference in that space, but we were not ready for the leap.

We are currently providing CamelSources in Knative, and they are really cool because they allow users to write a small piece of Camel DSL to produce any kind of event, taking data from any system that Camel supports.

So, what's wrong with CamelSources? They are really cool from my point of view. But the problem is that Knative users are usually not Camel users. As a Camel user, I can write a piece of Camel DSL in a few minutes and make it work. But people with a background in Go or Python... they don't even know what Camel actually is.

But that's where a Kamelet can help.

Camel developers and passionate contributors will write the Camel DSL, creating "connectors" for external sources using the language of Kamelets. In future versions of Camel K, we'll create a catalog of curated Kamelets that will be installed together with Camel K. Those Kamelets will be available to anyone who installs Camel K on a Kubernetes cluster.

Without any knowledge of Apache Camel, people can list the catalog of the available Kamelets with standard Kubernetes tools:

$ kubectl get kamelets
NAME
inventory-source
twitter-source
slack-source
telegram-source
...

Kamelets declare the list of expected parameters using a JSON-schema format, so it will be easy to check them and provide values.

Users can then decide to use a Kamelet e.g. to push some twitter data into the Knative broker. To do so, they can create a KameletBinding.

apiVersion: camel.apache.org/v1alpha1
kind: KameletBinding
metadata:
  name: twitter-source-to-knative
spec:
  source:
    ref:
      apiVersion: camel.apache.org/v1alpha1
      kind: Kamelet
      name: twitter-source
    properties:
      keywords: Apache Camel
  sink:
    ref:
      apiVersion: eventing.knative.dev/v1
      kind: Broker
      name: default

As you see, the final user will not need to know anything about Camel, only use the source.

Of course, the KameletBinding resource is a wrapper for an Integration. I.e. under the hood the operator will create the Camel DSL corresponding to the binding, which is something like:

# only for demonstration, we don't generate groovy code ;)
from('kamelet:twitter-source?keywords=Apache+Camel')
  .to('knative:event')

This makes Kamelets available to different kind of users: Camel users and Knative users, with a different interface.

So, is it only about Knative?

Of course not: Kamelets are general purpose connectors, not directly linked to Knative.

If I want to send data from my Twitter Kamelet to a Kafka topic, I just need to create the following code and run it:

from('kamelet:twitter-source?keywords=Apache+Camel')
  .to('kafka:topic')

All Kamelets can be used to feed Kafka instead of Knative with the same exact approach. In fact, we've also extended the KameletBinding mechanism to Kafka via Strimzi.

What makes Kamelets great?

The main difference between a Kamelet and a Camel component is its purpose. While a component can serve several purposes by specifying different combinations of the parameters, Kamelets are driven by use cases. A Kamelet can let you perform a specific action on a system or gather data from another system, with a limited degree of flexibility.

Reducing the scope helps designing CLI or UI tools around them.

A Kamelet contains (see the Kamelet specification for more details):

Title and description of the purpose of the Kamelet and how to use it
A nice icon
The set of expected parameters in JSON-schema format
The schema of the output they produce (or the input they require)
Implementation (Camel DSL)

Writing a UI that allows browsing the available Kamelets, configure them and bind them to a destination should be pretty easy.

There are many possible use cases:

A specialized UI, e.g. in the OpenShift dev console, can let the user browse the available Kamelets and configure them via a graphical form to send data to a particular Knative destination
A second specialized UI about Kafka can include a section where the users can instantiate the Kamelets to bring data to a particular topic
Another platform, e.g. like Syndesis can use the Kamelets to provide additional connectors for creating end-to-end integrations

The great thing about Kamelets is that they are not bound to a spcific technology: they are reusable connectors.

Try them out!

Camel K 1.2.0 has already been released, so, why don't you try this new feature?

Feedback is welcome, as well as any kind of contribution!

Meet us on Zulip!

Debugging Camel K

Nicola Ferraro — Mon, 30 Nov 2020 00:00:00 GMT

Camel K introduces new ways of dealing with integration problems by letting you develop and run routes in the cloud, where the services that you need to connect live. This has many advantages, since it makes it easy to use all the features available in the cloud environment from the very beginning. For example, you can leverage service discovery, you can make your integration passive and let it receive push notifications from Knative with real data... But this comes with a cost.

When something goes wrong while you're developing a new integration, you'd like to easily understand what's going on and fix your code. Since the integration is not running locally, it's not easy to leverage all your IDE tools to troubleshoot. But we're working to improve the experience on many fronts. I'm describing here the new kamel debug command.

Debug command

Suppose you're working on an integration problem and you've designed your route as:

import org.apache.camel.Header;
import org.apache.camel.builder.RouteBuilder;

public class Flaky extends RouteBuilder {
  @Override
  public void configure() throws Exception {

    from("timer:clock?period=5000")
    .bean(this, "createPayload")
    .to("https://postman-echo.com/post")
    .log("sent!");

  }

  public String createPayload(@Header("CamelTimerCounter") Integer tick) {
    return "" + (100 / (tick - 1));
  }

}

Note: I'm writing the "createPayload" method directly in the route file, but likely you'll put it in a separate library.

This is a simple route that every 5 seconds builds a payload and sends it to a HTTPS destination.

You can run it with:

kamel run Flaky.java

Once it starts, you notice that there's an issue in the way the payload is built, since the output signals an error in the first exchange.

[1] 2020-11-30 09:22:02,377 WARN  [org.apa.cam.com.tim.TimerConsumer] (Camel (camel-1) thread #0 - timer://clock) Error processing exchange. Exchange[746A22E1144A36D-0000000000000000]. Caused by: [java.lang.ArithmeticException - / by zero]: java.lang.ArithmeticException: / by zero
[1]     at Flaky.createPayload(Flaky.java:18)
[1]     at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[1]     at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[1]     at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[1]     at java.base/java.lang.reflect.Method.invoke(Method.java:566)
[1]     at org.apache.camel.support.ObjectHelper.invokeMethodSafe(ObjectHelper.java:208)
[1]     at org.apache.camel.component.bean.MethodInfo.invoke(MethodInfo.java:425)
[1]     at org.apache.camel.component.bean.MethodInfo$1.doProceed(MethodInfo.java:247)
[1]     at org.apache.camel.component.bean.MethodInfo$1.proceed(MethodInfo.java:217)
[1]     at org.apache.camel.component.bean.AbstractBeanProcessor.process(AbstractBeanProcessor.java:154)
[1]     at org.apache.camel.component.bean.BeanProcessor.process(BeanProcessor.java:56)
[1]     at org.apache.camel.processor.errorhandler.RedeliveryErrorHandler$SimpleTask.run(RedeliveryErrorHandler.java:404)
[1]     at org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.schedule(DefaultReactiveExecutor.java:148)
[1]     at org.apache.camel.impl.engine.DefaultReactiveExecutor.scheduleMain(DefaultReactiveExecutor.java:60)
[1]     at org.apache.camel.processor.Pipeline.process(Pipeline.java:147)
[1]     at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:287)
[1]     at org.apache.camel.component.timer.TimerConsumer.sendTimerExchange(TimerConsumer.java:207)
[1]     at org.apache.camel.component.timer.TimerConsumer$1.run(TimerConsumer.java:76)
[1]     at java.base/java.util.TimerThread.mainLoop(Timer.java:556)
[1]     at java.base/java.util.TimerThread.run(Timer.java:506)
[1] 
[1] 2020-11-30 09:22:08,088 INFO  [route1] (Camel (camel-1) thread #0 - timer://clock) sent!
[1] 2020-11-30 09:22:13,236 INFO  [route1] (Camel (camel-1) thread #0 - timer://clock) sent!

It's easy to understand what's wrong here, because this is a hand crafted example, but in a real integration case, your business logic may be much more complex and it won't be easy to understand the problem, until you actually look into the running code.

There's a new command in Camel K 1.3.0 that allows you to switch a remote integration to "debug mode". This way the integration can be connected to your IDE and you can use breakpoints to check specific sections of the code for bugs. The kamel CLI will take care of setting the right flags on the integration and also establishing a secure tunnel to let your IDE connect to the integration.

To switch to the debug mode, just run:

kamel debug flaky

Where flaky is just the name of the integration. This is an example of output:

$ kamel debug flaky
Enabling debug mode on integration "flaky"...
Forwarding from 127.0.0.1:5005 -> 5005
Forwarding from [::1]:5005 -> 5005
[1] Monitoring pod flaky-76f67fb58-rk8g8
[1] exec java -agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=*:5005 -cp ./resources:[...] io.quarkus.runner.GeneratedMain
[1] Listening for transport dt_socket at address: 5005

As you see, the integration is waiting for you to connect your IDE on localhost on port 5005 using the debug protocol. You can configure your IDE to attach to the process via that host an port. This is IDE specific, but all modern IDEs support remote debugging.

If you're using VSCode, you can put a launch.json file in the .vscode dir at the root of your project, with the following content:

{
    "version": "0.2.0",
    "configurations": [
        {
            "type": "java",
            "name": "Debug Camel K Integration",
            "request": "attach",
            "hostName": "localhost",
            "port": 5005
        }
    ]
}

This is enough to connect. The final result is shown in the following image:

<p style="text-align: center"> <img src="/images/camel-k-debug.gif" alt="IDE Debug"/> <caption align="bottom"><i>IDE Debug</i></caption> </p>

At the end of the debug session, the integration will be restored to its normal state.

Have fun with Camel K!

Kamelet Catalog

Nicola Ferraro — Wed, 24 Feb 2021 00:00:00 GMT

Note: this post has been published also in the Apache Camel blog

We're starting a new initiative at Apache Camel to create a community-driven catalog of reusable Kamelets (Camel route snippets, i.e. connectors) that can be used to stream data from/to external systems into any platform powered by Apache Camel.

The "Apache Camel Kamelet catalog" is available here and it already contains a collection of useful Kamelets: we would like to extend it with help of the community.

Kamelets are currently supported out-of-the-box by the Apache Camel K project and we're working to support them also in Camel core, so that they can run eventually in any Apache Camel subproject, like Camel Kafka Connector (but also camel-quarkus, camel-spring-boot, ...).

Why are they useful?

The most important reason why the new Kamelet catalog is so useful is that it provides a collection of connectors that are not tied to a particular platform, but they can be reused in multiple contexts. The new OpenShift 4.7 developer console is a clear evidence of this pattern. You can look at what it provides in the following animation:

The console shows a serverless environment where a Knative channel has been created and it's ready to receive and dispatch events to a registered subscribing service. From the console, the user can add a new "Event Source" and the UI displays a collection of possibilities: well, starting from Camel K 1.4.0, that collection of sources will display the official Apache Camel Kamelet catalog. This means that contributing new Kamelets to the catalog will allow you to enhance the OpenShift catalog of Knative sources, for instance.

But there's nothing in the Kamelet catalog specifically related to OpenShift: we show this to encourage people to take the same approach for other platforms (in-house or open source, especially Kubernetes-based) and use cases. One of the advantages of this approach is that, since Kamelets provide a well defined JSON-schema interface for their configuration, they can be made available for any kind of users, including those who don't even know anything about Apache Camel.

So, no matter if you need ingress/egress for a specific destination (like in the Knative example above) or you're working on an event orchestration/workflow tool, or you're just creating a IFTTT clone: if you need connectors, look at Kamelets.

Kamelets are not only useful in Web/UI projects, they can be also used from a CLI. For instance, there's a plugin in the making in the Knative sandbox that will allow users to easily create Knative sources from Kamelets using the Knative kn CLI.

Leaving the Kubernetes world, you can also imagine how the Kamelet Catalog can improve the user experience in Camel Kafka Connectors. As of today, A Camel Kafka connector can be created from a single Camel component: this is a quite powerful model, since components have a high degree of configurability, but it has severe limitations because complex use cases cannot be fulfilled by configuring a single Camel component. The Kamelet model will allow Camel Kafka Connector users to develop more complex use cases (because a Kamelet is a route template that can use multiple components and enterprise integration patterns) and share their connectors with other users via the Kamelet catalog.

Kamelet vs. Component

There's a question that often arises and it relates to the difference between a Camel component and a Kamelet.

From a pure user perspective, the main difference between them is that the Camel component is a highly configurable resource that is intended to be used in a Camel development environment. Components contain lots of properties that allow to customize every aspect of their behavior and how they interact with the Camel context, the registry and so on. Additionally, a single Camel component can serve multiple purposes: e.g. the camel-facebook component allows receiving any kind of event or execute any action on Facebook; the camel-http component allows to contact any external http or https endpoint.

By contrast, Kamelets are tailored around actual use cases. A todo-sink Kamelet may be created in the catalog to transform an event into an entry of a "To-Do Application" (I refer here to that kind of applications where you add the things you need to, and later mark them as resolved). It may use the camel-http component under the hood, but that will be an implementation detail. The todo-sink Kamelet will also expose to the end user not the full list of parameters available in the camel-http component, but a reduced set of configuration properties that are relevant for the business purpose (i.e. adding an entry into a "To-Do Application"). Similarly, when the Kamelet model will evolve with the help of the community, there won't be a single facebook-source Kamelet, but there will be probably many facebook-related sources, one for each use case that the community finds relevant.

An important corollary of all this is that we're not going to auto-generate the Kamelet catalog from the components, instead, we'll follow a more opinionated approach taking into consideration the user needs rather than what the components already provide.

Contributing

Contributions are welcome for the Kamelet catalog. All you need to do is to follow the guidelines in the official repository for the Kamelet catalog on Github.

Here's a list of relevant resources that you may want to learn before starting creating your own Kamelets:

Camel K 1.4.0 Released

Nicola Ferraro — Tue, 20 Apr 2021 00:00:00 GMT

Note: this post has been published also in the Apache Camel blog

Apache Camel K 1.4.0 has just been released!

This is a new major release of Camel K with an improved stability over previous versions, but also adding new features that simplify the overall user experience.

It is based on Camel 3.9.0 and Camel-Quarkus 1.8.1, providing all improvements that they bring, plus much more. In this blog post, we're going to describe the most important changes.

Embedded Kamelet catalog

Camel K 1.4.0 comes with an embedded Kamelet catalog containing multiple connectors ready for use.

When installing the operator into a namespace (but also globally in the cluster), the operator installs all the kamelets from the catalog (version 0.2.1), so that any integration can use them directly.

Users can bind them to a specific destination by writing a YAML binding file, as explained in the specific documentation related to each Kamelet.

Or, you can use the new kamel bind command (see below).

Note: it's easy to write your own Kamelet and publish it to the Apache Catalog. Take a look at the Kamelets developer guide.

Kamel bind command

We've added a bind subcommand to the kamel CLI that provides a new way to use Kamelets directly when you need to connect them to Knative channels, Kafka topics and any other endpoint.

E.g. Suppose that you want to get events of earthquakes happening around the world, as JSON objects, in your Knative channel named earthquakes. All you need to do is to install Camel K on your cluster and then execute the following command:

kamel bind earthquake-source channel:earthquakes

This command creates the KameletBinding resource for you and the Camel K operator does the rest to bring that data into your channel. Data is produced using the Earthquake Source Kamelet available in the embedded Katalog.

You can use any other Kamelet from the catalog using the kamel bind command.

You can also target any other Kubernetes reference that is supported by Camel K, for example, sink into a Strimzi KafkaTopic, using a fully qualified reference, for example:

kamel bind earthquake-source kafka.strimzi.io/v1beta1:KafkaTopic:mytopic

Of course, the command also supports plain Camel URIs, which are useful especially when you're developing a new Kamelet. For example, you can write:

kamel bind earthquake-source log:info?showHeaders=true

And the command will create a binding that just prints to the log the JSON data produced by the source.

Kamel dump command

When users have issues understanding why Camel K is not behaving as expected, they often need to provide useful information about the current state of their cluster, to let Camel K developers investigate the issue and provide a solution or a quick workaround (e.g. in a Github issue, or in the Zulip chat).

Usually, to identify the root cause of an issue, developers need to know things like:

What routes the user is trying to run
What the Camel K operator is doing
What images have been built, which versions of all libraries are they using
What's the state of the Camel K custom resources
What errors do Camel K integration throw when they start

Providing such information has always been hard, but we now have a quick way to obtain all that.

kamel dump status.log

This simple command will store in a text file all the information needed to investigate a possible issue in the cluster. The user can now edit the file to remove sensitive information (which the command may not be able to tell apart), then share it with developers to have much better insights.

Stability and compatibility

We focused a lot on stability and improved compatibility with other tools of the ecosystem.

Knative support (0.22.0) has been improved by fixing compatibility issues that sometimes caused multiple revisions to be present for the same service. We've also changed the way channels and brokers are bound to the integrations. Now it's possible to bind integrations to multiple channels and even create sequences of integrations attached to channels without any issue (e.g. #2190, #2115).

We've improved installation options, letting you configure things that may be important in a production environment, like setting toleration or using a secured maven repository. At the same time, we've fixed compatibility with recent dev environments, e.g. letting you smoothly install Camel K in K3S.

We also kept doing changes to continuously improve speed. On the runtime side of Camel K, we now use the Quarkus fast-jar format to reduce boot times. And last but not least on the operator side, it's possible to install Camel K globally in a cluster and have much faster build times by sharing base images across the cluster.

The list of important changes in the 1.4.0 release is too long for this blog post. There have never been so many contributors as in this release and we thank them all for their awesome work!

From Camel to Kamelets: new connectors for event-driven applications

Nicola Ferraro — Sun, 10 Oct 2021 00:00:00 GMT

I've done two similar talks this year at ApacheCon @Home and Asia, both on the ease of use of the new Kamelet paradigm when building event-driven applications.

Low Code Camel

Nicola Ferraro — Wed, 03 Nov 2021 00:00:00 GMT

Apache Camel is a project in constant transformation. Anyone knows that Camel has been able to adapt to any new kinds of protocols and systems that have emerged in the past 15 years. But it's not only that: Camel is also able to adapt to any new ways of dealing with integration problems in the cloud era. While Camel K represented a fundamental shift towards a new approach to cloud-native integration, "Kamelets" are driving a deeper transformation towards "low code" development.

When I talk about "low code", I refer to a platform for developers where you can achieve most of your goals without writing any code. But at the same time, a low code platform should be customizable enough to let you show your development skills when it's time to solve critical issues. It needs to be abstract in order to easily deal with the most common problems, even using a UI. But it should not prevent a developer to play with the low level details in order to extend the capabilities of the platform itself.

If you're a bit familiar with how Kamelets work, I think you've already got my point here.

Bindings and Karavans

It's really easy to use Kamelets, especially in combination with Camel K. I've shown in a recent presentation at ApacheCon @Home that moving data from Knative or Kafka into an S3 bucket using streaming upload is as simple as writing something like this:

apiVersion: camel.apache.org/v1alpha1
kind: KameletBinding
metadata:
  name: events-to-s3
spec:
  source:
    ref:
      apiVersion: messaging.knative.dev/v1
      kind: Channel
      name: messages
      # Or a Strimzi "KafkaTopic", if you prefer
  sink:
    ref:
      kind: Kamelet
      apiVersion: camel.apache.org/v1alpha1
      name: aws-s3-streaming-upload-sink
    properties:
      bucketNameOrArn: "my-bucket"
      accessKey: "<your-key>"
      # ...

(source, S3 Kamelet Documentation)

You write this file, apply it to a Kubernetes cluster where Camel K is installed: done!

And it's completely declarative. You just specify where you want to move data from and to: no need to care about lower level details.

If writing YAML is not your favourite sport, you can also opt for a visual UI. I've shown in the previous video how the OpenShift dev console is already instructed to use Kamelets in combination with Knative. But there's a new project in the horizon at Apache that will bring this model even further...

Karavan is a new project started at Apache by Marat Gubaidullin. It is a generic visual UI that can design integrations based on Kamelets. If you either need to create a simple source/sink integration or a more complex one, Karavan can help designing it with ease. There's also a blog post about Karavan if you want to learn more.

The Karavan UI

After you finish to configure the integration, the output you get from the UI is an Integration custom resource, that you can direcly apply to any Kubernetes cluster (having Camel K installed) and it will automatically run.

We have now a wider catalog of Kamelets available upstream and it will cover many use cases that you may have in mind. They are the building blocks for letting you connect systems without writing any code: just choose if you want to use the YAML configuration or let the Karavan UI create it for you.

Where's my Kamelet?

There's no doubt you'll find a use case which is not covered by a Kamelet. For example, what about moving data between parts of your organization?

In traditional Camel the approach you would have taken to address this problem is to create one or more ad hoc integrations to fullfill any of your needs. You'd develop a custom integration route for each use case you have in mind.

The Kamelet approach is completely different. You develop a reusable Kamelet that is responsible of taking specific data out of your system (a source), or putting some data back into your system (a sink). Once you've done that, you can treat it as a high level connector in the Karavan UI and use it, whenever you need, in multiple scenarios where you need to move data around, without having to code everything from scratch each time.

Writing a Kamelet is easy, much easier than writing a Camel component, if you've ever tried doing so. We've written an extended Kamelet developer guide that covers both easy and complex scenarios.

We know that creating a source or a sink and also doing it right may be difficult (integration is hard, that's why we have Apache Camel) and involve multiple enterprise integration patterns. The Kamelet developer guide contains a section about an "Earthquake" source that is complicated on purpose, to mimic a Kamelet that may be developed for handling data inside an organization.

When writing a new Kamelet, you can use plain YAML to describe a route template, add Java (or Groovy, or ...) code snippets if you need them, or also link an external Maven project in case you want to add a more complex behavior to it (we did it in the camel-kamelets repository, where we build the default catalog of Kamelets).

What does not change is that, once a Kamelet is written, the logic inside it is well encapsulated and you can treat it as any other connector, in a Binding or in the Karavan UI.

A Kamelet is then the element that enables low code development: it's easy to configure and embeddable in any UI (such as we've done in Karavan), and allows you to fully customize its behavior, even writing various degrees of custom code, up to the lowest level elements of a Camel integration route.

Where to go next

Check out the Karavan repository, where you can build an run the Karavan designer. There's also a VSCode extension that will let you design integrations directly from the IDE.

Then start playing with Karavan in combination with Camel K and the Kamelet catalog.

And finally look at how easy is to create and use your own Kamelets.

We'd love to hear your feedback!

Camel meets KEDA

Nicola Ferraro — Fri, 21 Jan 2022 00:00:00 GMT

KEDA (Kubernetes Event Driven Autoscalers) is a fantastic project (currently CNCF incubating) that provides Kubernetes-based autoscalers to help applications to scale out according to the number of incoming events when they are listening to several kinds of event sources. In Camel K we've long supported Knative for providing a similar functionality for integrations that are triggered by HTTP calls, so supporting KEDA was something planned since long time, because it enables full autoscaling from a wider collection of sources. The KEDA integration has now been merged and it will be available in Camel K 1.8.0. This post highlights some details of the solution. If you want to see it in action, you can just jump to the end to see the demo recording.

Why KEDA and Camel K?

Users may want to set up integrations that automatically scale, e.g. depending on the number of messages left in a Kafka topic (for their consumer group). The integration code may do e.g. transformations, aggregations and send data do a destination and it would be great if the number of instances deployed to a Kubernetes cluster could increase when there's work left behind, or they can scale to zero when there's no more data in the topic. This is what KEDA does by itself with scalers (Kafka is one of the many scalers available in KEDA). What you have now is that KEDA is now automatically configured by Camel K when you run integrations, so you get the autoscaling features out of the box (you just need to turn a flag on).

How does it work?

In Camel K 1.8.0 a new KEDA trait has been introduced. The trait allows to manually tweak the KEDA configuration to make sure that some ScaledObjects (KEDA concept) are generated as part of the Integration reconciliation, but this is mostly an internal detail. The interesting part about the KEDA trait is that it can recognize special KEDA markers in Kamelets and automatically create a KEDA valid configuration when those Kamelets are used as sources. So users can just use Kamelets to create bindings as usual and, if they enable a KEDA flag via an annotation, they get an event driven autoscaler automatically configured.

The Kamelet catalog embedded in next release (v0.7.0) contains two Kamelets enhanced with KEDA metadata: aws-sqs-source and kafka-source. These are just two examples of the many Kamelets that can be augmented in the future. The metadata configuration system is open and Kamelets can be marked at any time to work with KEDA: this means that you don't need to wait for a new Camel K release to enable KEDA on a different source and, more importantly, that you can mark your own Kamelets with KEDA metadata to enable autoscaling from your internal organization components.

The Kamelet developer guide contains a new section on how to mark a Kamelet with KEDA metadata, but essentially markers are used to map Kamelet properties into KEDA configuration options, so that when you provide a Kamelet configuration, the corresponding KEDA options can be generated from it (all the work is done under the cover by the Camel K operator).

A binding example

Before looking at the demo, here's an example of autoscaling binding that you can create with the latest Camel K:

apiVersion: camel.apache.org/v1alpha1
kind: KameletBinding
metadata:
  name: kafka-to-sink
  annotations:
    trait.camel.apache.org/keda.enabled: "true"
spec:
  source:
    ref:
      apiVersion: camel.apache.org/v1alpha1
      kind: Kamelet
      name: kafka-source
    properties:
      bootstrapServers: "<-- bootstrap servers -->"
      consumerGroup: my-group
      topic: "<-- the topic -->"
      user: "<-- user -->"
      password: "<-- pwd -->"
  sink:
    # ...

You can notice that the only difference from a standard binding is the presence of the trait.camel.apache.org/keda.enabled=true annotation that enables the KEDA trait in Camel K. The information about how to map Kamelet properties into KEDA options is encoded in the Kamelet definition.

Demo

Time for the demonstration. You'll see both the aws-sqs-source and the kafka-source in action with Camel K and KEDA.

The code for the demo is in the nicolaferraro/camel-k-keda-demo repository on GitHub.

Next steps

There are many other Kamelets to enhance in apache/camel-kamelets and things to improve in Camel K that are listed in the area/KEDA section of the issue tracker.

Contributions are always welcome!