Nicola Ferraro

Camel meets KEDA

2022-01-21T00:00:00+01:00

KEDA (Kubernetes Event Driven Autoscalers) is a fantastic project (currently CNCF incubating) that provides Kubernetes-based autoscalers to help applications to scale out according to the number of incoming events when they are listening to several kinds of event sources. In Camel K we’ve long supported Knative for providing a similar functionality for integrations that are triggered by HTTP calls, so supporting KEDA was something planned since long time, because it enables full autoscaling from a wider collection of sources. The KEDA integration has now been merged and it will be available in Camel K 1.8.0. This post highlights some details of the solution. If you want to see it in action, you can just jump to the end to see the demo recording.

Why KEDA and Camel K?

Users may want to set up integrations that automatically scale, e.g. depending on the number of messages left in a Kafka topic (for their consumer group). The integration code may do e.g. transformations, aggregations and send data do a destination and it would be great if the number of instances deployed to a Kubernetes cluster could increase when there’s work left behind, or they can scale to zero when there’s no more data in the topic. This is what KEDA does by itself with scalers (Kafka is one of the many scalers available in KEDA). What you have now is that KEDA is now automatically configured by Camel K when you run integrations, so you get the autoscaling features out of the box (you just need to turn a flag on).

How does it work?

In Camel K 1.8.0 a new KEDA trait has been introduced. The trait allows to manually tweak the KEDA configuration to make sure that some ScaledObjects (KEDA concept) are generated as part of the Integration reconciliation, but this is mostly an internal detail. The interesting part about the KEDA trait is that it can recognize special KEDA markers in Kamelets and automatically create a KEDA valid configuration when those Kamelets are used as sources. So users can just use Kamelets to create bindings as usual and, if they enable a KEDA flag via an annotation, they get an event driven autoscaler automatically configured.

The Kamelet catalog embedded in next release (v0.7.0) contains two Kamelets enhanced with KEDA metadata: aws-sqs-source and kafka-source. These are just two examples of the many Kamelets that can be augmented in the future. The metadata configuration system is open and Kamelets can be marked at any time to work with KEDA: this means that you don’t need to wait for a new Camel K release to enable KEDA on a different source and, more importantly, that you can mark your own Kamelets with KEDA metadata to enable autoscaling from your internal organization components.

The Kamelet developer guide contains a new section on how to mark a Kamelet with KEDA metadata, but essentially markers are used to map Kamelet properties into KEDA configuration options, so that when you provide a Kamelet configuration, the corresponding KEDA options can be generated from it (all the work is done under the cover by the Camel K operator).

A binding example

Before looking at the demo, here’s an example of autoscaling binding that you can create with the latest Camel K:

apiVersion: camel.apache.org/v1alpha1
kind: KameletBinding
metadata:
  name: kafka-to-sink
  annotations:
    trait.camel.apache.org/keda.enabled: "true"
spec:
  source:
    ref:
      apiVersion: camel.apache.org/v1alpha1
      kind: Kamelet
      name: kafka-source
    properties:
      bootstrapServers: "<-- bootstrap servers -->"
      consumerGroup: my-group
      topic: "<-- the topic -->"
      user: "<-- user -->"
      password: "<-- pwd -->"
  sink:
    # ...

You can notice that the only difference from a standard binding is the presence of the trait.camel.apache.org/keda.enabled=true annotation that enables the KEDA trait in Camel K. The information about how to map Kamelet properties into KEDA options is encoded in the Kamelet definition.

Demo

Time for the demonstration. You’ll see both the aws-sqs-source and the kafka-source in action with Camel K and KEDA.

The code for the demo is in the nicolaferraro/camel-k-keda-demo repository on GitHub.

Next steps

There are many other Kamelets to enhance in apache/camel-kamelets and things to improve in Camel K that are listed in the area/KEDA section of the issue tracker.

Contributions are always welcome!

Low Code Camel

2021-11-03T00:00:00+01:00

Apache Camel is a project in constant transformation. Anyone knows that Camel has been able to adapt to any new kinds of protocols and systems that have emerged in the past 15 years. But it’s not only that: Camel is also able to adapt to any new ways of dealing with integration problems in the cloud era. While Camel K represented a fundamental shift towards a new approach to cloud-native integration, “Kamelets” are driving a deeper transformation towards “low code” development.

When I talk about “low code”, I refer to a platform for developers where you can achieve most of your goals without writing any code. But at the same time, a low code platform should be customizable enough to let you show your development skills when it’s time to solve critical issues. It needs to be abstract in order to easily deal with the most common problems, even using a UI. But it should not prevent a developer to play with the low level details in order to extend the capabilities of the platform itself.

If you’re a bit familiar with how Kamelets work, I think you’ve already got my point here.

Bindings and Karavans

It’s really easy to use Kamelets, especially in combination with Camel K. I’ve shown in a recent presentation at ApacheCon @Home that moving data from Knative or Kafka into an S3 bucket using streaming upload is as simple as writing something like this:

apiVersion: camel.apache.org/v1alpha1
kind: KameletBinding
metadata:
  name: events-to-s3
spec:
  source:
    ref:
      apiVersion: messaging.knative.dev/v1
      kind: Channel
      name: messages
      # Or a Strimzi "KafkaTopic", if you prefer
  sink:
    ref:
      kind: Kamelet
      apiVersion: camel.apache.org/v1alpha1
      name: aws-s3-streaming-upload-sink
    properties:
      bucketNameOrArn: "my-bucket"
      accessKey: "<your-key>"
      # ...

(source, S3 Kamelet Documentation)

You write this file, apply it to a Kubernetes cluster where Camel K is installed: done!

And it’s completely declarative. You just specify where you want to move data from and to: no need to care about lower level details.

If writing YAML is not your favourite sport, you can also opt for a visual UI. I’ve shown in the previous video how the OpenShift dev console is already instructed to use Kamelets in combination with Knative. But there’s a new project in the horizon at Apache that will bring this model even further…

Karavan is a new project started at Apache by Marat Gubaidullin. It is a generic visual UI that can design integrations based on Kamelets. If you either need to create a simple source/sink integration or a more complex one, Karavan can help designing it with ease. There’s also a blog post about Karavan if you want to learn more.

The Karavan UI

After you finish to configure the integration, the output you get from the UI is an Integration custom resource, that you can direcly apply to any Kubernetes cluster (having Camel K installed) and it will automatically run.

We have now a wider catalog of Kamelets available upstream and it will cover many use cases that you may have in mind. They are the building blocks for letting you connect systems without writing any code: just choose if you want to use the YAML configuration or let the Karavan UI create it for you.

Where’s my Kamelet?

There’s no doubt you’ll find a use case which is not covered by a Kamelet. For example, what about moving data between parts of your organization?

In traditional Camel the approach you would have taken to address this problem is to create one or more ad hoc integrations to fullfill any of your needs. You’d develop a custom integration route for each use case you have in mind.

The Kamelet approach is completely different. You develop a reusable Kamelet that is responsible of taking specific data out of your system (a source), or putting some data back into your system (a sink). Once you’ve done that, you can treat it as a high level connector in the Karavan UI and use it, whenever you need, in multiple scenarios where you need to move data around, without having to code everything from scratch each time.

Writing a Kamelet is easy, much easier than writing a Camel component, if you’ve ever tried doing so. We’ve written an extended Kamelet developer guide that covers both easy and complex scenarios.

We know that creating a source or a sink and also doing it right may be difficult (integration is hard, that’s why we have Apache Camel) and involve multiple enterprise integration patterns. The Kamelet developer guide contains a section about an “Earthquake” source that is complicated on purpose, to mimic a Kamelet that may be developed for handling data inside an organization.

When writing a new Kamelet, you can use plain YAML to describe a route template, add Java (or Groovy, or …) code snippets if you need them, or also link an external Maven project in case you want to add a more complex behavior to it (we did it in the camel-kamelets repository, where we build the default catalog of Kamelets).

What does not change is that, once a Kamelet is written, the logic inside it is well encapsulated and you can treat it as any other connector, in a Binding or in the Karavan UI.

A Kamelet is then the element that enables low code development: it’s easy to configure and embeddable in any UI (such as we’ve done in Karavan), and allows you to fully customize its behavior, even writing various degrees of custom code, up to the lowest level elements of a Camel integration route.

Where to go next

Check out the Karavan repository, where you can build an run the Karavan designer. There’s also a VSCode extension that will let you design integrations directly from the IDE.

Then start playing with Karavan in combination with Camel K and the Kamelet catalog.

And finally look at how easy is to create and use your own Kamelets.

We’d love to hear your feedback!

From Camel to Kamelets: new connectors for event-driven applications

2021-10-10T00:00:00+02:00

I’ve done two similar talks this year at ApacheCon @Home and Asia, both on the ease of use of the new Kamelet paradigm when building event-driven applications.

Camel K 1.4.0 Released

2021-04-20T00:00:00+02:00

Note: this post has been published also in the Apache Camel blog

Apache Camel K 1.4.0 has just been released!

This is a new major release of Camel K with an improved stability over previous versions, but also adding new features that simplify the overall user experience.

It is based on Camel 3.9.0 and Camel-Quarkus 1.8.1, providing all improvements that they bring, plus much more. In this blog post, we’re going to describe the most important changes.

Embedded Kamelet catalog

Camel K 1.4.0 comes with an embedded Kamelet catalog containing multiple connectors ready for use.

When installing the operator into a namespace (but also globally in the cluster), the operator installs all the kamelets from the catalog (version 0.2.1), so that any integration can use them directly.

Users can bind them to a specific destination by writing a YAML binding file, as explained in the specific documentation related to each Kamelet.

Or, you can use the new kamel bind command (see below).

Note: it’s easy to write your own Kamelet and publish it to the Apache Catalog. Take a look at the Kamelets developer guide.

Kamel bind command

We’ve added a bind subcommand to the kamel CLI that provides a new way to use Kamelets directly when you need to connect them to Knative channels, Kafka topics and any other endpoint.

E.g. Suppose that you want to get events of earthquakes happening around the world, as JSON objects, in your Knative channel named earthquakes. All you need to do is to install Camel K on your cluster and then execute the following command:

kamel bind earthquake-source channel:earthquakes

This command creates the KameletBinding resource for you and the Camel K operator does the rest to bring that data into your channel. Data is produced using the Earthquake Source Kamelet available in the embedded Katalog.

You can use any other Kamelet from the catalog using the kamel bind command.

You can also target any other Kubernetes reference that is supported by Camel K, for example, sink into a Strimzi KafkaTopic, using a fully qualified reference, for example:

kamel bind earthquake-source kafka.strimzi.io/v1beta1:KafkaTopic:mytopic

Of course, the command also supports plain Camel URIs, which are useful especially when you’re developing a new Kamelet. For example, you can write:

kamel bind earthquake-source log:info?showHeaders=true

And the command will create a binding that just prints to the log the JSON data produced by the source.

Kamel dump command

When users have issues understanding why Camel K is not behaving as expected, they often need to provide useful information about the current state of their cluster, to let Camel K developers investigate the issue and provide a solution or a quick workaround (e.g. in a Github issue, or in the Zulip chat).

Usually, to identify the root cause of an issue, developers need to know things like:

What routes the user is trying to run
What the Camel K operator is doing
What images have been built, which versions of all libraries are they using
What’s the state of the Camel K custom resources
What errors do Camel K integration throw when they start

Providing such information has always been hard, but we now have a quick way to obtain all that.

kamel dump status.log

This simple command will store in a text file all the information needed to investigate a possible issue in the cluster. The user can now edit the file to remove sensitive information (which the command may not be able to tell apart), then share it with developers to have much better insights.

Stability and compatibility

We focused a lot on stability and improved compatibility with other tools of the ecosystem.

Knative support (0.22.0) has been improved by fixing compatibility issues that sometimes caused multiple revisions to be present for the same service. We’ve also changed the way channels and brokers are bound to the integrations. Now it’s possible to bind integrations to multiple channels and even create sequences of integrations attached to channels without any issue (e.g. #2190, #2115).

We’ve improved installation options, letting you configure things that may be important in a production environment, like setting toleration or using a secured maven repository. At the same time, we’ve fixed compatibility with recent dev environments, e.g. letting you smoothly install Camel K in K3S.

We also kept doing changes to continuously improve speed. On the runtime side of Camel K, we now use the Quarkus fast-jar format to reduce boot times. And last but not least on the operator side, it’s possible to install Camel K globally in a cluster and have much faster build times by sharing base images across the cluster.

The list of important changes in the 1.4.0 release is too long for this blog post. There have never been so many contributors as in this release and we thank them all for their awesome work!

Kamelet Catalog

2021-02-24T00:00:00+01:00

Note: this post has been published also in the Apache Camel blog

We’re starting a new initiative at Apache Camel to create a community-driven catalog of reusable Kamelets (Camel route snippets, i.e. connectors) that can be used to stream data from/to external systems into any platform powered by Apache Camel.

The “Apache Camel Kamelet catalog” is available here and it already contains a collection of useful Kamelets: we would like to extend it with help of the community.

Kamelets are currently supported out-of-the-box by the Apache Camel K project and we’re working to support them also in Camel core, so that they can run eventually in any Apache Camel subproject, like Camel Kafka Connector (but also camel-quarkus, camel-spring-boot, …).

Why are they useful?

The most important reason why the new Kamelet catalog is so useful is that it provides a collection of connectors that are not tied to a particular platform, but they can be reused in multiple contexts. The new OpenShift 4.7 developer console is a clear evidence of this pattern. You can look at what it provides in the following animation:

The console shows a serverless environment where a Knative channel has been created and it’s ready to receive and dispatch events to a registered subscribing service. From the console, the user can add a new “Event Source” and the UI displays a collection of possibilities: well, starting from Camel K 1.4.0, that collection of sources will display the official Apache Camel Kamelet catalog. This means that contributing new Kamelets to the catalog will allow you to enhance the OpenShift catalog of Knative sources, for instance.

But there’s nothing in the Kamelet catalog specifically related to OpenShift: we show this to encourage people to take the same approach for other platforms (in-house or open source, especially Kubernetes-based) and use cases. One of the advantages of this approach is that, since Kamelets provide a well defined JSON-schema interface for their configuration, they can be made available for any kind of users, including those who don’t even know anything about Apache Camel.

So, no matter if you need ingress/egress for a specific destination (like in the Knative example above) or you’re working on an event orchestration/workflow tool, or you’re just creating a IFTTT clone: if you need connectors, look at Kamelets.

Kamelets are not only useful in Web/UI projects, they can be also used from a CLI. For instance, there’s a plugin in the making in the Knative sandbox that will allow users to easily create Knative sources from Kamelets using the Knative kn CLI.

Leaving the Kubernetes world, you can also imagine how the Kamelet Catalog can improve the user experience in Camel Kafka Connectors. As of today, A Camel Kafka connector can be created from a single Camel component: this is a quite powerful model, since components have a high degree of configurability, but it has severe limitations because complex use cases cannot be fulfilled by configuring a single Camel component. The Kamelet model will allow Camel Kafka Connector users to develop more complex use cases (because a Kamelet is a route template that can use multiple components and enterprise integration patterns) and share their connectors with other users via the Kamelet catalog.

Kamelet vs. Component

There’s a question that often arises and it relates to the difference between a Camel component and a Kamelet.

From a pure user perspective, the main difference between them is that the Camel component is a highly configurable resource that is intended to be used in a Camel development environment. Components contain lots of properties that allow to customize every aspect of their behavior and how they interact with the Camel context, the registry and so on. Additionally, a single Camel component can serve multiple purposes: e.g. the camel-facebook component allows receiving any kind of event or execute any action on Facebook; the camel-http component allows to contact any external http or https endpoint.

By contrast, Kamelets are tailored around actual use cases. A todo-sink Kamelet may be created in the catalog to transform an event into an entry of a “To-Do Application” (I refer here to that kind of applications where you add the things you need to, and later mark them as resolved). It may use the camel-http component under the hood, but that will be an implementation detail. The todo-sink Kamelet will also expose to the end user not the full list of parameters available in the camel-http component, but a reduced set of configuration properties that are relevant for the business purpose (i.e. adding an entry into a “To-Do Application”). Similarly, when the Kamelet model will evolve with the help of the community, there won’t be a single facebook-source Kamelet, but there will be probably many facebook-related sources, one for each use case that the community finds relevant.

An important corollary of all this is that we’re not going to auto-generate the Kamelet catalog from the components, instead, we’ll follow a more opinionated approach taking into consideration the user needs rather than what the components already provide.

Contributing

Contributions are welcome for the Kamelet catalog. All you need to do is to follow the guidelines in the official repository for the Kamelet catalog on Github.

Here’s a list of relevant resources that you may want to learn before starting creating your own Kamelets:

Debugging Camel K

2020-11-30T00:00:00+01:00

Camel K introduces new ways of dealing with integration problems by letting you develop and run routes in the cloud, where the services that you need to connect live. This has many advantages, since it makes it easy to use all the features available in the cloud environment from the very beginning. For example, you can leverage service discovery, you can make your integration passive and let it receive push notifications from Knative with real data… But this comes with a cost.

When something goes wrong while you’re developing a new integration, you’d like to easily understand what’s going on and fix your code. Since the integration is not running locally, it’s not easy to leverage all your IDE tools to troubleshoot. But we’re working to improve the experience on many fronts. I’m describing here the new kamel debug command.

Debug command

Suppose you’re working on an integration problem and you’ve designed your route as:

import org.apache.camel.Header;
import org.apache.camel.builder.RouteBuilder;

public class Flaky extends RouteBuilder {
  @Override
  public void configure() throws Exception {

    from("timer:clock?period=5000")
    .bean(this, "createPayload")
    .to("https://postman-echo.com/post")
    .log("sent!");

  }

  public String createPayload(@Header("CamelTimerCounter") Integer tick) {
    return "" + (100 / (tick - 1));
  }

}

Note: I’m writing the “createPayload” method directly in the route file, but likely you’ll put it in a separate library.

This is a simple route that every 5 seconds builds a payload and sends it to a HTTPS destination.

You can run it with:

kamel run Flaky.java

Once it starts, you notice that there’s an issue in the way the payload is built, since the output signals an error in the first exchange.

[1] 2020-11-30 09:22:02,377 WARN  [org.apa.cam.com.tim.TimerConsumer] (Camel (camel-1) thread #0 - timer://clock) Error processing exchange. Exchange[746A22E1144A36D-0000000000000000]. Caused by: [java.lang.ArithmeticException - / by zero]: java.lang.ArithmeticException: / by zero
[1]     at Flaky.createPayload(Flaky.java:18)
[1]     at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[1]     at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[1]     at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[1]     at java.base/java.lang.reflect.Method.invoke(Method.java:566)
[1]     at org.apache.camel.support.ObjectHelper.invokeMethodSafe(ObjectHelper.java:208)
[1]     at org.apache.camel.component.bean.MethodInfo.invoke(MethodInfo.java:425)
[1]     at org.apache.camel.component.bean.MethodInfo$1.doProceed(MethodInfo.java:247)
[1]     at org.apache.camel.component.bean.MethodInfo$1.proceed(MethodInfo.java:217)
[1]     at org.apache.camel.component.bean.AbstractBeanProcessor.process(AbstractBeanProcessor.java:154)
[1]     at org.apache.camel.component.bean.BeanProcessor.process(BeanProcessor.java:56)
[1]     at org.apache.camel.processor.errorhandler.RedeliveryErrorHandler$SimpleTask.run(RedeliveryErrorHandler.java:404)
[1]     at org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.schedule(DefaultReactiveExecutor.java:148)
[1]     at org.apache.camel.impl.engine.DefaultReactiveExecutor.scheduleMain(DefaultReactiveExecutor.java:60)
[1]     at org.apache.camel.processor.Pipeline.process(Pipeline.java:147)
[1]     at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:287)
[1]     at org.apache.camel.component.timer.TimerConsumer.sendTimerExchange(TimerConsumer.java:207)
[1]     at org.apache.camel.component.timer.TimerConsumer$1.run(TimerConsumer.java:76)
[1]     at java.base/java.util.TimerThread.mainLoop(Timer.java:556)
[1]     at java.base/java.util.TimerThread.run(Timer.java:506)
[1] 
[1] 2020-11-30 09:22:08,088 INFO  [route1] (Camel (camel-1) thread #0 - timer://clock) sent!
[1] 2020-11-30 09:22:13,236 INFO  [route1] (Camel (camel-1) thread #0 - timer://clock) sent!

It’s easy to understand what’s wrong here, because this is a hand crafted example, but in a real integration case, your business logic may be much more complex and it won’t be easy to understand the problem, until you actually look into the running code.

There’s a new command in Camel K 1.3.0 that allows you to switch a remote integration to “debug mode”. This way the integration can be connected to your IDE and you can use breakpoints to check specific sections of the code for bugs. The kamel CLI will take care of setting the right flags on the integration and also establishing a secure tunnel to let your IDE connect to the integration.

To switch to the debug mode, just run:

kamel debug flaky

Where flaky is just the name of the integration. This is an example of output:

$ kamel debug flaky
Enabling debug mode on integration "flaky"...
Forwarding from 127.0.0.1:5005 -> 5005
Forwarding from [::1]:5005 -> 5005
[1] Monitoring pod flaky-76f67fb58-rk8g8
[1] exec java -agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=*:5005 -cp ./resources:[...] io.quarkus.runner.GeneratedMain
[1] Listening for transport dt_socket at address: 5005

As you see, the integration is waiting for you to connect your IDE on localhost on port 5005 using the debug protocol. You can configure your IDE to attach to the process via that host an port. This is IDE specific, but all modern IDEs support remote debugging.

If you’re using VSCode, you can put a launch.json file in the .vscode dir at the root of your project, with the following content:

{
    "version": "0.2.0",
    "configurations": [
        {
            "type": "java",
            "name": "Debug Camel K Integration",
            "request": "attach",
            "hostName": "localhost",
            "port": 5005
        }
    ]
}

This is enough to connect. The final result is shown in the following image:

IDE Debug

At the end of the debug session, the integration will be restored to its normal state.

Have fun with Camel K!

Kamelets

2020-10-12T00:00:00+02:00

Kamelets are the most important feature released with Apache Camel K 1.2.0. Apart from their cool name, Kamelets represent a significant change in the whole Camel ecosystem, because they introduce new ways of using Apache Camel in the cloud and a novel approach for contributing new connectors.

What is a Kamelet?

A Kamelet is a “Kamel Route Snippet”. Before going into the details of what this actually means, let’s make a step backward to add some background context.

Traditionally, the building blocks of Apache Camel have always been the components. Camel users can write complex routes by leveraging the 350+ components that are available in Apache Camel. It’s not a fixed pool: Camel developers and contributors are constantly increasing the collection of supported components at each new release.

This model has worked very well for all these years and it will continue to work. But it misses an important feature that we want to cover with Kamelets: the ability to abstract.

In Camel, if I want to publish a tweet in response to a Knative event, I’d do something like this:

from('knative:event/public.post')
  .to('twitter-timeline://user')

Every time some application sends an event of type “public.post” to the Knative broker, that event is published in my Twitter timeline.

As a user, I don’t care about what the “twitter-timeline” component is doing under the covers: it may contact a single API, or more than one, or it may establish several connections to various systems using strange protocols. I’m only interested in the result.

Now, being an enterprise user, I would like to do a similar thing with my own systems and create a new component to add an item to the inventory of my e-commerce application. I decide to create a new component named “company-inventory” that provides an “add” endpoint to implement such feature. So:

from('knative:event/new.item')
  .to('company-inventory://add') // easy, but now try to implement it

Now it’s time to implement the “company-inventory” component. When adding an item to the inventory, we need to call an HTTP API to get the ID of the actual item type, call another API if the ID is not already present, in order to create a new one, then a third API to add the item, then add another event to a third-party Kafka topic to synchronize downstream services.

Easy? Of course, with Apache Camel. But guess what? You can’t use Apache Camel when writing a new component.

True. Developers who contribute code to Apache Camel usually adapt existing libraries to the Camel APIs, but they can’t leverage existing components. There’s been an attempt in Camel 2.x to bring such possibility with the Routebox component, but it was not intuitive in some parts and missed a good delivery model, so it wasn’t widely used and it was finally removed in 3.x in favor of this new idea of Kamelets.

Kamelets come to the rescue. With Kamelets you can encapsulate the logic to connect to a specific system into route templates (new feature of Camel 3.5.0). And guess what? Kamelets are made of pure Camel DSL.

Kamelets are Kubernetes resources. We’ll see shortly how to write them. As any Kubernetes resource, you can write a Kamelet into a file and install it on a cluster using Kubectl:

company-inventory-add.kamelet.yaml

apiVersion: camel.apache.org/v1alpha1
kind: Kamelet
metadata:
  name: company-inventory-add
spec:
  # ...
  # The Kamelet will declare all accepted parameters
  # in JSON-schema format.
  # 
  # Skipping the details here.
  # ...
  flow: # Here's the route
    from:
      uri: "kamelet:source"
      steps:
        - to: "http://first-endpoint"
        - to: "https://second-endpoint/{{itemType}}"
        - choice:
          # ...
          - to: "kafka:downstream"

I’m not going into the details of how to write a Kamelet, that is covered in the Camel K user guide about Kamelets. We need just to know that:

A Kamelet exposes a well defined JSON-schema interface, that documents its purpose and defines the accepted parameters.
A Kamelet can be a source of data or a sink (consumer or producer, in the Camel jargon). The “company-inventory-add” Kamelet is a sink.
A Kamelet defines what Camel code should be executed when the Kamelet is used inside an integration: the code is expressed as route template that can use properties (e.g. “{{itemType}}”)

You can install the Kamelet on a namespace by simply executing:

kubectl apply -f company-inventory-add.kamelet.yaml

Once done so, the Kamelet becomes available in all integrations that are deployed in the same Kubernetes namespace. So you can write a route like this to use the Kamelet:

example.groovy

from('knative:event/new.item')
  .to('kamelet://company-inventory-add?itemType=grocery') // implementing this is much simpler

And run it with:

kamel run example.groovy

The logic about what it means to add an inventory item is encapsulated in the Kamelet (multiple HTTP calls, enterprise integration patterns and sync with Kafka), that can be shared among all other integrations. And it’s all Camel DSL.

This is only a simple example of what a Kamelet can do, there are many more interesting use cases out there.

What about Knative sources?

The first time we thought about Kamelets was in the context of Knative Eventing sources. It was the end of 2018, believe it or not. We thought that they could really make the difference in that space, but we were not ready for the leap.

We are currently providing CamelSources in Knative, and they are really cool because they allow users to write a small piece of Camel DSL to produce any kind of event, taking data from any system that Camel supports.

So, what’s wrong with CamelSources? They are really cool from my point of view. But the problem is that Knative users are usually not Camel users. As a Camel user, I can write a piece of Camel DSL in a few minutes and make it work. But people with a background in Go or Python… they don’t even know what Camel actually is.

But that’s where a Kamelet can help.

Camel developers and passionate contributors will write the Camel DSL, creating “connectors” for external sources using the language of Kamelets. In future versions of Camel K, we’ll create a catalog of curated Kamelets that will be installed together with Camel K. Those Kamelets will be available to anyone who installs Camel K on a Kubernetes cluster.

Without any knowledge of Apache Camel, people can list the catalog of the available Kamelets with standard Kubernetes tools:

$ kubectl get kamelets
NAME
inventory-source
twitter-source
slack-source
telegram-source
...

Kamelets declare the list of expected parameters using a JSON-schema format, so it will be easy to check them and provide values.

Users can then decide to use a Kamelet e.g. to push some twitter data into the Knative broker. To do so, they can create a KameletBinding.

apiVersion: camel.apache.org/v1alpha1
kind: KameletBinding
metadata:
  name: twitter-source-to-knative
spec:
  source:
    ref:
      apiVersion: camel.apache.org/v1alpha1
      kind: Kamelet
      name: twitter-source
    properties:
      keywords: Apache Camel
  sink:
    ref:
      apiVersion: eventing.knative.dev/v1
      kind: Broker
      name: default

As you see, the final user will not need to know anything about Camel, only use the source.

Of course, the KameletBinding resource is a wrapper for an Integration. I.e. under the hood the operator will create the Camel DSL corresponding to the binding, which is something like:

# only for demonstration, we don't generate groovy code ;)
from('kamelet:twitter-source?keywords=Apache+Camel')
  .to('knative:event')

This makes Kamelets available to different kind of users: Camel users and Knative users, with a different interface.

So, is it only about Knative?

Of course not: Kamelets are general purpose connectors, not directly linked to Knative.

If I want to send data from my Twitter Kamelet to a Kafka topic, I just need to create the following code and run it:

from('kamelet:twitter-source?keywords=Apache+Camel')
  .to('kafka:topic')

All Kamelets can be used to feed Kafka instead of Knative with the same exact approach. In fact, we’ve also extended the KameletBinding mechanism to Kafka via Strimzi.

What makes Kamelets great?

The main difference between a Kamelet and a Camel component is its purpose. While a component can serve several purposes by specifying different combinations of the parameters, Kamelets are driven by use cases. A Kamelet can let you perform a specific action on a system or gather data from another system, with a limited degree of flexibility.

Reducing the scope helps designing CLI or UI tools around them.

A Kamelet contains (see the Kamelet specification for more details):

Title and description of the purpose of the Kamelet and how to use it
A nice icon
The set of expected parameters in JSON-schema format
The schema of the output they produce (or the input they require)
Implementation (Camel DSL)

Writing a UI that allows browsing the available Kamelets, configure them and bind them to a destination should be pretty easy.

There are many possible use cases:

A specialized UI, e.g. in the OpenShift dev console, can let the user browse the available Kamelets and configure them via a graphical form to send data to a particular Knative destination
A second specialized UI about Kafka can include a section where the users can instantiate the Kamelets to bring data to a particular topic
Another platform, e.g. like Syndesis can use the Kamelets to provide additional connectors for creating end-to-end integrations

The great thing about Kamelets is that they are not bound to a spcific technology: they are reusable connectors.

Try them out!

Camel K 1.2.0 has already been released, so, why don’t you try this new feature?

Feedback is welcome, as well as any kind of contribution!

Meet us on Zulip!

Serverless Integration on Kubernetes with Apache Camel K

2020-09-08T00:00:00+02:00

I’ve presented Apache Camel at the last Kubecon EU conference that took place on August 19th, 2020.

As for many conferences this year, Kubecon EU was held online, not in the sunny Amsterdam as it was planned. No time to meet in person the developers that are making really good things in the Knative and Kubernetes space, but the talks were pretty awesome, and now everybody can watch them for free on YouTube.

I’m proud that Apache Camel was there, thanks to the hard work that all the team has been doing over the last two years, with the new Camel 3.x releases, Camel Quarkus and especially Camel K that bring all those features to Kubernetes.

The serverless space is an area where Camel K can make the difference. Camel K integration with Knative really rocks and we’re working every day on improving it.

If you want to know what I’m talking about, why don’t you listen it from myself?

As you’ll learn from the video, many cool features are cooking in the pot. So, stay tuned!

Event-driven serverless applications with Camel K

2020-07-09T00:00:00+02:00

One of the talks I’ve most enjoyed doing in 2020. A DevNation Tech Talk explaining where do we see Camel K in the context of event-driven applications.

Camel K 1.0 is here

2020-06-08T00:00:00+02:00

Apache Camel K has made a lot of progress since its inception and we’re now proud to announce the 1.0 release. We’ve been working hard in the past months to add more awesome features to Camel K, but also to improve stability and performance. This post contains a list of cool stuff that you’ll find in the 1.0 GA release.

NOTE: this article has been first published on the Apache Camel’s Blog.

First of all, if you’re living under a rock and it’s the first time you hear about Camel K, you can read some introductory blog posts here (1 - introducing camel k, 2 - camel k on knative) or look at the Apache Camel website that contains a Camel K section with a lot of material that is automatically generated from the Github repository.

User experience

Camel K development style is minimalistic: you need just to write a single file with your integration routes and you can immediately run them on any Kubernetes cluster. This way of defining things is common to many FaaS platforms (although Camel K is not a proper FaaS platform, but a lightweight integration platform) and it’s technically difficult to provide IDE support, such as code completion and other utilities, to developers.

But now we’ve done it. The integration tooling team has created some cool extensions for VS Code that make the development experience with Camel K even more exciting. You don’t need to remember the Camel DSL syntax, the IDE will give you suggestions and error highlighting.

IDE Autocompletion

Code completion works with Java code, but it’s not only limited to it: you also have suggestions and documentation out of the box when writing the Camel URIs and property files. And you also have many options to run integrations and interact with them, all integrated in the IDE.

Just install the VS Code Extension Pack for Apache Camel to have all these features available.

Getting started tutorials

Good tools are fundamental to have a great development experience with Camel K, but then you need to learn what you can do with such a great power. We’ve created a new repository in the Apache organization that hosts getting started examples: the camel-k-examples repository.

So far we’ve added guides that drive you through:

01 Basic: Learn the basics of Camel K and some interesting use cases
02 Serverless APIs: How to design a serverless (i.e. auto-scaling, scaling to zero) API and run it in a few minutes

The basic quickstart is also available online, so you can have a look at how camel k works without installing anything on your laptop.

More tutorials are expected to come in the following months. You are also welcome if you want to help us by contributing your own. They are based on the VSCode Didact project, that provides an awesome user experience.

If you are looking for Camel K code samples that you can just pick and run using the CLI, the examples directory of the Camel K main repository contains a lot of them. You can also run them directly from Github:

kamel run https://raw.githubusercontent.com/apache/camel-k/master/examples/Sample.java

You can find ready-to-use examples written in different languages (e.g. XML, JavaScript and others).

Serverless

Serverless is the most important area where we’re focusing the new developments in Apache Camel K, although, you should remember, you can have a wonderful Camel K experience even without serverless features. To enable the serverless profile in Camel K, you just need to have Knative installed.

In recent releases, we have added support for the most recent advancements in Knative, for example, Camel K is very well integrated with the Knative event broker and you can easily produce or consume events from it.

With 2 lines of code you can transfer events (e.g. generated by IoT devices) from your MQTT broker to the mesh:

bridge.groovy

from('paho:mytopic?brokerUrl=tcp://broker-address:1883&clientId=knative-bridge')
  .to('knative:event/device-event')

No kidding, you just need to write those two lines of code in a file and run it with kamel run bridge.groovy to push data into the Knative broker.

And you can also scale the Integration out (Integration is a Kubernetes custom resource, kubectl get integrations to see all of them) to have a higher throughput. Scaling here is manual because the source of events is a MQTT broker (but we’ve plans to put auto-scaling also in this scenario).

The Camel K embedded auto-scaling feature works really well when you want to react to some Knative events:

listener.groovy

from('knative:event/device-event')
  .to('http://myhost/webhook/random-id')

This integration is configured to receive all events with type=device-event and scales automatically with the load because it is materialized into a Knative Serving Service and automatically subscribed to the Eventing Broker via a Trigger.

It then receives a CloudEvent when your IoT devices produce something and scales down to zero if there’s no data coming. You just need to create it (as before, just kamel run listener.groovy), all the remaining configuration is done automatically by the Camel K operator.

We’ve added much more features for having a better integration with the Knative ecosystem and we’ve also fixed some compatibility and performance issues that were present in previous versions. The user experience is now much smoother.

If you are a Knative YAML developer (!), instead of using Camel K directly, you also have the option to use Knative Camel Sources which are part of the Knative release. They are wrappers for Camel K integrations that are compatible with all the tools used by Knative developers (such as the kn CLI or the OpenShift serverless console). Sources in Knative can only push data into the various Knative endpoints, but not the other way around (i.e. they cannot be used to publish data from Knative to the outside). In Camel K you don’t have this limitation: the Route is the fundamental building block of a Camel integration and you can do whatever you want with it.

Fast startup and low memory

We cannot say we’re serverless without mentioning the work that we’ve been doing in improving the performance of Camel K integrations.

Starting from Camel 3.3.0 which is the default version used by Camel K 1.0.0, you can benefit from all improvements that have been made directly in the Camel core to make it much more lightweight. More in depth details of the Camel core improvements can be found the following blog series that highlights what has been changed in the 3.x Camel timeline to reduce memory footprint and speedup the startup time, which is foundamental when running integrations in a serverless environment: part 1, part 2, part 3, part 4.

But improvements are not only limited to the Camel core: we’re doing much more. Several months ago we’ve started a new subproject of Apache Camel named “Camel Quarkus” with the goal of seamlessly running integrations on top of the Quarkus framework. As you probably know, Quarkus is able to reduce the memory footprint of Java applications and improve the startup time, because it moves much startup logic to the build phase. And Quarkus applications can also be compiled to a native binary, allowing a dramatic improvements in startup performance and very low memory footprint.

In Camel K 1.0.0 we support Camel Quarkus in JVM mode. A goal is to have also the in-cluster native compilation soon (for some DSL languages, such as YAML), in one of next releases!

To use Quarkus as underlying runtime, you just need to enable the Quarkus trait when running an integration:

kamel run myintegration.groovy -t quarkus.enabled=true

Quarkus is expected to be the default underlying runtime in the next release, and support for Standalone mode (via camel-main) will be deprecated and removed. This means that you won’t need to enable Quarkus manually in the next releases, but you still need to do it in 1.0.

Fast build time

Every application running on Kubernetes needs to be packaged in a container image, but in Camel K you only provide the integration DSL and the operator does what it takes to run it, including building images directly in the cluster.

The operator manages a pool of reusable container images and if you redeploy your integration code, it does try to reuse existing images from the pool rather than building a new one at each change, because it takes some time to build a new one. It was 1 minute at the beginning…

But Kubernetes is moving so fast that you cannot solve a problem once and forget about it, you need to take care of it continuously. It happened that some of our third party dependencies that we used for doing builds in “vanilla Kube” has slowly degraded in performance up to a point where Camel K user experience was highly affected.

We decided to work harder on the build system in order to dramatically improve (again!) the build phase of Camel K integrations.

Build time can be be now measured in seconds in dev environments such as Minikube. A bunch of seconds, most of the times. This is more than a simple improvement!

Better CLI

The ‘kamel’ CLI is the main tool we provide to developers to run integrations. It’s not a mandatory requirement: at the end, an Integration is a Kubernetes custom resources and you can manage it with any Kubernetes standard tool (e.g. kubectl). But the kamel CLI adds a lot of value for integration developers.

For example, if you’re a Camel Java developer it’s not super easy to remember the boilerplate that you have to write in order to instantiate a Camel route builder. Now you don’t have to remember that:

kamel init Handler.java

You get a Java file with all the boilerplate written for you and you just have to write your integration routes.

It works also with all other languages: Groovy, XML, YAML, Kotlin and JavaScript. For example you can write:

kamel init foo.js

This way you get a simple route written in JavaScript.

It’s not just that. Often Camel K developers need to add a lot of command line options to configure the final behavior of their integration. For example, you may want to add a custom library with the -d option or configure a trait with -t. E.g.:

kamel run -d mvn:org.my:lib:1.0.0 -d mvn:org.my:otherlib:2.0.0 -t quarkus.enabled=true Handler.java

Sometimes the number of command line parameters you’ve to add can become too many. For this reason we’ve added the possibility to specify them as modeline options in the integration file (done by adding a comment line with camel-k: as prefix).

Handler.java

// camel-k: dependency=mvn:org.my:lib:1.0.0 dependency=mvn:org.my:otherlib:2.0.0 trait=quarkus.enabled=true

// ...
// your routes here

Once the options are written in the file, you can run the routes with just:

// simply this, additional args are read from the file
kamel run Handler.java

The other options are taken automatically from the file modeline. The CLI also displays the full command to let you know what’s running.

This kind of configuration is extremely useful in CI/CD scenarios because it allows you to have self-contained integration files and you don’t need to change the pipeline to setup additional options. If you’re curious about the CI/CD configurations, you can follow the tutorial about Tekton pipelines to have more information.

Monitoring and Tracing

Ok, you’ve finished level 1 of Camel K development and you want to make serious things. You’re in a very good position because Camel K provides a lot of useful tools to add visibility on what your integration routes are doing.

Let’s suppose you’ve a Prometheus instance in your namespace and you want to publish your integration metrics:

kamel run Routes.java -t prometheus.enabled=true

That’s it. No need to setup services and labels to enable scraping. A default prometheus configuration file is also provided for the integration, with sensible defaults. Of course you also have the option to provide your own configuration for advanced use cases.

Now, let’s suppose you want to see what your routes are doing and trace the execution flow of an integration. What you need to do is to install an opentracing compatible application in the namespace, such as Jaeger, and run the integration as:

kamel run Routes.java -t prometheus.enabled=true -t tracing.enabled=true

That’s it again. The Camel K operator will add the camel-opentracing library and connect it to the Jaeger collector that is available in the namespace. Here again, advanced use cases are supported.

Master routes

Good old Camel users know why and when master routes are useful, but for those who are not familiar with the term, I’m going to provide a brief explanation.

Whenever you have an integration route that must be running, at any point in time, in at most one single Camel instance, you need to use a master route. Master routes can be declared by simply prefixing the consumer endpoint by the ‘master’ keyword and a name that will be used to create a named lock, e.g.

from('master:mylock:telegram:bots')
  .to('log:info')

It can be used to print all messages that are sent to your Telegram bot. Since the Telegram API support a single consumer only, you can guard the route with a master prefix to have the guarantee that there will be at most only one consumer at any given time.

If you’re wondering how there can be two instances running of you deploy one, well, think just to when you change your code and need to do a rolling update: for some time there’ll be two pods running in parallel. In some cases, you may decide to scale your service out but keep only one instance of a particular route among all the pods of your service. Or you may want to embed a master route in a Knative autoscaling service: in this case, the service can scale autonomously based on the load, but there’ll be only one telegram consumer at any time.

Master routes work out of the box in Camel K, you just need to put a prefix in your endpoint uri. A leader election protocol based on Kubernetes APIs resource locks will be automatically configured for you!

CronJobs

All complex enough systems contain several scheduled jobs. This is especially true for that part of the system that handles integration with the outside.

Ideally, if you need to execute a quick periodic task, say, every two seconds, you would startup an integration with a route based on timer to execute the periodic task. E.g.

from("timer:task?period=2000")
  .to(this, "businessLogic")

But if the period between two executions, instead of 2 seconds (“2000” in the Camel URI, which is measured in milliseconds) is 2 minutes (“120000”) or 2 hours (“7200000”)?

You can see that keeping a container with a JVM running for a task that should be executed once every two minutes may be overkill (it is overkill for sure when the period is 2 hours). We live in a time where resources such as memory and CPU are really valuable.

So the Camel K operator automatically handles this situation by deploying your integration not as a Kubernetes deployment, but as a Kubernetes CronJob. This saves a lot of resources, especially when the period between executions is high. When it’s time to run your integration code, a container starts, triggers the execution and then gracefully terminates. Everything is handled automatically by Camel K and Kubernetes.

There are cases when you don’t want this feature to be enabled, for example, when your code makes use of in memory caches that is better to keep between executions. In these cases, you can safely turn off the feature by passing the flag -t cron.enabled=false to the kamel run command.

The Cron feature does not only work with the timer component. We’ve also added a cron component since Camel 3.1 that works really well in combination with the cron trait.

So you can also write the cron expression in the route directly:

from("cron:job?schedule=0/5+*+*+*+?")
  .to(this, "businessLogic")

In this case, a new pod with a JVM is started every 5 minutes to execute your scheduled task. For the remaining 4+ minutes you don’t use any resource.

Transparency

Camel K does a lot of work for you when you run your integration code in the cluster and it’s possible that you put some errors in the code that can block the deployment process. We’ve added a lot of visibility on the deployment process that now communicates with the users via Kubernetes events that are printed to the console when you use the CLI.

This way you’re always notified of problems in the code and you can better understand what to fix to make your integration run.

How to try Camel K 1.0

The first step is to go to the Camel K release page on Github (or the official Apache release repository), download the kamel CLI for your OS and put it in your system path.

Installation is done usually using the kamel install command, but, depending on the kind of Kubernetes cluster you’re using, you may need to execute additional configuration steps. The Camel K documentation contains a section about installing it on various types of Kubernetes clusters.

If you have trouble or you need to install it on a particular cluster that is not listed, just reach out in the Gitter chat and we’ll do our best to help you.

Future

We’ve reached version 1.0.0 and this is a great milestone for us. But we are not going to stop now: we’ve big plans for the future and we’ll continue to develop awesome new features.

We need your help to improve Camel K and we love contributions!

Join us on:

Gitter: https://gitter.im/apache/camel-k
GitHub: https://github.com/apache/camel-k