Talend Best Practice

Good Job design is at the heart of your Talend development. Here, we look at the different aspects of designing your Jobs. We can then, through the tutorial and reference pages, look at each of these in a little more depth, and how we can achieve our design goals.

 Read about these items in detail by viewing our Talend Best Practice menu items.

• Purpose of Job
• Job Architecture
• Reusability
• Error Handling
• Restart & Recovery
• Robustness
• Performance (speed of execution)
• Logging

Purpose of Job

The first and possibly easiest of our design goals is What's the purpose of the Job?. Unfortunately, this is often where it ends. This may be ok for your first simple Job; but this is not a good basis for working on a larger project either on your own or with a team of developers.

As with any software development, you want to write reusable and reliable code that is written once and run many.

It is the goal of this website to lay the foundations and show the building blocks of good Talend design – through tutorials, working examples and reference material.

Job Architecture

Even the simplest of Jobs needs some design thought. A trivial Job that reads data from a source, performs some manipulation and then writes the results to a target, may only require three basic components. But what if the Job fails? You may then need to start thinking about recovery and adding some error handling. Suddenly, the Job becomes much more complex than first thought. Often, it’s these aspects of Jobs that gets left for another day.

As Jobs become more complex, it’s important to think about the modularity of your work. What code is common across many Jobs and is reusable? Can a large task be broken down in to several smaller Jobs that can beorchestrated by a master Job?

Reusability

A key aspect to our overall design strategy is reusability. Each time we write some reusable and robust code, whether it’s just a simple fragment of code or an entire Job, we are closer to our goal of quickly writing fast and robust Job.

Talend provides many features to help you write reusable code. These include Jobs, Joblets, (Enterprise only), Context Groups, Repository Code Routines, Custom Components and other Repository objects.

As well as the features provided by Talend, there are other techniques that we can use. I usually have a Job in each of my projects named Palette. Here I store all of my pre-configured Components so that I can copy-and-paste them in to my Jobs with the minimal of effort. Of course, they are pre-configured using Context Variables so there is the minimum of effort each time they’re used.

For more on reusability within Talend, read our Talend Reusability Reference.

Error Handling

As with any programming, your Job may receive errors and these need to be handled.

These may be Exceptions thrown by the components in your Job or they may be other softer errors that you also need to handle in some way.

You may, for example, refer to a null Object and receive a NullPointerException. Unless you catch this exception and deal with it, your Job will usually terminate immediately. This may or may not be your desired result.

Sometimes, you may receive softer errors. Your Job may call a Web Service which is not available. No exception will be thrown; but the request may return an HTTP Status Code 503. This will not be fatal to Talend but, again, you may want to deal with this in a special way, for example, you may want to retry the request at a later time.

Talend provides a selection of components and techniques for handling errors.

Restart & Recovery

Now that you’re handling errors, you’re in a much better position to deal with your Job should it receive a fatal error.

Can your Job simply be restarted from the beginning? Is that practicable? Or should you restart the Job at a logical point of failure?

There are many techniques for restarting your work at a point of failure and we’ll look at some of these in more detail in a later article.

Robustness

Once the architecture of your Job is in place, errors are being handled, and there is restart and recovery capability, your Job is well on the way to being reliable and robust.

Performance

Performance (usually meaning speed of execution) is often a key metric of a Jobs success or failure. There are many aspects to tuning your Job, including: General Job design, Memory management, I/O, Sharing the load with inputs and outputs (e.g. Databases) and Parallelism. There are also factors that may not be under the direct control of the Talend Developer such as Network performance and the configuration of external sources and targets. These should all be taken in to consideration.

Logging

Finally, it’s good practice to record information about the execution of your Job, over time. This helps with performance tuning and resolving issues.