This post uses the add() and remove() Java Collections calls to the set difference of two lists of files. Files from 2 directories are listed and one directory's files are removed from the other. The result is a difference used in later processing.
java.util.List.add() and List.remove() can gather the results of a tFileList for later processing. Pulling in a library like Commons Collections, these lists can be easily searched, sorted, transformed, and operated on with a variety of set-based operations (intersection, subtraction, etc.).
tFileList produces an iteration that can be converted to a flow using the tIterateToFlow. The flow can then drive a tJavaRow. The tIterateToFlow has the benefit of conveniently defining a flow schema. The schema in this example is a single field (the full file path), but could be extended with additional columns for the other variables produced by tFileList.
tSetGlobalVar
The job starts with a tSetGlobalVar, listed as "Init Builder List". You may have seen examples of setting variables of simple types like "FileName" set to "C:\data\file1.txt". This job sets a variable that is a Java object which will be unpacked in the other components.
Use the following as the Key and Value of the tSetGlobalVar component. There are no quotes around the 'new' statement.
"builderList" / new ArrayList<String>()
tFileList
Both tFileLists -- "List csv files" and "List txt files" -- are standard. They are searching for files of pattern "*.csv" and "*.txt". (In a production job, these components should use context variables that parameterize the directory. )
tFlowToIterate
The tFlowToIterate components connect the tFileList with custom Java code (tJavaRow). The tFlowToIterates defined a single-field schema. If additional file list information were needed, the schema could be expanded to include other tFileList variables.
Both tFlowToIterate components are similar except for the tFileList source.
tJavaRow
Two tJavaRows work with the list using the following algorithm.
This component uses the following tJavaRow code
List<String> builderList = (List<String>)globalMap.get("builderList");
builderList.add( row1.filename.substring(0, row1.filename.length() -4) );
In the Advanced settings tab, java.util.List is imported. (There is a line that can be uncommented.)
Remove from builder list
The second tJavaRow component uses
List<String> builderList = (List<String>)globalMap.get("builderList");
builderList.remove(row2.filename.substring(0, row2.filename.length()-4));
Remaining Work
The job ends with the execution of a tJava. This tJava can deal with the build-up lists or could be replaced with another set of iteration components.
You can always integrate large blocks of Java code into Talend for some functionality that's missing off-the-shelf. However, it's usually more maintainable to keep the jobs as standard as possible. If you do choose to write Java code, try packaging the Java code in a routine or component rather than embedding chunks throughout the job.
java.util.List.add() and List.remove() can gather the results of a tFileList for later processing. Pulling in a library like Commons Collections, these lists can be easily searched, sorted, transformed, and operated on with a variety of set-based operations (intersection, subtraction, etc.).
tFileList produces an iteration that can be converted to a flow using the tIterateToFlow. The flow can then drive a tJavaRow. The tIterateToFlow has the benefit of conveniently defining a flow schema. The schema in this example is a single field (the full file path), but could be extended with additional columns for the other variables produced by tFileList.
Job Comparing 2 Directories' Files |
The job starts with a tSetGlobalVar, listed as "Init Builder List". You may have seen examples of setting variables of simple types like "FileName" set to "C:\data\file1.txt". This job sets a variable that is a Java object which will be unpacked in the other components.
Use the following as the Key and Value of the tSetGlobalVar component. There are no quotes around the 'new' statement.
"builderList" / new ArrayList<String>()
tFileList
Both tFileLists -- "List csv files" and "List txt files" -- are standard. They are searching for files of pattern "*.csv" and "*.txt". (In a production job, these components should use context variables that parameterize the directory. )
tFlowToIterate
The tFlowToIterate components connect the tFileList with custom Java code (tJavaRow). The tFlowToIterates defined a single-field schema. If additional file list information were needed, the schema could be expanded to include other tFileList variables.
tFlowToIterate Configuration |
tJavaRow
Two tJavaRows work with the list using the following algorithm.
- Strip the suffix from a CSV file
- Add the CSV file (minus the .csv) to the builderList
- Strip the suffix from a TXT file
- Remove the file name from builderList
- The result is a list of suffix-less filenames
This component uses the following tJavaRow code
List<String> builderList = (List<String>)globalMap.get("builderList");
builderList.add( row1.filename.substring(0, row1.filename.length() -4) );
In the Advanced settings tab, java.util.List is imported. (There is a line that can be uncommented.)
Remove from builder list
The second tJavaRow component uses
List<String> builderList = (List<String>)globalMap.get("builderList");
builderList.remove(row2.filename.substring(0, row2.filename.length()-4));
Remaining Work
The job ends with the execution of a tJava. This tJava can deal with the build-up lists or could be replaced with another set of iteration components.
You can always integrate large blocks of Java code into Talend for some functionality that's missing off-the-shelf. However, it's usually more maintainable to keep the jobs as standard as possible. If you do choose to write Java code, try packaging the Java code in a routine or component rather than embedding chunks throughout the job.
Great Blog. It was very useful. Thanks.
ReplyDelete