28
Dec
13

The 3 R’s of Spring Batch 3.0.x with Annotations


This page describes how you can Read, wRrite and perform aRithmetic on flat files using The Spring Batch Framework. We will take a comma separated file (csv) that contain employee information, add some information to it, and write it back to the file system.

Full downloadable source for this page is available here. Corrections and enhancements are welcome, fork, change and push back to GitHub.

The basic building blocks of any batch process is

  1. Reading a Item
  2. Performing an operation on it
  3. Writing the Item back

Please take some time to review The Domain Language of Batch before proceeding. It covers much of the fundamental concepts we will be covering here.

Batch Steps

This page is focused on an individual step of the batch process.

The following is from the spring batch documentation

A Step is a domain object that encapsulates an independent, sequential phase of a batch job. Therefore, every Job is composed entirely of one or more steps. A Step contains all of the information necessary to define and control the actual batch processing. This is a necessarily vague description because the contents of any given Step are at the discretion of the developer writing a Job. A Step can be as simple or complex as the developer desires. A simple Step might load data from a file into the database, requiring little or no code. (depending upon the implementations used) A more complex Step may have complicated business rules that are applied as part of the processing.

Step Processing types

There are 2 ways a step can process data,

Tasklet

If the step requires only to execute a single task then you can use a tasklet. Typical use case for this is when you need to run a stored procedure, or copy a file from one location to the other. In the “Hello World” example we used a Tasklet to print the message to the console.

Chunk oriented

Chunk oriented processing involves specifying a reader, processor and writer. The input is read one item at a time in sequence and passed to the processor and eventually to the writer in chunks within a transaction boundary. Once the commit interval is reached the items are committed to the writer. Chunk oriented processing is what we will cover on this page.

Library Versions

  • Spring Batch 3.0.0-M3 or above

Input Data

The following is the input csv file that will be read. Please create the following file in the projects resource directory.

vi src/main/resources/input_data.txt

7876,ADAMS,CLERK,1100
7499,ALLEN,SALESMAN,1600
7698,BLAKE,MANAGER,2850
7782,CLARK,MANAGER,2450
7902,FORD,ANALYST,3000
7900,JAMES,CLERK,950
7566,JONES,MANAGER,2975
7839,KING,PRESIDENT,-5000
7654,MARTIN,SALESMAN,1250
7934,MILLER,CLERK,1300
7788,SCOTT,ANALYST,3000
7369,SMITH,CLERK,800
7844,TURNER,SALESMAN,1500
7521,WARD,SALESMAN,1250

Employee Bean

This is a simple bean that represents a single Employee.

vi src/main/java/com/test/Employee.java

package com.test;

public class Employee {

	private Integer empId;
	private String lastName;
	private String title;
	private Integer salary;
	private String rank;
	
	public Integer getEmpId() {
		return empId;
	}
	public void setEmpId(Integer empId) {
		this.empId = empId;
	}
	public String getLastName() {
		return lastName;
	}
	public void setLastName(String lastName) {
		this.lastName = lastName;
	}
	public String getTitle() {
		return title;
	}
	public void setTitle(String title) {
		this.title = title;
	}
	public Integer getSalary() {
		return salary;
	}
	public void setSalary(Integer salary) {
		this.salary = salary;
	}
	public void setRank(String rank) {
		this.rank = rank;
	}
	public String getRank() {
		return rank;
	}
	@Override
	public String toString() {
		return "Employee [empId=" + empId + ", lastName=" + lastName
				+ ", title=" + title + ", salary=" + salary + ", rank=" + rank
				+ "]";
	}	
}

Reading

The reader is configured in the ThreeRJobConfig.java ( see reader() method )

Arithmetic

Not really! All we are doing is assigning a Rank based on the salary amount. The item processor takes an input Bean and converts it to an output bean. In this case the beans are the same but they don’t have to be.

vi src/main/java/com/test/EmployeeProcessor.java

package com.test;

import org.springframework.batch.item.ItemProcessor;

public class EmployeeProcessor implements ItemProcessor<Employee, Employee> {

	public Employee process(Employee emp) throws Exception {
		// if salary >= 2500 then set rank as "Director"		
		if(emp.getSalary() >= 2500 ) {
			emp.setRank("Director");			
		} else {
			emp.setRank("N/A");
		}
		return emp;
	}

}

Writing

The reader is configured in the ThreeRJobConfig.java ( See writer() method )

Job Configuration

vi src/main/java/com/test/config/ThreeRJobConfig.java

package com.test.config;

import java.io.File;

import javax.sql.DataSource;

import org.springframework.batch.core.Job;
import org.springframework.batch.core.Step;
import org.springframework.batch.core.configuration.annotation.JobBuilderFactory;
import org.springframework.batch.core.configuration.annotation.StepBuilderFactory;
import org.springframework.batch.item.ItemProcessor;
import org.springframework.batch.item.ItemReader;
import org.springframework.batch.item.ItemWriter;
import org.springframework.batch.item.file.FlatFileItemReader;
import org.springframework.batch.item.file.FlatFileItemWriter;
import org.springframework.batch.item.file.LineMapper;
import org.springframework.batch.item.file.mapping.BeanWrapperFieldSetMapper;
import org.springframework.batch.item.file.mapping.DefaultLineMapper;
import org.springframework.batch.item.file.transform.BeanWrapperFieldExtractor;
import org.springframework.batch.item.file.transform.DelimitedLineAggregator;
import org.springframework.batch.item.file.transform.DelimitedLineTokenizer;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Import;
import org.springframework.core.io.ClassPathResource;
import org.springframework.core.io.FileSystemResource;

import com.test.Employee;
import com.test.EmployeeProcessor;

@Configuration
@Import(StandaloneInfrastructureConfiguration.class)
public class ThreeRJobConfig {

	@Autowired
	private JobBuilderFactory jobBuilders;
	
	@Autowired
	private StepBuilderFactory stepBuilders;
	
	@Autowired
	private InfrastructureConfiguration infrastructureConfiguration;
	
	@Autowired
	private DataSource dataSource; // just for show...
	
	@Bean
	public Job threeRJob(){
		return jobBuilders.get("threeRJob")
				.start(step())
				.build();
	}
	
	@Bean
	public Step step(){
		return stepBuilders.get("step")
				.<Employee,Employee>chunk(1)
				.reader(reader())
				.processor(processor())
				.writer(writer())
				.build();
	}

	private ItemWriter<Employee> writer() {
		FlatFileItemWriter<Employee> itemWriter = new FlatFileItemWriter<Employee>();
		DelimitedLineAggregator<Employee> la = new DelimitedLineAggregator<Employee>();
		la.setDelimiter(",");
		BeanWrapperFieldExtractor<Employee> fieldExtractor = new BeanWrapperFieldExtractor<Employee>();
		fieldExtractor.setNames(new String[]{"empId","lastName","title","salary","rank"});
		la.setFieldExtractor(fieldExtractor);
		itemWriter.setLineAggregator(la);
		
		itemWriter.setResource(new FileSystemResource(new File("target/output_data.txt")));
		return itemWriter;
	}

	private ItemProcessor<Employee,Employee> processor() {
		return new EmployeeProcessor();
	}

	private ItemReader<Employee> reader() {
		FlatFileItemReader<Employee> itemReader = new FlatFileItemReader<Employee>();
		itemReader.setLineMapper(lineMapper());
		itemReader.setResource(new ClassPathResource("input_data.txt"));
		return itemReader;
	}

	private LineMapper<Employee> lineMapper() {
		DefaultLineMapper<Employee> lineMapper = new DefaultLineMapper<Employee>();
		DelimitedLineTokenizer lineTokenizer = new DelimitedLineTokenizer();
		lineTokenizer.setNames(new String[]{"empId","lastName","title","salary"});
		lineTokenizer.setIncludedFields(new int[]{0,1,2,3});
		BeanWrapperFieldSetMapper<Employee> fieldSetMapper = new BeanWrapperFieldSetMapper<Employee>();
		fieldSetMapper.setTargetType(Employee.class);
		lineMapper.setLineTokenizer(lineTokenizer);
		lineMapper.setFieldSetMapper(fieldSetMapper);
		return lineMapper;
	}
	

}

Execute the job

Go to the command line and type the following:

mvn compile exec:java -Dexec.mainClass=org.springframework.batch.core.launch.support.CommandLineJobRunner -Dexec.args="com.test.config.ThreeRJobConfig threeRJob"

View the Results

The output file will appear in the target/ folder of the project.

Further Reading

To keep things simple we were reading and writing files located in the project own folders. There are many enterprise design patterns that describe the best practices for feeding data into the batch programs. For further reading on this topic please see the Spring Integrations Framework Homepage.

Advertisements

3 Responses to “The 3 R’s of Spring Batch 3.0.x with Annotations”


  1. January 3, 2014 at 6:39 pm

    Good one for reference to switch to Java config based Spring batch !

  2. January 25, 2016 at 6:00 am

    Here i need to send a feed back mail so how to configure the job


Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


Enter your email address to subscribe to this blog and receive notifications of new posts by email.

Join 77 other followers

December 2013
S M T W T F S
« Oct   Feb »
1234567
891011121314
15161718192021
22232425262728
293031  

Blog Stats

  • 830,680 hits

%d bloggers like this: