12c SOA - To Process Large Volume Files Using- SOA File Adapter Chunked Read Option

ChunkedRead is a feature introduced in Oracle SOA Suite 12c File Adapter to process large files in smaller chunks instead of loading the entire file into memory at once. With SOA 12c, the File Adapter provides a dedicated ChunkedRead operation, making the configuration much simpler.

Why do we use ChunkedRead?

The main advantage is memory optimization.

Suppose we have a 500 MB or 1 GB CSV file:

Normal Read:

Large File
   ↓
File Adapter
   ↓
Load entire file into memory
   ↓
Transformation / Processing
   ↓
Higher memory consumption

ChunkedRead:

Large File
   ↓
File Adapter
   ↓
Read Chunk 1 → Process
   ↓
Read Chunk 2 → Process
   ↓
Read Chunk 3 → Process
   ↓
...
   ↓
Read final chunk → Process

Only a portion of the file is handled at a time, which helps reduce memory pressure and makes better use of the translator resources.

The key detail is that SOA 12c File Adapter ChunkedRead is implemented by putting the ChunkedRead Invoke inside a BPEL while loop and using JCA message properties to maintain the current file position between iterations.

Below is the complete implementation flow, step by step, including the variables, .jca properties, while loop, batch processing, EOF handling, and file deletion.

SOA 12c – Process Large File Using ChunkedRead

1. Business requirement

Assume we receive a large CSV file:

customers.csv

with:

Total Records = 12( for better understanding for now consider less no of records)
Chunk Size    = 3
SOA to process it like this:
Employee.csv
    │
    ├── Chunk 1 → Records 1-3
    │
    ├── Chunk 2 → Records 4-6
    │
    ├── Chunk 3 → Records 7-9
    │
    └── Chunk 4 → Records 10-12
So the File Adapter reads only 3 records in each iteration, rather than loading all 12 records together. 
2. Overall SOA design
The design is essentially:
                  FILE SYSTEM
                       │
                       │
                customers.csv
                       │
                       ▼
              ┌─────────────────┐
              │   File Adapter  │
              │   ChunkedRead   │
              └────────┬────────┘
                       │
                       ▼
              ┌─────────────────┐
              │      BPEL       │
              │                 │
              │ Assign          │
              │   ↓             │
              │ While Loop      │
              │   ↓             │
              │ ChunkedRead     │
              │   ↓             │
              │ Process Batch   │
              │   ↓             │
              │ Assign          │
              │   ↓             │
              │ EOF ?           │
              └────────┬────────┘
                       │
                  EOF = true
                       │
                       ▼
              Delete / Archive
                       │
                       ▼
                    END
3. Create the File Adapter
In JDeveloper:
Composite → External References → File Adapter
Create a File Adapter.
In the wizard, select the operation:
Chunked Read
This is important.
In earlier versions, the workaround involved SynchRead and manually modifying the JCA configuration. In SOA 12c, ChunkedRead is directly available as an operation.









4 – Create BPEL variables-This is one of the most important parts of the implementation.
Create these variables:

Variable
Purpose
lineNumber
Input/current line position
columnNumber
Input/current column position
isEOF
Loop/EOF control
returnLineNumber
Line position returned by File Adapter
returnColumnNumber
Column position returned by File Adapter
returnIsEOF
EOF returned by File Adapter
returnIsMessageRejected
Indicates rejected message
returnRejectionReason
Rejection reason
returnNoDataFound
Indicates no data found

Variables needs to be created are ->

<variable name="lineNumber" type="xsd:string"/>
<variable name="columnNumber" type="xsd:string"/>
<variable name="isEOF" type="xsd:string"/>

<variable name="returnLineNumber" type="xsd:string"/>
<variable name="returnColumnNumber" type="xsd:string"/>
<variable name="returnIsEOF" type="xsd:string"/>

<variable name="returnIsMessageRejected" type="xsd:string"/>
<variable name="returnRejectionReason" type="xsd:string"/>
<variable name="returnNoDataFound" type="xsd:string"/>
For the first read and before entering the while loop, create an Assign activity to set 
lineNumber   = 1
columnNumber = 1
returnIsEOF = 'false'

Create the while loop with condition isEOF = 'false'




Configure To-properties
These are values going from BPEL → File Adapter:
<bpelx:toProperties>
    <bpelx:toProperty
        name="jca.file.LineNumber"
        variable="lineNumber"/>
    <bpelx:toProperty
        name="jca.file.ColumnNumber"
        variable="columnNumber"/>
</bpelx:toProperties>
Configure From-properties
Now the File Adapter returns information back to BPEL.
These are:
<bpelx:fromProperties>
    <bpelx:fromProperty
        name="jca.file.LineNumber"
        variable="returnLineNumber"/>
    <bpelx:fromProperty
        name="jca.file.ColumnNumber"
        variable="returnColumnNumber"/>
    <bpelx:fromProperty
        name="jca.file.IsEOF"
        variable="returnIsEOF"/>
    <bpelx:fromProperty
        name="jca.file.IsMessageRejected"
        variable="returnIsMessageRejected"/>
    <bpelx:fromProperty
        name="jca.file.RejectionReason"
        variable="returnRejectionReason"/>
    <bpelx:fromProperty
        name="jca.file.NoDataFound"
        variable="returnNoDataFound"/>
</bpelx:fromProperties>




The while loop will iterate till the isEOF = true

As here there are 12 records and batch size is 3 there will be 4 batches to complete the chunk read process.

To remove to file after read
It needs to create another File Adapter with Synchronous read file operation specifically for deleting the file after chunk processing.

External References
│
├── FileAdapter_ChunkedRead
│
└── FileAdapter_Delete