Search 800 + Posts

Feb 5, 2024

Bulk loading data into Oracle Autonomous Data Warehouse (ADW) using Oracle Integration Cloud (OIC).

This document explores bulk loading data into Oracle Autonomous Data Warehouse (ADW) using Oracle Integration Cloud (OIC). It covers the benefits, steps involved, configuration details, and best practices for a successful implementation.

For technical details please refer my other post at Bulk Load ADW Feature with OIC

Benefits of Bulk Loading with OIC:

  • Performance: Bulk loading significantly improves data loading speed compared to traditional methods like SQL*Loader , Loading data using Read/Write file in OIC or Insert Feature of ADW Database adapter.
  • Security: Utilize OIC's built-in security features and access control for secure data transfer.
  • Automation: Set up scheduled integrations for regular data updates without manual intervention.
  • Scalability: Handle large datasets efficiently with OIC's cloud-based scalability.

Configuration Involved:

  1. Ensure your data is in a supported format (CSV, JSON, etc.).
  2. Structure the data according to your target ADW table schema.
  3. Create OIC Integration , App Driven Orchestration (ADO) or for event-triggered loads or Schedule Based Orchestration (SBO) for regular schedules.
  4. Configure Adapter:
    1. Specify the ADW connection details and target table.
    2. Select the "Invoke Bulk Load from Object Storage" operation.
    3. Provide the Object Storage bucket and file information.
    4. Optionally, configure file format settings.
    5. Configure Table name that you would like to load and column names that you would like to populate.
  5. Test and Deploy:
    • Test the integration thoroughly with test data.
    • Deploy the integration and monitor its performance.

Process Flow :

  1. Once OIC flow triggered it will download data file(s) from SFTP server
  2. Load data file into temp memory
  3. Call ADW adapter , and since we have configure ADW adapter with Operation : "Perform Bulk Data Import Operation" , it will load data file into OCI bucket and then load data into ADW Table



No comments:

Post a Comment