Data Migration, a Crucial Step in Digital Transformation
Data migration is a significant task for companies planning a system transformation. This is due to the paradigm shift happening in the technology world—the need to act on data. To make the migration process seamless, SAP has provided several automated Data migration tools since its inception.
This blog covers the latest migration tool of SAP, S/4HANA Data Migration Cockpit (Migrate Your Data), for a new system implementation.
Overview of SAP S/4HANA Data Migration Cockpit (DMC)
SAP S/4HANA Data Migration Cockpit (DMC) is SAP’s latest data migration tool to transfer data from a legacy system (SAP and non-SAP) to an SAP S/4HANA system. This tool provides an integrated environment allowing users to extract, transform, and load (ETL) data from legacy systems into SAP S/4HANA.
The usage of this tool is based on the following scenarios.
- When enterprises adopt a completely new SAP S/4HANA implementation.
Note: Historical data is not migrated during the greenfield implementation. - When enterprises want to rapidly modernize their existing legacy system to SAP S/4HANA— system conversion (brownfield migration).
- When enterprises want to reengineer/optimize their business processes— selective data transition.
However, understanding data migration and the underlying details is key to selecting the most appropriate strategy during implementation. Also, this sets the stage for a successful business transformation.
What is SAP Data Migration?
SAP Data migration is the sequential process of transferring data from a source system to the target system. SAP stores all the business data—cost center, profit center, bank, etc.—in the respective migration objects. In other words, each migration object refers to a collection of business data and a specific business entity in SAP.
Importance of SAP Data Migration
Therefore, migrating data needs careful assessment of what needs to be moved and what needs to be left. Such a tedious task will consume a lot of time and effort, while resulting in a few errors if done manually. And SAP S/4HANA DMC is the latest in the line-up of migration tools offered by SAP that automates the data migration process.
A Brief History and Evolution of SAP Data Migration Tools
SAP has offered different Data migration tools over a 5-decade long journey. Each iteration of the migration tool aligns well with the technological transition of that time.
The following timeline provides a glimpse of the data migration tool evolution.
From Mainframe to Client-Server
Until the transition from R/2 (mainframe) to R/3 (first client-server model), SAP used ‘throw-away programs’ to migrate data. Throw-away programs are one-time codes written by programmers to migrate data from R/2 to R/3. This transition enabled SAP to introduce its first data migration tool—Legacy System Migration Workbench (LSMW). Though the automation capabilities of this migration tool are not on par with today’s automation standards.
Legacy System Migration Workbench (LSMW) (1999)
LSMW is the first data migration tool introduced by SAP to transfer data from SAP/non-SAP systems to SAP. In LSMW, the target system was SAP R/3 and ECC systems.
Functionalities of LSMW:
- Followed a traditional files-based approach.
- Used standard interfaces, such as BAPIs, IDocs, Direct Input and Batch Input programs, and recordings.
- Business Application Programming Interfaces (BAPI): Standard interface that allows external applications to interact with SAP systems
- IDocs: Standard containers used to exchange data between applications.
- Direct input: This process eliminated programming efforts through Batch Data Communication.
- Batch input: This process required ABAP programs only for field mapping.
- Data transfer was done all-at-once or periodically.
- Data validation efforts were handled only after migrating the data and are limited.
- Supported only the legacy data formats.
SAP S/4HANA Data Migration Cockpit (Legacy Transfer Migration Cockpit (LTMC))
Migration Cockpit was released in 2016 as the successor to LSMW. Migration Cockpit is also a web-based migration tool and it came along with the release of SAP S/4HANA 1610. LTMC is the transaction code used within Migration Cockpit to simplify data migration process.
This was the first time when SAP delivered predefined migration objects, making it easy for new users of SAP S/4HANA. The tool also had templates and built-in mapping capabilities. In LTMC, S/4HANA (on-prem and cloud) were the target systems.
Enhanced functionalities of LTMC
- Supports multiple data migration approaches, such as file-based migration, staging tables, and direct transfer.
- Supports multiple data formats, including Excel.
- Uses a more modern and user-friendly interface.
- Data validation efforts were handled even before migrating.
- Allows mass transfer of master data with a single Mass Transfer ID available across Development, Quality, and Production. Mass transfer ID is a system-generated ID when a new migration project is created. This also differentiates projects across systems.
There is also a version of LTMC that came with SAP Fiori App since the release of SAP S/4HANA 2020.
SAP S/4HANA Data Migration Cockpit (Migrate Your Data)
Starting with the release of SAP S/4HANA 2020, the migration cockpit started using ‘Migrate Your Data’ app in SAP Fiori to handle data migration tasks.
This replaced the earlier LTMC interface, offering the following.
- Better migration project management.
- More predefined migration objects.
- Enhanced data transformation capabilities.
Foundation : SAP Fiori was integrated into the Migration Cockpit with the release of SAP S/4HANA 1809. This gave birth to the new enhanced version of Migration Cockpit, using a new Fiori-based interface.
The new version can be accessed through Migrate Your Data – Migration Cockpit in SAP Fiori app. Like the earlier version using LTMC, the target system here is SAP S/4HANA.
Enhanced functionalities of SAP S/4HANA Data Migration Cockpit:
- Merged two migration approaches— files-based and staging tables—into staging tables approach.
- Allows creation of migration projects only in the development landscape.
- Migration projects already created in S/4HANA 2020 using transaction LTMC can only be executed (read-only mode) in S/4HANA 2021
- The implementation team is guided at each stage of the data migration process.
- Data validation efforts are handled in the Prepare phase.
Migration Approach
Moving on to data migration approaches, finding the right approach is key to faster data migration. The current version of SAP S/4HANA DMC supports the following.
- Staging Tables Approach
- Direct Transfer Approach
Staging Tables Approach
A staging table is the database table that holds the data (to be migrated) of the source system and acts as a temporary staging buffer. Staging tables reside in the SAP S/4HANA DMC. In this approach, the user holds the responsibility to provision data. In other words, the user is responsible for selecting the data that needs to be moved into the staging table. There are two ways to use Staging Tables.
- Using template files in XML or CSV formats is the standard option.
- Users can also choose SAP-delivered or third-party ETL tool of their choice. The following are a few of the SAP-delivered ETL tools.
- SAP Data Services (DS) is used in complex data migration projects. This tool allows users to extract data from non-SAP or SAP systems, enhancing data quality and cleaning the data.
- SAP Landscape Transformation (LT) is used during mergers & acquisitions and system conversions. This tool performs real-time data replication and transformation across SAP landscapes.
- SAP HANA Smart Data Integration (SDI) is used when data from multiple sources, such as Oracle or legacy PoS systems, need to be mapped to the target system’s structure. This native SAP HANA tool performs continuous data replication from different non-SAP sources.
Here is a step-by-step procedure using the Staging Tables approach. Images used throughout the blog are extracted from SAP Learning.
1. Access Migrate Your Data (Migration Cockpit) from the Fiori launchpad.
2. Create a migration project.
- The current version only supports the creation of a migration project in Development (DEV) system.
- DEV-QAS-PRD constitutes the complete system landscape.
- Quality and Production: Non-modifiable environments that support only execution of the projects.
- Development: Modifiable environment that supports adding or editing projects.
Note: DEV-QAS-PRD are part of the Transport Concept in SAP S/4HANA Data Migration Cockpit.
3. Establish a database connection between source system and target system (either locally or through remote connection).
- Source system – the new SAP S/4HANA.
- Target system – legacy system (ECC or any other non-SAP systems) containing business and application data.
The reason behind establishing a database connection is to align data (to be migrated) to the internal schema of the SAP S/4HANA database.
In a remote system set-up, a secondary database needs to be established with a remote SAP S/4HANA system or SAP HANA Box. SAP HANA Box is a hardware or virtual environment where SAP HANA is installed and ready to use.
In a remote system set-up (once the connection is established), the system generates a staging table that contains the same internal schema of the target HANA database.
4. Continuing further, the user is prompted to choose a development package in the Migration Cockpit. Once the user chooses a development package, it cannot be changed later.
5. Migration Scope
a. The user selects the necessary data migration objects depending on their business requirements. This is available under Available Migration Objects in the SAP Help portal.
b. If a migration object is not available, users can still use Legacy Transfer Migration Object Modeler (LTMOM) to:
- design their business-specific migration objects.
- add/edit migration objects or add rules to a transport request.
Transport request is a package containing the changes made to SAP objects, such as programs, configurations, or customizations, which need to be moved from source system to target. LTMOM is the transaction code in Migration Cockpit to access MOM.
6. Download
The Migration Cockpit system will auto populate the staging tables. Some migration objects may contain more underlying data structures. This indicates a direct relationship between the staging tables and number of data structures. Hence, the number of staging tables will increase in proportion to the underlying data structures.
- Once the migration objects are selected, the system prompts the user to download an XML or CSV template.
- Downloaded template
- Contains information on how to work with the worksheets provided in Excel.
- Provides necessary documentation to work with each migration object.
- Shows the mandatory field lists to be filled manually.
- Difference between XML and CSV
- XML template provides one single file for each migration object. The downloaded file contains:
- Detailed information about how to map the fields between two systems.
- A detailed guide on relevant data structures.
- First introduced with SAP S/4HANA 2022 version, CSV format is considered to be the most flexible way to populate staging tables. SAP recommends this as an expert option because of the following reasons.
- Contains separate CSV files for each data structure.
- Unlike XML template, there are multiple files for one migration object.
- If the user prefers to use the naming logic delivered by SAP S/4HANA Data Migration Cockpit, ZIP CSV file performs field mapping (automatically) between the source and target systems.
- XML template provides one single file for each migration object. The downloaded file contains:
7. Upload
Once the downloaded template is prepared, the Migration Cockpit prompts the user to upload it.
8. Prepare
The system validates the data available in the uploaded template file. In the previous version of Migration Cockpit, data validation was a separate task.
Some migration objects may contain open mapping tasks. These mapping tasks need to be completed before the start of migration. In this case, Migration Cockpit allows the user to perform relevant value mapping between the source and target systems.
Also in this phase, the user must decide whether a data transformation is required.
- Once mapping is done, the mapped file can be downloaded and sent to respective business unit stakeholders.
- SAP uses control parameters to control specific data migration procedures for certain migration objects. In such scenarios, mapping needs to be done prior to data loading.
9. Simulate
The simulation phase checks whether all the migration objects contain necessary information to satisfy the underlying APIs.
SAP S/4HANA Data Migration Cockpit uses standard APIs to migrate data into the target system.
10. Migrate
Migration is the final phase of successful data migration. The Data Migration Cockpit also contains tabs, such as Monitor, Status, Job Management, to ensure flawless migration through each stage of migration.
Direct Transfer (DT) Approach
This approach was first introduced into SAP S/4HANA Data Migration Cockpit during SAP S/4HANA 1909’s release. This is applicable for SAP system users that have an ABAP-based source system in place. Also, any existing SAP user can upgrade to a new generation of SAP product using this approach.
In DT, data provisioning is done by establishing an RFC (Remote Function Call) connection between the source and target systems—source system is mostly an SAP ERP system. Remote Function Call (RFC) is the standard SAP interface for communication between SAP systems.
In a DT approach, the Migration cockpit is completely responsible for data provisioning in the source system. In other words, data selection is highly automated. Other than the Migration Scope phase, this approach follows the same process as that of Staging Tables.
DT is applicable for the following source systems. The target system here would be SAP S/4HANA.
- SAP ERP 6.0 (EHP6 and above)
- SAP Advanced Planner and Optimizer (APO)
- SAP Apparel and Footwear (AFS)
- SAP Customer Relationship Management (CRM)
- SAP Extended Warehouse Management (EWM)
Difference between Staging Tables and Direct Transfer
SAP Data Migration Best Practices
We have listed the best practices to be followed during data migration.
- Adhere to the detailed documentation of each Migration Object.
- Impact of release upgrade.
- Adding custom fields.
- Performance factors.
Detailed Information on SAP Data Migration Objects
The latest version of SAP S/4HANA Data Migration Cockpit contains 486 pre-defined migration objects. Users can now filter the migration objects based on the following.
- Object type
- Master data
- Transaction data
- Migration approach
- Staging table
- Direct transfer
- Custom field support
- Components, such as migration object modeler (MOM), migration cockpit workbench, and migration monitor.
- Migration object modeler: Defines data validation rules and maps fields.
- Migration cockpit workbench: User-friendly interface to upload, clean, and validate data against rules set by MOM.
- Migration monitor: Provides status, tracks errors, and reconciles data between the source & target systems.
Certain migration objects contain a ‘prerequisite’ section. The section contains information on validity period, object dependencies, mapping instructions, and post migration activities. The pre-requisites must be fulfilled before the start of data migration.
This information for a given migration object is detailed in a collective SAP Note. SAP Note contains a set of instructions to avoid known errors from SAP systems.
Impact of Release Upgrade
SAP recommends not to upgrade to a newer release during data migration, because of the following reasons.
- Any new version of SAP always comes with changes in their respective application logic and customization, especially in terms of object dependencies. Users can access this Object Dependencies part through documentation.
- Migration Objects containing such inconsistencies in release upgrade will not function as expected.
Supporting Custom Fields
Additional custom fields can be created using Custom Fields app in the Data Migration Cockpit. For example, users can now add a product-related bonus to an already available Product Master. However, the Migration Cockpit does not support adding custom fields for all migration objects.
Performance Factors in SAP Data Migration
SAP provides Knowledge Based Articles (KBAs) that contain technical and non-technical information for any product/service.
Non-technical: To ensure a seamless data migration, please follow these measures.
- Categorize data available in the source system before extraction. This is to avoid migrating outdated, incorrect or archived data.
- Close all the open processes in the source system to minimize the amount of open data.
- If there are too many master data articles, consider a phased approach. This allows the user to prioritize master data that needs to be migrated upfront.
- Run minimal migration objects in parallel to ensure the system is not impacted.
- Track the migration objects that are active during downtime to skip or postpone them as needed.
- SAP recommends testing in a quality/pre-production environment.
Technical: Consider the following technical factors during SAP data migration.
- Limit the number of data transfer jobs per migration object and migration project. Immediately after the Prepare phase, the migration cockpit will suggest an ideal number of data transfer jobs for a given migration object.
- Upload only a minimum number of migration objects in parallel to avoid the object queue from flooding. However, there are also a few migration objects that cannot run in parallel.
- The performance of an API—specific to a migration object—also influences the system performance.
Key Things to be Noted
- The migration user must be assigned with required roles for each migration object.
- Following guided principles is recommended for first time users.
- As of now, data migration from SAP S/4HANA to SAP S/4HANA is not possible.
- SAP S/4HANA Finance is considered an SAP S/4HANA system. Hence, data migration from SAP S/4HANA Finance is also not possible using SAP S/4HANA Data Migration Cockpit.
Conclusion
Migrating data to a new system is a challenge and SAP is constantly enhancing its data migration tools to tackle the challenge. Let us keep our fingers crossed to see the nuanced enhancements in the next version.
