Author(s): Rohit Singhal
Most organizations undergo rigorous Enterprise resource planning (ERP) system implementation projects along their journey towards digital transformation and business systems modernization. These ERP systems meet the business requirements of integrating data and processes across the various business domains of finance, sales, warehouse, manufacturing, global trade, human resources, etc. A critical aspect of any ERP implementation is data migration for a source or legacy system to the newly implemented target ERP system. For an organization to truly leverage the big data, machine learning and data automation capabilities offered by modern ERP systems, it is imperative that project implementation teams define and executed an enhanced data migration plan ensuring data quality, accuracy and completeness, as well as establishing data governance processes for ongoing operations. The article aims to explore the components of such an effective, all-encompassing data migration strategy and execution and the future trends around the same.
An ERP implementation project involves installing, configuring and enabling an ERP software for an organization to run day to day business transactions across various functions like procurement, manufacturing, global trade, sales, finance, human resources and many others. In addition to business process re-engineering and underlying systems configuration, a vital step in an organization’s journey toward ERP enablement is typically a one-time data migration effort which involves transferring data from multiple sources such as legacy ERP or other business systems, spreadsheets and tools into the target ERP system’s associated database. A data migration is limited to a one-time scope, which means that the data in the target ERP system starts its independent information life cycle after Go-live. For a successful data migration, an organization must closely assess the migration approach and strategy best suited to the ERP implementation scenario(s) at hand and draft a detailed data migration plan that sets strong control parameters for the migration scope and execution process. This article aims to dive deeper into some of the important areas of a data migration plan and possibly serve as a guide to organizations looking to embark on such a data migration journey.
Large ERP transformations typically follow a ‘big bang migration’ strategy which means that the data migration is performed with the full project scope and all the data, applications and configuration move to the target environment at once. At the end of the cutover activity and the associated data migration, all end to end business processes within the project scope are live and available to be used by users. The advantages of simplicity, speed, and lower costs make this approach attractive for transformations where there is a hard breakaway from legacy systems and associated processes. However, such an intense effort often requires carefully coordinated system downtime which might impact both internal and external business stakeholders. Project teams must ensure data quality, fallbacks and adequate post go live support to avoid any major business disruptions.
On the other hand, a phased data migration approach allows a more controlled and less effort intensive migration with focus on only a few business functions at a time from the project scope catalog. An organization may choose to migrate the basic building blocks of enterprise structure, chart of accounts, company codes and associated finance configurations and data followed by materials or customer related data. In other cases, global organizations may follow a geographically phased approach for various global sites with solution deployments in a country/cluster of countries. A phased approach is less stressful for all stakeholders in business, project and IT teams and usually allows for more focused testing. However, this usually extends the project timelines and subsequently, the project costs based on the need for resources for a longer period. Legacy systems must be maintained for a longer time as well in this scenario in addition to interim interfaces between the target and the legacy systems [1].
It is recommended that decisions around data migration strategy be discussed and documented in focused workshops with business, IT, project management and other important stakeholders.
The data migration process in an ERP implementation project is a carefully crafted, multi-layered process that is critical to the enablement of underlying business processes and goals of the project. During the planning and scoping phase of the project, the project teams aim to identify the key business processes, user stories, user requirements, change impacts, configurations, and the underlying data objects that require migration to the target ERP system. Identification of such data specific scope combined with data profiling and cleansing requirements analysis in the source system will give the team an estimate of the work ahead. The project team can then allocate resources dedicated to performing detailed source to target mapping exercises and build programs/ applications to transform data based on these mapping rules. Detailed data quality test scripts execution during multiple mock cycles will help the project management gain confidence in the quality of data being migrated during final production cutover. The figure below captures the end-to-end migration process along with key milestones.
Figure 1: Data Migration Process Overview with Key Milestones
A data migration plan describes the overall strategy, preparation and specifications for converting data from source system(s) to the target ERP system. From a technical perspective, a data migration plan contains the overall approach, assumptions processes, tools and mechanisms used for conversion along with data inventory and cross references between source and target data elements, schema, and metadata. It also addresses other elements such as the process and structures at each stage of the extraction, transformation and loading (ETL) cycle and the respective tools required for executing the same. From a management perspective, the plan addresses themes such as dedicated resources viz. data managers, business analysis, developers etc., data governance mechanisms and business stakeholders’ communication plans [6]. An ideal plan also dives into the data quality assurance and control aspect of the migration. Listed below are some of the typical objectives of a data migration plan:
In the context of ERP implementation in the pharmaceutical manufacturing industry, no data migration plan is complete without due consideration to GxP compliance to ensure safety, efficacy, traceability and quality of products and services delivered to a patient. Enforced by major health agencies around the world including the Food and drug administration (FDA), GxP compliance encapsulates a set of guidelines and regulations in process industries such as pharmaceuticals, biotechnology and medical devices. Depending on the industry or the nature of business, GxP can stand for Good Manufacturing Practices (GMP), Good Laboratory Practices (GLP), or Good Clinical Practices (GCP) [5]. Although GxP regulations encompass all aspects of the pharmaceutical manufacturing business, data management holds a foundational place in the guidelines published by the FDA, the European Medicines Agency (EMA), UK’s medicines and Healthcare products Regulatory Agency (MHRA), World Health Organization (WHO), and the International Council for Harmonisation or Technical Requirements for Pharmaceuticals for Human Use (ICH) to name a few. These guidelines cover aspects such as electronic data standards for reporting and regulatory submissions, data integrity, data governance, proper validation and documentation practices, data risk management, implementing data audit trails, organizational training on data management, and harmonization of technical standards globally.
Out of the variety of data objects within the scope of a large-scale SAP implementation, some of the common GxP data objects include product specifications, manufacturing process related objects such as BoMs, recipes, production versions, material masters, quality control objects such as inspection methods and associated results, clinical trial data, inventory/ distribution records and many more. The implementation project team must do a thorough analysis of all the data objects and associated attributes in the project scope to determine the extent of applicability to GxP and accordingly solidify a GxP validation process.
When executing a data migration plan, it is important for the project team to minimize system downtime, manual effort required to transform data, and provide a repository of record for all the data migrated. To this end, there are multiple automated data transformation tools available in the market with enhanced ability to connect with the source and target systems whilst creating a workflow that will fulfill requirements of data accountability, security and user friendliness [2]. An example of such a tool is SAP Advanced Data Migration and Management (ADMM) by Syniti [3] which provides businesses a customizable environment to monitor data quality and integrity, automate data transformation rules, and generate detailed reports depicting conversion results. An automated data transformation tool offers multiple advantages of handling large volumes of data, eliminating repetitive tasks, and the ability to capture complex business transformation rules. On the other hand, developing such a level of automation takes up budget and time. Significant resources might be required to build, test and validate the workflows built to execute data transformation. Even then, there may be some data defects that cannot be handled through automation and can only be fixed manually through supplemental data scrubbing [1].
Given the criticality of data migration to the success of an ERP implementation, there is considerable focus on the application of cloud based artificial intelligence (AI) and machine learning (ML) applications to various facets of the migration process. Intelligent data profiling, predictive data mapping, and automation of the data cleansing process are just some of the examples where cloud-based AI platforms can make significant impacts. Market leading ERP packages come equipped with a technology platform to enable easy integration of internal/external on-premise/cloud-based AI/ ML services which, in turn, can be utilized to build intelligent features to address the dynamic data management needs of an organization [4]. For instance, the SAP AI Launchpad is a central hub that helps businesses integrate AI/ML models developed using SAP’s AI Core and AI foundational services to master/ transactional data that may exist in SAP S4 HANA. These models may then use APIs and Services to access table level data in the HANA database, address tasks like customer insights, predictive maintenance, demand forecasting, or automated invoice processing and return the model results back to SAP S4 HANA. These results may be reviewed by SAP users and translated to actionable tasks to adjust inventory levels or creating purchase orders.
Modern ERPs like SAP provide the option for a business to interface with Natural Language Processing (NLP) concepts such as Large Language Models (LLMs) to process unstructured datasets for data migration. Advanced LLMs such as GPT 3.5 and GPT 4 provide the ERP system a framework for analyzing, cleansing and formatting an unstructured data source [4].
The discussion in this article highlights the importance of data migration planning and execution in ERP implementations. A well-defined data migration process in synchronization with a detailed migration plan covers all aspects of data management, clearly identifies the source and target systems architecture and paves the way for a successful data transformation. The current ERP market provides a number of automated tools for executing data transformation which may be leveraged by project teams to program complex business transformation rules in an accelerated manner with minimal manual effort. Several such automated tools continue to evolve with incorporation of AI/ML based modules that help businesses move faster through the migration process with increased systems intelligence.
