What CEO and IT Directors Need To Know in 2020
As companies prepare for 2020 and move their business into the 21st century, many CEO’s and IT Directors are struggling to support their legacy mainframe systems. Legacy data from databases, reports, files, and software assets stored in mainframes are currently in use today – mostly for legal reasons or to maintain old accounting data but it poses a financial risk to organizations that still support these systems.
CEOs are focused on high performance, training, security, operational expenses, everyday expenditure, execution of the data plan, high-level data protection and increased data quality. Companies spend billions of dollars on migrating data, however, companies need to consider the style of the data migration and implement consolidation techniques when working with new technologies, vendors and tools.
One of the oldest legacy mainframes in use today is the IBM 2098 Model E-10 from the Defense Contract Management Agency – operated by the United States Department of Defense. As you can imagine, transferring data to a new system is a massive undertaking.
Key Problems with Legacy Mainframes
Heavy Maintenance Costs
These old legacy platforms are obsolete, costly to operate and repair and in general difficult to understand. Performance costs to uphold these mainframe databases can’t support new business initiatives.
Daily System Failures
Are an everyday occurrence, since most of the people who have implemented these mainframes are now long gone.
Poor IT infrastructure
IT Directors are struggling to support old technology. It’s time-consuming and they don’t have the knowledge or expertise to run these systems efficiently.
Legacy Systems Concerns
Challenges that plague organizations are real concerns and the legacy design around the old system is unfamiliar to the people who are supporting it.
- Knowledge is limited among the team and the skills to properly maintain the system are inadequate.
- The cost to repair old systems is costly and has become outrageously complicated to mange and the platform doesn’t serve critical business needs.
- The operating system is no longer in widespread use and the legacy files are problematic. Identifying disparate databases and securing data and systems is an everyday problem.
- Within the legacy system the old technology, method, computer system, or application programming language, software application, or process is outdated and can no longer receive support or maintenance updates.
- Replacing or updating cannot be done economically or easily and the legacy software is discontinued and unable to receive patch or hotfix updates.
- The process of data migration and moving data from one system to another, changing storage, database, dataset and applications is challenging, expensive and overwhelming for organizations.
Data Migration Facts
Executives are feeling the pressure to modernize their system and move into a cloud-based BI platform with automated data management.
Enterprise data migration projects can cost up to $875,000, with $250,000 in cost overruns.
- On average 38% of data transfer projects fail ¾ this is due to budget, time restrictions, resources, and poor team readiness.
- Up to 40% of the project cost is dedicated to data migration.
- 70% of BI implementations are unsuccessful and fail to meet business expectations and goals.
- The biggest problem with moving data is getting all of the stakeholders on board and signing off on the new system.
- Oddly enough, the most interesting tricky problem after that is recognizing the various ways databases deal with NULLs, NILs, N/As, blanks, 0s, spaces, leading and trailing spaces.
- It’s staggering to see the inconsistent mapping each of these have when moving from application to application.
The Benefits and Advantages of Business Intelligence Software
- Faster analysis, reporting, and planning
- More accurate reporting, analysis
- Improved data quality
- Better business decisions
- Improved operational efficiency
- Increased competitive advantage
- Increased revenues
- Reduced costs
- Valuable insights into business operations
- Excellent data visualization capabilities
- The ability to seek patterns in data, identify trends
The 5 Pitfalls in Data Migration
- Poor Data Quality – The biggest challenge for any company is data clean up and data conversion. Unlinked Data and Missing Links to External Data are just a few of the problems companies encounter. At the beginning of the project, not all the data will be coming from a single centralized database. Older databases, spreadsheets are all part of the data collection.When migrating old data, some original links may not be working and are broken links. Metadata is incomplete, outdated and the converted data after data conversion is disastrous. In-depth data analysis, data profiling, data structure, data cleansing and understanding the elaborate business rules for the legacy system is necessary for a smooth roll out.Duplicate Entries, Incomplete Data and Binary Data, and Encrypted Data are equally painful and time-consuming. Most of the project will be dedicated to removing unnecessary, unused, old, outdated, duplicate data or incorrect data.
- Lack of Company-Wide Adoption – To have a successful rollout of a migration project, all teams need to be on deck and focused on the mission. Data silos need to be broken down and disjointed groups, isolated teams or lagging responses will bring down the program and halt the project in its tracks.Consulting all key stakeholders and business owners is a necessary evil and will directly impact the success of the data migration implementation. Having the team see the cloud-based enterprise BI tool in action will help enforce the message.
- Creating a set of business requirements, timelines, and targeted weekly goals will ensure everyone is on the same page. Empower staff to have access to graphs, charts, and graphs including dashboards and encourage comments to be submitted on a day-to-day basis. Ask the business user what is working and what is failing.
Business Owners and Users Are Not Interested or Concerned
- Build a Data Stewardship portal that includes a data governance team structure with team calendar, dashboard, projects, tasks, business and IT policies, data quality expectations and protocols, tasks and issue log. Show how the data quality effort is progressing and timelines of what data items are left to still a process. Create an ad hoc question reference document to optimizing and quality efforts.
- Add all the data stewards on the team and which project is assigned to whom. Other items you can include are data services, business functions, and business applications, alternatives, and testing.
- Data Sources Needs to Analyzed and Data Scrutinized – Multiple ERP systems, CRM/CMS databases and Excel spreadsheets including Web Services, Social Media accounts, SQL Servers help in connecting an accurate picture of your organization. The project team will discover new data developments from trends, historical data, and the data sets can be merged quickly using the BI tools.By having a project management team focused on data sources and data quality will help in the data process and accelerate the migration development project.
- Determine the data project’s scope and create a data timeline plan with team leads
- Ensure that the data source migration plan is well-suited with existing strategy
- Establish a migration chronological order of events with dates, actions, and phrases. Build a project scope, breakdown structure, tasks, project dependencies, the total time needed for each action, identify resource availability, identify important milestones
- Validate and test data post-migration – Audit and document every step of the process.
- Slow SQL Queries – Bad Processing Database Performance – Businesses have reported that slow query performance as the top frustrating reason and number reason for problems in the data migration project. Recent advancements in technology and improvements in databases may help the user experience.Data Analysts and DBA’s, developers are finding new ways to improve processing by optimizing queries by adding query optimizers such as Ever SQL Query Optimizer, My SQL, Maris DB, and Percona DB helps to speed up query and performance and most are free.
- Allocate more memory will run your database more efficiently and boost system overall performance. Stay up to date with the latest version of your database will help processing times.
- Utilizing the correct type of disk for your server can significantly improve query operations. Another solution is data defragmentation allows you to free up more disk space for your database.
- The First Full-Volume Migration Test – The source data goes into a trial run of the migration data load to see if the content data and metadata is implementation ready; however, it comes with unexpected outcomes, scenarios, and unexplained problems.The source data needs to be thoroughly validated from the outset and a meticulous review of the source data is a must. Overlooking this phase will only guarantee a costly mistake, and the target system will not perform correctly.
- Corrective data cleansing work will need to done quickly. The target system vendor will need to be notified and a plan of action and in-house expertise will be called in.
- Outsourcing any re-data cleansing is a must and discovery efforts will need to put in place such as finding missing links and establishing an in-depth repository of knowledge, data validation, efficient data processes will also help to deliver accurate data transferring faster.
For the project to go smoothly, there are a few considerations to keep in mind.
Migrations Categorized into Four Types
- Storage Migration
- Application Migration
- Database Migration
- Business Process Migration
Migration Steps for the Source and Target Database Systems
- Planning Phrase
- Assessing the current environment
- Estimate the migration effort
- Choose the right tools and build a robust infrastructure
- Execution phase
- Unit Testing
- Integration Testing
- Validation and certification Testing
- Deployment Phase
- Performance Modifications
- Applications set up on the new systems
The Data Migration Process – Mainly Split into Two Projects:
- Migrating the raw data (tables, indexes, application data, and constraints)
- Migration of related infrastructure (stored procedures, database triggers, SQL queries, and functions)
Questions to Ask before Starting a Data Project
- Do the business owners have access to the source database? If so, for how long should it be for?
- What objects are we looking to migrate to the new system?
- Do we have the resources needed for the project team?
- Are the data requirements clear to the business owners?
- Are the timelines realistic?
Data Migration Prep Strategy
Is everyone on board with the data migration from business users to executive staff?
Can the product handle a large number of users and large data volumes?
Is the migration team able to get data from all the systems?
Is the software easy to use and flexible enough if changes need to be made quickly?
Will query times and cross-referencing query performance meet expectations?
Does the product include all the key features my organization needs?
Types of Data
What kinds of data relationships does my organization have that need to stay intact?
Proof of Concept
Can the data migration services company walk my organization through a step-by-step past migration project?
Organizations wanting to implement a legacy application data migration project need to follow these five rules.
- Follow a Best Practices Document Include skills, tools, plans, timelines, budget and data stewards and contact information for all teams, groups, and activities.
- Implement a Data Discovery Plan
- Data Team needs to understand the data source, target application data and address the concerns and how to access it over time.
- Complete Data Movement Strategy Create a document that includes how the data will be moved and strategies for access, validation, and audits.
- Collaborations of all Business Users and Data Stewards
- Add accountability to each team member and reward groups for participation and involvement. Stress the importance of supporting the migration project and the efforts in verifying data.
- Suitable Open Source Data Migration Tools Incorporate the right tools to support your data migration include processes, procedure, and workflows. Ask IT for advice and utilize the most popular and best suited for cloud data migration. The tool repository needs to support homogeneous and heterogeneous migrations and remember the migration tool needs to be fully operational throughout the data migration project.
Steps in Data Migration
- Identify the data format and location and data sensitivity
- Extensive planning and accurate timelines for the size and scope of the project
- Backup all data and remove obvious data duplications
- Assess staff skills, business owners and users
- Understand the migration tools capabilities
- Execution of the data migration plan
- Testing of the final system
- Follow-up and make the necessary tweaks in the maintenance of the data
Successfully planning an application data migration project that involves a legacy system and moving to modernized BI cloud-based software in order to improve performance, speed and move the organization into the digital environment is usually initiated by IT Directors.
On average the implementation cost for a data migration project can start at around $100,000 and the project will last from 12-26 months with a team of up to 15-30 developers.
Ask your data migration provider for testimonials, case studies, reports, graphs, charts and info-graphics proving their past project success. Inquire about past data migration projects and request examples of a comprehensive methodology and deliverables including what the project looked like across all migration phases?
Companies should use an experienced partner, get experts involved at an early stage, have a clear understanding of your data, and budget accurately plus plan the full scope of the data migration project, get extra internal and external resources and future proof of your new system. Remember: Self-Service tool software can simplify the process, reduce costs and help plan the migration.
Keep Moving Forward with Aptude
Aptude is your own personal IT professional services firm. We provide our clients with first class resources in a continuous, cost-containment fashion.
Our support services will free up your senior IT staff from the overwhelming burden of day-to-day maintenance issues. Now they’ll have time to launch those new projects and applications you’ve been waiting for. Simply put, we can free up your resources and contain your costs. Let’s have a quick chat to discuss our exclusive services.
What Are You Working On?
Looking for intelligent technological solutions? Seeking consultation on your upcoming projects? Need a quote for services? Contact Aptude’s executive team directly. It’s amazing just how much one little email can rapidly accelerate your productivity.
Some of our Clients Include…
Gain Time, Increase Currency, Contact Us
It’s amazing how one quick email can change your life. Give us a shout! We’ll get back to you right away with the right person for what you’re looking to accomplish.
What our clients are saying…
Aptude provides onsite and offshore Oracle DBA support, which includes troubleshooting, back-up, recovery, migration, upgrades, and daily maintenance of Oracle database servers. Aptude has been working with our team for the past four years and we continue to use them and are satisfied with their work
Aptude provided Build.com a Java, MySQL, Webservices and other UI based solution in the business domain of analyzing and reporting on user activities for our ecommerce website. Utilizing Omniture’s APIs to download, parse, and regenerate and upload back so that we could be more effective in our marketing. I was satisfied with their project work and delivery and would consider utilizing them for future projects.” Build.com
Aptude provided us with Oracle DBA migration support, including an upgrade from Oracle 11.1 to Oracle 11.2, and the project was completed on time and to specifications. The project manager and project consultants were responsive and proactive, resulting in a successful conclusion to the work. I would definitely contract with them again, and have recommended them to other technical offices at the University of Georgia.
Thank you for the hard work your team has put forth to staff the contract positions at Wolters Kluwer. Aptude has consistently scored high in our supplier carding and even more important you are a vendor we can always trust. I am especially impressed with your ability to tackle our positions that other vendors have not been able to fill.