Populating Databases¶
There are several ways to populate databases. Contingent upon your environment configuration, data location, and a range of other nuances, this process may vary among customers and applications. Please let the Game Warden team know (proactively) if you require data seeding or migration. Generally, you must have services for seeding and migrating data, or you must provide scripts for our team to execute for you.
Data Seeding vs Data Migration¶
Seeding and migrating both involve handling or managing data. However, each has a distinct purpose.
- Seeding
- Data seeding is the process of populating a database, providing an application with data to allow it to start or run for the first time.
- Migration
- Data migration involves moving data from one location to another, typically involving a database dump.
Note
You do not have direct access to your databases in IL4+ environments.
Data Seeding¶
Data seeding provides an application with its first set of data. For example, you might need to establish an Admin user who – in turn – invites other Admins. These Admin users might need application access to establish or configure the environment. In another scenario, you might seed a database to test automation or application functionality to ensure your solution works as designed. Similarly, you might need to seed data by adding continents, countries, states, and cities. You might need to add user accounts, the list of buildings at the university, or all bank branch offices in each state. The data you seed depends on the type of application in use coupled with the requirements or purpose for the initial set of data. Seed data provides foundational data for application function.
Seed data can be test data or real data. Test data might be helpful if you need to simulate application functionality.
The Game Warden team prefers that you perform all data seeding, assuming responsibility for managing this process. When you manage your data seeding process, there is greater flexibility – as you can use preferred data formats and determine the least disruptive seeding time periods. The Game Warden engineers are available to provide guidance. For example, you can provide us with your seed data, if you prefer, and we can add this data to a Docker volume to facilitate your access. Docker volumes ensure data persistence.
Seeding data commonly occurs before the application runs. If our engineers handle seeding, they must perform this process before the Functional Assessment because you must populate the database prior to testing. Our team’s data seeding process is manual and includes executing specific command-line statements to aid in this process. You must have services and/or scripts for seeding data.
Process Steps¶
The process for seeding data to Development (DEV), Staging (STG), or Production (PRD) might depend, in part, on your existing standards, tools, and application endpoints. If you have enabled endpoints for seeding data, you may fully manage this process. Alternatively, the Game Warden team might perform this process if endpoints do not exist or if required based on data sensitivity. In this case, you must provide our team with your seeding scripts and seed data. The Game Warden team would provide you with AWS credentials, using the Department of Defense (DoD) Secure Access File Exchange (SAFE), and you would then send your seeding scripts to an S3 bucket associated with our account.
Data Migration¶
Migrating data involves moving data from one location to another, typically via a database dump and transfer or via interaction with API endpoints in your application. Most commonly, migrations are needed to move Controlled Unclassified Information (CUI), that must be safeguarded, from the PRD environment of an external accredited platform into Game Warden. The process for this type of data migration is referenced in the External Accredited Platform Migrations content section.
If data sensitivity and application configuration allow, you can manage your data migration. When you manage your data migration process, there is greater flexibility – as you can use preferred data formats and determine the least disruptive migration time periods.
Process Steps¶
You can use one of two methods to migrate data:
From an external accredited platform, you must submit a request (typically via the external accredited platform’s ticketing system) to have your data sent to the Game Warden team. We then provide you with AWS credentials using DoD’s SAFE to have the external accredited platform's team send your data to an AWS S3 bucket associated with our account. The Game Warden team, via migration scripts or instructions you provided, then manages the process of getting this data into your Game Warden database.
Alternatively, from a database you control, you can initiate a data migration request via your customer Slack channel. Our team likely would already be aware of the need for the data migration and would work to schedule a time for the transfer. We then provide you with AWS credentials using DoD SAFE to send your data to an S3 bucket associated with our account. The Game Warden team, via migration scripts or instructions you provided, manages the process of getting this data into your Game Warden database. Alternatively, if you had endpoints to manage the migration, you could manage your own data migration. In this case, you must be mindful of any data sensitivity or restrictions.
Development-Staging-Production¶
Seeding and migrating data can be environment-dependent processes. While most environments require seeding to get applications running initially, data migrations usually occur in the STG and/or PRD environments just before you go live with Game Warden. Data migration would likely occur during deployment to STG for final application validation using mission data before deployment to PRD.
How and when you seed or migrate data depends on several conditions and, primarily, the goal you are attempting to accomplish. You might, for example, use the same seed data in DEV, STG, and PRD. The steps, generally, would be the same for each of the three environments. Another scenario might include seeding data in a lower Impact Level (IL) for testing then seeding this data differently in a CUI environment – such as IL4+. As another example, excluding IL2 STG for demo purposes, the Game Warden team commonly suggests customers drop a snapshot of their data into STG for testing before making the full cutover/migration in PRD. They test in PRD. If testing is successful, it is considered complete. If testing is unsuccessful, they revert and retry the process. Relative to timing, you should perform data migrations during the least disruptive time periods.
Note
You cannot have CUI at IL2. When testing application functionality in DEV, you need data; however, it cannot be CUI. You must use mock or synthetic data.
External Accredited Platform Migrations¶
There are two common methods to move data from an external accredited platform into the Game Warden environment:
- The migration occurs via the external accredited platform’s engineering team, typically triggered and documented via their ticketing system – making the external provider responsible for the migration to Game Warden.
- The Game Warden team manages the migration in collaboration with the external accredited platform’s engineering team. Typically, in this scenario, Game Warden engineers send AWS access keys to the external accredited platform. Access keys grant permission to copy files into an S3 bucket or similar object store (with no other permissions). Access keys for IL4 and IL5 are always sent via an encrypted and approved method, such as DoD SAFE, which is a secure means of sending and receiving files.
Staffers with the external accredited platform provider typically proceed by performing a database dump. Next, they push data to a Game Warden associated S3 bucket or similar object store. After the Game Warden team receives the data, they work with migration scripts or instructions you provided to ingest your data.
Downtime¶
During any data migration there is likely downtime, particularly if data is migrated from an external accredited platform. The Game Warden team will work to minimize this downtime based on mission needs. Generally, we advise you not to attempt to migrate data from an external accredited platform into Game Warden while your application is active and running on that platform. This action is not recommended because it will likely lead to missing or corrupted data, jeopardizing your data integrity. We will collaborate with you and your team to proactively plan data migrations, and we will consider the least disruptive time periods and ways to limit downtime.
Best Practices¶
Whether seeding or migrating data, you should align your actions with the following best practices to promote greater efficiency:
- Ensure you understand the exact data that requires seeding or migrating along with the need to comply with any existing company policies or procedures.
- Pre-determine the least disruptive data migration time periods and proceed accordingly.
- Manage data migrations within your application using tools such as Flyway, which facilitates the data migration process. If preferred, you also may use Flyway and similar tools for data seeding.
- Perform data migrations via automation using code, as this is required for higher Impact Levels where database access is restrictive (if not prevented).
- Test thoroughly after all data migrations.