Migration to Data Pipeline
Postgres Read Replica -> Snowflake Data Pipeline
As of April 1st 2023, access to the Read Replica will be deprecated and access to data will be managed through the Data Pipeline.
Previously customers had read-only access to their production data through a Read Replica of their production database. This was a temporary solution with a few different downsides.
In order to better serve the data needs of our customers, we have built a Data Pipeline which we will be transitioning to in April 2023.
- Data Pipeline is using Snowflake instead of Postgres
- New credentials and endpoints that will provided
- Customers will then need to update any existing connections to the new credentials.
- Existing queries will be backwards compatible with tables/joins
- All old tables/columns will exist. New columns are just being added to existing tables.
- There are some slight variations in functions between Snowflake and Postgres but should be very uncommon.
- Data in the pipeline will not be instantaneous since since it is undergoing significant transformation.
- Current Refresh Rate is 30 min
- Pre-calculated Fields
- There are some calculated fields that don't exist in the Read Replica because they are dynamically calculated in the app.
- Previously this logic needed to be copied and maintained in SQL but now will be directly in the database (i.e. Inventory Status, Run Status, etc.)
- Flattening of data on the base tables to make querying simpler w/ less joins
- i.e. User Name, Email in Run table directly instead of joining through the users table
- Will continue to add more functionality/simplification on top of this as well
- i.e. flattening of custom attributes onto the base tables instead of having to join into the attributes tables
- Access to data in both Staging and Production environments
Last modified 2mo ago