Replicating Data to Google BigQuery

With Syniti Replicate, you can replicate relational data to a BigQuery dataset. BigQuery is currently supported as a target both in refresh and mirroring. Before setting up Syniti Replicateto replicate data to BigQuery, install the BigQuery and AVRO Serialization libraries.

Google client library to access the BigQuery API

  1. Download the Google.Cloud.BigQuery.V2 (version 1.4.0) nuget package from: https://www.nuget.org/packages/Google.Cloud.BigQuery.V2/.

  2. Copy the assemblies to the Syniti Data Replication Plugins/BigQuery folder. See below for the list of files.

  • BigQueryPlugin.dll

  • BigQueryPlugin.pdb

  • Google.Api.Gax.dll

  • Google.Api.Gax.Rest.dll

  • Google.Apis.Auth.dll

  • Google.Apis.Auth.PlatformServices.dll

  • Google.Apis.Bigquery.v2.dll

  • Google.Apis.Core.dll

  • Google.Apis.dll

  • Google.Apis.PlatformServices.dll

  • Google.Cloud.BigQuery.V2.dll

  • Newtonsoft.Json.dll

  • System.Interactive.Async.dll

AVRO™ Data Serialization

This is optional, and only needed if you are using AVRO serialization.

  1. Download the Confluent.Apache.Avro (version 1.7.7.7) from: https://www.nuget.org/packages/Confluent.Apache.Avro/

  2. Copy the assemblies to the Syniti Data Replication Plugins/AvroSerializer folder. See below for the list of files.

  • AvroSerializer.dll

  • Confluent.Apache.Avro.dll

  • Confluent.SchemaRegistry.dll

  • Confluent.SchemaRegistry.Serdes.dll

To define a target connection for BigQuery:

  1. In the Metadata Explorer, choose Targets, then Add New Connection from the right mouse button.

  2. In the Add Target Connection wizard, type a name for the connection and choose Google BigQuery in the Database field.

  3. In the Set Connection String screen, enter values for the following properties:

    Project ID The ID of the BigQuery project under which you want to upload data.
    Service Account Key Path to the local service account key file..
    Output folder Local folder where to create the temporary files before loading them to BigQuery.
    Serialization The type of serialization used to send data to BigQuery. Options available are:
    1 - CSV (Comma Separated Values)
    2 - JSON
    3 – AVRO (Typically recommended for performance and to maintain and support compatibility with different versions of schemas.)

  1. Complete the fields in the Target Connection wizard. Use the User Guide available from the Management Center Help menu or the Setup Guide, available for download in the Help Center as needed.

    2. In the Select Tables screen, if tables already exist, choose the tables to use in replication. Note that you can also create tables, or add them at a later time.

  2. Click Next to display the Actions screen.

  3. Optional. Select the next wizard that you would like to run.

  4. Click Next to display the summary, then click Finish to create the connection.

  5. The next step is to add target output representation to the Metadata Explorer. This will be represented as relational tables.

Now you can set up replications from whichever source connection you have defined.