Python Api To Launch Template Unknown Name Cannot Find Field

I've created and run a DataPrep job, and am trying to use the template from python on app engine. I can successfully start a job using gcloud dataflow jobs run --parameters 'i

Solution 1:

The issue is with the format that you are passing input1 and output1. They need to be between quotation marks like this:

input1 = '{"location1":"' + input + '" }'
output1 = '{"location1":"' + output + '" }'

I have tried sending the request with the same approach than you and it fails. It also fails if I later parse it back to string or json because it doesn't parse quotes correctly.

Solution 2:

Surely the format is something to do with your problem. I had the same use case to solve, but the output would be the files, instead of google bigquery dataset. and for me, the code with the following BODY parameter is initiating the google dataflow pipeline:

BODY = {
        "jobName": "{jobname}".format(jobname=JOBNAME),
        "parameters": {
            "inputLocations" : "{{\"location1\":\"gs://{bucket}/employee/input/patient.json\"}}".format(bucket=BUCKET),
            "outputLocations": "{{\"location1\":\"gs://{bucket}/employee/employees.json/file\",\"location2\":\"gs://{bucket}/jobrun/employees_314804/.profiler/profilerTypeCheckHistograms.json/file\",\"location3\":\"gs://{bucket}/jobrun/employees_314804/.profiler/profilerValidValueHistograms.json/file\"}}".format(bucket=BUCKET)
         },
         "environment": {
            "tempLocation": "gs://{bucket}/employee/temp".format(bucket=BUCKET),
            "zone": "us-central1-f"
         }
    }