Create Batch

create batch create a batch to run up to 15,000 searches (100 when adding searches with include html=true set) on a schedule you can have up to 10,000 batches set up on your scale serp account at any one time batches are created by making an http post to the /batches endpoint post parameters can be either supplied as x www form urlencoded parameters or by post'ing a json object to the /batches endpoint using include html=true in batches when adding searches with include html=true to a batch the maximum number of searches per batch is 100 (rather than 15,000) because including the html within the response makes the batch result sets much larger the limit is in place to ensure result set files are of a manageable size if you have need to run a large number of searches all with include html=true then simply split the searches across multiple 100 search batches example post post /batches let's start with a simple example; we'll create a batch that is set to run every day at 9am and 5pm we'll use the notification email parameter to specify an email address to send notifications to (when the batch completes), the notification as csv parameter to request the batch results be delivered in csv format and the destination ids parameter to request that the result sets generated by running the batch be uploaded to the specified destination id of "abcdefg" $ curl "https //api scaleserp com/batches?api key=demo" \\ d name="my first batch" \\ d enabled=true \\ d schedule type="daily" \\ d priority="normal" \\ d schedule hours="9,17" \\ d destination ids="abcdefg" \\ d notification email="john smith\@example com" \\ d notification as csv=trueconst axios = require('axios'); const body = { name 'my first batch', enabled true, schedule type 'daily', priority 'normal', schedule hours \[9,17], destination ids \[ 'abcdefg' ], notification email 'john smith\@example com', notification as csv true } axios post('https //api scaleserp com/batches?api key=demo', body) then(response => { const apiresponse = response data; console log('batch created ' + json stringify(apiresponse, 0, 2)); }) catch(error => { console log(error); });import requests body = { "name" "my first batch", "enabled" true, "schedule type" "daily", "priority" "normal", "schedule hours" \[9,17], "destination ids" \[ "abcdefg" ], "notification email" "john smith\@example com", "notification as csv" true } api result = requests post('https //api scaleserp com/batches?api key=demo', json=body) api response = api result json() print "batch created ", json dumps(api response)\<?php $body = http build query(\[ "name" => "my first batch", "enabled" => true, "schedule type" => "daily", "priority" => "normal", "schedule hours" => "9,17", "destination ids" => "abcdefg", "notification email" => "john smith\@example com", "notification as csv" => true ]); $ch = curl init('https //api scaleserp com/batches?api key=demo'); curl setopt($ch, curlopt returntransfer, true); curl setopt($ch, curlopt followlocation, true); \# the following options are required if you're using an outdated openssl version \# more details https //www openssl org/blog/blog/2021/09/13/letsencryptrootcertexpire/ curl setopt($ch, curlopt ssl verifyhost, false); curl setopt($ch, curlopt ssl verifypeer, false); curl setopt($ch, curlopt timeout, 180); curl setopt($ch, curlopt customrequest, "post"); curl setopt($ch, curlopt postfields, $body); $json = curl exec($ch); curl close($ch); $api result = json decode($json, true); print r($api result); echo "batch created", php eol; ?> scale serp responds with the following json confirming the batch has been successfully created note the batch id , you'll use the batch id in subsequent calls to update https //docs trajectdata com/scaleserp/batches api/batches/update , delete https //docs trajectdata com/scaleserp/batches api/batches/delete and add searches https //docs trajectdata com/scaleserp/batches api/searches/create to the batch { "request info" { "success" true }, "batch" { "id" "f994420d", "created at" "2020 01 01 00 00 000z", "name" "my first batch", "schedule type" "daily", "priority" "normal", "schedule hours" \[ 9, 17 ], "destination ids" \[ "abcdefg" ], "enabled" true, "status" "idle", "search total count" 0, "search page count" 0, "next result set id" 1, "results count" 0, "notification email" "john smith\@example com", "notification as json" false, "notification as csv" true, "search type" "mixed", "search type locked" false } } scheduling batches batches can be scheduled on a monthly, weekly or daily basis you can also set a batch to be started manually, either via the scale serp dashboard https //app scaleserp com/batches or the start batch api https //docs trajectdata com/scaleserp/batches api/batches/start a batches' schedule is defined by the schedule type parameter the values of schedule type , and other associated parameters are shown below batch schedules are executed in the timezone set up on your profile https //app scaleserp com/profile schedule type scheduling parameters monthly schedule days of month array of days of the month to run the batch, i e \[1,20] for the 1st & 20th schedule hours array of hours of the day to run the batch, i e \[9,17] for 9am & 5pm weekly schedule days of week array of days of the week (as integers) to run the batch where 0=sunday and 6=saturday, i e \[0,2] for sunday & tuesday schedule hours array of hours of the day to run the batch, i e \[9,17] for 9am & 5pm daily schedule hours array of hours of the day to run the batch, i e \[9,17] for 9am & 5pm manual when schedule type=manual the batch must be manually started, either via the scale serp dashboard https //app scaleserp com/batches or the start batch api https //docs trajectdata com/scaleserp/batches api/batches/start batch parameters post post /batches the following parameters can be used when creating a batch the api key parameter should be supplied as a querystring parameter on the request url parameters other than api key should be supplied in either x www form urlencoded or json format, in the request body parameter required description api key required the api key for your account, should be supplied on the request querystring, i e api key=demo name required the name of the batch enabled optional determines whether the batch is enabled or not disabled batches do not automatically start on their schedule parameter values true batch is enabled false batch is disabled schedule type optional determines the type of schedule the batches uses parameter values monthly batch is run on a monthly schedule, use schedule days of month to determine which days of the month and schedule hours to determine which hours on those days weekly batch is run on a weekly schedule, use schedule days of week to determine which days of the week and schedule hours to determine which hours on those days daily batch is run on a daily schedule, use schedule hours to determine which hours of the day the batch is run minutes batch is run on a per x minutes schedule, use schedule minutes to determine the minutes frequency that the batch is run you may optionally use schedule hours , schedule days of week and schedule days of month to refine the hours of the day, days of the week or days of the month that you wish the per x minutes schedule to be run manual batch is run manually, use the scale serp dashboard https //app scaleserp com/batches or the start batch api https //docs trajectdata com/scaleserp/batches api/batches/start to start the batch priority optional determines the priority of the batch when multiple batches are queued, scale serp starts them in priority order learn more about priorities https //docs trajectdata com/scaleserp/batches api/priorities parameter values highest high normal low lowest schedule days of month optional determines the days of the month the batch is run when schedule type=monthly or schedule type=minutes , expressed as an array of integers, i e \[1,3,20] for the 1st, 3rd and 20th of the month note when using schedule days of month in combination with schedule type=minutes the batch is executed at the per minute frequency set in schedule minutes on the days of the month set in schedule days of month for example, if you set schedule type=minutes , schedule minutes=every 5 minutes and schedule days of month=\[1,3,20] your batch would be run every 5 minutes on the 1st, 3rd and 20th of the month when schedule type=minutes you can use either schedule days of month or schedule days of week , not both simultaneously schedule days of week optional determines the days of the week the batch is run when schedule type=weekly or schedule type=minutes , expressed as an array of integers where 0=sunday and 6=saturday, i e \[0,2] for sunday & tuesday note when using schedule days of week in combination with schedule type=minutes the batch is executed at the per minute frequency set in schedule minutes on the days of the week set in schedule days of week for example, if you set schedule type=minutes , schedule minutes=every 5 minutes and schedule days of week=\[0,2] your batch would be run every 5 minutes on sunday & tuesday when schedule type=minutes you can use either schedule days of week or schedule days of month , not both simultaneously schedule hours optional determines the hours of the day the batch is run when schedule type=minutes , schedule type=monthly , schedule type=weekly or schedule type=daily expressed as an array of integers between 0 (midnight) and 23 (11pm), i e \[9,17] for 9am & 5pm note batch schedules are executed in the timezone set up on your profile https //app scaleserp com/profile note when using schedule hours in combination with schedule type=minutes the batch is executed at the per minute frequency set in schedule minutes throughout the hours set in schedule hours for example, if you set schedule type=minutes , schedule minutes=every 5 minutes and schedule hours=\[10,11,12] your batch would be run every 5 minutes between the hours of 10am and 1pm if you wish for a schedule type=minutes batch to be run continuously 24/7 then you shoud omit the schedule hours parameter schedule minutes optional determines the minutes frequency that the batch is run when schedule type=minutes parameter values every minute run the batch every minute every 5 minutes run the batch every 5 minutes every 10 minutes run the batch every 10 minutes every 15 minutes run the batch every 15 minutes every 20 minutes run the batch every 20 minutes every 25 minutes run the batch every 25 minutes every 30 minutes run the batch every 30 minutes every hour run the batch every hour note when using schedule type=minutes batches are run at the selected frequency 24/7 unless specific hours are chosen in the schedule hours parameter, specific days of the week are chosen in the schedule days of week parameter or specific days of the month are chosen in the schedule days of month parameter destination ids optional specifies an array of destination ids (i e amazon s3 buckets, google cloud storage buckets, microsoft azure blob storage or alibaba cloud oss buckets) that result sets from this batch are uploaded to destination ids can be retrieved through the dashboard or via the destinations api notification email optional the email address to send notifications to when new result sets https //docs trajectdata com/scaleserp/batches api/results/list are available for this batch notification webhook optional the url scale serp will send a webhook https //docs trajectdata com/scaleserp/batches api/batches/webhook post to when new result sets https //docs trajectdata com/scaleserp/batches api/results/list are available for this batch should be a fully qualified url, i e https //yourserver com/scaleserp notification as json optional determines whether scale serp sends download links (in the email notification if notification email is set, or in the body of the webhook https //docs trajectdata com/scaleserp/batches api/batches/webhook post if notification webhook is set) for the batch result sets https //docs trajectdata com/scaleserp/batches api/results/list in json format, or not if destination ids are configured for this batch then this setting determines whether result set files in json format are uploaded to the destinations on batch completion parameter values true json output format enabled / sent false json output format disabled / not sent notification as jsonlines optional determines whether scale serp sends download links (in the email notification if notification email is set, or in the body of the webhook https //docs trajectdata com/scaleserp/batches api/batches/webhook post if notification webhook is set) for the batch result sets https //docs trajectdata com/scaleserp/batches api/results/list in json lines format, or not if destination ids are configured for this batch then this setting determines whether result set files in json lines format are uploaded to the destinations on batch completion parameter values true json lines output format enabled / sent false json lines output format disabled / not sent notification as csv optional determines whether sends download links (in the email notification if notification email is set, or in the body of the webhook https //docs trajectdata com/scaleserp/batches api/batches/webhook post if notification webhook is set) for the batch result sets https //docs trajectdata com/scaleserp/batches api/results/list csv format, or not to set the csv fields included in the result set use the csv fields parameter if destination ids are configured for this batch then this setting determines whether result set files in csv format are uploaded to the destinations on batch completion parameter values true csv output format enabled / sent false csv output format disabled / not sent notification csv fields optional determines the fields that are returned when notification as csv=true should be specified as a comma seperated list of fields (in nested field, dot notation, format) for more information see the csv fields reference https //docs trajectdata com/scaleserp/search api/reference/csv fields searches type optional determines whether this batch is locked to only allow searches of the specified type to be added to it locking a batch to a specific search type has several benefits including allowing scale serp to automatically choose the correct csv fields, appropriate to the searches in the batch, to be selected by default when exporting a result set in csv mode it can also help organize your account when you have many batches set up to allow any searches type to be added either omit searches type entirely, or set it to mixed when set, the searches type locked=true property is present when retrieving the batch letting you know that the batch is locked if you leave the batch as unlocked (i e by setting searches type=mixed ) then searches type locked=false will be present when retrieving the batch and searches type will be set to either mixed (in the case of the batch containing mixed searches types) or a searches type value automatically detected based on the type of searches that have been subsequently added to the batch note the searches type cannot be changed after the batch is created for more information see the batch locking https //docs trajectdata com/scaleserp/batches api/locking documentation valid values for searches type are parameter values mixed allow searches of any type to be added to this batch web lock the batch to allow only searches of type 'web' to be added news lock the batch to allow only searches of type 'news' to be added images lock the batch to allow only searches of type 'images' to be added videos lock the batch to allow only searches of type 'videos' to be added places lock the batch to allow only searches of type 'places' to be added shopping lock the batch to allow only searches of type 'shopping' to be added autocomplete lock the batch to allow only searches of type 'autocomplete' to be added place details lock the batch to allow only searches of type 'place details' to be added place reviews lock the batch to allow only searches of type 'place reviews' to be added place products lock the batch to allow only searches of type 'place products' to be added place posts lock the batch to allow only searches of type 'place posts' to be added scholar lock the batch to allow only searches of type 'scholar' to be added product lock the batch to allow only searches of type 'product' to be added product sellers online lock the batch to allow only searches of type 'product sellers online' to be added product sellers local lock the batch to allow only searches of type 'product sellers local' to be added product reviews lock the batch to allow only searches of type 'product reviews' to be added product specifications lock the batch to allow only searches of type 'product specifications' to be added next steps update batches https //docs trajectdata com/scaleserp/batches api/batches/update