Create Batch
Create a Batch to run up to 15,000 Searches (100 when adding searches with include_html=true set) on a schedule. You can have up to 10,000 Batches set up on your Scale SERP account at any one time.
Batches are created by making an HTTP POST to the /batches endpoint. POST parameters can be either supplied as x-www-form-urlencoded parameters or by POST'ing a JSON object to the /batches endpoint.
Using include_html=true in Batches When adding searches with include_html=true to a Batch the maximum number of searches per Batch is 100 (rather than 15,000) because including the HTML within the response makes the Batch Result Sets much larger. The limit is in place to ensure Result Set files are of a manageable size. If you have need to run a large number of searches all with include_html=true then simply split the searches across multiple 100-search Batches.
POST /batches
Let's start with a simple example; we'll create a Batch that is set to run every day at 9am and 5pm. We'll use the notification_email parameter to specify an email address to send notifications to (when the Batch completes), the notification_as_csv parameter to request the Batch results be delivered in CSV format and the destination_ids parameter to request that the Result Sets generated by running the Batch be uploaded to the specified Destination ID of "ABCDEFG":
Scale SERP responds with the following JSON confirming the Batch has been successfully created. Note the Batch id , you'll use the Batch ID in subsequent calls to update, delete and add Searches to the Batch.
Batches can be scheduled on a monthly, weekly or daily basis. You can also set a Batch to be started manually, either via the Scale SERP Dashboard or the Start Batch API.
A Batches' schedule is defined by the schedule_type parameter. The values of schedule_type , and other associated parameters are shown below:
Batch schedules are executed in the timezone set up on your profile
schedule_type | Scheduling Parameters |
|---|---|
monthly |
|
weekly |
|
daily |
|
manual | When schedule_type=manual the Batch must be manually started, either via the Scale SERP Dashboard or the Start Batch API . |
POST /batches
The following parameters can be used when creating a Batch. The api_key parameter should be supplied as a querystring parameter on the request URL.
Parameters other than api_key should be supplied in either x-www-form-urlencoded or JSON format, in the request body.
Parameter | Required | Description |
|---|---|---|
api_key | required | The API key for your account, should be supplied on the request querystring, i.e. api_key=demo |
name | required | The name of the Batch. |
enabled | optional | Determines whether the Batch is enabled or not. Disabled Batches do not automatically start on their schedule. Parameter Values
|
schedule_type | optional | Determines the type of schedule the Batches uses. Parameter Values
|
priority | optional | Determines the priority of the Batch. When multiple Batches are queued, Scale SERP starts them in priority order. Learn more about priorities . Parameter Values
|
schedule_days_of_month | optional | Determines the days of the month the Batch is run when schedule_type=monthly or schedule_type=minutes , expressed as an array of integers, i.e. [1,3,20] for the 1st, 3rd and 20th of the month. Note When using schedule_days_of_month in combination with schedule_type=minutes the Batch is executed at the per-minute frequency set in schedule_minutes on the days of the month set in schedule_days_of_month . For example, if you set schedule_type=minutes , schedule_minutes=every_5_minutes and schedule_days_of_month=[1,3,20] your Batch would be run every 5 minutes on the 1st, 3rd and 20th of the month. When schedule_type=minutes you can use either schedule_days_of_month or schedule_days_of_week , not both simultaneously. |
schedule_days_of_week | optional | Determines the days of the week the Batch is run when schedule_type=weekly or schedule_type=minutes , expressed as an array of integers where 0=Sunday and 6=Saturday, i.e. [0,2] for Sunday & Tuesday. Note When using schedule_days_of_week in combination with schedule_type=minutes the Batch is executed at the per-minute frequency set in schedule_minutes on the days of the week set in schedule_days_of_week . For example, if you set schedule_type=minutes , schedule_minutes=every_5_minutes and schedule_days_of_week=[0,2] your Batch would be run every 5 minutes on Sunday & Tuesday. When schedule_type=minutes you can use either schedule_days_of_week or schedule_days_of_month , not both simultaneously. |
schedule_hours | optional | Determines the hours of the day the Batch is run when schedule_type=minutes , schedule_type=monthly , schedule_type=weekly or schedule_type=daily . Expressed as an array of integers between 0 (midnight) and 23 (11pm), i.e. [9,17] for 9am & 5pm. Note Batch schedules are executed in the timezone set up on your profile . Note When using schedule_hours in combination with schedule_type=minutes the Batch is executed at the per-minute frequency set in schedule_minutes throughout the hours set in schedule_hours . For example, if you set schedule_type=minutes , schedule_minutes=every_5_minutes and schedule_hours=[10,11,12] your Batch would be run every 5 minutes between the hours of 10am and 1pm. If you wish for a schedule_type=minutes Batch to be run continuously 24/7 then you shoud omit the schedule_hours parameter. |
schedule_minutes | optional | Determines the minutes frequency that the Batch is run when schedule_type=minutes . Parameter Values
Note When using schedule_type=minutes Batches are run at the selected frequency 24/7 unless specific hours are chosen in the schedule_hours parameter, specific days of the week are chosen in the schedule_days_of_week parameter or specific days of the month are chosen in the schedule_days_of_month parameter. |
destination_ids | optional | Specifies an array of Destination IDs (i.e. Amazon S3 Buckets, Google Cloud Storage buckets, Microsoft Azure Blob Storage or Alibaba Cloud OSS Buckets) that Result Sets from this Batch are uploaded to. Destination IDs can be retrieved through the Dashboard or via the Destinations API. |
notification_email | optional | The email address to send notifications to when new Result Sets are available for this Batch. |
notification_webhook | optional | The URL Scale SERP will send a webhook POST to when new Result Sets are available for this Batch. Should be a fully qualified URL, i.e. https://yourserver.com/scaleserp |
notification_as_json | optional | Determines whether Scale SERP sends download links (in the email notification if notification_email is set, or in the body of the webhook POST if notification_webhook is set) for the Batch Result Sets in JSON format, or not. If destination_ids are configured for this Batch then this setting determines whether Result Set files in JSON format are uploaded to the Destinations on Batch completion. Parameter Values
|
notification_as_jsonlines | optional | Determines whether Scale SERP sends download links (in the email notification if notification_email is set, or in the body of the webhook POST if notification_webhook is set) for the Batch Result Sets in JSON Lines format, or not. If destination_ids are configured for this Batch then this setting determines whether Result Set files in JSON Lines format are uploaded to the Destinations on Batch completion. Parameter Values
|
notification_as_csv | optional | Determines whether sends download links (in the email notification if notification_email is set, or in the body of the webhook POST if notification_webhook is set) for the Batch Result Sets CSV format, or not. To set the CSV fields included in the Result Set use the csv_fields parameter. If destination_ids are configured for this Batch then this setting determines whether Result Set files in CSV format are uploaded to the Destinations on Batch completion. Parameter Values
|
notification_csv_fields | optional | Determines the fields that are returned when notification_as_csv=true . Should be specified as a comma seperated list of fields (in nested field, dot notation, format). For more information see the CSV Fields Reference . |
searches_type | optional | Determines whether this Batch is locked to only allow searches of the specified type to be added to it. Locking a Batch to a specific search type has several benefits including allowing Scale SERP to automatically choose the correct CSV fields, appropriate to the searches in the Batch, to be selected by default when exporting a Result Set in CSV mode. It can also help organize your account when you have many Batches set up. To allow any searches type to be added either omit searches_type entirely, or set it to mixed . When set, the searches_type_locked=true property is present when retrieving the Batch letting you know that the Batch is locked. If you leave the Batch as unlocked (i.e. by setting searches_type=mixed ) then searches_type_locked=false will be present when retrieving the Batch and searches_type will be set to either mixed (in the case of the Batch containing mixed searches types) or a searches_type value automatically detected based on the type of searches that have been subsequently added to the Batch. Note The searches_type cannot be changed after the Batch is created. For more information see the Batch Locking documentation. Valid values for searches_type are: Parameter Values
|
Next Steps Update Batches