Polling Triggers
Polling triggers periodically check an API for new or updated data, for example every 15 minutes. The polling schedule and frequency is configurable by end users in the Automation Editor's dashboard.
Polling triggers should only be used when equivalent webhooks are not made available by the API provider. Polling triggers are less efficient, consume more system resources (as well as for the API itself), and can potentially miss new/updated data when many data rows are added/updated in the API.
Polling triggers begin by fetching a list of data from the API:
The example above is a polling trigger for new forms.
Polling triggers that monitor for new data must receive the data sorted by newest items first. Polling triggers that monitor for updated data must receive the data sorted by most recently updated first. Polling triggers will not operate correctly if the data is not sorted appropriately from the API!
Most API's return a fixed amount of data rows. Polling trigger developers need to be aware that if more items are added/update in the app than are retrieved from the API, those items will be missed and never processed by the automation.
For example, if a polling trigger fetches 100 data rows from the API and the automation runs every 15 minutes, then if the end user adds or updates 150 items in the app within 15 minutes 50 items will not be processed.
Running the automation at a faster polling speed can be one way to reduce the possibility of missed items. Retrieving more data rows can be another.
If an API supports fetching new or update items that have been added or updated since a given timestamp, it is usually best to take advantage of that as a way to avoid possibly missed data rows:
In this example the API has a minDate parameter that is being configured to fetch canceled appointments. The Utility – When In modules are being used to configure different values depending on whether the assembly is being run in an automation on the server rather than in the assembly or automation editors.
When running in an automation it is desirable to fetch a manageable amount of data rows such that the automation won't hit its timeout limits. In the above example up to 250 items are fetched for the previous 24 hours.
When running in the automation editor and assembly editor, just a single data row is needed. If the Trigger module is configured to emit dynamic data fields like is done for the above example, then the trigger must try to fetch at least one data row so its data fields can be parsed and emitted by the Trigger module to facilitate data field mapping in the automation editor. So the example above searches for one data row up to 999 days old, just to make an attempt to find at least one data row to facilitate the parsing of dynamic data fields.
After fetching data, polling triggers should then use the Trigger – Emit New Items module to determine which data rows are new or updated:
The example above is for a new form trigger. The Trigger – Emit New Items module is configured to examine all form ids. The module stores the form ids in the database. Each time the trigger is executed the ids stored in the database are compared against the form ids received from the API. Only data rows with form ids not currently in the database are emitted, representing new forms.
Note that the selected unique row identifier should usually be the same in the Trigger – Emit New Items module and the Trigger module. If they aren't the same the editor will display a warning when the assembly is saved. If you know what you are doing, you can choose to save the assembly anyway.
Triggers for updated data need logic to create a unique identifier that is a combination of the item id plus its last updated timestamp for use with the Trigger – Emit New Items module. In this manner, the Trigger – Emit New Items module will know when an item has been updated when its last updated timestamp changes:
The above example is the general pattern to use to build unique identifiers for updated data items.
Use a Loop module with a nested Text Builder module to concatenate the item ids with the last updated timestamp, then configure the Trigger – Emit New Items module to use the generated update identifier.
Don't forget that the Trigger module also needs to use the generated update identifier, too.
Any processing that a polling trigger needs to perform on data rows, like transforming data or making additional API calls to fetch additional details for each data row, should be performed AFTER the Trigger – Emit New Items module, since it emits only the data rows being emitted by the trigger. It would be wasteful processing to perform functionality on data rows that later get discarded and not emitted by the trigger.
All polling trigger assemblies must have a Trigger module at the end:
The Trigger module emits data fields for each data row that can be mapped to actions in automations. The example above shows static data fields being keyed in manually.
Data fields can also be parsed and extracted dynamically from the data:
The Trigger - Extract Data Fields module is able to parse out all data fields from most API's. Its output data stream can then be selected in the Trigger module for emitting the dynamically parsed data fields.
When a trigger emitting dynamic data fields is configured in the Automation Editor, the trigger is executed in order to fetch a record and parse its fields. The trigger's API call should use the Utility – When In module to configure the API parameters to try to fetch a single record, so end users don't have to wait any longer than needed when configuring the automation.
The Trigger module can be configured with default data fields if no item can be fetched from the API when configuring the trigger in the automation editor.
To configure default data fields, run the trigger in the Assembly Editor so that at least one trigger record is emitted. Open the Stream Inspector for the Trigger module, copy its output, and paste it into the bottom of the Trigger module.
Default output fields should only be defined for triggers that always emit the same fields, e.g. not for triggers that emit custom fields.
Polling triggers store data row identifiers in the database in order to avoid reprocessing records that have already been processed.
The Trigger - Emit New Items module reads row ids from the database. Each data row having an id already in the database is filtered from its input data stream. Only data rows with ids not in the database are emitted:
Data row ids are stored in the database by the automation execution engine when the first action completes its execution, regardless if the action succeeded or failed.
Consider if the trigger emits 5 data rows to be processed. The automation execution engine processes each data row serially, one after another. As the first data row begins to process, after the first action completes the data row’s id is stored in the database. As each action completes its execution the state of the engine is updated in the database so that if a failure occurs a retry can be performed from the point of failure.
Now consider if something interrupts the system while the second data row’s actions are being processed. The data row ids for the remaining third, fourth, and fifth data rows are not yet stored in the database. So the next time the polling trigger is executed, they can be emitted by the Trigger - Emit New Items module, assuming they are present in the API’s response of fetched data.
