March 18, 2019 Timothy Liu Engineering, Product Five Hidden Obstacles with the Salesforce Streaming API Introduction A previous blog post of ours provided a brief overview of the different options available for Salesforce events. Today, we’ll look at some hidden obstacles developers encounter when designing a solution that uses the Salesforce Streaming API, as well as just with Salesforce’s APIs in general. 1. Rate Limits We start with what seems to be the most obvious, yet non-obvious, pitfall with the Salesforce Streaming API—rate limits. Salesforce documents their rate limits based on each tenant’s license type here. Salesforce even provides an API endpoint to programmatically query this information at /vXX.X/limits/ called Limits. However, Salesforce often states that using the CometD protocol allows you to bypass the API request limits because tenants may exceed 15,000 or even 1,000,000 events in a single 24-hour period. Retrieving these events will not count against the daily API request quota. Check out our previous blog post on Salesforce’s unique tenant-based API request quota structure here for more information. The really interesting aspect to Limits is that Salesforce documents a separate quota for each of the types below: Daily Streaming API events (API version 36.0 and earlier)Daily durable Streaming API events (API version 37.0 and later)Streaming API concurrent clients (API version 36.0 and earlier)Durable Streaming API concurrent clients (API version 37.0 and later)Daily generic streaming events (API version 36.0 and earlier)Daily durable generic streaming events (API version 37.0 and later)Daily standard-volume platform events delivered to CometD clients 2. Limits again! After designing a Streaming API client that doesn’t encounter the previous rate limits, developers usually encounter a similar issue with PushTopics. Here is a brief description of what each error developers commonly receive means and how it affects an implementation: Maximum number of topics per tenant Salesforce only allows a certain number of PushTopic objects per tenant. This means that: The tenant may face resource utilization issues if multiple teams request events, due to the limited number of PushTopic queries available.Developers are discouraged from customizing the PushTopic queries for separate event streams because the number of PushTopic queries is capped. Maximum length of the SOQL query in the Query field of a PushTopic record The name of the game has been and always will be customization. Salesforce offers and encourages the use of custom objects and custom fields. However, when specifying the SOQL query in the topic, there is a limit to how long and detailed the query can be when retrieving information. Maximum length for a PushTopic name This seems nonsensical, but it truly exists! Imagine a tenant has a common custom object called SomeCustomObjectName. Don’t forget that Salesforce will now add the suffix __c to custom objects. If a developer wanted to create a PushTopic for this custom object and differentiate it from other topics, the object name SomeCustomObjectName__c would be too long since it crosses the limit of 25 characters! 3. PushTopics again! We’ve yet to begin talking about the logic behind handling streaming events, yet we run into another hidden obstacle on our path there. Although PushTopic objects are crucial to configure events, developers need to understand the underlying SOQL queries to create the right PushTopic objects. This is because some SOQL queries are invalid, such as the unsupported ones documented here. There are three inconsistencies between SOQL and SOQL in PushTopic queries that are particularly important: A developer can only query a single entity (SObject type) for each topic. This makes it difficult for developers to monitor more than a select number of objects (standard or custom) within Salesforce. Example query: SELECT Id, Name FROM Opportunities A developer cannot use aggregate queries. This makes transforming and reporting on data less valuable since it requires additional logic to be included in the app instead. Example unsupported query: SELECT Id, AVG(AnnualRevenue) FROM Account No support for Text Area fields. This means that a developer would have to ensure the exact field names and types for each of their queries is not a Text Area field type. The most common and popular field in Salesforce SObject objects is the description attribute, yet this remains unsupported in SOQL queries. 4. Message Durability and the 24-hour retention period Developers should make note of how Salesforce retains events for a tenant within a 24-hour period. The logic to handle processing an event stream for multiple push topics has the following caveats: Replay ID values are unique to each event stream. Be sure to store the subscription channel and replay ID values for each event stream to continuously process events.Replay ID values are not guaranteed to be contiguous for consecutive events. Do not rely on any numerical incrementing to validate streaming events.The 24-hour retention period is not a rolling 24-hours, but an absolute time period, so clients must ensure events are retrieved before the 24-hour retention period ends.Furthermore, the stored replayId may become invalid when the retention period resets, so make sure to reset with the default replayId values:-1: the event stream will begin retrieving events from the present time onwards-2: the event stream will begin retrieving events from the start of the 24-hour retention period 5. Supported Objects Finally, developers could run into an issue that may be a dealbreaker depending on the requirements of their app—the Streaming API supports all custom objects, but only supports a subset of standard objects. Here are the standard objects supported: AccountCampaignCaseContactContractLineItemEntitlementLeadLiveChatTranscriptOpportunityQuoteQuoteLineItemServiceAppointmentServiceContractTaskWorkOrderWorkOrderLineItem As mentioned in our previous blog post on tracking activity in Salesforce, developers can use the Change Data Capture or Generic Events framework, but those have similar limitations. However, developers can also simply poll for changes by using the exact same SOQL queries used for PushTopic objects to track only newly modified objects. Here’s an example query: SELECT Id,Name FROM ApexPage WHERE SystemModstamp > 2019-02-20T00:00:00Z AND SystemModstamp <= 2019-02-21T00:00Z Salesforce supports an IsDeleted field for some objects, enabling the query above to track deleted objects in addition to newly created and updated ones. This is equivalent to using any of the three streaming options mentioned earlier, other than the SOQL query counting towards the tenant’s daily quota. The Kloudless Events API Kloudless provides a unified API to connect several different CRM apps with a single implementation. Instead of dealing with all of the obstacles above, check out the Kloudless Events API. We implement the CometD protocol and handle all of the handshakes, connection resets, and stream processing required. We also enable apps to track changes to Salesforce objects that the Streaming API does not support. To learn more about how Kloudless can help your application’s integration strategy when it comes to Salesforce integration, check out our new official guide to API integration now!