May 9, 2019 David Hallinan Engineering, Guides, Storage API The Pitfalls of API Integration Services: DropBox Edition “If you light a lamp for somebody, it will also brighten your path.” Buddha Here at Kloudless, tens of thousands of hours have been spent learning the intricacies of the APIs that we provide connections with. Our developers have toiled long days and nights in the pursuit of a deep understanding of the services we allow applications to integrate with, not only in the aim of bolstering our own product but also for the self-betterment that comes with the true comprehension of an outside service. They say that mastery of a subject or profession comes with 10,000 hours of practice. If that much is true, we like to think we’ve mastered a couple of APIs at this point. In the vein of the quote that preceded this piece, we’d like to light the metaphorical lamp of API integrations for the others that come after us. In this article, the first in a series of guides aimed to help you navigate the pitfalls of specific APIs, we will attempt to walk you through the trials and tribulations that we went through when integrating with DropBox, one of our more popular API offerings, in the hope that you can be prepared for the development speedbumps you might encounter while incorporating DropBox into your application. DropBox Rising Boasting more than 500 million users, DropBox has been a bona fide giant of the cloud-based file storage world since its release in 2008. DropBox became the first of the ‘folder sync’ storage applications to become a household name, having been originally conceived when one of its co-founders left his USB flash drive at home and was stuck with no way to access his necessary files for a college class. As a result, DropBox has become ubiquitous with users looking to manage their files and documents in the cloud, and heavily utilized by applications looking to provide their users with storage integrations. The Problem Dropbox operates in a unique fashion when it comes to file synchronization. The storage application creates a folder on the user’s computer where files are placed to be stored both locally and in the cloud. Every time a user updates a file or places a new file in the folder, DropBox monitors that change and then syncs the changes with the cloud, updating the new file being stored or creating a new copy of the file being placed there. Because of this operating procedure, a feature of major importance to Kloudless when integrating DropBox into our Unified Storage API was being able to track the events such as files being created or updated and relay that information to the user. This became a problem for us when DropBox launched V2 of their events API, drastically changing the schema of the objects being returned. While this was positive for Kloudless in the long run, since the new objects contained far more key-value pairs of metadata, it posed an immediate issue when connecting to our own Events API, as we had built integrations with DropBox’s V1 events API. In order to provide users with all of the new features and data being returned from DropBox’s V2 events API, we would have to make immediate changes to the way our integrations were set up. The solution Li-Hsing Chen, one of Kloudless’ developers at our Taipei office, was tasked with the work of integrating with the new DropBox V2 events API. To further complicate the job at hand, Kloudless had recently released V2 of our own Events API and Li-Hsing was responsible for making sure that both Dropbox V1&V2 events would work with Kloudless V1&V2 events. Part of the magic behind a unified API is the unseen data manipulation that goes on behind the scenes. End users who are accessing an API’s endpoint generally have no idea what version of an API they are using, but on the unified API’s side, there is a considerable amount of work that needs to go into making sure that all of the data returned is formatted into a unified data model. Kloudless Unified APIs work by creating a data model schema that is compiled to be constructed identically across all of the cloud storage service providers we connect with. Therefore, if a return in response data changes its structure, we have no choice but to immediately address the change, or else that particular connector will fail to be parsed correctly. When changes like these version updates occur, our developers often need to dig deep into the documentation. Because these changes are not always documented well at the time of release, our developers like Li-Hsing sometimes have to figure out much of the functionality themselves. They do this by rigorously testing and seeing what data is returned by performing a wide range of different calls to the API in the aim of seeing each and every possible response. Since the DropBox V2 events API was brand new when he began work on the integration updates, Li-Hsing had to generate all literal events from the API in order to see how the event format would return. This was an incredibly time-consuming process, but he had no alternative means of knowing the exact format he would encounter as their documentation at the time did not depict all of the details yet. This proved to be a time-consuming process and illustrated a major problem when relying on integrations in an application: changes to an API are not always easy to resolve or well documented. Through his work on this feature, however, Li-Hsing was able to document his own struggles with the API’s new features and provide Kloudless with a set of guidelines going forward with the popular cloud storage provider. In Closing Li-Hsing relented that while his process worked wonderfully when it finally came time to release the new integration features, his time spent was more than he would have liked. Because of so many different factors needing to be tweaked when they are mapped to a unified data model, his logic was quite lengthy and complex. Still, he credited this feature with giving him an in-depth and first-hand look at the inner workings of the DropBox events API and admitted that after the work was said and done, he was in a much better place going forward because he now knew exactly what to expect from every possible event format that the DropBox events API would return. Often, updating functionality in API integrations is not exactly what you would call a “glamorous process.” There is a profound amount of things that can break when an API changes anything about the format of their returns, and dealing with the ensuing chaos of broken functionality can be enough to make some developers throw in the towel. As a result, some developers and product managers choose to go with a unified API, such as Kloudless, for their integrations to save themselves the time and stress that goes along with updating functionality after an API update or versioning change. At the end of the day, however, Li-Hsing rose to the occasion and built an integral part of our DropBox functionality in the Kloudless Unified Storage API. We hope that when deciding to incorporate DropBox events API functionality into your application, you’ll go in prepared for what to expect thanks to Li-Hsing’s experience. We would love to hear from you about any pitfalls you’ve had while integrating with any of the APIs covered in this series. Please write to us at firstname.lastname@example.org and let us know any and all of the problems you may have encountered while wading through the swamp of API integrations. Please stay tuned for our next installment in the series, The Pitfalls of API Integration Services: Google Drive Edition.