Kloudless Blog

Tutorials, case studies and how-tos from our experts

Use pre-signed URLs to improve the performance of uploads and downloads

In our last post, we described the contrast between user-facing cloud storage services and object storage services used by applications. A key differentiating factor is how object storage services allow applications to upload and download data in a variety of ways. This enables apps to maximize both their computational as well as economic efficiency.

One way to maximize performance and cost savings is to avoid having client-side downloads and uploads proxy through your web server. Clients that access the source of application data directly, usually an object store of some kind, receive better throughput, can leverage CDNs, and avoid burdening your web server with unnecessary processing cycles.

Apps can utilize this strategy without inadvertently granting access to all their backend data by generating URLs for clients to use for a specific purpose. These “pre-signed URLs” include a cryptographically signed payload—by the application’s secret key—that contains the details of the source or destination of content. In this manner, backend servers can provide client-side applications with a specific URL to use rather than proxy the request. These client-side apps include browsers and browser-based applications.

Using pre-signed URLs

Amazon S3 is an example of a service that provides pre-signed URLs. Check out their docs for more information on generating one. Alternatively, the Kloudless Unified Storage API also supports obtaining pre-signed URLs where available, such as for S3.

The Storage API also allows links to any file to be generated for download purposes, regardless of cloud service. An indirect Kloudless Link redirects to either a signed URL or the actual file URL during downloads. This Link abstraction can be created as shown below (docs):

Similarly, Upload URLs generate a URL signed by the upstream storage service, such as S3, to return to the client-side for uploads. Create one as shown below:

When are pre-signed URLs available?

Most object storage services such as Amazon S3 offer pre-signed URLs. Check out our Storage API docs for a comprehensive overview of the API requests described above.

File storage services that focus on content collaboration such as Dropbox and Box usually don’t intend to be used as an app’s backend data store. This implies they wouldn’t expect a multitude of users uploading arbitrary content to a single storage account. Therefore, third-party apps accessing these services’ APIs rarely find the need for pre-signed URLs. These collaboration platforms prefer links that guide the user to the platform itself, such as a link to a file in Box, or a link to a folder in Dropbox to upload data to.

Check out our previous blog post for more information on the differences between user-facing and app-facing cloud storage services.

The Kloudless roadmap for pre-signed URLs

Kloudless currently proxies upload and download requests via nginx by default. Expanding this setup to multiple AWS regions has enabled us to stream content more efficiently for simple data transfer requests such as file downloads. We plan to expose this capability further to allow pre-signed URLs for any service, rather than only ones that support it natively.

Let us know which storage service you’d like to see pre-signed URLs for in the comments below or at hello@kloudless.com!

Published By

Vinod Chandru