It’s easy to make mistakes when developing multi-cloud applications, even when only dealing with object storage API. Amazon S3 and Azure Blob Storage are similar models but with differing semantics and APIs, just like Google Cloud Storage API. Amazon S3 is a RESTful API providing command syntax for create (PUT), access (GET), and delete (DELETE) operations on both buckets and objects, plus access to bucket metadata (HEAD).
Applications that need to support both API have to be developed very carefully to manage all corner cases and different implementations of the clouds. Luckily, Zenko’s team is dedicated to finding those corner cases and solve them once for everybody. Zenko CloudServer translates standard Amazon S3 calls to Azure Blob Storage, abstracting complexity. The design philosophy of CloudServer’s translations are:
- S3 API calls follow the Amazon S3 API specification for mandatory and optional headers, and for response and error codes.
- The Azure Blob Storage container is created when the application calls S3 PUT bucket, and the container is assigned the name given in the PUT bucket request.
- Bucket names must follow AWS naming conventions and limitations.
Try Zenko S3-to-Azure in a free sandbox
Non-exhaustive API Comparison: AWS versus Azure
Put Bucket / Create Container
- Bucket naming restrictions are similar but not the same.
- CloudServer returns an InvalidBucketName error for a bucket name with “.” even though allowed on AWS S3.
- Canned ACLs can be sent as part of the header in an AWS S3 bucket put call.
- CloudServer uses Azure metadata x-ms-meta-scality_md_x-amz-acl header to store canned ACLs in Azure containers.
Get Bucket / List Blobs
- The AWS S3 “Marker” parameter expects a object key value, but Azure does not have an implementation to retrieve object listings after a certain key name alphabetically (can only retrieve blobs after an opaque continuation token).
- AWS S3 sends back the object owner in each listing entry XML but Azure does not include object owner information in listings.
Delete Bucket / Delete Container
- While AWS S3 returns an error if a bucket is non-empty, Azure deletes containers regardless of contents. Zenko CloudServer makes a call to lists blobs in the container first and returns the AWS S3 BucketNotEmpty error if not empty.
Put Object / Put Blob
- CloudServer only allows canned ACLs, except aws-exec-read and log-delivery-write. ACLs are stored as blob metadata. From the Azure side, there are no object ACLs so behavior is based on container settings.
- Only the STANDARD setting is allowed as “storage class”
- Setting object-level encryption is not allowed through headers. The user must set encryption through Azure on an account basis.
Delete Object / Delete Blob
- AWS S3 has delete versions and offers an MFA requirement for delete. MFA header is not supported in CloudServer.
Get Service / ListContainers
- AWS S3 returns a creation date in its listing, while Azure only stores the last-modified date.
Initiate Multi-part Upload (MPU) / no correspondent on Azure
- A MPU is treated as a regular Put Blob call in Azure. CloudServer cannot allow users to initiate more than one MPU at a time because there is no way of renaming or copying a committed block blob to the correct name efficiently, and any uncommitted blocks on a blob are deleted when the block blob is committed (preventing an upload to the same key name). To allow for initiate MPU, Zenko CloudServer creates a “hidden” blob with a unique prefix that is used for saving the metadata/ACL/storage class/encryption of the future objectListing of ongoing MPUs.
Put Part / Put Block
- Azure has a size limit of 100 MB per block blob. AWS S3 has a max part size of 5 GB.
- Azure also has a 50,000-block maximum. At 100 MB max per block, this comes out to around 5 TB, which is the maximum size for an AWS S3 MPU. Putting the same part number to an MPU multiple times may also risk running out of blocks before 5 TB size limit is reached.
The easiest way to write multi-cloud applications is to use the open source projects Zenko and Zenko CloudServer.
Photo by Simon Buchou on Unsplash
