We have been working hard and are thrilled to give you a sneak peek of the Zenko 1.1 release. Here is the list of new treats for you:
Lifecycle management
Control over your data.
The new release is bringing you more tools to control your data workflow in the multi-cloud environment.
What is lifecycle policy anyway? Basically, you can create rules that will specify actions to be taken on the objects in Zenko after a certain period of time. In the previous Zenko release you could set an expiration date on objects, meaning you could simply tell Zenko to delete certain objects after a specified period of time.
We are adding another powerful policy to the lifecycle management – transition. Transition policies allow moving object data from one location to another automatically instead of deleting it. Usually, you would like to move old or infrequently-accessed data to a slower but cheaper location. On versioned buckets, you can apply this policy to current version data, as well as noncurrent versions.
Support for Ceph-based storage
Liberating freedom of choice
Zenko is striving to be up-to-date with all current cloud storage offerings on the market. Our goal is to provide as much flexibility as possible and avoid vendor lock-in.
Ceph is an open source software put together to facilitate highly scalable object, block and file-based storage under one whole system.
Zenko 1.1 adds Ceph to the list of possible storage locations. Just go to the Orbit -> Storage Locations -> Add, in a dropdown menu you will find Ceph.
Out-of-band (OOB) updates from Scality RING to Zenko
If you are already utilizing our on-prem object storage software, RING, we are excited to let you know that now you can go multi-cloud with all your stored data. Previously you could use Zenko capabilities to manage data across different clouds, but only for the inbound data stream. What about all the data you already had stored in the RING?
That’s where out-of-band updates are coming in to save the day.
Cosmos
Our team created an extensible framework (a.k.a Cosmos) that will allow Zenko to manage data stored on various kinds of backends such as filesystems, block storage devices, and any other storage platform. Pre-existing data on these storage systems and data not created through Zenko will be chronologically ingested/synchronized.
Enter SOFS (Scale-Out-File-System)
To accommodate a variety of architectures and use cases, RING allows native file system access to RING storage through the integrated SOFS with NFS, SMB and FUSE Connectors for access over these well-known file protocols. SOFS is more precisely a virtual file system on top of the RING’s storage services. It is a commercial product from Scality and you can learn more about it in this whitepaper.
These updates essentially enable Zenko to discover and import metadata from files stored on existing Scality RING file system(s), as well as to receive ongoing asynchronous (out-of-band) updates for changes to the target file systems such as new file creates, deletes and metadata updates. Once the metadata is imported into Zenko, key Zenko functionality can be used.
To name a few key features:
Cross-region replication
Lifecycle management
Metadata search
OOB from RING S3 Connector(S3C) to Zenko
The Scality S3 Connector provides a modern S3-compatible application interface to the Scality RING. The AWS S3 API has become the industry’s default cloud storage API and has furthermore emerged as the standard RESTful dialect for object storage.
Zenko 1.1 has a new service (ingestion-populator and ingestion-processor) in the Backbeat component to discover and import metadata from Scality S3C, as well as to receive ongoing asynchronous (out-of-band) updates for changes to the target bucket such as new objects created, deleted and metadata updates. Once the metadata is imported into Zenko, key Zenko functionality can be used on the associated RING object data.
Let us know
We would love to hear your thoughts on this updates to Zenko or If you want to contribute to the Zenko roadmap, check our GitHub repository and leave your comments on the forum.
Hybrid cloud headlines have dominated tech news in the last year, with Microsoft investing in Azure Stack, AWS building OutPosts and not to forget, Google who led the pack with the announcement of their Kubernetes on-premises offering last July. These are advancements in computing, but what about storage? Any truly scalable cloud architecture is build around an object storage interface that serves as a building block for many services, to name a few:
repository of system images
target for snapshots and backups of virtual machine images
long term repository of logs
main storage location for user generated content like images, videos or document uploads depending on the application
What about applications which need to store petabytes of data?
In a cloud native world, Kubernetes is the fundamental building block that makes running and managing applications in hybrid environments easier. The cloud native movement promised and has delivered improvements on many important axis:
faster application development
better quality and agility thanks to automation
flexibility in deployment options with hardware and infrastructure abstraction
less human errors in operations via automation and common practises
For example, today it’s possible to develop an application on a laptop with a self contained Kubernetes environment that runs all its dependencies, build a helm package and deploy the exact same code on a public cloud environment running in Google, AWS or Azure or on premise with a local K8s distribution like Scality MetalK8 or Redhat Openshift.
For stateless applications, like a web proxy or serving static content that do not require storage capacity beyond what can be offered inside a given hyper converged compute cluster, this is a reality. On the over end, if your application requires more capacity, like storing user documents, photos, or videos, the dreams of deploy anywhere and automated operations fails short.
It’s not practical to think that an application can be developed to support Azure Blob Store and its object snapshots and Append Blobs features while keeping this application portable to AWS. With such a set-up, your service could become global with a need to run in a region where Microsoft is not present. Do you really want to rewrite significant piece of your code for each storage service?
There should be a way to stay independent of storage clouds interfaces and vendors, the same way that it’s possible to abstract compute clouds thanks to solution like Kubernetes.
Why is object storage a good fit for cloud native architectures?
For one they share a lot of similarities:
Stateless: Great fit the Kubernetes/Pod model, greater flexibility
Abstraction of location: containers and data life cycles are very different, it’s a good way to separate operation concerns, right-sized capacity
Predictable: no locking and no complex state management
Optimised for heterogeneous networks: Multi-cloud and geo distributed by nature
Rich Metadata: metadata search is key, especially over multiple clusters and multiple geographies
What about if you could access data easily and transparently wherever it’s stored? No matter if it’s on prem, in a different data center or in any public cloud storage service.
To my knowledge, Zenko is the only solution today that can unify the leading 6 public cloud storage providers with a single API and support lifecycle policies between these heterogeneous environments. But don’t trust me, you can try Zenko in a few minutes by creating a sandbox environment in Orbit, Zenko’s hosted management portal!
And of course feel free to disagree and comment in our forums over here. We’re eager to get object storage as part of the reference architectures.
Congratulations to all of us for winning Product of the Year Award at the National Association of Broadcasters Show 2019 in Las Vegas. The Award recognizes the most significant and promising new products and technologies being exhibited at the NAB Show. It is an independently judged awards program recognizing product innovation, offered for the first time in 2019.
There were 16 categories in total and Zenko won the Storage category with six other products. With 1,700 vendors and exhibitors at NAB Show, we are among an exclusive group. An open source project at the roots of an enterprise product recognized among well established brands is quite an achievement. We’re in good company with vendors focused only on the media & entertainment space. Zenko being such a general-purpose multi-cloud data management and governance tool, makes the award even more precious.
This is a great, great acknowledgement of everyone’s efforts over the past couple of years. From Engineering to Tech Services to Marketing to Sales to Finance to Zenko’s customers and early adopters – so many people have contributed to the Zenko product and business. Thank you!
Since the first 1.0 release in late 2017, containerd has earned a large list of new adopters, including two major public clouds and recently graduated as a CNCF project. It’s a major milestone for the project that provides a “boring core container runtime,” demonstrating that it has reached a high level of maturity.
Phil Estes, IBM Distinguished Engineer for IBM Cloud division, gave an update on the status of containerd and shared a glimpse to the roadmap. In 2018, containerd and its CRI plugin was marked as general availability for use in Kubernetes, followed by two major public clouds–Google’s GKE and IBM Cloud’s IKS–offering containerd as their managed Kubernetes cluster’s runtime. Containerd continues to be used by all Docker releases, and Docker has plans to remove more code from the Moby project and utilize stable features within containerd instead, furthering adoption of containerd’s codebase.
Additionally, the attractive API and stability of containerd has brought quite a few A-list adopters since last year including AWS Firecracker VMM, Kata Containers, Alibaba PouchContainer; Microsoft is also contributing and hardening the complete Windows container support in the containerd 1.2 release.
The Zenko community has had a fantastic 2018! We’re celebrating by giving out $100 Amazon gift certificates.
We’ll be selecting winners from new users who provide feedback on Zenko Orbit. Since the first release of Orbit a year ago, a lot of new features have been added. Users can now replicate and control data seamlessly in Amazon S3, Google Cloud Storage, Azure Blob Storage, Wasabi, Digital Ocean Spaces and private cloud storage compatible with S3 protocol, like Scality RING.
What are the advantages? Some Zenko users see spreading data across multiple storage systems as a great way to reduce lock-in or gain negotiating power with the cloud providers. Others love how easy it is to put movie files in multiple locations: objects in Azure use Video Indexer machine-learning capabilities to automatically add metadata, which feed into Zenko searchable database and then replicate to Amazon buckets for worldwide distribution.
Lots more capabilities are coming to Zenko in 2019!
Your feedback will help shape its future: Hurry up and try Zenko Orbit (redirects to Google Login). All you have to do is fill out a short survey based on your first experience with it for the chance to win $100.
Just like you want to keep control of your kitchen and of your ingredients while cooking, we believe that everybody needs to keep control of their data. Zenko was designed from the ground up to be the chef’s choice of those who create value with data. Like picking the best ingredients from the shelf, Zenko’s multi-cloud support for S3, Azure and Google APIs makes it easy to prepare cloud applications with the best storage for the task.
We’re happy to share what’s in Zenko 1.0 and a brief view of what’s cooking for the rest of 2018.
Multi-Cloud Storage Backends
Zenko is the Esperanto of cloud storage APIs. At the moment, Zenko supports Amazon S3, Google Cloud Storage (GCS), Microsoft Azure, Scality S3Connector RING, Scality sproxyd RING, DigitalOcean Spaces and Wasabi Storage. Other S3-compatible backends are easy to add by extending the open source CloudServer component, with Blackblaze support already under development.
One-to-Many Replication
One-to-many cross-region replication (CRR) is a bucket-level configuration that enables automatic, asynchronous copying of objects across buckets in different clouds, private or public. This style of replication works particularly well for securing data in different locations, for example, copying video files to the closest distribution networks or keeping multiple copies for backups . Of course our favorite CRR use case is using replication to protect against rising cloud egress costs. Like others have painfully discovered, once multiple terabytes of data is stored in one cloud, moving away will incur in a very high bill. By starting with multiple copies of the data, most of the control goes back to the user: switching from AWS to Azure won’t be as expensive or time consuming if the same data is uploaded in both.
Zenko Orbit offers an easy UI to track the progress of all replication processes across each target cloud. Replication jobs can also be paused when a target cloud is known to be down, and then resumed (manually or based on time) when it’s back up. Of course replication jobs that fail are automatically retried until they succeed, too.
Aside from replication-related data, like status and throughput, Orbit also provides other useful statistics like total data managed, Zenko capacity, memory usage, number of objects/ object versions managed, and amount of data managed between cloud destinations.
Local Cache (w/ Storage Limit)
Without a local cache, each objects would be stored immediately in a bucket on a public cloud. In that case, any replication will incur in egress costs because each object would have to go from the first cloud destination to the new ones, like a daisy chain. With a local cache, the objects are replicated by going straight to their destination, like a broadcast to the cloud.
Overview of Zenko’s replication engine.
Zenko’s local cache saves charges by queuing objects in a user-defined local cache before they’re uploaded to a target cloud. Users can set a local cache location on their preferred storage backend, set a storage limit for how large the cache can become. All objects are automatically cleared from the cache after the replication process completes.
Metadata Search
Objects are usually stored with metadata to describe the object itself. For a video production company for example, metadata can include details like “production ready” or the department that produced the file or the rockstar featured in a video.
With metadata search, users can actually search metadata on objects written through Zenko. Simply add tags and metadata to the object (in Orbit), write object to target cloud and then search for the object later.
Future versions will allow users to also ingest metadata for objects that are written natively to the clouds, so that Zenko users will be able to easily import existing catalogues.
Lifecycle: Expiration Policies
Reclaim storage space by defining lifecycle expiration policies (specified in Amazon’s S3 API) to any bucket managed through Zenko. The policies can be applied to both versioned and non-versioned buckets, and are triggered after a number of days (defined by user) have passed since the object’s creation.
Zenko features offer users the ability to personalize data workflows based on metadata, treating Zenko as an S3-compatible target that replicates data to remote servers in the cloud.
Future versions will add storage targets based on network filesystems like NFS and SMB. Also there are plans to radically improve the workflow management capabilities of Zenko with powerful API and a graphical UI. Get involved with Zenko development by following Zenko on GitHub and get on Zenko Orbit to test all Zenko features.
