Announcing Apache Knox Gateway 0.6.0

With Apache Hadoop YARN as its architectural center, Apache Hadoop continues to attract new engines to run within the data platform, as organizations want to efficiently store their data in a single repository and interact with it in different ways. As YARN propels Hadoop’s emergence as a business-critical data platform, the enterprise requires more stringent data security capabilities. The Apache Knox Gateway (“Knox”) provides HTTP based access to resources of the Hadoop ecosystem so that enterprises can confidently extend Hadoop access to more users, while maintaining compliance with enterprise security policies.

On May 6th, the community announced the release of Apache Knox Gateway 0.6.0. With this release, the community addressed over 40 JIRA issues. Knox 0.6.0 delivers many new features, fixes and enhancements. Among these improvements, five stand out as particularly important.

This blog provides an overview of the new features and how they integrate with other Hadoop services, as well as provide a preview of enhancements that we have planned for upcoming releases.

Support for Storm UI REST API

The Storm UI daemon provides a REST API that allows users to interact with a Storm cluster. This includes retrieving metrics data and information related to cluster configuration and operations management such as starting or stopping topologies. By providing access to this API, through a new Knox routing service for Storm, users now have the ability to securely access these resources. This feature will be available as a technical preview in HDP 2.3, to be released in later half of this year.

Optimized LDAP Authentication through Caching

In this release of Apache Knox, we reduce the load on the LDAP or Active Directory server and optimize consecutive API calls by reducing the number of authentication attempts. By adding support for cached LDAP authentication information in Knox, eliminates the need for client sessions via JSESSIONID.

Configuration Driven REST API Integration

The objective of this improvement is for contributors to easily provide routing services to Knox in order to support additional APIs. A basic integration now simply consists of providing a Service Definition file and a Rewrite file. In case of more involved integrations, users may have to implement service specific protocols for requests by providing a custom Dispatch class.

The Knox community will benefit from simpler and dynamic REST API integrations through this feature.

SSL Mutual Authentication

SSL Mutual Authentication provides a mechanism for establishing a strong trust between clients and servers based on exchanging certificates to prove their identity. The explicit trust of a client’s certificate using mutual authentication allows Knox to unambiguously identify a client that is presenting a HTTP header or other assertion that needs to be trusted. This feature is especially beneficial for deployments that leverage the pre-authenticated SSO feature.

Improved Configuration for Load Balancers

This feature provides a configuration facility to indicate the “frontend” URL that is used for URL rewriting for deployments with load balancers. This enhancement reduces the processing that load balancer needs to perform for the rewriting task.

Preview of Features to Come

The Knox release would not have been possible without excellent contributions from the dedicated, talented community members who have done a great job understanding the needs of the user community and trying to deliver on them. Based on demand from the user community, we will continue to focus our efforts in three primary areas:

Providing access to the evolving REST API of the modern data architectureIn the future releases, we plan to expose more APIs to new and interesting clients and applications. We believe that our configuration-driven approach for API integrations will facilitate accelerated support for new API integrations.
Authentication and SSO capabilities for the REST APIs and web UIs within the Hadoop ecosystem

We will continue to expand authentication, federation and SSO capabilities to meet user and developer needs. We plan to deliver support for Keystone tokens and JWT capabilities in the upcoming releases.

Reducing integration burden for authentication and load balancing solutionsIntroducing new authentication and federation providers to Knox and providing support for advanced load balancing features such as relevant aspects of the Forwarded HTTP Extension.