Nerve reporting to Kubernetes API Server

[tejokumar]

First, apologies to raise this question/request here.
I am looking for Nerve implementation to report the service to Kubernetes API server. What kind of challenges and issues to keep in mind while doing such implementation. Any pointers to this topic is appreciated.
Thank You

[igor47]

hi, i saw your note in my email as well. i haven't really kept up with nerve development. i'm also not too familiar with service discovery in k8s.

off the top of my head, the biggest problem is going to be freshness. the original ZK implementation used ephemeral nodes that would expire after some time. the etcd implementation i see in the code also seems to use a ping mechanism that resets a TTL. for kubernetes API, i doubt there's an auto-expiring entry type.

it would be helpful to know what's consuming the entries you're putting into k8s. are you trying to discover out-of-cluster services the same as in-cluster services, perhaps by using coredns? without an auto-expiry, you'll end up with a bunch of stale entries in the dns name. if you want to use the k8s api directly in your application, you can probably set metadata about last ping time, and exclude entries that are too old -- but i don't recommend this pattern. one of the goals of smartstack was to remove service discovery concepts from inside applications.

a more smart-stacky approach would be to run a sidecar container as part of your pod, containing just a smartstack-aware proxy like haproxy w/ synapse or envoy. if you want to talk to your services outside of k8s from your services inside k8s, talk to them through the sidecar.

if your goal is to talk to your services inside of k8s from your sevices outside of k8s, then you'll probably want to direct your attention to synapse instead of nerve.

[tejokumar]

Thanks for response @igor47 . We are using Synapse on both K8S and outside K8S for service discovery, and Envoy as proxy service. On non-K8S side, we are using nerve to register services with Zookeeper. K8S registers services with K8S API server. Synapse running on both non-K8S and K8S discover services. Envoy is used to communicate between services.
In this setup, Synapse discovers services from both Zookeeper and K8S api server. We want Synapse to discover services from single location. Hence looking at options for Nerve to register services directly with K8S api server.
As you mentioned, the problem is with freshness. We are looking at couple of options like

1. A proxy layer between nerve and K8S, which will take care of freshness
2. Modify nerve to update K8S api server directly, use Envoy health checks and TTL stale entries
   Both of above approaches have its own complexities. I am looking at any other ways of nerve registering services to K8S Api server