WIP: Implement logic to send service logs to syslog

[keymon]

Motivation

Several of the CF and Diego platform services send logs to the local syslog, tagged as vcap.*. . The metron agent from loggregrator configures the local syslog to send the logs to metron so that later can be sent to a centralised logging service.

In some cases it is the service application itself who sends the logs to syslog, but it seems to be a common pattern across CF and Diego services to let the service log to stdout/stderr, and redirect that output to local files or syslog using the logger command.

Not all the services implement this feature, and instead they only log to local files. In order to centralise all the logs, so we will add the logic required to send the logs of the missing services.

We will implement the log to stdout and redirect to logger pattern whenever is possible.

Implementation in logsearch bosh release

We want to implement the same idea in all the graphite and statsd services.

For that we we add several properties: <service>.log_to_syslog. When true, we redirect the stdout and stderr in the ctl script into syslog following the standard vcap pattern, in addition to files.

In the case of carbon, in order to get the logs sent to stdout, we add the option --debug.

In the case of graphite-web, we shell out the Log directives to stream the logs to logger.

How to review?

Enable the *.log_to_syslog and deploy. Logs should be sent to syslog.

If metron agent is installed locally, the logs will be redirected to metron agent the loggregrator.

Open questions

This PR is open to debate for several open questions:

• Is OK to add a specific property to enable this feature for each service? Shall we use one unique property? or should this be enabled by default?
• Is the tag vcap.* the most appropriate one? This is not a CF official release. But the generic tag vcap.* might be handy to refer to a service deployed with bosh.
• tee and logger run as root for statsd and carbon Is that fine?