Interesting, I had previously found that many (but not all) ES params are extensively processed by code fairly deep in the guts of ES itself (unrelated to Metron’s use of it), to allow this kind of flexibility.
Thanks very much for sharing your results, Mike.  Looks like we’ll continue with the release.

Laurens, I hope you can find what’s going on, on your system.  I would suggest that, if it isn’t being written through to the yml file, it seems to be an Ambari or install issue rather than an ES issue per se.

Thanks,
--Matt

From: Michael Miklavcic <[EMAIL PROTECTED]>
Date: Wednesday, September 13, 2017 at 2:08 PM
To: Matt Foley <[EMAIL PROTECTED]>
Cc: Laurens Vets <[EMAIL PROTECTED]>, "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
Subject: Re: Elasticsearch masters_also_are_datanodes doesn't work.

Here's what I found. Data is flowing into ES indexes. I can change the "masters_also_are_datanodes" property and it is reflected in the node.data property in elasticsearch.yml.

Here's my search for this property in the source code. It is properly being used int the ES mpack scripts afaik.

Targets
    Occurrences of 'masters_also_are_datanodes' in Directory devprojects/metron/metron-deployment
Found Occurrences  (6 usages found)
    Unclassified occurrence  (5 usages found)
        metron-deployment  (2 usages found)
            metron-deployment/other-examples/manual-install  (1 usage found)
                Manual_Install_CentOS6.md  (1 usage found)
                    810 "masters_also_are_datanodes" to "true"
            metron-deployment/roles/ambari_config/vars  (1 usage found)
                single_node_vm.yml  (1 usage found)
                    104 masters_also_are_datanodes: "1"
        metron_mpack  (3 usages found)
            common-services.ELASTICSEARCH.2.3.3.configuration  (1 usage found)
                elastic-site.xml  (1 usage found)
                    30 <name>masters_also_are_datanodes</name>
            common-services.ELASTICSEARCH.2.3.3.package.scripts  (1 usage found)
                params.py  (1 usage found)
                    26 masters_also_are_datanodes = config['configurations']['elastic-site']['masters_also_are_datanodes']
            common-services.ELASTICSEARCH.2.3.3.package.templates  (1 usage found)
                elasticsearch.master.yaml.j2  (1 usage found)
                    36   data: {{ masters_also_are_datanodes }}
    Usage in string constants  (1 usage found)
        metron_mpack  (1 usage found)
            common-services.ELASTICSEARCH.2.3.3.package.scripts  (1 usage found)
                params.py  (1 usage found)
                    26 masters_also_are_datanodes = config['configurations']['elastic-site']['masters_also_are_datanodes']

It should be noted that regardless of the ES documentation, this seems to work with a variety of values including "true" (with quotes) and 1 (without quotes). When I set the value to garbage through Ambari, I see this in /var/log/elasticsearch/metron.log
java.lang.IllegalArgumentException: value cannot be parsed to boolean [ true/1/on/yes OR false/0/off/no ]
        at org.elasticsearch.common.Booleans.parseBooleanExact(Booleans.java:97)
        at org.elasticsearch.cluster.node.DiscoveryNode.dataNode(DiscoveryNode.java:256)
        at org.elasticsearch.cluster.node.DiscoveryNodes$Builder.build(DiscoveryNodes.java:659)
        at org.elasticsearch.cluster.ClusterState$Builder.nodes(ClusterState.java:571)
        at org.elasticsearch.cluster.service.InternalClusterService.doStart(InternalClusterService.java:205)
        at org.elasticsearch.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:68)
        at org.elasticsearch.node.Node.start(Node.java:279)
        at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:206)
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:272)
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)

Apparently, even yes/no will work! I tried it, and can confirm they work with quotes. But surprisingly, this does NOT work if you pass true/false or yes/no without quotes. In summary, here are the supported values:

  *   1
  *   0
  *   "true"
  *   "false"
  *   "yes"
  *   "no"
Mike
On Wed, Sep 13, 2017 at 2:17 PM, Matt Foley <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
Sorry, you’ve beat me.

Mike, what have you found in full-dev?  Does ES work correctly there?
From: Laurens Vets <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
Date: Wednesday, September 13, 2017 at 1:14 PM
To: Matt Foley <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
Cc: "[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>" <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>, Michael Miklavcic <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>

Subject: Re: Elasticsearch masters_also_are_datanodes doesn't work.
Yes, I'm saving the changes. The orange bar pops up saying I have to restart elasticsearch, which I do. I'm logged in with the admin user, it has privileges to do everything else. I'm using a bare metal install via Ambari. Everything else seems to work through Ambari except this :)

On 2017-09-13 12:04, Matt Foley wrote:
That it doesn't work in Ambari, at least to the point of writing thru to the config file (whether or not it actually works with ES), is quite mystifying.  Silly question, but, are you pressing the "Save" button after changing the field?  Are you perhaps not logged into Ambari with an ambari user id that has privs to change this config?  Did you use non-default user configuration with Ambari, and if so are you running with an admin-priv ambari account?

From: Laurens Vets <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
Date: Tuesday, September 12, 2017 at 8:32 PM
To: "[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>" <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
Cc: Michael Miklavcic <[EMAIL PROTECTED]<mailto:michael.miklavci
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB