Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # user - How to use HTTPS with HTTPSource?


Copy link to this message
-
Re: How to use HTTPS with HTTPSource?
Ashish 2013-06-26, 09:28
The example I shared works for Jetty 9, Flume uses 6.1.26.

Here is a basic version. Add these lines in HTTPSource in start() method.
This is a basic version. You would like to read these properties from flume
config in configure(). This is small stuff.

SslSocketConnector sslSocketConnector = new SslSocketConnector();
sslSocketConnector.setKeystore("jettykeystore");
sslSocketConnector.setKeyPassword("password");
sslSocketConnector.setPort(8443);
srv.setConnectors(new Connector[] { connector, sslSocketConnector });

Here is how it would like

  @Override
  public void start() {
    Preconditions.checkState(srv == null,
            "Running HTTP Server found in source: " + getName()
            + " before I started one."
            + "Will not attempt to start.");
    srv = new Server();
    SocketConnector connector = new SocketConnector();
    connector.setPort(port);
    connector.setHost(host);

        SslSocketConnector sslSocketConnector = new SslSocketConnector();
        sslSocketConnector.setKeystore("jettykeystore");
        sslSocketConnector.setKeyPassword("password");
        sslSocketConnector.setPort(8443);

    srv.setConnectors(new Connector[] { connector, sslSocketConnector });
    try {
      org.mortbay.jetty.servlet.Context root               new org.mortbay.jetty.servlet.Context(
              srv, "/", org.mortbay.jetty.servlet.Context.SESSIONS);
      root.addServlet(new ServletHolder(new FlumeHTTPServlet()), "/");
      srv.start();
........

To submit a patch, would need to refine the code a bit and add test cases.
Shall take a while. HTH !
On Mon, Jun 24, 2013 at 8:24 PM, shushuai zhu <[EMAIL PROTECTED]> wrote:

> Ashish, thanks again. Could you elaborate a little more what I should do?
> I am relatively new to Flume (just started using it a couple of weeks ago)
> and also new to open source code base.
>
> Shushuai
>    *From:* Ashish <[EMAIL PROTECTED]>
> *To:* [EMAIL PROTECTED]; shushuai zhu <[EMAIL PROTECTED]>
> *Sent:* Friday, June 21, 2013 10:38 PM
>
> *Subject:* Re: How to use HTTPS with HTTPSource?
>
> HTTP source has not changed much :) One way to patch in additional SSL
> stuff in jetty code. The second more tedious way is to use NIO framework
> with SSL code like Netty or MINA. We are working on MINA M1 release and
> perhaps after that shall take a tab at writing an HTTP source based on that.
>
> HTH !
>
>
> On Sat, Jun 22, 2013 at 3:07 AM, shushuai zhu <[EMAIL PROTECTED]> wrote:
>
> Ashish,
>
> Thanks for the pointer. So I will create my own source, say HTTPSSource,
> which extends the HTPSource to add the https connection, then use the
> custom HTTPSSource in flume.conf.
>
> Is this the right source code you referred (I am using flume 1.3.1)?
>
>
> https://git-wip-us.apache.org/repos/asf?p=flume.git;a=tree;f=flume-ng-core/src/main/java/org/apache/flume/source/http;h=fa5477033ca23c075fec6af2c391b01d1d49f69a;hb=trunk
>
>
> Shushuai
>
>    *From:* Ashish <[EMAIL PROTECTED]>
> *To:* [EMAIL PROTECTED]; shushuai zhu <[EMAIL PROTECTED]>
> *Sent:* Friday, June 21, 2013 5:09 AM
> *Subject:* Re: How to use HTTPS with HTTPSource?
>
> AFAIK, HTTPSource doesn't support HTTPS. HTTP Source uses embedded jetty,
> so it would extending HTTP source to add SSL configuration for this.
>
> This link should help
> http://git.eclipse.org/c/jetty/org.eclipse.jetty.project.git/tree/examples/embedded/src/main/java/org/eclipse/jetty/embedded/ManyConnectors.java
>
> It should be just few lines of code change.
>
> Please note I am referring to source code in master branch.
>
>
> On Thu, Jun 20, 2013 at 1:40 AM, shushuai zhu <[EMAIL PROTECTED]> wrote:
>
> Hi,
>
> I am using HTTP to transfer data from remote machine to HTTPSource of
> Flume agent (the flume conf. file looks like below). Is HTTPS supported?
> Anyone has examples to use HTTPS to transfer the data?
>
> agent1.sources = r1
> agent1.sources.r1.type = org.apache.flume.source.http.HTTPSource
> agent1.sources.r1.port = 41415
> agent1.sources.r1.channels = ch1
thanks
ashish

Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal