Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # user - SolrCell help!


Copy link to this message
-
Re: SolrCell help!
Flavio Pompermaier 2013-07-23, 07:51
I tried to download the current trunk but it doesn't compile..for example
it hangs on
https://repository.cloudera.com/artifactory/cloudera-repos/com/twitter/parquet-avro/1.0.0-SNAPSHOT/maven-metadata.xml
that doesn't exists anymore..
On Mon, Jul 22, 2013 at 11:14 PM, Flavio Pompermaier
<[EMAIL PROTECTED]>wrote:

> You couldn't be more precise ;)
>
> Thanks,
> Flavio
>
> On Mon, Jul 22, 2013 at 11:02 PM, Wolfgang Hoschek <[EMAIL PROTECTED]>wrote:
>
>> Docs for the xquery and xslt morphline commands are here (look for
>> xquery"):
>> https://github.com/cloudera/cdk/blob/master/cdk-morphlines/src/site/confluence/morphlinesReferenceGuide.confluence
>>
>> Example morphlines for the new xquery and xslt commands are here:
>> https://github.com/cloudera/cdk/tree/master/cdk-morphlines/cdk-morphlines-saxon/src/test/resources/test-morphlines
>>
>> Sample input data is here:
>> https://github.com/cloudera/cdk/tree/master/cdk-morphlines/cdk-morphlines-saxon/src/test/resources/test-documents
>>
>> Unit tests are here:
>> https://github.com/cloudera/cdk/blob/master/cdk-morphlines/cdk-morphlines-saxon/src/test/java/com/cloudera/cdk/morphline/saxon/SaxonMorphlineTest.java
>>
>> Wolfgang.
>>
>> On Jul 22, 2013, at 1:41 PM, Flavio Pompermaier wrote:
>>
>> > Ok, I'll try to follow the code! Just one last thing: for morphine-neon
>> I manage to find the test (in cdk repository) but for the new xslt and
>> xquery I'm not able to find the tests code..could you give me an hook?
>> >
>> > On Mon, Jul 22, 2013 at 9:21 PM, Wolfgang Hoschek <
>> [EMAIL PROTECTED]> wrote:
>> > There are many tests for this in the morphlines repo.
>> >
>> > Wolfgang.
>> >
>> > On Jul 22, 2013, at 11:43 AM, Flavio Pompermaiert wrote:
>> >
>> > >
>> > > Thank you for the great support Wolfgang!
>> > > Flume + Morphlines is undoubtedly an exciting road but its taking me
>> too much time :(
>> > > Do you think you could add some more tests including readJson and the
>> new xquery and xslt in trunk?
>> > >
>> > > Best,
>> > > Flavio
>> > > On Mon, Jul 22, 2013 at 8:12 PM, Wolfgang Hoschek <
>> [EMAIL PROTECTED]> wrote:
>> > > Looks like the DcXMLParser spits out a metadata field called "title"
>> and another title as part of the Tika XML stream. That metadata field is
>> then added to the solr document by solrcell. If you add "title" to the
>> captures the title from the XML stream gets added as well by solrcell.
>> > >
>> > > JSON support has been released in morphlines-0.4.1 (which flume trunk
>> is now depending on):
>> http://cloudera.github.io/cdk/docs/0.4.1/cdk-morphlines/morphlinesReferenceGuide.html#readJson
>> > >
>> > > Note that Tika XML doesn't really support/capture XPath extraction
>> with SolrCell. We have added proper support for reading, extracting and
>> transforming XML and HTML with XPath, XQuery and XSLT on the current
>> morphlines trunk (not yet released), similar to the way we already support
>> JSON and Avro. This should make XML handling a lot more straightforward,
>> and make the very limited XML SolrCell approach obsolete. Look for the new
>> "xquery" and "xslt" command in
>> https://github.com/cloudera/cdk/blob/master/cdk-morphlines/src/site/confluence/morphlinesReferenceGuide.confluence
>> > >
>> > > Meanwhile, consider using these new commands or, use JSON or Avro, or
>> write your own custom morphline commands that extract whatever you want
>> from your XML data.
>> > >
>> > > Wolfgang.
>> > >
>> > > On Jul 22, 2013, at 9:18 AM, Flavio Pompermaier wrote:
>> > >
>> > > > Hi to all,
>> > > > I'm trying to understand how to "master" Morphline configuration
>> files in order to put some data into Solr but I'm facing some problem with
>> TestMorphlineSolrSink. This is what I done:
>> > > >
>> > > > 1) Since I want to index the title of the testXML.xml (i.e. "Tika
>> test document") so I commented out all the parsers except
>> org.apache.tika.parser.xml.DcXMLParser (which parse Doublin Core metadata)
>> > > > 2) In schema.xml I added the following field: