Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Sqoop >> mail # dev >> Review Request: Fix for SQOOP-937


+
Venkat Ranganathan 2013-03-13, 19:18
+
Jarek Cecho 2013-03-15, 04:38
+
Venkat Ranganathan 2013-03-15, 04:46
+
Venkat Ranganathan 2013-03-19, 21:25
+
Jarek Cecho 2013-03-20, 21:30
+
Venkat Ranganathan 2013-03-19, 16:45
+
Venkat Ranganathan 2013-03-19, 18:06
+
Venkat Ranganathan 2013-03-21, 00:08
+
Jarek Cecho 2013-03-22, 03:22
+
Jarek Cecho 2013-03-22, 03:24
+
Jarek Cecho 2013-03-20, 21:16
+
Venkat Ranganathan 2013-03-21, 03:41
+
Jarek Cecho 2013-03-21, 02:17
+
Venkat Ranganathan 2013-03-21, 01:38
+
Jarek Cecho 2013-03-21, 00:41
+
Venkat Ranganathan 2013-03-20, 23:48
+
Venkat Ranganathan 2013-03-20, 23:43
Copy link to this message
-
Re: Review Request: Fix for SQOOP-932


> On March 20, 2013, 9:16 p.m., Jarek Cecho wrote:
> > src/java/org/apache/sqoop/manager/DirectNetezzaManager.java, lines 78-83
> > <https://reviews.apache.org/r/10018/diff/2/?file=271931#file271931line78>
> >
> >     My Netezza knowledge is a bit rusty these day, but I do have feeling that the external table parameter "NULLVALUE" is used only for string based columns (varchar, ...). For all other column types (int, float, ...) empty string is used to encode NULL value. On precondition that this is still the case, shouldn't the condition be more if(nullNonStrValue != null) { error; }?
>
> Venkat Ranganathan wrote:
>     Thanks Jarcec
>    
>     The Netezza external table feature supports NULLVALUE for nonstring columns also.  The unit tests tests it with an Integer column if you see.
>    
>     Thanks
>     Venkat
>
> Jarek Cecho wrote:
>     Hi Venkat,
>     thank you for your feedback. I was actually looking at that particular test before, but I came to a conclusion that it's passing simply because "\N" is invalid value for integer based column. To verify my theory, I've added another test to DirectNetezzaExportManualTest that is using string "1" for null escape character (e.g. value that is valid for integer based column):
>    
>       @Test
>       public void testNullStringExport2() throws Exception {
>    
>         String [] extraArgs = {
>             "--input-null-string", "1",
>             "--input-null-non-string", "1",
>             "--input-escaped-by", "\\",
>         };
>         ColumnGenerator[] extraCols = new ColumnGenerator[] {
>            new ColumnGenerator() {
>              @Override
>              public String getExportText(int rowNum) {
>                return "1";
>              }
>    
>              @Override
>              public String getVerifyText(int rowNum) {
>                return null;
>              }
>    
>              @Override
>              public String getType() {
>                return "INTEGER";
>              }
>            },
>         };
>    
>         String[] argv = getArgv(true, 10, 10, extraArgs);
>         runNetezzaTest(getTableName(), argv, extraCols);
>       }
>    
>     And this particular test is failing for me:
>    
>     Testcase: testNullStringExport2 took 2.528 sec
>     FAILED
>     Got unexpected column value expected:<null> but was:<1>
>     junit.framework.ComparisonFailure: Got unexpected column value expected:<null> but was:<1>
>     at com.cloudera.sqoop.TestExport.assertColValForRowId(TestExport.java:380)
>     at com.cloudera.sqoop.TestExport.assertColMinAndMax(TestExport.java:398)
>     at com.cloudera.sqoop.manager.DirectNetezzaExportManualTest.runNetezzaTest(DirectNetezzaExportManualTest.java:131)
>     at com.cloudera.sqoop.manager.DirectNetezzaExportManualTest.testNullStringExport2(DirectNetezzaExportManualTest.java:205)
>    
>    
>     I've also tried similar test for direct import by adding following test to NetezzaImportManualTest:
>    
>       @Test
>       public void testDirectNullStringValue() throws Exception {
>    
>    
>          String [] extraArgs = {
>              "--null-string", "\\\\N",
>              "--null-non-string", "\\\\N",
>           };
>    
>          String[] expectedResultsWithNulls >            getExpectedResultsWithNulls();
>          String tableNameWithNull = getTableName() + "_W_N";
>    
>          runNetezzaTest(true, tableNameWithNull, expectedResultsWithNulls,
>             extraArgs);
>       }
>    
>     Generated output seems to be suggesting that the substitution character is not being used for the integer column:
>    
>     22218 [main] INFO com.cloudera.sqoop.manager.NetezzaImportManualTest  - Line read from file = 1,Aaron,2009-05-14,1000000,T,engineering,,1
>     22218 [main] INFO com.cloudera.sqoop.manager.NetezzaImportManualTest  - Line read from file = 3,Fred,2009-01-23,15,F,marketing,,3
>     22218 [main] INFO com.cloudera.sqoop.manager.NetezzaImportManualTest  - Line read from file = 2,Bob,2009-04-20,400,T,sales,,2

Thank you for creating the documentation JIRA Venkat! I've tried yet another test case:
  @Test
  public void testNullStringExport3() throws Exception {

    String [] extraArgs = {
      "--input-null-string", "N",
      "--input-null-non-string", "N",
      "--input-escaped-by", "\\",

    };
    ColumnGenerator[] extraCols = new ColumnGenerator[] {
      new ColumnGenerator() {
        @Override
        public String getExportText(int rowNum) {
          return "B";
        }

        @Override
        public String getVerifyText(int rowNum) {
          return null;
        }

        @Override
        public String getType() {
          return "INTEGER";
        }
      },
    };

    String[] argv = getArgv(true, 10, 10, extraArgs);
    runNetezzaTest(getTableName(), argv, extraCols);
  }

Interestingly this test case actually fails. I found the behavior of the external table feature quite buggy on Netezza side. I think that your change to documentation explains it correctly, so I'll go ahead and commit it.
- Jarek
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10018/#review18174
On March 21, 2013, 12:08 a.m., Venkat Ranganathan wrote:
+
Venkat Ranganathan 2013-03-21, 18:14
+
Venkat Ranganathan 2013-03-22, 03:29
+
Venkat Ranganathan 2013-03-21, 04:34
+
Venkat Ranganathan 2013-03-21, 04:34
+
Venkat Ranganathan 2013-03-22, 03:40
+
Jarek Cecho 2013-03-22, 03:52
+
Venkat Ranganathan 2013-04-15, 14:46
+
Venkat Ranganathan 2013-04-15, 14:57
+
Jarek Cecho 2013-04-15, 17:09
+
Venkat Ranganathan 2013-04-16, 14:47
+
Jarek Cecho 2013-04-16, 16:57
+
Raghav Gautam 2013-05-29, 17:19