Not exactly what you may want - but could you try using a HTTP client
in Java ? Some of them have the ability to automatically follow
redirects, manage cookies etc.
On Thu, Dec 9, 2010 at 4:35 PM, edward choi <[EMAIL PROTECTED]> wrote:
> Excuse me for asking a general Java question here.
> I tried to find Java mailing list from Google but none of them were active.
> There is a problem that's been driving me crazy for a while.
> I am trying to download webpages from New York Times.
> With Java URL.openStream(), I can't get past the login requirement.
> But with c++ socket programming (using read() and write()), I can download
> any webpage just fine.
> Interesting thing is that with c++, I get redirected like 10 times. Below is
> the content of the header of the firstly redirected webpage when I try to
> HTTP/1.1 302 Moved Temporarily
> Server: Sun-ONE-Web-Server/6.1
> Date: Thu, 09 Dec 2010 08:42:35 GMT
> Content-type: text/html
> Set-cookie: RMID=0b5d4aea392d4d00967bfaf1; expires=Friday, 09-Dec-2011
> 08:42:35 GMT; path=/; domain=.nytimes.com
> Set-cookie: NYT_GR=4d009b2b-yJ4V047ooAmPtGcvASTmng; path=/; domain=.
> expires=Saturday, 08-Jan-2011 08:42:35 GMT; path=/; domain=.nytimes.com
> Set-cookie: NYT-Pref=hppznw|^creator|NYTD.Cookies; path=/; domain=.
> Expires: Thu, 01 Dec 1994 16:00:00 GMT
> Cache-control: no-cache
> Pragma: no-cache
> Connection: close
> But with Java, I get redirected only once to a https:// webpage and it's a
> dead end. Below is the result of java.net.URLConnection.getHeaderFiles()
> HTTP/1.1 301 Moved Permanently,
> Date: Thu, 09 Dec 2010 10:50:53 GMT,
> Content-type: text/html,
> Content-length: 0,
> Server: Sun-ONE-Web-Server/6.1,
> There is a clear difference between the two. I don't know why and it's been
> driving me crazy.
> My guess is that c++ write() function can create some kind of cookie by
> itself, but Java URL.openStream() can't.
> Am I right? Or can anyone explain this for me?