Comments on: Checking if a remote file exists in python http://timbroder.com/2009/09/checking-if-remote-file-exists-in.html code. comics. crossfit. Thu, 11 Aug 2011 19:51:00 +0000 hourly 1 http://wordpress.org/?v=3.3 By: woxidu http://timbroder.com/2009/09/checking-if-remote-file-exists-in.html/comment-page-1#comment-20 woxidu Thu, 01 Oct 2009 00:55:54 +0000 http://beta.timbroder.com/2009/09/30/checking-if-a-remote-file-exists-in-python/#comment-20 I noticed that this got the wrong answer for some okcupid URLs that I threw at it. I don't think OKWS prints out "ERROR 404" for its 404 pages, but it does put "HTTP/1.0 404" in its response headesr. I think it might work better if you disregard whatever's coming out of stdout/stderr and just go by wget's return code. I'm pretty sure wget will return with 0 for all pages it fetches successfully and 1 otherwise.<br /><br />While you're disregarding the output, you don't need Popen or PIPE or communicate or stdin/stderr or whatever. We can just use the os.system call for this (though I admit it's kind of nice having the command components in a python list) and save a bunch of trouble. One key is using wget's '-q" option which makes it stop outputting anything (even on error). Here's a simper version I put together:<br /><br /><b><br />from os import system<br />def url_exists(url):<br />    command = "wget -qS --spider %s" % url<br />    res = system(command)<br />    return not res # "Success" is backwards for processes<br /></b><br /><br />Cheers!<br />Eli<br /><br /><i>PS, does blogger support code/pre blocks in comments?</i> I noticed that this got the wrong answer for some okcupid URLs that I threw at it. I don't think OKWS prints out "ERROR 404" for its 404 pages, but it does put "HTTP/1.0 404" in its response headesr. I think it might work better if you disregard whatever's coming out of stdout/stderr and just go by wget's return code. I'm pretty sure wget will return with 0 for all pages it fetches successfully and 1 otherwise.

While you're disregarding the output, you don't need Popen or PIPE or communicate or stdin/stderr or whatever. We can just use the os.system call for this (though I admit it's kind of nice having the command components in a python list) and save a bunch of trouble. One key is using wget's '-q" option which makes it stop outputting anything (even on error). Here's a simper version I put together:


from os import system
def url_exists(url):
    command = "wget -qS –spider %s" % url
    res = system(command)
    return not res # "Success" is backwards for processes

Cheers!
Eli

PS, does blogger support code/pre blocks in comments?

]]>