Problem while reading files from hdfs using python
Dave Angel
davea at davea.name
Tue Feb 3 18:02:52 EST 2015
On 01/25/2015 03:23 PM, Shalini Ravishankar wrote:
> Hello Everyone,
>
> I am trying to read(open) and write files in hdfs inside a python script. But having error.
Please copy/paste the full error traceback.
> Can someone tell me what is wrong here.
>
> Code (full): sample.py
>
> #!/usr/bin/python
>
>
> from subprocess import Popen, PIPE
>
> print "Before Loop"
>
> cat = Popen(["hadoop", "fs", "-cat", "./sample.txt"],
> stdout=PIPE)
I don't know anything about hadoop, and when you run it separately, you
used different parameters. So you can do a lot towards testing it yourself.
Start by running hadoop fs -cat ... from shell to see whether it
displays anything. You should be able to use exactly the same arguments
as you use in the Popen call.
Then if that seems to work as you expect, comment out your 'put' code
below, and add some prints to the loop. Does that look reasonable?
At that point, if both look reasonable, then try the inverse. Write
some known data to the 'put' command, and see if it makes it into the
appropriate file. Once again, you should be able to also test the
program parameters and behavior from the shell, typing manually into stdin.
> put = Popen(["hadoop", "fs", "-put", "-", "./modifiedfile.txt"],
> stdin=PIPE)
> for line in cat.stdout:
> line += "Blah"
> print line
> put.stdin.write(line)
>
> cat.stdout.close()
> cat.wait()
> put.stdin.close()
> put.wait()
>
> When I execute :
>
> hadoop jar /usr/local/hadoop/share/hadoop/tools/lib/hadoop-streaming-2.5.1.jar -file ./sample.py -mapper './sample.py' -input sample.txt -output fileRead
>
> It executes properly I couldn't find the file which supposed to create in hdfs modifiedfile
>
> And When I execute :
>
> hadoop fs -getmerge ./fileRead/ file.txt
>
> Inside the file.txt, I got :
>
> Before Loop
> Before Loop
>
> Can someone please tell me what I am doing wrong here ?? I dont think it reads from the sample.txt
>
> I would really appreciate the help.
>
>
> --
> Thanks & Regards,
> Shalini Ravishankar.
>
--
DaveA
More information about the Python-list
mailing list