FTP client

Post by Pontus Ãstlund
Hello Pike List!
I was wondering if there's any built-in FTP client in Pike. I sure can't
find one. Does anyone have one laying around to spare, or some pointers to
an easy way to implement one?

I don't know of one in the standard library, and I don't have actual
code lying around, but it's not a terribly complicated protocol; you
could probably build it on top of basic Stdio.File sockets in under an
hour, especially if your needs are fairly simple. I'd be happy to help
out if you need a hand.

ChrisA

Pontus Östlund

2015-01-15 15:28:01 UTC

Well, I've actually started on one already based off of how the SMTP client
is implemented. So my FTP class inherits Protocols.NNTP which inherits
Stdio.FILE and I've then overridden some of the methods in NNTP in my FTP
class.

I have no problems connecting to the server, change working directory,
entering passive mode and such. The problem starts when I try to issue a
LIST command. Nothing happens and the read-loop hangs until the server
closes the connection. And I don't know why that is!

Post by Pontus Ãstlund
Hello Pike List!
I was wondering if there's any built-in FTP client in Pike. I sure can't
find one. Does anyone have one laying around to spare, or some pointers

Post by Pontus Ãstlund
an easy way to implement one?

Chris Angelico

2015-01-15 15:34:09 UTC

Post by Pontus Ãstlund
I have no problems connecting to the server, change working directory,
entering passive mode and such. The problem starts when I try to issue a
LIST command. Nothing happens and the read-loop hangs until the server
closes the connection. And I don't know why that is!

That's where you need a secondary connection. When you enter passive
mode, the server informs you of an IP and port to connect to, and then
you need to establish that connection before you send LIST. The file
listing will come up on that secondary conn.

ChrisA

Pontus Östlund

2015-01-15 15:43:30 UTC

Ah, but of course. Now I get it! I read about it but didn't fully get it
when I read it.
Back to the keyboard ;)

# Pontus

Pontus Östlund

2015-01-15 17:03:32 UTC

I almost got it ;)
I haven't done very much I/O stuff in my days so give me a pointer here
please.

Should I replace the current fd with a new one connected to the new port?
Or do I need a separate fd?

Post by Pontus Ãstlund
Ah, but of course. Now I get it! I read about it but didn't fully get it
when I read it.
Back to the keyboard ;)
# Pontus

Chris Angelico

2015-01-15 17:04:57 UTC

Post by Pontus Ãstlund
I almost got it ;)
I haven't done very much I/O stuff in my days so give me a pointer here
please.
Should I replace the current fd with a new one connected to the new port? Or
do I need a separate fd?

Separate one. You start with the control connection, then you
establish one or more data connections. You probably don't need more
than one data connection at a time, though I believe it is legal and
sometimes useful.

ChrisA

Bertrand LUPART - Linkeo.com

2015-01-15 17:51:06 UTC

Hello,

A Pike FTP client is something that sounds useful.

Please share on modules.gotpike.org or make your code available on github or something.

Thank you :)

--
Bertrand LUPART
29, rue du Colisée - 75008 Paris
+33 1 72 71 71 84

H. William Welliver III

2015-01-16 03:15:08 UTC

Agreed… I’ll also add that I’d love to have an SSH/SFTP protocol module, but that’s not at all a one hour project :)

Bill

Post by Bertrand LUPART - Linkeo.com
Hello,
A Pike FTP client is something that sounds useful.
Please share on modules.gotpike.org or make your code available on github or something.
Thank you :)

Chris Angelico

2015-01-16 03:40:25 UTC

On Fri, Jan 16, 2015 at 2:15 PM, H. William Welliver III

Post by H. William Welliver III
Agreed… I’ll also add that I’d love to have an SSH/SFTP protocol module, but that’s not at all a one hour project :)

Definitely! Pike does have a fairly awesome crypto module, though...
how much of the existing SSL work can be reused? It would be rather
awesome.

ChrisA

Pontus Östlund

2015-01-16 13:31:14 UTC

Okey, I now have a simple FTP client that at least seems to be working
somewhat. The only problem right now is that binary data seems to be
scrambled in some way since binary files are broken when retrieved and
written to local disk. Any ideas why?

https://github.com/poppa/Pike-Modules/blob/master/Protocols.pmod/FTP.pmod

A simple use case:

int main(int argc, array(string) argv)
{
Protocols.FTP.Client cli = Protocols.FTP.Client("graveyard");

if (!cli->login("poppa", "some-password-dude")) {
werror("Login failed!\n");
return 1;
}

cli->binary_mode();

cli->cwd("/Main");

cli->retr("house.jpg", __DIR__);

cli->quit();

return 0;
}

# Pontus

Post by Chris Angelico
On Fri, Jan 16, 2015 at 2:15 PM, H. William Welliver III

AgreedâŠ Iâll also add that Iâd love to have an SSH/SFTP protocol module,

but thatâs not at all a one hour project :)
Definitely! Pike does have a fairly awesome crypto module, though...
how much of the existing SSL work can be reused? It would be rather
awesome.
ChrisA

Arne Goedeke

2015-01-16 13:55:51 UTC

On first glance it seems like you are using Stdio.FILE()->gets() to read
the file. FILE does charset conversion and gets() will also throw array
line endings, so that might be a reason for the corrupted data.

arne

Post by Pontus Ãstlund
Okey, I now have a simple FTP client that at least seems to be working
somewhat. The only problem right now is that binary data seems to be
scrambled in some way since binary files are broken when retrieved and
written to local disk. Any ideas why?
https://github.com/poppa/Pike-Modules/blob/master/Protocols.pmod/FTP.pmod
int main(int argc, array(string) argv)
{
Protocols.FTP.Client cli = Protocols.FTP.Client("graveyard");
if (!cli->login("poppa", "some-password-dude")) {
werror("Login failed!\n");
return 1;
}
cli->binary_mode();
cli->cwd("/Main");
cli->retr("house.jpg", __DIR__);
cli->quit();
return 0;
}
# Pontus

Post by Chris Angelico
On Fri, Jan 16, 2015 at 2:15 PM, H. William Welliver III

Post by H. William Welliver III
Agreed… I’ll also add that I’d love to have an SSH/SFTP protocol module,

but that’s not at all a one hour project :)
Definitely! Pike does have a fairly awesome crypto module, though...
how much of the existing SSL work can be reused? It would be rather
awesome.
ChrisA

Chris Angelico

2015-01-16 13:57:08 UTC

You're using gets(), which expects line-based results. Then you
concatenate them all, and in the process, lose the newlines.

I've sent you through a GitHub PR with a fix that appears to work on my system.

ChrisA

Pontus Östlund

2015-01-16 14:09:29 UTC

Sweet, that did it. Thank you!

# Pontus

https://github.com/poppa/Pike-Modules/blob/master/Protocols.pmod/FTP.pmod
You're using gets(), which expects line-based results. Then you
concatenate them all, and in the process, lose the newlines.
I've sent you through a GitHub PR with a fix that appears to work on my system.
ChrisA

Pontus Östlund

2015-01-19 13:25:09 UTC

Okey, I've implemented most (common) commands - like file transferring,
directory listing and so on - and it seems to be working fairly well.

https://github.com/poppa/Pike-Modules/blob/master/Protocols.pmod/FTP.pmod

And any feedback would be appreciated :)

# Pontus

Post by Pontus Ãstlund
Sweet, that did it. Thank you!
# Pontus

Peter Bortas

2015-01-19 15:29:00 UTC

After a quick look:

I think "cmd2()" could do with a more descriptive name.

fd and fd2 would be more understandable as ctrlfd and listenfd or
something similar.
--
Peter Bortas

Post by Pontus Ãstlund
Okey, I've implemented most (common) commands - like file transferring,
directory listing and so on - and it seems to be working fairly well.
https://github.com/poppa/Pike-Modules/blob/master/Protocols.pmod/FTP.pmod
And any feedback would be appreciated :)
# Pontus

Post by Pontus Ãstlund
Sweet, that did it. Thank you!
# Pontus

You're using gets(), which expects line-based results. Then you
concatenate them all, and in the process, lose the newlines.
I've sent you through a GitHub PR with a fix that appears to work on my system.
ChrisA

Pontus Östlund

2015-01-19 15:36:29 UTC

Good point. Perhaps "passive_cmd" or "data_cmd" would work?

Post by Peter Bortas
I think "cmd2()" could do with a more descriptive name.
fd and fd2 would be more understandable as ctrlfd and listenfd or
something similar.
--
Peter Bortas

Post by Pontus Ãstlund
Okey, I've implemented most (common) commands - like file transferring,
directory listing and so on - and it seems to be working fairly well.

https://github.com/poppa/Pike-Modules/blob/master/Protocols.pmod/FTP.pmod

Post by Pontus Ãstlund
And any feedback would be appreciated :)
# Pontus

Post by Pontus Ãstlund
Sweet, that did it. Thank you!
# Pontus

Post by Pontus Ãstlund
Okey, I now have a simple FTP client that at least seems to be

working

Post by Pontus Ãstlund
somewhat. The only problem right now is that binary data seems to be
scrambled in some way since binary files are broken when retrieved

and

Post by Pontus Ãstlund
written to local disk. Any ideas why?

https://github.com/poppa/Pike-Modules/blob/master/Protocols.pmod/FTP.pmod

Post by Chris Angelico
You're using gets(), which expects line-based results. Then you
concatenate them all, and in the process, lose the newlines.
I've sent you through a GitHub PR with a fix that appears to work on my system.
ChrisA

Peter Bortas

2015-01-19 15:41:06 UTC

As long as they are easily memorable without having to remember in
which order they where created it works for me.
--
Peter Bortas

Post by Pontus Ãstlund
Good point. Perhaps "passive_cmd" or "data_cmd" would work?

Post by Peter Bortas
I think "cmd2()" could do with a more descriptive name.
fd and fd2 would be more understandable as ctrlfd and listenfd or
something similar.
--
Peter Bortas

Post by Pontus Ãstlund
Sweet, that did it. Thank you!
# Pontus

On Sat, Jan 17, 2015 at 12:31 AM, Pontus Östlund

You're using gets(), which expects line-based results. Then you
concatenate them all, and in the process, lose the newlines.
I've sent you through a GitHub PR with a fix that appears to work on
my
system.
ChrisA

Pontus Östlund

2015-01-20 09:39:06 UTC

I came to the conclusion that the other "cmd" method wasn't needed so I
removed it all together. The logic for the passive stuff is handled in
"cmd()".

Post by Peter Bortas
As long as they are easily memorable without having to remember in
which order they where created it works for me.
--
Peter Bortas

Post by Pontus Ãstlund
Good point. Perhaps "passive_cmd" or "data_cmd" would work?

Post by Peter Bortas
I think "cmd2()" could do with a more descriptive name.
fd and fd2 would be more understandable as ctrlfd and listenfd or
something similar.
--
Peter Bortas

Post by Pontus Ãstlund
Okey, I've implemented most (common) commands - like file

transferring,

Post by Pontus Ãstlund
directory listing and so on - and it seems to be working fairly well.

https://github.com/poppa/Pike-Modules/blob/master/Protocols.pmod/FTP.pmod

Post by Pontus Ãstlund
And any feedback would be appreciated :)
# Pontus

Post by Pontus Ãstlund
Sweet, that did it. Thank you!
# Pontus

On Sat, Jan 17, 2015 at 12:31 AM, Pontus Ãstlund

Post by Pontus Ãstlund
Okey, I now have a simple FTP client that at least seems to be working
somewhat. The only problem right now is that binary data seems to

Post by Pontus Ãstlund
scrambled in some way since binary files are broken when retrieved and
written to local disk. Any ideas why?

https://github.com/poppa/Pike-Modules/blob/master/Protocols.pmod/FTP.pmod

You're using gets(), which expects line-based results. Then you
concatenate them all, and in the process, lose the newlines.
I've sent you through a GitHub PR with a fix that appears to work on
my
system.
ChrisA

Stephen R. van den Berg

2015-01-15 15:34:23 UTC

Post by Pontus Ãstlund
entering passive mode and such. The problem starts when I try to issue a
LIST command. Nothing happens and the read-loop hangs until the server
closes the connection. And I don't know why that is!

That's because any file transfer command (including LIST, which transfers the content
of the directory(file)) results in a temporary separate TCP connection which is
being used to transfer the filecontent.

See: https://www.ietf.org/rfc/rfc959.txt

--
Stephen.

Henrik Grubbström

2015-01-26 17:25:29 UTC