[File Upload] How to Increase the Speed of File transfer ??

Discussion:

Bikash Paul

2003-08-19 08:50:47 UTC

Hi gang,

I have developed one file transfer(means file
uploading) application for that I have used
swing,servlet and Tomcat4.1.24.Now my problem is the
speed. The speed of manual copy and paste of file on
hard disk of remote computer is faster than my
application.I want to increase the speed of file
transfer.Networking is done through 2MBPS dedicated
ISDN lease line.Any help will be highly appreciated.

I want some suggestion about below factors:

1.Iam sending file to server as block wise of 1024
bytes and my servlet also writes file in remote
destination folder as a block wise of 1024 bytes.If I
increase the block size then is it increase the speed
?

2. I have used DataOutputStream for writing on
outputstream and InputStream for reading,if I used
BufferWriter and BufferReader then is it increase the
speed ?

3.Is there any tuning of tomcat,so that it increase
the speed ?

4.Lastly I have closed all my Input and output stream
object and I have also closed all connection at the
end of file transfer method.

Or is there any other factor for increasing Speed ?

Thanks & Regards
Bikash

__________________________________
Do you Yahoo!?
Yahoo! SiteBuilder - Free, easy-to-use web site design software
http://sitebuilder.yahoo.com

John Turner

2003-08-19 12:06:08 UTC

Permalink

OK, take a step back and think about this.

ATA-66 disk bus: 533 Mbps
ATA-100 disk bus: 800 Mbps
Ultra-160 SCSI disk bus: 1300 Mbps

Now, you're saying that you have a network connection of 2 Mbps (I think
you mean T1 or E1 as ISDN is not > 128 Kbps unless you have a bunch of
lines bonded together but that's not important). There's no way to
change Tomcat to make 2 Mbps be equal to 533 Mbps or even 20 Mbps.

A basic 200 MHz Pentium class server with even rudimentary disks can
easily saturate a 10 Mbps LAN connection. Unless your server was built
prior to 1997 or so, my guess is your network connection can never match
your server's internal bus "speed".

I think your solution is to stop writing blocks to disk as soon as you
receive them, and instead assemble your file in a buffer, then write it
out to disk all at once. Or switch to a dedicated GIGAbit line, which
with overhead would get you around ATA-100 speed.

John

Post by Bikash Paul
Hi gang,
I have developed one file transfer(means file
uploading) application for that I have used
swing,servlet and Tomcat4.1.24.Now my problem is the
speed. The speed of manual copy and paste of file on
hard disk of remote computer is faster than my
application.I want to increase the speed of file
transfer.Networking is done through 2MBPS dedicated
ISDN lease line.Any help will be highly appreciated.
1.Iam sending file to server as block wise of 1024
bytes and my servlet also writes file in remote
destination folder as a block wise of 1024 bytes.If I
increase the block size then is it increase the speed
?
2. I have used DataOutputStream for writing on
outputstream and InputStream for reading,if I used
BufferWriter and BufferReader then is it increase the
speed ?
3.Is there any tuning of tomcat,so that it increase
the speed ?
4.Lastly I have closed all my Input and output stream
object and I have also closed all connection at the
end of file transfer method.
Or is there any other factor for increasing Speed ?
Thanks & Regards
Bikash
__________________________________
Do you Yahoo!?
Yahoo! SiteBuilder - Free, easy-to-use web site design software
http://sitebuilder.yahoo.com
---------------------------------------------------------------------

Justin Ruthenbeck

2003-08-19 16:19:46 UTC

Permalink

If speed really is of paramount importance, consider directly opening
sockets and transfering data directly between code running on Tomcat and
your client application/applet/etc -- essentially taking Tomcat out of the
loop for the actual file transfer.

We've seen significant performance increases by doing this (though our
messages are short enough that the http/Tomcat overhead is
significant). Think long and hard before doing this as you'll give up most
of the benefits you get from Tomcat in the first place. It's an option,
though.

justin

Post by John Turner
OK, take a step back and think about this.
ATA-66 disk bus: 533 Mbps
ATA-100 disk bus: 800 Mbps
Ultra-160 SCSI disk bus: 1300 Mbps
Now, you're saying that you have a network connection of 2 Mbps (I think
you mean T1 or E1 as ISDN is not > 128 Kbps unless you have a bunch of
lines bonded together but that's not important). There's no way to change
Tomcat to make 2 Mbps be equal to 533 Mbps or even 20 Mbps.
A basic 200 MHz Pentium class server with even rudimentary disks can
easily saturate a 10 Mbps LAN connection. Unless your server was built
prior to 1997 or so, my guess is your network connection can never match
your server's internal bus "speed".
I think your solution is to stop writing blocks to disk as soon as you
receive them, and instead assemble your file in a buffer, then write it
out to disk all at once. Or switch to a dedicated GIGAbit line, which
with overhead would get you around ATA-100 speed.
John

---------------------------------------------------------------------

____________________________________
Justin Ruthenbeck
Software Engineer, NextEngine Inc.
justinr - AT - nextengine DOT com
Confidential
See http://www.nextengine.com/confidentiality.php
____________________________________

Shapira, Yoav

2003-08-19 13:07:58 UTC

Permalink

Howdy,
You will be hard-pressed to find any network file transmission
mechanism, even on unlimited bandwidth, that exceeds local hard-drive
folder to folder copy. Good luck ;)

Post by Bikash Paul
transfer.Networking is done through 2MBPS dedicated
ISDN lease line.Any help will be highly appreciated.

Verify you really have 2mbps bandwidth.

Post by Bikash Paul
1.Iam sending file to server as block wise of 1024
bytes and my servlet also writes file in remote
destination folder as a block wise of 1024 bytes.If I
increase the block size then is it increase the speed
?

Up to a certain block size, which is the MTU for the link with the
lowest MTU along your transmission path, yes.

Post by Bikash Paul
2. I have used DataOutputStream for writing on
outputstream and InputStream for reading,if I used
BufferWriter and BufferReader then is it increase the
speed ?

Actually it might decrease it. Give it a shot though ;)

Post by Bikash Paul
3.Is there any tuning of tomcat,so that it increase
the speed ?

Not really: I don't think tomcat would be your bottleneck. You could
try to implement a raw socket connector and use it instead of coyote,
but I doubt you'd see much performance difference.

Post by Bikash Paul
4.Lastly I have closed all my Input and output stream
object and I have also closed all connection at the
end of file transfer method.

That's a good practice that doesn't have much to do with transmission
speed.

Post by Bikash Paul
Or is there any other factor for increasing Speed ?

Compress your file better?

Yoav Shapira

This e-mail, including any attachments, is a confidential business communication, and may contain information that is confidential, proprietary and/or privileged. This e-mail is intended only for the individual(s) to whom it is addressed, and may not be saved, copied, printed, disclosed or used by anyone else. If you are not the(an) intended recipient, please immediately delete this e-mail from your computer system and notify the sender. Thank you.

Bikash Paul

2003-08-20 04:52:57 UTC

Permalink

Hi Gang,

Lot of thanks for all of urs suggestion.Now By
modifying some factors in my code like by compressing
file on the fly using util.zip package and using
Buffer stream and also increasing block size(16384
bytes) I can increase speed of file transfer upto some
extent means now speed of manual copy and my
application's file transfer speed both are same.Any
way Lot of thanks.

Thanks
Bikash

Post by Shapira, Yoav
Howdy,
You will be hard-pressed to find any network file
transmission
mechanism, even on unlimited bandwidth, that exceeds
local hard-drive
folder to folder copy. Good luck ;)

Post by Bikash Paul
transfer.Networking is done through 2MBPS dedicated
ISDN lease line.Any help will be highly

appreciated.
Verify you really have 2mbps bandwidth.

Post by Bikash Paul
1.Iam sending file to server as block wise of 1024
bytes and my servlet also writes file in remote
destination folder as a block wise of 1024 bytes.If

Post by Bikash Paul
increase the block size then is it increase the

speed

Post by Bikash Paul
?

Up to a certain block size, which is the MTU for the
link with the
lowest MTU along your transmission path, yes.

Post by Bikash Paul
2. I have used DataOutputStream for writing on
outputstream and InputStream for reading,if I used
BufferWriter and BufferReader then is it increase

the

Post by Bikash Paul
speed ?

Actually it might decrease it. Give it a shot
though ;)

Post by Bikash Paul
3.Is there any tuning of tomcat,so that it increase
the speed ?

Not really: I don't think tomcat would be your
bottleneck. You could
try to implement a raw socket connector and use it
instead of coyote,
but I doubt you'd see much performance difference.

Post by Bikash Paul
4.Lastly I have closed all my Input and output

stream

Post by Bikash Paul
object and I have also closed all connection at the
end of file transfer method.

That's a good practice that doesn't have much to do
with transmission
speed.

Post by Bikash Paul
Or is there any other factor for increasing Speed ?

Compress your file better?
Yoav Shapira
This e-mail, including any attachments, is a
confidential business communication, and may contain
information that is confidential, proprietary and/or
privileged. This e-mail is intended only for the
individual(s) to whom it is addressed, and may not
be saved, copied, printed, disclosed or used by
anyone else. If you are not the(an) intended
recipient, please immediately delete this e-mail
from your computer system and notify the sender.
Thank you.

---------------------------------------------------------------------
__________________________________
Do you Yahoo!?
Yahoo! SiteBuilder - Free, easy-to-use web site design software
http://sitebuilder.yahoo.com

Pablo Mayrgundter

2003-08-19 14:50:15 UTC

Permalink

Yeah.. test with higher block sizes.

Post by Bikash Paul
2. I have used DataOutputStream for writing on
outputstream and InputStream for reading,if I used
BufferWriter and BufferReader then is it increase the
speed ?

Defnitely a good idea.

--
Pablo Mayrgundter
Director of Applications and Services
www.reeltwo.com

Micael

2003-08-19 17:59:13 UTC

Permalink

You definitely should use a buffered stream. Get a book on networking and
take a look at these issues. There are many good ones out there. Network
Programming by Hughes, et al, is one I have enjoyed.

Micael

LEGAL NOTICE

This electronic mail transmission and any accompanying documents contain
information belonging to the sender which may be confidential and legally
privileged. This information is intended only for the use of the
individual or entity to whom this electronic mail transmission was sent as
indicated above. If you are not the intended recipient, any disclosure,
copying, distribution, or action taken in reliance on the contents of the
information contained in this transmission is strictly prohibited. If you
have received this transmission in error, please delete the message. Thank
you

Luciano Kiniti Issoe

2003-08-19 19:21:56 UTC

Permalink

1) Yes
2) Yes
3) No (AFAIK)
4) OK
5) Split your file and use threads in you application.

----- Original Message -----
From: "Micael" <***@harbornet.com>
To: "Tomcat Users List" <tomcat-***@jakarta.apache.org>;
<tomcat-***@jakarta.apache.org>
Sent: Tuesday, August 19, 2003 2:59 PM
Subject: Re: [File Upload] How to Increase the Speed of File transfer ??

Post by Micael
You definitely should use a buffered stream. Get a book on networking and
take a look at these issues. There are many good ones out there. Network
Programming by Hughes, et al, is one I have enjoyed.
Micael

LEGAL NOTICE
This electronic mail transmission and any accompanying documents contain
information belonging to the sender which may be confidential and legally
privileged. This information is intended only for the use of the
individual or entity to whom this electronic mail transmission was sent as
indicated above. If you are not the intended recipient, any disclosure,
copying, distribution, or action taken in reliance on the contents of the
information contained in this transmission is strictly prohibited. If you
have received this transmission in error, please delete the message.

Thank

Post by Micael
you
---------------------------------------------------------------------