Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
155 commits
Select commit Hold shift + click to select a range
82fb651
Fix read when message size wasn't read in one step
pankdm Apr 10, 2013
728edf8
Throw AnnounceException when current tracker client isn't available.
sroze Jul 18, 2013
9370d86
Fix order of uploaded/downloaded parameters in HTTPAnnounceRequestMes…
mpetazzoni Jul 22, 2013
1ad97ef
Merge pull request #45 from sroze/trackerless-patch
mpetazzoni Jul 22, 2013
2e643f4
add the ability to set upload/download rate-limits (in kB/sec.) on sh…
ptgoetz Jul 19, 2013
b15ea5e
add javadoc documentation for upload/download rate-limit functionality
ptgoetz Jul 22, 2013
73b78f2
reduce rate limit code duplication by refactoring code present in bot…
ptgoetz Jul 22, 2013
3ef9e9b
add example of upload/download rate limiting to the client usage sample.
ptgoetz Jul 22, 2013
13db7ab
add comments documenting the drawbacks/potential improvements for the…
ptgoetz Jul 22, 2013
00bc8f2
minor formatting/style modifications based on feedback from pull requ…
ptgoetz Jul 22, 2013
42775d5
Merge pull request #49 from ptgoetz/rate-limits
mpetazzoni Jul 23, 2013
59a2ec1
Fix a few styling issues from recent contributions
mpetazzoni Jul 23, 2013
c004fa8
Merge pull request #40 from pankdm/fix-incomplete-read
mpetazzoni Jul 23, 2013
64b8865
fixes Issue #47
oxlade39 Jul 22, 2013
a47a275
fixes turn/ttorrent#51
pwoodworth Jul 24, 2013
0dcb033
Merge pull request #50 from oxlade39/bug_fix
mpetazzoni Jul 24, 2013
bf4d679
Merge pull request #53 from pwoodworth/issue-51
mpetazzoni Jul 25, 2013
c1dbcd9
Update pom for compatibility with the maven-release-plugin
pwoodworth Jul 30, 2013
9e8abed
Use maven-shade-plugin to produce executable jar
pwoodworth Jul 30, 2013
5a499ce
Clean up javadoc problems
pwoodworth Jul 30, 2013
c06b110
Merge pull request #54 from pwoodworth/shaded
mpetazzoni Jul 31, 2013
f69190b
Merge pull request #55 from pwoodworth/releaseplugin
mpetazzoni Jul 31, 2013
89a6754
Merge pull request #56 from pwoodworth/javadoc
mpetazzoni Jul 31, 2013
0080296
Fix pom parent relative path
pwoodworth Aug 14, 2013
9e09ee5
Adding more platform agnostic shell scripts
pwoodworth Aug 15, 2013
d50eb99
Removing old shell scripts
pwoodworth Aug 15, 2013
86f4a2f
Merge pull request #57 from pwoodworth/scripts
mpetazzoni Aug 16, 2013
1c8dbfc
Move entry-point main methods to separate package
pwoodworth Aug 26, 2013
d33c361
Merge pull request #59 from pwoodworth/climove
mpetazzoni Aug 26, 2013
dbfd004
Fixes turn/ttorrent#42
pwoodworth Aug 28, 2013
06a3dfc
Merge pull request #60 from pwoodworth/issue-42
mpetazzoni Aug 28, 2013
dd8e7c9
Use commons-io throughout
pwoodworth Sep 11, 2013
f72705f
Merge pull request #63 from pwoodworth/commonsio
mpetazzoni Sep 12, 2013
d243274
Use commons-codec for hex and digest encoding
pwoodworth Aug 28, 2013
6fd448f
Merge pull request #64 from pwoodworth/codec
mpetazzoni Sep 17, 2013
869f664
Fix Piece compareTo ordering
pwoodworth Sep 20, 2013
32c66e1
Correcting improper use of Java 7 API
pwoodworth Sep 26, 2013
d787d9c
Merge pull request #65 from pwoodworth/piececompare
mpetazzoni Sep 30, 2013
cce0c02
Developer contact info updates
mpetazzoni Oct 24, 2013
b540ebd
Change FileCollectionStorage to not write 0 bytes to the next file at…
camerondavison Nov 25, 2013
053b13d
Instead of abusing SocketChannel.read in a tight loop, use a Selector
maxaf Oct 27, 2013
6ff991d
Fixing email address of contributor in README
mpetazzoni Nov 26, 2013
37bff2a
Code style fixes
mpetazzoni Nov 26, 2013
186dbb3
Merge pull request #73 from a86c6f7964/collectionfileselect
mpetazzoni Nov 26, 2013
d403b7a
Create Travis-CI configuration
mpetazzoni Dec 6, 2013
9dac22a
Add Travis-CI status image
mpetazzoni Dec 6, 2013
5c9f3cf
Bump to version 1.4; first release to Maven Central
mpetazzoni Dec 6, 2013
51f0676
[maven-release-plugin] prepare release ttorrent-1.4
mpetazzoni Dec 6, 2013
efe3a26
[maven-release-plugin] prepare for next development iteration
mpetazzoni Dec 6, 2013
9550b79
Add documentation about declaring ttorrent as a POM dependency
mpetazzoni Dec 8, 2013
ac0e8c0
SharedTorrent: Permit custom TorrentByteStorage implementations.
shevek Dec 25, 2013
7cef1ec
Torrent: 25% performance improvement in hashing.
shevek Dec 26, 2013
8ec410a
Performance improvements to torrent validation.
shevek Dec 26, 2013
4c146ff
Client: Don't mask unforseen exceptions.
shevek Dec 26, 2013
26c5548
SharedTorrent: The storage bucket isn't authoritative for size.
shevek Dec 26, 2013
14524ae
TorrentCreator: Correctly compute hash of last block.
shevek Dec 26, 2013
bfef6c3
HTTPTrackerClient: If you can't talk to a URL, say which URL.
shevek Dec 30, 2013
ededeea
Client: Include the IOException when a peer connection fails.
shevek Dec 30, 2013
a9ef4bb
Tracker: Wholesale clean-up and efficiency work.
shevek Dec 30, 2013
08095b2
HTTPTrackerClient: Fix error handling.
shevek Dec 30, 2013
b5559d0
TrackedPeer, TrackedTorrent: Efficiency and thread-correctness.
shevek Dec 30, 2013
ada03d4
Netty work.
shevek Jan 17, 2014
cbce422
Standardize software layout.
shevek Jan 20, 2014
b8c0761
Compiles; start writing tests.
shevek Jan 21, 2014
e0e9415
Port to JCL.
shevek Jan 21, 2014
9de9783
Towards testability.
shevek Jan 21, 2014
5209594
Clean up, split out to interfaces, start writing tests.
shevek Jan 22, 2014
1c8f245
Towards more testing.
shevek Jan 22, 2014
83c932f
ReplicationTest and associated startup work.
shevek Jan 23, 2014
3c74a3b
Factor out Peer, Torrent to clarify static analysis. Connection routi…
shevek Jan 24, 2014
88b6c48
Rutime peer management distinguishes between connected and unconnecte…
shevek Jan 24, 2014
7f59770
Fix tracker rescheduling.
shevek Jan 25, 2014
e926c00
More testing.
shevek Jan 26, 2014
f864b61
Make all tests pass.
shevek Jan 26, 2014
876cd97
ReplicationTest: Make deterministic.
shevek Jan 27, 2014
5b8c535
Remove Piece; replace with AtomicIntegerArray.
shevek Jan 28, 2014
c25071a
HTTPTrackerClient: CheckForNull.
shevek Jan 28, 2014
d28273f
Integrate yammer metrics. Fix double-counting of peers. Useful loggin…
shevek Jan 28, 2014
30f85c1
Correctness audits.
shevek Jan 28, 2014
140ac0c
Fix piece pipelining as well as blocks.
shevek Jan 28, 2014
235b996
ttorrent: Improve performance against lossy peers.
shevek Feb 6, 2014
e9dae9b
Fix buffer resizing on dropped packets.
shevek Feb 6, 2014
8bf7b76
Large swarm test now passes, but somewhat messy.
shevek Feb 6, 2014
1c85747
Use ephemeral ports for each tracker in the test suite.
shevek Feb 6, 2014
c049c29
HandshakeMessage: Only require enough data, but allow pipelining.
shevek Feb 7, 2014
e9bd106
Lots of performance work on larger clusters.
shevek Feb 7, 2014
dd977b8
findbugs fixes
shevek Feb 7, 2014
0ec0fea
More production API. Start BEP0010: Extended messages. Overhaul BEValue.
shevek Feb 10, 2014
8226200
TrackerMessage: Regularize and reduce cost of inheritance hierarchy.
shevek Feb 10, 2014
d533b0a
TorrentHandler: Handle single file torrents better.
shevek Feb 12, 2014
cdd6dd5
Tracker: Put IP into Tracker URI.
shevek Feb 12, 2014
459f3a6
add eclipse support to gradle config
christf Feb 18, 2014
4df8271
Fix formatting of eclipse patch.
shevek Feb 24, 2014
64bcdc2
Tracker supports multiple listen addresses.
shevek Feb 25, 2014
fddfb43
More testing.
shevek Feb 25, 2014
f5119fb
Test tracker migration.
shevek Feb 26, 2014
c6781cc
Support BEP0007: IPv6 in tracker responses.
shevek Feb 26, 2014
020f2c6
Normalize toHex into TorrentUtils.
shevek Feb 26, 2014
daa3229
Support multiple ipv4 and ipv6 addresses per peer on the tracker.
shevek Feb 27, 2014
c78f8fa
Client now works with zeroconf ipv6 and multiple paths to a destination.
shevek Feb 28, 2014
ceb76cf
TrackedTorrent: Fix byte[]->String in hash removals (findbugs).
shevek Feb 28, 2014
12bc779
Clean up SwarmHandler logic.
shevek Mar 1, 2014
c1c93d9
SwarmHandler: Closed wrong connection on duplicate.
shevek Mar 3, 2014
346c940
Towards PEX.
shevek Jul 24, 2014
cd8937c
Use errorprone compiler.
shevek Sep 4, 2014
ecbc286
Allow specifying listen port on Client.
shevek Sep 25, 2014
807c48f
Bump all deps; add incoming peers to known peers.
shevek Sep 25, 2014
c2e7a61
Finish PEX.
shevek Oct 2, 2014
fb4dba3
PeerServer implements PeerAddressProvider for local addresses so chan…
shevek Oct 9, 2014
4239b4f
PeerHandler: Fix request expiry (operator precedence bug)
shevek Oct 9, 2014
5d35d5e
Split into multi-module project.
shevek Oct 10, 2014
e8c772c
More project split work.
shevek Oct 10, 2014
7a7ed54
FileCollectionStorage: Fix out-by-one in select()
shevek Oct 10, 2014
78b94a0
Build useful application packages.
shevek Oct 11, 2014
727a995
Some Spring work. Split ReplicationTest.
shevek Oct 11, 2014
536ea9c
Servlet implementation. Fix some javadoc warnings.
shevek Oct 11, 2014
e36a563
TorrentCreator: Faster using MappedByteBuffer.
shevek Oct 13, 2014
df95b39
SwarmHandler: Defend against NPE when adding PEX seeds to a non-start…
shevek Oct 14, 2014
ca0aff1
Use more NIO-like calls for storage.
shevek Oct 15, 2014
4a5189c
Merge both EventLoopGroup(s) and the ScheduledExecutorService.
shevek Oct 16, 2014
dbc5334
ClientEnvironment: Support epoll. Clean ClientEnvironment startup and…
shevek Oct 21, 2014
32628dc
More Netty optimization. ReplicationTimeoutTest inexplicably passing …
shevek Oct 21, 2014
7f65eff
Gradle: Apply gradle-info-plugin.
shevek Oct 21, 2014
455599b
Fix PEX test - send local addresses in extended handshake and PEX.
shevek Oct 22, 2014
5b9e984
PeerExchangeTest: Use 8 peers.
shevek Oct 22, 2014
87f5daa
Major documentation update.
shevek Oct 23, 2014
7e3b49e
FIXUP: README
shevek Oct 23, 2014
db23f55
Fix mis-named tracker package.
shevek Oct 23, 2014
a30a163
RawFileStorage: Use flush() instead of inline code.
shevek Oct 25, 2014
92696ed
TrackerHandlerTest: Fix test case.
shevek Nov 3, 2014
ee80657
Fix issues with coalesced frames on the wire.
shevek Nov 5, 2014
88f5ad8
Fix #10: Check for query string in URI.
shevek Nov 6, 2014
2a9759c
Add empty PeerMessageTrafficShapingHandler to attempt to remove manua…
shevek Nov 6, 2014
fb59b82
PeerHandshakeHandler: Don't include length byte in lengthAdjustment.
shevek Nov 7, 2014
8b0e3fb
Reduce INFO logging -> DEBUG.
shevek Nov 7, 2014
a9f4729
Logging: Generally sort out.
shevek Dec 3, 2014
3128b0e
-> gradle 2.2.1
shevek Dec 3, 2014
8242d44
Clean up build setup.
shevek Dec 16, 2014
0669163
PeerServer: Fix findbugs NPE warning.
shevek Dec 29, 2014
37afd66
Fix NPE on Client shutdown with HTTPTrackerClient stop
jfriedly Dec 30, 2014
d068f76
Remove unused PeerServer
jfriedly Dec 30, 2014
694cdf3
Fix unknown port -1 issue when peer doesn't send a port
shevek Jan 6, 2015
4cbb66a
Add UbuntuImageDownloadTest
jfriedly Dec 30, 2014
5419df6
Dyanimcally compute requestedPieces instead of storing it
jfriedly Jan 7, 2015
25b5265
Documentation and log message fixes
jfriedly Jan 7, 2015
7244013
More javadoc
jfriedly Jan 8, 2015
349d5c4
Remove errorprone
shevek Mar 10, 2015
ce2434d
TorrentCreator: Fix support for large files by not using mmap.
shevek Mar 11, 2015
99915ed
More JSR305 annotations in TorrentCreator.
shevek Mar 11, 2015
d3d6b5c
Update to stdproject.
shevek May 22, 2015
548764a
Group javadocs usefully.
shevek May 22, 2015
edee8e4
Obsolete custom SuppressWarnings annotation in favour of findbugs.
shevek Jun 23, 2015
bbdc7fa
Update to Gradle 5.6.
shevek Jan 5, 2020
54fd497
Test needs more memory to complete.
shevek Jan 5, 2020
f8d80c6
Bump test memory on CORRECT test suite.
shevek Jan 5, 2020
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 6 additions & 4 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,11 +1,9 @@
# Git ignore patterns
# Author: Maxime Petazzoni <mpetazzoni@turn.com>
# Author: Shevek

# Ignore build output
/build/*

# Ignore Javadoc output
/doc/*
build

# Ignore any eventual Eclipse project files, these don't belong in the
# repository.
Expand All @@ -18,3 +16,7 @@
*.bak
*~
*~

/.gradle
/.nb-gradle

1 change: 1 addition & 0 deletions .travis.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
language: java
125 changes: 66 additions & 59 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,15 +10,23 @@ metainfo files creation and parsing capabilities. It is designed to be embedded
into larger applications, but its components can also be used as standalone
programs.

This fork of ttorrent is a complete structural overhaul for
correctness, improved performance using [Netty](http://netty.io/),
vastly reduced memory consumption, monitoring and metrics, and
optional integration with [Spring Framework](http://spring.io/) and
[Jetty](http://www.eclipse.org/jetty/).

Ttorrent supports the following BEPs (BitTorrent enhancement proposals):

* `BEP#0003`: The BitTorrent protocol specification
* `BEP#0003`: The BitTorrent protocol specification (complete)
This is the base official protocol specification, which Ttorrent implements
fully.
* `BEP#0007`: IPv6 Tracker Extension
* `BEP#0010`: Extension Protocol
* `BEP#0012`: Multi-tracker metadata extension
Full support for the `announce-list` meta-info key providing a tiered tracker
list.
* `BEP#0015`: UDP Tracker Protocol for BitTorrent
* `BEP#0015`: UDP Tracker Protocol for BitTorrent (partial)
The UDP tracker protocol is fully supported in the BitTorrent client to make
announce requests to UDP trackers. UDP tracker support itself is planned.
* `BEP#0020`: Peer ID conventions
Expand All @@ -28,6 +36,12 @@ Ttorrent supports the following BEPs (BitTorrent enhancement proposals):
Compact peer lists are supported in both the client and the tracker.
Currently the tracker only supports sending back compact peer lists
to an announce request.
* `BEP#0024`: Tracker Returns External IP

In addition, the following extensions are supported:

* Peer Exchange (ut\_pex) with PEX seeding for fast trackerless operation.
* Linux epoll() (See ClientEnvironment#setEventLoopType()).

History
-------
Expand All @@ -44,15 +58,9 @@ to re-integrate into another application;
* Snark's, which is old, and unfortunately unstable;
* bitext, which was also unfortunately unstable, and extremely slow.

This implementation aims at providing a down-to-earth, simple to use library.
No fancy protocol extensions are implemented here: just the basics that allows
for the exchange and distribution of files through the BitTorrent protocol.

Although the write performance of the BitTorrent client is currently quite poor
(~10MB/sec/connected peer), it has been measured that the distribution of a
150MB file to thousands of machines across several datacenters took no more
than 30 seconds, with very little network overhead for the initial seeder (only
125% of the original file size uploaded by the initial seeder).
The APIs and implementation were thoroughly overhauled in this fork,
with the objective of sharing a multi-gigabyte file to a thousand
hosts in a matter of seconds.


How to use
Expand All @@ -65,65 +73,67 @@ usage message on the console when invoked with the ``-h`` command-line flag.

### As a library

*Thanks to Anatoli Vladev for the code examples in #16.*
To use ``ttorrent`` is a library in your project, all you need is to
declare the dependency on the latest version of ``ttorrent``. For
example, if you use Maven, add the following in your POM's dependencies
section:

```xml
<dependencies>
...
<dependency>
<groupId>org.anarres.ttorrent</groupId>
<artifactId>ttorrent-client</groupId>
<version>1.8.0-SNAPSHOT</version>
</dependency>
</dependencies>
```

#### Client code

```java
// First, instantiate the Client object.
Client client = new Client(
// This is the interface the client will listen on (you might need something
// else than localhost here).
InetAddress.getLocalHost(),

// Load the torrent from the torrent file and use the given
// output directory. Partials downloads are automatically recovered.
SharedTorrent.fromFile(
new File("/path/to/your.torrent"),
new File("/path/to/output/directory")));

// At this point, can you either call download() to download the torrent and
// stop immediately after...
client.download();

// Or call client.share(...) with a seed time in seconds:
// client.share(3600);
// Which would seed the torrent for an hour after the download is complete.

// Downloading and seeding is done in background threads.
// To wait for this process to finish, call:
client.waitForCompletion();

// At any time you can call client.stop() to interrupt the download.
Client client = new Client();

// Configure the client as desired.
client.getEnvironment.set*();

// Add one or more torrents.
Torrent torrent = new Torrent(new File("/path/to/your.torrent"))
client.addTorrent(torrent, new File("/path/to/output/directory"));

client.start();

```
At any time you can call `client.stop()` to interrupt the download.
You can also add a
[ClientListener](ttorrent-client/src/main/java/com/turn/ttorrent/client/ClientListener.java)
to wait for a download to complete.

#### Tracker code

For a tracker, there are a number of options:

Standalone:

```java
// First, instantiate a Tracker object with the port you want it to listen on.
// The default tracker port recommended by the BitTorrent protocol is 6969.
Tracker tracker = new Tracker(new InetSocketAddress(6969));

// Then, for each torrent you wish to announce on this tracker, simply created
// a TrackedTorrent object and pass it to the tracker.announce() method:
FilenameFilter filter = new FilenameFilter() {
@Override
public boolean accept(File dir, String name) {
return name.endsWith(".torrent");
}
};

for (File f : new File("/path/to/torrent/files").listFiles(filter)) {
tracker.announce(TrackedTorrent.load(f));
}

// Once done, you just have to start the tracker's main operation loop:
SimpleTracker tracker = new SimpleTracker(new InetSocketAddress(6969));
tracker.addTorrent(new Torrent("/path/to/file"));
tracker.start();

// You can stop the tracker when you're done with:
tracker.stop();
```

Servlets/Jetty: See [TrackerServlet](ttorrent-tracker-servlet/src/main/java/com/turn/ttorrent/tracker/servlet/TrackerServlet.java).

Spring Framework: See [TrackerController](ttorrent-tracker-spring/src/main/java/com/turn/ttorrent/tracker/spring/TrackerController.java).

#### JavaDoc API

The [JavaDoc API](http://shevek.github.io/ttorrent/docs/javadoc/)
is available.

License
-------

Expand All @@ -134,14 +144,14 @@ License version 2.0. See COPYING file for more details.
Authors and contributors
------------------------

* Maxime Petazzoni <<mpetazzoni@turn.com>> (Platform Engineer at Turn, Inc)
* Maxime Petazzoni <<maxime.petazzoni@bulix.org>> (Software Engineer at SignalFuse, Inc)
Original author, main developer and maintainer
* David Giffin <<david@etsy.com>>
Contributed parallel hashing and multi-file torrent support.
* Thomas Zink <<thomas.zink@uni-konstanz.de>>
Fixed a piece length computation issue when the total torrent size is an
exact multiple of the piece size.
* Johan Parent <<parent_johan@yahoo.com>>
* Johan Parent <<parent\_johan@yahoo.com>>
Fixed a bug in unfresh peer collection and issues on download completion on
Windows platforms.
* Dmitriy Dumanskiy
Expand All @@ -153,8 +163,5 @@ Authors and contributors
Caveats
-------

* Client write performance is a bit poor, mainly due to a (too?) simple piece
caching algorithm.

Contributions are welcome in all areas, even more so for these few points
above!
9 changes: 9 additions & 0 deletions TODO
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
Code required:

1) Protocol extension to specify maximum block size - we want 1Mb.
2) Keepalives.

Tests required:

Connect to peers which never offer anything. Make sure rarestPiece is null.

18 changes: 0 additions & 18 deletions bin/client

This file was deleted.

18 changes: 0 additions & 18 deletions bin/torrent

This file was deleted.

18 changes: 0 additions & 18 deletions bin/tracker

This file was deleted.

Loading