if all sources are expired forget mesh

Fixes i2cp bug while connecting to remote router

README.md

@@ -1,39 +1,56 @@
 # MuWire - Easy Anonymous File-Sharing
 
-MuWire is an easy to use file-sharing program which offers anonymity using [I2P technology](http://geti2p.net).
+MuWire is an easy to use file-sharing program which offers anonymity using [I2P technology](http://geti2p.net).  It works on any platform Java works on, including Windows,MacOS,Linux.
 
 It is inspired by the LimeWire Gnutella client and developped by a former LimeWire developer.
 
-The project is in development.  You can find technical documentation in the "doc" folder.
+The current stable release - 0.2.5 is avaiable for download at http://muwire.com.  You can find technical documentation in the "doc" folder.
 
 ### Building
 
-You need JDK 8 or newer.  After installing that and setting up the appropriate paths, just type
+You need JRE 8 or newer.  After installing that and setting up the appropriate paths, just type
 
 ```
-./gradlew assemble 
+./gradlew clean assemble 
 ```
 
 If you want to run the unit tests, type
 ```
-./gradlew build
+./gradlew clean build
 ```
 
 Some of the UI tests will fail because they haven't been written yet :-/
 
 ### Running
 
-You need to have an I2P router up and running on the same machine.  After you build the application, look inside "gui/build/distributions".  Untar/unzip one of the "shadow" files and then run the jar contained inside.  
+You need to have an I2P router up and running on the same machine.  After you build the application, look inside "gui/build/distributions".  Untar/unzip one of the "shadow" files and then run the jar contained inside by typing "java -jar MuWire-x.y.z.jar" in a terminal or command prompt.  If you use a custom I2CP host and port, create a file $HOME/.MuWire/i2p.properties and put "i2cp.tcp.host=<host>" and "i2cp.tcp.port=<port>" in there.
 
-The first time you run MuWire it will ask you to select a nickname.  This nickname will be displayed with search results, so that others can verify the file was shared by you.
+The first time you run MuWire it will ask you to select a nickname.  This nickname will be displayed with search results, so that others can verify the file was shared by you.  It is best to leave MuWire running all the time, just like I2P.
-
-At the moment there are very few nodes on the network, so you will see very few connections and search results.  It is best to leave MuWire running all the time, just like I2P.
 
 
 ### Known bugs and limitations
 
-* Any shared files get re-hashed on startup
-* Sometimes the list of shared files gets lost 
 * Many UI features you would expect are not there yet
 
+### Quick FAQ
 
+* why is MuWire slow ?
+  
+  - too few sources you're downloading from
+  - you can increase the number of tunnels by using more tunnels via Options->I2P Inbound/Outbound Quantity 
+    the default is 4 and you could raise up to as high as 16 ( Caution !!!!)
+
+* my search is not returning (enough) results !
+
+  - search is keyword or hash based
+  - keywords and hash(es) are NOT regexed or wildcarded so they have to be complete
+    so searching for 'musi' will not return results with 'music' - you have to search for 'music'
+  - ALL keywords have to match 
+  - only use space for keyword separation
+  - if you already have the file in question it is not displayed ( can be changed via Options )
+
+* what's this right click -> 'Copy hash to clipboard' for ?
+
+  - if you have a specific file you wish to share or download you can use the hash as a unique identifier
+    to make sure you have exactly the right file.
+  - you can share this hash with others to ensure they are getting the right file

TODO.md

@@ -0,0 +1,40 @@
+# TODO List
+
+Not in any particular order yet
+
+### Big Items
+
+##### Bloom Filters
+
+This reduces query traffic by not sending last hop queries to peers that definitely do not have the file
+
+##### Two-tier Topology
+
+This helps with scalability
+
+##### Trust List Sharing
+
+For helping users make better decisions whom to trust
+
+##### Content Control Panel
+
+To allow every user to not route queries for content they do not like.  This is mostly GUI work, the backend part is simple
+
+##### Packaging With JRE, Embedded Router
+
+For ease of deployment for new users, and so that users do not need to run a separate I2P router
+
+##### Web UI, REST Interface, etc.
+
+Basically any non-gui non-cli user interface
+
+##### Metadata editing and search
+
+To enable parsing of metadata from known file types and the user editing it or adding manual metadata
+
+### Small Items
+
+* Wrapper of some kind for in-place upgrades
+* Download file sequentially
+* Unsharing of files
+* Multiple-selection download, Ctrl-A

cli/build.gradle

@@ -1,7 +1,22 @@
-apply plugin : 'application'
+buildscript {
 
+    repositories {
+        jcenter()
+        mavenLocal()
+    }
+
+    dependencies {
+        classpath 'com.github.jengelman.gradle.plugins:shadow:2.0.4'
+    }
+}
+
+apply plugin : 'application'
 mainClassName = 'com.muwire.cli.Cli'
+apply plugin : 'com.github.johnrengelman.shadow'
+
+applicationDefaultJvmArgs = ['-Djava.util.logging.config.file=logging.properties']
 
 dependencies {
     compile project(":core")
 }
+

cli/src/main/groovy/com/muwire/cli/Cli.groovy

@@ -1,7 +1,19 @@
 package com.muwire.cli
 
+import java.util.concurrent.CountDownLatch
+
 import com.muwire.core.Core
 import com.muwire.core.MuWireSettings
+import com.muwire.core.UILoadedEvent
+import com.muwire.core.connection.ConnectionAttemptStatus
+import com.muwire.core.connection.ConnectionEvent
+import com.muwire.core.connection.DisconnectionEvent
+import com.muwire.core.files.AllFilesLoadedEvent
+import com.muwire.core.files.FileHashedEvent
+import com.muwire.core.files.FileLoadedEvent
+import com.muwire.core.files.FileSharedEvent
+import com.muwire.core.upload.UploadEvent
+import com.muwire.core.upload.UploadFinishedEvent
 
 class Cli {
     
@@ -23,20 +35,110 @@ class Cli {
         
         Core core 
         try {
-            core = new Core(props, home, "0.0.8")
+            core = new Core(props, home, "0.3.6")
         } catch (Exception bad) {
             bad.printStackTrace(System.out)
             println "Failed to initialize core, exiting"
             System.exit(1)
         }
+
+                
         
+        
+        def filesList
+        if (args.length == 0) {
+            println "Enter a file containing list of files to share"
+            def reader = new BufferedReader(new InputStreamReader(System.in))
+            filesList = reader.readLine()
+        } else 
+            filesList = args[0]
+        
+        Thread.sleep(1000)
+        println "loading shared files from $filesList"
+        
+        // listener for shared files
+        def sharedListener = new SharedListener()
+        core.eventBus.register(FileHashedEvent.class, sharedListener)
+        core.eventBus.register(FileLoadedEvent.class, sharedListener)
+        
+        // for connections
+        def connectionsListener = new ConnectionListener()
+        core.eventBus.register(ConnectionEvent.class, connectionsListener)
+        core.eventBus.register(DisconnectionEvent.class, connectionsListener)
+        
+        // for uploads
+        def uploadsListener = new UploadsListener()
+        core.eventBus.register(UploadEvent.class, uploadsListener)
+        core.eventBus.register(UploadFinishedEvent.class, uploadsListener)
+        
+        Timer timer = new Timer("status-printer", true)
+        timer.schedule({
+            println String.valueOf(new Date()) + " Connections $connectionsListener.connections Uploads $uploadsListener.uploads Shared $sharedListener.shared"
+        } as TimerTask, 60000, 60000)
+        
+        def latch = new CountDownLatch(1)
+        def fileLoader = new Object() {
+            public void onAllFilesLoadedEvent(AllFilesLoadedEvent e) {
+                latch.countDown()
+            }
+        }
+        core.eventBus.register(AllFilesLoadedEvent.class, fileLoader)
         core.startServices()
-        
+
+        core.eventBus.publish(new UILoadedEvent())        
+        println "waiting for files to load"
+        latch.await()
         // now we begin
         println "MuWire is ready"
-        println "Enter a file containing list of files to share"
+        
-        def reader = new BufferedReader(new InputStreamReader(System.in))
+        filesList = new File(filesList)
-        def filesList = reader.readLine()
+        filesList.withReader { 
+            def toShare = it.readLine()
+            core.eventBus.publish(new FileSharedEvent(file : new File(toShare)))
+        }
+        Runtime.getRuntime().addShutdownHook({
+            println "shutting down.."
+            core.shutdown()
+            println "shutdown."
+        })
         Thread.sleep(Integer.MAX_VALUE)
     }
-}
+    
+    static class ConnectionListener {
+        volatile int connections
+        public void onConnectionEvent(ConnectionEvent e) {
+            if (e.status == ConnectionAttemptStatus.SUCCESSFUL)
+                connections++
+        }
+        public void onDisconnectionEvent(DisconnectionEvent e) {
+            connections--
+        }
+    }
+    
+    static class UploadsListener {
+        volatile int uploads
+        public void onUploadEvent(UploadEvent e) {
+            uploads++
+            println String.valueOf(new Date()) + " Starting upload of ${e.uploader.file.getName()} to ${e.uploader.request.downloader.getHumanReadableName()}"
+        }
+        public void onUploadFinishedEvent(UploadFinishedEvent e) {
+            uploads--
+            println String.valueOf(new Date()) + " Finished upload of ${e.uploader.file.getName()} to ${e.uploader.request.downloader.getHumanReadableName()}"
+        }
+    }
+    
+    static class SharedListener {
+        volatile int shared
+        void onFileHashedEvent(FileHashedEvent e) {
+            if (e.error != null)
+                println "ERROR $e.error"
+            else {
+                println "Shared file : $e.sharedFile.file"
+                shared++
+            }
+        }
+        void onFileLoadedEvent(FileLoadedEvent e) {
+            shared++
+        }
+    }
+}

cli/src/main/groovy/com/muwire/cli/CliDownloader.groovy

@@ -0,0 +1,166 @@
+package com.muwire.cli
+
+import java.util.concurrent.ConcurrentHashMap
+import java.util.concurrent.CountDownLatch
+
+import com.muwire.core.Core
+import com.muwire.core.MuWireSettings
+import com.muwire.core.connection.ConnectionAttemptStatus
+import com.muwire.core.connection.ConnectionEvent
+import com.muwire.core.download.DownloadStartedEvent
+import com.muwire.core.download.Downloader
+import com.muwire.core.download.UIDownloadEvent
+import com.muwire.core.search.QueryEvent
+import com.muwire.core.search.SearchEvent
+import com.muwire.core.search.UIResultEvent
+
+import net.i2p.data.Base64
+
+class CliDownloader {
+    
+    private static final List<Downloader> downloaders = Collections.synchronizedList(new ArrayList<>())
+    private static final Map<UUID,ResultsHolder> resultsListeners = new ConcurrentHashMap<>()
+    
+    public static void main(String []args) {
+        def home = System.getProperty("user.home") + File.separator + ".MuWire"
+        home = new File(home)
+        if (!home.exists())
+            home.mkdirs()
+        
+        def propsFile = new File(home,"MuWire.properties")
+        if (!propsFile.exists()) {
+            println "create props file ${propsFile.getAbsoluteFile()} before launching MuWire"
+            System.exit(1)
+        }
+        
+        def props = new Properties()
+        propsFile.withInputStream { props.load(it) }
+        props = new MuWireSettings(props)
+        
+        def filesList
+        int connections
+        int resultWait
+        if (args.length != 3) {
+            println "Enter a file containing list of hashes of files to download, " + 
+                "how many connections you want before searching" +
+                "and how long to wait for results to arrive"
+                System.exit(1)
+        } else {
+            filesList = args[0]
+            connections = Integer.parseInt(args[1])
+            resultWait = Integer.parseInt(args[2])
+        }
+
+        Core core
+        try {
+            core = new Core(props, home, "0.3.6")
+        } catch (Exception bad) {
+            bad.printStackTrace(System.out)
+            println "Failed to initialize core, exiting"
+            System.exit(1)
+        }
+        
+        
+        def latch = new CountDownLatch(connections)
+        def connectionListener = new ConnectionWaiter(latch : latch)
+        core.eventBus.register(ConnectionEvent.class, connectionListener)
+        
+        core.startServices()
+        println "starting to wait until there are $connections connections"
+        latch.await()
+        
+        println "connected, searching for files"
+        
+        def file = new File(filesList)
+        file.eachLine {
+            String[] split = it.split(",")
+            UUID uuid = UUID.randomUUID()
+            core.eventBus.register(UIResultEvent.class, new ResultsListener(fileName : split[1]))
+            def hash = Base64.decode(split[0])
+            def searchEvent = new SearchEvent(searchHash : hash, uuid : uuid)
+            core.eventBus.publish(new QueryEvent(searchEvent : searchEvent, firstHop:true,
+                replyTo: core.me.destination, receivedOn : core.me.destination, originator: core.me)) 
+        }
+        
+        println "waiting for results to arrive"
+        Thread.sleep(resultWait * 1000)
+        
+        core.eventBus.register(DownloadStartedEvent.class, new DownloadListener())
+        resultsListeners.each { uuid, resultsListener ->
+            println "starting download of $resultsListener.fileName from ${resultsListener.getResults().size()} hosts"
+            File target = new File(resultsListener.fileName)
+            
+            core.eventBus.publish(new UIDownloadEvent(target : target, result : resultsListener.getResults()))
+        }
+        
+        Thread.sleep(1000)
+        
+        Timer timer = new Timer("stats-printer")
+        timer.schedule({
+            println "==== STATUS UPDATE ==="
+            downloaders.each {
+                int donePieces = it.donePieces()
+                int totalPieces = it.nPieces
+                int sources = it.activeWorkers.size()
+                def root = Base64.encode(it.infoHash.getRoot())
+                def state = it.getCurrentState()
+                println "file $it.file hash: $root progress: $donePieces/$totalPieces sources: $sources status: $state}"
+                it.resume()
+            }
+            println "==== END ==="
+        } as TimerTask, 60000, 60000)
+        
+        println "waiting for downloads to finish"
+        while(true) {
+            boolean allFinished = true
+            for (Downloader d : downloaders) {
+                allFinished &= d.getCurrentState() == Downloader.DownloadState.FINISHED
+            }
+            if (allFinished)
+                break
+            Thread.sleep(1000)
+        }
+        
+        println "all downloads finished"
+    }
+    
+    static class ResultsHolder {
+        final List<UIResultEvent> results = Collections.synchronizedList(new ArrayList<>())
+        String fileName
+        void add(UIResultEvent e) {
+            results.add(e)
+        }
+        List getResults() {
+            results
+        }
+    }
+    
+    static class ResultsListener {
+        UUID uuid
+        String fileName
+        public onUIResultEvent(UIResultEvent e) {
+            println "got a result for $fileName from ${e.sender.getHumanReadableName()}"
+            ResultsHolder listener = resultsListeners.get(e.uuid)
+            if (listener == null) {
+                listener = new ResultsHolder(fileName : fileName)
+                resultsListeners.put(e.uuid, listener)
+            }
+            listener.add(e)
+        }
+    }
+    
+    static class ConnectionWaiter {
+        CountDownLatch latch
+        public void onConnectionEvent(ConnectionEvent e) {
+            if (e.status == ConnectionAttemptStatus.SUCCESSFUL)
+                latch.countDown()
+        }
+    }
+    
+
+    static class DownloadListener {
+        public void onDownloadStartedEvent(DownloadStartedEvent e) {
+            downloaders.add(e.downloader)
+        }
+    }
+}

cli/src/main/groovy/com/muwire/cli/FileList.groovy

@@ -0,0 +1,23 @@
+package com.muwire.cli
+
+import com.muwire.core.util.DataUtil
+
+import groovy.json.JsonSlurper
+import net.i2p.data.Base64
+
+class FileList {
+    public static void main(String [] args) {
+        if (args.length < 1) {
+            println "pass files.json as argument"
+            System.exit(1)
+        }
+        
+        def slurper = new JsonSlurper()
+        File filesJson = new File(args[0])
+        filesJson.eachLine { 
+            def json = slurper.parseText(it)
+            String name = DataUtil.readi18nString(Base64.decode(json.file))
+            println "$name,$json.length,$json.pieceSize,$json.infoHash"
+        }
+    }
+}

core/src/main/groovy/com/muwire/core/Constants.groovy

@@ -11,5 +11,5 @@ class Constants {
     
     public static final float DOWNLOAD_SEQUENTIAL_RATIO = 0.8f
     
-    public static final String SPLIT_PATTERN = "[\\.,_-]"
+    public static final String SPLIT_PATTERN = "[\\+\\-,\\.:;\\(\\)=_/\\\\\\!\\\"\\\'\\\$%\\|]"
 }

core/src/main/groovy/com/muwire/core/Core.groovy

@@ -1,6 +1,7 @@
 package com.muwire.core
 
 import java.nio.charset.StandardCharsets
+import java.util.concurrent.atomic.AtomicBoolean
 
 import com.muwire.core.connection.ConnectionAcceptor
 import com.muwire.core.connection.ConnectionEstablisher
@@ -12,7 +13,11 @@ import com.muwire.core.connection.I2PConnector
 import com.muwire.core.connection.LeafConnectionManager
 import com.muwire.core.connection.UltrapeerConnectionManager
 import com.muwire.core.download.DownloadManager
+import com.muwire.core.download.SourceDiscoveredEvent
+import com.muwire.core.download.UIDownloadCancelledEvent
 import com.muwire.core.download.UIDownloadEvent
+import com.muwire.core.download.UIDownloadPausedEvent
+import com.muwire.core.download.UIDownloadResumedEvent
 import com.muwire.core.files.FileDownloadedEvent
 import com.muwire.core.files.FileHashedEvent
 import com.muwire.core.files.FileHasher
@@ -22,9 +27,11 @@ import com.muwire.core.files.FileSharedEvent
 import com.muwire.core.files.FileUnsharedEvent
 import com.muwire.core.files.HasherService
 import com.muwire.core.files.PersisterService
+import com.muwire.core.files.DirectoryWatcher
 import com.muwire.core.hostcache.CacheClient
 import com.muwire.core.hostcache.HostCache
 import com.muwire.core.hostcache.HostDiscoveredEvent
+import com.muwire.core.mesh.MeshManager
 import com.muwire.core.search.QueryEvent
 import com.muwire.core.search.ResultsEvent
 import com.muwire.core.search.ResultsSender
@@ -43,6 +50,7 @@ import net.i2p.client.I2PSession
 import net.i2p.client.streaming.I2PSocketManager
 import net.i2p.client.streaming.I2PSocketManagerFactory
 import net.i2p.client.streaming.I2PSocketOptions
+import net.i2p.client.streaming.I2PSocketManager.DisconnectListener
 import net.i2p.crypto.DSAEngine
 import net.i2p.crypto.SigType
 import net.i2p.data.Destination
@@ -56,6 +64,8 @@ public class Core {
     final EventBus eventBus
     final Persona me
     final File home
+    final Properties i2pOptions
+    final MuWireSettings muOptions
     
     private final TrustService trustService
     private final PersisterService persisterService
@@ -66,9 +76,15 @@ public class Core {
     private final ConnectionAcceptor connectionAcceptor
     private final ConnectionEstablisher connectionEstablisher
     private final HasherService hasherService
+    private final DownloadManager downloadManager
+    private final DirectoryWatcher directoryWatcher
+    final FileManager fileManager
+    
+    final AtomicBoolean shutdown = new AtomicBoolean()
         
     public Core(MuWireSettings props, File home, String myVersion) {
         this.home = home		
+        this.muOptions = props
         log.info "Initializing I2P context"
         I2PAppContext.getGlobalContext().logManager()
         I2PAppContext.getGlobalContext()._logManager = new MuWireLogManager()
@@ -83,15 +99,35 @@ public class Core {
 			}
 		}
 		
-		def sysProps = System.getProperties().clone()
+        i2pOptions = new Properties()
-		sysProps["inbound.nickname"] = "MuWire"
+        def i2pOptionsFile = new File(home,"i2p.properties")
+        if (i2pOptionsFile.exists()) {
+            i2pOptionsFile.withInputStream { i2pOptions.load(it) }
+
+            if (!i2pOptions.containsKey("inbound.nickname"))
+                i2pOptions["inbound.nickname"] = "MuWire"
+            if (!i2pOptions.containsKey("outbound.nickname"))
+                i2pOptions["outbound.nickname"] = "MuWire"
+        } else {
+            i2pOptions["inbound.nickname"] = "MuWire"
+            i2pOptions["outbound.nickname"] = "MuWire"
+            i2pOptions["inbound.length"] = "3"
+            i2pOptions["inbound.quantity"] = "4"
+            i2pOptions["outbound.length"] = "3"
+            i2pOptions["outbound.quantity"] = "4"
+            i2pOptions["i2cp.tcp.host"] = "127.0.0.1"
+            i2pOptions["i2cp.tcp.port"] = "7654"
+        }
+        
+        // options like tunnel length and quantity
 		I2PSession i2pSession
 		I2PSocketManager socketManager
 		keyDat.withInputStream {
-			socketManager = new I2PSocketManagerFactory().createManager(it, sysProps)
+			socketManager = new I2PSocketManagerFactory().createManager(it, i2pOptions["i2cp.tcp.host"], i2pOptions["i2cp.tcp.port"].toInteger(), i2pOptions)
 		}
 		socketManager.getDefaultOptions().setReadTimeout(60000)
 		socketManager.getDefaultOptions().setConnectTimeout(30000)
+        socketManager.addDisconnectListener({eventBus.publish(new RouterDisconnectedEvent())} as DisconnectListener)
 		i2pSession = socketManager.getSession()
 		
         def destination = new Destination()
@@ -129,15 +165,20 @@ public class Core {
 		
 		
 		log.info "initializing file manager"
-		FileManager fileManager = new FileManager(eventBus)
+		fileManager = new FileManager(eventBus, props)
 		eventBus.register(FileHashedEvent.class, fileManager)
 		eventBus.register(FileLoadedEvent.class, fileManager)
 		eventBus.register(FileDownloadedEvent.class, fileManager)
 		eventBus.register(FileUnsharedEvent.class, fileManager)
 		eventBus.register(SearchEvent.class, fileManager)
+        
+        log.info("initializing mesh manager")
+        MeshManager meshManager = new MeshManager(fileManager, home)
+        eventBus.register(SourceDiscoveredEvent.class, meshManager)
 		
 		log.info "initializing persistence service"
-		persisterService = new PersisterService(new File(home, "files.json"), eventBus, 5000, fileManager)
+		persisterService = new PersisterService(new File(home, "files.json"), eventBus, 15000, fileManager)
+        eventBus.register(UILoadedEvent.class, persisterService)
         
 		log.info("initializing host cache")
 		File hostStorage = new File(home, "hosts.json")
@@ -147,7 +188,8 @@ public class Core {
 		
 		log.info("initializing connection manager")
 		connectionManager = props.isLeaf() ? 
-			new LeafConnectionManager(eventBus, me, 3, hostCache) : new UltrapeerConnectionManager(eventBus, me, 512, 512, hostCache, trustService)
+			new LeafConnectionManager(eventBus, me, 3, hostCache, props) : 
+            new UltrapeerConnectionManager(eventBus, me, 512, 512, hostCache, trustService, props)
 		eventBus.register(TrustEvent.class, connectionManager)
 		eventBus.register(ConnectionEvent.class, connectionManager)
 		eventBus.register(DisconnectionEvent.class, connectionManager)
@@ -171,31 +213,40 @@ public class Core {
 		eventBus.register(ResultsEvent.class, searchManager)
 		
         log.info("initializing download manager")
-        DownloadManager downloadManager = new DownloadManager(eventBus, i2pConnector, new File(home, "incompletes"), me)
+        downloadManager = new DownloadManager(eventBus, trustService, meshManager, props, i2pConnector, home, me)
         eventBus.register(UIDownloadEvent.class, downloadManager)
-        
+        eventBus.register(UILoadedEvent.class, downloadManager)
+        eventBus.register(FileDownloadedEvent.class, downloadManager)
+        eventBus.register(UIDownloadCancelledEvent.class, downloadManager)
+        eventBus.register(SourceDiscoveredEvent.class, downloadManager)
+        eventBus.register(UIDownloadPausedEvent.class, downloadManager)
+        eventBus.register(UIDownloadResumedEvent.class, downloadManager)
+                
         log.info("initializing upload manager")
-        UploadManager uploadManager = new UploadManager(eventBus, fileManager)
+        UploadManager uploadManager = new UploadManager(eventBus, fileManager, meshManager, downloadManager)
         
+        log.info("initializing connection establisher")
+        connectionEstablisher = new ConnectionEstablisher(eventBus, i2pConnector, props, connectionManager, hostCache)
         
 		log.info("initializing acceptor")
 		I2PAcceptor i2pAcceptor = new I2PAcceptor(socketManager)
 		connectionAcceptor = new ConnectionAcceptor(eventBus, connectionManager, props, 
-            i2pAcceptor, hostCache, trustService, searchManager, uploadManager)
+            i2pAcceptor, hostCache, trustService, searchManager, uploadManager, connectionEstablisher)
 		
-        
+        log.info("initializing directory watcher")
-        connectionEstablisher = new ConnectionEstablisher(eventBus, i2pConnector, props, connectionManager, hostCache)
+        directoryWatcher = new DirectoryWatcher(eventBus, fileManager)
+        eventBus.register(FileSharedEvent.class, directoryWatcher)
         
         log.info("initializing hasher service")
-        hasherService = new HasherService(new FileHasher(), eventBus)
+        hasherService = new HasherService(new FileHasher(), eventBus, fileManager)
         eventBus.register(FileSharedEvent.class, hasherService)
 	}
     
     public void startServices() {
         hasherService.start()
+        directoryWatcher.start()
         trustService.start()
         trustService.waitForLoad()
-        persisterService.start()
         hostCache.start()
         connectionManager.start()
         cacheClient.start()
@@ -206,6 +257,19 @@ public class Core {
     }
     
     public void shutdown() {
+        if (!shutdown.compareAndSet(false, true)) {
+            log.info("already shutting down")
+            return
+        }
+        log.info("shutting down download manageer")
+        downloadManager.shutdown()
+        log.info("shutting down connection acceeptor")
+        connectionAcceptor.stop()
+        log.info("shutting down connection establisher")
+        connectionEstablisher.stop()
+        log.info("shutting down directory watcher")
+        directoryWatcher.stop()
+        log.info("shutting down connection manager")
         connectionManager.shutdown()
     }
 
@@ -233,7 +297,7 @@ public class Core {
             }
         }
         
-        Core core = new Core(props, home, "0.0.8")
+        Core core = new Core(props, home, "0.3.6")
         core.startServices()
         
         // ... at the end, sleep or execute script

core/src/main/groovy/com/muwire/core/EventBus.groovy

@@ -3,6 +3,7 @@ package com.muwire.core
 import java.util.concurrent.CopyOnWriteArrayList
 import java.util.concurrent.Executor
 import java.util.concurrent.Executors
+import java.util.logging.Level
 
 import com.muwire.core.files.FileSharedEvent
 
@@ -23,14 +24,18 @@ class EventBus {
 	}
 	
 	private void publishInternal(Event e) {
-		log.fine "publishing event $e of type ${e.getClass().getSimpleName()}"
+		log.fine "publishing event $e of type ${e.getClass().getSimpleName()} event $e"
 		def currentHandlers
 		final def clazz = e.getClass()
 		synchronized(this) {
 			currentHandlers = handlers.getOrDefault(clazz, [])
 		}
 		currentHandlers.each {
-			it."on${clazz.getSimpleName()}"(e)
+            try {
+                it."on${clazz.getSimpleName()}"(e)
+            } catch (Exception bad) {
+                log.log(Level.SEVERE, "exception dispatching event",bad)
+            }
 		}
 	}
 	

core/src/main/groovy/com/muwire/core/MuWireSettings.groovy

@@ -1,6 +1,11 @@
 package com.muwire.core
 
+import java.util.stream.Collectors
+
 import com.muwire.core.hostcache.CrawlerResponse
+import com.muwire.core.util.DataUtil
+
+import net.i2p.data.Base64
 
 class MuWireSettings {
 	
@@ -10,8 +15,9 @@ class MuWireSettings {
     int updateCheckInterval
     String nickname
     File downloadLocation
-    String sharedFiles
     CrawlerResponse crawlerResponse
+    boolean shareDownloadedFiles
+    Set<String> watchedDirectories
     
 	MuWireSettings() {
         this(new Properties())
@@ -24,9 +30,16 @@ class MuWireSettings {
         nickname = props.getProperty("nickname","MuWireUser")
         downloadLocation = new File((String)props.getProperty("downloadLocation", 
             System.getProperty("user.home")))
-        sharedFiles = props.getProperty("sharedFiles")
+        downloadRetryInterval = Integer.parseInt(props.getProperty("downloadRetryInterval","1"))
-        downloadRetryInterval = Integer.parseInt(props.getProperty("downloadRetryInterval","15"))
+        updateCheckInterval = Integer.parseInt(props.getProperty("updateCheckInterval","24"))
-        updateCheckInterval = Integer.parseInt(props.getProperty("updateCheckInterval","36"))
+        shareDownloadedFiles = Boolean.parseBoolean(props.getProperty("shareDownloadedFiles","true"))
+        
+        watchedDirectories = new HashSet<>()
+        if (props.containsKey("watchedDirectories")) {
+            String[] encoded = props.getProperty("watchedDirectories").split(",")
+            encoded.each { watchedDirectories << DataUtil.readi18nString(Base64.decode(it)) }
+        }
+        
 	}
     
     void write(OutputStream out) throws IOException {
@@ -38,8 +51,15 @@ class MuWireSettings {
         props.setProperty("downloadLocation", downloadLocation.getAbsolutePath())
         props.setProperty("downloadRetryInterval", String.valueOf(downloadRetryInterval))
         props.setProperty("updateCheckInterval", String.valueOf(updateCheckInterval))
-        if (sharedFiles != null)
+        props.setProperty("shareDownloadedFiles", String.valueOf(shareDownloadedFiles))
-            props.setProperty("sharedFiles", sharedFiles)
+        
+        if (!watchedDirectories.isEmpty()) {
+            String encoded = watchedDirectories.stream().
+                map({Base64.encode(DataUtil.encodei18nString(it))}).
+                collect(Collectors.joining(","))
+            props.setProperty("watchedDirectories", encoded)
+        }
+        
         props.store(out, "")
     }
 

core/src/main/groovy/com/muwire/core/RouterDisconnectedEvent.groovy

@@ -0,0 +1,4 @@
+package com.muwire.core
+
+class RouterDisconnectedEvent extends Event {
+}

core/src/main/groovy/com/muwire/core/UILoadedEvent.groovy

@@ -0,0 +1,4 @@
+package com.muwire.core
+
+class UILoadedEvent extends Event {
+}

core/src/main/groovy/com/muwire/core/connection/Connection.groovy

@@ -6,6 +6,7 @@ import java.util.concurrent.atomic.AtomicBoolean
 import java.util.logging.Level
 
 import com.muwire.core.EventBus
+import com.muwire.core.MuWireSettings
 import com.muwire.core.Persona
 import com.muwire.core.hostcache.HostCache
 import com.muwire.core.hostcache.HostDiscoveredEvent
@@ -26,7 +27,8 @@ abstract class Connection implements Closeable {
 	final boolean incoming
 	final HostCache hostCache
     final TrustService trustService
-	
+    final MuWireSettings settings
+    	
 	private final AtomicBoolean running = new AtomicBoolean()
 	private final BlockingQueue messages = new LinkedBlockingQueue()
 	private final Thread reader, writer
@@ -35,12 +37,14 @@ abstract class Connection implements Closeable {
 	
 	long lastPingSentTime, lastPongReceivedTime
 	
-	Connection(EventBus eventBus, Endpoint endpoint, boolean incoming, HostCache hostCache, TrustService trustService) {
+	Connection(EventBus eventBus, Endpoint endpoint, boolean incoming, 
+        HostCache hostCache, TrustService trustService, MuWireSettings settings) {
 		this.eventBus = eventBus
 		this.incoming = incoming
 		this.endpoint = endpoint
 		this.hostCache = hostCache
         this.trustService = trustService
+        this.settings = settings
 		
 		this.name = endpoint.destination.toBase32().substring(0,8)
 		
@@ -72,9 +76,9 @@ abstract class Connection implements Closeable {
 			return
 		}
         log.info("closing $name")
-        endpoint.close()
 		reader.interrupt()
 		writer.interrupt()
+		endpoint.close()
 		eventBus.publish(new DisconnectionEvent(destination: endpoint.destination))
 	}
 	
@@ -123,6 +127,9 @@ abstract class Connection implements Closeable {
         query.uuid = e.searchEvent.getUuid()
         query.firstHop = e.firstHop
         query.keywords = e.searchEvent.getSearchTerms()
+        query.oobInfohash = e.searchEvent.oobInfohash
+        if (e.searchEvent.searchHash != null)
+            query.infohash = Base64.encode(e.searchEvent.searchHash)
         query.replyTo = e.replyTo.toBase64()
         if (e.originator != null)
             query.originator = e.originator.toBase64()
@@ -151,15 +158,22 @@ abstract class Connection implements Closeable {
     
     protected void handleSearch(def search) {
         UUID uuid = UUID.fromString(search.uuid)
-        if (search.infohash != null)
+        byte [] infohash = null
+        if (search.infohash != null) {
             search.keywords = null
+            infohash = Base64.decode(search.infohash)
+        }
             
         Destination replyTo = new Destination(search.replyTo)
-        if (trustService.getLevel(replyTo) == TrustLevel.DISTRUSTED) {
+        TrustLevel trustLevel = trustService.getLevel(replyTo)
+        if (trustLevel == TrustLevel.DISTRUSTED) {
             log.info "dropping search from distrusted peer"
             return
         }
-        // TODO: add option to respond only to trusted peers
+        if (trustLevel == TrustLevel.NEUTRAL && !settings.allowUntrusted()) {
+            log.info("dropping search from neutral peer")
+            return
+        }
         
         Persona originator = null
         if (search.originator != null) {
@@ -170,10 +184,14 @@ abstract class Connection implements Closeable {
             }
         }
         
+        boolean oob = false
+        if (search.oobInfohash != null)
+            oob = search.oobInfohash
         
         SearchEvent searchEvent = new SearchEvent(searchTerms : search.keywords,
-                                            searchHash : search.infohash,
+                                            searchHash : infohash,
-                                            uuid : uuid)
+                                            uuid : uuid,
+                                            oobInfohash : oob)
         QueryEvent event = new QueryEvent ( searchEvent : searchEvent,
                                             replyTo : replyTo,
                                             originator : originator,

core/src/main/groovy/com/muwire/core/connection/ConnectionAcceptor.groovy

@@ -17,6 +17,8 @@ import com.muwire.core.upload.UploadManager
 import com.muwire.core.search.InvalidSearchResultException
 import com.muwire.core.search.ResultsParser
 import com.muwire.core.search.SearchManager
+import com.muwire.core.search.UIResultBatchEvent
+import com.muwire.core.search.UIResultEvent
 import com.muwire.core.search.UnexpectedResultsException
 
 import groovy.json.JsonOutput
@@ -34,13 +36,17 @@ class ConnectionAcceptor {
 	final TrustService trustService
     final SearchManager searchManager 
     final UploadManager uploadManager
+    final ConnectionEstablisher establisher
 	
 	final ExecutorService acceptorThread
 	final ExecutorService handshakerThreads
+    
+    private volatile shutdown
 	
 	ConnectionAcceptor(EventBus eventBus, UltrapeerConnectionManager manager,
 		MuWireSettings settings, I2PAcceptor acceptor, HostCache hostCache,
-		TrustService trustService, SearchManager searchManager, UploadManager uploadManager) {
+		TrustService trustService, SearchManager searchManager, UploadManager uploadManager,
+        ConnectionEstablisher establisher) {
 		this.eventBus = eventBus
 		this.manager = manager
 		this.settings = settings
@@ -49,7 +55,8 @@ class ConnectionAcceptor {
 		this.trustService = trustService
         this.searchManager = searchManager
         this.uploadManager = uploadManager
-		
+		this.establisher = establisher
+        
 		acceptorThread = Executors.newSingleThreadExecutor { r -> 
 			def rv = new Thread(r)
 			rv.setDaemon(true)
@@ -70,11 +77,13 @@ class ConnectionAcceptor {
 	}
 	
 	void stop() {
+        shutdown = true
 		acceptorThread.shutdownNow()
 		handshakerThreads.shutdownNow()
 	}
 	
 	private void acceptLoop() {
+        try {
 		while(true) {
 			def incoming = acceptor.accept()
 			log.info("accepted connection from ${incoming.destination.toBase32()}")
@@ -90,6 +99,11 @@ class ConnectionAcceptor {
 			}
 			handshakerThreads.execute({processIncoming(incoming)} as Runnable)
 		}
+        } catch (Exception e) {
+            log.log(Level.WARNING, "exception in accept loop",e)
+            if (!shutdown)
+                throw e
+        }
 	}
 	
 	private void processIncoming(Endpoint e) {
@@ -105,6 +119,9 @@ class ConnectionAcceptor {
 				case (byte)'G':
 					processGET(e)
 					break
+                case (byte)'H':
+                    processHashList(e)
+                    break
                 case (byte)'P':
                     processPOST(e)
                     break
@@ -140,7 +157,9 @@ class ConnectionAcceptor {
     }
 
 	private void handleIncoming(Endpoint e, boolean leaf) {
-		boolean accept = !manager.isConnected(e.destination) && (leaf ? manager.hasLeafSlots() : manager.hasPeerSlots())
+		boolean accept = !manager.isConnected(e.destination) && 
+            !establisher.isInProgress(e.destination) &&
+            (leaf ? manager.hasLeafSlots() : manager.hasPeerSlots())
 		if (accept) {
 			log.info("accepting connection, leaf:$leaf")
 			e.outputStream.write("OK".bytes)
@@ -173,9 +192,18 @@ class ConnectionAcceptor {
         dis.readFully(et)
         if (et != "ET ".getBytes(StandardCharsets.US_ASCII))
             throw new IOException("Invalid GET connection")
-        uploadManager.processEndpoint(e)
+        uploadManager.processGET(e)
 	}
     
+    private void processHashList(Endpoint e) {
+        byte[] ashList = new byte[8]
+        final DataInputStream dis = new DataInputStream(e.getInputStream())
+        dis.readFully(ashList)
+        if (ashList != "ASHLIST ".getBytes(StandardCharsets.US_ASCII))
+            throw new IOException("Invalid HASHLIST connection")
+        uploadManager.processHashList(e)
+    }
+    
     private void processPOST(final Endpoint e) throws IOException {
         byte [] ost = new byte[4]
         final DataInputStream dis = new DataInputStream(e.getInputStream())
@@ -199,13 +227,15 @@ class ConnectionAcceptor {
             if (sender.destination != e.getDestination())
                 throw new IOException("Sender destination mismatch expected $e.getDestination(), got $sender.destination")
             int nResults = dis.readUnsignedShort()
+            UIResultEvent[] results = new UIResultEvent[nResults]
             for (int i = 0; i < nResults; i++) {
                 int jsonSize = dis.readUnsignedShort()
                 byte [] payload = new byte[jsonSize]
                 dis.readFully(payload)
                 def json = slurper.parse(payload)
-                eventBus.publish(ResultsParser.parse(sender, resultsUUID, json))
+                results[i] = ResultsParser.parse(sender, resultsUUID, json)
             }
+            eventBus.publish(new UIResultBatchEvent(uuid: resultsUUID, results: results))
         } catch (IOException | UnexpectedResultsException | InvalidSearchResultException bad) {
             log.log(Level.WARNING, "failed to process POST", bad)
         } finally {

core/src/main/groovy/com/muwire/core/connection/ConnectionEstablisher.groovy

@@ -35,6 +35,8 @@ class ConnectionEstablisher {
 	
 	final Set inProgress = new ConcurrentHashSet()
 	
+    ConnectionEstablisher(){}
+    
 	ConnectionEstablisher(EventBus eventBus, I2PConnector i2pConnector, MuWireSettings settings,
 		ConnectionManager connectionManager, HostCache hostCache) {
 		this.eventBus = eventBus
@@ -176,4 +178,8 @@ class ConnectionEstablisher {
 			e.close()
 		}
 	}
+    
+    public boolean isInProgress(Destination d) {
+        inProgress.contains(d)
+    }
 }

core/src/main/groovy/com/muwire/core/connection/ConnectionManager.groovy

@@ -1,6 +1,7 @@
 package com.muwire.core.connection
 
 import com.muwire.core.EventBus
+import com.muwire.core.MuWireSettings
 import com.muwire.core.Persona
 import com.muwire.core.hostcache.HostCache
 import com.muwire.core.search.QueryEvent
@@ -19,13 +20,15 @@ abstract class ConnectionManager {
 	
 	protected final HostCache hostCache
     protected final Persona me
+    protected final MuWireSettings settings
 	
 	ConnectionManager() {}
 	
-	ConnectionManager(EventBus eventBus, Persona me, HostCache hostCache) {
+	ConnectionManager(EventBus eventBus, Persona me, HostCache hostCache, MuWireSettings settings) {
 		this.eventBus = eventBus
         this.me = me
 		this.hostCache = hostCache
+        this.settings = settings
 		this.timer = new Timer("connections-pinger",true)
 	}
 	

core/src/main/groovy/com/muwire/core/connection/Endpoint.groovy

@@ -1,6 +1,7 @@
 package com.muwire.core.connection
 
 import java.util.concurrent.atomic.AtomicBoolean
+import java.util.logging.Level
 
 import groovy.util.logging.Log
 import net.i2p.data.Destination
@@ -24,7 +25,7 @@ class Endpoint implements Closeable {
 	@Override
 	public void close() {
 		if (!closed.compareAndSet(false, true)) {
-			log.warning("Close loop detected for ${destination.toBase32()}", new Exception())
+			log.log(Level.WARNING,"Close loop detected for ${destination.toBase32()}", new Exception())
 			return
 		}
 		if (inputStream != null) {
@@ -34,7 +35,7 @@ class Endpoint implements Closeable {
 			try {outputStream.close()} catch (Exception ignore) {}
 		}
         if (toClose != null) {
-            try {toClose.close()} catch (Exception ignore) {}
+            try {toClose.reset()} catch (Exception ignore) {}
         }
 	}
 	

core/src/main/groovy/com/muwire/core/connection/LeafConnection.groovy

@@ -4,6 +4,7 @@ import java.io.InputStream
 import java.io.OutputStream
 
 import com.muwire.core.EventBus
+import com.muwire.core.MuWireSettings
 import com.muwire.core.hostcache.HostCache
 import com.muwire.core.trust.TrustService
 
@@ -16,8 +17,9 @@ import net.i2p.data.Destination
  */
 class LeafConnection extends Connection {
 
-	public LeafConnection(EventBus eventBus, Endpoint endpoint, HostCache hostCache, TrustService trustService) {
+	public LeafConnection(EventBus eventBus, Endpoint endpoint, HostCache hostCache, 
-		super(eventBus, endpoint, true, hostCache, trustService);
+        TrustService trustService, MuWireSettings settings) {
+		super(eventBus, endpoint, true, hostCache, trustService, settings);
 	}
 
 	@Override

core/src/main/groovy/com/muwire/core/connection/LeafConnectionManager.groovy

@@ -3,6 +3,7 @@ package com.muwire.core.connection
 import java.util.concurrent.ConcurrentHashMap
 
 import com.muwire.core.EventBus
+import com.muwire.core.MuWireSettings
 import com.muwire.core.Persona
 import com.muwire.core.hostcache.HostCache
 import com.muwire.core.search.QueryEvent
@@ -17,8 +18,9 @@ class LeafConnectionManager extends ConnectionManager {
 	
 	final Map<Destination, UltrapeerConnection> connections = new ConcurrentHashMap()
 	
-	public LeafConnectionManager(EventBus eventBus, Persona me, int maxConnections, HostCache hostCache) {
+	public LeafConnectionManager(EventBus eventBus, Persona me, int maxConnections, 
-		super(eventBus, me, hostCache)
+        HostCache hostCache, MuWireSettings settings) {
+		super(eventBus, me, hostCache, settings)
 		this.maxConnections = maxConnections
 	}
 	

core/src/main/groovy/com/muwire/core/connection/PeerConnection.groovy

@@ -4,6 +4,7 @@ import java.io.InputStream
 import java.io.OutputStream
 
 import com.muwire.core.EventBus
+import com.muwire.core.MuWireSettings
 import com.muwire.core.hostcache.HostCache
 import com.muwire.core.trust.TrustService
 import com.muwire.core.util.DataUtil
@@ -29,8 +30,9 @@ class PeerConnection extends Connection {
 	private final JsonSlurper slurper = new JsonSlurper()
 
 	public PeerConnection(EventBus eventBus, Endpoint endpoint,
-			boolean incoming, HostCache hostCache, TrustService trustService) {
+			boolean incoming, HostCache hostCache, TrustService trustService,
-		super(eventBus, endpoint, incoming, hostCache, trustService)
+            MuWireSettings settings) {
+		super(eventBus, endpoint, incoming, hostCache, trustService, settings)
 		this.dis = new DataInputStream(endpoint.inputStream)
 		this.dos = new DataOutputStream(endpoint.outputStream)
 	}

core/src/main/groovy/com/muwire/core/connection/UltrapeerConnectionManager.groovy

@@ -4,6 +4,7 @@ import java.util.Collection
 import java.util.concurrent.ConcurrentHashMap
 
 import com.muwire.core.EventBus
+import com.muwire.core.MuWireSettings
 import com.muwire.core.Persona
 import com.muwire.core.hostcache.HostCache
 import com.muwire.core.search.QueryEvent
@@ -17,15 +18,15 @@ class UltrapeerConnectionManager extends ConnectionManager {
 	
 	final int maxPeers, maxLeafs
     final TrustService trustService
-	
+    
 	final Map<Destination, PeerConnection> peerConnections = new ConcurrentHashMap()
 	final Map<Destination, LeafConnection> leafConnections = new ConcurrentHashMap()
     
 	UltrapeerConnectionManager() {}
 
 	public UltrapeerConnectionManager(EventBus eventBus, Persona me, int maxPeers, int maxLeafs, 
-        HostCache hostCache, TrustService trustService) {
+        HostCache hostCache, TrustService trustService, MuWireSettings settings) {
-		super(eventBus, me, hostCache)
+		super(eventBus, me, hostCache, settings)
 		this.maxPeers = maxPeers
 		this.maxLeafs = maxLeafs
         this.trustService = trustService
@@ -85,8 +86,8 @@ class UltrapeerConnectionManager extends ConnectionManager {
 			return
 		
 		Connection c = e.leaf ? 
-			new LeafConnection(eventBus, e.endpoint, hostCache, trustService) : 
+			new LeafConnection(eventBus, e.endpoint, hostCache, trustService, settings) : 
-			new PeerConnection(eventBus, e.endpoint, e.incoming, hostCache, trustService)
+			new PeerConnection(eventBus, e.endpoint, e.incoming, hostCache, trustService, settings)
 		def map = e.leaf ? leafConnections : peerConnections
 		map.put(e.endpoint.destination, c)
 		c.start()
@@ -103,8 +104,8 @@ class UltrapeerConnectionManager extends ConnectionManager {
     
     @Override
     void shutdown() {
-        peerConnections.each {k,v -> v.close() }
+        peerConnections.values().stream().parallel().forEach({v -> v.close()})
-        leafConnections.each {k,v -> v.close() }
+        leafConnections.values().stream().parallel().forEach({v -> v.close()})
         peerConnections.clear()
         leafConnections.clear()
     }

core/src/main/groovy/com/muwire/core/download/DownloadManager.groovy

@@ -1,28 +1,53 @@
 package com.muwire.core.download
 
 import com.muwire.core.connection.I2PConnector
+import com.muwire.core.files.FileDownloadedEvent
+import com.muwire.core.files.FileHasher
+import com.muwire.core.mesh.Mesh
+import com.muwire.core.mesh.MeshManager
+import com.muwire.core.trust.TrustLevel
+import com.muwire.core.trust.TrustService
+import com.muwire.core.util.DataUtil
 
+import groovy.json.JsonBuilder
+import groovy.json.JsonOutput
+import groovy.json.JsonSlurper
 import net.i2p.data.Base64
 import net.i2p.data.Destination
+import net.i2p.util.ConcurrentHashSet
 
 import com.muwire.core.EventBus
+import com.muwire.core.InfoHash
+import com.muwire.core.MuWireSettings
 import com.muwire.core.Persona
+import com.muwire.core.UILoadedEvent
 
+import java.util.concurrent.ConcurrentHashMap
 import java.util.concurrent.Executor
 import java.util.concurrent.Executors
 
 public class DownloadManager {
     
     private final EventBus eventBus
+    private final TrustService trustService
+    private final MeshManager meshManager
+    private final MuWireSettings muSettings
     private final I2PConnector connector
     private final Executor executor
-    private final File incompletes
+    private final File incompletes, home
     private final Persona me
     
-    public DownloadManager(EventBus eventBus, I2PConnector connector, File incompletes, Persona me) {
+    private final Map<InfoHash, Downloader> downloaders = new ConcurrentHashMap<>()
+    
+    public DownloadManager(EventBus eventBus, TrustService trustService, MeshManager meshManager, MuWireSettings muSettings,
+        I2PConnector connector, File home, Persona me) {
         this.eventBus = eventBus
+        this.trustService = trustService
+        this.meshManager = meshManager
+        this.muSettings = muSettings
         this.connector = connector
-        this.incompletes = incompletes
+        this.incompletes = new File(home,"incompletes")
+        this.home = home
         this.me = me
         
         incompletes.mkdir()
@@ -46,15 +71,131 @@ public class DownloadManager {
         e.result.each { 
             destinations.add(it.sender.destination)
         }
+        destinations.addAll(e.sources)
+        destinations.remove(me.destination)
         
-        def downloader = new Downloader(this, me, e.target, size,
+        Pieces pieces = getPieces(infohash, size, pieceSize)
+        
+        def downloader = new Downloader(eventBus, this, me, e.target, size,
             infohash, pieceSize, connector, destinations,
-            incompletes)
+            incompletes, pieces)
+        downloaders.put(infohash, downloader)
+        persistDownloaders()
         executor.execute({downloader.download()} as Runnable)
         eventBus.publish(new DownloadStartedEvent(downloader : downloader))
     }
     
+    public void onUIDownloadCancelledEvent(UIDownloadCancelledEvent e) {
+        downloaders.remove(e.downloader.infoHash)
+        persistDownloaders()
+    }
+    
+    public void onUIDownloadPausedEvent(UIDownloadPausedEvent e) {
+        persistDownloaders()
+    }
+    
+    public void onUIDownloadResumedEvent(UIDownloadResumedEvent e) {
+        persistDownloaders()
+    }
+    
     void resume(Downloader downloader) {
         executor.execute({downloader.download() as Runnable})
     }
+    
+    void onUILoadedEvent(UILoadedEvent e) {
+        File downloadsFile = new File(home, "downloads.json")
+        if (!downloadsFile.exists())
+            return
+        def slurper = new JsonSlurper()
+        downloadsFile.eachLine {
+            def json = slurper.parseText(it)
+            File file = new File(DataUtil.readi18nString(Base64.decode(json.file)))
+            def destinations = new HashSet<>()
+            json.destinations.each { destination -> 
+                destinations.add new Destination(destination)
+            }
+            InfoHash infoHash
+            if (json.hashList != null) {
+                byte[] hashList = Base64.decode(json.hashList)
+                infoHash = InfoHash.fromHashList(hashList)
+            } else {
+                byte [] root = Base64.decode(json.hashRoot)
+                infoHash = new InfoHash(root)
+            }
+            
+            Pieces pieces = getPieces(infoHash, (long)json.length, json.pieceSizePow2)
+            
+            def downloader = new Downloader(eventBus, this, me, file, (long)json.length,
+                infoHash, json.pieceSizePow2, connector, destinations, incompletes, pieces)
+            if (json.paused != null)
+                downloader.paused = json.paused
+            downloaders.put(infoHash, downloader)
+            downloader.readPieces()
+            if (!downloader.paused)
+                downloader.download()
+            eventBus.publish(new DownloadStartedEvent(downloader : downloader))
+        }
+    }
+    
+    private Pieces getPieces(InfoHash infoHash, long length, int pieceSizePow2) {
+        int pieceSize = 0x1 << pieceSizePow2
+        int nPieces = (int)(length / pieceSize)
+        if (length % pieceSize != 0)
+            nPieces++
+        Mesh mesh = meshManager.getOrCreate(infoHash, nPieces)
+        mesh.pieces
+    }
+    
+    void onSourceDiscoveredEvent(SourceDiscoveredEvent e) {
+        Downloader downloader = downloaders.get(e.infoHash)
+        if (downloader == null)
+            return
+        boolean ok = false
+        switch(trustService.getLevel(e.source.destination)) {
+            case TrustLevel.TRUSTED: ok = true; break
+            case TrustLevel.NEUTRAL: ok = muSettings.allowUntrusted; break
+            case TrustLevel.DISTRUSTED: ok = false; break
+        }
+        
+        if (ok)
+            downloader.addSource(e.source.destination)
+    }
+    
+    void onFileDownloadedEvent(FileDownloadedEvent e) {
+        downloaders.remove(e.downloader.infoHash)
+        persistDownloaders()
+    }
+    
+    private void persistDownloaders() {
+        File downloadsFile = new File(home,"downloads.json")
+        downloadsFile.withPrintWriter { writer -> 
+            downloaders.values().each { downloader ->
+                if (!downloader.cancelled) {
+                    def json = [:]
+                    json.file = Base64.encode(DataUtil.encodei18nString(downloader.file.getAbsolutePath()))
+                    json.length = downloader.length
+                    json.pieceSizePow2 = downloader.pieceSizePow2
+                    def destinations = []
+                    downloader.destinations.each {
+                        destinations << it.toBase64()
+                    }
+                    json.destinations = destinations
+                    
+                    InfoHash infoHash = downloader.getInfoHash()
+                    if (infoHash.hashList != null)
+                        json.hashList = Base64.encode(infoHash.hashList)
+                    else
+                        json.hashRoot = Base64.encode(infoHash.getRoot())
+                        
+                    json.paused = downloader.paused
+                    writer.println(JsonOutput.toJson(json))
+                }
+            }
+        }
+    }
+    
+    public void shutdown() {
+        downloaders.values().each { it.stop() }
+        Downloader.executorService.shutdownNow()
+    }
 }

core/src/main/groovy/com/muwire/core/download/DownloadSession.groovy

@@ -3,8 +3,12 @@ package com.muwire.core.download;
 import net.i2p.data.Base64
 
 import com.muwire.core.Constants
+import com.muwire.core.EventBus
 import com.muwire.core.InfoHash
+import com.muwire.core.Persona
 import com.muwire.core.connection.Endpoint
+import com.muwire.core.util.DataUtil
+
 import static com.muwire.core.util.DataUtil.readTillRN
 
 import groovy.util.logging.Log
@@ -16,36 +20,40 @@ import java.nio.file.Files
 import java.nio.file.StandardOpenOption
 import java.security.MessageDigest
 import java.security.NoSuchAlgorithmException
+import java.util.logging.Level
 
 @Log
 class DownloadSession {
     
     private static int SAMPLES = 10
     
+    private final EventBus eventBus
     private final String meB64
-    private final Pieces downloaded, claimed
+    private final Pieces pieces
     private final InfoHash infoHash
     private final Endpoint endpoint
     private final File file
     private final int pieceSize
     private final long fileLength
+    private final Set<Integer> available
     private final MessageDigest digest
     
-    private final ArrayDeque<Long> timestamps = new ArrayDeque<>(SAMPLES)
+    private final LinkedList<Long> timestamps = new LinkedList<>()
-    private final ArrayDeque<Integer> reads = new ArrayDeque<>(SAMPLES)
+    private final LinkedList<Integer> reads = new LinkedList<>()
     
     private ByteBuffer mapped
     
-    DownloadSession(String meB64, Pieces downloaded, Pieces claimed, InfoHash infoHash, Endpoint endpoint, File file, 
+    DownloadSession(EventBus eventBus, String meB64, Pieces pieces, InfoHash infoHash, Endpoint endpoint, File file, 
-        int pieceSize, long fileLength) {
+        int pieceSize, long fileLength, Set<Integer> available) {
+        this.eventBus = eventBus
         this.meB64 = meB64
-        this.downloaded = downloaded
+        this.pieces = pieces
-        this.claimed = claimed
         this.endpoint = endpoint
         this.infoHash = infoHash
         this.file = file
         this.pieceSize = pieceSize
         this.fileLength = fileLength
+        this.available = available
         try {
             digest = MessageDigest.getInstance("SHA-256")
         } catch (NoSuchAlgorithmException impossible) {
@@ -64,19 +72,14 @@ class DownloadSession {
         InputStream is = endpoint.getInputStream()
         
         int piece
-        while(true) {
+        if (available.isEmpty())
-            piece = downloaded.getRandomPiece()
+            piece = pieces.claim()
-            if (claimed.isMarked(piece)) {
+        else
-                if (downloaded.donePieces() + claimed.donePieces() == downloaded.nPieces) {
+            piece = pieces.claim(available)
-                    log.info("all pieces claimed")
+        if (piece == -1)
-                    return false
+            return false
-                }
+        boolean unclaim = true
-                continue
+            
-            }
-            break
-        }
-        claimed.markDownloaded(piece)
-        
         log.info("will download piece $piece")
         
         long start = piece * pieceSize
@@ -85,91 +88,129 @@ class DownloadSession {
         
         String root = Base64.encode(infoHash.getRoot())
                 
-        FileChannel channel
         try {
             os.write("GET $root\r\n".getBytes(StandardCharsets.US_ASCII))
             os.write("Range: $start-$end\r\n".getBytes(StandardCharsets.US_ASCII))
-            os.write("X-Persona: $meB64\r\n\r\n".getBytes(StandardCharsets.US_ASCII))
+            os.write("X-Persona: $meB64\r\n".getBytes(StandardCharsets.US_ASCII))
+            String xHave = DataUtil.encodeXHave(pieces.getDownloaded(), pieces.nPieces)
+            os.write("X-Have: $xHave\r\n\r\n".getBytes(StandardCharsets.US_ASCII)) 
             os.flush()
-            String code = readTillRN(is)
+            String codeString = readTillRN(is)
-            if (code.startsWith("404 ")) {
+            int space = codeString.indexOf(' ')
+            if (space > 0)
+                codeString = codeString.substring(0, space)
+            
+            int code = Integer.parseInt(codeString.trim())
+            
+            if (code == 404) {
                 log.warning("file not found")
                 endpoint.close()
-                return
+                return false
             }
                 
-            if (code.startsWith("416 ")) {
+            if (!(code == 200 || code == 416)) {
-                log.warning("range $start-$end cannot be satisfied")
-                return // leave endpoint open
-            }
-            
-            if (!code.startsWith("200 ")) {
                 log.warning("unknown code $code")
                 endpoint.close()
-                return
+                return false
             }
 
             // parse all headers
-            Set<String> headers = new HashSet<>()
+            Map<String,String> headers = new HashMap<>()
             String header
-            while((header = readTillRN(is)) != "" && headers.size() < Constants.MAX_HEADERS) 
+            while((header = readTillRN(is)) != "" && headers.size() < Constants.MAX_HEADERS) {
-                headers.add(header)
+                int colon = header.indexOf(':')
-
+                if (colon == -1 || colon == header.length() - 1)
-            long receivedStart = -1
+                    throw new IOException("invalid header $header")
-            long receivedEnd = -1
+                String key = header.substring(0, colon)
-            for (String receivedHeader : headers) {
+                String value = header.substring(colon + 1) 
-                def group = (receivedHeader =~ /^Content-Range: (\d+)-(\d+)$/)
+                headers[key] = value.trim()
-                if (group.size() != 1) {
-                    log.info("ignoring header $receivedHeader")
-                    continue
-                } 
-
-                receivedStart = Long.parseLong(group[0][1])
-                receivedEnd = Long.parseLong(group[0][2])
             }
             
+            // prase X-Alt if present
+            if (headers.containsKey("X-Alt")) {
+                headers["X-Alt"].split(",").each {
+                    if (it.length() > 0) {
+                        byte [] raw = Base64.decode(it)
+                        Persona source = new Persona(new ByteArrayInputStream(raw))
+                        eventBus.publish(new SourceDiscoveredEvent(infoHash : infoHash, source : source))
+                    }
+                }
+            }
+
+            // parse X-Have if present
+            if (headers.containsKey("X-Have")) { 
+                DataUtil.decodeXHave(headers["X-Have"]).each {
+                    available.add(it)
+                }
+                if (!available.contains(piece))
+                    return true // try again next time
+            } else {
+                if (code != 200)
+                    throw new IOException("Code $code but no X-Have")
+                available.clear()
+            }
+            
+            if (code != 200)
+                return true
+            
+            String range = headers["Content-Range"]
+            if (range == null) 
+                throw new IOException("Code 200 but no Content-Range")
+            
+            def group = (range =~ /^(\d+)-(\d+)$/)
+            if (group.size() != 1) 
+                throw new IOException("invalid Content-Range header $range")
+
+            long receivedStart = Long.parseLong(group[0][1])
+            long receivedEnd = Long.parseLong(group[0][2])
+
             if (receivedStart != start || receivedEnd != end) {
                 log.warning("We don't support mismatching ranges yet")
                 endpoint.close()
-                return
+                return false
             }
             
             // start the download
-            channel = Files.newByteChannel(file.toPath(), EnumSet.of(StandardOpenOption.READ, StandardOpenOption.WRITE, 
+            FileChannel channel
-                StandardOpenOption.SPARSE, StandardOpenOption.CREATE)) // TODO: double-check, maybe CREATE_NEW
+            try {
-            mapped = channel.map(FileChannel.MapMode.READ_WRITE, start, end - start + 1)
+                channel = Files.newByteChannel(file.toPath(), EnumSet.of(StandardOpenOption.READ, StandardOpenOption.WRITE,
-            
+                        StandardOpenOption.SPARSE, StandardOpenOption.CREATE)) // TODO: double-check, maybe CREATE_NEW
-            byte[] tmp = new byte[0x1 << 13]
+                mapped = channel.map(FileChannel.MapMode.READ_WRITE, start, end - start + 1)
-            while(mapped.hasRemaining()) {
+
-                if (mapped.remaining() < tmp.length)
+                byte[] tmp = new byte[0x1 << 13]
-                    tmp = new byte[mapped.remaining()]
+                while(mapped.hasRemaining()) {
-                int read = is.read(tmp)
+                    if (mapped.remaining() < tmp.length)
-                if (read == -1)
+                        tmp = new byte[mapped.remaining()]
-                    throw new IOException()
+                    int read = is.read(tmp)
-                synchronized(this) {
+                    if (read == -1)
-                    mapped.put(tmp, 0, read)
+                        throw new IOException()
-                    
+                    synchronized(this) {
-                    if (timestamps.size() == SAMPLES) {
+                        mapped.put(tmp, 0, read)
-                        timestamps.removeFirst()
+
-                        reads.removeFirst()
+                        if (timestamps.size() == SAMPLES) {
+                            timestamps.removeFirst()
+                            reads.removeFirst()
+                        }
+                        timestamps.addLast(System.currentTimeMillis())
+                        reads.addLast(read)
                     }
-                    timestamps.addLast(System.currentTimeMillis())
-                    reads.addLast(read)
                 }
+
+                mapped.clear()
+                digest.update(mapped)
+                byte [] hash = digest.digest()
+                byte [] expected = new byte[32]
+                System.arraycopy(infoHash.getHashList(), piece * 32, expected, 0, 32)
+                if (hash != expected)
+                    throw new BadHashException()
+            } finally {
+                try { channel?.close() } catch (IOException ignore) {}
             }
-            
+            pieces.markDownloaded(piece)
-            mapped.clear()
+            unclaim = false
-            digest.update(mapped)
-            byte [] hash = digest.digest()
-            byte [] expected = new byte[32]
-            System.arraycopy(infoHash.getHashList(), piece * 32, expected, 0, 32)    
-            if (hash != expected) 
-                throw new BadHashException()
-            
-            downloaded.markDownloaded(piece)        
         } finally {
-            claimed.clear(piece)
+            if (unclaim)
-            try { channel?.close() } catch (IOException ignore) {}
+                pieces.unclaim(piece)
         }
         return true
     }
@@ -183,9 +224,22 @@ class DownloadSession {
     synchronized int speed() {
         if (timestamps.size() < SAMPLES)
             return 0
-        long interval = timestamps.last - timestamps.first
         int totalRead = 0
-        reads.each { totalRead += it }
+        int idx = 0
+        final long now = System.currentTimeMillis()
+        
+        while(idx < SAMPLES && timestamps.get(idx) < now - 1000)
+            idx++
+        if (idx == SAMPLES)
+            return 0
+        if (idx == SAMPLES - 1)
+            return reads[idx]
+            
+        long interval = timestamps.last - timestamps[idx]
+        if (interval == 0) 
+            interval = 1
+        for (int i = idx; i < SAMPLES; i++)
+            totalRead += reads[idx]
         (int)(totalRead * 1000.0 / interval)
     }
 }

core/src/main/groovy/com/muwire/core/download/Downloader.groovy

@@ -4,21 +4,29 @@ import com.muwire.core.InfoHash
 import com.muwire.core.Persona
 import com.muwire.core.connection.Endpoint
 
+import java.nio.file.AtomicMoveNotSupportedException
+import java.nio.file.Files
+import java.nio.file.StandardCopyOption
 import java.util.concurrent.ConcurrentHashMap
 import java.util.concurrent.ExecutorService
 import java.util.concurrent.Executors
+import java.util.concurrent.atomic.AtomicBoolean
 import java.util.logging.Level
 
 import com.muwire.core.Constants
+import com.muwire.core.DownloadedFile
+import com.muwire.core.EventBus
 import com.muwire.core.connection.I2PConnector
+import com.muwire.core.files.FileDownloadedEvent
 
 import groovy.util.logging.Log
 import net.i2p.data.Destination
+import net.i2p.util.ConcurrentHashSet
 
 @Log
 public class Downloader {
-    public enum DownloadState { CONNECTING, DOWNLOADING, FAILED, CANCELLED, FINISHED }
+    public enum DownloadState { CONNECTING, HASHLIST, DOWNLOADING, FAILED, CANCELLED, PAUSED, FINISHED }
-    private enum WorkerState { CONNECTING, DOWNLOADING, FINISHED}
+    private enum WorkerState { CONNECTING, HASHLIST, DOWNLOADING, FINISHED}
     
     private static final ExecutorService executorService = Executors.newCachedThreadPool({r ->
         Thread rv = new Thread(r)
@@ -27,25 +35,33 @@ public class Downloader {
         rv
     })
 
+    private final EventBus eventBus
     private final DownloadManager downloadManager 
     private final Persona me   
     private final File file
-    private final Pieces downloaded, claimed
+    private final Pieces pieces
     private final long length
-    private final InfoHash infoHash
+    private InfoHash infoHash
     private final int pieceSize
     private final I2PConnector connector
     private final Set<Destination> destinations
     private final int nPieces
     private final File piecesFile
+    private final File incompleteFile
+    final int pieceSizePow2
     private final Map<Destination, DownloadWorker> activeWorkers = new ConcurrentHashMap<>()
+    private final Set<Destination> successfulDestinations = new ConcurrentHashSet<>()
     
     
-    private volatile boolean cancelled
+    private volatile boolean cancelled, paused
+    private final AtomicBoolean eventFired = new AtomicBoolean()
+    private boolean piecesFileClosed
 
-    public Downloader(DownloadManager downloadManager, Persona me, File file, long length, InfoHash infoHash, 
+    public Downloader(EventBus eventBus, DownloadManager downloadManager, 
+        Persona me, File file, long length, InfoHash infoHash, 
         int pieceSizePow2, I2PConnector connector, Set<Destination> destinations,
-        File incompletes) {
+        File incompletes, Pieces pieces) {
+        this.eventBus = eventBus
         this.me = me
         this.downloadManager = downloadManager
         this.file = file
@@ -54,17 +70,19 @@ public class Downloader {
         this.connector = connector
         this.destinations = destinations
         this.piecesFile = new File(incompletes, file.getName()+".pieces")
+        this.incompleteFile = new File(incompletes, file.getName()+".part")
+        this.pieceSizePow2 = pieceSizePow2
         this.pieceSize = 1 << pieceSizePow2
-        
+        this.pieces = pieces
-        int nPieces
+        this.nPieces = pieces.nPieces
-        if (length % pieceSize == 0)
+    }
-            nPieces = length / pieceSize
+    
-        else
+    public synchronized InfoHash getInfoHash() {
-            nPieces = length / pieceSize + 1
+        infoHash
-        this.nPieces = nPieces
+    }
-        
+    
-        downloaded = new Pieces(nPieces, Constants.DOWNLOAD_SEQUENTIAL_RATIO)
+    private synchronized void setInfoHash(InfoHash infoHash) {
-        claimed = new Pieces(nPieces)
+        this.infoHash = infoHash
     }
     
     void download() {
@@ -81,22 +99,26 @@ public class Downloader {
     void readPieces() {
         if (!piecesFile.exists())
             return
-        piecesFile.withReader { 
+        piecesFile.eachLine { 
-            int piece = Integer.parseInt(it.readLine())
+            int piece = Integer.parseInt(it)
-            downloaded.markDownloaded(piece)
+            pieces.markDownloaded(piece)
         }
     }
     
     void writePieces() {
-        piecesFile.withPrintWriter { writer ->
+        synchronized(piecesFile) {
-            downloaded.getDownloaded().each { piece -> 
+            if (piecesFileClosed)
-                writer.println(piece)
+                return
+            piecesFile.withPrintWriter { writer ->
+                pieces.getDownloaded().each { piece ->
+                    writer.println(piece)
+                }
             }
         }
     }
     
     public long donePieces() {
-        downloaded.donePieces()
+        pieces.donePieces()
     }
     
     
@@ -104,7 +126,8 @@ public class Downloader {
         int total = 0
         if (getCurrentState() == DownloadState.DOWNLOADING) {
             activeWorkers.values().each {
-                total += it.speed()
+                if (it.currentState == WorkerState.DOWNLOADING)
+                    total += it.speed()
             }
         }
         total
@@ -113,12 +136,15 @@ public class Downloader {
     public DownloadState getCurrentState() {
         if (cancelled)
             return DownloadState.CANCELLED
+        if (paused)
+            return DownloadState.PAUSED
+            
         boolean allFinished = true
         activeWorkers.values().each { 
             allFinished &= it.currentState == WorkerState.FINISHED
         }
         if (allFinished) {
-            if (downloaded.isComplete())
+            if (pieces.isComplete())
                 return DownloadState.FINISHED
             return DownloadState.FAILED
         }
@@ -135,11 +161,37 @@ public class Downloader {
         if (oneDownloading)
             return DownloadState.DOWNLOADING
         
+        // at least one is requesting hashlist
+        boolean oneHashlist = false
+        activeWorkers.values().each { 
+            if (it.currentState == WorkerState.HASHLIST) {
+                oneHashlist = true
+                return
+            }
+        }
+        if (oneHashlist)
+            return DownloadState.HASHLIST
+            
         return DownloadState.CONNECTING
     }
     
     public void cancel() {
         cancelled = true
+        stop()
+        synchronized(piecesFile) {
+            piecesFileClosed = true
+            piecesFile.delete()
+        }
+        incompleteFile.delete()
+        pieces.clearAll()
+    }
+    
+    public void pause() {
+        paused = true
+        stop()
+    }
+    
+    void stop() {
         activeWorkers.values().each { 
             it.cancel()
         }
@@ -155,7 +207,30 @@ public class Downloader {
     }
     
     public void resume() {
-        downloadManager.resume(this)
+        paused = false
+        readPieces()
+        destinations.each { destination ->
+            def worker = activeWorkers.get(destination)
+            if (worker != null) {
+                if (worker.currentState == WorkerState.FINISHED) {
+                    def newWorker = new DownloadWorker(destination)
+                    activeWorkers.put(destination, newWorker)
+                    executorService.submit(newWorker)
+                }
+            } else {
+                worker = new DownloadWorker(destination)
+                activeWorkers.put(destination, worker)
+                executorService.submit(worker)
+            }
+        }
+    }
+    
+    void addSource(Destination d) {
+        if (activeWorkers.containsKey(d))
+            return
+        DownloadWorker newWorker = new DownloadWorker(d)
+        activeWorkers.put(d, newWorker)
+        executorService.submit(newWorker)
     }
     
     class DownloadWorker implements Runnable {
@@ -164,6 +239,7 @@ public class Downloader {
         private volatile Thread downloadThread
         private Endpoint endpoint
         private volatile DownloadSession currentSession
+        private final Set<Integer> available = new HashSet<>()
                 
         DownloadWorker(Destination destination) {
             this.destination = destination
@@ -175,19 +251,44 @@ public class Downloader {
             Endpoint endpoint = null
             try {
                 endpoint = connector.connect(destination)
+                while(getInfoHash().hashList == null) {
+                    currentState = WorkerState.HASHLIST
+                    HashListSession session = new HashListSession(me.toBase64(), infoHash, endpoint)
+                    InfoHash received = session.request()
+                    setInfoHash(received)
+                }
                 currentState = WorkerState.DOWNLOADING
                 boolean requestPerformed
-                while(!downloaded.isComplete()) {
+                while(!pieces.isComplete()) {
-                    currentSession = new DownloadSession(me.toBase64(), downloaded, claimed, infoHash, endpoint, file, pieceSize, length)
+                    currentSession = new DownloadSession(eventBus, me.toBase64(), pieces, getInfoHash(), 
+                        endpoint, incompleteFile, pieceSize, length, available)
                     requestPerformed = currentSession.request()
                     if (!requestPerformed)
                         break
+                    successfulDestinations.add(endpoint.destination)
                     writePieces()
                 }
             } catch (Exception bad) {
                 log.log(Level.WARNING,"Exception while downloading",bad)
             } finally {
                 currentState = WorkerState.FINISHED
+                if (pieces.isComplete() && eventFired.compareAndSet(false, true)) {
+                    synchronized(piecesFile) {
+                        piecesFileClosed = true
+                        piecesFile.delete()
+                    }
+                    try {
+                        Files.move(incompleteFile.toPath(), file.toPath(), StandardCopyOption.ATOMIC_MOVE)
+                    } catch (AtomicMoveNotSupportedException e) {
+                        Files.copy(incompleteFile.toPath(), file.toPath(), StandardCopyOption.REPLACE_EXISTING)
+                        incompleteFile.delete()
+                    }
+                    eventBus.publish(
+                        new FileDownloadedEvent(
+                            downloadedFile : new DownloadedFile(file, getInfoHash(), pieceSizePow2, successfulDestinations),
+                        downloader : Downloader.this))
+                            
+                } 
                 endpoint?.close()
             }
         }

core/src/main/groovy/com/muwire/core/download/HashListSession.groovy

@@ -0,0 +1,82 @@
+package com.muwire.core.download
+
+import java.nio.ByteBuffer
+import java.nio.charset.StandardCharsets
+import java.security.MessageDigest
+import java.security.NoSuchAlgorithmException
+
+import com.muwire.core.Constants
+import com.muwire.core.InfoHash
+import com.muwire.core.connection.Endpoint
+
+import groovy.util.logging.Log
+
+import static com.muwire.core.util.DataUtil.readTillRN
+
+import net.i2p.data.Base64
+
+@Log
+class HashListSession {
+    private final String meB64
+    private final InfoHash infoHash
+    private final Endpoint endpoint
+    
+    HashListSession(String meB64, InfoHash infoHash, Endpoint endpoint) {
+        this.meB64 = meB64
+        this.infoHash = infoHash
+        this.endpoint = endpoint
+    }
+    
+    InfoHash request() throws IOException {
+        InputStream is = endpoint.getInputStream()
+        OutputStream os = endpoint.getOutputStream()
+        
+        String root = Base64.encode(infoHash.getRoot())
+        os.write("HASHLIST $root\r\n".getBytes(StandardCharsets.US_ASCII))
+        os.write("X-Persona: $meB64\r\n\r\n".getBytes(StandardCharsets.US_ASCII))
+        os.flush()
+        
+        String code = readTillRN(is)
+        if (!code.startsWith("200"))
+            throw new IOException("unknown code $code")
+        
+        // parse all headers
+        Set<String> headers = new HashSet<>()
+        String header
+        while((header = readTillRN(is)) != "" && headers.size() < Constants.MAX_HEADERS)
+            headers.add(header)
+            
+        long receivedStart = -1
+        long receivedEnd = -1
+        for (String receivedHeader : headers) {
+            def group = (receivedHeader =~ /^Content-Range: (\d+)-(\d+)$/)
+            if (group.size() != 1) {
+                log.info("ignoring header $receivedHeader")
+                continue
+            }
+
+            receivedStart = Long.parseLong(group[0][1])
+            receivedEnd = Long.parseLong(group[0][2])
+        }
+        
+        if (receivedStart != 0)
+            throw new IOException("hashlist started at $receivedStart")
+        
+        byte[] hashList = new byte[receivedEnd]
+        ByteBuffer hashListBuf = ByteBuffer.wrap(hashList)
+        byte[] tmp = new byte[0x1 << 13]
+        while(hashListBuf.hasRemaining()) {
+            if (hashListBuf.remaining() > tmp.length)
+                tmp = new byte[hashListBuf.remaining()]
+            int read = is.read(tmp)
+            if (read == -1)
+                throw new IOException()
+            hashListBuf.put(tmp, 0, read)
+        }
+        
+        InfoHash received = InfoHash.fromHashList(hashList)
+        if (received.getRoot() != infoHash.getRoot())
+            throw new IOException("fetched list doesn't match root")
+        received
+    }
+}

core/src/main/groovy/com/muwire/core/download/Pieces.groovy

@@ -1,7 +1,7 @@
 package com.muwire.core.download
 
 class Pieces {
-    private final BitSet bitSet
+    private final BitSet done, claimed
     private final int nPieces
     private final float ratio
     private final Random random = new Random()
@@ -13,52 +13,74 @@ class Pieces {
     Pieces(int nPieces, float ratio) {
         this.nPieces = nPieces
         this.ratio = ratio
-        bitSet = new BitSet(nPieces)
+        done = new BitSet(nPieces)
+        claimed = new BitSet(nPieces)
     }
     
-    synchronized int getRandomPiece() {
+    synchronized int claim() {
-        int cardinality = bitSet.cardinality()
+        int claimedCardinality = claimed.cardinality()
-        if (cardinality == nPieces)
+        if (claimedCardinality == nPieces)
             return -1
         
         // if fuller than ratio just do sequential
-        if ( (1.0f * cardinality) / nPieces > ratio) {
+        if ( (1.0f * claimedCardinality) / nPieces > ratio) {
-            return bitSet.nextClearBit(0)
+            int rv = claimed.nextClearBit(0)
+            claimed.set(rv)
+            return rv
         }
         
         while(true) {
             int start = random.nextInt(nPieces)
-            if (bitSet.get(start))
+            if (claimed.get(start))
                 continue
+            claimed.set(start)
             return start
         }
     }
     
-    def getDownloaded() {
+    synchronized int claim(Set<Integer> available) {
+        for (int i = claimed.nextSetBit(0); i >= 0; i = claimed.nextSetBit(i+1))
+            available.remove(i)
+        if (available.isEmpty())
+            return -1
+        List<Integer> toList = available.toList()
+        Collections.shuffle(toList)
+        int rv = toList[0]
+        claimed.set(rv)
+        rv
+    }
+    
+    synchronized def getDownloaded() {
         def rv = []
-        for (int i = bitSet.nextSetBit(0); i >= 0; i = bitSet.nextSetBit(i+1)) {
+        for (int i = done.nextSetBit(0); i >= 0; i = done.nextSetBit(i+1)) {
             rv << i
         }
         rv
     }
     
     synchronized void markDownloaded(int piece) {
-        bitSet.set(piece)
+        done.set(piece)
+        claimed.set(piece)
     }
     
-    synchronized void clear(int piece) {
+    synchronized void unclaim(int piece) {
-        bitSet.clear(piece)
+        claimed.clear(piece)
     }
     
     synchronized boolean isComplete() {
-        bitSet.cardinality() == nPieces
+        done.cardinality() == nPieces
-    }
-    
-    synchronized boolean isMarked(int piece) {
-        bitSet.get(piece)
     }
     
     synchronized int donePieces() {
-        bitSet.cardinality()
+        done.cardinality()
+    }
+    
+    synchronized boolean isDownloaded(int piece) {
+        done.get(piece)
+    }
+    
+    synchronized void clearAll() {
+        done.clear()
+        claimed.clear()
     }
 }

core/src/main/groovy/com/muwire/core/download/SourceDiscoveredEvent.groovy

@@ -0,0 +1,10 @@
+package com.muwire.core.download
+
+import com.muwire.core.Event
+import com.muwire.core.InfoHash
+import com.muwire.core.Persona
+
+class SourceDiscoveredEvent extends Event {
+    InfoHash infoHash
+    Persona source
+}

core/src/main/groovy/com/muwire/core/download/UIDownloadCancelledEvent.groovy

@@ -0,0 +1,7 @@
+package com.muwire.core.download
+
+import com.muwire.core.Event
+
+class UIDownloadCancelledEvent extends Event {
+    Downloader downloader
+}

core/src/main/groovy/com/muwire/core/download/UIDownloadEvent.groovy

@@ -3,8 +3,11 @@ package com.muwire.core.download
 import com.muwire.core.Event
 import com.muwire.core.search.UIResultEvent
 
+import net.i2p.data.Destination
+
 class UIDownloadEvent extends Event {
     
     UIResultEvent[] result
+    Set<Destination> sources
     File target
 }

core/src/main/groovy/com/muwire/core/download/UIDownloadPausedEvent.groovy

@@ -0,0 +1,6 @@
+package com.muwire.core.download
+
+import com.muwire.core.Event
+
+class UIDownloadPausedEvent extends Event {
+}

core/src/main/groovy/com/muwire/core/download/UIDownloadResumedEvent.groovy

@@ -0,0 +1,6 @@
+package com.muwire.core.download
+
+import com.muwire.core.Event
+
+class UIDownloadResumedEvent extends Event {
+}

core/src/main/groovy/com/muwire/core/files/AllFilesLoadedEvent.groovy

@@ -0,0 +1,6 @@
+package com.muwire.core.files
+
+import com.muwire.core.Event
+
+class AllFilesLoadedEvent extends Event {
+}

core/src/main/groovy/com/muwire/core/files/DirectoryWatcher.groovy

@@ -0,0 +1,141 @@
+package com.muwire.core.files
+
+import java.nio.file.FileSystem
+import java.nio.file.FileSystems
+import java.nio.file.Path
+import java.nio.file.Paths
+import static java.nio.file.StandardWatchEventKinds.*
+
+import java.nio.file.ClosedWatchServiceException
+import java.nio.file.WatchEvent
+import java.nio.file.WatchKey
+import java.nio.file.WatchService
+import java.util.concurrent.ConcurrentHashMap
+
+import com.muwire.core.EventBus
+import com.muwire.core.SharedFile
+
+import groovy.util.logging.Log
+import net.i2p.util.SystemVersion
+
+@Log
+class DirectoryWatcher {
+    
+    private static final long WAIT_TIME = 1000
+    
+    private static final WatchEvent.Kind[] kinds
+    static {
+        if (SystemVersion.isMac())
+            kinds = [ENTRY_MODIFY, ENTRY_DELETE]
+        else
+            kinds = [ENTRY_CREATE, ENTRY_MODIFY, ENTRY_DELETE]
+    }
+    
+    private final EventBus eventBus
+    private final FileManager fileManager
+    private final Thread watcherThread, publisherThread
+    private final Map<File, Long> waitingFiles = new ConcurrentHashMap<>()
+    private WatchService watchService
+    private volatile boolean shutdown
+    
+    DirectoryWatcher(EventBus eventBus, FileManager fileManager) {
+        this.eventBus = eventBus
+        this.fileManager = fileManager
+        this.watcherThread = new Thread({watch() } as Runnable, "directory-watcher")
+        watcherThread.setDaemon(true)
+        this.publisherThread = new Thread({publish()} as Runnable, "watched-files-publisher")
+        publisherThread.setDaemon(true)
+    }
+    
+    void start() {
+        watchService = FileSystems.getDefault().newWatchService()
+        watcherThread.start()
+        publisherThread.start()
+    }
+    
+    void stop() {
+        shutdown = true
+        watcherThread.interrupt()
+        publisherThread.interrupt()
+        watchService.close()
+    }
+    
+    void onFileSharedEvent(FileSharedEvent e) {
+        if (!e.file.isDirectory())
+            return
+        Path path = e.file.getCanonicalFile().toPath()
+        path.register(watchService, kinds)
+        
+    }
+
+    private void watch() {
+        try {
+            while(!shutdown) {
+                WatchKey key = watchService.take()
+                key.pollEvents().each {
+                    switch(it.kind()) {
+                        case ENTRY_CREATE: processCreated(key.watchable(), it.context()); break
+                        case ENTRY_MODIFY: processModified(key.watchable(), it.context()); break
+                        case ENTRY_DELETE: processDeleted(key.watchable(), it.context()); break
+                    }
+                }
+                key.reset()
+            }
+        } catch (InterruptedException|ClosedWatchServiceException e) {
+            if (!shutdown)
+                throw e
+        }
+    }
+    
+
+    private void processCreated(Path parent, Path path) {
+        File f= join(parent, path)
+        log.fine("created entry $f")
+        if (f.isDirectory())
+            f.toPath().register(watchService, kinds)
+        else
+            waitingFiles.put(f, System.currentTimeMillis())
+    }
+        
+    private void processModified(Path parent, Path path) {
+        File f = join(parent, path)
+        log.fine("modified entry $f")
+        waitingFiles.put(f, System.currentTimeMillis())
+    }
+    
+    private void processDeleted(Path parent, Path path) {
+        File f = join(parent, path)
+        log.fine("deleted entry $f")
+        SharedFile sf = fileManager.fileToSharedFile.get(f)
+        if (sf != null)
+            eventBus.publish(new FileUnsharedEvent(unsharedFile : sf))
+    }
+    
+    private static File join(Path parent, Path path) {
+        File parentFile = parent.toFile().getCanonicalFile()
+        new File(parentFile, path.toFile().getName())
+    }
+    
+    private void publish() {
+        try {
+            while(!shutdown) {
+                Thread.sleep(WAIT_TIME)
+                long now = System.currentTimeMillis()
+                def published = []
+                waitingFiles.each { file, timestamp ->
+                    if (now - timestamp > WAIT_TIME) {
+                        log.fine("publishing file $file")
+                        eventBus.publish new FileSharedEvent(file : file)
+                        published << file
+                    }
+                }
+                published.each {
+                    waitingFiles.remove(it)
+                }
+            }
+        } catch (InterruptedException e) {
+            if (!shutdown)
+                throw e
+        }
+    }
+}

core/src/main/groovy/com/muwire/core/files/FileDownloadedEvent.groovy

@@ -2,10 +2,11 @@ package com.muwire.core.files
 
 import com.muwire.core.DownloadedFile
 import com.muwire.core.Event
+import com.muwire.core.download.Downloader
 
 import net.i2p.data.Destination
 
 class FileDownloadedEvent extends Event {
-
+    Downloader downloader
 	DownloadedFile downloadedFile
 }

core/src/main/groovy/com/muwire/core/files/FileHasher.groovy

@@ -1,6 +1,9 @@
 package com.muwire.core.files
 
 import com.muwire.core.InfoHash
+
+import net.i2p.data.Base64
+
 import java.nio.MappedByteBuffer
 import java.nio.channels.FileChannel
 import java.nio.channels.FileChannel.MapMode
@@ -17,12 +20,12 @@ class FileHasher {
 	 * @return the size of each piece in power of 2
 	 */
 	static int getPieceSize(long size) {
-		if (size <= 0x1 << 25)
+		if (size <= 0x1 << 30)
-			return 18
+			return 17
 		
-		for (int i = 26; i <= 37; i++) {
+		for (int i = 31; i <= 37; i++) {
 			if (size <= 0x1L << i) {
-				return i-7
+				return i-13
 			}
 		}
 		
@@ -52,11 +55,11 @@ class FileHasher {
 		try {
 			MappedByteBuffer buf
 			for (int i = 0; i < numPieces - 1; i++) {
-				buf = raf.getChannel().map(MapMode.READ_ONLY, size * i, size)
+				buf = raf.getChannel().map(MapMode.READ_ONLY, ((long)size) * i, size)
 				digest.update buf
 				output.write(digest.digest(), 0, 32)
 			}
-			def lastPieceLength = length - (numPieces - 1) * size
+			def lastPieceLength = length - (numPieces - 1) * ((long)size)
 			buf = raf.getChannel().map(MapMode.READ_ONLY, length - lastPieceLength, lastPieceLength)
 			digest.update buf
 			output.write(digest.digest(), 0, 32)
@@ -67,4 +70,18 @@ class FileHasher {
 		byte [] hashList = output.toByteArray()
 		InfoHash.fromHashList(hashList)
 	}
+    
+    public static void main(String[] args) {
+        if (args.length != 1) {
+            println "This utility computes an infohash of a file"
+            println "Pass absolute path to a file as an argument"
+            System.exit(1)
+        }
+        
+        def file = new File(args[0])
+        file = file.getAbsoluteFile()
+        def hasher = new FileHasher()
+        def infohash = hasher.hashFile(file)
+        println Base64.encode(infohash.getRoot())
+    }
 }

core/src/main/groovy/com/muwire/core/files/FileManager.groovy

@@ -2,7 +2,9 @@ package com.muwire.core.files
 
 import com.muwire.core.EventBus
 import com.muwire.core.InfoHash
+import com.muwire.core.MuWireSettings
 import com.muwire.core.SharedFile
+import com.muwire.core.UILoadedEvent
 import com.muwire.core.search.ResultsEvent
 import com.muwire.core.search.SearchEvent
 import com.muwire.core.search.SearchIndex
@@ -14,27 +16,31 @@ class FileManager {
 
 
 	final EventBus eventBus
+    final MuWireSettings settings
 	final Map<InfoHash, Set<SharedFile>> rootToFiles = Collections.synchronizedMap(new HashMap<>())
 	final Map<File, SharedFile> fileToSharedFile = Collections.synchronizedMap(new HashMap<>())
 	final Map<String, Set<File>> nameToFiles = new HashMap<>()
 	final SearchIndex index = new SearchIndex()
 	
-	FileManager(EventBus eventBus) {
+	FileManager(EventBus eventBus, MuWireSettings settings) {
+        this.settings = settings
 		this.eventBus = eventBus
 	}
 	
-	void onFileHashedEvent(FileHashedEvent e) {
+    void onFileHashedEvent(FileHashedEvent e) {
-		if (e.sharedFile != null)
+        if (e.sharedFile != null)
-			addToIndex(e.sharedFile)
+            addToIndex(e.sharedFile)
-	}
+    }
 	
 	void onFileLoadedEvent(FileLoadedEvent e) {
 		addToIndex(e.loadedFile)
 	}
 	
-	void onFileDownloadedEvent(FileDownloadedEvent e) {
+    void onFileDownloadedEvent(FileDownloadedEvent e) {
-		addToIndex(e.downloadedFile)
+        if (settings.shareDownloadedFiles) {
-	}
+            addToIndex(e.downloadedFile)
+        }
+    }
 	
 	private void addToIndex(SharedFile sf) {
         log.info("Adding shared file " + sf.getFile())
@@ -100,20 +106,33 @@ class FileManager {
 		if (e.searchHash != null) {
             Set<SharedFile> found
             found = rootToFiles.get new InfoHash(e.searchHash)
+            found = filter(found, e.oobInfohash)
 			if (found != null && !found.isEmpty())
-				re = new ResultsEvent(results: found.asList(), uuid: e.uuid)
+				re = new ResultsEvent(results: found.asList(), uuid: e.uuid, searchEvent: e)
 		} else {
 			def names = index.search e.searchTerms
 			Set<File> files = new HashSet<>()
 			names.each { files.addAll nameToFiles.getOrDefault(it, []) }
 			Set<SharedFile> sharedFiles = new HashSet<>()
 			files.each { sharedFiles.add fileToSharedFile[it] }
+            files = filter(sharedFiles, e.oobInfohash)
 			if (!sharedFiles.isEmpty())
-				re = new ResultsEvent(results: sharedFiles.asList(), uuid: e.uuid)
+				re = new ResultsEvent(results: sharedFiles.asList(), uuid: e.uuid, searchEvent: e)
 			
 		}
 		
 		if (re != null)
 			eventBus.publish(re)
 	}
+    
+    private static Set<SharedFile> filter(Set<SharedFile> files, boolean oob) {
+        if (!oob)
+            return files
+        Set<SharedFile> rv = new HashSet<>()
+        files.each { 
+            if (it.getPieceSize() != 0)
+                rv.add(it)
+        }
+        rv
+    }
 }

core/src/main/groovy/com/muwire/core/files/HasherService.groovy

@@ -10,11 +10,13 @@ class HasherService {
 
 	final FileHasher hasher
 	final EventBus eventBus
+    final FileManager fileManager
 	Executor executor
 	
-	HasherService(FileHasher hasher, EventBus eventBus) {
+	HasherService(FileHasher hasher, EventBus eventBus, FileManager fileManager) {
 		this.hasher = hasher
 		this.eventBus = eventBus
+        this.fileManager = fileManager
 	}
 	
 	void start() {
@@ -22,13 +24,15 @@ class HasherService {
 	}
 	
 	void onFileSharedEvent(FileSharedEvent evt) {
+        if (fileManager.fileToSharedFile.containsKey(evt.file))
+            return
 		executor.execute( { -> process(evt.file) } as Runnable)
 	}
 	
 	private void process(File f) {
 		f = f.getCanonicalFile()
 		if (f.isDirectory()) {
-			f.listFiles().each {onFileSharedEvent new FileSharedEvent(file: it) }
+			f.listFiles().each {eventBus.publish new FileSharedEvent(file: it) }
 		} else {
 			if (f.length() == 0) {
 				eventBus.publish new FileHashedEvent(error: "Not sharing empty file $f")
@@ -36,7 +40,7 @@ class HasherService {
 				eventBus.publish new FileHashedEvent(error: "$f is too large to be shared ${f.length()}")
 			} else {
 				def hash = hasher.hashFile f
-				eventBus.publish new FileHashedEvent(sharedFile: new SharedFile(f, hash))
+				eventBus.publish new FileHashedEvent(sharedFile: new SharedFile(f, hash, FileHasher.getPieceSize(f.length())))
 			}
 		}
 	}

core/src/main/groovy/com/muwire/core/files/PersisterService.groovy

@@ -1,5 +1,8 @@
 package com.muwire.core.files
 
+import java.nio.file.CopyOption
+import java.nio.file.Files
+import java.nio.file.StandardCopyOption
 import java.util.logging.Level
 import java.util.stream.Collectors
 
@@ -8,6 +11,7 @@ import com.muwire.core.EventBus
 import com.muwire.core.InfoHash
 import com.muwire.core.Service
 import com.muwire.core.SharedFile
+import com.muwire.core.UILoadedEvent
 import com.muwire.core.util.DataUtil
 
 import groovy.json.JsonOutput
@@ -33,14 +37,14 @@ class PersisterService extends Service {
 		timer = new Timer("file persister", true)
 	}
 	
-	void start() {
-		timer.schedule({load()} as TimerTask, 1000)
-	}
-	
 	void stop() {
 		timer.cancel()
 	}
-	
+
+    void onUILoadedEvent(UILoadedEvent e) {
+        timer.schedule({load()} as TimerTask, 1)
+    }
+    	
 	void load() {
 		if (location.exists() && location.isFile()) {
 			def slurper = new JsonSlurper()
@@ -55,10 +59,13 @@ class PersisterService extends Service {
 						}
 					}
 				}
+                listener.publish(new AllFilesLoadedEvent())
 			} catch (IllegalArgumentException|NumberFormatException e) {
                 log.log(Level.WARNING, "couldn't load files",e)
 			}
-		}
+		} else {
+            listener.publish(new AllFilesLoadedEvent())
+        }
 		timer.schedule({persistFiles()} as TimerTask, 0, interval)
 		loaded = true
 	}
@@ -94,28 +101,37 @@ class PersisterService extends Service {
 		if (!Arrays.equals(root, ih.getRoot()))
 			return null
 			
-		if (json.sources != null) {
+        int pieceSize = 0
+        if (json.pieceSize != null)
+            pieceSize = json.pieceSize
+            
+        if (json.sources != null) {
 			List sources = (List)json.sources
 			Set<Destination> sourceSet = sources.stream().map({d -> new Destination(d.toString())}).collect Collectors.toSet()
-			DownloadedFile df = new DownloadedFile(file, ih, sourceSet)
+			DownloadedFile df = new DownloadedFile(file, ih, pieceSize, sourceSet)
 			return new FileLoadedEvent(loadedFile : df)
 		}
+        
 		 
-		SharedFile sf = new SharedFile(file, ih)
+		SharedFile sf = new SharedFile(file, ih, pieceSize)
 		return new FileLoadedEvent(loadedFile: sf)
 		
 	}
 	
 	private void persistFiles() {
-		location.delete()
 		def sharedFiles = fileManager.getSharedFiles()
-		location.withPrintWriter { writer ->
+        
+        File tmp = File.createTempFile("muwire-files", "tmp")
+        tmp.deleteOnExit()
+		tmp.withPrintWriter { writer ->
 			sharedFiles.each { k, v ->
 				def json = toJson(k,v)
 				json = JsonOutput.toJson(json)
 				writer.println json
 			}
 		}
+        Files.copy(tmp.toPath(), location.toPath(), StandardCopyOption.REPLACE_EXISTING)
+        tmp.delete()
 	}
 	
 	private def toJson(File f, SharedFile sf) {
@@ -124,6 +140,7 @@ class PersisterService extends Service {
 		json.length = f.length()
 		InfoHash ih = sf.getInfoHash()
 		json.infoHash = Base64.encode ih.getRoot()
+        json.pieceSize = sf.getPieceSize()
 		byte [] tmp = new byte [32]
 		json.hashList = []
 		for (int i = 0;i < ih.getHashList().length / 32; i++) {

core/src/main/groovy/com/muwire/core/hostcache/CacheClient.groovy

@@ -140,7 +140,7 @@ class CacheClient {
 		pong.pongs.asList().each { 
 			Destination dest = new Destination(it)
 			if (!session.getMyDestination().equals(dest))
-				eventBus.publish(new HostDiscoveredEvent(destination: dest)) 
+				eventBus.publish(new HostDiscoveredEvent(destination: dest, fromHostcache : true)) 
 		}
 		
 	}

core/src/main/groovy/com/muwire/core/hostcache/Host.groovy

@@ -5,9 +5,11 @@ import net.i2p.data.Destination
 class Host {
 
 	private static final int MAX_FAILURES = 3
+    private static final int CLEAR_INTERVAL = 60 * 60 * 1000
 	
 	final Destination destination
 	int failures,successes
+    long lastAttempt
 	
 	public Host(Destination destination) {
 		this.destination = destination
@@ -16,11 +18,13 @@ class Host {
 	synchronized void onConnect() {
 		failures = 0
 		successes++
+        lastAttempt = System.currentTimeMillis()
 	}
 	
 	synchronized void onFailure() {
 		failures++
 		successes = 0
+        lastAttempt = System.currentTimeMillis()
 	}
 	
 	synchronized boolean isFailed() {
@@ -30,4 +34,12 @@ class Host {
 	synchronized boolean hasSucceeded() {
 		successes > 0
 	}
+    
+    synchronized void clearFailures() {
+        failures = 0
+    }
+    
+    synchronized void canTryAgain() {
+        System.currentTimeMillis() - lastAttempt > CLEAR_INTERVAL
+    }
 }

core/src/main/groovy/com/muwire/core/hostcache/HostCache.groovy

@@ -46,8 +46,12 @@ class HostCache extends Service {
 	void onHostDiscoveredEvent(HostDiscoveredEvent e) {
 		if (myself == e.destination)
 			return
-		if (hosts.containsKey(e.destination))
+		if (hosts.containsKey(e.destination)) {
-			return
+            if (!e.fromHostcache)
+                return
+            hosts.get(e.destination).clearFailures()
+            return
+		}
 		Host host = new Host(e.destination)
 		if (allowHost(host)) {
 			hosts.put(e.destination, host)
@@ -105,6 +109,8 @@ class HostCache extends Service {
 				Host host = new Host(dest)
 				host.failures = Integer.valueOf(String.valueOf(entry.failures))
 				host.successes = Integer.valueOf(String.valueOf(entry.successes))
+                if (entry.lastAttempt != null)
+                    host.lastAttempt = entry.lastAttempt
 				if (allowHost(host))
 					hosts.put(dest, host)
 			}
@@ -114,7 +120,7 @@ class HostCache extends Service {
 	}
 	
 	private boolean allowHost(Host host) {
-		if (host.isFailed())
+		if (host.isFailed() && !host.canTryAgain())
 			return false
 		if (host.destination == myself)
 			return false
@@ -139,6 +145,7 @@ class HostCache extends Service {
 					map.destination = dest.toBase64()
 					map.failures = host.failures
 					map.successes = host.successes
+                    map.lastAttempt = host.lastAttempt
 					def json = JsonOutput.toJson(map)
 					writer.println json
 				}

core/src/main/groovy/com/muwire/core/hostcache/HostDiscoveredEvent.groovy

@@ -7,9 +7,10 @@ import net.i2p.data.Destination
 class HostDiscoveredEvent extends Event {
 
 	Destination destination
-	
+    boolean fromHostcache
+    
 	@Override
 	public String toString() {
-		"HostDiscoveredEvent ${super.toString()} destination:${destination.toBase32()}"
+		"HostDiscoveredEvent ${super.toString()} destination:${destination.toBase32()} from hostcache $fromHostcache"
 	}
 }

core/src/main/groovy/com/muwire/core/mesh/Mesh.groovy

@@ -0,0 +1,28 @@
+package com.muwire.core.mesh
+
+import com.muwire.core.InfoHash
+import com.muwire.core.Persona
+import com.muwire.core.download.Pieces
+
+import net.i2p.data.Destination
+import net.i2p.util.ConcurrentHashSet
+
+class Mesh {
+    private final InfoHash infoHash
+    private final Set<Persona> sources = new ConcurrentHashSet<>()
+    private final Pieces pieces
+    
+    Mesh(InfoHash infoHash, Pieces pieces) {
+        this.infoHash = infoHash
+        this.pieces = pieces
+    }
+    
+    Set<Persona> getRandom(int n, Persona exclude) {
+        List<Persona> tmp = new ArrayList<>(sources)
+        tmp.remove(exclude)
+        Collections.shuffle(tmp)
+        if (tmp.size() < n)
+            return tmp
+        tmp[0..n-1]
+    }
+}

core/src/main/groovy/com/muwire/core/mesh/MeshManager.groovy

@@ -0,0 +1,96 @@
+package com.muwire.core.mesh
+
+import java.util.stream.Collectors
+
+import com.muwire.core.Constants
+import com.muwire.core.InfoHash
+import com.muwire.core.Persona
+import com.muwire.core.download.Pieces
+import com.muwire.core.download.SourceDiscoveredEvent
+import com.muwire.core.files.FileManager
+
+import groovy.json.JsonOutput
+import groovy.json.JsonSlurper
+import net.i2p.data.Base64
+
+class MeshManager {
+    
+    private static final int EXPIRATION = 60 * 60 * 1000
+    
+    private final Map<InfoHash, Mesh> meshes = Collections.synchronizedMap(new HashMap<>())
+    private final FileManager fileManager
+    private final File home
+    
+    MeshManager(FileManager fileManager, File home) {
+        this.fileManager = fileManager
+        this.home = home
+        load()
+    }
+    
+    Mesh get(InfoHash infoHash) {
+        meshes.get(infoHash)
+    }
+    
+    Mesh getOrCreate(InfoHash infoHash, int nPieces) {
+        synchronized(meshes) {
+            if (meshes.containsKey(infoHash))
+                return meshes.get(infoHash)
+            Pieces pieces = new Pieces(nPieces, Constants.DOWNLOAD_SEQUENTIAL_RATIO)
+            if (fileManager.rootToFiles.containsKey(infoHash)) {
+                for (int i = 0; i < nPieces; i++)
+                    pieces.markDownloaded(i)
+            }
+            Mesh rv = new Mesh(infoHash, pieces)
+            meshes.put(infoHash, rv)
+            return rv
+        }
+    }
+    
+    void onSourceDiscoveredEvent(SourceDiscoveredEvent e) {
+        Mesh mesh = meshes.get(e.infoHash)
+        if (mesh == null)
+            return
+       if (mesh.sources.add(e.source))
+            save()
+    }
+    
+    private void save() {
+        File meshFile = new File(home, "mesh.json")
+        synchronized(meshes) {
+            meshFile.withPrintWriter { writer ->
+                meshes.values().each { mesh ->
+                    def json = [:]
+                    json.timestamp = System.currentTimeMillis()
+                    json.infoHash = Base64.encode(mesh.infoHash.getRoot())
+                    json.sources = mesh.sources.stream().map({it.toBase64()}).collect(Collectors.toList())
+                    json.nPieces = mesh.pieces.nPieces
+                    writer.println(JsonOutput.toJson(json))
+                }
+            }
+        }
+    }
+    
+    private void load() {
+        File meshFile = new File(home, "mesh.json")
+        if (!meshFile.exists())
+            return
+        long now = System.currentTimeMillis()
+        JsonSlurper slurper = new JsonSlurper()
+        meshFile.eachLine { 
+            def json = slurper.parseText(it)
+            if (now - json.timestamp > EXPIRATION)
+                return
+            InfoHash infoHash = new InfoHash(Base64.decode(json.infoHash))
+            Pieces pieces = new Pieces(json.nPieces, Constants.DOWNLOAD_SEQUENTIAL_RATIO)
+            
+            Mesh mesh = new Mesh(infoHash, pieces)
+            json.sources.each { source -> 
+                Persona persona = new Persona(new ByteArrayInputStream(Base64.decode(source)))
+                mesh.sources.add(persona)
+            }
+            
+            if (!mesh.sources.isEmpty())
+                meshes.put(infoHash, mesh)
+        }
+    }
+}

core/src/main/groovy/com/muwire/core/search/QueryEvent.groovy

@@ -13,4 +13,8 @@ class QueryEvent extends Event {
     Persona originator
 	Destination receivedOn
 
+    String toString() {
+        "searchEvent: $searchEvent firstHop:$firstHop, replyTo:${replyTo.toBase32()}" +
+        "originator: ${originator.getHumanReadableName()} receivedOn: ${receivedOn.toBase32()}"
+    }
 }

core/src/main/groovy/com/muwire/core/search/ResultsEvent.groovy

@@ -5,6 +5,7 @@ import com.muwire.core.SharedFile
 
 class ResultsEvent extends Event {
 
+    SearchEvent searchEvent
 	SharedFile[] results
 	UUID uuid
 }

core/src/main/groovy/com/muwire/core/search/ResultsParser.groovy

@@ -1,5 +1,7 @@
 package com.muwire.core.search
 
+import java.util.stream.Collectors
+
 import javax.naming.directory.InvalidSearchControlsException
 
 import com.muwire.core.InfoHash
@@ -7,13 +9,25 @@ import com.muwire.core.Persona
 import com.muwire.core.util.DataUtil
 
 import net.i2p.data.Base64
+import net.i2p.data.Destination
 
 class ResultsParser {
     public static UIResultEvent parse(Persona p, UUID uuid, def json) throws InvalidSearchResultException {
         if (json.type != "Result")
             throw new InvalidSearchResultException("not a result json")
-        if (json.version != 1)
+        switch(json.version) {
-            throw new InvalidSearchResultException("unknown version $json.version")
+            case 1:
+                return parseV1(p, uuid, json)
+            case 2:
+                return parseV2(p, uuid, json)
+            default:
+                throw new InvalidSearchResultException("unknown version $json.version")
+
+        }
+
+    }
+    
+    private static parseV1(Persona p, UUID uuid, def json) {
         if (json.name == null)
             throw new InvalidSearchResultException("name missing")
         if (json.size == null)
@@ -47,9 +61,45 @@ class ResultsParser {
                  size : size,
                  infohash : parsedIH,
                  pieceSize : pieceSize,
+                 sources : Collections.emptySet(),
                  uuid : uuid)
         } catch (Exception e) {
             throw new InvalidSearchResultException("parsing search result failed",e)
         }
     }
+    
+    private static UIResultEvent parseV2(Persona p, UUID uuid, def json) {
+        if (json.name == null)
+            throw new InvalidSearchResultException("name missing")
+        if (json.size == null)
+            throw new InvalidSearchResultException("length missing")
+        if (json.infohash == null)
+            throw new InvalidSearchResultException("infohash missing")
+        if (json.pieceSize == null)
+            throw new InvalidSearchResultException("pieceSize missing")
+        if (json.hashList != null)
+            throw new InvalidSearchResultException("V2 result with hashlist")
+        try {
+            String name = DataUtil.readi18nString(Base64.decode(json.name))
+            long size = json.size
+            byte [] infoHash = Base64.decode(json.infohash)
+            if (infoHash.length != InfoHash.SIZE)
+                throw new InvalidSearchResultException("invalid infohash size $infoHash.length")
+            int pieceSize = json.pieceSize
+            
+            Set<Destination> sources = Collections.emptySet()
+            if (json.sources != null) 
+                sources = json.sources.stream().map({new Destination(it)}).collect(Collectors.toSet())
+            
+            return new UIResultEvent( sender : p,
+                name : name,
+                size : size,
+                infohash : new InfoHash(infoHash),
+                pieceSize : pieceSize,
+                sources : sources,
+                uuid: uuid)
+        } catch (Exception e) {
+            throw new InvalidSearchResultException("parsing search result failed",e)
+        }
+    }
 }

core/src/main/groovy/com/muwire/core/search/ResultsSender.groovy

@@ -11,7 +11,10 @@ import java.util.concurrent.Executor
 import java.util.concurrent.Executors
 import java.util.concurrent.ThreadFactory
 import java.util.concurrent.atomic.AtomicInteger
+import java.util.logging.Level
+import java.util.stream.Collectors
 
+import com.muwire.core.DownloadedFile
 import com.muwire.core.EventBus
 import com.muwire.core.InfoHash
 
@@ -46,22 +49,30 @@ class ResultsSender {
         this.me = me
     }
     
-    void sendResults(UUID uuid, SharedFile[] results, Destination target) {
+    void sendResults(UUID uuid, SharedFile[] results, Destination target, boolean oobInfohash) {
-        log.info("Sending $results.length results for uuid $uuid to ${target.toBase32()}")
+        log.info("Sending $results.length results for uuid $uuid to ${target.toBase32()} oobInfohash : $oobInfohash")
         if (target.equals(me.destination)) {
             results.each { 
                 long length = it.getFile().length()
+                int pieceSize = it.getPieceSize()
+                if (pieceSize == 0)
+                    pieceSize = FileHasher.getPieceSize(length)
+                Set<Destination> suggested = Collections.emptySet()
+                if (it instanceof DownloadedFile) 
+                    suggested = it.sources
                 def uiResultEvent = new UIResultEvent( sender : me,
                     name : it.getFile().getName(),
                     size : length,
                     infohash : it.getInfoHash(),
-                    pieceSize : FileHasher.getPieceSize(length),
+                    pieceSize : pieceSize,
-                    uuid : uuid
+                    uuid : uuid,
+                    sources : suggested
                     )
                     eventBus.publish(uiResultEvent)
             }
         } else {
-            executor.execute(new ResultSendJob(uuid : uuid, results : results, target: target))
+            executor.execute(new ResultSendJob(uuid : uuid, results : results, 
+                target: target, oobInfohash : oobInfohash))
         }
     }
     
@@ -69,48 +80,58 @@ class ResultsSender {
         UUID uuid
         SharedFile [] results
         Destination target
+        boolean oobInfohash
         
         @Override
         public void run() {
-            byte [] tmp = new byte[InfoHash.SIZE]
-            JsonOutput jsonOutput = new JsonOutput()
-            Endpoint endpoint = null;
             try {
-                endpoint = connector.connect(target)
+                byte [] tmp = new byte[InfoHash.SIZE]
-                DataOutputStream os = new DataOutputStream(endpoint.getOutputStream())
+                JsonOutput jsonOutput = new JsonOutput()
-                os.write("POST $uuid\r\n\r\n".getBytes(StandardCharsets.US_ASCII))
+                Endpoint endpoint = null;
-                me.write(os)
+                try {
-                os.writeShort((short)results.length)
+                    endpoint = connector.connect(target)
-                results.each {
+                    DataOutputStream os = new DataOutputStream(endpoint.getOutputStream())
-                    byte [] name = it.getFile().getName().getBytes(StandardCharsets.UTF_8)
+                    os.write("POST $uuid\r\n\r\n".getBytes(StandardCharsets.US_ASCII))
-                    def baos = new ByteArrayOutputStream()
+                    me.write(os)
-                    def daos = new DataOutputStream(baos)
+                    os.writeShort((short)results.length)
-                    daos.writeShort((short) name.length)
+                    results.each {
-                    daos.write(name)
+                        byte [] name = it.getFile().getName().getBytes(StandardCharsets.UTF_8)
-                    daos.flush()
+                        def baos = new ByteArrayOutputStream()
-                    String encodedName = Base64.encode(baos.toByteArray())
+                        def daos = new DataOutputStream(baos)
-                    def obj = [:]
+                        daos.writeShort((short) name.length)
-                    obj.type = "Result"
+                        daos.write(name)
-                    obj.version = 1
+                        daos.flush()
-                    obj.name = encodedName
+                        String encodedName = Base64.encode(baos.toByteArray())
-                    obj.infohash = Base64.encode(it.getInfoHash().getRoot())
+                        def obj = [:]
-                    obj.size = it.getFile().length()
+                        obj.type = "Result"
-                    obj.pieceSize = FileHasher.getPieceSize(it.getFile().length())
+                        obj.version = oobInfohash ? 2 : 1
-                    byte [] hashList = it.getInfoHash().getHashList()
+                        obj.name = encodedName
-                    def hashListB64 = []
+                        obj.infohash = Base64.encode(it.getInfoHash().getRoot())
-                    for (int i = 0; i < hashList.length / InfoHash.SIZE; i++) {
+                        obj.size = it.getFile().length()
-                        System.arraycopy(hashList, InfoHash.SIZE * i, tmp, 0, InfoHash.SIZE)
+                        obj.pieceSize = it.getPieceSize()
-                        hashListB64 << Base64.encode(tmp)
+                        if (!oobInfohash) {
-                    }
+                            byte [] hashList = it.getInfoHash().getHashList()
-                    obj.hashList = hashListB64
+                            def hashListB64 = []
+                            for (int i = 0; i < hashList.length / InfoHash.SIZE; i++) {
+                                System.arraycopy(hashList, InfoHash.SIZE * i, tmp, 0, InfoHash.SIZE)
+                                hashListB64 << Base64.encode(tmp)
+                            }
+                            obj.hashList = hashListB64
+                        }
 
-                    def json = jsonOutput.toJson(obj)
+                        if (it instanceof DownloadedFile)
-                    os.writeShort((short)json.length())
+                            obj.sources = it.sources.stream().map({dest -> dest.toBase64()}).collect(Collectors.toSet())
-                    os.write(json.getBytes(StandardCharsets.US_ASCII))
+
+                        def json = jsonOutput.toJson(obj)
+                        os.writeShort((short)json.length())
+                        os.write(json.getBytes(StandardCharsets.US_ASCII))
+                    }
+                    os.flush()
+                } finally {
+                    endpoint?.close()
                 }
-                os.flush()
+            } catch (Exception e) {
-            } finally {
+                log.log(Level.WARNING, "problem sending results",e)
-                endpoint?.close()
             }
         }
     }

core/src/main/groovy/com/muwire/core/search/SearchEvent.groovy

@@ -1,10 +1,19 @@
 package com.muwire.core.search
 
 import com.muwire.core.Event
+import com.muwire.core.InfoHash
 
 class SearchEvent extends Event {
 
 	List<String> searchTerms
 	byte [] searchHash
 	UUID uuid
+    boolean oobInfohash
+    
+    String toString() {
+        def infoHash = null
+        if (searchHash != null)
+            infoHash = new InfoHash(searchHash)
+        "searchTerms: $searchTerms searchHash:$infoHash, uuid:$uuid oobInfohash:$oobInfohash"
+    }
 }

core/src/main/groovy/com/muwire/core/search/SearchIndex.groovy

@@ -33,7 +33,10 @@ class SearchIndex {
 	
 	private static String[] split(String source) {
 		source = source.replaceAll(Constants.SPLIT_PATTERN, " ").toLowerCase()
-		source.split(" ")
+		String [] split = source.split(" ")
+        def rv = []
+        split.each { if (it.length() > 0) rv << it }
+        rv.toArray(new String[0])
 	}
 	
 	String[] search(List<String> terms) {
@@ -42,7 +45,7 @@ class SearchIndex {
 		terms.each {
 			Set<String> forWord = keywords.getOrDefault(it,[])
 			if (rv == null) {
-				rv = forWord
+				rv = new HashSet<>(forWord)
 			} else {
 				rv.retainAll(forWord)
 			}

core/src/main/groovy/com/muwire/core/search/SearchManager.groovy

@@ -44,7 +44,7 @@ public class SearchManager {
             log.info("No results for search uuid $event.uuid")
             return
         }
-        resultsSender.sendResults(event.uuid, event.results, target)
+        resultsSender.sendResults(event.uuid, event.results, target, event.searchEvent.oobInfohash)
     }
     
     boolean hasLocalSearch(UUID uuid) {

core/src/main/groovy/com/muwire/core/search/UIResultBatchEvent.groovy

@@ -0,0 +1,8 @@
+package com.muwire.core.search
+
+import com.muwire.core.Event
+
+class UIResultBatchEvent extends Event {
+    UUID uuid
+    UIResultEvent[] results
+}

core/src/main/groovy/com/muwire/core/search/UIResultEvent.groovy

@@ -4,11 +4,19 @@ import com.muwire.core.Event
 import com.muwire.core.InfoHash
 import com.muwire.core.Persona
 
+import net.i2p.data.Destination
+
 class UIResultEvent extends Event {
     Persona sender
+    Set<Destination> sources
     UUID uuid
     String name
     long size
     InfoHash infohash
     int pieceSize
+    
+    @Override
+    public String toString() {
+        super.toString() + "name:$name size:$size sender:${sender.getHumanReadableName()} pieceSize $pieceSize"
+    }
 }

core/src/main/groovy/com/muwire/core/update/UpdateAvailableEvent.groovy

@@ -1,8 +1,10 @@
 package com.muwire.core.update
 
 import com.muwire.core.Event
+import com.muwire.core.InfoHash
 
 class UpdateAvailableEvent extends Event {
     String version
     String signer
+    String infoHash
 }

core/src/main/groovy/com/muwire/core/update/UpdateClient.groovy

@@ -36,7 +36,7 @@ class UpdateClient {
     
     void start() {
         session.addMuxedSessionListener(new Listener(), I2PSession.PROTO_DATAGRAM, 2)
-        timer.schedule({checkUpdate()} as TimerTask, 30000, 60 * 60 * 1000)
+        timer.schedule({checkUpdate()} as TimerTask, 60000, 60 * 60 * 1000)
     }
     
     void stop() {
@@ -107,7 +107,7 @@ class UpdateClient {
                 }
                 
                 log.info("new version $payload.version available, publishing event")
-                eventBus.publish(new UpdateAvailableEvent(version : payload.version, signer : payload.signer))
+                eventBus.publish(new UpdateAvailableEvent(version : payload.version, signer : payload.signer, infoHash : payload.infoHash))
                 
             } catch (Exception e) {
                 log.log(Level.WARNING,"Invalid datagram",e)

core/src/main/groovy/com/muwire/core/upload/ContentRequest.groovy

@@ -0,0 +1,6 @@
+package com.muwire.core.upload
+
+class ContentRequest extends Request {
+    Range range
+    boolean have
+}

core/src/main/groovy/com/muwire/core/upload/ContentUploader.groovy

@@ -0,0 +1,108 @@
+package com.muwire.core.upload
+
+import java.nio.ByteBuffer
+import java.nio.channels.FileChannel
+import java.nio.charset.StandardCharsets
+import java.nio.file.Files
+import java.nio.file.StandardOpenOption
+import java.util.stream.Collectors
+
+import com.muwire.core.Persona
+import com.muwire.core.connection.Endpoint
+import com.muwire.core.mesh.Mesh
+import com.muwire.core.util.DataUtil
+
+import net.i2p.data.Destination
+
+class ContentUploader extends Uploader {
+    
+    private final File file
+    private final ContentRequest request
+    private final Mesh mesh
+    private final int pieceSize
+    
+    ContentUploader(File file, ContentRequest request, Endpoint endpoint, Mesh mesh, int pieceSize) {
+        super(endpoint)
+        this.file = file
+        this.request = request
+        this.mesh = mesh
+        this.pieceSize = pieceSize
+    }
+    
+    @Override
+    void respond() {
+        OutputStream os = endpoint.getOutputStream()
+        Range range = request.getRange()
+        boolean satisfiable = true
+        final long length = file.length()
+        if (range.start >= length || range.end >= length)
+            satisfiable = false
+        if (satisfiable) {
+            int startPiece = range.start / (0x1 << pieceSize)
+            int endPiece = range.end / (0x1 << pieceSize)
+            for (int i = startPiece; i <= endPiece; i++)
+                satisfiable &= mesh.pieces.isDownloaded(i)
+        }
+        if (!satisfiable) {
+            os.write("416 Range Not Satisfiable\r\n".getBytes(StandardCharsets.US_ASCII))
+            writeMesh(request.downloader)
+            os.write("\r\n".getBytes(StandardCharsets.US_ASCII))
+            os.flush()
+            return
+        }
+
+        os.write("200 OK\r\n".getBytes(StandardCharsets.US_ASCII))
+        os.write("Content-Range: $range.start-$range.end\r\n".getBytes(StandardCharsets.US_ASCII))
+        writeMesh(request.downloader)
+        os.write("\r\n".getBytes(StandardCharsets.US_ASCII))
+        
+        FileChannel channel
+        try {
+            channel = Files.newByteChannel(file.toPath(), EnumSet.of(StandardOpenOption.READ))
+            mapped = channel.map(FileChannel.MapMode.READ_ONLY, range.start, range.end - range.start + 1)
+            byte [] tmp = new byte[0x1 << 13]
+            while(mapped.hasRemaining()) {
+                int start = mapped.position()
+                synchronized(this) {
+                    mapped.get(tmp, 0, Math.min(tmp.length, mapped.remaining()))
+                }
+                int read = mapped.position() - start
+                endpoint.getOutputStream().write(tmp, 0, read)
+            }
+        } finally {
+            try {channel?.close() } catch (IOException ignored) {}
+            endpoint.getOutputStream().flush()
+        }
+    }
+    
+    private void writeMesh(Persona toExclude) {
+        String xHave = DataUtil.encodeXHave(mesh.pieces.getDownloaded(), mesh.pieces.nPieces)
+        endpoint.getOutputStream().write("X-Have: $xHave\r\n".getBytes(StandardCharsets.US_ASCII))
+        
+        Set<Persona> sources = mesh.getRandom(3, toExclude)
+        if (!sources.isEmpty()) {
+            String xAlts = sources.stream().map({ it.toBase64() }).collect(Collectors.joining(","))
+            endpoint.getOutputStream().write("X-Alt: $xAlts\r\n".getBytes(StandardCharsets.US_ASCII))
+        }
+    }
+
+    @Override
+    public String getName() {
+        return file.getName();
+    }
+
+    @Override
+    public synchronized int getProgress() {
+        if (mapped == null)
+            return 0
+        int position = mapped.position()
+        int total = request.getRange().end - request.getRange().start
+        (int)(position * 100.0 / total)
+    }
+
+    @Override
+    public String getDownloader() {
+        request.downloader.getHumanReadableName()
+    }
+
+}

core/src/main/groovy/com/muwire/core/upload/HashListRequest.groovy

@@ -0,0 +1,4 @@
+package com.muwire.core.upload
+
+class HashListRequest extends Request {
+}

core/src/main/groovy/com/muwire/core/upload/HashListUploader.groovy

@@ -0,0 +1,55 @@
+package com.muwire.core.upload
+
+import java.nio.ByteBuffer
+import java.nio.charset.StandardCharsets
+
+import com.muwire.core.InfoHash
+import com.muwire.core.connection.Endpoint
+
+import net.i2p.data.Base64
+
+class HashListUploader extends Uploader {
+    private final InfoHash infoHash
+    private final HashListRequest request
+    
+    HashListUploader(Endpoint endpoint, InfoHash infoHash, HashListRequest request) {
+        super(endpoint)
+        this.infoHash = infoHash
+        mapped = ByteBuffer.wrap(infoHash.getHashList())
+        this.request = request
+    }
+    
+    void respond() {
+        OutputStream os = endpoint.getOutputStream()
+        os.write("200 OK\r\n".getBytes(StandardCharsets.US_ASCII))
+        os.write("Content-Range: 0-${mapped.remaining()}\r\n\r\n".getBytes(StandardCharsets.US_ASCII))
+        
+        byte[]tmp = new byte[0x1 << 13]
+        while(mapped.hasRemaining()) {
+            int start = mapped.position()
+            synchronized(this) {
+                mapped.get(tmp, 0, Math.min(tmp.length, mapped.remaining()))
+            }
+            int read = mapped.position() - start
+            endpoint.getOutputStream().write(tmp, 0, read)
+        }
+        endpoint.getOutputStream().flush()
+    }
+
+    @Override
+    public String getName() {
+        return "Hash list for " + Base64.encode(infoHash.getRoot());
+    }
+
+    @Override
+    public synchronized int getProgress() {
+        (int)(mapped.position() * 100.0 / mapped.capacity())
+    }
+
+    @Override
+    public String getDownloader() {
+        request.downloader.getHumanReadableName()
+    }
+    
+    
+}

core/src/main/groovy/com/muwire/core/upload/Request.groovy

@@ -5,6 +5,7 @@ import java.nio.charset.StandardCharsets
 import com.muwire.core.Constants
 import com.muwire.core.InfoHash
 import com.muwire.core.Persona
+import com.muwire.core.util.DataUtil
 
 import groovy.util.logging.Log
 import net.i2p.data.Base64
@@ -16,57 +17,12 @@ class Request {
     private static final byte N = "\n".getBytes(StandardCharsets.US_ASCII)[0]
     
     InfoHash infoHash
-    Range range
     Persona downloader
     Map<String, String> headers
     
-    static Request parse(InfoHash infoHash, InputStream is) throws IOException {
+    static Request parseContentRequest(InfoHash infoHash, InputStream is) throws IOException {
-        Map<String,String> headers = new HashMap<>()
+        
-        byte [] tmp = new byte[Constants.MAX_HEADER_SIZE]
+        Map<String, String> headers = parseHeaders(is)
-        while(headers.size() < Constants.MAX_HEADERS) {
-            boolean r = false
-            boolean n = false
-            int idx = 0
-            while (true) {
-                byte read = is.read()
-                if (read == -1)
-                    throw new IOException("Stream closed")
-                    
-                if (!r && read == N)
-                    throw new IOException("Received N before R")
-                if (read == R) {
-                    if (r) 
-                        throw new IOException("double R")
-                    r = true
-                    continue
-                }
-                
-                if (r && !n) {
-                    if (read != N) 
-                        throw new IOException("R not followed by N")
-                    n = true
-                    break
-                }
-                if (idx == 0x1 << 14)
-                    throw new IOException("Header too long")    
-                tmp[idx++] = read
-            }
-            
-            if (idx == 0)
-                break
-                
-            String header = new String(tmp, 0, idx, StandardCharsets.US_ASCII)
-            log.fine("Read header $header")
-            
-            int keyIdx = header.indexOf(":")
-            if (keyIdx < 1)
-                throw new IOException("Header key not found")
-            if (keyIdx == header.length())
-                throw new IOException("Header value not found")
-            String key = header.substring(0, keyIdx)
-            String value = header.substring(keyIdx + 1)
-            headers.put(key, value)
-        }
         
         if (!headers.containsKey("Range"))
             throw new IOException("Range header not found")
@@ -93,7 +49,75 @@ class Request {
             def decoded = Base64.decode(encoded)
             downloader = new Persona(new ByteArrayInputStream(decoded))
         }
-        new Request( infoHash : infoHash, range : new Range(start, end), headers : headers, downloader : downloader)
+        
+        boolean have = false
+        if (headers.containsKey("X-Have")) {
+            def encoded = headers["X-Have"].trim()
+            have = DataUtil.decodeXHave(encoded).size() > 0
+        }
+        new ContentRequest( infoHash : infoHash, range : new Range(start, end), 
+            headers : headers, downloader : downloader, have : have)
+    }
+    
+    static Request parseHashListRequest(InfoHash infoHash, InputStream is) throws IOException {
+        Map<String,String> headers = parseHeaders(is)
+        Persona downloader = null
+        if (headers.containsKey("X-Persona")) {
+            def encoded = headers["X-Persona"].trim()
+            def decoded = Base64.decode(encoded)
+            downloader = new Persona(new ByteArrayInputStream(decoded))
+        }
+        new HashListRequest(infoHash : infoHash, headers : headers, downloader : downloader)
+    }
+    
+    private static Map<String, String> parseHeaders(InputStream is) {
+        Map<String,String> headers = new HashMap<>()
+        byte [] tmp = new byte[Constants.MAX_HEADER_SIZE]
+        while(headers.size() < Constants.MAX_HEADERS) {
+            boolean r = false
+            boolean n = false
+            int idx = 0
+            while (true) {
+                byte read = is.read()
+                if (read == -1)
+                    throw new IOException("Stream closed")
+                    
+                if (!r && read == N)
+                    throw new IOException("Received N before R")
+                if (read == R) {
+                    if (r)
+                        throw new IOException("double R")
+                    r = true
+                    continue
+                }
+                
+                if (r && !n) {
+                    if (read != N)
+                        throw new IOException("R not followed by N")
+                    n = true
+                    break
+                }
+                if (idx == 0x1 << 14)
+                    throw new IOException("Header too long")
+                tmp[idx++] = read
+            }
+            
+            if (idx == 0)
+                break
+                
+            String header = new String(tmp, 0, idx, StandardCharsets.US_ASCII)
+            log.fine("Read header $header")
+            
+            int keyIdx = header.indexOf(":")
+            if (keyIdx < 1)
+                throw new IOException("Header key not found")
+            if (keyIdx == header.length())
+                throw new IOException("Header value not found")
+            String key = header.substring(0, keyIdx)
+            String value = header.substring(keyIdx + 1)
+            headers.put(key, value)
+        }
+        headers
     }
     
 }

core/src/main/groovy/com/muwire/core/upload/UploadManager.groovy

@@ -6,7 +6,12 @@ import com.muwire.core.EventBus
 import com.muwire.core.InfoHash
 import com.muwire.core.SharedFile
 import com.muwire.core.connection.Endpoint
+import com.muwire.core.download.DownloadManager
+import com.muwire.core.download.Downloader
+import com.muwire.core.download.SourceDiscoveredEvent
 import com.muwire.core.files.FileManager
+import com.muwire.core.mesh.Mesh
+import com.muwire.core.mesh.MeshManager
 
 import groovy.util.logging.Log
 import net.i2p.data.Base64
@@ -15,17 +20,22 @@ import net.i2p.data.Base64
 public class UploadManager {
     private final EventBus eventBus
     private final FileManager fileManager
+    private final MeshManager meshManager
+    private final DownloadManager downloadManager
     
     public UploadManager() {}
     
-    public UploadManager(EventBus eventBus, FileManager fileManager) {
+    public UploadManager(EventBus eventBus, FileManager fileManager, 
+        MeshManager meshManager, DownloadManager downloadManager) {
         this.eventBus = eventBus
         this.fileManager = fileManager
+        this.meshManager = meshManager
+        this.downloadManager = downloadManager
     }
     
-    public void processEndpoint(Endpoint e) throws IOException {
+    public void processGET(Endpoint e) throws IOException {
         byte [] infoHashStringBytes = new byte[44]
-                DataInputStream dis = new DataInputStream(e.getInputStream())
+        DataInputStream dis = new DataInputStream(e.getInputStream())
         boolean first = true
         while(true) {
             if (first)
@@ -44,8 +54,10 @@ public class UploadManager {
             log.info("Responding to upload request for root $infoHashString")
 
             byte [] infoHashRoot = Base64.decode(infoHashString)
+            InfoHash infoHash = new InfoHash(infoHashRoot)
             Set<SharedFile> sharedFiles = fileManager.getSharedFiles(infoHashRoot)
-            if (sharedFiles == null || sharedFiles.isEmpty()) {
+            Downloader downloader = downloadManager.downloaders.get(infoHash)
+            if (downloader == null && (sharedFiles == null || sharedFiles.isEmpty())) {
                 log.info "file not found"
                 e.getOutputStream().write("404 File Not Found\r\n\r\n".getBytes(StandardCharsets.US_ASCII))
                 e.getOutputStream().flush()
@@ -61,13 +73,31 @@ public class UploadManager {
                 return
             }
 
-            Request request = Request.parse(new InfoHash(infoHashRoot), e.getInputStream())
+            ContentRequest request = Request.parseContentRequest(infoHash, e.getInputStream())
             if (request.downloader != null && request.downloader.destination != e.destination) {
                 log.info("Downloader persona doesn't match their destination")
                 e.close()
                 return
             }
-            Uploader uploader = new Uploader(sharedFiles.iterator().next().file, request, e)
+            
+            if (request.have) 
+                eventBus.publish(new SourceDiscoveredEvent(infoHash : request.infoHash, source : request.downloader))
+            
+            Mesh mesh
+            File file
+            int pieceSize
+            if (downloader != null) {
+                mesh = meshManager.get(infoHash)
+                file = downloader.incompleteFile
+                pieceSize = downloader.pieceSizePow2
+            } else {
+                SharedFile sharedFile = sharedFiles.iterator().next();
+                mesh = meshManager.getOrCreate(request.infoHash, sharedFile.NPieces)
+                file = sharedFile.file
+                pieceSize = sharedFile.pieceSize
+            }
+                
+            Uploader uploader = new ContentUploader(file, request, e, mesh, pieceSize)
             eventBus.publish(new UploadEvent(uploader : uploader))
             try {
                 uploader.respond()
@@ -75,7 +105,131 @@ public class UploadManager {
                 eventBus.publish(new UploadFinishedEvent(uploader : uploader))
             }
         }
+    }
+    
+    public void processHashList(Endpoint e) {
+        byte [] infoHashStringBytes = new byte[44]
+        DataInputStream dis = new DataInputStream(e.getInputStream())
+        dis.readFully(infoHashStringBytes)
+        String infoHashString = new String(infoHashStringBytes, StandardCharsets.US_ASCII)
+        log.info("Responding to hashlist request for root $infoHashString")
         
+        byte [] infoHashRoot = Base64.decode(infoHashString)
+        InfoHash infoHash = new InfoHash(infoHashRoot)
+        Downloader downloader = downloadManager.downloaders.get(infoHash)
+        Set<SharedFile> sharedFiles = fileManager.getSharedFiles(infoHashRoot)
+        if (downloader == null && (sharedFiles == null || sharedFiles.isEmpty())) {
+            log.info "file not found"
+            e.getOutputStream().write("404 File Not Found\r\n\r\n".getBytes(StandardCharsets.US_ASCII))
+            e.getOutputStream().flush()
+            e.close()
+            return
+        }
+        
+        byte [] rn = new byte[2]
+        dis.readFully(rn)
+        if (rn != "\r\n".getBytes(StandardCharsets.US_ASCII)) {
+            log.warning("Malformed HASHLIST header")
+            e.close()
+            return
+        }
+        
+        Request request = Request.parseHashListRequest(infoHash, e.getInputStream())
+        if (request.downloader != null && request.downloader.destination != e.destination) {
+            log.info("Downloader persona doesn't match their destination")
+            e.close()
+            return
+        }
+        
+        InfoHash fullInfoHash
+        if (downloader == null) {
+            fullInfoHash = sharedFiles.iterator().next().infoHash
+        } else {
+            byte [] hashList = downloader.getInfoHash().getHashList()
+            if (hashList != null && hashList.length > 0)
+                fullInfoHash = downloader.getInfoHash()
+            else {
+                log.info("infohash not found in downloader")
+                e.getOutputStream().write("404 File Not Found\r\n\r\n".getBytes(StandardCharsets.US_ASCII))
+                e.getOutputStream().flush()
+                e.close()
+                return
+            }
+        }
+        
+        Uploader uploader = new HashListUploader(e, fullInfoHash, request)
+        eventBus.publish(new UploadEvent(uploader : uploader))
+        try {
+            uploader.respond()
+        } finally {
+            eventBus.publish(new UploadFinishedEvent(uploader : uploader))
+        }
+        
+        // proceed with content
+        while(true) {
+            byte[] get = new byte[4]
+            dis.readFully(get)
+            if (get != "GET ".getBytes(StandardCharsets.US_ASCII)) {
+                log.warning("received a method other than GET on subsequent call")
+                e.close()
+                return
+            }
+            dis.readFully(infoHashStringBytes)
+            infoHashString = new String(infoHashStringBytes, StandardCharsets.US_ASCII)
+            log.info("Responding to upload request for root $infoHashString")
+
+            infoHashRoot = Base64.decode(infoHashString)
+            infoHash = new InfoHash(infoHashRoot)
+            sharedFiles = fileManager.getSharedFiles(infoHashRoot)
+            downloader = downloadManager.downloaders.get(infoHash)
+            if (downloader == null && (sharedFiles == null || sharedFiles.isEmpty())) {
+                log.info "file not found"
+                e.getOutputStream().write("404 File Not Found\r\n\r\n".getBytes(StandardCharsets.US_ASCII))
+                e.getOutputStream().flush()
+                e.close()
+                return
+            }
+
+            rn = new byte[2]
+            dis.readFully(rn)
+            if (rn != "\r\n".getBytes(StandardCharsets.US_ASCII)) {
+                log.warning("Malformed GET header")
+                e.close()
+                return
+            }
+
+            request = Request.parseContentRequest(new InfoHash(infoHashRoot), e.getInputStream())
+            if (request.downloader != null && request.downloader.destination != e.destination) {
+                log.info("Downloader persona doesn't match their destination")
+                e.close()
+                return
+            }
+            
+            if (request.have)
+                eventBus.publish(new SourceDiscoveredEvent(infoHash : request.infoHash, source : request.downloader))
+            
+            Mesh mesh
+            File file
+            int pieceSize
+            if (downloader != null) {
+                mesh = meshManager.get(infoHash)
+                file = downloader.incompleteFile
+                pieceSize = downloader.pieceSizePow2
+            } else {
+                SharedFile sharedFile = sharedFiles.iterator().next();
+                mesh = meshManager.getOrCreate(request.infoHash, sharedFile.NPieces)
+                file = sharedFile.file
+                pieceSize = sharedFile.pieceSize
+            }
+
+            uploader = new ContentUploader(file, request, e, mesh, pieceSize)
+            eventBus.publish(new UploadEvent(uploader : uploader))
+            try {
+                uploader.respond()
+            } finally {
+                eventBus.publish(new UploadFinishedEvent(uploader : uploader))
+            }
+        }
     }
 }
 

core/src/main/groovy/com/muwire/core/upload/Uploader.groovy

@@ -8,52 +8,28 @@ import java.nio.file.StandardOpenOption
 
 import com.muwire.core.connection.Endpoint
 
-class Uploader {
+abstract class Uploader {
-    private final File file
+    protected final Endpoint endpoint
-    private final Request request
+    protected ByteBuffer mapped
-    private final Endpoint endpoint
-    private ByteBuffer mapped
     
-    Uploader(File file, Request request, Endpoint endpoint) {
+    Uploader(Endpoint endpoint) {
-        this.file = file
-        this.request = request
         this.endpoint = endpoint
     }
     
-    void respond() {
+    abstract void respond()
-        OutputStream os = endpoint.getOutputStream()
+        
-        Range range = request.getRange()
-        if (range.start >= file.length() || range.end >= file.length()) {
-            os.write("416 Range Not Satisfiable\r\n\r\n".getBytes(StandardCharsets.US_ASCII))
-            os.flush()
-            return
-        }
-
-        os.write("200 OK\r\n".getBytes(StandardCharsets.US_ASCII))
-        os.write("Content-Range: $range.start-$range.end\r\n\r\n".getBytes(StandardCharsets.US_ASCII))        
-
-        FileChannel channel
-        try {
-            channel = Files.newByteChannel(file.toPath(), EnumSet.of(StandardOpenOption.READ))
-            mapped = channel.map(FileChannel.MapMode.READ_ONLY, range.start, range.end - range.start + 1)
-            byte [] tmp = new byte[0x1 << 13]
-            while(mapped.hasRemaining()) {
-                int start = mapped.position()
-                synchronized(this) {
-                    mapped.get(tmp, 0, Math.min(tmp.length, mapped.remaining()))
-                }
-                int read = mapped.position() - start
-                endpoint.getOutputStream().write(tmp, 0, read)
-            }
-        } finally {
-            try {channel?.close() } catch (IOException ignored) {}
-            endpoint.getOutputStream().flush()
-        }     
-    }
-    
     public synchronized int getPosition() {
         if (mapped == null)
             return -1
         mapped.position()
     }
+    
+    abstract String getName();
+    
+    /**
+     * @return an integer between 0 and 100
+     */
+    abstract int getProgress();
+    
+    abstract String getDownloader();
 }

core/src/main/groovy/com/muwire/core/util/DataUtil.groovy

@@ -4,6 +4,8 @@ import java.nio.charset.StandardCharsets
 
 import com.muwire.core.Constants
 
+import net.i2p.data.Base64
+
 class DataUtil {
 	
 	private final static int MAX_SHORT = (0x1 << 16) - 1
@@ -79,4 +81,32 @@ class DataUtil {
         }
         new String(baos.toByteArray(), StandardCharsets.US_ASCII)
     }
+    
+    public static String encodeXHave(List<Integer> pieces, int totalPieces) {
+        int bytes = totalPieces / 8
+        if (totalPieces % 8 != 0)
+            bytes++
+        byte[] raw = new byte[bytes]
+        pieces.each {
+            int byteIdx = it / 8
+            int offset = it % 8
+            int mask = 0x80 >>> offset
+            raw[byteIdx] |= mask
+        }
+        Base64.encode(raw)
+    }
+    
+    public static List<Integer> decodeXHave(String xHave) {
+        byte [] availablePieces = Base64.decode(xHave)
+        List<Integer> available = new ArrayList<>()
+        availablePieces.eachWithIndex {b, i ->
+            for (int j = 0; j < 8 ; j++) {
+                byte mask = 0x80 >>> j
+                if ((b & mask) == mask) {
+                    available.add(i * 8 + j)
+                }
+            }
+        }
+        available
+    }
 }

core/src/main/groovy/com/muwire/core/util/JULLog.groovy

@@ -55,21 +55,21 @@ class JULLog extends Log {
 
     @Override
     public boolean shouldDebug() {
-        level.intValue().intValue() >= Level.FINE.intValue()
+        level.intValue().intValue() <= Level.FINE.intValue()
     }
 
     @Override
     public boolean shouldInfo() {
-        level.intValue().intValue() >= Level.INFO.intValue()
+        level.intValue().intValue() <= Level.INFO.intValue()
     }
 
     @Override
     public boolean shouldWarn() {
-        level.intValue().intValue() >= Level.WARNING.intValue()
+        level.intValue().intValue() <= Level.WARNING.intValue()
     }
 
     @Override
     public boolean shouldError() {
-        level.intValue().intValue() >= Level.SEVERE.intValue()
+        level.intValue().intValue() <= Level.SEVERE.intValue()
     }
 }

core/src/main/java/com/muwire/core/DownloadedFile.java

@@ -9,8 +9,8 @@ public class DownloadedFile extends SharedFile {
 	
 	private final Set<Destination> sources;
 
-	public DownloadedFile(File file, InfoHash infoHash, Set<Destination> sources) {
+	public DownloadedFile(File file, InfoHash infoHash, int pieceSize, Set<Destination> sources) {
-		super(file, infoHash);
+		super(file, infoHash, pieceSize);
 		this.sources = sources;
 	}
 	

core/src/main/java/com/muwire/core/InfoHash.java

@@ -7,6 +7,7 @@ import java.util.Arrays;
 import java.util.List;
 
 import net.i2p.data.Base32;
+import net.i2p.data.Base64;
 
 public class InfoHash {
 
@@ -76,14 +77,16 @@ public class InfoHash {
 	}
 	
 	public String toString() {
-		String rv = "InfoHash[root:"+Base32.encode(root) + " hashList:";
+		String rv = "InfoHash[root:"+Base64.encode(root) + " hashList:";
-		List<String> b32HashList = new ArrayList<>(hashList.length / SIZE);
+		List<String> b64HashList = new ArrayList<>();
-		byte [] tmp = new byte[SIZE];
+		if (hashList != null) {       
-		for (int i = 0; i < hashList.length / SIZE; i++) {
+		    byte [] tmp = new byte[SIZE];
-			System.arraycopy(hashList, SIZE * i, tmp, 0, SIZE);
+		    for (int i = 0; i < hashList.length / SIZE; i++) {
-			b32HashList.add(Base32.encode(tmp));
+		        System.arraycopy(hashList, SIZE * i, tmp, 0, SIZE);
+		        b64HashList.add(Base64.encode(tmp));
+		    }
 		}
-		rv += b32HashList.toString();
+		rv += b64HashList.toString();
 		rv += "]";
 		return rv;
 	}

core/src/main/java/com/muwire/core/SharedFile.java

@@ -6,10 +6,12 @@ public class SharedFile {
 
 	private final File file;
 	private final InfoHash infoHash;
+	private final int pieceSize;
 	
-	public SharedFile(File file, InfoHash infoHash) {
+	public SharedFile(File file, InfoHash infoHash, int pieceSize) {
 		this.file = file;
 		this.infoHash = infoHash;
+		this.pieceSize = pieceSize;
 	}
 
 	public File getFile() {
@@ -20,4 +22,29 @@ public class SharedFile {
 		return infoHash;
 	}
 	
+	public int getPieceSize() {
+	    return pieceSize;
+	}
+	
+	public int getNPieces() {
+	    long length = file.length();
+	    int rawPieceSize = 0x1 << pieceSize;
+	    int rv = (int) (length / rawPieceSize);
+	    if (length % pieceSize != 0)
+	        rv++;
+	    return rv;
+	}
+	
+	@Override
+	public int hashCode() {
+	    return file.hashCode() ^ infoHash.hashCode();
+	}
+	
+	@Override
+	public boolean equals(Object o) {
+	    if (!(o instanceof SharedFile))
+	        return false;
+	    SharedFile other = (SharedFile)o;
+	    return file.equals(other.file) && infoHash.equals(other.infoHash);
+	}
 }

core/src/test/groovy/com/muwire/core/connection/ConnectionAcceptorTest.groovy

@@ -43,6 +43,9 @@ class ConnectionAcceptorTest {
     
     def uploadManagerMock
     UploadManager uploadManager
+    
+    def connectionEstablisherMock
+    ConnectionEstablisher connectionEstablisher
 	
 	ConnectionAcceptor acceptor
 	List<ConnectionEvent> connectionEvents
@@ -57,6 +60,7 @@ class ConnectionAcceptorTest {
 		trustServiceMock = new MockFor(TrustService.class)
         searchManagerMock = new MockFor(SearchManager.class)
         uploadManagerMock = new MockFor(UploadManager.class)
+        connectionEstablisherMock = new MockFor(ConnectionEstablisher.class)
 	}
 	
 	@After
@@ -68,6 +72,7 @@ class ConnectionAcceptorTest {
 		trustServiceMock.verify trustService
         searchManagerMock.verify searchManager
         uploadManagerMock.verify uploadManager
+        connectionEstablisherMock.verify connectionEstablisher
 		Thread.sleep(100)
 	}
 	
@@ -87,8 +92,10 @@ class ConnectionAcceptorTest {
 		trustService = trustServiceMock.proxyInstance()
         searchManager = searchManagerMock.proxyInstance()
         uploadManager = uploadManagerMock.proxyInstance()
+        connectionEstablisher = connectionEstablisherMock.proxyInstance()
 		
-		acceptor = new ConnectionAcceptor(eventBus, connectionManager, settings, i2pAcceptor, hostCache, trustService, searchManager, uploadManager)
+		acceptor = new ConnectionAcceptor(eventBus, connectionManager, settings, i2pAcceptor, 
+            hostCache, trustService, searchManager, uploadManager, connectionEstablisher)
 		acceptor.start()
 		Thread.sleep(100)
 	}
@@ -108,6 +115,7 @@ class ConnectionAcceptorTest {
 			new Endpoint(destinations.dest1, is, os, null)
 		}
 		i2pAcceptorMock.demand.accept { Thread.sleep(Integer.MAX_VALUE) }
+        connectionEstablisherMock.demand.isInProgress(destinations.dest1) { false }
         connectionManagerMock.demand.isConnected { dest ->
             assert dest == destinations.dest1
             false
@@ -150,6 +158,7 @@ class ConnectionAcceptorTest {
 			new Endpoint(destinations.dest1, is, os, null)
 		}
 		i2pAcceptorMock.demand.accept { Thread.sleep(Integer.MAX_VALUE) }
+        connectionEstablisherMock.demand.isInProgress(destinations.dest1) { false }
         connectionManagerMock.demand.isConnected { dest ->
             assert dest == destinations.dest1
             false
@@ -264,6 +273,7 @@ class ConnectionAcceptorTest {
 			new Endpoint(destinations.dest1, is, os, null)
 		}
 		i2pAcceptorMock.demand.accept { Thread.sleep(Integer.MAX_VALUE) }
+        connectionEstablisherMock.demand.isInProgress(destinations.dest1) { false }
         connectionManagerMock.demand.isConnected { dest ->
             assert dest == destinations.dest1
             false
@@ -310,6 +320,7 @@ class ConnectionAcceptorTest {
 			new Endpoint(destinations.dest1, is, os, null)
 		}
 		i2pAcceptorMock.demand.accept { Thread.sleep(Integer.MAX_VALUE) }
+        connectionEstablisherMock.demand.isInProgress(destinations.dest1) { false }
         connectionManagerMock.demand.isConnected { dest ->
             assert dest == destinations.dest1
             false
@@ -356,6 +367,7 @@ class ConnectionAcceptorTest {
 			new Endpoint(destinations.dest1, is, os, null)
 		}
 		i2pAcceptorMock.demand.accept { Thread.sleep(Integer.MAX_VALUE) }
+        connectionEstablisherMock.demand.isInProgress(destinations.dest1) { false }
         connectionManagerMock.demand.isConnected { dest ->
             assert dest == destinations.dest1
             false

core/src/test/groovy/com/muwire/core/download/DownloadSessionTest.groovy

@@ -1,21 +1,30 @@
 package com.muwire.core.download
 
+import static org.junit.Assert.fail
+
 import org.junit.After
+import org.junit.Before
 import org.junit.Test
 
+import com.muwire.core.EventBus
 import com.muwire.core.InfoHash
+import com.muwire.core.Persona
+import com.muwire.core.Personas
 import com.muwire.core.connection.Endpoint
 import com.muwire.core.files.FileHasher
 import static com.muwire.core.util.DataUtil.readTillRN
+import static com.muwire.core.util.DataUtil.encodeXHave
 
 import net.i2p.data.Base64
+import net.i2p.util.ConcurrentHashSet
 
 class DownloadSessionTest {
     
+    private EventBus eventBus
     private File source, target
     private InfoHash infoHash
     private Endpoint endpoint
-    private Pieces pieces, claimed
+    private Pieces pieces
     private String rootBase64
     
     private DownloadSession session
@@ -24,6 +33,16 @@ class DownloadSessionTest {
     private InputStream fromDownloader, fromUploader
     private OutputStream toDownloader, toUploader
     
+    private volatile boolean performed
+    private Set<Integer> available = new ConcurrentHashSet<>()
+    private volatile IOException thrown
+    
+    
+    @Before
+    public void setUp() {
+        eventBus = new EventBus()
+    }
+    
     private void initSession(int size, def claimedPieces = []) {
         Random r = new Random()
         byte [] content = new byte[size]
@@ -48,8 +67,7 @@ class DownloadSessionTest {
         else
             nPieces = size / pieceSize + 1
         pieces = new Pieces(nPieces)
-        claimed = new Pieces(nPieces)
+        claimedPieces.each {pieces.claimed.set(it)}
-        claimedPieces.each {claimed.markDownloaded(it)}
         
         fromDownloader = new PipedInputStream()
         fromUploader = new PipedInputStream()
@@ -57,12 +75,20 @@ class DownloadSessionTest {
         toUploader = new PipedOutputStream(fromDownloader)
         endpoint = new Endpoint(null, fromUploader, toUploader, null)
         
-        session = new DownloadSession("",pieces, claimed, infoHash, endpoint, target, pieceSize, size)
+        session = new DownloadSession(eventBus, "",pieces, infoHash, endpoint, target, pieceSize, size, available)
-        downloadThread = new Thread( { session.request() } as Runnable)
+        downloadThread = new Thread( { perform() } as Runnable)
         downloadThread.setDaemon(true)
         downloadThread.start()
     }
     
+    private void perform() {
+        try { 
+            performed = session.request()
+        } catch (IOException e) {
+            thrown = e
+        }
+    }
+    
     @After
     public void teardown() {
         source?.delete()
@@ -77,6 +103,7 @@ class DownloadSessionTest {
         assert "GET $rootBase64" == readTillRN(fromDownloader)
         assert "Range: 0-19" == readTillRN(fromDownloader)
         readTillRN(fromDownloader)
+        readTillRN(fromDownloader)
         assert "" == readTillRN(fromDownloader)
         
         toDownloader.write("200 OK\r\n".bytes)
@@ -88,6 +115,9 @@ class DownloadSessionTest {
         
         assert pieces.isComplete()
         assert target.bytes == source.bytes
+        assert performed
+        assert available.isEmpty()
+        assert thrown == null
     }
     
     @Test
@@ -99,6 +129,7 @@ class DownloadSessionTest {
         assert "GET $rootBase64" == readTillRN(fromDownloader)
         readTillRN(fromDownloader)
         readTillRN(fromDownloader)
+        readTillRN(fromDownloader)
         assert "" == readTillRN(fromDownloader)
         
         toDownloader.write("200 OK\r\n".bytes)
@@ -109,6 +140,9 @@ class DownloadSessionTest {
         Thread.sleep(150)
         assert pieces.isComplete()
         assert target.bytes == source.bytes
+        assert performed
+        assert available.isEmpty()
+        assert thrown == null
     }
     
     @Test
@@ -126,6 +160,7 @@ class DownloadSessionTest {
         assert (start == 0 && end == ((1 << pieceSize) - 1)) || 
             (start == (1 << pieceSize) && end == (1 << pieceSize))
         
+        readTillRN(fromDownloader)
         readTillRN(fromDownloader)
         assert "" == readTillRN(fromDownloader)
         
@@ -139,6 +174,9 @@ class DownloadSessionTest {
         Thread.sleep(150)
         assert !pieces.isComplete()
         assert 1 == pieces.donePieces()
+        assert performed
+        assert available.isEmpty()
+        assert thrown == null
     }
     
     @Test
@@ -146,7 +184,10 @@ class DownloadSessionTest {
         initSession(20, [0])
         long now = System.currentTimeMillis()
         downloadThread.join(100)
-        assert 100 > (System.currentTimeMillis() - now)
+        assert 100 >= (System.currentTimeMillis() - now)
+        assert !performed
+        assert available.isEmpty()
+        assert thrown == null
     }
     
     @Test
@@ -154,7 +195,7 @@ class DownloadSessionTest {
         int pieceSize = FileHasher.getPieceSize(1)
         int size = (1 << pieceSize) * 10
         initSession(size, [1,2,3,4,5,6,7,8,9])
-        assert !claimed.isMarked(0)
+        assert !pieces.claimed.get(0)
         
         assert "GET $rootBase64" == readTillRN(fromDownloader)
         String range = readTillRN(fromDownloader)
@@ -162,7 +203,131 @@ class DownloadSessionTest {
         int start = Integer.parseInt(matcher[0][1])
         int end = Integer.parseInt(matcher[0][2])
         
-        assert claimed.isMarked(0)
+        assert pieces.claimed.get(0)
         assert start == 0 && end == (1 << pieceSize) - 1
     }
+    
+    @Test
+    public void test416NoHave() {
+        initSession(20)
+        readAllHeaders(fromDownloader)
+        
+        toDownloader.write("416 don't have it\r\n\r\n".bytes)
+        toDownloader.flush()
+        Thread.sleep(150)
+        assert !performed
+        assert available.isEmpty()
+        assert thrown != null
+    }
+    
+    @Test
+    public void test416Have() {
+        initSession(20)
+        readAllHeaders(fromDownloader)
+        
+        toDownloader.write("416 don't have it\r\n".bytes)
+        toDownloader.write("X-Have: ${encodeXHave([0], 1)}\r\n\r\n".bytes)
+        toDownloader.flush()
+        
+        Thread.sleep(150)
+        assert performed
+        assert available.contains(0)
+        assert thrown == null
+    }
+    
+    @Test
+    public void test416Have2Pieces() {
+        int pieceSize = FileHasher.getPieceSize(1)
+        int size = (1 << pieceSize) + 1
+        initSession(size)
+        readAllHeaders(fromDownloader)
+        
+        toDownloader.write("416 don't have it\r\n".bytes)
+        toDownloader.write("X-Have: ${encodeXHave([1], 2)}\r\n\r\n".bytes)
+        toDownloader.flush()
+        
+        Thread.sleep(150)
+        assert performed
+        assert available.contains(1)
+        assert thrown == null
+    }
+    
+    @Test
+    public void test200TwoPieces1Available() {
+        int pieceSize = FileHasher.getPieceSize(1)
+        int size = (1 << pieceSize) * 9 + 1
+        initSession(size)
+        
+        Set<String> headers = readAllHeaders(fromDownloader)
+        def matcher = null
+        headers.each {
+            if (it.startsWith("Range"))
+                matcher = (it =~ /^Range: (\d+)-(\d+)$/)
+        }
+        assert matcher.groupCount() > 0
+        int start = Integer.parseInt(matcher[0][1])
+        int end = Integer.parseInt(matcher[0][2])
+        
+        if (start == 0) 
+            fail("inconlcusive")
+        
+        toDownloader.write("416 don't have it \r\n".bytes)
+        toDownloader.write("X-Have: ${encodeXHave([0],2)}\r\n\r\n".bytes)
+        toDownloader.flush()
+        downloadThread.join()
+        
+        assert performed
+        performed = false
+        assert available.contains(0)
+        assert thrown == null
+        
+        // request same session
+        downloadThread = new Thread( { perform() } as Runnable)
+        downloadThread.setDaemon(true)
+        downloadThread.start()
+        
+        Thread.sleep(150)
+        
+        headers = readAllHeaders(fromDownloader)
+        matcher = null
+        headers.each {
+            if (it.startsWith("Range"))
+            matcher = (it =~ /^Range: (\d+)-(\d+)$/)
+        }
+        assert matcher.groupCount() > 0
+        start = Integer.parseInt(matcher[0][1])
+        end = Integer.parseInt(matcher[0][2])
+        assert start == 0
+    }
+    
+    @Test
+    public void testXAlt() throws Exception {
+        Personas personas = new Personas()
+        def sources = []
+        def listener = new Object() {
+            public void onSourceDiscoveredEvent(SourceDiscoveredEvent e) {
+                sources << e.source
+            }
+        }
+        eventBus.register(SourceDiscoveredEvent.class, listener)
+        
+        initSession(20)
+        readAllHeaders(fromDownloader)
+        toDownloader.write("416 don't have it\r\n".bytes)
+        toDownloader.write("X-Alt: ${personas.persona1.toBase64()},${personas.persona2.toBase64()}\r\n\r\n".bytes)
+        toDownloader.flush()
+        
+        Thread.sleep(150)
+        assert sources.contains(personas.persona1)
+        assert sources.contains(personas.persona2)
+        assert 2 == sources.size()
+    }
+    
+    private static Set<String> readAllHeaders(InputStream is) {
+        Set<String> rv = new HashSet<>()
+        String header
+        while((header = readTillRN(is)) != "")
+            rv.add(header)
+        rv
+    }
 }

core/src/test/groovy/com/muwire/core/download/PiecesTest.groovy

@@ -16,7 +16,7 @@ class PiecesTest {
     public void testSinglePiece() {
         pieces = new Pieces(1)
         assert !pieces.isComplete()
-        assert pieces.getRandomPiece() == 0
+        assert pieces.claim() == 0
         pieces.markDownloaded(0)
         assert pieces.isComplete()
     }
@@ -25,13 +25,28 @@ class PiecesTest {
     public void testTwoPieces() {
         pieces = new Pieces(2)
         assert !pieces.isComplete()
-        int piece = pieces.getRandomPiece()
+        int piece = pieces.claim()
         assert piece == 0 || piece == 1
         pieces.markDownloaded(piece)
         assert !pieces.isComplete()
-        int piece2 = pieces.getRandomPiece()
+        int piece2 = pieces.claim()
         assert piece != piece2
         pieces.markDownloaded(piece2)
         assert pieces.isComplete()
     }
+    
+    @Test
+    public void testClaimAvailable() {
+        pieces = new Pieces(2)
+        int claimed = pieces.claim([0].toSet())
+        assert claimed == 0
+        assert -1 == pieces.claim([0].toSet())
+    }
+    
+    @Test
+    public void testClaimNoneAvailable() {
+        pieces = new Pieces(20)
+        int claimed = pieces.claim()
+        assert -1 == pieces.claim([claimed].toSet())
+    }
 }

core/src/test/groovy/com/muwire/core/files/FileHasherTest.groovy

@@ -24,9 +24,9 @@ class FileHasherTest extends GroovyTestCase {
 	
 	@Test
 	void testPieceSize() {
-		assert 18 == FileHasher.getPieceSize(1000000)
+		assert 17 == FileHasher.getPieceSize(1000000)
-		assert 20 == FileHasher.getPieceSize(100000000)
+		assert 17 == FileHasher.getPieceSize(100000000)
-		assert 30 == FileHasher.getPieceSize(FileHasher.MAX_SIZE)
+		assert 24 == FileHasher.getPieceSize(FileHasher.MAX_SIZE)
 		shouldFail IllegalArgumentException, {
 			FileHasher.getPieceSize(Long.MAX_VALUE)
 		}
@@ -48,7 +48,7 @@ class FileHasherTest extends GroovyTestCase {
 		fos.write b
 		fos.close()
 		def ih = hasher.hashFile tmp
-		assert ih.getHashList().length == 32
+		assert ih.getHashList().length == 64
 	}
 	
 	@Test
@@ -58,7 +58,7 @@ class FileHasherTest extends GroovyTestCase {
 		fos.write b
 		fos.close()
 		def ih = hasher.hashFile tmp
-		assert ih.getHashList().length == 64
+		assert ih.getHashList().length == 96
 	}
 	
 	@Test
@@ -68,7 +68,7 @@ class FileHasherTest extends GroovyTestCase {
 		fos.write b
 		fos.close()
 		def ih = hasher.hashFile tmp
-		assert ih.getHashList().length == 64
+		assert ih.getHashList().length == 128
 	}
 	
 	@Test
@@ -78,6 +78,6 @@ class FileHasherTest extends GroovyTestCase {
 		fos.write b
 		fos.close()
 		def ih = hasher.hashFile tmp
-		assert ih.getHashList().length == 32 * 3
+		assert ih.getHashList().length == 160
 	}
 }

core/src/test/groovy/com/muwire/core/files/FileManagerTest.groovy

@@ -5,6 +5,7 @@ import org.junit.Test
 
 import com.muwire.core.EventBus
 import com.muwire.core.InfoHash
+import com.muwire.core.MuWireSettings
 import com.muwire.core.SharedFile
 import com.muwire.core.search.ResultsEvent
 import com.muwire.core.search.SearchEvent
@@ -26,7 +27,7 @@ class FileManagerTest {
 	void before() {
 		eventBus = new EventBus()
 		eventBus.register(ResultsEvent.class, listener)
-		manager = new FileManager(eventBus)
+		manager = new FileManager(eventBus, new MuWireSettings())
 		results = null
 	}
 	
@@ -34,7 +35,7 @@ class FileManagerTest {
 	void testHash1Result() {
 		File f = new File("a b.c")
 		InfoHash ih = InfoHash.fromHashList(new byte[32])
-		SharedFile sf = new SharedFile(f,ih)
+		SharedFile sf = new SharedFile(f,ih, 0)
 		FileHashedEvent fhe = new FileHashedEvent(sharedFile: sf)
 		manager.onFileHashedEvent(fhe)
 		
@@ -53,8 +54,8 @@ class FileManagerTest {
 	@Test
 	void testHash2Results() {
 		InfoHash ih = InfoHash.fromHashList(new byte[32])
-		SharedFile sf1 = new SharedFile(new File("a b.c"), ih)
+		SharedFile sf1 = new SharedFile(new File("a b.c"), ih, 0)
-		SharedFile sf2 = new SharedFile(new File("d e.f"), ih)
+		SharedFile sf2 = new SharedFile(new File("d e.f"), ih, 0)
 		manager.onFileLoadedEvent new FileLoadedEvent(loadedFile : sf1)
 		manager.onFileLoadedEvent new FileLoadedEvent(loadedFile : sf2)
 
@@ -75,7 +76,7 @@ class FileManagerTest {
 	void testHash0Results() {
 		File f = new File("a b.c")
 		InfoHash ih = InfoHash.fromHashList(new byte[32])
-		SharedFile sf = new SharedFile(f,ih)
+		SharedFile sf = new SharedFile(f,ih, 0)
 		FileHashedEvent fhe = new FileHashedEvent(sharedFile: sf)
 		manager.onFileHashedEvent(fhe)
 		
@@ -89,7 +90,7 @@ class FileManagerTest {
 	void testKeyword1Result() {
 		File f = new File("a b.c")
 		InfoHash ih = InfoHash.fromHashList(new byte[32])
-		SharedFile sf = new SharedFile(f,ih)
+		SharedFile sf = new SharedFile(f,ih,0)
 		FileHashedEvent fhe = new FileHashedEvent(sharedFile: sf)
 		manager.onFileHashedEvent(fhe)
 		
@@ -107,12 +108,12 @@ class FileManagerTest {
 	void testKeyword2Results() {
 		File f1 = new File("a b.c")
 		InfoHash ih1 = InfoHash.fromHashList(new byte[32])
-		SharedFile sf1 = new SharedFile(f1, ih1)
+		SharedFile sf1 = new SharedFile(f1, ih1, 0)
 		manager.onFileLoadedEvent new FileLoadedEvent(loadedFile: sf1)
 		
 		File f2 = new File("c d.e")
 		InfoHash ih2 = InfoHash.fromHashList(new byte[64])
-		SharedFile sf2 = new SharedFile(f2, ih2)
+		SharedFile sf2 = new SharedFile(f2, ih2, 0)
 		manager.onFileLoadedEvent new FileLoadedEvent(loadedFile: sf2)
 		
 		UUID uuid = UUID.randomUUID()
@@ -130,7 +131,7 @@ class FileManagerTest {
 	void testKeyword0Results() {
 		File f = new File("a b.c")
 		InfoHash ih = InfoHash.fromHashList(new byte[32])
-		SharedFile sf = new SharedFile(f,ih)
+		SharedFile sf = new SharedFile(f,ih,0)
 		FileHashedEvent fhe = new FileHashedEvent(sharedFile: sf)
 		manager.onFileHashedEvent(fhe)
 		
@@ -143,8 +144,8 @@ class FileManagerTest {
 	@Test
 	void testRemoveFileExistingHash() {
 		InfoHash ih = InfoHash.fromHashList(new byte[32])
-		SharedFile sf1 = new SharedFile(new File("a b.c"), ih)
+		SharedFile sf1 = new SharedFile(new File("a b.c"), ih, 0)
-		SharedFile sf2 = new SharedFile(new File("d e.f"), ih)
+		SharedFile sf2 = new SharedFile(new File("d e.f"), ih, 0)
 		manager.onFileLoadedEvent new FileLoadedEvent(loadedFile : sf1)
 		manager.onFileLoadedEvent new FileLoadedEvent(loadedFile : sf2)
 		
@@ -161,12 +162,12 @@ class FileManagerTest {
 	void testRemoveFile() {
 		File f1 = new File("a b.c")
 		InfoHash ih1 = InfoHash.fromHashList(new byte[32])
-		SharedFile sf1 = new SharedFile(f1, ih1)
+		SharedFile sf1 = new SharedFile(f1, ih1, 0)
 		manager.onFileLoadedEvent new FileLoadedEvent(loadedFile: sf1)
 		
 		File f2 = new File("c d.e")
 		InfoHash ih2 = InfoHash.fromHashList(new byte[64])
-		SharedFile sf2 = new SharedFile(f2, ih2)
+		SharedFile sf2 = new SharedFile(f2, ih2, 0)
 		manager.onFileLoadedEvent new FileLoadedEvent(loadedFile: sf2)
 		
 		manager.onFileUnsharedEvent new FileUnsharedEvent(unsharedFile: sf2)

core/src/test/groovy/com/muwire/core/files/HasherServiceTest.groovy

@@ -8,6 +8,7 @@ import org.junit.Before
 import org.junit.Test
 
 import com.muwire.core.EventBus
+import com.muwire.core.MuWireSettings
 
 class HasherServiceTest {
 
@@ -24,8 +25,9 @@ class HasherServiceTest {
 	void before() {
         eventBus = new EventBus()
 		hasher = new FileHasher()
-		service = new HasherService(hasher, eventBus)
+		service = new HasherService(hasher, eventBus, new FileManager(eventBus, new MuWireSettings()))
         eventBus.register(FileHashedEvent.class, listener)
+        eventBus.register(FileSharedEvent.class, service)
 		service.start()
 	}
 	

core/src/test/groovy/com/muwire/core/files/PersisterServiceLoadingTest.groovy

@@ -78,7 +78,7 @@ class PersisterServiceLoadingTest {
 		persisted.write json
 		
 		PersisterService ps = new PersisterService(persisted, eventBus, 100, null)
-		ps.start()
+		ps.onUILoadedEvent(null)
 		Thread.sleep(2000)
 		
 		assert listener.publishedFiles.size() == 1
@@ -99,7 +99,7 @@ class PersisterServiceLoadingTest {
 		FileHasher fh = new FileHasher()
 		InfoHash ih1 = fh.hashFile(sharedFile1)
 		
-		assert ih1.getHashList().length == 2 * 32
+		assert ih1.getHashList().length == 96
 		
 		def json = [:]
 		json.file = getSharedFileJsonName(sharedFile1)
@@ -111,7 +111,9 @@ class PersisterServiceLoadingTest {
 		String hash1 = Base64.encode(tmp)
 		System.arraycopy(ih1.getHashList(), 32, tmp, 0, 32)
 		String hash2 = Base64.encode(tmp)
-		json.hashList = [hash1, hash2]
+        System.arraycopy(ih1.getHashList(), 64, tmp, 0, 32)
+        String hash3 = Base64.encode(tmp)
+		json.hashList = [hash1, hash2, hash3]
 		
 		json = JsonOutput.toJson(json)
 		
@@ -119,7 +121,7 @@ class PersisterServiceLoadingTest {
 		persisted.write json
 		
 		PersisterService ps = new PersisterService(persisted, eventBus, 100, null)
-		ps.start()
+		ps.onUILoadedEvent(null)
 		Thread.sleep(2000)
 		
 		assert listener.publishedFiles.size() == 1
@@ -161,7 +163,7 @@ class PersisterServiceLoadingTest {
 		persisted.append "$json2\n"
 		
 		PersisterService ps = new PersisterService(persisted, eventBus, 100, null)
-		ps.start()
+		ps.onUILoadedEvent(null)
 		Thread.sleep(2000)
 		
 		assert listener.publishedFiles.size() == 2
@@ -193,7 +195,7 @@ class PersisterServiceLoadingTest {
 		persisted.write json1
 		
 		PersisterService ps = new PersisterService(persisted, eventBus, 100, null)
-		ps.start()
+		ps.onUILoadedEvent(null)
 		Thread.sleep(2000)
 		
 		assert listener.publishedFiles.size() == 1

core/src/test/groovy/com/muwire/core/files/PersisterServiceSavingTest.groovy

@@ -8,6 +8,7 @@ import com.muwire.core.Destinations
 import com.muwire.core.DownloadedFile
 import com.muwire.core.EventBus
 import com.muwire.core.InfoHash
+import com.muwire.core.MuWireSettings
 import com.muwire.core.SharedFile
 import com.muwire.core.util.DataUtil
 
@@ -31,7 +32,7 @@ class PersisterServiceSavingTest {
 		f = new File("build.gradle")
 		f = f.getCanonicalFile()
 		ih = fh.hashFile(f)
-		fileSource = new FileManager(eventBus) {
+		fileSource = new FileManager(eventBus, new MuWireSettings()) {
 					Map<File, SharedFile> getSharedFiles() {
 						Map<File, SharedFile> rv = new HashMap<>()
 						rv.put(f, sf)
@@ -54,10 +55,10 @@ class PersisterServiceSavingTest {
 	
 	@Test
 	void testSavingSharedFile() {
-		sf = new SharedFile(f, ih)
+		sf = new SharedFile(f, ih, 0)
 		
 		ps = new PersisterService(persisted, eventBus, 100, fileSource)
-		ps.start()
+		ps.onUILoadedEvent(null)
 		Thread.sleep(1500)
 	
 		JsonSlurper jsonSlurper = new JsonSlurper()	
@@ -73,10 +74,10 @@ class PersisterServiceSavingTest {
 	@Test
 	void testSavingDownloadedFile() {
 		Destinations dests = new Destinations()
-		sf = new DownloadedFile(f, ih, new HashSet([dests.dest1, dests.dest2]))
+		sf = new DownloadedFile(f, ih, 0, new HashSet([dests.dest1, dests.dest2]))
 		
 		ps = new PersisterService(persisted, eventBus, 100, fileSource)
-		ps.start()
+		ps.onUILoadedEvent(null)
 		Thread.sleep(1500)
 	
 		JsonSlurper jsonSlurper = new JsonSlurper()

core/src/test/groovy/com/muwire/core/search/SearchIndexTest.groovy

@@ -30,7 +30,18 @@ class SearchIndexTest {
 		assert found.size() == 2
 		assert found.contains("a b.c")
 		assert found.contains("c d.e")
+        
 	}
+    
+    @Test
+    public void testDrillDownDoesNotModifyIndex() {
+        initIndex(["a b.c", "c d.e"])
+        index.search(["c","e"])
+        def found = index.search(["c"])
+        assert found.size() == 2
+        assert found.contains("a b.c")
+        assert found.contains("c d.e")
+    }
 	
 	@Test
 	void testDrillDown() {
@@ -72,4 +83,11 @@ class SearchIndexTest {
 		assert found.size() == 1
 		assert found.contains("b c.d")
 	}
+    
+    @Test
+    void testDuplicateTerm() {
+        initIndex(["MuWire-0.3.3.jar"])
+        def found = index.search(["muwire", "0", "3", "jar"])
+        assert found.size() == 1
+    }
 }

core/src/test/groovy/com/muwire/core/upload/RequestParsingTest.groovy

@@ -9,18 +9,18 @@ import com.muwire.core.InfoHash
 
 class RequestParsingTest {
     
-    Request request
+    ContentRequest request
     
     private void fromString(String requestString) {
         def is = new ByteArrayInputStream(requestString.getBytes(StandardCharsets.US_ASCII))
-        request = Request.parse(new InfoHash(new byte[InfoHash.SIZE]), is)
+        request = Request.parseContentRequest(new InfoHash(new byte[InfoHash.SIZE]), is)
     }
 
     
     private static void failed(String requestString) {
         try {
             def is = new ByteArrayInputStream(requestString.getBytes(StandardCharsets.US_ASCII))
-            Request.parse(new InfoHash(new byte[InfoHash.SIZE]), is)
+            Request.parseContentRequest(new InfoHash(new byte[InfoHash.SIZE]), is)
             assert false
         } catch (IOException expected) {}
     }

core/src/test/groovy/com/muwire/core/upload/UploaderTest.groovy

@@ -19,7 +19,7 @@ class UploaderTest {
     InputStream is
     OutputStream os
     
-    Request request
+    ContentRequest request
     Uploader uploader
     
     byte[] inFile
@@ -52,7 +52,7 @@ class UploaderTest {
     }
     
     private void startUpload() {
-        uploader = new Uploader(file, request, endpoint)
+        uploader = new ContentUploader(file, request, endpoint)
         uploadThread = new Thread(uploader.respond() as Runnable)
         uploadThread.setDaemon(true)
         uploadThread.start()
@@ -77,7 +77,7 @@ class UploaderTest {
     @Test
     public void testSmallFile() {
         fillFile(20)
-        request = new Request(range : new Range(0,19))
+        request = new ContentRequest(range : new Range(0,19))
         startUpload()
         assert "200 OK" == readUntilRN()
         assert "Content-Range: 0-19" == readUntilRN()
@@ -92,7 +92,7 @@ class UploaderTest {
     @Test
     public void testRequestMiddle() {
         fillFile(20)
-        request = new Request(range : new Range(5,15))
+        request = new ContentRequest(range : new Range(5,15))
         startUpload()
         assert "200 OK" == readUntilRN()
         assert "Content-Range: 5-15" == readUntilRN()
@@ -108,7 +108,7 @@ class UploaderTest {
     @Test
     public void testOutOfRange() {
         fillFile(20)
-        request = new Request(range : new Range(0,20))
+        request = new ContentRequest(range : new Range(0,20))
         startUpload()
         assert "416 Range Not Satisfiable" == readUntilRN()
         assert "" == readUntilRN()
@@ -118,7 +118,7 @@ class UploaderTest {
     public void testLargeFile() {
         final int length = 0x1 << 14
         fillFile(length)
-        request = new Request(range : new Range(0, length - 1))
+        request = new ContentRequest(range : new Range(0, length - 1))
         startUpload()
         readUntilRN()
         readUntilRN()

doc/design.md

@@ -49,7 +49,7 @@ Files are transferred over HTTP1.1 protocol with some custom headers added for d
 
 ### Mesh management
 
-Download mesh management is identical to Gnutella, except instead of ip addresses MuWire personas are used.  [More information](http://rfc-gnutella.sourceforge.net/developer/tmp/download-mesh.html) 
+Download mesh management is a simplified version of Gnutella's "Alternate Location" system.  For more information see the "download-mesh" document.
 
 ### In-Network updates
 

doc/download-mesh.md

@@ -0,0 +1,15 @@
+# Download Mesh / Partial Sharing
+
+MuWire uses a system similar to Gnutella's "Alternate Location" download mesh management system, however it is simplified to account for I2P's strengths and borrows a bit from BitTorrent's "Have" message.
+
+### "X-Have" header
+
+With every request a downloader makes it sends an "X-Have" header containing the Base64-encoded representation of a bitfield where bits set to 1 represent pieces of the file that the downloader already has.  To make partial file sharing possible, if the uploader does not have the complete file it also sends this header in every response.  If the header is missing it is assumed the uploader has the complete file.
+
+### "X-Alt" header
+
+The uploader can recommend other uploaders to the downloader via the "X-Alt" header.  The format of this header is a comma-separated list of Base64-encoded Personas that have previously reported having at least one piece of the file to the uploader via the "X-Have" header.
+
+### Differences from Gnutella
+
+Unlike Gnutella the uploader is the sole repository where possible sources of the file are tracked.  There is no negative "X-Nalt" header to prevent attacking the download mesh by mass downvoting of sources.

doc/infohash-upgrade.md

@@ -0,0 +1,37 @@
+# InfoHash Upgrade
+
+An infohash is a list of hashes of the pieces of the file.  In MuWire 0.1.0 the piece size is determined by policy based on the file size, with the intention being to keep the list of hashes to maximum 128 in number.  The reason for this is that infohashes get returned with search results, and smaller piece size will result in larger infohash which will slow down the transmission of search results.
+
+### The problem
+
+This presents the following problem - larger files have larger piece sizes: a 2GB file will have a 16MB piece size, a 4GB file 32MB and so on.  Pieces are atomic, i.e. if a download fails halfway through a piece it will resume since the beginning of the piece.  Unfortunately in the current state of I2P the failure rate of streaming connections is too high and transmitting an entire piece over a single connection is not likely to succeed as the size of the piece increases.  This makes downloading multi-gigabyte files nearly impossible.
+
+### Out-of-band request proposal
+
+Barring any improvement to the reliability of I2P connections, the following approach can be used to enable smaller piece sizes and the corresponding increase in download success rate of large files:
+
+* Search results do carry the full infohash of the file, instead they carry just the root and the number of 32-byte hashes in the infohash
+* When a downloader begins a download, it issues a request for the full infohash first.  Only after that is fetched and verified, the download proceeds as usual.
+
+Such approach is more complicated in nature than the current simplistic one, but it has the additional benefit of reducing the size of search results.
+
+### Wire protocol changes
+
+A new request method - "HASHLIST" is introduced.  It is immediately followed by the Base64 encoded root of the infohash and '\r\n'.  The request may contain HTTP headers in the same format as in HTTP 1.1.  One such header may be the X-Persona header, but that is optional.  After all the headers a blank line with just '\r\n' is sent.
+
+The response is identical to that of regular GET request, with the same response codes.  The response may also carry headers, and is also followed by a blank line with '\r\n'.  What follows immediately after that is a binary representation of the hashlist.  After sending the full hashlist, the uploader keeps the connection open in anticipation of the first content GET request.
+
+The downloader verifies the hashlist by hashing it with SHA256 and comparing it to the advertised infohash root.  If there is a match, it proceeds with the rest of the download as in MuWire 0.1.0.
+
+### Necessary changes to MuWire 0.1.0
+
+To accommodate this proposal in a backwards compatible manner, it is necessary to first de-hardcode the piece count computation logic which is currently hardcoded in a few places.  Then it is necessary to:
+
+* persist the piece size to disk when a file is being shared so that it can be returned in search results
+* search queries need to carry a flag of some kind that indicates support for out-of-band infohash support
+  * that in turn requires nodes to support passing of that flag as the queries are being routed through the network
+* the returned results need to indicate whether they are returning a full infohash or just a root; the "version" field in the json can be used for that
+
+### Roadmap
+
+Support for this proposal is currently intended for MuWire 0.2.0.  However, in order to make rollout smooth, in MuWire 0.1.1 support for the first two items will be introduced.  Since there already are users on the network who have shared files without persisting the size of their pieces on disk, those files will not be eligible to participate in this scheme unless re-shared (which implies re-hashing).

doc/wire-protocol.md

@@ -131,12 +131,18 @@ Sent by a leaf or ultrapeer when performing a search.  Contains the reply-to per
     firstHop: false,
     keywords : ["keyword1","keyword2"...]
     infohash: "asdfasdf...",
-    replyTo : "asdfasf...b64"
+    replyTo : "asdfasf...b64",
+    originator : "asfasdf...",
+    "oobHashlist" : true
 }
 ```
 
 A search can contain either the query entered by the user in the UI or the infohash if the user is looking for a specific file.  If both are present, the infohash takes precedence and the keyword query is ignored.
 
+The "originator" field contains the Base64-encoded persona of the originator of the query.  It is used for display purposes only.  The I2P destination in that persona must match the one in the "replyTo" field.
+
+The oobHashlist flag indicates support for out-of-band hashlist delivery, which is not yet implemented.  Nevertheless, this flag gets propagated through the network for future-proofing.
+
 ### Ultrapeer to leaf
 
 The "Search" message is also sent from an ultrapeer to a leaf.
@@ -175,6 +181,8 @@ Search results are sent through and HTTP POST method from the responder to the o
 * The "altlocs" list contains list of alternate personas that the responder thinks may also have the file.
 * The "pieceSize" field is the size of the each individual file piece (except possibly the last) in powers of 2
 
+Results version 1 contain the full hashlist, version 2 does not contain that list.  See the "infohash-upgrade" document for more information.
+
 ### "Who do you trust" query - any node to any node
 (See the "web-of-trust" document for more info on this query)
 

gradle.properties

@@ -1,5 +1,8 @@
 group         = com.muwire
-version       = 0.0.8
+version       = 0.3.6
 groovyVersion = 2.4.15
 slf4jVersion  = 1.7.25
 spockVersion  = 1.1-groovy-2.4
+
+sourceCompatibility=1.8
+targetCompatibility=1.8

gui/griffon-app/controllers/com/muwire/gui/MainFrameController.groovy

@@ -7,13 +7,18 @@ import griffon.core.mvc.MVCGroup
 import griffon.core.mvc.MVCGroupConfiguration
 import griffon.inject.MVCMember
 import griffon.metadata.ArtifactProviderFor
+import net.i2p.data.Base64
+
 import javax.annotation.Nonnull
 import javax.inject.Inject
 
 import com.muwire.core.Constants
 import com.muwire.core.Core
 import com.muwire.core.download.DownloadStartedEvent
+import com.muwire.core.download.UIDownloadCancelledEvent
 import com.muwire.core.download.UIDownloadEvent
+import com.muwire.core.download.UIDownloadPausedEvent
+import com.muwire.core.download.UIDownloadResumedEvent
 import com.muwire.core.search.QueryEvent
 import com.muwire.core.search.SearchEvent
 import com.muwire.core.trust.TrustEvent
@@ -36,21 +41,60 @@ class MainFrameController {
         cardsPanel.getLayout().show(cardsPanel, "search window")
         
         def search = builder.getVariable("search-field").text
+        search = search.trim()
+        if (search.length() == 0)
+            return
         def uuid = UUID.randomUUID()
         Map<String, Object> params = new HashMap<>()
         params["search-terms"] = search
         params["uuid"] = uuid.toString()
         def group = mvcGroup.createMVCGroup("SearchTab", uuid.toString(), params)
         model.results[uuid.toString()] = group
+
+        boolean hashSearch = false
+        byte [] root = null
+        if (search.length() == 44 && search.indexOf(" ") < 0) {
+            try {
+                root = Base64.decode(search)
+                hashSearch = true
+            } catch (Exception e) {
+                // not a hash search
+            }
+        }
         
-        // this can be improved a lot
+        def searchEvent
-        def terms = search.toLowerCase().trim().split(Constants.SPLIT_PATTERN)
+        if (hashSearch) {
-        def searchEvent = new SearchEvent(searchTerms : terms, uuid : uuid)
+            searchEvent = new SearchEvent(searchHash : root, uuid : uuid, oobInfohash: true)
+        } else {
+            // this can be improved a lot
+            def replaced = search.toLowerCase().trim().replaceAll(Constants.SPLIT_PATTERN, " ")
+            def terms = replaced.split(" ")
+            def nonEmpty = []
+            terms.each { if (it.length() > 0) nonEmpty << it }
+            searchEvent = new SearchEvent(searchTerms : nonEmpty, uuid : uuid, oobInfohash: true)
+        }
         core.eventBus.publish(new QueryEvent(searchEvent : searchEvent, firstHop : true, 
             replyTo: core.me.destination, receivedOn: core.me.destination,
             originator : core.me))
     }
     
+    void search(String infoHash, String tabTitle) {
+        def cardsPanel = builder.getVariable("cards-panel")
+        cardsPanel.getLayout().show(cardsPanel, "search window")
+        def uuid = UUID.randomUUID()
+        Map<String, Object> params = new HashMap<>()
+        params["search-terms"] = tabTitle
+        params["uuid"] = uuid.toString()
+        def group = mvcGroup.createMVCGroup("SearchTab", uuid.toString(), params)
+        model.results[uuid.toString()] = group
+        
+        def searchEvent = new SearchEvent(searchHash : Base64.decode(infoHash), uuid:uuid,
+            oobInfohash: true)
+        core.eventBus.publish(new QueryEvent(searchEvent : searchEvent, firstHop : true,
+            replyTo: core.me.destination, receivedOn: core.me.destination,
+            originator : core.me))
+    }
+    
     private def selectedResult() {
         def selected = builder.getVariable("result-tabs").getSelectedComponent()
         def group = selected.getClientProperty("mvc-group")
@@ -58,28 +102,40 @@ class MainFrameController {
         int row = table.getSelectedRow()
         if (row == -1)
             return
+        def sortEvt = group.view.lastSortEvent
+        if (sortEvt != null) {
+            row = group.view.resultsTable.rowSorter.convertRowIndexToModel(row)
+        } 
         group.model.results[row]        
     }
     
-    private def selectedDownload() {
+    private int selectedDownload() {
-        def selected = builder.getVariable("downloads-table").getSelectedRow()
+        def downloadsTable = builder.getVariable("downloads-table")
-        model.downloads[selected].downloader
+        def selected = downloadsTable.getSelectedRow()
+        def sortEvt = mvcGroup.view.lastDownloadSortEvent
+        if (sortEvt != null)
+            selected = downloadsTable.rowSorter.convertRowIndexToModel(selected)
+        selected
     }
     
     @ControllerAction
     void download() {
         def result = selectedResult()
         if (result == null)
-            return // TODO disable button
+            return
-            
+        
+        if (!model.canDownload(result.infohash))
+            return   
+             
         def file = new File(application.context.get("muwire-settings").downloadLocation, result.name)
         
         def selected = builder.getVariable("result-tabs").getSelectedComponent()
         def group = selected.getClientProperty("mvc-group")
         
         def resultsBucket = group.model.hashBucket[result.infohash]
+        def sources = group.model.sourcesBucket[result.infohash]
          
-        core.eventBus.publish(new UIDownloadEvent(result : resultsBucket, target : file))
+        core.eventBus.publish(new UIDownloadEvent(result : resultsBucket, sources: sources, target : file))
     }
     
     @ControllerAction
@@ -100,14 +156,24 @@ class MainFrameController {
     
     @ControllerAction 
     void cancel() {
-        def downloader = selectedDownload()
+        def downloader = model.downloads[selectedDownload()].downloader
         downloader.cancel()
+        model.downloadInfoHashes.remove(downloader.getInfoHash())
+        core.eventBus.publish(new UIDownloadCancelledEvent(downloader : downloader))
     }
     
     @ControllerAction
     void resume() {
-        def downloader = selectedDownload()
+        def downloader = model.downloads[selectedDownload()].downloader
         downloader.resume()
+        core.eventBus.publish(new UIDownloadResumedEvent())
+    }
+    
+    @ControllerAction
+    void pause() {
+        def downloader = model.downloads[selectedDownload()].downloader
+        downloader.pause()
+        core.eventBus.publish(new UIDownloadPausedEvent())
     }
 
     private void markTrust(String tableName, TrustLevel level, def list) {
@@ -137,6 +203,17 @@ class MainFrameController {
         markTrust("trusted-table", TrustLevel.NEUTRAL, model.trusted)
     }
     
+    void unshareSelectedFiles() {
+        println "unsharing selected files"
+    }
+    
+    void saveMuWireSettings() {
+        File f = new File(core.home, "MuWire.properties")
+        f.withOutputStream { 
+            core.muOptions.write(it)
+        }
+    }
+    
     void mvcGroupInit(Map<String, String> args) {
         application.addPropertyChangeListener("core", {e-> 
             core = e.getNewValue()

gui/griffon-app/controllers/com/muwire/gui/OptionsController.groovy

@@ -6,6 +6,8 @@ import griffon.inject.MVCMember
 import griffon.metadata.ArtifactProviderFor
 import javax.annotation.Nonnull
 
+import com.muwire.core.Core
+
 @ArtifactProviderFor(GriffonController)
 class OptionsController {
     @MVCMember @Nonnull
@@ -15,21 +17,91 @@ class OptionsController {
 
     @ControllerAction
     void save() {
-        String text = view.retryField.text
+        String text
-        model.downloadRetryInterval = text
+        Core core = application.context.get("core")
         
+        def i2pProps = core.i2pOptions
+        
+        text = view.inboundLengthField.text
+        model.inboundLength = text
+        i2pProps["inbound.length"] = text
+        
+        text = view.inboundQuantityField.text
+        model.inboundQuantity = text
+        i2pProps["inbound.quantity"] = text
+        
+        text = view.outboundQuantityField.text
+        model.outboundQuantity = text
+        i2pProps["outbound.quantity"] = text
+        
+        text = view.outboundLengthField.text
+        model.outboundLength = text
+        i2pProps["outbound.length"] = text
+        
+        File i2pSettingsFile = new File(core.home, "i2p.properties")
+        i2pSettingsFile.withOutputStream { 
+            i2pProps.store(it,"")
+        }
+                
+        text = view.retryField.text
+        model.downloadRetryInterval = text
+
         def settings = application.context.get("muwire-settings")
         settings.downloadRetryInterval = Integer.valueOf(text)
 
         text = view.updateField.text
         model.updateCheckInterval = text
         settings.updateCheckInterval = Integer.valueOf(text)
-                
+
-        File settingsFile = new File(application.context.get("core").home, "MuWire.properties")
+        boolean onlyTrusted = view.allowUntrustedCheckbox.model.isSelected()
+        model.onlyTrusted = onlyTrusted
+        settings.setAllowUntrusted(!onlyTrusted)
+        
+        boolean shareDownloaded = view.shareDownloadedCheckbox.model.isSelected()
+        model.shareDownloadedFiles = shareDownloaded
+        settings.shareDownloadedFiles = shareDownloaded
+                        
+        File settingsFile = new File(core.home, "MuWire.properties")
         settingsFile.withOutputStream { 
             settings.write(it)
         }
         
+        // UI Setttings
+        
+        UISettings uiSettings = application.context.get("ui-settings")
+        text = view.lnfField.text
+        model.lnf = text
+        uiSettings.lnf = text
+        
+        text = view.fontField.text
+        model.font = text
+        uiSettings.font = text
+        
+//        boolean showMonitor = view.monitorCheckbox.model.isSelected()
+//        model.showMonitor = showMonitor
+//        uiSettings.showMonitor = showMonitor
+        
+        boolean clearCancelledDownloads = view.clearCancelledDownloadsCheckbox.model.isSelected()
+        model.clearCancelledDownloads = clearCancelledDownloads
+        uiSettings.clearCancelledDownloads = clearCancelledDownloads
+        
+        boolean clearFinishedDownloads = view.clearFinishedDownloadsCheckbox.model.isSelected()
+        model.clearFinishedDownloads = clearFinishedDownloads
+        uiSettings.clearFinishedDownloads = clearFinishedDownloads
+        
+        boolean excludeLocalResult = view.excludeLocalResultCheckbox.model.isSelected()
+        model.excludeLocalResult = excludeLocalResult
+        uiSettings.excludeLocalResult = excludeLocalResult
+        
+//        boolean showSearchHashes = view.showSearchHashesCheckbox.model.isSelected()
+//        model.showSearchHashes = showSearchHashes
+//        uiSettings.showSearchHashes = showSearchHashes
+        
+        File uiSettingsFile = new File(core.home, "gui.properties")
+        uiSettingsFile.withOutputStream { 
+            uiSettings.write(it)
+        }
+        
         cancel()
     }
     

gui/griffon-app/lifecycle/Initialize.groovy

@@ -1,17 +1,25 @@
 import griffon.core.GriffonApplication
 import groovy.util.logging.Log
+import net.i2p.util.SystemVersion
 
 import org.codehaus.griffon.runtime.core.AbstractLifecycleHandler
 
 import com.muwire.core.Core
 import com.muwire.core.MuWireSettings
+import com.muwire.gui.UISettings
 
 import javax.annotation.Nonnull
 import javax.inject.Inject
+import javax.swing.JTable
+import javax.swing.LookAndFeel
+import javax.swing.UIManager
 
 import static griffon.util.GriffonApplicationUtils.isMacOSX
 import static groovy.swing.SwingBuilder.lookAndFeel
 
+import java.awt.Font
+import java.util.logging.Level
+
 @Log
 class Initialize extends AbstractLifecycleHandler {
     @Inject
@@ -21,11 +29,87 @@ class Initialize extends AbstractLifecycleHandler {
 
     @Override
     void execute() {
-        if (isMacOSX()) {
+        log.info "Loading home dir"
-            lookAndFeel('nimbus') // otherwise the file chooser doesn't open???
+        def portableHome = System.getProperty("portable.home")
-        } else {
+        def home = portableHome == null ?
-            lookAndFeel('system', 'gtk')
+            selectHome() :
+            portableHome
+            
+        home = new File(home)
+        if (!home.exists()) {
+            log.info("creating home dir $home")
+            home.mkdirs()
         }
+        
+        application.context.put("muwire-home", home.getAbsolutePath())
+        
+        def guiPropsFile = new File(home, "gui.properties")
+        UISettings uiSettings
+        if (guiPropsFile.exists()) {
+            Properties props = new Properties()
+            guiPropsFile.withInputStream { props.load(it) }
+            uiSettings = new UISettings(props)
+            
+            log.info("settting user-specified lnf $uiSettings.lnf")
+            try {
+                lookAndFeel(uiSettings.lnf)
+            } catch (Throwable bad) {
+                log.log(Level.WARNING,"couldn't set desired look and feeel, switching to defaults", bad)
+                uiSettings.lnf = lookAndFeel("system","gtk","metal").getID()
+            }
+            
+            if (uiSettings.font != null) {
+                log.info("setting user-specified font $uiSettings.font")
+                Font font = new Font(uiSettings.font, Font.PLAIN, 12)
+                def defaults = UIManager.getDefaults()
+                defaults.put("Button.font", font)
+                defaults.put("RadioButton.font", font)
+                defaults.put("Label.font", font)
+                defaults.put("CheckBox.font", font)
+                defaults.put("Table.font", font)
+                defaults.put("TableHeader.font", font)
+                // TODO: add others
+            }
+        } else {
+            Properties props = new Properties()
+            uiSettings = new UISettings(props)
+            log.info "will try default lnfs"
+            if (isMacOSX()) {
+                if (SystemVersion.isJava9()) {
+                    uiSettings.lnf = "metal"
+                    lookAndFeel("metal")
+                } else {
+                    uiSettings.lnf = "nimbus"
+                    lookAndFeel('nimbus') // otherwise the file chooser doesn't open???
+                }
+            } else {
+                LookAndFeel chosen = lookAndFeel('system', 'gtk')
+                uiSettings.lnf = chosen.getID()
+                log.info("ended up applying $chosen.name")
+            }
+        }
+        
+        application.context.put("ui-settings", uiSettings)
+    }
+    
+    private static String selectHome() {
+        def home = new File(System.properties["user.home"])
+        def defaultHome = new File(home, ".MuWire")
+        if (defaultHome.exists())
+            return defaultHome.getAbsolutePath()
+        if (SystemVersion.isMac()) {
+            def library = new File(home, "Library")
+            def appSupport = new File(library, "Application Support")
+            def muwire = new File(appSupport,"MuWire")
+            return muwire.getAbsolutePath()
+        }
+        if (SystemVersion.isWindows()) {
+            def appData = new File(home,"AppData")
+            def roaming = new File(appData, "Roaming")
+            def muwire = new File(roaming, "MuWire")
+            return muwire.getAbsolutePath()
+        }
+        defaultHome.getAbsolutePath()
     }
 }
 

gui/griffon-app/lifecycle/Ready.groovy

@@ -1,11 +1,14 @@
+
 import griffon.core.GriffonApplication
 import griffon.core.env.Metadata
 import groovy.util.logging.Log
+import net.i2p.util.SystemVersion
 
 import org.codehaus.griffon.runtime.core.AbstractLifecycleHandler
 
 import com.muwire.core.Core
 import com.muwire.core.MuWireSettings
+import com.muwire.core.UILoadedEvent
 import com.muwire.core.files.FileSharedEvent
 
 import javax.annotation.Nonnull
@@ -32,17 +35,8 @@ class Ready extends AbstractLifecycleHandler {
     @Override
     void execute() {
         log.info "starting core services"
-        def portableHome = System.getProperty("portable.home")
-        def home = portableHome == null ? 
-            System.getProperty("user.home") + File.separator + ".MuWire" :
-            portableHome
-            
-        home = new File(home)
-        if (!home.exists()) {
-            log.info("creating home dir")
-            home.mkdir()
-        }
         
+        def home = new File(application.getContext().getAsString("muwire-home"))
         def props = new Properties()
         def propsFile = new File(home, "MuWire.properties")
         if (propsFile.exists()) {
@@ -104,6 +98,9 @@ class Ready extends AbstractLifecycleHandler {
                 "Can't connect to I2P router", JOptionPane.WARNING_MESSAGE)
             System.exit(0)
         }
+        Runtime.getRuntime().addShutdownHook({
+            core.shutdown() 
+        })
         core.startServices()
         application.context.put("muwire-settings", props)
         application.context.put("core",core)
@@ -111,11 +108,7 @@ class Ready extends AbstractLifecycleHandler {
             it.propertyChange(new PropertyChangeEvent(this, "core", null, core)) 
         }
         
-        if (props.sharedFiles != null) {
+        core.eventBus.publish(new UILoadedEvent())
-            props.sharedFiles.split(",").each {
-                core.eventBus.publish(new FileSharedEvent(file : new File(it)))
-            }
-        }
     }
 }
 

gui/griffon-app/models/com/muwire/gui/MainFrameModel.groovy

@@ -9,16 +9,21 @@ import javax.swing.JTable
 
 import com.muwire.core.Core
 import com.muwire.core.InfoHash
+import com.muwire.core.MuWireSettings
 import com.muwire.core.Persona
+import com.muwire.core.RouterDisconnectedEvent
 import com.muwire.core.connection.ConnectionAttemptStatus
 import com.muwire.core.connection.ConnectionEvent
 import com.muwire.core.connection.DisconnectionEvent
 import com.muwire.core.download.DownloadStartedEvent
 import com.muwire.core.download.Downloader
+import com.muwire.core.files.FileDownloadedEvent
 import com.muwire.core.files.FileHashedEvent
 import com.muwire.core.files.FileLoadedEvent
 import com.muwire.core.files.FileSharedEvent
+import com.muwire.core.files.FileUnsharedEvent
 import com.muwire.core.search.QueryEvent
+import com.muwire.core.search.UIResultBatchEvent
 import com.muwire.core.search.UIResultEvent
 import com.muwire.core.trust.TrustEvent
 import com.muwire.core.trust.TrustService
@@ -28,17 +33,22 @@ import com.muwire.core.upload.UploadFinishedEvent
 
 import griffon.core.GriffonApplication
 import griffon.core.artifact.GriffonModel
+import griffon.core.env.Metadata
 import griffon.core.mvc.MVCGroup
 import griffon.inject.MVCMember
 import griffon.transform.FXObservable
 import griffon.transform.Observable
+import net.i2p.data.Base64
 import net.i2p.data.Destination
 import griffon.metadata.ArtifactProviderFor
 
 @ArtifactProviderFor(GriffonModel)
 class MainFrameModel {
+    @Inject Metadata metadata
     @MVCMember @Nonnull
     FactoryBuilderSupport builder
+    @MVCMember @Nonnull
+    MainFrameController controller
     @Inject @Nonnull GriffonApplication application
     @Observable boolean coreInitialized = false
     
@@ -46,6 +56,7 @@ class MainFrameModel {
     def downloads = []
     def uploads = []
     def shared = []
+    def watched = []
     def connectionList = []
     def searches = new LinkedList()
     def trusted = []
@@ -53,15 +64,22 @@ class MainFrameModel {
     
     @Observable int connections
     @Observable String me
-    @Observable boolean searchButtonsEnabled
+    @Observable boolean downloadActionEnabled
+    @Observable boolean trustButtonsEnabled
     @Observable boolean cancelButtonEnabled
     @Observable boolean retryButtonEnabled
+    @Observable boolean pauseButtonEnabled
+    @Observable String resumeButtonText
     
     private final Set<InfoHash> infoHashes = new HashSet<>()
+    
+    private final Set<InfoHash> downloadInfoHashes = new HashSet<>()
 
     volatile Core core    
 
     private long lastRetryTime = System.currentTimeMillis()
+    
+    UISettings uiSettings
         
     void updateTablePreservingSelection(String tableName) {
         def downloadTable = builder.getVariable(tableName)
@@ -72,11 +90,28 @@ class MainFrameModel {
     
     void mvcGroupInit(Map<String, Object> args) {
         
+        uiSettings = application.context.get("ui-settings")
+        
         Timer timer = new Timer("download-pumper", true)
         timer.schedule({
             runInsideUIAsync {
                 if (!mvcGroup.alive)
                     return
+                
+                // remove cancelled or finished downloads
+                def toRemove = []    
+                downloads.each {
+                    if (uiSettings.clearCancelledDownloads &&
+                        it.downloader.getCurrentState() == Downloader.DownloadState.CANCELLED)
+                        toRemove << it
+                    if (uiSettings.clearFinishedDownloads &&
+                        it.downloader.getCurrentState() == Downloader.DownloadState.FINISHED)
+                        toRemove << it
+                }
+                toRemove.each {
+                    downloads.remove(it)
+                }    
+                
                 builder.getVariable("uploads-table")?.model.fireTableDataChanged()
                 
                 updateTablePreservingSelection("downloads-table")
@@ -90,6 +125,7 @@ class MainFrameModel {
             core = e.getNewValue()
             me = core.me.getHumanReadableName()
             core.eventBus.register(UIResultEvent.class, this)
+            core.eventBus.register(UIResultBatchEvent.class, this)
             core.eventBus.register(DownloadStartedEvent.class, this)
             core.eventBus.register(ConnectionEvent.class, this)
             core.eventBus.register(DisconnectionEvent.class, this)
@@ -100,9 +136,14 @@ class MainFrameModel {
             core.eventBus.register(TrustEvent.class, this)
             core.eventBus.register(QueryEvent.class, this)
             core.eventBus.register(UpdateAvailableEvent.class, this)
+            core.eventBus.register(FileDownloadedEvent.class, this)
+            core.eventBus.register(FileUnsharedEvent.class, this)
+            core.eventBus.register(RouterDisconnectedEvent.class, this)
             
             timer.schedule({
-                int retryInterval = application.context.get("muwire-settings").downloadRetryInterval
+                if (core.shutdown.get())
+                    return
+                int retryInterval = core.muOptions.downloadRetryInterval
                 if (retryInterval > 0) {
                     retryInterval *= 60000
                     long now = System.currentTimeMillis()
@@ -110,7 +151,9 @@ class MainFrameModel {
                         lastRetryTime = now
                         runInsideUIAsync {
                             downloads.each {
-                                if (it.downloader.currentState == Downloader.DownloadState.FAILED)
+                                def state = it.downloader.currentState
+                                if (state == Downloader.DownloadState.FAILED || 
+                                    state == Downloader.DownloadState.DOWNLOADING)
                                     it.downloader.resume()
                                 updateTablePreservingSelection("downloads-table")
                             }
@@ -123,6 +166,12 @@ class MainFrameModel {
             runInsideUIAsync {
                 trusted.addAll(core.trustService.good.values())
                 distrusted.addAll(core.trustService.bad.values())
+                
+                watched.addAll(core.muOptions.watchedDirectories)
+                builder.getVariable("watched-directories-table").model.fireTableDataChanged()
+                watched.each { core.eventBus.publish(new FileSharedEvent(file : new File(it))) }
+                
+                resumeButtonText = "Retry"
             }
         })
         
@@ -133,9 +182,15 @@ class MainFrameModel {
         resultsGroup?.model.handleResult(e)
     }
     
+    void onUIResultBatchEvent(UIResultBatchEvent e) {
+        MVCGroup resultsGroup = results.get(e.uuid)
+        resultsGroup?.model.handleResultBatch(e.results)
+    }
+    
     void onDownloadStartedEvent(DownloadStartedEvent e) {
         runInsideUIAsync {
             downloads << e
+            downloadInfoHashes.add(e.downloader.infoHash)
         }
     }
     
@@ -150,7 +205,8 @@ class MainFrameModel {
                 topPanel.getLayout().show(topPanel, "top-search-panel")
             }
             
-            connectionList.add(e.endpoint.destination)
+            UIConnection con = new UIConnection(destination : e.endpoint.destination, incoming : e.incoming)
+            connectionList.add(con)
             JTable table = builder.getVariable("connections-table")
             table.model.fireTableDataChanged()
         }
@@ -165,7 +221,8 @@ class MainFrameModel {
                 topPanel.getLayout().show(topPanel, "top-connect-panel")
             }
             
-            connectionList.remove(e.destination)
+            UIConnection con = new UIConnection(destination : e.destination)
+            connectionList.remove(con)
             JTable table = builder.getVariable("connections-table")
             table.model.fireTableDataChanged()
         }
@@ -195,6 +252,17 @@ class MainFrameModel {
         }
     }
     
+    void onFileUnsharedEvent(FileUnsharedEvent e) {
+        InfoHash infohash = e.unsharedFile.infoHash
+        if (!infoHashes.remove(infohash))
+            return
+        runInsideUIAsync {
+            shared.remove(e.unsharedFile)
+            JTable table = builder.getVariable("shared-files-table")
+            table.model.fireTableDataChanged()
+        }
+    }
+    
     void onUploadEvent(UploadEvent e) {
         runInsideUIAsync {
             uploads << e.uploader
@@ -222,8 +290,10 @@ class MainFrameModel {
             updateTablePreservingSelection("trusted-table")
             updateTablePreservingSelection("distrusted-table")
             
-            results.values().each { 
+            results.values().each { MVCGroup group -> 
-                it.view.pane.getClientProperty("results-table")?.model.fireTableDataChanged()
+                if (group.alive) {
+                    group.view.pane.getClientProperty("results-table")?.model.fireTableDataChanged()
+                }
             }
         }
     }
@@ -231,14 +301,23 @@ class MainFrameModel {
     void onQueryEvent(QueryEvent e) {
         if (e.replyTo == core.me.destination)
             return
-        StringBuilder sb = new StringBuilder()
+        
-        e.searchEvent.searchTerms?.each {
+        def search
-            sb.append(it)
+        if (e.searchEvent.searchHash != null) {
-            sb.append(" ")
+            if (!uiSettings.showSearchHashes) {
+                return
+            }
+            search = Base64.encode(e.searchEvent.searchHash)
+        } else {
+            StringBuilder sb = new StringBuilder()
+            e.searchEvent.searchTerms?.each {
+                sb.append(it)
+                sb.append(" ")
+            }
+            search = sb.toString()
+            if (search.trim().size() == 0)
+                return
         }
-        def search = sb.toString()
-        if (search.trim().size() == 0)
-            return
         runInsideUIAsync {
             searches.addFirst(new IncomingSearch(search : search, replyTo : e.replyTo, originator : e.originator))
             while(searches.size() > 200)
@@ -256,7 +335,54 @@ class MainFrameModel {
     
     void onUpdateAvailableEvent(UpdateAvailableEvent e) {
         runInsideUIAsync {
-            JOptionPane.showMessageDialog(null, "A new version of MuWire is available from $e.signer.  Please update to $e.version")
+            
+            int option = JOptionPane.showConfirmDialog(null, 
+                "MuWire $e.version is available from $e.signer.  You have "+ metadata["application.version"]+" Update?",
+                "New MuWire version availble", JOptionPane.OK_CANCEL_OPTION)
+            if (option == JOptionPane.CANCEL_OPTION)
+                return
+            controller.search(e.infoHash,"MuWire update")
         }
     }
+    
+    void onRouterDisconnectedEvent(RouterDisconnectedEvent e) {
+        runInsideUIAsync {
+            JOptionPane.showMessageDialog(null, "MuWire lost connection to the I2P router and will now exit.",
+                "Connection to I2P router lost", JOptionPane.WARNING_MESSAGE)
+            System.exit(0)
+        }
+    }
+    
+    void onFileDownloadedEvent(FileDownloadedEvent e) {
+        if (!core.muOptions.shareDownloadedFiles)
+            return
+        infoHashes.add(e.downloadedFile.infoHash)
+        runInsideUIAsync {
+            shared << e.downloadedFile
+            JTable table = builder.getVariable("shared-files-table")
+            table.model.fireTableDataChanged()
+        }
+    }
+    
+    private static class UIConnection {
+        Destination destination
+        boolean incoming
+        
+        @Override
+        public int hashCode() {
+            destination.hashCode()
+        }
+        
+        @Override
+        public boolean equals(Object o) {
+            if (!(o instanceof UIConnection))
+                return false
+            UIConnection other = (UIConnection) o
+            return destination == other.destination
+        }
+    }
+    
+    boolean canDownload(InfoHash hash) {
+        !downloadInfoHashes.contains(hash)
+    }
 }

gui/griffon-app/models/com/muwire/gui/OptionsModel.groovy

@@ -1,5 +1,8 @@
 package com.muwire.gui
 
+import com.muwire.core.Core
+import com.muwire.core.MuWireSettings
+
 import griffon.core.artifact.GriffonModel
 import griffon.transform.Observable
 import griffon.metadata.ArtifactProviderFor
@@ -8,9 +11,44 @@ import griffon.metadata.ArtifactProviderFor
 class OptionsModel {
     @Observable String downloadRetryInterval 
     @Observable String updateCheckInterval
+    @Observable boolean onlyTrusted
+    @Observable boolean shareDownloadedFiles
+    
+    // i2p options
+    @Observable String inboundLength
+    @Observable String inboundQuantity
+    @Observable String outboundLength
+    @Observable String outboundQuantity
+    
+    // gui options
+    @Observable boolean showMonitor
+    @Observable String lnf
+    @Observable String font
+    @Observable boolean clearCancelledDownloads
+    @Observable boolean clearFinishedDownloads
+    @Observable boolean excludeLocalResult
+    @Observable boolean showSearchHashes
     
     void mvcGroupInit(Map<String, String> args) {
-        downloadRetryInterval = application.context.get("muwire-settings").downloadRetryInterval
+        MuWireSettings settings = application.context.get("muwire-settings")
-        updateCheckInterval = application.context.get("muwire-settings").updateCheckInterval
+        downloadRetryInterval = settings.downloadRetryInterval
+        updateCheckInterval = settings.updateCheckInterval
+        onlyTrusted = !settings.allowUntrusted()
+        shareDownloadedFiles = settings.shareDownloadedFiles
+        
+        Core core = application.context.get("core")
+        inboundLength = core.i2pOptions["inbound.length"]
+        inboundQuantity = core.i2pOptions["inbound.quantity"]
+        outboundLength = core.i2pOptions["outbound.length"]
+        outboundQuantity = core.i2pOptions["outbound.quantity"]
+        
+        UISettings uiSettings = application.context.get("ui-settings")
+        showMonitor = uiSettings.showMonitor
+        lnf = uiSettings.lnf
+        font = uiSettings.font
+        clearCancelledDownloads = uiSettings.clearCancelledDownloads
+        clearFinishedDownloads = uiSettings.clearFinishedDownloads
+        excludeLocalResult = uiSettings.excludeLocalResult
+        showSearchHashes = uiSettings.showSearchHashes
     }
 }

gui/griffon-app/models/com/muwire/gui/SearchTabModel.groovy

@@ -19,13 +19,16 @@ class SearchTabModel {
     FactoryBuilderSupport builder
     
     Core core
+    UISettings uiSettings
     String uuid
     def results = []
     def hashBucket = [:]
+    def sourcesBucket = [:]
     
     
     void mvcGroupInit(Map<String, String> args) {
         core = mvcGroup.parentGroup.model.core
+        uiSettings = application.context.get("ui-settings")
         mvcGroup.parentGroup.model.results[UUID.fromString(uuid)] = mvcGroup
     }
     
@@ -34,6 +37,9 @@ class SearchTabModel {
     }
     
     void handleResult(UIResultEvent e) {
+        if (uiSettings.excludeLocalResult &&
+            core.fileManager.rootToFiles.containsKey(e.infohash))
+            return
         runInsideUIAsync {
             def bucket = hashBucket.get(e.infohash)
             if (bucket == null) {
@@ -41,10 +47,44 @@ class SearchTabModel {
                 hashBucket[e.infohash] = bucket
             }
             bucket << e
+            
+            Set sourceBucket = sourcesBucket.get(e.infohash)
+            if (sourceBucket == null) {
+                sourceBucket = new HashSet()
+                sourcesBucket.put(e.infohash, sourceBucket)
+            }
+            sourceBucket.addAll(e.sources)
                             
             results << e
             JTable table = builder.getVariable("results-table")
             table.model.fireTableDataChanged()
         }
     }
+    
+    void handleResultBatch(UIResultEvent[] batch) {
+        runInsideUIAsync {
+            batch.each { 
+                if (uiSettings.excludeLocalResult && 
+                    core.fileManager.rootToFiles.containsKey(it.infohash))
+                    return
+                def bucket = hashBucket.get(it.infohash)
+                if (bucket == null) {
+                    bucket = []
+                    hashBucket[it.infohash] = bucket
+                }
+                
+                Set sourceBucket = sourcesBucket.get(it.infohash)
+                if (sourceBucket == null) {
+                    sourceBucket = new HashSet()
+                    sourcesBucket.put(it.infohash, sourceBucket)
+                }
+                sourceBucket.addAll(it.sources)
+                
+                bucket << it
+                results << it
+            }
+            JTable table = builder.getVariable("results-table")
+            table.model.fireTableDataChanged()
+        }
+    }
 }

gui/griffon-app/views/com/muwire/gui/MainFrameView.groovy

@@ -1,18 +1,25 @@
 package com.muwire.gui
 
 import griffon.core.artifact.GriffonView
+import griffon.core.env.Metadata
 import griffon.inject.MVCMember
 import griffon.metadata.ArtifactProviderFor
+import net.i2p.data.Base64
 import net.i2p.data.DataHelper
 
 import javax.swing.BorderFactory
 import javax.swing.Box
 import javax.swing.BoxLayout
 import javax.swing.JFileChooser
+import javax.swing.JLabel
+import javax.swing.JMenuItem
+import javax.swing.JPopupMenu
 import javax.swing.JSplitPane
+import javax.swing.JTable
 import javax.swing.ListSelectionModel
 import javax.swing.SwingConstants
 import javax.swing.border.Border
+import javax.swing.table.DefaultTableCellRenderer
 
 import com.muwire.core.Constants
 import com.muwire.core.download.Downloader
@@ -24,9 +31,14 @@ import java.awt.FlowLayout
 import java.awt.GridBagConstraints
 import java.awt.GridBagLayout
 import java.awt.Insets
+import java.awt.Toolkit
+import java.awt.datatransfer.StringSelection
+import java.awt.event.MouseAdapter
+import java.awt.event.MouseEvent
 import java.nio.charset.StandardCharsets
 
 import javax.annotation.Nonnull
+import javax.inject.Inject
 
 @ArtifactProviderFor(GriffonView)
 class MainFrameView {
@@ -35,11 +47,19 @@ class MainFrameView {
     @MVCMember @Nonnull
     MainFrameModel model
     
+    @Inject Metadata metadata
+    
+    def downloadsTable
+    def lastDownloadSortEvent
+    def lastSharedSortEvent
+    
     void initUI() {
+        UISettings settings = application.context.get("ui-settings")
         builder.with {
             application(size : [1024,768], id: 'main-frame',
             locationRelativeTo : null,
-            title: application.configuration['application.title'],
+            title: application.configuration['application.title'] + " " + 
+                metadata["application.version"] + " revision " + metadata["build.revision"],
             iconImage:   imageIcon('/griffon-icon-48x48.png').image,
             iconImages: [imageIcon('/griffon-icon-48x48.png').image,
                 imageIcon('/griffon-icon-32x32.png').image,
@@ -58,7 +78,8 @@ class MainFrameView {
                         gridLayout(rows:1, cols: 2)
                         button(text: "Searches", actionPerformed : showSearchWindow)
                         button(text: "Uploads", actionPerformed : showUploadsWindow)
-                        button(text: "Monitor", actionPerformed : showMonitorWindow)
+                        if (settings.showMonitor)
+                            button(text: "Monitor", actionPerformed : showMonitorWindow)
                         button(text: "Trust", actionPerformed : showTrustWindow)
                     }
                     panel(id: "top-panel", constraints: BorderLayout.CENTER) {
@@ -89,22 +110,22 @@ class MainFrameView {
                                 borderLayout()
                                 tabbedPane(id : "result-tabs", constraints: BorderLayout.CENTER)
                                 panel(constraints : BorderLayout.SOUTH) {
-                                    button(text : "Download", enabled : bind {model.searchButtonsEnabled}, downloadAction)
+                                    button(text : "Download", enabled : bind {model.downloadActionEnabled}, downloadAction)
-                                    button(text : "Trust", enabled: bind {model.searchButtonsEnabled }, trustAction)
+                                    button(text : "Trust", enabled: bind {model.trustButtonsEnabled }, trustAction)
-                                    button(text : "Distrust", enabled : bind {model.searchButtonsEnabled}, distrustAction)
+                                    button(text : "Distrust", enabled : bind {model.trustButtonsEnabled}, distrustAction)
                                 }
                             }
                             panel (constraints : JSplitPane.BOTTOM) {
                                 borderLayout()
                                 scrollPane (constraints : BorderLayout.CENTER) {
-                                    table(id : "downloads-table") {
+                                    downloadsTable = table(id : "downloads-table", autoCreateRowSorter : true) {
                                         tableModel(list: model.downloads) {
                                             closureColumn(header: "Name", preferredWidth: 350, type: String, read : {row -> row.downloader.file.getName()})
-                                            closureColumn(header: "Status", preferredWidth: 50, type: String, read : {row -> row.downloader.getCurrentState()})
+                                            closureColumn(header: "Status", preferredWidth: 50, type: String, read : {row -> row.downloader.getCurrentState().toString()})
                                             closureColumn(header: "Progress", preferredWidth: 20, type: String, read: { row ->
                                                 int pieces = row.downloader.nPieces
                                                 int done = row.downloader.donePieces()
-                                                "$done/$pieces pieces"
+                                                "$done/$pieces pieces".toString()
                                             })
                                             closureColumn(header: "Sources", preferredWidth : 10, type: Integer, read : {row -> row.downloader.activeWorkers()})
                                             closureColumn(header: "Speed", preferredWidth: 50, type:String, read :{row -> 
@@ -114,8 +135,9 @@ class MainFrameView {
                                     }
                                 }
                                 panel (constraints : BorderLayout.SOUTH) {
+                                    button(text: "Pause", enabled : bind {model.pauseButtonEnabled}, pauseAction)
                                     button(text: "Cancel", enabled : bind {model.cancelButtonEnabled }, cancelAction )
-                                    button(text: "Retry", enabled : bind {model.retryButtonEnabled}, resumeAction)
+                                    button(text: bind { model.resumeButtonText }, enabled : bind {model.retryButtonEnabled}, resumeAction)
                                 }
                             }
                         }
@@ -123,17 +145,32 @@ class MainFrameView {
                     panel (constraints: "uploads window"){
                         gridLayout(cols : 1, rows : 2)
                         panel {
-                            borderLayout()
+                            gridLayout(cols : 2, rows : 1) 
-                            panel (constraints : BorderLayout.NORTH) {
+                            panel {
-                                button(text : "Click here to share files", actionPerformed : shareFiles)
+                                borderLayout()
+                                panel (constraints : BorderLayout.NORTH) {
+                                    button(text : "Add directories to watch", actionPerformed : watchDirectories)
+                                }
+                                scrollPane (constraints : BorderLayout.CENTER) {
+                                    table(id : "watched-directories-table", autoCreateRowSorter: true) {
+                                        tableModel(list : model.watched) {
+                                            closureColumn(header: "Watched Directories", type : String, read : { it })
+                                        }
+                                    }
+                                }
                             }
-                            scrollPane ( constraints : BorderLayout.CENTER) {
+                            panel {
-                                table(id : "shared-files-table") {
+                                borderLayout()
-                                     tableModel(list : model.shared) {
+                                panel (constraints : BorderLayout.NORTH) {
-                                         closureColumn(header : "Name", preferredWidth : 550, type : String, read : {row -> row.file.getAbsolutePath()})
+                                    button(text : "Share files", actionPerformed : shareFiles)
-                                         closureColumn(header : "Size", preferredWidth : 50, type : String, 
+                                }
-                                             read : {row -> DataHelper.formatSize2Decimal(row.file.length(),false) + "B"})
+                                scrollPane(constraints : BorderLayout.CENTER) {
-                                     }   
+                                    table(id : "shared-files-table", autoCreateRowSorter: true) {
+                                        tableModel(list : model.shared) {
+                                            closureColumn(header : "Name", preferredWidth : 500, type : String, read : {row -> row.file.getAbsolutePath()})
+                                            closureColumn(header : "Size", preferredWidth : 100, type : Long, read : {row -> row.file.length() })
+                                        }
+                                    }
                                 }
                             }
                         }
@@ -145,16 +182,13 @@ class MainFrameView {
                             scrollPane (constraints : BorderLayout.CENTER) {
                                 table(id : "uploads-table") {
                                     tableModel(list : model.uploads) {
-                                        closureColumn(header : "Name", type : String, read : {row -> row.file.getName() })
+                                        closureColumn(header : "Name", type : String, read : {row -> row.getName() })
                                         closureColumn(header : "Progress", type : String, read : { row ->
-                                            int position = row.getPosition()
+                                            int percent = row.getProgress()
-                                            def range = row.request.getRange()
-                                            int total = range.end - range.start
-                                            int percent = (int)((position * 100.0) / total)
                                             "$percent%"
                                         })
                                         closureColumn(header : "Downloader", type : String, read : { row -> 
-                                            row.request.downloader?.getHumanReadableName()
+                                            row.getDownloader()
                                         })
                                     }
                                 }
@@ -171,7 +205,13 @@ class MainFrameView {
                             scrollPane(constraints : BorderLayout.CENTER) {
                                 table(id : "connections-table") {
                                     tableModel(list : model.connectionList) {
-                                        closureColumn(header : "Destination", type: String, read : { row -> row.toBase32() })
+                                        closureColumn(header : "Destination", preferredWidth: 250, type: String, read : { row -> row.destination.toBase32() })
+                                        closureColumn(header : "Direction", preferredWidth: 20, type: String, read : { row ->
+                                            if (row.incoming)
+                                                return "In"
+                                            else 
+                                                return "Out"
+                                        })
                                     }
                                 }
                             }
@@ -205,7 +245,7 @@ class MainFrameView {
                         panel (border : etchedBorder()){
                             borderLayout()
                             scrollPane(constraints : BorderLayout.CENTER) {
-                                table(id : "trusted-table") {
+                                table(id : "trusted-table", autoCreateRowSorter : true) {
                                     tableModel(list : model.trusted) {
                                         closureColumn(header : "Trusted Users", type : String, read : { it.getHumanReadableName() } )
                                     }
@@ -220,7 +260,7 @@ class MainFrameView {
                         panel (border : etchedBorder()){
                             borderLayout()
                             scrollPane(constraints : BorderLayout.CENTER) {
-                                table(id : "distrusted-table") {
+                                table(id : "distrusted-table", autoCreateRowSorter : true) {
                                     tableModel(list : model.distrusted) {
                                         closureColumn(header: "Distrusted Users", type : String, read : { it.getHumanReadableName() } )
                                     }
@@ -252,25 +292,208 @@ class MainFrameView {
         def selectionModel = downloadsTable.getSelectionModel()
         selectionModel.setSelectionMode(ListSelectionModel.SINGLE_SELECTION)
         selectionModel.addListSelectionListener({
-            int selectedRow = downloadsTable.getSelectedRow()
+            int selectedRow = selectedDownloaderRow()
-            def downloader = model.downloads[selectedRow].downloader
+            if (selectedRow < 0) {
+                model.cancelButtonEnabled = false
+                model.retryButtonEnabled = false
+                model.pauseButtonEnabled = false
+                return
+            }
+            def downloader = model.downloads[selectedRow]?.downloader
+            if (downloader == null)
+                return
             switch(downloader.getCurrentState()) {
                 case Downloader.DownloadState.CONNECTING :
                 case Downloader.DownloadState.DOWNLOADING :
+                case Downloader.DownloadState.HASHLIST:
                 model.cancelButtonEnabled = true
+                model.pauseButtonEnabled = true
                 model.retryButtonEnabled = false
                 break
                 case Downloader.DownloadState.FAILED:
-                model.cancelButtonEnabled = false
+                model.cancelButtonEnabled = true
                 model.retryButtonEnabled = true
+                model.resumeButtonText = "Retry"
+                model.pauseButtonEnabled = false
+                break
+                case Downloader.DownloadState.PAUSED:
+                model.cancelButtonEnabled = true
+                model.retryButtonEnabled = true
+                model.resumeButtonText = "Resume"
+                model.pauseButtonEnabled = false
                 break
                 default:
                 model.cancelButtonEnabled = false
                 model.retryButtonEnabled = false
+                model.pauseButtonEnabled = false
             }
         })
+        
+        def centerRenderer = new DefaultTableCellRenderer()
+        centerRenderer.setHorizontalAlignment(JLabel.CENTER)
+        downloadsTable.setDefaultRenderer(Integer.class, centerRenderer)
+        
+        downloadsTable.rowSorter.addRowSorterListener({evt -> lastDownloadSortEvent = evt})
+        downloadsTable.rowSorter.setSortsOnUpdates(true)
+        
+        downloadsTable.addMouseListener(new MouseAdapter() {
+            @Override
+            public void mouseReleased(MouseEvent e) {
+                if (e.isPopupTrigger())
+                    showDownloadsMenu(e)
+            }
+            @Override
+            public void mousePressed(MouseEvent e) {
+                if (e.isPopupTrigger())
+                    showDownloadsMenu(e)
+            }
+        })
+        
+        // shared files table
+        def sharedFilesTable = builder.getVariable("shared-files-table")
+        sharedFilesTable.columnModel.getColumn(1).setCellRenderer(new SizeRenderer())
+        
+        sharedFilesTable.rowSorter.addRowSorterListener({evt -> lastSharedSortEvent = evt})
+        sharedFilesTable.rowSorter.setSortsOnUpdates(true)
+        
+        JPopupMenu sharedFilesMenu = new JPopupMenu()
+//        JMenuItem unshareSelectedFiles = new JMenuItem("Unshare selected files")
+//        unshareSelectedFiles.addActionListener({mvcGroup.controller.unshareSelectedFiles()})
+        JMenuItem copyHashToClipboard = new JMenuItem("Copy hash to clipboard")
+        copyHashToClipboard.addActionListener({mvcGroup.view.copyHashToClipboard(sharedFilesTable)})
+        sharedFilesMenu.add(copyHashToClipboard)
+        sharedFilesTable.addMouseListener(new MouseAdapter() {
+            @Override
+            public void mouseReleased(MouseEvent e) {
+                if (e.isPopupTrigger())
+                    showPopupMenu(sharedFilesMenu, e)
+            }
+            @Override
+            public void mousePressed(MouseEvent e) {
+                if (e.isPopupTrigger())
+                    showPopupMenu(sharedFilesMenu, e)
+            }
+        })
+        
+        // searches table
+        def searchesTable = builder.getVariable("searches-table")
+        JPopupMenu searchTableMenu = new JPopupMenu()
+        JMenuItem copySearchToClipboard = new JMenuItem("Copy search to clipboard")
+        copySearchToClipboard.addActionListener({mvcGroup.view.copySearchToClipboard(searchesTable)})
+        searchTableMenu.add(copySearchToClipboard)
+        searchesTable.addMouseListener(new MouseAdapter() {
+            @Override
+            public void mouseReleased(MouseEvent e) {
+                if (e.isPopupTrigger())
+                    showPopupMenu(searchTableMenu, e)
+            }
+            @Override
+            public void mousePressed(MouseEvent e) {
+                if (e.isPopupTrigger())
+                    showPopupMenu(searchTableMenu, e)
+            }
+        })
+        
+    }
+    
+    private static void showPopupMenu(JPopupMenu menu, MouseEvent event) {
+        menu.show(event.getComponent(), event.getX(), event.getY())
+    }
+    
+    def copyHashToClipboard(JTable sharedFilesTable) {
+        int selected = sharedFilesTable.getSelectedRow()
+        if (selected < 0)
+            return
+        if (lastSharedSortEvent != null) 
+            selected = sharedFilesTable.rowSorter.convertRowIndexToModel(selected)
+        String root = Base64.encode(model.shared[selected].infoHash.getRoot())
+        StringSelection selection = new StringSelection(root)
+        def clipboard = Toolkit.getDefaultToolkit().getSystemClipboard()
+        clipboard.setContents(selection, null)
+    }
+    
+    def copySearchToClipboard(JTable searchesTable) {
+        int selected = searchesTable.getSelectedRow()
+        if (selected < 0)
+            return
+        String search = model.searches[selected].search
+        StringSelection selection = new StringSelection(search)
+        def clipboard = Toolkit.getDefaultToolkit().getSystemClipboard()
+        clipboard.setContents(selection, null)
     }
 
+    int selectedDownloaderRow() {
+        int selected = builder.getVariable("downloads-table").getSelectedRow()
+        if (lastDownloadSortEvent != null)
+            selected = lastDownloadSortEvent.convertPreviousRowIndexToModel(selected)
+        selected
+    }
+    
+    def showDownloadsMenu(MouseEvent e) {
+        int selected = selectedDownloaderRow()
+        if (selected < 0)
+            return
+        boolean pauseEnabled = false
+        boolean cancelEnabled = false
+        boolean retryEnabled = false
+        String resumeText = "Retry"
+        Downloader downloader = model.downloads[selected].downloader
+        switch(downloader.currentState) {
+            case Downloader.DownloadState.DOWNLOADING:
+            case Downloader.DownloadState.HASHLIST:
+            case Downloader.DownloadState.CONNECTING:
+                pauseEnabled = true
+                cancelEnabled = true
+                retryEnabled = false
+                break
+            case Downloader.DownloadState.FAILED:
+                pauseEnabled = false
+                cancelEnabled = true
+                retryEnabled = true
+                break
+            case Downloader.DownloadState.PAUSED:
+                pauseEnabled = false
+                cancelEnabled = true
+                retryEnabled = true
+                resumeText = "Resume"
+                break
+            default :
+                pauseEnabled = false
+                cancelEnabled = false
+                retryEnabled = false
+        }
+        
+        JPopupMenu menu = new JPopupMenu()
+        JMenuItem copyHashToClipboard = new JMenuItem("Copy hash to clipboard")
+        copyHashToClipboard.addActionListener({
+            String hash = Base64.encode(downloader.infoHash.getRoot())
+            StringSelection selection = new StringSelection(hash)
+            def clipboard = Toolkit.getDefaultToolkit().getSystemClipboard()
+            clipboard.setContents(selection, null)
+        })
+        menu.add(copyHashToClipboard)
+        
+        if (pauseEnabled) {
+            JMenuItem pause = new JMenuItem("Pause")
+            pause.addActionListener({mvcGroup.controller.pause()})
+            menu.add(pause)
+        }
+        
+        if (cancelEnabled) {
+            JMenuItem cancel = new JMenuItem("Cancel")
+            cancel.addActionListener({mvcGroup.controller.cancel()})
+            menu.add(cancel)
+        }
+        
+        if (retryEnabled) {
+            JMenuItem retry = new JMenuItem(resumeText)
+            retry.addActionListener({mvcGroup.controller.resume()})
+            menu.add(retry)
+        }
+        
+        showPopupMenu(menu, e)
+    }
+    
     def showSearchWindow = {
         def cardsPanel = builder.getVariable("cards-panel")
         cardsPanel.getLayout().show(cardsPanel, "search window")
@@ -293,11 +516,26 @@ class MainFrameView {
     
     def shareFiles = {
         def chooser = new JFileChooser()
-        chooser.setDialogTitle("Select file or directory to share")
+        chooser.setDialogTitle("Select file to share")
-        chooser.setFileSelectionMode(JFileChooser.FILES_AND_DIRECTORIES)
+        chooser.setFileSelectionMode(JFileChooser.FILES_ONLY)
         int rv = chooser.showOpenDialog(null)
         if (rv == JFileChooser.APPROVE_OPTION) {
             model.core.eventBus.publish(new FileSharedEvent(file : chooser.getSelectedFile()))
         }
     }
+    
+    def watchDirectories = {
+        def chooser = new JFileChooser()
+        chooser.setDialogTitle("Select directory to watch")
+        chooser.setFileSelectionMode(JFileChooser.DIRECTORIES_ONLY)
+        int rv = chooser.showOpenDialog(null)
+        if (rv == JFileChooser.APPROVE_OPTION) {
+            File f = chooser.getSelectedFile()
+            model.watched << f.getAbsolutePath()
+            application.context.get("muwire-settings").watchedDirectories << f.getAbsolutePath()
+            mvcGroup.controller.saveMuWireSettings()
+            builder.getVariable("watched-directories-table").model.fireTableDataChanged()
+            model.core.eventBus.publish(new FileSharedEvent(file : f))
+        }
+    }
 }

gui/griffon-app/views/com/muwire/gui/OptionsView.groovy

@@ -5,8 +5,11 @@ import griffon.inject.MVCMember
 import griffon.metadata.ArtifactProviderFor
 
 import javax.swing.JDialog
+import javax.swing.JPanel
+import javax.swing.JTabbedPane
 import javax.swing.SwingConstants
 
+import java.awt.BorderLayout
 import java.awt.event.WindowAdapter
 import java.awt.event.WindowEvent
 
@@ -21,8 +24,29 @@ class OptionsView {
 
     def d
     def p
+    def i
+    def u
+    
     def retryField
     def updateField
+    def allowUntrustedCheckbox
+    def shareDownloadedCheckbox
+    
+    def inboundLengthField
+    def inboundQuantityField
+    def outboundLengthField
+    def outboundQuantityField
+
+    def lnfField
+    def monitorCheckbox
+    def fontField
+    def clearCancelledDownloadsCheckbox
+    def clearFinishedDownloadsCheckbox
+    def excludeLocalResultCheckbox
+    def showSearchHashesCheckbox
+    
+    def buttonsPanel    
+    
     def mainFrame
     
     void initUI() {
@@ -38,14 +62,63 @@ class OptionsView {
             label(text : "Check for updates every", constraints : gbc(gridx : 0, gridy: 1))
             updateField = textField(text : bind {model.updateCheckInterval }, columns : 2, constraints : gbc(gridx : 1, gridy: 1))
             label(text : "hours", constraints : gbc(gridx: 2, gridy : 1))
+
+            label(text : "Allow only trusted connections", constraints : gbc(gridx: 0, gridy : 2))
+            allowUntrustedCheckbox = checkBox(selected : bind {model.onlyTrusted}, constraints : gbc(gridx: 1, gridy : 2))
             
+            label(text : "Share downloaded files", constraints : gbc(gridx : 0, gridy:3))
+            shareDownloadedCheckbox = checkBox(selected : bind {model.shareDownloadedFiles}, constraints : gbc(gridx :1, gridy:3))
+                        
+        }
+        i = builder.panel {
+            gridBagLayout()
+            label(text : "Changing these settings requires a restart", constraints : gbc(gridx : 0, gridy : 0, gridwidth: 2))
+            label(text : "Inbound Length", constraints : gbc(gridx:0, gridy:1))
+            inboundLengthField = textField(text : bind {model.inboundLength}, columns : 2, constraints : gbc(gridx:1, gridy:1))
+            label(text : "Inbound Quantity", constraints : gbc(gridx:0, gridy:2))
+            inboundQuantityField = textField(text : bind {model.inboundQuantity}, columns : 2, constraints : gbc(gridx:1, gridy:2))
+            label(text : "Outbound Length", constraints : gbc(gridx:0, gridy:3))
+            outboundLengthField = textField(text : bind {model.outboundLength}, columns : 2, constraints : gbc(gridx:1, gridy:3))
+            label(text : "Outbound Quantity", constraints : gbc(gridx:0, gridy:4))
+            outboundQuantityField = textField(text : bind {model.outboundQuantity}, columns : 2, constraints : gbc(gridx:1, gridy:4))
+        }
+        u = builder.panel {
+            gridBagLayout()
+            label(text : "Changing these settings requires a restart", constraints : gbc(gridx : 0, gridy : 0, gridwidth: 2))
+            label(text : "Look And Feel", constraints : gbc(gridx: 0, gridy:1))
+            lnfField = textField(text : bind {model.lnf}, columns : 4, constraints : gbc(gridx : 1, gridy : 1))
+            label(text : "Font", constraints : gbc(gridx: 0, gridy : 2))
+            fontField = textField(text : bind {model.font}, columns : 4, constraints : gbc(gridx : 1, gridy:2))
+//            label(text : "Show Monitor", constraints : gbc(gridx :0, gridy: 3))
+//            monitorCheckbox = checkBox(selected : bind {model.showMonitor}, constraints : gbc(gridx : 1, gridy: 3))
+            label(text : "Clear Cancelled Downloads", constraints: gbc(gridx: 0, gridy:4))
+            clearCancelledDownloadsCheckbox = checkBox(selected : bind {model.clearCancelledDownloads}, constraints : gbc(gridx : 1, gridy:4))
+            label(text : "Clear Finished Downloads", constraints: gbc(gridx: 0, gridy:5))
+            clearFinishedDownloadsCheckbox = checkBox(selected : bind {model.clearFinishedDownloads}, constraints : gbc(gridx : 1, gridy:5))
+            label(text : "Exclude Local Files From Results", constraints: gbc(gridx:0, gridy:6))
+            excludeLocalResultCheckbox = checkBox(selected : bind {model.excludeLocalResult}, constraints : gbc(gridx: 1, gridy : 6))
+//            label(text : "Show Hash Searches In Monitor", constraints: gbc(gridx:0, gridy:7))
+//            showSearchHashesCheckbox = checkBox(selected : bind {model.showSearchHashes}, constraints : gbc(gridx: 1, gridy: 7))
+        }
+        buttonsPanel = builder.panel {
+            gridBagLayout()
             button(text : "Save", constraints : gbc(gridx : 1, gridy: 2), saveAction)
             button(text : "Cancel", constraints : gbc(gridx : 2, gridy: 2), cancelAction)
         }
     }
     
     void mvcGroupInit(Map<String,String> args) {
-        d.getContentPane().add(p)
+        def tabbedPane = new JTabbedPane()
+        tabbedPane.addTab("MuWire", p)
+        tabbedPane.addTab("I2P", i)
+        tabbedPane.addTab("GUI", u)
+                
+        JPanel panel = new JPanel()
+        panel.setLayout(new BorderLayout())
+        panel.add(tabbedPane, BorderLayout.CENTER)
+        panel.add(buttonsPanel, BorderLayout.SOUTH)
+                
+        d.getContentPane().add(panel)
         d.pack()
         d.setLocationRelativeTo(mainFrame)
         d.setDefaultCloseOperation(JDialog.DISPOSE_ON_CLOSE)

gui/griffon-app/views/com/muwire/gui/SearchTabView.groovy

@@ -4,14 +4,26 @@ import griffon.core.artifact.GriffonView
 import griffon.core.mvc.MVCGroup
 import griffon.inject.MVCMember
 import griffon.metadata.ArtifactProviderFor
+import net.i2p.data.Base64
 import net.i2p.data.DataHelper
 
+import javax.swing.JComponent
 import javax.swing.JLabel
+import javax.swing.JMenuItem
+import javax.swing.JPopupMenu
+import javax.swing.JTable
 import javax.swing.ListSelectionModel
 import javax.swing.SwingConstants
 import javax.swing.table.DefaultTableCellRenderer
 
+import com.muwire.core.util.DataUtil
+
 import java.awt.BorderLayout
+import java.awt.Color
+import java.awt.Toolkit
+import java.awt.datatransfer.StringSelection
+import java.awt.event.MouseAdapter
+import java.awt.event.MouseEvent
 
 import javax.annotation.Nonnull
 
@@ -26,19 +38,21 @@ class SearchTabView {
     def parent
     def searchTerms
     def resultsTable
+    def lastSortEvent
     
     void initUI() {
         builder.with {
             def resultsTable
             def pane = scrollPane {
-                resultsTable = table(id : "results-table") {
+                resultsTable = table(id : "results-table", autoCreateRowSorter : true) {
                     tableModel(list: model.results) {
-                        closureColumn(header: "Name", preferredWidth: 350, type: String, read : {row -> row.name})
+                        closureColumn(header: "Name", preferredWidth: 350, type: String, read : {row -> row.name.replace('<','_')})
-                        closureColumn(header: "Size", preferredWidth: 50, type: String, read : {row -> DataHelper.formatSize2Decimal(row.size, false)+"B"})
+                        closureColumn(header: "Size", preferredWidth: 20, type: Long, read : {row -> row.size})
-                        closureColumn(header: "Sources", preferredWidth: 10, type : Integer, read : { row -> model.hashBucket[row.infohash].size()})
+                        closureColumn(header: "Direct Sources", preferredWidth: 50, type : Integer, read : { row -> model.hashBucket[row.infohash].size()})
+                        closureColumn(header: "Possible Sources", preferredWidth : 50, type : Integer, read : {row -> model.sourcesBucket[row.infohash].size()})
                         closureColumn(header: "Sender", preferredWidth: 170, type: String, read : {row -> row.sender.getHumanReadableName()})
                         closureColumn(header: "Trust", preferredWidth: 50, type: String, read : {row ->
-                          model.core.trustService.getLevel(row.sender.destination)
+                          model.core.trustService.getLevel(row.sender.destination).toString()
                         })
                     }
                 }
@@ -53,7 +67,13 @@ class SearchTabView {
             def selectionModel = resultsTable.getSelectionModel()
             selectionModel.setSelectionMode(ListSelectionModel.SINGLE_SELECTION)
             selectionModel.addListSelectionListener( {
-                mvcGroup.parentGroup.model.searchButtonsEnabled = true
+                int row = resultsTable.getSelectedRow()
+                if (row < 0)
+                    return
+                if (lastSortEvent != null)
+                    row = resultsTable.rowSorter.convertRowIndexToModel(row)
+                mvcGroup.parentGroup.model.trustButtonsEnabled = true
+                mvcGroup.parentGroup.model.downloadActionEnabled = mvcGroup.parentGroup.model.canDownload(model.results[row].infohash)
             })
         }
     }
@@ -62,7 +82,7 @@ class SearchTabView {
         searchTerms = args["search-terms"]
         parent = mvcGroup.parentGroup.view.builder.getVariable("result-tabs")
         parent.addTab(searchTerms, pane)
-        int index = parent.indexOfTab(searchTerms)
+        int index = parent.indexOfComponent(pane)
         parent.setSelectedIndex(index)
         
         def tabPanel
@@ -84,12 +104,60 @@ class SearchTabView {
         resultsTable.columnModel.getColumn(1).setCellRenderer(centerRenderer)
         resultsTable.setDefaultRenderer(Integer.class,centerRenderer)
         resultsTable.columnModel.getColumn(4).setCellRenderer(centerRenderer)
+        
+        resultsTable.columnModel.getColumn(1).setCellRenderer(new SizeRenderer())
+        
+        
+        resultsTable.rowSorter.addRowSorterListener({ evt -> lastSortEvent = evt})
+        resultsTable.rowSorter.setSortsOnUpdates(true)
+        
+        
+        resultsTable.addMouseListener(new MouseAdapter() {
+            @Override
+            public void mouseClicked(MouseEvent e) {
+                if (e.button == MouseEvent.BUTTON3)
+                    showPopupMenu(e)
+                else if (e.button == MouseEvent.BUTTON1 && e.clickCount == 2)
+                    mvcGroup.parentGroup.controller.download()
+            }
+            @Override
+            public void mouseReleased(MouseEvent e) {
+                if (e.button == MouseEvent.BUTTON3)
+                    showPopupMenu(e)
+            }
+        })
     }
     
     def closeTab = {
         int index = parent.indexOfTab(searchTerms)
         parent.removeTabAt(index)
-        mvcGroup.parentGroup.model.searchButtonsEnabled = false
+        mvcGroup.parentGroup.model.trustButtonsEnabled = false
+        mvcGroup.parentGroup.model.downloadActionEnabled = false
         mvcGroup.destroy()
     }
+    
+    def showPopupMenu(MouseEvent e) {
+        JPopupMenu menu = new JPopupMenu()
+        if (mvcGroup.parentGroup.model.downloadActionEnabled) {
+            JMenuItem download = new JMenuItem("Download")
+            download.addActionListener({mvcGroup.parentGroup.controller.download()})
+            menu.add(download)
+        }
+        JMenuItem copyHashToClipboard = new JMenuItem("Copy hash to clipboard")
+        copyHashToClipboard.addActionListener({mvcGroup.view.copyHashToClipboard()})
+        menu.add(copyHashToClipboard)
+        menu.show(e.getComponent(), e.getX(), e.getY())
+    }
+    
+    def copyHashToClipboard() {
+        int selected = resultsTable.getSelectedRow()
+        if (selected < 0)
+            return
+        if (lastSortEvent != null)
+            selected = resultsTable.rowSorter.convertRowIndexToModel(selected)
+        String hash = Base64.encode(model.results[selected].infohash.getRoot())
+        StringSelection selection = new StringSelection(hash)
+        def clipboard = Toolkit.getDefaultToolkit().getSystemClipboard()
+        clipboard.setContents(selection, null)
+    }
 }

gui/src/main/groovy/com/muwire/gui/SizeRenderer.groovy

@@ -0,0 +1,29 @@
+package com.muwire.gui
+
+import javax.swing.JComponent
+import javax.swing.JLabel
+import javax.swing.JTable
+import javax.swing.table.DefaultTableCellRenderer
+
+import net.i2p.data.DataHelper
+
+class SizeRenderer extends DefaultTableCellRenderer {
+    SizeRenderer() {
+        setHorizontalAlignment(JLabel.CENTER)
+    }
+    @Override
+    JComponent getTableCellRendererComponent(JTable table, Object value,
+        boolean isSelected, boolean hasFocus, int row, int column) {
+        Long l = (Long) value
+        String formatted = DataHelper.formatSize2Decimal(l, false)+"B"
+        setText(formatted)
+        if (isSelected) {
+            setForeground(table.getSelectionForeground())
+            setBackground(table.getSelectionBackground())
+        } else {
+            setForeground(table.getForeground())
+            setBackground(table.getBackground())
+        }
+        this
+    }
+}

gui/src/main/groovy/com/muwire/gui/UISettings.groovy

@@ -0,0 +1,37 @@
+package com.muwire.gui
+
+class UISettings {
+    
+    String lnf
+    boolean showMonitor
+    String font
+    boolean clearCancelledDownloads
+    boolean clearFinishedDownloads
+    boolean excludeLocalResult
+    boolean showSearchHashes
+    
+    UISettings(Properties props) {
+        lnf = props.getProperty("lnf", "system")
+        showMonitor = Boolean.parseBoolean(props.getProperty("showMonitor", "false"))
+        font = props.getProperty("font",null)
+        clearCancelledDownloads = Boolean.parseBoolean(props.getProperty("clearCancelledDownloads","true"))
+        clearFinishedDownloads = Boolean.parseBoolean(props.getProperty("clearFinishedDownloads","false"))
+        excludeLocalResult = Boolean.parseBoolean(props.getProperty("excludeLocalResult","true"))
+        showSearchHashes = Boolean.parseBoolean(props.getProperty("showSearchHashes","true"))
+    }
+    
+    void write(OutputStream out) throws IOException {
+        Properties props = new Properties()
+        props.setProperty("lnf", lnf)
+        props.setProperty("showMonitor", String.valueOf(showMonitor))
+        props.setProperty("clearCancelledDownloads", String.valueOf(clearCancelledDownloads))
+        props.setProperty("clearFinishedDownloads", String.valueOf(clearFinishedDownloads))
+        props.setProperty("excludeLocalResult", String.valueOf(excludeLocalResult))
+        props.setProperty("showSearchHashes", String.valueOf(showSearchHashes))
+        if (font != null)
+            props.setProperty("font", font)
+            
+            
+        props.store(out, "UI Properties")
+    }
+}

host-cache/build.gradle

@@ -1,2 +1,3 @@
 apply plugin : 'application'
 mainClassName = 'com.muwire.hostcache.HostCache'
+applicationDefaultJvmArgs = ['-Djava.util.logging.config.file=logging.properties']

host-cache/src/main/groovy/com/muwire/hostcache/Crawler.groovy

@@ -1,9 +1,12 @@
 package com.muwire.hostcache
 
+import java.util.logging.Level
 import java.util.stream.Collectors
 
+import groovy.util.logging.Log
 import net.i2p.data.Destination
 
+@Log
 class Crawler {
 
     final def pinger
@@ -22,12 +25,14 @@ class Crawler {
     
     synchronized def handleCrawlerPong(pong, Destination source) {
         if (!inFlight.containsKey(source)) {
+            log.info("response from host that hasn't been crawled")
             return
         }
         Host host = inFlight.remove(source)
         
         if (pong.uuid == null || pong.leafSlots == null || pong.peerSlots == null || pong.peers == null) {
             hostPool.fail(host)
+            log.info("invalid crawler pong")
             return
         }
         
@@ -40,6 +45,7 @@ class Crawler {
         }
         
         if (!uuid.equals(currentUUID)) {
+            log.info("uuid mismatch")
             hostPool.fail(host)
             return
         }
@@ -50,7 +56,9 @@ class Crawler {
         def peers
         try {
             peers = pong.peers.stream().map({b64 -> new Destination(b64)}).collect(Collectors.toSet())
+            log.info("received ${peers.size()} peers")
         } catch (Exception e) {
+            log.log(Level.WARNING,"couldn't parse peers", e)
             hostPool.fail(host)
             return
         }

host-cache/src/main/groovy/com/muwire/hostcache/HostCache.groovy

@@ -1,9 +1,11 @@
 package com.muwire.hostcache
 
+import java.util.logging.Level
 import java.util.stream.Collectors
 
 import groovy.json.JsonOutput
 import groovy.json.JsonSlurper
+import groovy.util.logging.Log
 import net.i2p.client.I2PClientFactory
 import net.i2p.client.I2PSession
 import net.i2p.client.I2PSessionMuxedListener
@@ -12,6 +14,7 @@ import net.i2p.client.datagram.I2PDatagramMaker
 import net.i2p.util.SystemVersion
 import net.i2p.data.*
 
+@Log
 public class HostCache {
 
     public static void main(String[] args) {
@@ -53,7 +56,7 @@ public class HostCache {
         myDest = session.getMyDestination()
         
 		// initialize hostpool and crawler
-		HostPool hostPool = new HostPool(3, 60 * 1000 * 1000)
+		HostPool hostPool = new HostPool(3, 60 * 60 * 1000)
 		Pinger pinger = new Pinger(session)
 		Crawler crawler = new Crawler(pinger, hostPool, 5)
 		
@@ -64,7 +67,7 @@ public class HostCache {
         session.addMuxedSessionListener(new Listener(hostPool: hostPool, toReturn: 2, crawler: crawler), 
             I2PSession.PROTO_DATAGRAM, I2PSession.PORT_ANY)
         session.connect()
-        println "INFO: connected, going to sleep"
+        log.info("connected, going to sleep")
         Thread.sleep(Integer.MAX_VALUE)
         
     }
@@ -77,16 +80,16 @@ public class HostCache {
         
         void reportAbuse(I2PSession sesison, int severity) {}
         void disconnected(I2PSession session) {
-            println "ERROR: session disconnected, exiting"
+            log.severe("session disconnected, exiting")
             System.exit(1)
         }
         void errorOccurred(I2PSession session, String message, Throwable error) {
-            println "ERROR: ${message} ${error}"
+            log.warning("${message} ${error}")
         }
         void messageAvailable(I2PSession session, int msgId, long size, int proto,
             int fromport, int toport) {
             if (proto != I2PSession.PROTO_DATAGRAM) {
-                println "WARN: received unexpected protocol ${proto}"
+                log.warning("received unexpected protocol ${proto}")
                 return
             }
 
@@ -95,19 +98,19 @@ public class HostCache {
             try {
                 dissector.loadI2PDatagram(payload)
                 def sender = dissector.getSender()
-                println "INFO: Received something from ${sender.toBase32()}"
+                def b32 = sender.toBase32()
                 
                 payload = dissector.getPayload()
                 payload = json.parse(payload)
                 if (payload.type == null) {
-                    println "WARN: type field missing"
+                    log.warning("type field missing from $b32")
                     return
                 }
                 switch(payload.type) {
-                    case "Ping" : 
+                    case "Ping" :
-                    println "Ping"
+                    log.info("ping from $b32") 
 					if (payload.leaf == null) {
-						println "WARN: ping didn't specify if leaf"
+						log.warning("ping didn't specify if leaf from $b32")
 						return
 					}
 					payload.leaf = Boolean.parseBoolean(payload.leaf.toString())
@@ -116,14 +119,14 @@ public class HostCache {
 					respond(session, sender, payload)
                     break
                     case "CrawlerPong":
-                    println "CrawlerPong"
+                    log.info("CrawlerPong from $b32")
 					crawler.handleCrawlerPong(payload, sender)
                     break
                     default:
-                    println "WARN: Unexpected message type ${payload.type}, dropping"
+                    log.warning("Unexpected message type ${payload.type}, dropping from $b32")
                 }
             } catch (Exception dfe) {
-                println "WARN: invalid datagram ${dfe}"
+                log.log(Level.WARNING,"invalid datagram", dfe)
             } 
         }
         void messageAvailable(I2PSession session, int msgId, long size) {

logging/1_logging.properties

@@ -0,0 +1,61 @@
+############################################################
+#  	Default Logging Configuration File
+#
+# You can use a different file by specifying a filename
+# with the java.util.logging.config.file system property.  
+# For example java -Djava.util.logging.config.file=myfile
+############################################################
+
+############################################################
+#  	Global properties
+############################################################
+
+# "handlers" specifies a comma separated list of log Handler 
+# classes.  These handlers will be installed during VM startup.
+# Note that these classes must be on the system classpath.
+# By default we only configure a ConsoleHandler, which will only
+# show messages at the INFO and above levels.
+handlers= java.util.logging.FileHandler
+
+# To also add the FileHandler, use the following line instead.
+#handlers= java.util.logging.FileHandler, java.util.logging.ConsoleHandler
+
+# Default global logging level.
+# This specifies which kinds of events are logged across
+# all loggers.  For any given facility this global level
+# can be overriden by a facility specific level
+# Note that the ConsoleHandler also has a separate level
+# setting to limit messages printed to the console.
+.level= SEVERE
+
+############################################################
+# Handler specific properties.
+# Describes specific configuration info for Handlers.
+############################################################
+
+# default file output is in user's home directory.
+java.util.logging.FileHandler.pattern = MuWire.log
+java.util.logging.FileHandler.limit = 50000000
+java.util.logging.FileHandler.count = 1
+java.util.logging.FileHandler.formatter = java.util.logging.SimpleFormatter
+
+# Limit the message that are printed on the console to INFO and above.
+java.util.logging.ConsoleHandler.level = INFO
+java.util.logging.ConsoleHandler.formatter = java.util.logging.SimpleFormatter
+
+# Example to customize the SimpleFormatter output format 
+# to print one-line log message like this:
+#     <level>: <log message> [<date/time>]
+#
+#java.util.logging.SimpleFormatter.format=%4$s: %5$s [%1$tc]%n
+
+java.util.logging.SimpleFormatter.format=%1$tY-%1$tm-%1$td %1$tH:%1$tM:%1$tS.%1$tL %4$s %2$s %5$s %6$s %n
+
+############################################################
+# Facility specific properties.
+# Provides extra control for each logger.
+############################################################
+
+# For example, set the com.xyz.foo logger to only log SEVERE
+# messages:
+com.xyz.foo.level = SEVERE

logging/2_logging.properties

@@ -0,0 +1,61 @@
+############################################################
+#  	Default Logging Configuration File
+#
+# You can use a different file by specifying a filename
+# with the java.util.logging.config.file system property.  
+# For example java -Djava.util.logging.config.file=myfile
+############################################################
+
+############################################################
+#  	Global properties
+############################################################
+
+# "handlers" specifies a comma separated list of log Handler 
+# classes.  These handlers will be installed during VM startup.
+# Note that these classes must be on the system classpath.
+# By default we only configure a ConsoleHandler, which will only
+# show messages at the INFO and above levels.
+handlers= java.util.logging.FileHandler
+
+# To also add the FileHandler, use the following line instead.
+#handlers= java.util.logging.FileHandler, java.util.logging.ConsoleHandler
+
+# Default global logging level.
+# This specifies which kinds of events are logged across
+# all loggers.  For any given facility this global level
+# can be overriden by a facility specific level
+# Note that the ConsoleHandler also has a separate level
+# setting to limit messages printed to the console.
+.level= WARNING
+
+############################################################
+# Handler specific properties.
+# Describes specific configuration info for Handlers.
+############################################################
+
+# default file output is in user's home directory.
+java.util.logging.FileHandler.pattern = MuWire.log
+java.util.logging.FileHandler.limit = 50000000
+java.util.logging.FileHandler.count = 1
+java.util.logging.FileHandler.formatter = java.util.logging.SimpleFormatter
+
+# Limit the message that are printed on the console to INFO and above.
+java.util.logging.ConsoleHandler.level = INFO
+java.util.logging.ConsoleHandler.formatter = java.util.logging.SimpleFormatter
+
+# Example to customize the SimpleFormatter output format 
+# to print one-line log message like this:
+#     <level>: <log message> [<date/time>]
+#
+#java.util.logging.SimpleFormatter.format=%4$s: %5$s [%1$tc]%n
+
+java.util.logging.SimpleFormatter.format=%1$tY-%1$tm-%1$td %1$tH:%1$tM:%1$tS.%1$tL %4$s %2$s %5$s %6$s %n
+
+############################################################
+# Facility specific properties.
+# Provides extra control for each logger.
+############################################################
+
+# For example, set the com.xyz.foo logger to only log SEVERE
+# messages:
+com.xyz.foo.level = SEVERE