tq: retry batch failures

If the batch operation fails due to an error instead of a bad HTTP
status code, we'll abort the batch operation and retry.  This appears to
be a regression from 1412d6e ("Don't fail if we lack objects the server
has", 2019-04-30), which caused us to handle errors differently.

Since there are two error returns from enqueueAndCollectRetriesFor,
let's wrap the batch error case as a retriable error and not abort if we
find a retriable error later on.  This lets us continue to abort if we
get a missing object, which should be fatal, but retry in the more
common network failure case.

Author: bk2204

tq/transfer_queue.go

@@ -442,7 +442,7 @@ func (q *TransferQueue) collectBatches() {
 		// don't process further batches.  Abort the wait queue so that
 		// we don't deadlock waiting for objects to complete when they
 		// never will.
-		if err != nil {
+		if err != nil && !errors.IsRetriableError(err) {
 			q.wait.Abort()
 			break
 		}
@@ -538,7 +538,7 @@ func (q *TransferQueue) enqueueAndCollectRetriesFor(batch batch) (batch, error)
 				}
 			}
 
-			return next, err
+			return next, errors.NewRetriableError(err)
 		}
 	}