Is there a way to make the MoveNext() part of foreach loop retry the operation when applied to result of Azure Storage Library ListBlobs()?

Question

I have code using Azure Storage Library that goes like this:

CloudBlobContainer container = obtainContainer();

var blobList = container.ListBlobs(options:OptionsWithLinearRetry);
foreach (var blobItem in blobList) // << exception happens here sometimes
{
     //process item
}

and it works okay most of the time. But sometimes something goes wrong with the network and then I get:

Microsoft.WindowsAzure.Storage.StorageException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host

and the stack is like this:

// lots of Azure Storage Library stuff here
 at Microsoft.WindowsAzure.Storage.Blob.CloudBlobContainer.<>c__DisplayClass13.<ListBlobs>b__12(IContinuationToken token)
at Microsoft.WindowsAzure.Storage.Core.Util.General.<LazyEnumerable>d__0`1.MoveNext()
// my code with foreach line here

and so it looks like the foreach causes a MoveNext() call and the latter fails when obtaining the new chunk of data.

Can I somehow cause that MoveNext() call to be retried? Is there a way to have a "try-catch-retry" logic in foreach loop?

Answer 1

No matter how retry in next milliseconds, it will throw same exception, until it's totally closed. because your storage account is hitting the thresholds/limit which is specified by Windows Azure Storage: https://blogs.msdn.microsoft.com/windowsazurestorage/2012/11/04/cross-post-windows-azures-flat-network-storage-and-2012-scalability-targets/

Actually not only Azure, also Google services and AWS are same, all of them have thresholds on remote calls.

The MSDN cloudblobcontainer.listblobs says:

Returns an enumerable collection of the blobs in the container that are retrieved lazily.

you can setup

1. BlobRequestOptions.RetryPolicy in next 0-30 seconds.
2. BlobRequestOptions.AbsorbConditionalErrorsOnRetry
3. BlobRequestOptions.MaximumExecutionTime

You detect retry by the events:

1. OperationContext.GlobalRetrying
2. OperationContext.Retrying

You could also try from Linq:

1. .ToList()
2. .Where(x=>x.Conditions).ToList() filter useless data in server side.
3. .Take(10) to page your result, because no one need all results in 1 page.