How to upload large size file/blob to azure storage using ASP.Net,C#

Tool to upload local files into azure storage.

We have already demonstrated an application to upload files into azure storage using simple C# application in one of our previous post. Using that application we can upload local folder files into azure storage very easily and we can see that error log for each file uploading process. That is working fine for up to some size of the file (we are checking with 1 GB ) it is working fine. But when we are trying to upload larger than 3 GB files using same codes in one of our live project, got error in the middle of uploading files into azure storage. Then we are found out another solution for uploading very large size file into azure storage blob.

How can upload large size files into azure storage using C#, ASP.Net

When we are trying to upload very large size local files into azure storage blob, we are getting error related to timeout. To resolve this issue, we need to split up our local large size files into different small packages, then upload and after the upload successfully re pack the file again. This parallel file upload can be achieved by following code. We can upload very huge files to azure blob from a web page in ASP.Net,C# by using following code.

public void ParallelDownloadToFile(CloudBlockBlob blob,
string fileName, int maxBlockSize)
{
try
{
// refresh the values
blob.FetchAttributes();
long fileSize = blob.Attributes.Properties.Length;
var filePath = Path.GetDirectoryName(fileName);
var fileNameWithoutPath = Path.GetFileNameWithoutExtension(fileName);

// let's figure out how big the file is here
long leftToRead = fileSize;
int startPosition = 0;

// have 1 block for every maxBlockSize bytes plus 1 for the remainder
var blockCount =
((int)Math.Floor((double)(fileSize / maxBlockSize))) + 1;

// setup the control array
BlockTransferDetail[] transferDetails =
new BlockTransferDetail[blockCount];

// create an array of block keys
string[] blockKeys = new string[blockCount];
var blockIds = new List<string>();

// populate the control array...
for (int j = 0; j < transferDetails.Length; j++)
{
int toRead = (int)(maxBlockSize < leftToRead ?
maxBlockSize :
leftToRead);

string blockId = Path.Combine(filePath,
string.Format("{0}_{1}.dat",
fileNameWithoutPath,
j.ToString("00000000000")));

if (startPosition < 0)
startPosition = startPosition * -1;
if (toRead < 0)
toRead = toRead * -1;
transferDetails[j] = new BlockTransferDetail()
{
StartPosition = startPosition,
BytesToRead = toRead,
BlockId = blockId
};

if (toRead > 0)
{
blockIds.Add(blockId);
}

// increment the starting position
startPosition += toRead;
leftToRead -= toRead;
}

// now we do a || download of the file.
var result = Parallel.For(0, transferDetails.Length, j =>
{
// get the blob as a stream
try
{
using (BlobStream stream = blob.OpenRead())
{
Thread.Sleep(10000);
stream.Seek(transferDetails[j].StartPosition, SeekOrigin.Begin);

// setup a buffer with the proper size
byte[] buff = new byte[transferDetails[j].BytesToRead];

// read into the buffer
stream.Read(buff, 0, transferDetails[j].BytesToRead);

using (Stream fileStream = new FileStream(transferDetails[j].BlockId,
    FileMode.Create, FileAccess.Write, FileShare.None))
{
using (BinaryWriter bw = new BinaryWriter(fileStream))
{
bw.Write(buff);
bw.Close();
}
}
buff = null;
}
}
catch (Exception ex)
{
throw;
}
});

// assemble the file into one now...
using (Stream fileStream = new FileStream(fileName,
FileMode.Append, FileAccess.Write, FileShare.None))
{
using (BinaryWriter bw = new BinaryWriter(fileStream))
{
// loop through each of the files on the disk
for (int j = 0; j < transferDetails.Length; j++)
{
// read them into the file (append)
bw.Write(File.ReadAllBytes(transferDetails[j].BlockId));

// and then delete them
File.Delete(transferDetails[j].BlockId);
}
}
}

transferDetails = null;
}
catch (Exception ex)
{
throw;
}
}

public static class BlobExtensions
{
static readonly string DFDriveEnvVarName = "AZURE_DRIVE_DEV_PATH";
static readonly string containername =
RoleEnvironment.GetConfigurationSettingValue("Container")
.ToLowerInvariant();

public static bool Exists(this CloudBlob blob, string folderName)
{
try
{
//test doesnt work in df
if (RoleEnvironment.DeploymentId.ToLowerInvariant().StartsWith("deployment("))
{
string path = Environment.GetEnvironmentVariable(DFDriveEnvVarName);
path += "\\devstoreaccount1\\";
path += containername;
path += "\\";
path += folderName;
//path += blob.Uri.Segments.Last();
if (Directory.Exists(path))
return true;
else
return false;
}
else
{
blob.FetchAttributes();
return true;
}

}
catch (StorageClientException e)
{
if (e.ErrorCode == StorageErrorCode.ResourceNotFound)
{
return false;
}
else
{
throw;
}
}
}
}

By using the above code we can easily upload very large size files from local folder to azure blob storage. What the function does is, splitting the file stream into different byte packets and start uploading these small pieces of files into blob storage so there is no timeout issue generated. The above function is working well and we are implemented in one of our project also. We can uploaded large size files (Up to larger than 2GB checked, shall working for heavy large file also) into azure blob storage in our ASP.Net mvc application.