We have been using EPiServer’s blob storage for saving large chunks of JSON data to disk (please see Storing your own data in EPiServer’s blob store). As these JSON blobs takes a relatively long time to write, chances are the writing is interrupted causing broken JSON blobs.
This is especially prone to happen in develop environments where a process may be interrupted suddenly. If you add the aspect where several developer uses the same file share, all doing writes to the file, the chances increases.
To avoid having this behavior in our production environment, a fault tolerant way of writing data to disk seemed appropriate. This is by no means 100% bullet proof, but it is considerably more tolerant than just writing to disk.
Rather than just applying this for our own file management, here is a solution switching out EPiServer‘s FileBlobProvider to the fault tolerant version.
Fault tolerant FileBlobProvider for EPiServer blob storage
First of all, in order to make EPiServer use our fault tolerant blob provider as the default provider, we can register it in the episerver.framework portion of our web.config file.
<episerver.framework>
<blob defaultProvider="faultTolerantProvider">
<providers>
<add name="faultTolerantProvider" type="MyNamespace.FaultTolerantFileBlobProvider, MyAssembly" />
It may also be registered as default provider using initializable modules.
What we need to override in EPiServer’s implementation of the FileBlobProvider is not much. Only the GetBlob method and the Delete method, as below.
The main difference is that instead of using EPiServer FileBlobs, our code will be using a new class called FaultTolerantFileBlob.
public class FaultTolerantFileBlobProvider : FileBlobProvider
{
public override Blob GetBlob(Uri id)
{
string orgPath = OriginalPathFrom(id);
return new FaultTolerantFileBlob(id, orgPath);
}
public override void Delete(Uri id)
{
base.Delete(id);
if (id.Segments.Length == 3)
{
string orgPath = OriginalPathFrom(id);
string tmpPath = FaultTolerantFileBlob.TmpPathFor(orgPath);
File.Delete(tmpPath);
string bakPath = FaultTolerantFileBlob.BakPathFor(orgPath);
File.Delete(bakPath);
}
}
private string OriginalPathFrom(Uri id)
{
return System.IO.Path.Combine(this.Path, id.AbsolutePath.Substring(1));
}
}
The Delete method will be doing exactly the same as EPiServer’s implementation, with the addition that we will also be deleting any occurrences of temporary files and backup files.
So in addition to EPiServer’s original file, we will be using one with the extension .tmp, and one with extension .bak. The FaultTolerantFileBlob will be responsible for these paths.
The two methods that we need to override in EPiServer’s FileBlob are OpenRead and OpenWrite. Again, the difference here is that we will not be opening a FileStream, but rather our new FaultTolerantFileStream. To this, we will be supplying paths to the original file, the temporary file as well as the backup file.
public class FaultTolerantFileBlob : FileBlob
{
private readonly string _tmpPath;
private readonly string _bakPath;
public FaultTolerantFileBlob(Uri id, string filePath) : base(id, filePath)
{
_tmpPath = TmpPathFor(filePath);
_bakPath = BakPathFor(filePath);
}
public override Stream OpenRead()
{
return FaultTolerantFileStream.OpenRead(FilePath, _bakPath);
}
public override Stream OpenWrite()
{
DirectoryInfo directory = new DirectoryInfo(Path.GetDirectoryName(FilePath));
if (!directory.Exists)
{
directory.Create();
}
return FaultTolerantFileStream.CreateOpenWrite(FilePath, _tmpPath, _bakPath);
}
internal static string TmpPathFor(string path)
{
return string.Concat(path, ".tmp");
}
internal static string BakPathFor(string path)
{
return string.Concat(path, ".bak");
}
}
As you can see above, we are using static creation methods to retrieve instances of the FaultTolerantFileStream. This is the only way of creating this object, since the constructor is made private.
In the OpenRead creation method, there is a safety check to ensure that we do in fact have an original file to read from. It this is missing but we have a backup file, we will rename the backup and use it as the original (here there is a slight window where a conflict may occur with the rename dance described for the writing below).
public class FaultTolerantFileStream : Stream
{
private readonly ILogger _logger;
private readonly string _orgPath;
private readonly string _tmpPath;
private readonly string _bakPath;
private FileStream _fileStream;
private bool _disposed = false;
private bool IsOpenWriteStream { get { return _tmpPath != null; } }
private FaultTolerantFileStream(string originalPath, string temporaryPath, string backupPath, FileStream fileStream)
{
_orgPath = originalPath;
_tmpPath = temporaryPath;
_bakPath = backupPath;
_fileStream = fileStream;
_logger = LogManager.GetLogger(this.GetType());
}
public static FaultTolerantFileStream OpenRead(string originalPath, string backupPath)
{
if (!File.Exists(originalPath) && File.Exists(backupPath))
{
File.Move(backupPath, originalPath);
}
LogManager.GetLogger(typeof(FaultTolerantFileStream))
.Debug("Read original file: Creating new {0} for orgPath='{1}' tmpPath=null bakPath='{2}'",
nameof(FaultTolerantFileStream), originalPath, backupPath);
FileStream readStream = new FileStream(originalPath, FileMode.Open, FileAccess.Read, FileShare.Read);
return new FaultTolerantFileStream(originalPath, null, backupPath, readStream);
}
public static FaultTolerantFileStream CreateOpenWrite(string originalPath, string temporaryPath, string backupPath)
{
LogManager.GetLogger(typeof(FaultTolerantFileStream))
.Debug("Write temporary file: Creating new {0} for orgPath='{1}' tmpPath='{2}' bakPath='{3}'",
nameof(FaultTolerantFileStream), originalPath, temporaryPath, backupPath);
FileStream writeStream = new FileStream(temporaryPath, FileMode.Create, FileAccess.Write, FileShare.None);
return new FaultTolerantFileStream(originalPath, temporaryPath, backupPath, writeStream);
}
While the FaultTolerantFileStream inherits the abstract Stream class, it is actually creating and working a backing FileStream object. Because of this, we will be passing along most of the methods and properties to this backing stream.
public override bool CanRead => _fileStream.CanRead;
public override bool CanSeek => _fileStream.CanSeek;
public override bool CanWrite => _fileStream.CanWrite;
public override long Length => _fileStream.Length;
public override long Position
{
get => _fileStream.Position;
set => _fileStream.Position = value;
}
public override void Flush()
{
_fileStream.Flush();
}
public override int Read(byte[] buffer, int offset, int count)
{
return _fileStream.Read(buffer, offset, count);
}
public override long Seek(long offset, SeekOrigin origin)
{
return _fileStream.Seek(offset, origin);
}
public override void SetLength(long value)
{
_fileStream.SetLength(value);
}
public override void Write(byte[] buffer, int offset, int count)
{
_fileStream.Write(buffer, offset, count);
}
The only really interesting part is the Dispose method. This is were the fault tolerance is added. The thought is that when the system requests the FaultTolerantStream for writing, one is supplied as usual, but when the stream is disposed we engage in a renaming dance as seen below.
protected override void Dispose(bool disposing)
{
if (_disposed || !disposing)
{
return;
}
try
{
_logger.Debug("Closing underlying (IsOpenWriteStream='{0}') file stream '{1}'", IsOpenWriteStream, _fileStream.Name);
_fileStream.Flush(flushToDisk: true);
_fileStream.Close();
if (!IsOpenWriteStream)
{
return;
}
MoveReplace(_orgPath, _bakPath);
MoveReplace(_tmpPath, _orgPath);
File.Delete(_bakPath);
}
catch(Exception ex)
{
_logger.Error($"Rename dance error in {nameof(FaultTolerantFileStream)} for orgPath='{_orgPath}' tmpPath='{_tmpPath}' bakPath='{_bakPath}'", ex);
throw;
}
finally
{
_fileStream = null;
_disposed = true;
}
File.Delete(_tmpPath);
}
The first think that we want to do in our Dispose override is to flush and close the backing FileStream ensuring that it is in fact written to disk, if there is anything there. Note that the backing stream is written to the temporary file. If the stream we are closing was only open for read, there is no need to go any further.
After this, we create a new backup file from the old original (i.e. renaming the original file and adding the .bak suffix). Then we do the same with the temporary file to the original file (i.e. removing the .tmp suffix).
Finally, we remove the backup file in order to avoid having duplicates of files.
If anything goes wrong, it is logged and the exception rethrown. Whatever happens, the backing _fileStream reference is discarded and the stream marked as disposed. Should everything go as planned, we also delete the temporary file at the end.
Since System.IO.File.Move does not have support for overwriting files if the target name already exist, we first need to delete the target before attempting the rename.
private void MoveReplace(string source, string target)
{
for (int i = 0; i < 3; i++)
{
try
{
_logger.Debug("Deleting file '{0}'", target);
File.Delete(target);
_logger.Debug("Moving file '{0}' to '{1}'", source, target);
File.Move(source, target);
break;
}
catch (Exception ex)
{
_logger.Warning($"Failed try '{i}' MoveReplace source='{source}' target='{target}' with exception", ex);
Thread.Sleep(500);
}
}
}
}
In the above method, this is attempted 3 times in order to account for things like locks on files and so on.
